when should an agent need human approval to ship software?

Putting a human in every loop makes the agent wait for permission to do work that is already isolated and reversible. Removing the human from every loop gives the agent standing authority over customers, money, access, and state.

Both designs waste the main advantage of agents.

The useful boundary is consequence. Let the agent move quickly while it can create evidence without causing durable harm. Stop it when the next action changes who is affected, what can be reached, what will be spent, or what cannot be recovered.

human-in-the-loop is too vague

“Keep a human in the loop” sounds safe because it does not say where the human appears, what they review, or what the approval authorizes.

A person clicking approve on every pull request may still miss the destructive operation hidden inside one of them. A person approving a broad agent session may authorize ten later actions they never saw. A person reviewing code may not be the person who understands the customer, security, budget, or data consequence of applying it.

The first question should be:

What becomes possible after this approval that was not possible before it?

If the answer is only “the agent may run another test in an isolated preview,” the gate is probably too early. If the answer is “the agent may delete the shared database,” the gate is probably carrying real judgment.

Google Cloud's architecture guidance describes human intervention as a predefined checkpoint for tasks that require oversight, subjective judgment, or final approval for critical actions. The useful word is checkpoint. A checkpoint belongs at a transition in authority, not at an arbitrary interval in the agent's work.

do not gate authorship

An agent-written change is not inherently riskier than a human-written change. Humans write destructive migrations, expose administrative routes, and approve expensive infrastructure too.

Authorship is still valuable evidence. The platform should record who proposed the work, whether an agent executed it, and which human approved the consequential step. But “an agent touched this” is a poor risk classifier.

The same proposed operation should cross the same gate regardless of who typed the diff:

• adding a preview-only worker remains isolated and recoverable;
• exposing an internal service to the public internet changes access;
• increasing a production fleet from two instances to two hundred changes spend;
• removing a managed database creates data-loss risk.

This keeps policy legible. Teams do not maintain one set of controls for humans and another for agents. They define which consequences require judgment.

four practical gates

Most software-delivery decisions fit into four boundaries. They overlap, but each names a different thing at stake.

1. launch

The launch gate asks whether the change becomes customer-visible.

An agent can usually build, deploy, and verify a branch in a production-shaped preview without asking permission. Those actions create evidence inside a disposable environment.

The boundary arrives when traffic, users, or a release channel moves to the proposed version. A human may need to decide whether:

• the product behavior is ready;
• the release timing is appropriate;
• support, sales, or customers need notice;
• current incidents or business events make the launch unsafe;
• the evidence answers the questions that matter for this feature.

Not every team needs a manual launch gate for every release. Mature tests, progressive delivery, and a bounded rollback path can allow low-risk production changes to move automatically. The rule should reflect customer consequence, not ceremony.

2. access

The access gate asks whether the change expands who or what can reach the system.

Examples include:

• making a private application public;
• exposing a new route or service to the internet;
• granting an agent access to secrets or sensitive logs;
• widening a role or service account;
• changing which users can read or modify a dataset;
• attaching a production credential to a new process.

Access changes are easy to hide inside otherwise routine work. A deploy can be technically healthy while its trust boundary is wrong.

The reviewer needs the before and after policy, not only a statement that “permissions changed.” The approval should name the principal, resource, action, environment, and duration. If the agent only needs temporary permission to apply one reviewed plan, it should not receive a reusable credential for the rest of the session.

3. spend

The spend gate asks whether the action creates a material or open-ended financial commitment.

Adding one small preview database may remain below the gate. Increasing production capacity, enabling a high-volume external API, removing a budget cap, or creating an unbounded fleet may cross it.

A useful plan shows:

• the current and proposed capacity;
• the expected unit and monthly cost;
• whether usage can grow without another decision;
• which budget or quota contains the action;
• how the commitment can be reduced or removed.

The threshold should be configurable. Ten dollars may matter to an individual builder. Ten thousand may be routine for a larger team. The invariant is that the agent cannot redefine the threshold in the same change that spends through it.

4. data risk

The data-risk gate asks whether the action can destroy, expose, rewrite, or irreversibly transform state.

This includes obvious operations such as dropping a database. It also includes narrower changes whose impact is easy to underestimate:

• dropping a table or column;
• removing a managed stateful service;
• running a destructive or unbounded backfill;
• narrowing a type or adding a constraint that rejects historical rows;
• changing retention or deletion behavior;
• applying a migration that old code cannot survive after rollback;
• sending an irreversible command to an external system.

The agent migration test loop should run automatically against preview-owned state. The human boundary appears when preview evidence is about to authorize a durable transition against shared state.

Data loss is different from recoverable operational change. A platform may let teams configure whether restarts, rollbacks, or replacement of stateless resources need review. It should not allow the proposed change to disable the floor protecting irreversible state.

review the action, not just the code

Code review answers whether the proposed diff is understandable and appears correct. Operational approval answers whether a particular action should happen now, against this target, with these consequences.

They can happen in the same pull request, but they are not the same decision.

Consider a reviewed configuration change that removes a database declaration. The diff may be perfectly correct. The product may genuinely no longer need the database. Applying that diff still deserves a separate answer to:

• Which database will be affected?
• Which environments and customers depend on it?
• What data will be lost?
• Is there a backup or recovery path?
• Can the removal be rescinded before terminal deletion?
• What evidence shows the application no longer uses it?

The approval object should therefore contain the evaluated plan, target, risk class, evidence, proposer, approver, and expiry. “This pull request has one approval” is too weak a contract for a high-impact action.

the plan should be the permission

Broad standing credentials make approval ambiguous. If an agent already holds the authority to modify production, the human prompt is a social convention rather than an enforcement boundary.

A stronger sequence is:

1. the agent begins with read access and isolated execution;
2. it gathers evidence and proposes a concrete plan;
3. the platform classifies the plan by consequence;
4. a human reviews the exact actions and targets;
5. approval grants a short-lived capability scoped to that plan;
6. the platform applies the plan and records the result;
7. the agent returns to its lower-privilege posture.

Vercel Agent uses a current version of this model. It runs under its own identity, starts read-only, requests permission for a named plan, receives a short-lived capability for that plan, and returns to read-only afterward. Vercel calls this “the plan is the permission”. The important property is structural: model behavior cannot expand authority beyond what the platform granted.

The approval also needs drift protection. If the plan changes after the reviewer opens it, the old approval must no longer apply. The human should review the new plan rather than authorize it through a stale confirmation.

a gate needs an escape path

A paused action cannot wait forever without becoming operational debris.

Every gate should define:

• approve: apply exactly the reviewed plan;
• reject: terminate the proposed action and preserve the reason;
• supersede: let a corrected change retire the old request;
• expire: fail closed after a bounded time;
• re-evaluate: require a new decision when the plan or policy changes.

Self-approval should be prohibited where independent judgment is part of the control. An agent must never approve its own elevation. A human proposer should not approve their own high-impact operation when the policy promises two-party review.

GitHub environments support this shape for deployment jobs. Required reviewers can hold a job before it starts, environment secrets remain unavailable until protection rules pass, and teams can prevent the initiator from approving their own deployment. GitHub documents these as deployment protection rules.

The mechanism is less important than the invariant: the action cannot acquire its consequential authority until the required decision exists.

automate below the gate

The gate only works if teams can afford to keep it enabled. A policy that interrupts every agent action will be bypassed, weakened, or ignored.

The following work should usually proceed automatically when scoped to an isolated environment:

• reading code and non-sensitive runtime evidence;
• creating and updating a preview;
• running tests and migrations against preview-owned state;
• inspecting preview logs and diagnostics;
• resetting or deleting preview-owned resources;
• proposing a pull request and an infrastructure plan;
• retrying a failed, recoverable verification step.

Automation below the gate produces the evidence that makes the later human decision fast. The reviewer should receive a tested plan, not become the agent's interactive debugger.

This is the operating model for a software factory: machines handle the cheap, repeatable loop; humans own product judgment and consequential authority.

where floo fits

floo's guardrail model classifies infrastructure operations by consequence.

Additive, reversible work remains ungated. Recoverable operations have a configurable review posture by app and environment in the policy model. Data-loss operations form a non-overridable floor: they always require a human.

The current concrete gate covers removal of a floo-managed service through repository configuration. The pull request is evaluated before merge and receives a floo/infra-approval status with the exact gated plan. An org admin can approve or reject that plan in the dashboard. An API key cannot approve it, and the human who requested a deploy cannot approve their own deploy-time request.

If an unapproved change still reaches the deployment path, floo pauses before building or touching cloud resources. The deploy-time gate is the fail-closed backstop. A changed plan invalidates the old approval. A corrected push supersedes the paused request. An undecided request expires without running the destructive operation.

Today, the customer-facing editor for recoverable-operation policy is still being built. The server-owned policy model and resolver exist, while the platform default determines the posture when no explicit policy has been provisioned. The invariant is already fixed: routine additions do not enter a human queue, and data-loss operations cannot opt out of review.

This is a narrower promise than “humans approve everything.” It is also more useful. Agents can create previews, inspect runtime evidence, and test stateful changes without waiting. When the proposed work reaches a consequential boundary, floo turns the evaluated plan into a human decision.

The goal is not to slow agents down. It is to make their authority expand at an explicit point, for an explicit reason, and only for the action a human actually reviewed.