Human in the loop for AI agents: when the agent needs permission to change the plan

A business traveler needs to reach Berlin for a morning meeting. Their flight is canceled. An AI travel agent searches for replacements and finds an overnight train, a flight above the approved budget, and a cheaper flight that arrives after the meeting starts. It can book any of them, but none meets all the requirements it was given.

The traveler now faces a choice about sleep, spending, or the purpose of the trip. A company manager may also need to authorize the extra expense. Better search will not settle those questions if these are the available options. The people who own the requirements must decide which can change.

In a human-in-the-loop (HITL) workflow, the agent pauses for a person to make that decision. For an engineering lead building it, the design starts with what the person must decide and who has authority to decide it.

Each option changes a different requirement #

Before the agent acts, agree on what it may book without asking again. The traveler’s arrival deadline, permitted travel modes, and total spending limit define its delegated authority, the permission it has to act on their behalf. Cancellation rights and company policy further limit which changes it can make. A replacement inside those limits can proceed if booking has already been authorized.

Asking separately about a flight, hotel cancellation, and airport transfer leaves the traveler to reconstruct the trip. A flight can be affordable and still arrive too late. A hotel cancellation can save money and still leave them without a room they need. The approval screen should show the revised itinerary together, with the original requirement beside each proposed exception:

Each option needs departure and arrival times, total charges, and the effect on existing reservations. A label such as “recommended” needs an explanation of which requirement the recommendation prioritizes. If the agent favors the cheaper flight, the late arrival belongs beside the price, where it can affect the decision.

The workflow must distinguish an exception the right person can authorize from an action the system prohibits. The traveler’s willingness to take an overnight train cannot waive a company restriction on that travel mode. Part of the engineering work is identifying who can make each decision, instead of sending every question to whoever happens to be using the agent.

Repeated review can wear down attention #

There is a reasonable case for human oversight of every booking while a team learns what its agent gets wrong. A small pilot can give an experienced operator time to inspect each proposed change. We would keep that review until the team has evidence for a narrower scope of approval. Where the law requires a human decision, that decision stays in the workflow.

Repeated approval has limits, though. Anthropic, which sells access to AI models, tested human permission review with 1,053 paid testers. Partway through each coding session, it swapped one permission prompt for a clearly dangerous command, and reviewers caught it 13.6% of the time. They worked in a test environment, where approving the command caused no dangerous execution.

In the same experiment, detection fell from about 17% early in a session to about 5% after at least 50 preceding prompts. These are the vendor’s own findings about permission prompts in a coding study, rather than a measure of how often travelers will approve a bad itinerary in the real world. They give teams building AI agents a reason to test whether people still catch mistakes after repeated requests.

An approver needs evidence behind the proposal #

Even an occasional request needs more than a clear account of the options. The traveler needs the supplier’s cancellation conditions and the company’s applicable spending rule alongside the agent’s summary. If a fare condition is missing or contradictory, someone must resolve that uncertainty before purchase. Choosing to spend more does not establish whether a ticket is refundable.

Duolingo, which sells a language-learning app and the Duolingo English Test, tested how its exam proctors responded to fabricated cheating alerts. In a talk describing the experiment, Duolingo software engineer Angel Ortmann Lee reported that reviewers accepted half the false alerts despite strong calibration scores. The experiment used historical sessions and did not affect test takers.

The team then required reviewers to find independent evidence in the video before upholding a flag. Reported rejection of fabricated alerts rose from 50% to 71% across sequential studies. Reviewers also rejected a higher share of genuine alerts, so this result alone does not establish an overall accuracy improvement.

In the travel workflow, the traveler’s response should identify the requirement they are willing to change and the resulting itinerary. The budget owner’s response should record the additional spending they authorize. A general “looks good” attached to a conversation would leave the scope harder to reconstruct when something changes.

The booking must match the approved plan #

If the traveler selects the overnight train and that service is unavailable at booking time, the approved plan can no longer run as presented. Choosing another departure might change the arrival time or accommodation. The agent must return for a decision if the substitute falls outside the approved itinerary and any explicitly allowed variation.

In the Loopjacking preprint, researcher Adithyan Arun Kumar reproduced cases in selected agent products where the operation executed could differ from the one presented for approval. In some tested agentic AI systems, workflow data could change after approval, allowing a different operation to run. The findings concern the tested products and versions, without establishing how common the problem is.

Our engineering recommendation is to keep a fixed record of the approved itinerary, including the bookings to create or cancel, the permitted charges, and the approvers. Before each action, the booking service should compare the request with that record and check that the approval remains valid. An altered plan needs a new version. The same principle runs through our article on the agentic AI control plane: the rule that limits an action belongs in the service that executes it.

Each intended purchase also needs a unique identifier, or idempotency key, that stays the same on retries. The system receiving the request should use that key to recognize the same purchase and return its existing result or pending status instead of booking again.

On this trip, a replacement ticket and hotel change could each fit an individual spending threshold while exceeding the authorized total together. Rock Lambros of Zenity, which sells security tools for AI agents, illustrates the problem in a post on approval as a rubber stamp. Two agents that each see $100 in an account can each authorize a $60 withdrawal, and both checks pass. We would enforce the remaining budget in the service that commits the purchase, including amounts already committed or reserved for pending bookings.

Waiting and partial bookings need explicit rules #

Execution can also stop halfway through an approved itinerary. The train booking might succeed while the hotel change fails. The traveler needs to see in real time what has happened and what remains unresolved, while the agent attempts recovery within its existing authority. Keeping a record of completed steps lets the system resume without repeating purchases or presenting the original plan as untouched.

An unanswered approval needs a deadline. For this traveler, the deadline should reflect how long the proposed option remains available, and a backup approver must have the authority to decide.

If nobody can be reached, the agent should deny the pending changes by default, in line with the deny-by-default practice that Singapore’s Infocomm Media Development Authority recommends for when human supervisors are unreachable in its Model AI Governance Framework for Agentic AI. It should preserve existing reservations where possible and explain which proposed changes it has not made. A separately authorized fallback can proceed within its own limits. The product must make the cost of waiting visible before the deadline, while leaving the decision with the person entitled to make it.

Planted mistakes reveal whether review works #

An approval rate cannot show whether people understand these decisions. Frequent acceptance could mean that the agent proposes suitable itineraries, or that users miss the exceptions. The Financial Stability Board’s consultation report on AI adoption proposes investigating cases where people consistently approve agent recommendations without modification. A rejection rate also needs interpretation: someone may reject a technically correct plan because their preferences have changed.

We would test the workflow in an environment that cannot create bookings or send cancellation requests. Alongside valid plans, include an itinerary that arrives after the stated deadline, an itinerary with a missing cancellation fee, and a proposed purchase that differs from the approved one. Define the expected response for each case.

Reviewers should recognize the late arrival and obtain permission to change the deadline if they choose that option. They should investigate the missing fee before making a decision. The execution system should block the changed purchase.

Measure whether reviewers recognize each planted discrepancy and obtain any required authorization. Record mistakes about the itinerary separately from preference changes and authorized exceptions. Also test whether the agent routes exceptions to the right person in the first place, since an out-of-budget booking could bypass the review screen entirely. Include unanswered requests and partial bookings, so the exercise covers what happens after the user leaves the screen as well as what happens when they click.

Recorded decisions can refine the agent’s authority #

Recording the reason for a decision alongside the approved plan and eventual result lets the team distinguish a factual correction from a preference change or a budget exception. It also gives them something more useful than a rising approval rate when deciding what authority to grant next.

If travelers repeatedly choose an overnight train in the same circumstances, the team can offer to save that preference with explicit consent. If a manager repeatedly authorizes the same class of expense, the company can consider changing its policy. Until those permissions change, the agent should continue to ask. Evidence from the approval workflow can help the business make delegation more precise without silently turning yesterday’s exception into tomorrow’s authority.

Sources #

• Conner Phillippi and contributors, Anthropic. Auto mode is now the default in Claude Code for Pro, Max, and Team plans. August 7, 2026.
• Angel Ortmann Lee, Duolingo. Build AI Systems for Discernment, Not Approval. AI Engineer World’s Fair 2026 talk, San Francisco, June 29–July 2, 2026.
• Adithyan Arun Kumar. Loopjacking: Hijacking Human-in-the-Loop Approval. September 17, 2026. Preprint.
• Rock Lambros. Human in the Loop Is a Rubber Stamp. Its Replacements Are Too. September 29, 2026. Corrected October 3, 2026.
• Infocomm Media Development Authority. Model AI Governance Framework for Agentic AI. Version 1.5, published May 20, 2026. Updated June 5, 2026.
• Financial Stability Board. Sound Practices for Responsible Adoption of Artificial Intelligence (AI): Consultation Report. June 10, 2026.