JGJonathan Gobielweb platform engineer
Back

human in the loop AI development

Human-in-the-Loop: Where Engineers Should Still Approve

human-in-the-loopgovernanceriskapprovals
4 min read

I want agents to do most of the repetitive work on our H5 frontend. I do not want them to merge payment copy changes at 2 a.m. without a named human accepting responsibility.

Human-in-the-loop is not nostalgia for manual typing. It is risk partitioning: agents explore the solution space fast; humans hold accountability where mistakes are expensive or irreversible.

The Failure Mode of "Full Autonomy" Demos

Demos optimize for wow. Production optimizes for rollback stories.

When autonomy runs without gates, you get:

  • silent dependency upgrades,
  • merged PRs that passed lint but violated product policy,
  • E2E greens on staging configs that do not match production,
  • architecture drift that compounds until a human rewrite is cheaper than iteration.

Loops still need a choke point where someone with context says yes.

My Default Approval Gates

These are non-delegable on our team unless explicitly waived with written reason:

GateWhy human
Merge to shared mainlineRelease accountability
Payment, balance, or PII UIRegulatory and trust
Auth, session, crypto changesSecurity blast radius
New third-party SDK or scriptSupply chain
Feature flags affecting prod %Blast radius control
Deleting or migrating user dataIrreversibility

Agents may prepare the diff, tests, and docs. They stop at the merge button.

Approvals That Can Be Policy, Not Meetings

Not every decision needs me. Automation handles:

  • formatting and import order,
  • bundle increases under budget,
  • non-blocking review warnings,
  • retry loops on deterministic test failures under cap,
  • preview deploy smoke.

The distinction is reversibility and blast radius. If revert is one commit and users see no money impact, policy can decide. If not, human.

How I Make Human Approval Fast Instead of Bottleneck

Slow gates kill loops. I optimize the human step to minutes:

Pre-digest packets. Before I open a PR, bots post: risk tags, bundle delta, suites run, override history on touched modules.

Small PRs by agent contract. Implementers must split work when scope crosses risk tags. I reject elephant PRs even if agents could technically produce them.

Explicit waivers. If infra is down and E2E cannot run, waiver includes outage ticket and time-bounded follow-up. No silent merges.

Pair only on ambiguity. If acceptance criteria conflict, agents stop and ping me. They do not guess product intent.

Frontend Examples Where I Stay Hands-On

Copy that implies fees, limits, or success states. LLMs paraphrase dangerously. I read strings.

Cross-platform navigation changes. Deep links, back gestures, and in-app browser stacks break easily. Agent tests help; product sense still matters.

Performance tradeoffs with UX impact. Lazy loading that saved ~79% on initial JS is worthless if the skeleton lies about content. I judge the tradeoff, not just the metric.

Code-splitting boundary moves. An 18% page bundle win is real. So is shipping the wrong chunk to first paint. I verify route-level impact, not only total kb.

Teaching Agents Where Not to Spend Tokens

I encode "ask human" triggers in AGENTS.md:

  • requirements contain "TBD" or conflicting bullets,
  • ticket touches more than N packages without architecture doc,
  • external API returns undocumented status codes,
  • legal or compliance label in diff without linked spec.

That reduces confident wrong merges more than any post-merge revert policy.

Trust Calibration Over Time

Early on I approved everything. Monthly I promote categories to policy when:

  • override rate < 5%,
  • incident count zero for three releases,
  • rollbacks trivial.

Demote back to human when incidents spike. Trust is a meter, not a badge.

FAQ

Is human-in-the-loop the same as manual QA?

No. Humans approve high-risk decisions. Agents and Maestro still run most tests.

How do you avoid becoming the router again?

Structured handoffs and pre-digest packets. You approve outcomes, not re-run every command.

What about on-call at night?

Agents can open PRs and run tests. They do not merge without on-call policy explicitly allowing it for defined hotfix classes.

Do customers care?

They care that money UI is correct. They do not care whether a bot opened the PR.

Closing

Automate everything whose failure mode is a revert. Keep your name on everything whose failure mode is a headline.

Write your gate list before you add the tenth agent. Then tune loops so human approval is rare, fast, and always intentional.