The Trust Ladder

The interesting question about automated inspection is not whether a model can grade a module. It is how you hand it the authority to do so, and how you would know you had handed it over too early. That is a quality question, and we think it deserves a more careful answer than a go-live date.

Human-factors research has treated autonomy as a continuum rather than a switch for a long time. The canonical model describes automation as varying “across a continuum of levels, from the lowest level of fully manual performance to the highest level of full automation”^[1], building on a ten-point scale first proposed for undersea teleoperation^[2]. The gradations in that scale are strikingly close to what an inspection handover actually needs: at one level the system offers options but has no say in which is chosen; at a higher one it suggests a single alternative while the human retains authority to execute it; higher still, it acts unless a human vetoes within a set window^[1].

We are building toward four modes on that continuum. They are not milestones on a schedule. They are states with a gate between each one, and the gate is the point.

Observe

The system runs alongside the line and grades nothing. It sees what the inspection stations see, produces its outputs, and those outputs go nowhere near a disposition. The purpose is unglamorous: find out whether the imaging is stable, whether coverage is complete, whether the thing survives a shift. Nobody is deciding anything about quality yet, which is exactly why it is safe to be wrong here.

Shadow, which is the part that actually matters

In shadow, humans still grade every module and hold every disposition. The difference is that the model grades too, in parallel, and both records are kept. Where they disagree, a person adjudicates: which call was right, and why.

This pattern is established practice elsewhere. Clinical AI teams call it a silent trial, where model outputs are “produced in parallel to (and thus separate from) the standard of care; therefore, they do not influence clinicians”^[4]. A 2026 scoping review of that literature found something worth internalising before trusting any offline benchmark: “a performance drop is apparent when moving from retrospective to live evaluation, showing that models often perform less reliably during silent or prospective evaluation”^[4]. Those studies are clinical, and the specific causes they name are particular to health records. But the mechanism — a model meeting a distribution its training set did not describe — is not a medical phenomenon.

Shadow mode is not a rehearsal for autonomy. It is the instrument that tells you whether autonomy is warranted, and it is the only stage that manufactures ground truth as a by-product.

There is a second reason shadow deserves more respect than it usually gets, and it is uncomfortable. Human grading is not a clean label to measure a model against. In a controlled study of 82 professional inspectors examining precision manufactured parts, inspectors “correctly rejected 85% of defective items and incorrectly rejected 35% of acceptable parts”^[3]. Per-item agreement varied enormously: correct accepts for individual acceptable parts ranged from 17% to 95%, and detection of individual defective items ranged from 37% to 100%^[3]. Worse for anyone building a categorical classifier, correct rejections were “frequently achieved for the wrong reasons” — inspectors agreed a part was bad while spreading their explanation across five different defect categories^[3].

That population was deliberately conservative, which is why its false-alarm rate is so high, and we would not expect those exact numbers on a module line. The structural finding is what carries: a disagreement between a model and an inspector is not automatically the model’s error, and a single inspector’s label is not automatically truth. Adjudication is how you resolve that, and it is why the adjudication has to be a real decision by someone accountable for quality rather than a rubber stamp applied to whichever answer is more convenient.

Assist

In assist, the model grades first and a human confirms every single disposition. This sounds like a safe halfway house. It is the mode we are most wary of, because it is the one with a known failure mode: people asked to confirm machine output tend to “over-accept computer output ‘as a heuristic replacement of vigilant information seeking and processing’”^[5]. The systematic review that describes this splits it into errors of commission — following incorrect advice — and errors of omission, failing to act because nothing prompted you to^[5]. On an inspection line, omission is the dangerous half: nobody scrutinises the modules the model quietly passed. The same review notes the effect “occurs more often with task inexperienced users but can occur with more experienced users”^[5], which disposes of the obvious reassurance that senior people would not do this.

That literature is drawn from clinical decision support and aviation. We have not found an equivalent study in manufacturing visual inspection, and we are not going to pretend the transfer is measured rather than reasoned. It is a risk we are designing against rather than a rate we can quote: assist is a stage to move through deliberately, not a comfortable place to settle.

Autonomous

The model grades. Humans audit a sample, and they work the low-confidence queue — the calls the system itself flags as uncertain. Note what has not changed: a person still adjudicates every ambiguous case. The queue gets shorter and more interesting, not empty.

What governs a promotion

Each gate is a quality decision. Not an engineering decision, not a decision the people who built the model get to make on their own authority. The condition for moving from shadow to assist is an agreement rate that the quality function is willing to accept, on a defect distribution it recognises, and the same is true of every rung above it. If that sounds slow, it is worth remembering that the alternative is promoting a model on the strength of a benchmark that the silent-trial literature says will flatter it^[4].

A promotion is also reversible. A rung is a state you can be demoted from when the defect mix moves, when a material changes, or when the audit sample says something you did not expect.

What the work becomes

The inspection role does not disappear as the ladder is climbed; it moves. The work stops being look at this module and decide and becomes resolve the cases the system could not, and find out why they happened. Adjudication is a harder and more valuable job than grading, because every adjudication is simultaneously a disposition, a training label, and a pointer at a process problem upstream. The people who know the defects best stop spending their attention on the ninety-odd percent of modules that are obviously fine, and spend it on the ones that are genuinely ambiguous or genuinely wrong.

Automation research has been clear for a quarter century that this is what happens: automation “does not simply supplant human activity but rather changes it, often in ways unintended and unanticipated by the designers”^[1]. Designing the ladder is, in large part, deciding what you want that change to be before it happens to you.