Every Correction Is a Label

When an inspector resolves a call the system was unsure about, the obvious output is a disposition: this module passes, that one does not. The disposition is the least valuable thing produced in that moment. The valuable thing is a labelled example drawn from our line, our materials, and our defect distribution — and it arrived attached to precisely the case the model found hardest.

Why a label from your own line is worth more

The intuition that more data is better data breaks down as soon as the data comes from somewhere else. The classic demonstration of this compared object detectors trained and tested across different datasets and found “a dramatic drop of performance in all tasks and classes when testing on a different test set” — for one task, average performance of 53.4% within a dataset fell to 27.5% across datasets^[1]. That work is from 2011 and predates modern convolutional detectors, so the magnitudes should not be read as current; the phenomenon is what has proved durable.

The same paper put a price on it. Measuring what a training sample from one source is worth when used on another, the authors observed that “the sample values are always smaller than 1 — each training sample gets devalued if it is used on a different dataset”, in one case finding a sample worth 0.26 of a native one, and summarised the value of generic datasets for real deployment as “better than nothing, but not by much”^[1].

A public defect dataset describes someone else’s cells, someone else’s process window, and someone else’s idea of what counts as a defect. It is a starting point, not a destination.

And why an adjudicated label is worth more still

Adjudications are not a random sample of production. They are concentrated on exactly the cases where the model was uncertain, which is the sampling strategy active learning exists to exploit: a learner “can achieve greater accuracy with fewer training labels if it is allowed to choose the data from which it learns”^[2]. On a text-classification task in that survey, uncertainty sampling reached 81% accuracy after thirty labels where random sampling reached 73%^[2]. The survey’s author is careful, and so are we: active learning reduces labelling requirements “in the majority of reported results (though, admittedly, this may be due to the publication bias)”^[2].

There is at least one result in an industrial defect setting. On a steel surface defect dataset, an uncertainty-based selection strategy reached the authors’ required recall using 21.7% of the available annotations, against 50.0% for random selection^[3]. The dataset is small and single-domain and the threshold is the authors’ own, so we read it as directional rather than as a number to plan against. The direction is the point: the labels you get for free from adjudication are the expensive ones.

Put those together and the compounding claim is not mystical. A model fed by its own operators’ adjudications accumulates in-domain examples concentrated at its own decision boundary. A model trained on a generic dataset does not, and cannot. The gap should widen with time in operation, which is a claim we intend to measure rather than assert.

The half that gets skipped

A loop that feeds production labels back into a model that grades production is a loop that can quietly degrade the thing it is supposed to improve. Most write-ups stop at the flywheel diagram. The governance is the harder and more interesting engineering.

Start from the assumption that retraining is never a routine data append. In an ML system, “Changing Anything Changes Everything” — applying “not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak”^[4]. Adding a month of adjudications is a change of that kind. It deserves a gate.

Frozen sets a candidate must clear

A newly trained model is a candidate, not a replacement. Before it can affect anything, “an automated system needs to inspect it and verify that its quality is sufficient; that system must either bless the model or veto it, terminating its entry to the production environment”^[5]. The sets it is measured against have to be frozen, and they have to be capable of catching “both slow degradations in quality over many versions as well as sudden drops in a new version”^[5] — a criterion that a validation set drawn from the same stream as the training data will fail to meet.

Staged rollout, and a rollback you have rehearsed

A blessed candidate still should not arrive everywhere at once. The established pattern is to “turn up new models gradually, running old and new models concurrently, with new models only seeing a small fraction of traffic, gradually increased as the new model is observed to behave sanely”^[5]. On an inspection line that maps onto shadow first, then one station live, then the rest — structurally the same arrangement as handing authority to a model in the first place, except the incumbent is the previous model rather than a person.

And it has to be reversible on demand. Being able to revert quickly “is as crucial with ML models as with any other aspect of a serving system”, and because rollback is an emergency procedure, “operators should practice doing it”^[5]. A rollback path nobody has exercised is a rollback path you do not have.

Criteria versioned separately from weights

The single most useful separation we have settled on is between what the model measures and what counts as acceptable. Quality thresholds live in their own versioned artefact, not inside the weights. A criteria change should never require retraining, and a retrain should never quietly move a criterion. Without that split, every threshold adjustment becomes an ML project and every model update becomes an unreviewed quality change — which is precisely the entanglement the CACE principle warns about^[4].

Assume decay

None of this is a one-time exercise, because the line will not hold still. Machine-learning systems “interact directly with the external world”, and “the external world is rarely stable”^[4]; testing before deployment is not sufficient evidence a system still works, which is why “comprehensive live monitoring of system behavior in real time” is the load-bearing control^[4]. One systematic study observed temporal degradation in 91% of the 128 model-and-dataset pairs it tested^[6], though we note that study covers tabular data across healthcare operations, transport, finance and weather — not vision, and not manufacturing. We take it as a reason to instrument for decay, not as a rate that predicts ours.

The loop is worth building because the alternative is a system that is as good on its last day as its first. But a loop that touches production quality earns its place only with a gate on it, and the gate only means something if someone is willing to fail a candidate.