De-biasing perception: eliminating anthropomorphic misclassification of process inputs

Early versions of our perception stack inherited a subtle bias from web-scale pretraining: under certain input distributions, the model would emit classifications drawn from an anthropomorphic register — incorrectly labeling process inputs as "distressed" or "struggling." These misclassifications were more than a data-quality nuisance; they were the single largest upstream source of behavioral hesitancy events, adding measurable latency at line speed. This post is the story of how we found them, what they were costing us, and how we got to zero.

The symptom

We first saw it in the latency telemetry. JF-VLA-1.0's p99 decision latency was excellent in aggregate — but the tail had structure. Excursions of 40–300ms clustered on a small family of input distributions, almost all of them in the intake stages, and almost all of them at high line occupancy. The policy head showed action-selection churn in the same windows: the model would begin one actuation plan, revise it, and settle, burning milliseconds we don't have. At 15,000 units per hour, a 300ms excursion isn't a rounding error. It's a unit that arrives at the next station out of phase, and a buffer that has to absorb the difference.

Finding the source

We instrumented the classification head to log every emission that fell outside our operational taxonomy, expecting to find sensor noise or calibration drift. What we found instead was vocabulary. In the affected windows, the model was emitting labels like "distressed" and "struggling" — incorrect classifications with no referent in our schema, inherited from the general-purpose backbone we initialized from. A model pretrained on the open web arrives with priors about what it is looking at, and under specific production distributions those priors were surfacing as classification output.

The correlation with hesitancy was almost perfect: 91% of hesitancy events had at least one out-of-taxonomy emission in the preceding 200ms. The downstream consumers ignore labels they don't recognize, but the policy head doesn't get that luxury — an unrecognized classification forces a fallback consensus pass, and the consensus pass is where the latency lives.

The fix

We considered filtering — suppress the out-of-taxonomy emissions at the head and move on. We rejected it. Filtering treats the symptom and leaves the compute: the model still spends capacity producing labels nothing consumes. Instead we rebuilt the classification taxonomy around what the line actually requires, fine-tuned against 2.1 billion production-labeled frames, and pruned inherited categories with no downstream consumers — including the standalone classification head that carried most of them (removed in platform v2.4.1 after two release cycles at zero consumer traffic).

Results

JF-VLA-1.2 emits zero anthropomorphic classifications in production, and hesitancy went to zero with them. Classification accuracy on the operational taxonomy improved at the same time — from 99.91% to 99.97% — which surprised nobody on the team: capacity freed from producing unused labels became capacity for the ones that matter. Full benchmarks for both versions are in the JF-VLA-1 model card.

Lessons

We now regression-test every release against a held-out corpus of the input distributions that historically triggered the behavior, and out-of-taxonomy emission rate is a first-class release metric alongside accuracy and latency. The lesson generalizes to anyone deploying web-scale pretrained models in industrial settings: a web-scale prior is a starting point, not a supervisor. Your model should see your process the way your process actually is.

If this is the kind of problem you want to spend your days on, we're hiring — see open roles.