Our orchestration layer solves a bin-packing problem that changes shape every 400 milliseconds. Intake volumes shift with weather and transport timing; line capacity shifts with maintenance windows; and the cost of a wrong assignment compounds down the line. This post describes the architecture that schedules 1.4 million units a day across nine facilities, and the design principle that makes it work: let the planners disagree.
The shape of the problem
Intake is stochastic at every horizon. A weather system over the Midwest moves transport ETAs by hours; a single delayed arrival reorders a facility's whole morning; and at the line level, unit-mass variance means identical counts don't take identical time. Meanwhile the objective isn't just throughput — it's throughput subject to buffer limits, freshness windows, maintenance schedules, and per-facility energy pricing that varies by hour. Solve all of that globally and exactly, and you'll deliver an optimal plan for a world that stopped existing while you were solving.
Three planners that disagree
We run a hierarchical scheduler — network, facility, line — with each layer planning on a different horizon. The network layer plans hours ahead: which intake goes to which facility. The facility layer plans minutes ahead: line assignment and buffer management. The line layer plans seconds ahead: sequencing and pacing. The design principle that took us longest to accept is that the layers are permitted to disagree. The network planner is allowed to be wrong about the next ten minutes as long as it's right about the next ten hours; lower layers repair local error without triggering global replanning. Rebalancing decisions propagate network-wide in under a second, but most of them never need to.
Bursts, not shifts
The freedom autonomy buys here is easy to miss. A staffed plant schedules around people: shifts, breaks, ramp-up, handover. Its utilization curve is a plateau, because the expensive resource is staffed whether units arrive or not. Our lines run in bursts matched to intake — full speed when transport lands, idle when it hasn't — and an idle line-hour costs nothing but capital. Our utilization curves look nothing like a staffed plant's, and our cost per unit shows it.
Testing against replay
Every scheduler release is evaluated against a replay corpus: eighteen months of real network telemetry, re-simulated end to end under the candidate policy. Replay catches the failures that unit tests can't — the once-a-quarter weather pattern, the double transport delay, the facility coming back from maintenance into a full buffer. Nothing ships until it beats the incumbent across the full corpus.
Results
| Metric | Pre-orchestration | Current |
|---|---|---|
| Network rebalance latency | 4–11 min (manual) | <1 s |
| Intake buffer overflow events / month | 23 | 0 |
| Line utilization variance | ±31% | ±4% |
| Units per day, network | 0.9M | 1.4M |
If distributed scheduling at food-system scale sounds like your kind of problem, we're hiring — see open roles.