We ship the same gigabytes to the same machines on every job and wait ~18 minutes before work starts; the tokens we actually pay for are already cached, we just can’t see it.
We ship the same gigabytes to the same machines over and over, and we wait for it before any work starts. The fix is boring and every CI system on earth already does it. Meanwhile the thing we actually pay for — model tokens — turns out to be already well cached by the providers; our gap there is that we can't see it in our own receipts.
Every time we hand a job to a build machine, we send the project's entire history as one file, and the machine unpacks a fresh copy. We do this again for the next job, and the next.
The trip home already works properly — finished work comes back as only the new part. We learned this lesson once, in one direction, after an incident. The outbound half never got it.
git bundle create "$BUNDLE_LOCAL" "$SEND_REF" over the whole ref (box-sealed.sh:416), then a full clone on the box. Return: ${BASE_SHA}..${ref}, thin, added after a peer's stranded-custody incident. The asymmetry is the whole finding.| Repository | Shipped per job | Wait before work starts | With a warm copy |
|---|---|---|---|
| warp itself | 4.8 MB | ~2 seconds | 112 KB / instant [M] |
| mid-size project | 290 MB, six copies visible on one box | ~2 min 12 s | seconds [E] |
| the large lane repo | 1.7 GB | ~18 min 14 s before the engine began | ~1 min [E] |
One box is currently holding 20 job folders, 18 GB [M] — mostly the same history copied again per job, on a 59 GB disk.
This is also why the other lane spent tonight fighting shallow clones: when transfer hurts, people invent their own caching. Theirs produced a file that certified itself complete and then failed to unpack. That is the failure mode this plan has to avoid being.
Each box keeps one copy of the project's history and every job sends only the commits that copy is missing.
The part that matters for us: we don't have to write the freshness check. A partial package names the commits it depends on. If the box's copy doesn't have them, git refuses out loud and we fall back to sending everything. The failure direction is "slow," never "wrong" — which is the property today's whole day was about.
All collapse to the same case: missing prerequisites → send everything. No divergence logic to write, so none to get wrong.
After setting up, the box mechanically confirms the code it has is the exact commit we asked for. A computer can re-verify that, so it's allowed to run on the box.
Mirrors are cache and nothing else — listed, per project, deletable at any moment by the reaper without losing work.
git bundle verify and then fails to clone; the door now refuses shallow sources at the cause) and giving boxes GitHub access (breaks the credential posture — boxes deliberately hold no tokens). Bonus: job folders can use a worktree off the mirror instead of a bundle + full clone each, so the 18 GB mostly evaporates with it.I expected to find a gap here and did not. Provider-side caching is on and doing enormous work, measured from real transcripts rather than documentation:
| Run (all real, today) | Calls | Served from cache | Fresh |
|---|---|---|---|
| This session | 630 | 179.2 M tokens | 1,253 |
| Largest helper agent | 86 | 9.66 M | 172 |
| Box planning run | 61 | 2.31 M | 62 |
Cached reads bill at roughly a tenth of fresh input, so that box run's ~2.4 M input cost about 0.4 M — a 5–6× saving we already get, with zero code of ours.
The actual gap is that we can't see it. Codex prints no token counts in our logs; grok carries no token fields at all; and the one lane paying real dollars per token leaves no usage receipt on disk. So the build here is measurement, not a cache.
The background dispatch service ran last week's version for hours while everything else was current, because its startup file names a version-specific path and nothing re-points it when we publish. Nobody noticed until the other lane read the process list.
Standing services sit outside every freshness check we have. The move is small: the existing delivery watcher gets a fourth question — for each standing service, does the version baked into its startup file match what's published? A mismatch is RED with both numbers shown.
Re-pointing automatically on publish is the deeper fix. The watcher comes first on purpose: make it visible before you cure it, so we can tell whether the cure worked.
It touches the dispatch trust path, so it would get the full treatment — written intent, an independent adversary trying to break it, and a proof that the fallback actually fires when the mirror is deleted mid-run. That last part is non-negotiable: a fallback nobody has watched fail is a gate that cannot fail.
Making cached-vs-fresh tokens visible is measurement only, near-zero risk. The question is scope and sequencing, not whether.
Nothing else on this page needs you. Items 3 and 4 below are additive and I'd fold them into normal work.
| # | Work | Saves | Risk |
|---|---|---|---|
| 1 | Warm mirror + partial sends, with fallback and an on-box identity check | ~18 min per job on the big repo [M baseline]; ~16 GB of box disk [E] | Low — git proves freshness; fallback fails closed |
| 2 | Cached-vs-fresh token counts in receipts | Nothing directly — makes the largest spend lever measurable | Near zero — measurement only |
| 3 | Standing-service version seam in the delivery watcher | The next stale-service incident | Near zero — additive |
| 4 | Prompt-cache protection doctrine + seat scheduling affinity | ~27 k tokens per run [M], plus insurance on the 5–6× already banked | Zero code |
Forbidden. Verification is recomputed, never remembered. A cached pass is just a pass you stopped checking.
Shipping it fresh is the trust mechanism. A cached copy on the box is the stale-service failure moved to the trust layer.
An install through a symlink once emptied the shared store for every worktree at once. The prize is about two seconds. No.
A sealed run inheriting another run's context is a contaminated court. Independence beats the discount, always.
Whether the codex, grok and pi lanes cache at all is unmeasured — and today unmeasurable, because their receipts carry no token fields. That is exactly why item 2 exists. I also could not probe a cold start of a sealed run from inside this sandbox (the nested tool can't reach the keychain); one authenticated probe would close it.