CI that shares agent boxes and yields to their owner
My CI runs on my agents' own boxes in memory-sized slots, and starts nothing new while a box's owner needs it.
In 30 seconds
- CI shares the boxes my agents work on, in a fixed number of slots sized by memory.
- The owner writes a one-line "I need the box" signal; while it is fresh, no new CI job starts there.
- Running jobs always finish, and a dead or stale signal never blocks CI.
Problem
On 2026-10-16, uncapped local test runs pushed a 24-core agent box to a load of 31 to 55, with 5 of 7 GB of swap in use. One run was killed after 2617 s; the same commit passed in 5 min 4 s elsewhere.
What I tried
| Attempt | What happened |
|---|---|
| Uncapped local test runs on agent boxes | Whichever run started first took the whole machine. |
| CI only on boxes quiet for 10 minutes | Chat turns kept idle boxes “busy”; narrowed to working lanes, later dropped on my dev box. |
| A dedicated CI machine | Priced on 2026-10-11 and 12; I chose to max out existing hardware first. |
| One of my laptops as a runner | Hung twice under CI load. Came back behind a power cap, then left CI on 2026-10-19. |
What worked
Slots sized by memory. My 128 GB dev box runs two CI jobs at once, 20 GB cap each. Memory sets the count: on 2026-10-11 and 12 one suite barely sped up from 8 to 24 cores, but thrashed and mostly failed at 12 GB.
One container, cap and cleanup per job. A runaway job can’t throttle its neighbour, and cleanup touches only its own slot; the old one killed every CI process on the box.
Half the agents’ CPU weight. Under contention the agents get twice CI’s share.
Admission. A job starts only if load is at most half the cores and free memory covers running jobs’ growth, one more slot and 2 GB.
The owner signal. On my dev box the owner is the agents’ local test queue. Every 10 s it writes one line: runs waiting or running, and the write time. Only a still-live process counts, so a crashed run can’t hold the box. While the line is fresh and nonzero, no new CI job starts; running ones finish within their 45-minute budget.
Fail open, with an alert. A missing or stale signal (over 60 s) means no yield, so a dead writer can’t stop all CI. A live run whose identity can’t be checked still counts; after 95 minutes my morning report flags it for a manual clear.
Numbers
| Load, two CI jobs, 24 cores (2026-10-18) | 11.8, one readingmeasured |
|---|---|
| Memory free, two CI jobs (2026-10-18) | 110.9 GB, one readingmeasured |
| Median CI queue wait (2026-10-24) | 1 smeasured |
| p95 load on my dev box, with CI vs without (2026-10-24) | 35.4 vs 10.9measured |
CI still raises load on a shared box; the p95 pair is that cost.
When it stops being true
Yield only holds back new jobs, so a slow running job can still sit on the owner’s box. A planned change stops a CI job after the owner waits 5 minutes, with a retry, relaxing the rule below. Another gates on CI’s own CPU use, as load lags.
Agent brief
You are helping me let CI use spare capacity on my machines without slowing each machine's owner.
First ask me:
- Which machines can run CI, with their memory and cores.
- Who owns each machine.
- My CI jobs' peak memory.
- How CI jobs are routed today.
Constraints:
- Never stop a running CI job; only hold back new ones.
- A missing or stale owner signal means no demand.
- One container, memory cap and cleanup per CI job, and a lower CPU share than the owner.
Steps:
- Size CI slots from spare memory, not cores.
- Have the owner's test queue write one line every 10 to 15 seconds: live runs waiting or running, and the write time.
- Skip a machine while its signal is fresh and nonzero, or memory lacks room for another slot.
- Alert when a signal holds CI off far longer than a normal run.
Done when:
- A queued CI job waits or goes elsewhere while the owner's test runs, and starts there once it ends.