Skip to content

· 22 min read · Engineering

Three Kubernetes Projects Shipped a Brake Pedal This Month. Why?

Three Kubernetes Projects Shipped a Brake Pedal This Month. Why?

Agent Sandbox keeps a warm pool of already-booted sandboxes so an agent’s request doesn’t wait on a cold start. A loop watches the pool and refills it as requests draw it down. On August 13, 2026, Agent Sandbox v0.5.5 taught that loop to refill more slowly, and the rest of this post is why it had to.

A burst of requests empties the pool at once. The empty pool triggers a refill the same size as the burst, so the loop fires a wall of pod-creation writes at the same API server the incoming requests are already hammering. That’s a thundering herd, and it’s one the platform built for itself: the mechanism meant to make requests fast makes them slow, right when load is highest. Under pressure, the refill loop had become the fastest-moving thing in the system.

Why does a bug this old show up in three different projects in one week of 2026? A herd of clients stampeding a shared resource is a problem as old as the very concept of a shared resource. It’s because two pressures that were merely uncomfortable a year ago are now the design constraint of our time.

The first is pod churn. Kubernetes was built around pods that live for hours or days; a Deployment’s controllers, its informers, and its API-server write path all assume the population turns over slowly. A sandbox breaks that assumption by design. It’s short-lived and single-use: a client claims one, an agent runs its untrusted code inside it, and then it’s thrown away rather than reset and handed to the next request. Discarding it is the isolation: nothing an agent leaves behind—filesystem changes, a half-cleaned process, state of any kind—survives into the next request. The only safe sandbox is a fresh one. That safety has a price the control plane pays: single-use means a brand-new pod per request instead of one long-lived pod serving many. Inference workers scaled per token of traffic and gang jobs resized mid-flight add their own turnover on top. These examples churn pods on a timescale the original Kubernetes control plane design never budgeted for, and its write path is the part that falls down first.

The second is hardware scarcity. Accelerators are contended and expensive enough that nobody can afford to let them sit idle, so every serious platform runs its capacity tight: warm pools instead of cold starts, gang quota instead of best-effort placement, and per-role autoscaling instead of a fixed fleet. Scarcity is what forces the aggressive elasticity in the first place.

Put the two together and the timing isn’t coincidence. Scarcity forces more elasticity; more elasticity means more control loops; and more loops running against a control plane already at its churn limit means the loops stop smoothing and start amplifying. It’s a thundering herd of its own making. A control loop you add to smooth a system out will, past some load, outrun everything it was meant to help. What it needs then is not more speed but a bound—a limit on how fast and how often it’s allowed to act. Modern AI infrastructure platforms need to scale up and down, but those actions require an accelerator and brakes. The brake pedals are here.

The pressure that makes teams reach for a brake arrived across the Kubernetes landscape at once; the exact timing is partly luck, and I’ll come back to how much. I watched three unrelated projects reach the same conclusion in the same three days. Agent Sandbox v0.5.5 shipped on August 13, Kueue v0.19.1 on August 12, and LeaderWorkerSet v0.10.0 on August 11. They share the same diagnosis of the problem but the three repairs aren’t the same. That’s most of what follows—along with the honest counter-case, because the same week also shipped features that add elasticity.

The Brake Ships Installed but Not Pressed

The warm pool is the cleanest case, so I’ll start there and take it slowly.

A warm pool is a supply of pre-created, already-booted resources kept idle on purpose so a request can skip the slow part: a cold start. Agent Sandbox is a Kubernetes orchestrator for agent sandboxes—it hands out isolated environments for coding agents to run in, delegating the low-level isolation to a runtime like gVisor or Kata rather than being a runtime itself. Its SandboxWarmPool keeps a target number of booted sandbox pods sitting there doing nothing. When a client issues a SandboxClaim, the controller adopts a warm sandbox and hands it over. The client skips the entire cold start path—image pull, pod schedule, runtime boot, readiness probe—and gets a usable sandbox in the time it takes to reassign ownership.

You keep idle capacity because cold start is the expensive part, and for an interactive agent the cold start is what the user feels. The pool trades standing cost for tail latency. That trade is the reason warm pools exist, and it’s a good trade right up until the moment it inverts.

Here’s the inversion, in the words of the merged pull request that fixes it:

When a burst of SandboxClaims adopts warm sandboxes, the SandboxWarmPool controller immediately fires the entire deficit at the API server through slowStartBatch (up to --sandbox-warm-pool-max-batch-size creates in one reconcile). Those replacement CREATEs compete with the claim burst’s adoption writes for API server budget, slowing down exactly the traffic the warm pool exists to accelerate.

A burst of claims drains the pool. An empty pool triggers a refill sized to the burst. That refill is a wall of pod-creation writes aimed at the same API server the adoption path needs at the same instant. The mechanism built to absorb a spike faithfully reproduces the spike one level down against the control plane. The shock absorber is also a shock amplifier.

The pull request even measures the ceiling it’s fighting. In a 300-claim benchmark the isolated per-pool refill topped out around 70 to 85 creates per second—roughly 85/s at the sandbox CREATE stage with API Priority and Fairness queueing the creates, and about 70/s at pod scheduling given the kube-scheduler’s default --kube-api-qps=50. An unshaped 300-deficit refill burst therefore monopolizes several seconds of write budget precisely when adoption needs it most.

flowchart TD
  burst["Claim burst arrives<br/>(N SandboxClaims)"] --> adopt["Controller adopts N<br/>warm sandboxes"]
  adopt --> drain["Pool drops N below target"]
  drain --> refill["slowStartBatch fires the<br/>entire N-deficit at the apiserver"]
  adopt -.->|"needs API write budget"| api[["API server<br/>~70–85 creates/s ceiling"]]
  refill -.->|"competes for the same budget"| api
  api --> slow["Adoption writes slow down—<br/>the traffic the pool exists to speed up"]
  slow -.->|"burst still draining"| refill
  classDef hot fill:#7f1d1d,stroke:#f87171,color:#fef2f2;
  class api,slow hot;

The fix, in v0.5.5, is two opt-in controller flags on the SandboxWarmPool controller. One controls when refill starts, the other decides how fast it flows:

  • --sandbox-warm-pool-replenish-delay (a duration, default 0) defers the start of refill after members leave the pool, so a claim burst gets API-server priority before the controller starts replacing anything. The hold re-arms while members keep dropping and clears when the pool is full again. Initial fill and scale-ups are never deferred. There’s a documented sharp edge: under continuous arrivals the hold re-arms indefinitely and the pool just drains, so sustained load wants the rate flag with a small delay, not a big delay on its own.
  • --sandbox-warm-pool-max-refill-rate (a float, creates per second per pool, default 0) shapes the flow with a per-pool token bucket sized to one second of creates—max(1, rate)—turning a full-deficit burst into a smooth stream. These tokens aren’t the tokens an inference server bills for; here a token is a rate-limiter permit, and one token buys the right to issue exactly one pod-creation write to the API server. The bucket refills at max-refill-rate tokens per second, so the flag caps how many CREATEs the controller may fire per second per pool. When the bucket runs dry the reconcile creates what it was granted and requeues for exactly when the next token accrues, so pacing doesn’t depend on watch events firing. A permit spent on a failed CREATE is deliberately not returned to the bucket, because a failed POST spends the same API-server write budget a successful one would—refund it and a run of failures could multiply writes past the ceiling the flag exists to hold.

The two compose cleanly: the delay defers the start, the rate shapes the flow once started. Both flags default to 0, and at 0 the behavior is byte-identical to the old controller. The deferral bookkeeping and the token bucket are bypassed entirely, no state even allocated, and refill stays an immediate full-deficit slowStartBatch. You opt into it when your claim rate crosses the per-pool refill ceiling, and until then you pay nothing, not even the memory for the counters.

There’s a new end-to-end metric, agent_sandbox_client_claim_startup_latency_ms, that measures the delay from the moment a client’s SDK initiates a request until the sandbox is fully ready. The SDK stamps the start time on a agents.x-k8s.io/client-first-requested-at annotation and the reconciler records the final latency on readiness. Pod-ready starts its clock when Kubernetes begins acting on an object. This metric starts it when the client asked—so it captures the queueing and adoption-contention time that the refill burst inflates and that a pod-centric metric renders invisible. Shipping it is the team saying, in the language of instrumentation, that pod-ready was never the number the user actually felt.

None of this appeared from nowhere. The prior release, v0.5.4, is a two-weeks-earlier campaign against the same API-server pressure from the other direction: HTTP/2 connection sharding to stop watch traffic from starving everything else (--separate-watch-connection, --api-connections), a ReplicaSet-style expectations tracker so an informer-cache lag can’t trick the pool into over-creating, and an API Priority and Fairness insulation overlay to protect critical controller traffic. v0.5.4 fought the symptom at the transport and cache layers. v0.5.5 went after the cause and shaped the refill loop itself. Two releases, one realization: the platform’s own elasticity loop was the fastest-moving thing under load, and it spent a month learning to slow it down.

Kueue Freezes the Field That Drives the Quota Math

This one is a fix in Kueue, the job queue and quota scheduler, and it’s the sharpest of the three because it reads like a security bug that wandered into a scheduler. Kueue doesn’t run your pods; it decides when a workload is allowed to start and reserves the quota it will consume. A LeaderWorkerSet is only the shape of the workload Kueue is managing here—the fix ships in Kueue’s own codebase and changes what Kueue permits you to do to a workload it has already admitted. The next section is a different project changing the LeaderWorkerSet API itself; here, LeaderWorkerSet is just the thing being managed.

First, the primitive. A distributed training or inference job is a gang: a set of pods—a leader plus workers, or a set of ranks—that only do useful work when all of them are running together. They all-reduce gradients, they shard a KV cache across ranks, they hold consecutive stages of a pipeline. Place six of eight and leave two unschedulable, and the six you placed sit there pinning accelerators they can’t use for anything. It’s worse than waste: two half-placed jobs can deadlock, each holding half the GPUs the other needs to make progress. Partial placement isn’t a smaller success. It’s a failure that also happens to be expensive.

Gang admission—all-or-nothing admission—is the answer. The scheduler reserves quota for the entire gang atomically and admits the job only when the whole reservation can be satisfied; otherwise it admits nothing and the job waits its turn. Kueue implements this as workload admission against a ClusterQueue’s quota, and a LeaderWorkerSet is one gang arrangement: a leader pod plus a fixed-size group of workers. The invariant underneath is short, and everything depends on it. A job must never consume more than the quota reserved for it at admission time. Break that and gang admission’s promise—that an admitted job’s accelerators are genuinely, exclusively yours—quietly stops being true.

v0.19.1 fixes a break of exactly that invariant. The release note is worth quoting in full:

LeaderWorkerSet: Fixed a quota bypass where raising spec.leaderWorkerTemplate.size on an already-admitted, Kueue-managed LeaderWorkerSet ran more pods per group than the reserved quota covered. spec.leaderWorkerTemplate.size is now immutable while the LeaderWorkerSet is managed by Kueue, behind the new LWSImmutableGroupSize feature gate (Beta, enabled by default). spec.replicas stays mutable.

Let’s decompose it. spec.leaderWorkerTemplate.size is the number of pods per group. That’s the gang’s size. Admission reserved quota for replicas × size pods. Kubernetes is declarative and endlessly editable, so it invites you to kubectl edit that number on a running job. Nothing recomputed the reservation when you did. The group grew, the pods grew with it, and the extra pods ran against quota that was never set aside for them. A straight bypass.

The LWSImmutableGroupSize gate closes it by making spec.leaderWorkerTemplate.size immutable while the LWS is Kueue-managed. Beta, and enabled by default. The field that drives the quota math can no longer drift out from under the reservation. And notice what stays mutable: spec.replicas, the count of whole groups. Adding groups is a normal elastic scale that goes back through admission, where Kueue re-evaluates quota for the new groups and reserves or rejects accordingly. It was only resizing an already-admitted group, mid-flight, that skipped the recomputation. So the fix freezes the dangerous knob and leaves the safe one live.

If resizing an admitted gang is something you actually want to do, this is how, from the same operator guidance:

If you change spec.leaderWorkerTemplate.size on a Kueue-managed LeaderWorkerSet, recreate it at the new size instead, or disable the LWSImmutableGroupSize feature gate to keep the previous behavior, which also restores the quota bypass.

The gate is the fix; turning it off gives you back the bug. The escape hatch exists—some clusters genuinely want in-place resize and will accept the risk. That’s the brake-pedal pattern again: the governor is a feature gate, the gate ships on, and the release notes are clear on the consequence of switching it off.

This is the same self-limiting loop as the warm pool. Elastic resize of a running gang is an autoscaling loop. Left unbounded, it lets a job outgrow the reservation gang admission made on its behalf and eat quota that was promised to somebody else. The repair bounds the loop. Freeze the input that drives the quota calculation rather than removing elastic scale, because replicas scaling stays alive and correct.

LeaderWorkerSet Gives Every Role an Autoscaler, Then a Floor Under It

Now a different project. The last section was a fix in Kueue that happened to govern a LeaderWorkerSet-shaped workload; this one ships in LeaderWorkerSet itself in v0.10.0. Previously the concern was quota an admitted job could bypass; here it’s two autoscalers that can fight each other over one number. Same family of failure, different project and different solution. It’s the most forward-looking of the three, and it needs some explanation, because the primitive underneath is newer than the other two.

An LLM inference request runs in two phases with genuinely different appetites for hardware. Prefill processes your entire input prompt at once to build the KV cache and produce the first output token. Because every prompt token is available up front, prefill is a big, highly parallel matrix multiply that saturates compute—it’s FLOP-bound, and it sets your time to first token. Decode then generates output tokens one at a time, and each step reads the whole KV cache to produce a single next token. That’s a small matrix-vector operation that moves a lot of memory for very little arithmetic, so decode is bound by memory bandwidth, not compute, and it sets your time per output token.

flowchart LR
  prompt["Input prompt<br/>(all tokens available)"] --> prefill["Prefill<br/>one parallel matrix multiply<br/>over the whole prompt"]
  prefill --> kv["KV cache<br/>built once, then read every step"]
  prefill --> ttft["First output token<br/>(time to first token)"]
  kv --> decode["Decode<br/>one token at a time,<br/>reads the whole KV cache each step"]
  decode -->|"loop per output token"| decode
  decode --> tpot["Each next token<br/>(time per output token)"]
  classDef compute fill:#1e3a5f,stroke:#60a5fa,color:#eff6ff;
  classDef memory fill:#3b2f1e,stroke:#fbbf24,color:#fffbeb;
  class prefill compute;
  class decode,kv memory;

This isn’t a subtle distinction. The Splitwise paper (Patel et al., ISCA ‘24) states it from its own characterization: there are “two main phases during an LLM inference request: a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics,” and it notes that token generation “do[es] not require the compute capability of the latest GPUs, and can be run with lower power and cost.” The LWS maintainers use the identical framing in KEP-849: prefill and decode “have different bottlenecks (compute-bound vs memory-bandwidth-bound).”

If you run both phases on the same GPU they interfere—prefill’s compute bursts stall decode’s steady drip, and you plan one pool of hardware around two incompatible profiles. Splitting them onto phase-appropriate machines and scaling each independently pays off, and the numbers are large. DistServe (Zhong et al., OSDI ‘24) reports that disaggregating prefill and decode onto different GPUs, then co-optimizing each phase’s allocation and parallelism, can “serve 7.4x more requests or 12.6x tighter SLO” than colocated serving under the same latency requirements. Splitwise reports phase-split clusters delivering “1.4x higher throughput at 20% lower cost” or, held to the same budget, “2.35x more throughput with the same cost and power budgets.”

Disaggregation is clearly the right design. It also creates the problem, because now there are two control loops. The correct ratio of prefill capacity to decode capacity depends on your live traffic and each role needs to scale to its own bottleneck. Long prompts load prefill, long generations load decode. Give each role its own autoscaler and you have exactly what you wanted but it’s dangerous: for any single role, its replica count is now written by two loops that never coordinate. The role’s own autoscaler sets it from live traffic, and a rolling update sets it while standing up a new revision. Nothing arbitrates between them, so when both act at once they can drive that one number in opposite directions.

LWS v0.10.0 ships that per-role autoscaling for the DisaggregatedSet (which orchestrates one LeaderWorkerSet per role, e.g. prefill and decode) via KEP-849. Set a role’s scaling.mode to External and the DisaggregatedSet controller automatically creates and manages a DisaggregatedSetRoleScaler for it—you don’t author the scaler, and its name is deterministic, <disaggregatedset>-<role>, so a serving stack named my-llm-serving gets my-llm-serving-prefill. The scaler exposes the standard /scale subresource, so an HPA, a KEDA ScaledObject, or any /scale-aware controller can drive it. Roles you don’t opt in behave exactly as they do today.

Here’s a minimal version of the config a reader could actually act on:

disaggregated-serving.yaml — opt one role into external autoscaling
apiVersion: leaderworkerset.x-k8s.io/v1alpha1
kind: DisaggregatedSet
metadata:
  name: my-llm-serving
spec:
  roles:
    - name: prefill
      scaling:
        mode: External # controller auto-creates my-llm-serving-prefill
    - name: decode
      scaling:
        mode: External # controller auto-creates my-llm-serving-decode
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: prefill-hpa
spec:
  scaleTargetRef: # point the HPA at the stable scaler, not the LWS
    apiVersion: leaderworkerset.x-k8s.io/v1alpha1
    kind: DisaggregatedSetRoleScaler
    name: my-llm-serving-prefill
  minReplicas: 2
  maxReplicas: 20
  metrics: [] # your TTFT-driven metric here

The scaler is a separate object—rather than pointing the HPA straight at the role’s LeaderWorkerSet—because of a real operational trap. LWS names embed a revision hash (<ds>-<slice>-<revision>-<role>), so an HPA created against a specific LWS name becomes an orphan the instant a rolling update produces a new revision with a new name. The scaler’s name is revision-independent, so the HPA or KEDA object survives arbitrary rollouts and keeps driving the role across every new revision. The autoscaler drives scaler.spec.replicas, and the DisaggregatedSet controller reads that value every reconcile and applies it to the underlying LWS. (The alpha ships with its own guardrails: the webhook warns on External with spec.replicas > 1 and outright rejects External with spec.slices > 1.)

Now the brake. An autoscaler and a rolling update are two control loops writing to the same replica count, and left uncoordinated they fight. An HPA scale-down that lands in the middle of a rollout could shrink the new revision the rollout is still trying to stand up—the deployment building the fleet up, the autoscaler tearing it down, at the same time, on the same number. KEP-849’s no-shrink guard stops it:

One safety guard: between planner iterations, the new-revision target is never allowed to fall below the current new-revision replica count. If an HPA scale-down arrives mid-rollout, its requested value is floored to what’s already in flight—the new-revision fleet stops growing but does not shrink. Once the rollout completes, the guard releases and the target tracks the scaler exactly.

The guard is a floor, not a freeze. During a rollout the autoscaler can still push the target up; it just can’t pull it down below what’s already running on the new revision. The moment the rollout finishes, the floor lifts and the autoscaler regains full authority. The maintainers are candid about why the guard is this narrow: tighter coupling—accelerating the old revision’s drain on a scale-down, accelerating new-revision growth on a spike—is deliberately out of scope for alpha, because both interact with the planner’s surge and unavailability budgets and “are potentially destabilising if not carefully bounded” and “could produce oscillation.” Couple two control loops carelessly and you get oscillation, so this version ships the bounded one for now.

flowchart TD
  hpa["HPA (TTFT-driven)<br/>wants fewer prefill replicas"] --> scaler["DisaggregatedSetRoleScaler<br/>scaler.spec.replicas"]
  rollout["Rolling update<br/>standing up new revision"] --> target(("New-revision<br/>replica target"))
  scaler --> guard{"no-shrink guard:<br/>floor at in-flight count"}
  guard --> target
  guard -.->|"scale-down floored<br/>during rollout"| blocked["Fleet stops growing<br/>but does NOT shrink"]
  target --> done["Rollout completes →<br/>guard releases →<br/>target tracks scaler exactly"]
  classDef hot fill:#7f1d1d,stroke:#f87171,color:#fef2f2;
  class guard,blocked hot;

Same self-limiting loop. Two elasticity loops with no shared authority will, under a rollout, drive the same number in opposite directions. The fix doesn’t remove either loop. It bounds one of them for exactly as long as the conflict can occur, then hands authority back.

You Ship Half a Design Until You Ship the Bound

Line the three fixes up and the pattern is clear: a loop built to make the system faster or more elastic had, past some load, become the mechanism amplifying load onto the controller beneath it. Warm-pool refill amplified a claim burst into an API-server burst. Elastic gang resize amplified a running job past its own reservation. Two per-role autoscalers amplified a rollout into an oscillation risk. The repairs take three different approaches—a delay plus a token bucket, an immutability gate, a no-shrink floor—but they share a design pattern: give the loop a brake, ship the brake on (or opt-in and clearly labeled), and don’t pretend the loop was ever safe unbounded.

This is not a new idea. What these releases implemented has names already. The closest modern framing is metastable failure (Bronson et al., HotOS ‘21): a system sitting in a vulnerable-but-not-overloaded state hits a trigger and drops into a state where goodput—the rate of useful work completed, as opposed to raw request throughput including the failed and retried ones—stays “unusably low” because of a “sustaining effect—often involving work amplification”—a feedback loop that keeps the failure going even after the trigger is gone. The key point for us: “the strength of many feedback loops is proportional to the scale.” Bronson means scale as system size—the sustaining effect spreading across shards, clusters, and datacenters. The three loops here are load-driven, so the same relationship holds against runtime demand: a sustaining loop whose strength rises with the thing driving it will, past a threshold, outrun the corrective response.

The lineage goes back further, to TCP congestion collapse and Jacobson’s AIMD in 1988. A control loop deliberately designed to damp rather than amplify, laid out in Van Jacobson’s “Congestion Avoidance and Control” (SIGCOMM ‘88). Forward to Google SRE’s cascading-failure canon, where the brakes have standard names: load shedding, graceful degradation, brownout. A token-bucket refill rate is load shedding on the refill loop. A no-shrink guard is a bounded controller. An immutable group-size gate is admission-time back-pressure on elastic growth. The three releases reinvented the classics in three new domains.

There’s a nuance, though. The canonical metastable sustaining effect is retries—a failure-recovery loop turning malignant. These three are provisioning loops turning malignant: warm-pool refill, gang resize, per-role autoscale. The amplified resource is API-server write budget or reserved quota, not backend queue depth. That’s a real extension of the pattern, not a textbook instance of it. As careful readers we should hold it as a variant of the retry-driven canon rather than the same mechanism.

Three releases in one week is a good story, and a good story is exactly when to look for the counter-evidence—because the strong reading, “the whole field synchronously turned toward brakes,” isn’t what the evidence supports. In the same window, Karpenter v1.14.0 added a CapacityBuffer API that deliberately keeps spare, over-provisioned node capacity ahead of demand so pods schedule instantly—a warm pool for nodes, which is more proactive elasticity, not a brake, and its v1.14.1 landed 2026-08-21, squarely in the same release week. Even Kueue’s own v0.19.1, the release that shipped the immutability gate, also shipped a cluster of ElasticJobsViaWorkloadSlices fixes that extend elastic scaling. Within a single “brake” release, the project was widening another elasticity path.

A maturing AI-infrastructure discipline is learning to put governors on its elasticity loops. Two pressures drive it—pod churn against a control plane built for slower turnover, and accelerator scarcity that rules out idle capacity—and both run field-wide, which is why three independent projects bounded three different loops in the same season. That much is a genuine signal about where the field’s attention has gone. It’s the maturation half of a two-sided motion, and both halves will continuously evolve.

You build elasticity, and then you learn to bound it, sometimes in the same release. So the shared pressure is causal and the shared week is partly the coincidence of independent release cadences lining up—and I’d rather hand you the version that survives contact with the counter-case than the version that reads better in a headline.

The usable form to take away isn’t “add fewer control loops.” It’s this: when you add a loop to smooth a system, you have shipped half a design. The other half is the bound—the delay, the token bucket, the immutable field, the floor—and the phase where you write it is not a sign the loop failed. It’s the sign the loop grew up. If your platform has an elasticity loop with no brake on it yet, that’s not a feature you finished. It’s a brake pedal you haven’t installed.

Share: