Obtainability, Part 1: The Other Half of Scarcity

You went to Spot VMs for the discount, up to 91 percent off on-demand, and that was the right call. It’s the reason anyone reaches for them, but they’re a little different. You went to the docs to understand how to use them. So you did the homework the docs told you to do. You wrote the checkpoint code. You set terminationGracePeriodSeconds. You triggered a preemption on purpose, in staging, and watched the job pick itself back up. You’re prepared for the inevitable preemption, when you lose the spot instance. You shipped it.

Then the job sat in Pending for four days across three regions, never got a single L4.

Nothing preempted. There was nothing to preempt. The capacity you asked for was never there to give, and none of the resilience you built for losing a machine does anything about not getting one in the first place. You can’t checkpoint your way out of a stockout. There’s nothing to checkpoint.

Call that second failure obtainability. The rest of this post is what follows from taking it seriously.

Scarcity Has Two Halves

The published advice about Spot VMs is correct. It’s also only half of the problem, and until recently it was the only half that mattered.

Interruption is the first half. The capacity you’ve got gets reclaimed. Spot is best-effort and preemptible at any time, so the machine under your workload can vanish on a few seconds’ notice. This half is old and well handled. Every Spot tutorial covers it, and the mechanisms are mature: checkpoints, drain handlers, idempotent work units, pod disruption budgets. Follow them and an interruption costs you a little wasted compute and nothing else. It’s a solved problem.

Obtainability is the second half. The capacity you want was never there to rent. You ask for four L4s in a region and the region has zero. Not “zero right now, try again in thirty seconds.” Zero for hours, sometimes zero for days. This half is new. It arrived with the accelerator crunch. The advice was written when the assumption underneath it still held, back when asking for a machine and getting one were the same event.

Hold the two side by side and the asymmetry is painful. Interruption is a loss you can recover from, because the machine existed. You had it, you lost it, you resume. Obtainability is an absence you can’t engineer around inside the workload, because the machine never existed for you to hold. Checkpointing answers the first and is mute on the second. So is your pod disruption budget. So are your three carefully chosen node pools, if all three point at zones that are dry.

For a CPU workload in a healthy region, none of this bites; capacity is reliably there and always has been. Obtainability is a GPU problem, an accelerator problem, a frontier-hardware problem. But that’s precisely the workload you turned to Spot to afford, because on-demand L4s and A100s can be a tough bill to swallow. The discount that brought you here and the scarcity that ambushed you are bolted to the same chip.

What a Four-Day Stockout Looks Like

I want to be specific about this, because the specifics changed how I think.

I was trying to run a single LoRA fine-tune of a small Gemma model on one L4. One GPU. Not a training cluster, not a fleet. One accelerator for one job, the kind of thing the pricing page makes look trivial. The capacity advisor I’d built, a small tool that queries Google’s capacity-advice API and ranks zones by how likely you are to get a machine, told me to go to us-east1-c, and it was confident. It read an obtainability score of 0.90 for that zone, and 0.90 for essentially every US zone with g2 machine classes.

There were no L4s in us-east1-c. There were none in the next region, or the one after that. The score said 0.90 and the autoscaler said no, over and over, for about thirty-six hours across three regions. The number predicting obtainability and reality didn’t agree.

Two details from that run stuck with me. First, while the whole region read as dry for the shape I wanted, a throwaway g2-standard-8 Spot VM found stock in a zone where the g2-standard-4 next to it didn’t. Same zone, same accelerator family, different machine size, opposite answer. In that run, the capacity-advice API’s scores were only meaningful for an exact (machine type, zone) pair at an exact minute; a region-wide reading didn’t tell me what a given shape could get. Second, the advice landscape shifted materially in about four hours. One region’s score collapsed from 0.90 to 0.10 while I watched, and waited, and watched some more. Whatever that 0.90 had been, it was not a fact about the present. It was a guess with a short shelf life, and I wanted to treat it as inventory.

The fine-tune eventually ran. It ran on a g2-standard-8 in us-central1-a, found not by a better obtainability score but by a live attempt. Actually trying to get a machine and seeing whether one came back. That distinction, between a prediction that a zone is probably fine and a confirmation that a machine was just allocated, turns out to be the entire series.

Designing for Obtainability (in Practice)

If you read Compute Scarcity Is Permanent. Build a Ladder., it argued for a pattern: an ordered set of substitution rungs a workload descends when its preferred compute is unobtainable, a selector that picks the highest rung you can actually get, and a promotion mechanism that climbs back when the good hardware frees. That pattern isn’t a thought experiment. Google ships most of it as product: GKE custom ComputeClasses give you the rungs as a declared fallback priority and the promotion mechanism as active migration back to a preferred config. Rungs and a climb-back, running in production.

The selector is where the gap is. The pattern said the selector “picks the highest obtainable rung,” then left the word obtainable undefined. Picks it how? By what signal does a selector know, at the moment it chooses, that a rung is obtainable rather than merely listed? The ComputeClass influences autoscaling but isn’t consulted by the scheduler; it will try a rung and fall to the next when the try fails, which is descent by failure, not selection by knowledge. The pattern piece named the missing function, a probe() that answers can I get this rung right now. Not in general, not on average, but for this exact machine type in this exact zone at this exact minute.

My four-day stockout fell into this gap. My selector picked a rung because an obtainability number said 0.90. The number was a prior, and priors go stale. A selector that can’t tell a prior from a live fact will keep you trying a zone that has already refused you four times.

So this series builds the probe. It wires the selector to a beta capacity-advice API that scores obtainability before you ask for capacity. Then, because that score is a prior and priors lie, it adds a ledger that learns from what the autoscaler actually refused, gates every widening onto a scarce rung on a live attempt to get one, and runs the whole loop against real GPU scarcity. The ladder stops being a static priority list you wrote once and becomes a rung ordering that reorders itself against what the cluster will give you today.

flowchart LR
    have["Capacity you HAVE"] -->|"reclaimed"| interruption["Interruption"]
    want["Capacity you WANT"] -->|"never there"| obtainability["Obtainability"]
    interruption -->|"checkpoint, drain, retry"| solved["Solved: resume the work"]
    obtainability -->|"nothing to checkpoint"| open["Open: probe, then route around"]
    open -->|"this series"| probe["A probe() that runs"]

Why I’m the One Telling You This

Because I ran it. None of that’s a hypothetical I reasoned my way into. They’re a fine-tune I was trying to ship, on a cluster I was paying for, that refused to run because I’d armed myself for the wrong failure. Everything in the eight posts after this one is code I wrote, ran on a real GKE cluster, and tore down, with every measured claim tagged as measured and every projection tagged as a projection. I learned the second half of scarcity the expensive way so that you can learn it by reading.

What’s Next

Part 1 is the frame; it doesn’t ask you to do anything yet except stop trusting a capacity picture you haven’t checked. Part 2 makes the check concrete. There’s a beta Compute Engine endpoint that will tell you, right now, whether you can get twenty Spot VMs of a given shape in a given region, and it returns a field called obtainability. Most teams building on Spot have never used it. Next, we will.