Schedule the Accelerators: A GPU/TPU Puzzle

I’m hiring. I’m looking for a Senior Developer Relations Engineer for GKE and AI Infrastructure, and rather than tell you what the job is like, I’d rather show you.

Skip to the puzzle ↓

The day job is taking hard infrastructure problems—the kind that involve GPU memory limits, TPU topologies, and scheduling—and turning them into things developers actually want to read and watch. That’s a mouthful if you’ve never had to place a training job on a cluster. So here is one of those problems, small enough to hold in your hands.

What you’re looking at

A modern AI workload doesn’t just need a computer. It needs a specific kind of chip—a GPU or a TPU, the accelerators built to do the enormous parallel math that training and running models demands—and it needs enough of them, arranged the right way, all at once. Getting the right jobs onto the right chips is a packing problem, and it fights back.

Four rules make it fight back:

• A chip has a memory ceiling. Each accelerator has a fixed amount of high-bandwidth memory. Ask it to hold a model bigger than that and the job never even starts—it’s turned away at the door, not halfway through.
• A training job is all-or-nothing. Big models train across many chips working in lockstep, a pattern called gang scheduling. Land three of the four workers and the whole job still waits. Partial placement is worse than none: it holds chips hostage while making zero progress.
• Cheap capacity can be taken back. Some jobs run on spot capacity: the same chips as everyone else’s, sold at a steep, floating discount in exchange for one condition. A higher-priority job can reclaim them the instant it needs them, and yours stops mid-run. This isn’t leftover or idle hardware; it’s full-strength capacity you’re only allowed to borrow until someone with more claim shows up.
• A TPU slice is wired in a grid. This is the one that surprises people. A TPU pod isn’t a bucket of interchangeable chips; the chips are cabled to their neighbors in a mesh, and a multi-chip job needs a contiguous block of them—chips that actually sit next to each other. Scatter smaller jobs carelessly and you can end up with plenty of free chips and still no room, because no free chips are adjacent. The job sits Pending while the cluster looks half-empty.

That last rule is the hard one, and it’s nearly impossible to feel from a diagram. So play with it.

Play it

Watch the Events panel as you go—when a placement fails, it fails the way a real cluster tells you it failed, in the same format as kubectl describe output. Flip on the manifest to see your placement written out as YAML, the way you’d actually declare it.

Why this, and not a blog post about it

Because the difficulty is easier to understand once you’ve run into it yourself. If you’ve ever untangled a Pending pod that “should” fit—stared at a cluster with free capacity that refuses to schedule your job—you already know the exact feeling this is trying to reproduce. If you haven’t yet, now you have, and it cost you a few minutes instead of a Friday afternoon.

That feeling is most of the job I’m hiring for. You build first: the work is running actual training and inference workloads on Google’s accelerators and feeding what you learn back to the teams shipping them. It’s for someone who wants a real point of view and an audience that listens, and who finds a fragmented TPU slice more interesting than annoying.

A word on who you’d report to, since the rec won’t tell you: I’ve spent most of my career at the seam between infrastructure and the people who have to use it, and I care about this scheduling problem because I’ve watched good engineers lose whole afternoons to a cluster that wouldn’t explain itself. I’d rather you disagree with me early than nod along later.

If the puzzle made you want to argue with it, dig into it, or explain it to someone else—that instinct is the one I’m hiring. The apply link is at the top. It’s a standard Google hiring loop: a recruiter screen comes first, then a set of interviews that lean on real infrastructure judgment and on your ability to explain it to someone who isn’t in the room with you.