Not gracefully. Not with a polite exit code and a stack trace you could grep through in the morning. It kernel-panicked mid-pull, mid-load, in the middle of a model download that was sharing RAM with a test suite that had no business running at the same time. By the time the page landed on my phone, the logs were gone, the process table was a mess, and I was standing in my kitchen trying to remember if I'd ever written a runbook for "machine is unresponsive and also it's three in the morning."
I had not.
That panic didn't come from bad code. The model worked fine. The test suite worked fine. The problem was simpler and stupider than any of us wanted to admit: we'd let two resource-intensive operations start at the same time on the same machine, and the machine had opinions about that.
This is the failure mode nobody talks about in local LLM operations. Everyone optimizes prompts, quantization settings, inference throughput. Nobody talks about what happens in the ten seconds before the model even loads. Those ten seconds are where the real danger lives.
The host is the thing
When you're running AI workloads locally, actual models on actual hardware, not farmed out to an API, you inherit a class of problems that SREs have known about for decades but that the ML community keeps rediscovering the hard way. The model itself is rarely the issue. The issue is the host it lives on.
A large model download and a large model load look like completely different operations if you're thinking in ML pipelines. From the machine's perspective, they're the same thing: sustained, greedy, memory-and-I/O-intensive processes that each want the full attention of the system's resources. Run them concurrently and you're not running two things efficiently. You're running one thing badly while actively undermining the other.
The Google SRE book calls this "avoiding overload" and spends multiple chapters on it. The distributed systems literature calls it backpressure: the circuit-breaker pattern, the principle that the correct response to a system that can't absorb more load is to refuse the work at the door rather than accept it and collapse under its weight. macOS's launchd has had conditional execution primitives since before most ML engineers were writing Python: start a job only if a prior job has exited, throttle on certain path and daemon conditions. What launchd doesn't give you natively is load-threshold gating. The "don't start unless the host is healthy" check is something you build at the orchestration layer, as an explicit pre-flight gate. No scheduler hands you that for free.
The pattern is old. It just hasn't been applied consistently to local LLM ops.
Three things you need to know before you touch a heavy operation
After the 3am incident, we built a gate. Not a complicated one. The concept is embarrassingly simple in retrospect. Before any resource-intensive model operation starts, you check three things. All three have to pass. If any one of them fails, the operation doesn't start. It doesn't start later, or with reduced priority, or after a warning. It doesn't start.
The first check, and for model loads, the binding one, is available memory.
When you're pulling a large model into RAM or VRAM, the metric that will actually kill you isn't a stressed CPU scheduler. It's running out of addressable memory while the load is in progress. Load average can be high and the operation can still complete if you have genuine headroom. But if available RAM is below what the model actually needs to seat itself, there is no scenario where the operation ends well. The OOM killer steps in, something gets evicted or crashed, and you're back in the kitchen at 3am.
The specific headroom that's right depends on your models: their size, quantization level, how much the inference framework pre-allocates on initialization. What matters is that you measure actual free memory before the operation starts and compare it against the real lower bound, with margin. Not a tight margin. A real one. If the gap between what's free and what the model requires makes you uncomfortable, you don't start. Your discomfort is data.
Load average is worth checking too. A system that's already thrashing under existing demand is a system where "theoretically enough RAM" can evaporate fast, and a high scheduler load number is a useful early warning signal. But it's secondary. It tells you the system is stressed; it doesn't tell you whether the operation will succeed or fail. Available memory headroom is the thing that directly determines whether your model load lives or dies.
What happens when you skip both? You start a model pull because it "shouldn't take long" and you need it done before morning. Three minutes in, the page goes off. In the best case, the operation fails. In the worst case, the machine kernel-panics.
The second check is disk space.
This one feels obvious but it's the most common footgun in practice. Large model files are large. If you're managing multiple models, different sizes, different quantizations, different task specializations, the disk fills faster than your intuition tracks. A model pull that starts at barely enough free space and needs more does not fail cleanly. It fails messily, often mid-write, often corrupting whatever partial file it managed to lay down before the disk ran out.
There's a subtler version of this too. Even operations that aren't themselves disk-heavy can trigger cascading disk pressure. A full test sweep generates logs. A model load that fails and retries generates core dumps on some configurations. The test suite writes coverage reports. You end up in a situation where the disk fills from the combination of things you're running, none of which individually seemed like a disk risk.
The gate check here is simple: measure free space before the operation starts and compare it against a threshold that gives you real headroom, not just "technically fits." If you're not comfortable with the margin, don't start.
And when the download finishes, verify what actually landed before you trust it. A pre-flight check on disk space needs a post-download integrity check as its counterpart: compare the file against a published digest before you declare it ready to load. This matters more than most people realize. A corrupted model doesn't always fail loudly on load. Sometimes it loads, appears functional, and returns garbage outputs for hours before anyone notices something is wrong. The redownload never gets triggered because you never knew something was wrong. The digest check is what closes that loop: download, verify, then trust. Skip it and you've got a silent failure mode that no pre-flight gate can protect you from.
The third check is in-flight operations.
This is the one that got us at 3am, and it's the one most teams don't implement because it requires tracking state across processes in a way that feels like overkill until it isn't.
The check is simple to describe: before starting any heavy operation, verify that no other heavy operation is already running. Not "I think nothing's running." Verify it. Look at the process table. Check a lock file. Query whatever your scheduler or job runner knows about in-flight work. The mechanism is less important than the invariant: heavy operations serialize. They do not overlap.
This is where people push back. "But if I serialize everything, won't things take longer?" Yes. They'll take longer in the same way that a single-lane bridge is slower than a two-lane bridge. But if your vehicles are a model download and a full test sweep, neither of them is going to fit side by side without a collision. The slowdown is the point.
The forbidden-combination matrix
After we built the gate, we mapped out the combinations that had actually caused problems, or that we could reason would cause problems, and turned that map into a hard constraint list. Not a warning list. A constraint list.
Some combinations are simply forbidden, regardless of what the available resource numbers look like at the moment you're tempted to start:
A model pull and a model load cannot run concurrently. Both are sustained, high-memory, high-I/O operations. The intuition that they're "different enough" to overlap is wrong. They are not different enough.
A model load and a full test sweep cannot run concurrently. The test sweep competes for the exact system resources, CPU, memory, I/O, that the model is trying to claim for residence. The model wins in some scenarios. The tests win in others. Neither finishes cleanly when both are fighting for the same headroom.
Two in-flight heavy processes of any kind cannot run concurrently. This is the general case that subsumes the previous two. "Heavy" here is a label you assign in advance, not something you detect at runtime. If an operation is resource-intensive enough to require a pre-flight check before it starts, it's resource-intensive enough to be in the serial queue.
The subtlety that took us a while to internalize: "not running at the exact same second" is not the same as "not overlapping." A model pull that's been running for four minutes is still running when the second operation starts. The serialization check has to account for the duration of the in-flight operation, not just whether something started after something else finished. This distinction sounds pedantic until it's the thing that saves you.
The gate as a habit, not just a function
We eventually wired all three checks into a gate that lives in front of every heavy operation. The mechanics, how you implement it in whatever shell, scripting environment, or job scheduling system you're using, are less important than the conceptual shape.
The gate is a go/no-go decision that runs before every heavy operation and before the operation knows what resources it's actually going to need. It checks memory, it checks disk, it checks in-flight state. If any check fails, the gate returns a refusal. The operation does not start. If all three pass, the gate issues a lock: a signal that says "something heavy is now running," and the operation proceeds under the assumption that no other heavy operation will be permitted to start while the lock is held.
This is the pre-flight checklist pattern from aviation, applied to local compute. It's the circuit-breaker and backpressure pattern from the distributed systems world, made concrete at the orchestration layer. It's the load-threshold gating the schedulers don't give you natively, built explicitly where it actually needs to live. None of this is new. What's specific is applying it consistently, in front of every heavy operation, without exceptions for "this one will probably be fine."
The "probably fine" cases are exactly where the 3am pages come from.
One more thing about the gate: the refusal has to be visible. A silent refusal, an operation that silently skips without telling anyone, is worse than no gate at all. The gate's job is to surface the conflict so you can make a real decision: wait, reschedule, or cancel. If it hides the refusal, you'll spend an hour wondering why the model didn't load and eventually conclude it was "probably a transient thing."
Log the refusal. Surface it wherever you surface operational events. Make it loud enough to be findable. A gate that whispers isn't a gate.
What this actually buys you
The gate doesn't make your hardware faster. It doesn't give you more RAM or more disk or a faster I/O bus. What it buys you is something different: it makes resource contention a decision instead of an accident.
Before the gate, resource contention happened when two operations started too close together and the machine made a bad implicit tradeoff between them. After the gate, resource contention is surfaced explicitly, as a refusal, before either operation starts. You make the call. The machine doesn't make it for you at 3am.
That's the shift. From "the machine will figure it out" to "the gate will figure it out, in advance, with your thresholds, and tell you when the answer is no."
The teams I've seen skip this step share a common belief: that their setup is different, that their hardware is beefy enough, that the operations are short enough that the window of risk is negligible. They're not wrong about the hardware or the duration. They're wrong about the probability. The failure is rare. Until it isn't, and then it's 3am and the logs are gone.
Where this leads
Once you have a gate, you start seeing a natural split emerge in your local operations: the operations that pull and load models into memory (slow, RAM-heavy, irreversible once started), and the operations that do inference once the model is resident (potentially concurrent, potentially parallelizable, much lighter on the way in). These are different beasts. They want different scheduling treatment. They have different failure modes.
The pre-flight gate is the foundation. The next layer, keeping model loads in one serialized lane while inference runs in a separate bounded-parallel lane, is where the real throughput gains live. Two queues. You can't get to the two-queue thesis until you've first established that you will never let two resource-intensive operations collide at the bottom.
Serialization first. Parallelization second. That's the order.
The machine died at 3am because nobody had written the rule that said it shouldn't. The rule is not complicated. The rule is: don't start a heavy operation unless the host can take it, enough free memory, enough disk, nothing else already running, and never run two of them at the same time.
Write the rule before the machine writes it for you.