Cheap Hardware Works Great

Case Study: Running SpiceGrinder on the Cheapest Box AWS Sells

We already knew SpiceGrinder’s engine runs on modest hardware — the benchmarks page has the numbers across six real AWS instance types, from the smallest free-tier boxes up to the flex family. What we hadn’t done was point the actual Service API at the worst-performing one, throw a genuinely unpleasant workload at it, and see what broke.

And the answer was: nothing.

The setup

The box: a t4g.micro — 2 vCPUs, 1 GiB of RAM, the smallest, cheapest instance type AWS offers. The kind of instance you’d spin up almost by accident, not the one you’d choose if you were trying to look good in a benchmark.

The workload: register more models than the service is configured to keep warm at once — well over the default cap of 20 — pin a handful of them, and then flood the service with 16 concurrent clients hammering random model IDs, requesting 2,000-4,000 rows per call, sustained for three minutes straight. Then we did it again with the cache expanded to 50 slots and 75 registered models, and again at 100 slots and 150 models, to see whether any of this changed as the numbers grew.

Across all three runs: roughly 78,000 real requests, over nine minutes of sustained load, on a box with less RAM than a phone from a decade ago.

It just worked. Consistently.

Zero failures, across all three sizes, across all ~78,000 requests. And the numbers barely move as the cache and the model pool triple and quintuple in size: p50 sits at 105-108ms the whole way through, p95 stays under 200ms, p99 never breaks a quarter second. For a $6-a-month instance under a genuinely adversarial multi-model workload, that’s a small, competent, boring, forgettable table.

1 GB of RAM is fine

The one way to actually hurt this box is a request-size mistake, not a hardware limit. SpiceGrinder’s buffered /generate endpoint builds its entire response in memory before sending it — we confirmed directly during previous testing that a single 1,000,000-row request against a 1GB host runs the process out of heap. The identical request against /stream succeeds cleanly on the same box, because streaming sends chunks incrementally instead of holding the entire response at once.

This test never got near that line on purpose — 2,000 to 4,000 rows per call is the range we recommend as a starting point for sizing a /generate call, not a stress test of the memory ceiling itself. (It’s really bytes per call that matters, not row count, so that range is a starting point to adjust from based on your own model’s row width, not a hard rule — the same reasoning that already governs stream.batch.size’s default.) The box didn’t so much as hiccup across 78,000 of them.

Running on 1GB isn’t risky. This is just a caveat that there is an implicit upper bound on how many rows to request in a single chunk. That’s true on any size server, not just the 1GB one; the lower-end server just makes the consequences of getting it wrong show up quicker.

Model swapping cost is noise. Don’t pin for speed.

We split every request’s latency two ways: whether it hit a model already warm in the cache, or had to rebuild one that had been evicted. If swapping cost anything real, that split should show it clearly.

It doesn’t.

Warm and rebuilt requests land at statistically the same latency, at every cache size we tested. We went looking for why, and found the real answer sitting in the service’s own architecture: every generation call — regardless of which model, regardless of how many client connections are open — is serialized through a single worker thread per process, specifically to avoid a race on the shared random-number generator. Sixteen concurrent clients aren’t sixteen parallel generations; they’re sixteen requests standing in one line. Under that kind of load, the time you spend waiting in line dwarfs the few milliseconds a cache miss costs you. The swap is real, it’s measurable in isolation, and it is completely lost in the queueing noise the moment real concurrent traffic shows up.

Which means keepResident — the flag that pins a model against cache eviction — is not a performance knob, and should not be treated like one. Pinning ten models out of seventy-five required exactly zero forced rebuilds, which is the correct, working behavior. But it did not buy them a single millisecond of measurable speed over their unpinned neighbors. The reason to pin a model is that you need a specific guarantee — continuous RNG state across calls, a session that must never restart mid-sequence. That’s a correctness requirement, and keepResident is the right tool for it. “This model needs to run fast, better pin it” is trying to solve a problem doesn’t actually exist.

That said, this test also shows that if you do have need to keep additional models pinned to ensure uninterrupted RNG state, increasing the model cache size is an option available to you, even on this class of hardware.

The bottom line

If you’re already running in a cloud you control, the monthly number for hosting this comes out to about the price of a cup of coffee. At the time of this writing, running a t4g.micro server 24/7 on AWS in US-east-1 costs just over $6/month at full AWS list price. If you only run this server for 10-12 hours a day on weekdays, it’s about $2/month.

Including this as an aside because because it’s kind of funny: producing every number in this post — three fresh EC2 instances, launched, configured, flooded with 78,000 requests apiece, and torn back down — cost us well under one cent in AWS charges. Combined.