Runtime 2: Replay vs. Continue

The problem

Sometimes you want the exact same output back (reproducing a specific failure, asserting on fixed values in a test). Sometimes you want fresh draws from the same running model without restarting it. Both go through the same /generate call. The difference is one field, and it’s important to understand what that means and how it works.

Explicit seed always replays

POST /v1/models/m-8fabf14b.../generate
{"count": 3, "seed": 42}

{"generationId": "g-e335793c...", "seed": "42", "seedSource": "explicit",
 "rngContinued": false, "data": "1328.88944245958\n1032.9487578330304\n1998.4253348481675\n"}

Call it again, identical request:

{"generationId": "g-11c6af97...", "seed": "42", "seedSource": "explicit",
 "rngContinued": false, "data": "1328.88944245958\n1032.9487578330304\n1998.4253348481675\n"}

Same three values, byte for byte. rngContinued: false on both calls tells you that a resident graph already existed by the second call (the first call built and cached one), but supplying an explicit seed forced the random number generator to start over with the provided seed. Since the seed was identical, that is what made it a replay: an explicit seed doesn’t care what’s resident, it always gives you the same starting point.

No seed continues whatever’s resident

If you drop seed from the request:

POST /v1/models/m-8fabf14b.../generate
{"count": 3}

{"generationId": "g-751b2b76...", "seed": "42", "seedSource": "explicit",
 "rngContinued": true, "data": "3753.1974124840012\n1738.170392594539\n4814.593001737639\n"}

Now you get different results, not a repeat of the seed-42 output. However, it’s not a fresh random start, either. rngContinued: true means this call picked up the resident graph exactly where the last explicit-seed call left its RNG, and drew the next values forward from there. Call it again with no seed and it keeps going further still:

{"generationId": "g-f3994aa3...", "rngContinued": true,
 "data": "2592.4813592053542\n3215.1422253886203\n5767.530578644751\n"}

A field that’s easy to misread

Notice that seedSource still says "explicit" on both no-seed calls above. That’s not indicating a bug or a stale value — it’s reporting how the currently resident graph originally got seeded, not what this particular request passed. A no-seed “continue” call inherits whatever seed history the resident graph already has; seedSource is describing the graph’s provenance, not this call’s own input. Read it as “here’s where the state you’re continuing came from,” not “here’s what you asked for this time.”

The operational catch

Because an explicit seed always rebuilds — never continues — mixing one into a sequence of otherwise-unseeded calls resets the resident graph out from under anyone else relying on it to continue. If a CI job replays a fixed seed against a model another caller is mid-stream on, that caller’s next no-seed call continues from the CI job’s replay, not from its own prior draws. Same endpoint, same model, and one caller’s replay just became another caller’s discontinuity. If callers need independent continuation, they need independent models (or at minimum, awareness of who else is touching the same modelId).

That’s actually less difficult to manage than it sounds. The shared store does not check for duplicate model names or duplicate models at all. If you register the same model twice, you get different modelId values back. In the case above, the CI group might want to have their own private copy of the model in question so that they can ensure their runs don’t get interrupted by other people calling the same model (and that the CI run won’t introduce problems in other people’s data).

Combine that with the cache-size, idle-timeout, and pinning knobs from Runtime 1 to decide how to size and configure your server for your organization’s needs.