Showcases 3: Feeding a Pool Without Pro

Uses Only Free-Tier SpiceGrinder

Problem statement

Showcases 1 and Showcases 2 both lean on FlatFile, which is a Pro component. If you’re on Free, the obvious conclusion is that the whole pattern is closed to you.

It isn’t, and you should be aware of the workaround before you decide whether or not you need to upgrade — partly because the workaround is effective, and partly because understanding exactly where it stops being effective is the clearest way to see what FlatFile is actually buying you.

SpiceGrinder Free is numeric-only, so it can’t hand you a name. But it can hand you the row number of a name. Emit an index, do the lookup yourself downstream, and the pool never has to enter the model at all.

The model

Two numbers are needed: which pool, and which row in it. Categorical handles the first — it takes weights and emits the drawn 0-based index, which is exactly a market code. Uniform into ToInteger handles the second: draw a real number across the pool’s range and floor it.

<?xml version="1.0" encoding="UTF-8"?>
<dataset seed="42">
	<root node="Order"/>
	<nodes>
		<Append name="Order">
			<input name="Market"/>
			<input name="CustomerRow"/>
			<input name="ItemCount"/>
			<input name="OrderValue"/>
		</Append>

		<!-- 0 = DE, 1 = FR, 2 = ES -->
		<Categorical name="Market" weights="0.50,0.30,0.20"/>

		<!-- row index into that market's 200-row pool -->
		<ToInteger name="CustomerRow" mode="floor">
			<input name="RowDraw"/>
		</ToInteger>
		<Uniform name="RowDraw" min="0" max="200"/>

		<Poisson name="ItemCount" lambda="2.2"/>
		<Lognormal name="OrderValue" mu="4.1" sigma="0.8"/>
	</nodes>
</dataset>

Six nodes, every component Free. Run it (samples/software/orders-multi-locale-free.xml):

1,134,3,29.534151471601852
2,32,3,15.782629571402822
2,196,4,109.64989566069686
1,16,2,88.14631181927025
0,88,1,140.59989142221843

Join those against the pools — a SQL join, a pandas merge, ten lines of shell — and you have your data.

Here is the fun part: it is the exact same data. Market 1 row 134 is Hortense Diallo, and Hortense Diallo is the first row of the Pro model’s output in Showcases 2. Market 2 row 32 is Nicodemo Cabello Barrera, and so is Pro’s second row. The item counts and order values are byte-identical between the two models. The Pro version isn’t generating better data here — it’s doing the join for you, inside the model, at generate time.

The same trick works for Showcases 1, with one pool instead of three and no market code (samples/health/claims-from-synthea-free.xml):

27,4,269.411696423658
35,2,393.7695537208169
12,7,1178.44857884471

Row 27 of the member pool is the same Synthea member the Pro model drew first, with the same 4 service units and the same $269.41.

Why this all works

It works because the pool value is terminal. Nothing downstream in either model depends on who the customer is — the identity gets attached and carried out. That’s the condition, and it’s also the boundary.

The moment you need one of those data values in an upstream operation, the idea of using an index collapses. You can’t branch on a customer’s country to pick a tax rule, derive an email from the name you drew, vary claim severity by the member’s age, or compose the row into a business object.

An index is an opaque number until it leaves SpiceGrinder, so anything the model needs to know has to enter the tree upstream of the node that needs it. That’s what FlatFile is for, and that’s where using Pro-tier brings real value.

Nothing validates the range. The Uniform max has to match your pool size, and SpiceGrinder cannot check it, because the model never opens the file. Regenerate a 40-row pool as 35 rows and the model will happily emit index 38 forever — producing a join that silently drops rows, or an error somewhere far downstream with no obvious connection to the cause. FlatFile has no such failure mode: it reads the actual file and knows how long it is.

The first point draws the line at capability — where Free tier is limited and what Pro tier brings to the table. The second point is about data integrity and how brittle models can become when tied to external data.

Use floor, and make the range exclusive at the top. Uniform is [min, max) — the upper bound is already excluded — so min="0" max="200" with mode="floor" gives you exactly 0–199. Using round instead would fold everything in [199.5, 200) up to 200 — a quarter of a percent of your rows pointing at an index that doesn’t exist.