Grading cost as much as writing the question

Rubicon asks you a question and you type an answer; an LLM writes the question and another LLM decides whether your answer is right. Both of those are per-round costs, and for months we had only a vague sense of what they came to.

When we finally metered it properly, the answer was about $5 per daily active user per month, against a target of $0.50. This is the breakdown, the two things we had wrong, and the changes that brought it to an estimated $0.40–0.60.

Almost nobody publishes these numbers. Vendor pricing pages give you dollars per million tokens, which tells you nothing about what a feature costs, and "it depends" is the standard answer to anyone who asks. So here is one real application's, with the caveats attached.

What it actually cost

From our usage table, over six active days of real (small) traffic:

Purpose	Model	Calls	Avg tokens in / out	$ per call
Generate a question	Haiku 4.5	180	3,100 / 360	0.0049
Grade an answer	Haiku 4.5	126	3,100 / 190	0.0040
Bot's own answer	Haiku 4.5	60	375 / 21	0.0005
Translate a question	Haiku 4.5	42	270 / 330	0.0021
Appeal a score	Sonnet 4.5	4	1,630 / 175	0.0075
Generate / translate	Gemini	6	0 / 0	$0.00 — not metered

Two things in that table were news to us.

1. Grading cost as much as generating

$0.0040 against $0.0049. Marking an answer is intuitively the smaller job — the model reads a sentence and says yes or no — but the prompt is not smaller. Our judge carries a 3,100-token rubric into every single call: the question, the model answer, the points being looked for, the scoring guidance, the formatting rules. The answer being graded is maybe twenty tokens of that.

The cost of an LLM call is the cost of its prompt, and a rubric you wrote once and stopped looking at is a prompt you are paying for on every request forever. We had optimised the generation path repeatedly and never once looked at the grading path, because grading didn't feel like the expensive one.

2. One provider was invisible

Every Gemini call logged as $0.00, because our Gemini and Vertex adapters never surfaced usageMetadata. The calls happened, the tokens were spent, and our own cost dashboard showed nothing — and, worse, showed less the more traffic we moved to Gemini. A dashboard that improves when you shift load to the unmetered provider is not a dashboard, it is a trap.

That was the first thing we fixed, before any optimisation, because nothing else could be measured until it was. If you run more than one model provider, check that each one is actually reporting tokens before you trust a single number you have about cost.

Unit prices

Per million tokens, which is what makes the rest of this arithmetic checkable:

Model	Input	Output	What we use it for
claude-haiku-4-5	$1.00	$5.00	fallback for everything
claude-sonnet-4-5	$3.00	$15.00	appeals, when a player disputes a score
gemini-2.5-flash	$0.30	$2.50	question generation
gemini-2.5-flash-lite	$0.10	$0.40	grading
gemini-embedding-001	negligible	—	the scoring gate

At our prompt sizes that works out to:

Call	On Haiku	On Gemini
Generate a question	$0.0049	$0.0018 (Flash)
Grade an answer	$0.0040	$0.0004 (Flash-Lite)

One detail worth knowing if you are counting on prompt caching to save you: it did nothing for us on Haiku. Our 2,506-token prefix sits under the 4,096-token minimum for caching to engage. Just below a threshold is the worst place to be, because you get none of the benefit and no warning that you are missing it.

What we changed

Seven changes, none of which alter what a player sees:

1. Meter Gemini properly. Visibility, not saving. Everything else depended on it.
2. Generate on Flash instead of Haiku. About 2.7× cheaper per live question.
3. Pick topics that already have a question in the pool. Our multiplayer rounds were choosing topics at random and then generating live 84% of the time. Preferring topics with a stored question at the right difficulty took the pool hit rate from 16% upward, and a failed generation now serves a nearby stored row instead of breaking the round.
4. Stop judging things that aren't answers. Blank submissions, "idk", "no clue" and keyboard mash are scored zero before any model is called. Run over 539 real answers it caught 11, and the judge had scored all 11 as zero — no false positives.
5. Skip the judge when an embedding can settle it. About 52% of English free-text answers now never reach a model. That one has its own article, with the thresholds and the error rates.
6. Precompute the onboarding quiz verdicts. They were hand-written once; tapping a chip used to cost a judge call.
7. Grade on Flash-Lite instead of Haiku. About 10× cheaper per judge call.

The result

Per question answered, generation and grading together:

Surface	Before	After
Solo / campaign, English	~$0.004	~$0.0002
Multiplayer, Easy and Medium	~$0.008	~$0.0003
Multiplayer, Hard	~$0.008	~$0.002
Duels	~$0.009	~$0.002

Hard rounds and duels stay expensive because they still generate live — our stored questions sit at each topic's own difficulty tier, so a Hard round often has nothing to draw on, and duels use reasoning questions that were never pooled. That is the largest remaining cost and the next thing to fix.

At roughly 20 answers a day per player:

	Per player per day	Per player per month
Before (measured)	~$0.16	~$5
After (estimated)	~$0.015–0.02	~$0.45–0.60

What these numbers don't tell you

• The "after" column is an estimate, not a measurement. It assumes about 30% of questions still generate live and that the cheap paths mostly get served. The "before" column is metered. We will replace the estimate once there is a week of real traffic behind it, and we would rather say that than quietly present a projection as a result.
• The traffic was small. One to two daily active users over six days. Per-call costs are solid — a call costs what it costs — but the per-player figure rests on an assumed 20 answers a day.
• Cheap models are on shared capacity. We see 429s at four concurrent Gemini calls, and our embedding quota is about six calls a minute. Every one of those falls back to the expensive model, so the real bill depends on capacity we don't control. If you are budgeting on a cheap model's list price, budget for the fallback too.
• Cheaper grading is slightly different grading. Re-judging the answers our gate still sends to a model, Flash-Lite agreed with Haiku's bucket 70% of the time and landed within two points 78% of the time. The disagreements are mostly right-versus-partly-right rather than right-versus-wrong, and a disputed score escalates to a stronger model. That is a trade we chose with the numbers in front of us, not one we discovered afterwards.

If you want to do this on your own app

1. Log tokens and cost per call, tagged with what the call was for — not per model, per purpose. "Generation" and "grading" being separable is what made the first finding visible.
2. Check every provider is really reporting. A provider logging zero looks like a cheap provider.
3. Look at your prompt sizes before your model choice. A 3,100-token rubric on a cheaper model is still a 3,100-token rubric.
4. Find the calls that don't need a model at all. Empty answers, duplicate requests, anything with a precomputable result. These are free savings with no quality cost.
5. Then downgrade the model, and measure the disagreement rather than assuming it is fine.
6. Price the fallback path, because under load that is the path you are on.

The measurement itself cost almost nothing. Not having it cost us about ten times our target for months.

---

Rubicon is a trivia game where you answer in your own words and an AI judge marks what you meant, not how you spelled it. Every number above comes from the models that run it.