Rubicon asks you a question and you type an answer; an LLM writes the question and another LLM decides whether your answer is right. Both of those are per-round costs, and for months we had only a vague sense of what they came to.
When we finally metered it properly, the answer was about $5 per daily active user per month, against a target of $0.50. This is the breakdown, the two things we had wrong, the changes that brought it to an estimated $0.40–0.60, and what a week of live traffic measured afterwards.
Almost nobody publishes these numbers. Vendor pricing pages give you dollars per million tokens, which tells you nothing about what a feature costs, and "it depends" is the standard answer to anyone who asks. So here is one real application's, with the caveats attached.
From our usage table, over six active days of real (small) traffic:
| Purpose | Model | Calls | Avg tokens in / out | $ per call |
|---|---|---|---|---|
| Generate a question | Haiku 4.5 | 180 | 3,100 / 360 | 0.0049 |
| Grade an answer | Haiku 4.5 | 126 | 3,100 / 190 | 0.0040 |
| Bot's own answer | Haiku 4.5 | 60 | 375 / 21 | 0.0005 |
| Translate a question | Haiku 4.5 | 42 | 270 / 330 | 0.0021 |
| Appeal a score | Sonnet 4.5 | 4 | 1,630 / 175 | 0.0075 |
| Generate / translate | Gemini | 6 | 0 / 0 | $0.00 — not metered |
Two things in that table were news to us.
$0.0040 against $0.0049. Marking an answer is intuitively the smaller job — the model reads a sentence and says yes or no — but the prompt is not smaller. Our judge carries a 3,100-token rubric into every single call: the question, the model answer, the points being looked for, the scoring guidance, the formatting rules. The answer being graded is maybe twenty tokens of that.
The cost of an LLM call is the cost of its prompt, and a rubric you wrote once and stopped looking at is a prompt you are paying for on every request forever. We had optimised the generation path repeatedly and never once looked at the grading path, because grading didn't feel like the expensive one.
Every Gemini call logged as $0.00, because our Gemini and Vertex adapters never surfaced
usageMetadata. The calls happened, the tokens were spent, and our own cost dashboard showed
nothing — and, worse, showed less the more traffic we moved to Gemini. A dashboard that
improves when you shift load to the unmetered provider is not a dashboard, it is a trap.
That was the first thing we fixed, before any optimisation, because nothing else could be measured until it was. If you run more than one model provider, check that each one is actually reporting tokens before you trust a single number you have about cost.
Per million tokens, which is what makes the rest of this arithmetic checkable:
| Model | Input | Output | What we use it for |
|---|---|---|---|
| claude-haiku-4-5 | $1.00 | $5.00 | bot answers, and the last-resort fallback |
| claude-sonnet-5 | $2.00 | $10.00 | appeals, when a player disputes a score |
| gemini-2.5-flash | $0.30 | $2.50 | question generation and translation |
| gemini-2.5-flash-lite | $0.10 | $0.40 | grading, until 24 September |
| gemini-3.6-flash | $0.75 | $3.75 | grading since 24 September, and the Lyceum tutor (promotional price; $1.50 / $7.50 from 2027) |
| gemini-embedding-001 | negligible | — | the scoring gate |
At our prompt sizes that works out to:
| Call | On Haiku | On Gemini |
|---|---|---|
| Generate a question | $0.0049 | $0.0018 (Flash) |
| Grade an answer | $0.0040 | $0.0004 (Flash-Lite) |
One detail worth knowing if you are counting on prompt caching to save you: it did nothing for us on Haiku. Our 2,506-token prefix sits under the 4,096-token minimum for caching to engage. Just below a threshold is the worst place to be, because you get none of the benefit and no warning that you are missing it.
Seven changes, none of which alter what a player sees:
Per question answered, generation and grading together:
| Surface | Before | After |
|---|---|---|
| Solo / campaign, English | ~$0.004 | ~$0.0002 |
| Multiplayer, Easy and Medium | ~$0.008 | ~$0.0003 |
| Multiplayer, Hard | ~$0.008 | ~$0.002 |
| Duels | ~$0.009 | ~$0.002 |
Hard rounds and duels stay expensive because they still generate live — our stored questions sit at each topic's own difficulty tier, so a Hard round often has nothing to draw on, and duels use reasoning questions that were never pooled. That is the largest remaining cost and the next thing to fix.
At roughly 20 answers a day per player:
| Per player per day | Per player per month | |
|---|---|---|
| Before (measured) | ~$0.16 | ~$5 |
| After (estimated) | ~$0.015–0.02 | ~$0.45–0.60 |
| After (measured, 24–26 September) | ~$0.042 | ~$1.30 |
Added 27 September 2026.
Over 24–26 September the real figure was about $0.042 per player per day, roughly $1.30 a month: more than double the estimate. Most of the gap is a choice we made after writing it, not the estimate being wrong about what it covered.
| Purpose | Model | Calls | $ per call |
|---|---|---|---|
| Grade an answer | Gemini 3.6 Flash | 122 | 0.0016 |
| Generate a question | Gemini 2.5 Flash | 82 | 0.0013 |
| Translate a question | Gemini | 59 | 0.0008 |
| Lyceum tutor turn | Gemini 3.6 Flash | 27 | 0.0030 |
| Bot's own answer | Haiku 4.5 | 30 | 0.0005 |
| Appeal a score | Sonnet 5 | 2 | 0.0070 |
We moved the judge up a tier. Within two days, Flash-Lite was marking right answers written in plain words as blank or off-topic, and our cloud provider was throttling it under light load. We re-ran the candidates against answers we had marked by hand. Gemini 3.6 Flash had the smallest average error, 0.49 points out of 10, against 0.79 for Haiku and 1.00 for 2.5 Flash-Lite. It costs about four times what the estimate assumed per verdict ($0.0016 against $0.0004), so grading alone now comes to about $0.018 per player per day, which is roughly the whole of the old estimate. The price is also promotional: it doubles in January.
Lyceum is new. Our AI tutor launched on 23 September, after the estimate was written. It adds about $0.007 per player per day.
The rest held. 26% of questions were generated live, against the 30% we assumed, and nothing fell back to Haiku for grading or generation after 23 September. Take out the judge upgrade and the tutor, and the remainder comes to about $0.02 per player per day, at the top of the original range.
So the saving from the changes is real. We then spent part of it on grading, because a judge that marks right answers wrong costs more than any model's list price.
The measurement itself cost almost nothing. Not having it cost us about ten times our target for months.
Rubicon is a trivia game where you answer in your own words and an AI judge marks what you meant, not how you spelled it. Every number above comes from the models that run it.