We measured when an embedding can replace an LLM judge

Rubicon asks you a question and you type an answer in your own words. Something then has to decide whether "the bloke who did the printing press" means Johannes Gutenberg. We use an LLM for that, and it works — but when we finally metered it, grading cost almost exactly as much as generating the question in the first place: $0.0040 a call against $0.0049, because the judge carries a 3,100-token rubric into every single one.

The obvious idea is to skip the model when the answer is obviously right or obviously wrong, using a cheap embedding comparison, and only pay for a judgement in the middle. The non-obvious part is where "obviously" sits. We could not find published numbers for this, so we measured it on our own traffic.

Everything below is measured, not estimated. It is also specific to one embedding model and one judge — the method transfers, the constants do not.

What we measured

We exported every answer real players had submitted, joined to the question it answered, and scored each one two ways: the LLM judge's verdict (the label) and an embedding similarity (the candidate signal).

• Input: 408 submissions from production, later re-run on 539.
• Excluded: 63 answers that came from the multiple-choice lifeline. Their "answer" is a clicked option and often the correct text verbatim, so they would stack the curve with perfect matches the gate will never see in the wild. Also 4 rows with no answer or no model answer.
• Embedded: 927 distinct strings — every answer, every model answer, every asked-for bullet, deduplicated. gemini-embedding-001 at 768 dimensions, taskType=SEMANTIC_SIMILARITY.
• Label: the judge's own content score, normalised to 0–10. Right is ≥7, wrong is ≤3. The middle band is excluded from both error rates rather than forced to one side.

That left 341 free-text answers with a usable label: 168 judged right, 125 judged wrong, 48 in the middle.

The signal is a max, not an average

A question has a model answer and, usually, a list of points it is asking for. The similarity we use is:

max( cosine(answer, modelAnswer), max over i of cosine(answer, askedFor[i]) )

Either one alone is enough to be right. A terse answer nails one required point and ignores the model answer's phrasing entirely; an essayistic one echoes the model answer and never states a bullet cleanly. A gate that demanded both would fail the first kind of answer, which is the kind good players give.

The distributions overlap more than you would guess

group	n	p10	median	p90
judge said right	168	0.848	0.916	0.976
judge said wrong	125	0.731	0.791	0.870

This is the table that makes the exercise worth doing. The separation is real, but the cosines sit in a compressed high band: a wrong answer's median similarity is 0.79. Any intuition calibrated on "0.8 means similar" is simply wrong for this model. If you pick your threshold by feel, you will pick it far too low and start telling correct players they were wrong.

Note also that the two distributions overlap between roughly 0.73 and 0.98. There is no single threshold that separates them. That is the whole reason the design is a gate with two thresholds and an LLM in the middle, rather than a classifier.

Where it breaks: order and count

The wrong answers that scored highest on similarity were all the same shape — questions that ask for things in a particular order, or a particular number of them.

Cosine similarity is a bag-of-meaning measure. It cannot see sequence. "Name the planets outward from the Sun" scored 0 from the judge and 0.947 from the embedding, because the answer contained exactly the right words in the wrong order. No threshold fixes that; the signal is blind to the thing being marked.

The fix is not a better threshold, it is a narrower scope. Restricting the gate to plain free-text questions — dropping the ordered and styled-recall shapes — moved the 2%-error threshold from 0.948 down to 0.893 and nearly doubled what the gate could settle.

If you take one thing from this: check whether your signal can see the property you are grading, before you tune it.

The thresholds

Free-text questions only, n=268:

error budget	auto-correct above	auto-wrong below	settled without an LLM call	mistakes
1%	0.948	0.760	68/268 (25%)	0 of 40 passed, 0 of 28 failed
2%	0.893	0.795	161/268 (60%)	2 of 100 passed, 1 of 61 failed
5%	0.870	0.802	197/268 (74%)	6 of 129 passed, 3 of 68 failed

We took the 2% row. The 1% row is nearly mistake-free but only reaches a quarter of traffic; the 5% row buys 14 more points of coverage for triple the errors. Telling a correct player they were wrong is the expensive mistake here — far more expensive than a fraction of a cent on a judge call — so the error budget is not symmetric with the money.

One asymmetry is worth keeping in mind: auto-wrong is the safer direction. Nothing below 0.76 was ever judged right, while the auto-correct side needs 0.95 to be equally clean. Cheap confidence is easier to come by when you are ruling things out.

Re-cut on more data

We re-ran the same calibration later on 539 submissions (390 free-text, 179 judged right, 158 wrong, 21 Russian) and shipped these:

side	threshold	settles	disagreement with the judge
auto-correct	≥ 0.90	114 (29%)	2 judged wrong (1.8%)
auto-wrong	≤ 0.78	88 (23%)	2 judged right (2.3%)

About 52% of free-text English answers now never reach the model. The auto-correct side scores full marks; the auto-wrong side scores zero; both hand the player the model answer instead of the judge's usual explanation, since there is no judge to explain anything.

What this does not tell you

• The label is the judge's verdict, not the truth. This measures agreement with one LLM judge. A judge mistake looks like a gate mistake in these numbers. Spot-checking near-misses turned up at least one answer scored 0 that reads correct to a human — which is a judge problem, not a gate problem, but it is inside the error rates above either way.
• A few hundred rows is thin. These are starting thresholds, not constants. The gate records the same measurement on every live answer so the curve can be re-cut continuously.
• Language matters and we cannot yet say how much. 21 Russian rows is nowhere near enough to calibrate per-language, and their similarities ran higher than the English ones (wrong-answer median 0.853 against 0.790). A single global threshold probably under-serves Russian. Ours is English-only until there is enough non-English data to cut its own curve.
• The constants are married to the model. gemini-embedding-001, 768 dimensions, SEMANTIC_SIMILARITY. A model change or a dimension change invalidates every number in this article, which is why each recorded row stores both.

If you want to run this yourself

The shape is the useful part, and it is about an afternoon of work:

1. Export your graded answers with whatever your current judge said about each one.
2. Throw out the rows where the "answer" wasn't really typed by a person — clicked options, empties, anything mechanical.
3. Embed each answer and each thing you are comparing it to. Deduplicate first; you will have far fewer distinct strings than rows.
4. Plot the two distributions before you pick anything. If they overlap the way ours do, you are looking for two thresholds, not one.
5. Split by question shape before you tune. If one shape is unlearnable by your signal, exclude it rather than paying for it in the threshold.
6. Pick your error budget by what the mistake costs a user, not by what it saves you.

We spent about eight minutes of embedding quota on the whole run. The measurement is cheap; not having it is what is expensive.

---

Rubicon is a trivia game where you answer in your own words and an AI judge marks what you meant, not how you spelled it. The judge described here is the one that grades every answer in the game.