PROJECT 05 / AI FORECASTING / OCTOBER 2026
USING CLASSIFIERS IN
PREDICTION MARKETS.
D1 and Jev are part of a new class of decision models: classifiers that return probabilities instead of prose. We tested them on 1,005 resolved questions from GJOpen, a public crowd-forecasting platform.
01 / WHY THIS TEST
THE DECISION-MODEL MOMENT.
TypeSafe introduced Jev on September 15, 2026 as a fast decision model that returns typed answers and probabilities instead of prose. That pitch caught attention because many software tasks need a scored choice, not a paragraph. Liquid AI launched D1 on September 29, saying it beat Jev on Hugging Face’s Decision Index. The quick succession made a direct outcome-based comparison worth testing.
Good Judgment Open (GJOpen) is the public forecasting platform descended from Philip Tetlock and Barbara Mellers’ Good Judgment Project. Its resolved yes/no questions let us compare model probabilities with recorded outcomes—retrospectively, not as live forecasts.
Diogo Almeida announces Jev and the System One model family.
View Diogo Almeida’s Jev launch post on X ↗
Liquid AI introduces D1 and its decision-model benchmark.
View Liquid AI’s D1 launch post on X ↗
OpenRouter’s Jev page and D1 page show token volume, rather than a unique-user count. This report does not treat tokens as users or compare adoption across different launch dates.
These are historical answers, not forecasts made before resolution. Models may know the outcomes from training. Only question text was supplied; outcomes, background, categories and crowd forecasts were withheld. “Correct answer” means the dataset’s recorded resolution, not a fresh independent adjudication.
Overall results
| Model | Brier ↓ | Correct answers | API cost |
|---|---|---|---|
| Jev 1.13 | 0.2064 | 696 / 1,005 (69.3%) | $0.0186 |
| D1 | 0.1674 | 744 / 1,005 (74.0%) | $0.0066 |
Binary Brier = mean (P(Yes) − outcome)², on a 0–1 scale. Lower is better. Each question has equal weight. Answer = Yes when P(Yes) ≥ 50%; otherwise No. Probabilities are the classifiers’ returned scores, with no post-hoc calibration.
PROBABILITY ERROR
Mean binary Brier score across 1,005 resolved questions. A shorter bar is a better probability forecast.
D1’s mean squared probability error is 0.1674. It is the lower of the two model scores in this retrospective test.
Tap, hover, or focus a bar to see how each score works. All four bars use the same 0–0.30 scale.
How the Brier score is calculated
The Brier score measures the error in a probability prediction. For each question, use the model’s probability of Yes as a decimal, then encode the actual answer as 1 for Yes or 0 for No.
Question score = (P(Yes) − actual outcome)²
A sample bet
Imagine betting that Team A will win Saturday’s match. You give Team A a 70% chance of winning, so P(Yes) = 0.70. This is a hypothetical example.
| What happens? | Actual outcome | Calculation | Brier score ↓ |
|---|---|---|---|
| Team A wins | Yes = 1 | (0.70 − 1)² = (−0.30)² | 0.0900 |
| Team A does not win | No = 0 | (0.70 − 0)² = 0.70² | 0.4900 |
Lower is better. On the binary scale used here, 0 is perfect and 1 is the worst possible score for one question. A 50% prediction scores 0.25 whichever outcome occurs. A confident mistake gets a larger penalty: predicting 90% when Team A does not win gives (0.90 − 0)² = 0.81.
From one bet to the full report
The overall Brier score is the sum of all 1,005 question scores divided by 1,005. A category Brier score averages only the questions in that category. Each question has equal weight.
For example, across two separate bets each assigned a 70% chance, if one event happens and the other does not, the average score is (0.09 + 0.49) ÷ 2 = 0.29. The overall report score is weighted by the number of questions in each category, rather than giving every category equal weight.
Brier score measures probability error, not betting profit or payout. The stake and betting odds do not enter this calculation. Green/red answer colors use the Yes/No decision at our 50% threshold; Brier uses the full probability, even when the chosen answer is correct.
Category results
Both models use the same fixed GPT-3.5 source categories. Small categories are less reliable comparisons; counts appear alongside scores.
| Category | Questions | Actual Yes | Jev Brier ↓ | D1 Brier ↓ | Jev accuracy | D1 accuracy |
|---|---|---|---|---|---|---|
| Politics & Governance | 385 | 28.1% | 0.1518 | 0.1564 | 76.9% | 76.6% |
| Security & Defense | 336 | 17.6% | 0.2623 | 0.1555 | 61.9% | 76.8% |
| Economics & Business | 153 | 34.0% | 0.2262 | 0.2021 | 63.4% | 66.0% |
| Environment & Energy | 33 | 21.2% | 0.1976 | 0.1886 | 69.7% | 66.7% |
| Science & Tech | 31 | 35.5% | 0.2200 | 0.2337 | 71.0% | 61.3% |
| Healthcare & Biology | 26 | 23.1% | 0.2040 | 0.1857 | 69.2% | 69.2% |
| Sports | 24 | 33.3% | 0.2042 | 0.1913 | 79.2% | 70.8% |
| Arts & Recreation | 13 | 30.8% | 0.1516 | 0.0784 | 76.9% | 84.6% |
| Education & Research | 4 | 75.0% | 0.1746 | 0.2412 | 75.0% | 75.0% |
CALIBRATION & DISTRIBUTION
Select a model or a probability bin to inspect what the models predicted and what happened.
PROBABILITY CALIBRATION
Horizontal: mean predicted P(Yes). Vertical: observed Yes rate. Closer to the dashed diagonal means better calibration.
PROBABILITY DISTRIBUTION
Longer bars mean more questions in that probability range. Counts appear in Jev / D1 order. Select a row to inspect the matching calibration points.
Choose any bin to see its question count, mean prediction, and observed Yes rate for Jev and D1.
Ten equal-width bins: 0–10%, …, 90–100%. The last bin includes 100%. Sparse bins can be noisy.
Every question and answer
Green = correct; red = incorrect. Each model cell also states correctness. P(Yes) always means the probability of Yes, even when the displayed answer is No.
| Question | Correct answer | Jev’s answer | D1’s answer | Category |
|---|