A field note from a live forecasting system
We made the AIs sit a prediction exam.
Every tutoring company publishes “topics to revise” before the HSC. We are publishing something riskier: question-level probabilities from a six-model AI panel, pre-registered now and publicly scored against the real 2026 papers in November. This is how the system works.
↳ Every probability gets marked against the real paper in November 2026.
The short version
Anyone can publish a revision list.
We are publishing predictions that can be marked wrong.
“Focus on equilibrium this year.” “Projectile motion is due.” Advice like this circulates before every HSC, and it is unfalsifiable by design: vague enough that something on the paper will always seem to confirm it. Nobody goes back in November and marks the tipsters.
At Intuition Education we decided to do the opposite. We built a system in which six frontier models, given identical official evidence, independently predict what the 2026 NSW HSC papers will actually ask — down to individual questions, with a probability attached to every call. The predictions are being registered now, before the exams. In November, we score them against the real papers using Brier scores and publish the result, hit or miss.
Before asking anyone to trust a 2026 forecast, we made the panel sit an exam of its own: we hid the 2025 papers, had every model predict them from earlier years only, and graded the predictions strictly against what NESA actually set. The middle of this field note is that backtest — the hits, the misses, and the one failure mode that taught us the most.
01 The evidence pack
The markers telegraph next year’s paper.
Each subject gets one consolidated evidence pack built entirely from official NESA documents. No tutor folklore, no “my school reckons”. If a claim is not in the pack, the models are instructed not to make it.
For every year of the current syllabus — 2019 or 2020 through 2025, depending on the subject — the pack contains three documents: the exam paper itself, the marking guidelines, and the marking-centre feedback. The third one is the quiet gold. Each year, the people who marked tens of thousands of scripts write down exactly where students failed. Exam committees read those notes too, and they write questions where students demonstrably fail — that is what a discriminator is. Feedback weaknesses recur as future questions often enough that the prompt makes it an explicit analytical method.
The pack is completed by the full current syllabus, then consolidated to one plain-text file per subject — papers, guidelines and feedback in year order, every source delimited by a header. For the two pilot subjects that came to roughly 249k tokens for Chemistry and 151k tokens for Maths Extension 1: large, but comfortably inside a frontier context window, so every model reasons over the identical, complete record.
What was actually asked, section by section, mark by mark.
What earned the marks — the official criteria and sample answers.
Where students failed. The signal committees mine for next year’s discriminators.
The complete space of what is allowed to be examined.
One file per pilot subject. Identical for every model on the panel.
02 The panel & methods
Six models, five labs, one disciplined harness.
Every model receives an identical evidence prompt and must return a strictly validated JSON schema. No live tools, no browsing, no qualitative evasion. The only variable is the model.
Prompting frontier models for exam predictions without a schema yields qualitative fluff: long essays on "emerging themes" that cannot be evaluated, aggregated, or scored. To turn generative text into computable intelligence, we built a four-stage pipeline: taxonomy binding, schema decomposition, automated validation & review gates, and mathematical consensus synthesis.
The failure mode: Broad exams touch nearly every syllabus module. Topic-level predictions naturally saturated at p ≥ 0.90 across all models, providing zero actionable guidance and zero scoring discrimination.
The discipline: Schema v1.1 forces models to emit concrete items: specific scenario setups, mark allocations (e.g. 7-mark multi-step), paper section, cognitive role, MCQ distractor logic, and the exact past marking-feedback weakness targeted.
The Consensus Engine — 5 Synthesis Algorithms
Every response is checked against canonical NESA syllabus code trees. Any hallucinated topic code or missing module causes immediate rejection before ingestion.
When Anthropic sibling models (Fable 5 and Opus 5) agree within ±0.10, their individual weights are discounted to 0.75×. Correlated architectures cannot hijack the ensemble.
Standard deviation above 0.25 across the panel flags a genuine evidentiary fork. We never average away disagreement; we surface it as a high-value strategic branch.
Cross-model agreement on specifics absent from the evidence pack (e.g. memorised training-set artifacts) is collapsed to a single vote rather than treated as independent corroboration.
An automated checklist audits macro/legal currency drift and training contamination. 32 pre-launch flags render as "pending human verification" until audited by subject leads.
All agent traces are audited to ensure zero live-web retrieval tools were active during generation. Predictions must reason strictly within the closed universe of the evidence pack.
03 The backtest
We hid 2025 and made the panel predict it.
Before publishing a single 2026 number, we rebuilt the evidence packs with everything from 2025 onwards removed, had all six models predict the 2025 Chemistry and Maths Extension 1 papers from 2019/2020–2024 evidence only, and scored the predictions against the real papers using NESA’s official marking-guideline mapping grids. Two subjects, one year — a pilot, not a proof. But a revealing one.
Every p ≥ 0.9 “this topic will be examined” call, across the whole panel, was correct. Necessary — and it separates nothing, because real papers touch nearly every topic.
Graded strictly — the form must match, not just the topic — roughly half of all question-level predictions recognisably appeared, far above what naive guessing would land.
The field’s biggest collective hit: Chemistry Q33, the 7-mark back-titration, predicted by five models — DeepSeek nearly verbatim.
| Model | Chemistry | Maths Ext 1 |
|---|---|---|
| GPT-5.6 Sol | 0.179 | 0.325 |
| Claude Opus 5 | 0.192 | 0.187 |
| Grok 4.6 | 0.200 | 0.259 |
| DeepSeek V4 | 0.217 | 0.389 |
| Gemini 3.1 Pro | 0.251 | 0.512 |
| Claude Fable 5 | 0.281 | 0.285 |
| In-sample base-rate bar | 0.248 | 0.16 |
Chemistry: four of six models beat the base-rate bar. Extension 1: nobody did — the 2025 paper concentrated its ≥ 4-mark questions in just two topics, and the whole panel over-predicted breadth. The panel runs overconfident on “substantial”, and we say so.
| Model | Chemistry | Maths Ext 1 |
|---|---|---|
| GPT-5.6 Sol | 0.600 | 0.571 |
| Claude Fable 5 | 0.483 | 0.648 |
| Grok 4.6 | 0.500 | 0.558 |
| Gemini 3.1 Pro | 0.444 | 0.528 |
| Claude Opus 5 | 0.438 | 0.517 |
| DeepSeek V4 | 0.439 | 0.375 |
04 Different models, different edges
The ensemble is the product.
No single model won the backtest. Different models won different dimensions — which is the strongest argument for running a panel at all.
Best chemistry calibration and hit rate.
Top Brier on chemistry substantials (0.179) and the top chemistry hit rate (0.600), from the fewest, broadest predictions.
Most consistently calibrated.
Strong on both subjects (0.192 and 0.187 — the only model near the top of both Brier columns). The panellist you would trust with the probabilities.
Sharpest question-level maths — but overconfident on breadth.
The exact Q12 induction and the Q14(e) fusion were the board’s two sharpest calls, and its 0.648 led Extension 1. Its chemistry calibration (0.281) was the panel’s weakest.
Breadth backfired.
The most and most specific predictions — including a near-verbatim back-titration — but 18 of its 36 Extension 1 predictions missed, many in off-taxonomy topics it invented.
05 Economics & what ships
The expensive ingredient is not tokens.
The entire two-subject pilot — including the full 2025 backtest — cost about $2 of API spend on the metered models. The other four models ran through in-session and CLI agent harnesses on existing subscriptions.
Those numbers are worth sitting with. The marginal cost of having six frontier models read every paper of the current syllabus and produce structured, probabilistic forecasts is now pocket change. What actually consumed the effort was everything around the tokens: building leakage-guarded evidence packs, pinning the schema so outputs are comparable, validating every reply against the taxonomy, designing synthesis rules that survive corpus echoes, and grading a backtest strictly enough to trust. The moat is harness design and verification, not API budget.
What students actually get
Question styles to look out for
Per-subject study guides on the Intuition website: the question styles the panel agrees on, the skill chain each one demands, and the traps the marking centre flags year after year — with the panel’s probability attached as a confidence signal. Styles to prepare for, not guarantees: the backtest showed examiners keep the skill and twist the format, so the advice is to practise the skill chain, not memorise a question.
Predicted practice papers
Full practice papers that realise those styles in the real exam’s structure — something to sit under timed conditions, not a tip sheet.
The November scorecard
After the real 2026 papers are sat, every registered prediction gets Brier-scored and published — the misses alongside the hits.
The idea to keep
A revision list can never be wrong.
That is exactly what is wrong with it.
If exam-preparation insight is real, it should survive being written down as a probability and marked against the actual paper. Ours is registered. November will grade it — and whatever the scorecard says, we will publish it.
Disclosure & method note
I am the CEO of Intuition Education, which sells HSC preparation and will publish the prediction products described here. All backtest figures come from our holdout scoring of two pilot subjects — Chemistry and Maths Extension 1 — against the real 2025 papers, graded via NESA’s official mapping grids; they should be read as a pilot, not a proof. Model behaviour and prices change; the comparisons describe the runs we recorded, not a permanent ranking. NESA is not affiliated with, and does not endorse, this work.