A field note from a production AI system
The best model is not the best product.
The market keeps asking which AI is smartest. Building an exam-marking system taught me to ask a different question: which combination of models, software and human judgement makes the whole process work?
What the product needs
↳ A leaderboard ranks models. A business has to ship a system.
The short version
We have become very good at measuring models.
Businesses need to measure work.
Every model release now arrives like a sporting result. A point gained on a benchmark becomes evidence that one company is winning, another is falling behind, and artificial general intelligence has moved a little closer. The scoreboard is easy to understand, so it becomes the story.
I do not think public evals are the problem. We publish our own model evaluation. They are valuable indicators. The mistake is treating a general ranking as a product decision. As the market starts asking where the revenue and return from AI will come from, that distinction matters.
Commercial value rarely comes from giving one enormous prompt to the model at the top of the table. It comes from understanding a real process, deciding where judgement is needed, keeping ordinary logic in ordinary software, and evaluating models against the particular work they will perform.
That changes what “best” means. A public benchmark asks whether a model can solve a broad class of problems. A product evaluation asks whether a system can complete one particular job to the required standard, at a sustainable cost, and recover cleanly when something goes wrong. Both questions matter. They are not the same question.
01 The first attempt
We started with the obvious architecture.
Give a capable multimodal model the exam, its solutions and criteria, then ask it to mark everything. We tried Claude Sonnet, Claude Opus, GPT-5.5 and Gemini Pro. They could all do impressive work. None made the process commercially sound.
Exam marking makes the problem unusually visible. What looks like one instruction—“mark this exam”—is really a chain of different responsibilities: read handwriting, understand diagrams, locate answers, infer intent, apply question-specific criteria, award marks and write useful feedback. A model can be capable of every one of those things while the combined call remains fragile.
Our first version coupled all six responsibilities inside one model request. When it worked, the result looked magical. When it failed, content went missing, an answer could be associated with the wrong question, or the final mark drifted from the stored criteria. The output did not tell us which part of the process had broken.
No inspectable hand-off between stages
Looks complete
Failure location unknownThe answer was not to add more agents for the sake of it. It was to create explicit hand-offs. Once reading, matching and judgement produced inspectable outputs, we could test each boundary, use ordinary software where rules were sufficient, and choose a model for the work at that stage.
Content disappeared
Student work was sometimes missed during the same call that was also trying to judge it.
Answers moved
Responses could be linked to the wrong question—or not linked at all.
Marks drifted
Criteria adherence weakened when the model had too many different jobs in one request.
Errors hid
When the final mark was wrong, it was difficult to isolate whether reading, matching or judgement had failed.
02 Process first
The breakthrough was not a smarter model.
The biggest quality improvement came from separating image extraction from marking. Once reading and judgement became different stages, each could be tested, improved and routed independently.
The assessment is already structured before a student submits: the system knows the questions, solutions, available marks and marking criteria. MARK does not ask a model to rediscover that structure from a PDF. It turns the submission into records that can be compared with it.
Page splitting, content-density checks and answered-versus-unanswered checks use ordinary software rules. Pages are extracted concurrently, alone or in small batches, and only structured text—including descriptions of diagrams—crosses into the marking stage. The models remain configuration choices, not architectural dependencies.
Prepare & inspect
Split pages, inspect content density and choose single-page or small-batch extraction.
No AI modelRead the submission
Read handwriting and turn diagrams into structured descriptions, one page or small batch at a time.
Gemini 3 FlashMatch every answer
Identify answered questions, join responses to question numbers, then use page position and meaning when numbering is unclear.
Software + model fallbackMark each response
Compare the structured answer with the stored solution, available marks and criteria. Recheck work when confidence or software checks indicate risk.
Flash · Luna when neededPages run concurrently and can travel in small batches. Dense pages are isolated; sparse pages can be grouped. We avoid asking one model request to carry the whole exam.
The marking stage sees structured text, including descriptions of diagrams—not the source image. That makes extraction fidelity independently measurable.
An MCQ is normally re-run only when extraction fails. A diagram can receive a second look even when the first model is moderately confident.
Technical detail What happens when an answer runs across pages? +
Splitting the process has trade-offs. Students do not always answer in order, and one response can run across several pages. MARK preserves page position and extracted question numbers. It matches explicit numbers first, then uses position and meaning to reconnect fragments when the labelling is ambiguous. This has taken iteration, and it remains a good example of why the process—not only the model—needs evaluation.
03 What we measured
Our own evaluation changed the model decision.
MARK has been tested on approximately 200 real homework and exam submissions using blind human review. We assess transcription, awarded marks, false positives and negatives, confidence, feedback, failures, speed and API cost. The detailed rubric and data remain private, but the product decisions came from that evidence—not from a public ranking.
Real homework and exam submissions, reviewed blind and re-run when model choices change.
For the model mix versus using Claude Opus or GPT-5.6 Sol throughout the workflow.
In our blind review, the AI marking was more consistent than human marking and often produced better, more detailed comments.
What our product evaluation actually scores
Handwriting, notation and diagrams are checked independently of the mark.
Awarded marks, false positives and false negatives are compared with reviewed answers.
Different response types are judged against their own requirements, not one generic score.
Correctness, specificity, tone and usefulness to the student are reviewed blind.
Timeouts, failed endpoints, retries and confidence behaviour count as product quality.
Tokens, API spend and human review are measured per successful piece of work.
The result that did not fit the hype
The pipeline is not dramatically faster.
End-to-end processing is similar to the monolithic approach and can be a little longer. That is acceptable because marking is asynchronous: the student submits work and receives a notification when it is ready. Here, quality and economics matter more than producing a 500-millisecond demo.
04 What MARK changes
The value is larger than a cheaper first mark.
MARK changes who spends time on what, how quickly a student receives useful information, and how consistently the service can be delivered. Those benefits reinforce one another.
Feedback arrives while the work is still fresh.
Students receive detailed comments sooner and can immediately ask follow-up questions. We see that behaviour in product usage as well as in conversations with students.
Time moves from producing every comment to checking quality.
People currently verify the correctness of the mark and the usefulness of the feedback. They do not have to create every part of the first pass from a blank page.
The same criteria are applied to every response.
The system can be meticulous about state symbols, missing content and partial intent without becoming tired or varying between markers. Human QA remains the safeguard.
Better economics make richer feedback sustainable.
The current model mix costs roughly one fifth of using Opus or GPT-5.6 Sol throughout. Lower delivery cost supports scale without reducing the quality threshold.
We are not yet claiming a measured causal improvement in academic results. The nearer-term evidence is clearer: faster feedback, more follow-up questions, more consistent marking, stronger comments and less marking effort. Those are the conditions from which better learning outcomes can be built.
05 A model portfolio
“Best” depends on the work being bought.
I now find it more useful to think in roles. Frontier, workhorse and budget models are not a podium. They are different tools with different commercial jobs.
Long-form essays, ambiguity and second opinions where extra capability changes the answer.
Reading pages, routine marking and feedback where quality is proven and volume is high.
Simple classification and orchestration. MARK does not currently use this tier for marking.
MARK currently uses frontier and workhorse models. Intu AI uses budget models elsewhere for narrow orchestration tasks. The labels describe roles, not permanent identities; as models improve, today’s frontier capability becomes tomorrow’s workhorse economics.
This is why a cheaper model is not automatically the commercial choice, and a frontier model is not automatically wasteful. Quality remains the entry requirement. After that, the relevant comparison is the total cost of a successful task: how many tokens it uses, how often it must be retried, how dependable the serving layer is and how much human review its output creates.
For MARK, workhorse models cleared the quality bar for most bounded stages. We could then buy frontier reasoning only where it made a measurable difference—long-form essays, ambiguity and uncertain responses. That mix reduced model and API cost by roughly five times compared with running Opus or GPT-5.6 Sol throughout.
Where the return comes from
Intelligence becomes valuable when it changes a real process.
MARK reduces the time people spend marking, improves consistency and returns feedback to students far sooner. We see the shorter feedback loop in both usage data and conversations: students ask more follow-up questions while the work is still fresh. It also reduces the cost of delivering that service.
That is a more useful commercial story than saying we have access to the smartest model. The product advantage sits across the workflow, our structured assessment data, our own evaluation, the model-selection rules and the operational experience of making all of it dependable.
Human QA currently checks marking correctness and feedback quality. The long-term goal is more autonomous marking, but autonomy should be an outcome of accumulated evidence. It is not something we earn by putting a frontier model’s name on the architecture diagram.
This is where the return on AI becomes tangible. It is not only labour removed from an existing process. It is a better service: more consistent marking, richer comments, faster turnaround and a student who is more likely to ask the next question while the original work is still in mind. The system changes the learning loop, rather than merely placing AI inside the old one.
Map the process
Define inputs, transformations, judgement points and failure states before choosing a model.
Keep rules as software
Use ordinary code for work that does not require interpretation or judgement.
Build your own evaluation
Measure the quality, cost and reliability of your tasks—not somebody else’s average.
Buy capability selectively
Use frontier intelligence when the work demands it; let workhorses carry the volume.
The idea to keep
A benchmark tells you what a model can do.
A product proves what a system can do reliably and economically.
The commercialisation race will not be won only by the company with the most intelligent model. It will also be won by teams that turn increasingly capable models into well-designed, measurable processes—and know when not to use the biggest one.
Disclosure & method note
I am the CEO of Intuition Education and have been directly involved in building Intu AI. The MARK findings in this article come from our internal product evaluation of approximately 200 real submissions. Model performance, endpoint reliability and prices change; the named comparisons describe the systems and routes we tested, not a permanent universal ranking.