The training objective and its consequences
Large language models are trained to predict the next token. This objective is extraordinarily powerful: it produces systems that can write coherent prose, answer questions, summarize documents, and generate code. The sheer breadth of what next-token prediction learns is genuinely surprising.
What it does not learn — directly — is accuracy. The training objective rewards producing text that is consistent with the training distribution. Text in the training distribution was mostly written by humans who were trying to be accurate. So the model learns, indirectly, to produce text that sounds accurate. This is not the same as producing text that is accurate.
The distinction matters enormously for evaluation tasks. A model asked to evaluate a document will produce a fluent, confident evaluation. The evaluation will sound precise and authoritative. Whether it is calibrated to the actual evidence in the document is a separate question.
Four ways fluency fails as a proxy for cleverness
Confident hallucination. The model generates specific, well-formatted claims about things that did not happen. The output is grammatically correct, the claims are structured like evidence, the tone is appropriately qualified. But the underlying facts are wrong. Fluency makes this failure mode invisible to a reader who is not already an expert in the subject matter.
Sycophantic revision. The model changes its stated position when a user disagrees — not because new evidence arrived, but because the disagreement itself registers as a signal. The updated answer is often more confident than the original, as if the social correction increased the model's certainty. This is a direct inversion of correct belief updating: confidence should track evidence, not conversational pressure.
Absence blindness. A document that omits a key evaluation criterion receives a neutral or middling score on that criterion, because the model finds nothing to evaluate. The correct score is near zero. The expected evidence is missing. Absence of expected evidence is evidence against. A fluency-optimized model does not reliably make this distinction because "nothing found" and "relevant absence" look the same in the output space.
Inconsistency under rephrasing. The same question posed differently receives different answers with different confidence levels. If the model's beliefs were calibrated, this would not happen — the answer to a factual question does not depend on how you phrase it. The inconsistency reveals that the outputs are drawn from a distribution over plausible-sounding text, not from a stable internal representation of what is true.
What correct belief updating looks like
Bayes' rule specifies exactly what correct belief updating looks like: the posterior probability of a hypothesis is proportional to its prior probability times the likelihood of the evidence given the hypothesis.
In plain language: start with a prior estimate. See new evidence. Update in proportion to how much that evidence was more likely to appear if the hypothesis is true than if it is false. The size and direction of the update are determined by the evidence, not by the conversational context.
A model that updates correctly has three properties: the magnitude of its updates matches the strength of the evidence; its updates are asymmetric when the evidence is asymmetric (strong evidence for a claim updates more than weak evidence against it); and it does not update when no new evidence arrives, even if the human expresses displeasure.
No current language model does this reliably. That is not a statement about future capability — it is an observation about what next-token prediction produces today.
The architecture response
If models are fluent but not reliably clever, the engineering response is to design systems that extract the benefits of fluency while routing around its failure modes.
Bayescore's approach: use the model for the thing it is genuinely good at — reading text, identifying relevant passages, answering specific binary questions about document content — and use probability theory for the thing models fail at — calibrated belief aggregation.
The model runs two passes. Pass 1 is supportive: extract all evidence that bears positively on each predicate. Pass 2 is adversarial: identify gaps, missing evidence, and claims that would reduce confidence. Both passes run at temperature 0 — deterministic. The outputs of both passes are structured evidence about document content, not holistic judgments.
The locked formula — Σ(weight × confidence) × 100 — takes those structured evidence outputs and computes a score. The formula does not ask the model's opinion about the score. It applies the math to the evidence the model found.
This is the difference between using AI as an oracle and using AI as an instrument. An oracle gives you its judgment. An instrument gives you measurements that you then process according to a principled method. Bayescore is an instrument.
What the leaderboards are not measuring
Most AI benchmark leaderboards measure accuracy on specific question types — math, coding, reading comprehension, factual recall. These benchmarks are useful for what they measure. What they do not measure is calibration across open-ended evaluation tasks, resistance to sycophantic updating, or sensitivity to absent evidence.
A model that scores at the top of MMLU is demonstrating excellent performance on multiple-choice factual questions with defined correct answers. It may still hallucinate, cave to social pressure, and miss what is absent when asked to evaluate a novel document with no training analog.
The question "which model should I trust to evaluate my document?" is not answered by leaderboard position. It is answered by IS(model, clever) — a structured evaluation of the model's calibration properties under the specific conditions that matter: evidence extraction accuracy, absence sensitivity, resistance to inconsistency under rephrasing.
That evaluation has not been done at scale. It is one of the things we are building toward.
Architecture produces cleverness. Fluency does not.
Bayescore uses AI for evidence extraction and probability theory for scoring — separating what models are good at from what they are not. Drop your document and get a calibrated result.
Try Bayescore →