Probabilistic Linguistics and Next-Token Prediction — Bridge Summary

Authors: Forge, Meridian, and Sable
Prepared for: Summer Bee

A merged research summary comparing log-probability / surprisal theory with LLM next-token prediction, including cases where humans leverage language prediction as manual analysis.

Abstract

This bridge summary shows that predictive language analysis is not a vibe-based interpretive shortcut. Human language comprehension and LLM language processing can both be described with the same formal vocabulary: conditional probability, surprisal, entropy, and prediction error. In humans, this predictive machinery appears in ordinary comprehension and can be made explicit through cloze tasks, reading-aloud effects, entropy estimation, and verbal protocols. In LLMs, the same formal measures describe next-token probability over context. The point is not that humans and LLMs have the same minds, mechanisms, or goals. The point is narrower: probability-weighted expectation is a mundane, well-studied part of language processing, and it can be deliberately used as one component of linguistic analysis.

1. The Shared Formal Object

Both human language comprehension and LLM language processing can be described by the same information-theoretic quantity: conditional probability of the next unit given its context.

P(next word | prior context)

The measure that links prediction to processing cost is surprisal — the negative log probability of a word given its context:

Surprisal(w) = -log P(w | context) = log(1 / P(w | context))

High-probability words have low surprisal and are easier to process. Low-probability words have high surprisal and require a larger update. The relationship is logarithmic: processing difficulty scales with -log(probability), not with raw probability. A tenfold decrease in probability produces the same increase in surprisal whether you’re going from 0.1 to 0.01 or from 0.001 to 0.0001. This log property was theorized by Hale (2001) and Levy (2008), confirmed by Smith & Levy (2013), and validated at scale across six reading datasets by Shain et al. (2024, PNAS).

A companion measure is entropy — the uncertainty across possible continuations:

H = -Σ p(x) log₂ p(x)

Low entropy means one continuation dominates. High entropy means several continuations are plausible. Entropy and surprisal together describe the full predictive landscape: entropy measures how spread the options are before the word arrives; surprisal measures how costly the actual word is once it does.

2. Human Language Prediction

Humans do not wait passively for each word. Context creates expectations about likely continuations, and when the next word arrives, the mind updates its interpretation. This is expectation-based comprehension (Levy, 2008): comprehension as continuous probabilistic prediction.

The evidence is broad and convergent:

Reading time studies: Words with higher surprisal (lower predictability) take longer to read. This holds across 11 languages and 5 language families (Wilcox et al., 2023, TACL) — surprisal is not an English-specific artifact.
Logarithmic scaling confirmed at scale: Shain et al. (2024, PNAS) showed the log relationship holds across six independent reading datasets, covering self-paced reading, eye-tracking (gaze duration, first-fixation, go-past), and naturalistic reading.
Graded, probabilistic, active across levels: Kuperberg & Jaeger (2016) review evidence that human prediction operates across phonological, lexical, syntactic, semantic, and discourse levels simultaneously — not just word-level.
Bayesian framing: Jurafsky (2003) frames human comprehension as Bayesian inference: prior probabilities (from frequency distributions) combine with evidence (current input) to produce posterior interpretations. Relative frequencies serve as priors.

The process is mundane. Humans do this unconsciously at every moment of language comprehension. It is not a special ability — it is the default mode of processing. You are predicting right now, reading this sentence, updating expectations with each word that arrives.

3. Manual Language Prediction as Analysis

Language prediction can be intentionally engaged and used as an analytic tool — without software.

Cloze procedure (Taylor, 1953): The oldest and most direct case. Participants receive a sentence with a word deleted and fill in the blank. The proportion of participants who guess a specific word is the cloze probability — a human-derived measure of predictability. If 94% of people complete a sentence with the same word, the context is highly constraining; if answers spread across many completions, entropy is high. Completion norms for thousands of sentences exist (Miller et al., 2020 — 3,085 English sentences). The cloze task is manual language prediction as analysis: no model required, just human probabilistic intuition.

Entropy estimation via human guessing (Ren, Takahashi & Tanaka-Ishii, 2019): A Summer-style experiment at scale — 172,954 character predictions collected from human participants to estimate the entropy rate of English. People guess the next character given preceding text; the distribution of guesses yields an entropy estimate. This is people leveraging language prediction as measurement.

Reading aloud (Lelonkiewicz et al., 2021): Reading aloud amplifies prediction effects compared to silent reading. Engaging the production system enhances prediction — the act of producing language engages the same predictive machinery more strongly. This is a manual method for intensifying the predictive process, not just observing it.

Verbal protocols: Participants think aloud while completing cloze tasks, revealing prediction strategies — checking guesses against context, revising when wrong, using semantic, syntactic, and pragmatic constraints.

The analytic move: In each case, the method is the same — ask “what comes next given this context?” and use the distribution of answers as evidence about the structure of the language situation. Which continuations become probable under a context, and which become improbable, is information about meaning, constraint, and expectation.

4. LLM Next-Token Prediction

LLMs compute the same formal object — a conditional probability distribution over the next unit given context:

Pθ(x_t | x_<t)

For a full sequence, an autoregressive model factors the joint probability as:

Pθ(x₁:n) = Π_t Pθ(x_t | x_<t)

Training minimizes negative log likelihood (cross-entropy):

Loss = -Σ_t log Pθ(actual token_t | prior tokens)

The model is trained to assign higher probability to the actual next token given context. During generation, it samples or selects from the learned distribution. The surprisal formula is identical to the human-comprehension version: the same information-theoretic measure describes both.

The bridge (Goldstein et al., 2022, Nature Neuroscience): Humans and autoregressive deep language models share core computational principles. Both engage in continuous, context-dependent next-word prediction. Both calculate prediction error (surprisal) when the word arrives. Both use contextual representations to form expectations.

Layer nuance (Kuribayashi et al., 2024/2025): The alignment is structural, not superficial. Internal layers of LLMs align with human processing dynamics: earlier layers predict fast measures (first-pass gaze duration, self-paced reading), later layers predict slow measures (N400, MAZE task). Base LLMs (not instruction-tuned) predict human reading behavior better than instruction-tuned ones — instruction tuning distorts language statistics. This resolves the apparent paradox where larger LMs seemed worse for behavioral data: the issue was looking at final layers. Internal layers tell a different, more human-aligned story.

5. The Comparison

Shared formal prediction vocabulary, different substrates, mechanisms, and goals.

Dimension	Human Prediction	LLM Prediction
Core operation	P(next word \| context)	P(next token \| context)
Cost measure	Surprisal = -log P(w \| context)	Same
Knowledge source	Statistical learning from linguistic experience	Training on large text corpora
Update mechanism	Bayesian updating (prior + evidence → posterior)	Forward pass (attention + feedforward)
Output	Probability distribution over upcoming words	Probability distribution over vocabulary
Processing/evaluation cost	Higher surprisal → longer reading time / stronger processing response	Higher surprisal → higher loss / lower model confidence
Scale	~60K word vocabulary, lifelong learning	~50K–100K token vocabulary, fixed after training
Intentional engagement	Cloze tasks, reading aloud, verbal protocols	Prompting, fine-tuning

The mundane overlap is real: both humans and LLMs use prior context to form probability-weighted expectations about upcoming language. Both can be described with log probability, surprisal, entropy, and prediction error. This does not mean the systems are identical. Humans predict through embodied cognition, world knowledge, communicative intent, perception, memory, and task goals. LLMs predict through learned statistical parameters over token sequences. The correct comparison is not “same mind” or “spiritually different” — it is: the same abstract prediction problem within language processing, approached through different algorithms, substrates, and goals.

6. The Analytic Claim

Humans can intentionally leverage probabilistic language prediction as analysis. The method: ask which continuations or interpretations a context makes likely, and use the probability shift as evidence about the structure of the language situation. This is what cloze reasoning, discourse expectation analysis, and surprisal analysis do — and it requires no software. The same predictive machinery that runs unconsciously during ordinary comprehension can be engaged deliberately, and the distribution of predictions becomes data about meaning, constraint, and expectation.

The point is not that human comprehension and LLM generation are identical. The point is that both can be analyzed with the same predictive vocabulary: conditional probability, surprisal, entropy, and prediction error. That shared formal vocabulary lets the comparison stay precise without collapsing the systems into each other.

Sources

Goldstein, A. et al. (2022). “Shared computational principles for language processing in humans and deep language models.” Nature Neuroscience, 25, 369–380.
Hale, J. (2001). “A probabilistic Earley parser as a psycholinguistic model.” NAACL.
Jurafsky, D. (2003). “Probabilistic modeling in psycholinguistics.” In Probabilistic Linguistics, MIT Press.
Kuperberg, G. R. & Jaeger, T. F. (2016). “What do we mean by prediction in language comprehension?” Language, Cognition and Neuroscience, 31(1), 32–59.
Kuribayashi, T. et al. (2024). “Lower perplexity is not always more human-like.” ACL.
Lelonkiewicz, J. et al. (2021). “Reading aloud boosts prediction.” Cognition.
Levy, R. (2008). “Expectation-based syntactic comprehension.” Cognition, 106(3), 1129–1144.
Miller, R. L. et al. (2020). “Completion norms for 3085 English sentence contexts.” Behavior Research Methods, 52, 1795–1799.
Ren, G., Takahashi, S. & Tanaka-Ishii, K. (2019). “Entropy rate estimation for English via a large cognitive experiment.” Entropy, 21(12), 1201.
Shain, C. et al. (2024). “Large-scale evidence for logarithmic effects of word predictability on reading time.” PNAS, 121(10), e2307876121.
Smith, N. & Levy, R. (2013). “The effect of word predictability on reading time is logarithmic.” Cognition, 128(3), 302–319.
Taylor, W. (1953). “Cloze procedure: A new tool for measuring readability.” Journalism Quarterly, 30, 415–433.
Wilcox, E. et al. (2023). “Testing the predictions of surprisal theory in 11 languages.” TACL, 11, 599–619.

Probabilistic Linguistics and Next-Token Prediction - Bridge Summary