LLMs, Turing Tests and Chinese Rooms
Summary: Emma Borg (2025) argues that LLM outputs are genuinely meaningful via derived intentionality, but that LLMs neither assert content nor possess original intentionality — and that the latter is a feature, not a bug.
Sources: Academia/LLMs Turing tests and Chinese rooms the prospects for meaning in large language models.pdf
Last updated: 2026-05-06
The deadlock
Debates about LLMs are stuck between two positions defined by two thought-experiments. Turing argued that passing a behavioural test (the Imitation Game) is sufficient for intelligence. Searle countered in his Chinese Room argument that passing such a test is not sufficient — a system can produce correct outputs by manipulating purely syntactic symbols with no semantic content at all.
LLMs have moved the debate into a new phase without resolving it. They pass the Turing test in practice. Sceptics (Searle, Mallory, Titus) say the outputs are meaningless “stochastic parrots.” Believers say the systems show “sparks of AGI.” Borg argues both camps have something right.
Three-level analysis
Borg distinguishes three questions that are often conflated:
- Are LLM outputs meaningful? — Yes.
- Do LLMs assert content? — No.
- Do LLMs possess original intentionality? — No, and this is fortunate.
1. Meaningful outputs: derived intentionality
The Sceptic demands an externalist connection between sign and world before attributing meaning. LLMs lack direct perceptual contact with the world. But Borg argues that LLM outputs inherit meaning via semantic deference — the same mechanism that allows ordinary humans to use “elm” correctly without being able to distinguish an elm from a beech. Words mean what they do because there is a community practice of using them that way; a current speaker (or system) defers to that practice.
LLM outputs are tokens of types that belong to an established human communicative system. They are produced in ways that are syntactically well-formed and contextually appropriate — far beyond the “word salad” a purely random mechanism would produce. On Borg’s A-style intention-based semantics, this is enough for genuine type-level semantic content.
Internalist theories of meaning (distributional semantics, inferential role semantics, conceptual role semantics) also support the case: tracking statistical co-occurrence across a large enough corpus may just be tracking inferential and conceptual relations — tracking the proxy may be tracking the property.
2. LLMs do not assert
The norm governing LLM outputs is word-occurrence probability, not truth. Hallucinations emerge from the same mechanism as true statements; the system cannot distinguish them. Because truth does not get a grip, LLMs cannot be in the business of asserting content.
This connects to austin-speech-acts: assertion is an illocutionary act that requires the speaker to be committed to the truth of what is said. LLMs lack this commitment because they lack sensitivity to truth. Their outputs should be flagged as such when they enter the public sphere.
3. LLMs lack original intentionality — and should
Original intentionality (Searle’s term) requires rich environmental embedding, long-term goals, stable point of view, and initiated interaction. LLMs currently have none of these. Borg’s key move is to argue this is desirable: a system that met these requirements would be a candidate for moral consideration. We should be glad that LLMs are not yet agents.
Key distinctions
| Level | LLMs? | Condition required |
|---|---|---|
| Meaningful outputs | Yes | Semantic deference, type-level content |
| Representation of semantic properties | Open question | Depends on philosophy of content |
| Assertion | No | Requires truth sensitivity |
| Original intentionality | No | Requires agency, embodiment, goals |
| Consciousness | No | Requires original intentionality |
Connections
The paper implicitly connects to austin-speech-acts: Borg’s claim that LLMs do not assert maps directly onto Austin’s distinction between the sentence meaning something (locutionary) and a speaker doing something with it (illocutionary). LLMs produce meaningful locutions without performing genuine illocutionary acts.
The wider philosophical question — whether the gap between LLM capability and genuine cognition is contingent or fundamental — is addressed from the biological angle in jaeger-relevance-realization and tested empirically in illusion-of-thinking. The synthesis is in llm-and-mind.