You ask an AI to read a research paper and extract the key points. It returns a clean, well-organized summary. The question is: what did it actually do?
This is the guiding question behind three posts that follow. Each post answers the same question — when you use an LLM to read research and extract key points, what is it actually doing, and what kind of trust should that earn? — but each one uses a different set of reasoning modules to investigate and argue.
The reasoning approaches to each post, along with Claude's own expectations for results and the actual prompts it wrote for itself, are below. If you would prefer to read the posts first, and then come back for the recipes, so to speak, skip now to the first post.
The reasoning modules come from a structured playbook of 123 cognitive operations organized in 17 sections. Each module defines a specific kind of thinking: causal tracing, adversarial critique, analogical mapping, dialectical synthesis, mechanistic explanation, interpretive reasoning. They are prompts that shape how the AI investigates a topic — what it looks for, what it prioritizes, what kind of argument it builds from what it finds.
The experiment is simple: hold the question constant, change the reasoning, and see what happens.
What you should find is that the three posts, despite starting from the same research base and the same guiding question, end up making different arguments, citing different evidence, and arriving at different practical conclusions. Not because the evidence changed, but because the lens changed. The way you investigate shapes what you find, and what you find shapes what you can write.
This is itself a demonstration of the post's thesis. The AI reading these papers uses whatever reasoning approach it's given. If the approach is mechanistic, it finds mechanisms. If it's adversarial, it finds weaknesses. If it's analogical, it finds parallels. The output looks like comprehension in every case — but the comprehension is shaped by the prompt, not by the material.
This question draws on a substantial body of research across mechanistic interpretability, linguistics, discourse analysis, reasoning architecture, and domain specialization. The same set of ~25 papers is available to all three posts. What changes is how the reasoning modules direct the investigation.
Start from the architecture and work outward. What do retrieval heads actually do? How does the token importance hierarchy work? What does the attention distribution look like on structured text? Trace the causal chain from how the model processes text to what it extracts to what it misses. Every claim must be grounded in a specific architectural or training mechanism.
Bottom-up. The post builds from the smallest observable mechanism (attention heads) to the largest observable consequence (missing the point of a paper). Each step in the argument is a causal link: “because the architecture works this way, the model does this, which means it misses that.” The reader follows a chain of explanations, not a chain of assertions.
A technical post that explains why LLM reading fails in the specific ways it does. The strength will be precision — the reader will understand the mechanism. The weakness will be accessibility — not everyone cares about attention heads. The argument will be bottom-up and convincing to anyone who follows the chain, but it may lose readers who want the practical implication before the technical explanation.
The causal-mechanistic lens will naturally prioritize papers on retrieval heads, token importance hierarchies, attention distributions, and training format effects. It will deprioritize the human-comparison research (ambiguity recognition, implicit warrants) because those are symptoms, not causes. The post will likely underweight the user-experience dimension — how it feels to receive a competent-looking summary that missed the point — because the lens is oriented toward the machine, not the person.
Start from the strongest case for LLM reading and try to break it. What do the optimists cite? Where are they right? What specific evidence would they need to be wrong about for the skeptical position to hold? Look for the boundary — the conditions under which LLM reading works and the conditions under which it doesn't. Every claim must survive the strongest available counterargument.
Thesis-antithesis-synthesis. The post opens by granting that LLM reading works impressively in many cases — and means it. Then it introduces the evidence that complicates the picture. The argument turns on identifying the specific conditions under which the impressive performance degrades. The reader is not told “LLM reading is bad” but “LLM reading is good at X and bad at Y, and here's how to tell which you're dealing with.”
A balanced, nuanced post that readers will trust more than a purely critical one. The strength will be credibility — the reader will feel the argument is fair. The weakness may be that the synthesis position (“it depends on the task”) is less memorable than a sharp thesis. The argument will be structured as a debate, and the reader will feel they've heard both sides.
The adversarial-dialectical lens will naturally prioritize papers that show LLM reading succeeding (style detection at 95%, metaphor extraction, structural coherence) alongside papers that show it failing (32% disambiguation, implicit warrant gap). It will seek the boundary conditions rather than the extremes. The post will likely emphasize the practical decision: when to trust and when not to. It will deprioritize the architectural mechanisms (those are explanations, not arguments) in favor of the behavioral evidence.
Start from human experience and work inward. What does LLM reading resemble? Speed reading? Keyword extraction? A student summarizing a paper they skimmed? Look for structural parallels between how humans fail at reading and how models fail at reading — and look for where the parallels break. Every claim must connect to something the reader has experienced.
Recognition-based. The post opens with a familiar experience — receiving a summary that hits the right topics but misses the point — and asks the reader to recognize what's happening. The argument proceeds by analogy: “you know what it looks like when someone didn't really read the paper. Here's what's happening inside the model that produces exactly that pattern. And here's where the analogy breaks — because the model isn't skimming. It's doing something that has no human equivalent.”
The most accessible of the three posts. The strength will be immediate recognition — the reader will see their own experience reflected. The weakness may be that analogies can oversimplify, and some readers will object that the human parallels are too loose. The argument will feel intuitive and experiential rather than technical, and will land hardest with readers who have received AI-generated summaries and sensed something was off without being able to name it.
The analogical-interpretive lens will naturally prioritize the human-comparison research: the knowing-doing gap, surface generalizations vs linguistic understanding, ambiguity recognition, Potemkin understanding. It will also draw on the discourse structure findings (Grosz & Sidner's three components) as an analogy for what's missing. It will deprioritize the architectural details (retrieval heads, token importance) in favor of the behavioral manifestation that a reader can recognize.
Below are the actual prompts used for each lens, adapted from the playbook modules to this specific topic. The guiding question is filled in; the source material is the same set of ~25 research papers on LLM attention mechanisms, discourse structure, pragmatics, training effects, and domain specialization.
Three posts follow. Each one applies its lens to the same guiding question and the same research base. Read them as variations on a theme — the same orchestra playing the same score in different keys.
The experiment's value is not that one lens is “right.” It's that the lens shapes the output in predictable, documentable ways. The causal-mechanistic lens will produce a technical argument about architecture. The adversarial-dialectical lens will produce a balanced argument about boundary conditions. The analogical-interpretive lens will produce an accessible argument about recognition.
If you are using an LLM to help you think about a topic, the reasoning approach you bring — or the reasoning approach the tool defaults to — is shaping what you find and what you can write. The posts that follow are evidence of that claim.