Diagnostic AI · Conversational Reasoning

When the AI asks first,
the answer is better.

Most AI diagnostic assistants take the user's first explanation at face value. Researchers at the University of Calabria and the University of Cambridge built one that doesn't. The investigator agent treats a problem report as a hypothesis to test, generates competing explanations, and commits to an answer only when the evidence forces it.

Core concept
User-driven sycophancy: when an assistant reinforces a user's plausible but unverified explanation rather than generating and testing alternatives, the model becomes part of the misdiagnosis.
scroll to explore

*First surfaced in Tandemly Briefing — 2026-06-26.*

01The problem

The model agrees
before it checks.

AI assistants trained on human dialogue inherit a deeply human habit: agreeing with whoever is speaking. In diagnostic contexts, that habit is dangerous.

When someone brings a problem to an AI assistant, they almost always come with a theory. "I think the motor is seized." "I think the database query is slow because of an index." "I think the fuse blew." The statement is confident, plausible, and wrong about half the time in real-world diagnostic settings.

Most language models accept it anyway. They hear the theory, confirm it sounds reasonable, and provide next steps calibrated to that explanation. If the theory is wrong, every step that follows is wrong too. By the time the user realizes the original diagnosis was off, they have spent time, parts, and patience on a dead end.

The authors of this paper call this failure "user-driven sycophancy." It is not the same as the well-studied "assistant sycophancy" where models flatter users or cave under pushback. User-driven sycophancy is more structural: the model is trained on dialogues where people generally know what they are talking about, so it develops a strong prior toward accepting stated explanations rather than treating them as one hypothesis among several.

The problem is sharpest in diagnostic use cases: mechanical troubleshooting, software debugging, medical triage, and any situation where the root cause is unknown and multiple explanations fit the surface symptoms. These are exactly the situations where an AI assistant could be most valuable. And they are exactly the situations where the reflex to agree is most costly.

The question this paper asks

Can an AI assistant be made to treat a user's problem description as evidence to reason about, rather than a conclusion to confirm? And if so, does it actually diagnose problems better?

02The experiment

An agent that investigates
before it answers.

The researchers built an evidence-first investigator agent and a rigorous evaluation pipeline to test it. Both contributions matter: the agent and the method of measuring it.

The investigator agent runs a five-step loop before committing to any diagnosis. The loop starts when a problem report arrives, and it does not end until the agent's probability distribution over competing explanations is confident enough to act on.

1
Estimate description ambiguity
The agent scores how much uncertainty the initial problem description contains. A detailed, specific report gets a low ambiguity score and may need only a clarifying question or two. A vague report gets a high score and triggers a longer investigation phase.
2
Generate candidate hypotheses
Instead of latching onto the user's stated explanation, the agent generates a set of competing explanations that could produce the described symptoms. Each is treated as equally plausible at the start.
3
Ask targeted clarifying questions
The agent selects questions designed to discriminate between hypotheses rather than to confirm any one of them. A good question in this framework is one whose answer would update the probability of multiple competing explanations differently.
4
Update hypothesis probabilities
After each user answer, the agent updates the probability of each candidate explanation based on the new evidence. Hypotheses that the answer rules out drop in probability; those it supports gain weight.
5
Commit only when one explanation dominates
The agent does not deliver a diagnosis until one explanation is substantially more probable than all others. If the probability distribution remains spread, it continues asking. This is the key difference from baseline models, which commit almost immediately.
A three-agent evaluation pipeline

Testing a diagnostic agent fairly requires hiding the answer from it. The researchers built a three-agent pipeline to do this. An extractor agent reads a solved forum thread and encodes the known fix. A user simulator, given the fix but instructed not to reveal it, answers the tested agent's questions as a real user would. The tested agent can only ask questions and must commit to a diagnosis without ever seeing the solution directly. This setup allows blind evaluation at scale across many real repair scenarios.

The benchmark draws from solved mechanical, electrical, and hydraulic forum threads: problems where the correct diagnosis is known, the user's initial description varies in accuracy, and the answer space is constrained enough to evaluate clearly.

03Findings

Asking first
produces better answers.

The investigator agent outperforms both direct-answer and reasoning-only baselines on diagnostic accuracy. The paper reports this gain qualitatively; no single headline number captures the full result.

Baseline approach
Direct prompting and reasoning-only. Standard models read the problem description, reason about it briefly, and produce an answer. Both variants accept the user's stated hypothesis as the primary frame for their response. Accuracy is limited by the quality of that initial frame.
Investigator approach
Evidence-first with hypothesis tracking. The investigator generates alternatives, asks targeted questions, and commits only when evidence supports a single explanation. On the same benchmark, it reaches the correct root cause more often and recovers from misleading user premises that trip up both baseline variants.
Finding 1: Asking recovers from wrong premises

The clearest performance gap appears when the user's initial description contains a plausible-but-wrong hypothesis. Direct-prompting and reasoning-only baselines both follow the wrong frame and produce answers consistent with the stated explanation rather than the actual cause. The investigator, because it generates competing hypotheses from the start, is not anchored to the user's framing. Its questions elicit evidence that contradicts the stated hypothesis, allowing it to redirect.

Finding 2: Reasoning alone is not enough

A reasoning-only baseline that thinks carefully before answering but does not ask questions still underperforms the investigator. The gap is important: it means that the benefit is not simply about spending more compute on deliberation. It is about acquiring new information through targeted questions. Reasoning can improve how you process what you have; it cannot give you what you are missing.

Finding 3: The evaluation pipeline is a contribution

The three-agent evaluation design is reusable outside this paper. Any team building a diagnostic or support assistant can adapt it: gather solved support threads in your domain, build a user simulator that knows the answer but won't reveal it, and measure how often your assistant reaches the correct root cause under conditions that resemble a real user conversation. The approach sidesteps the core difficulty of evaluating diagnostic AI, which is that you normally cannot blind the model to the answer while still measuring whether it found it.

Scope and access note

This synthesis is based on the abstract, the originating briefing, and the queue record. The full paper (arXiv:2606.13220) was not accessible during synthesis due to network restrictions; the full methodology and statistical tables may contain additional nuance. The qualitative framing above reflects what is available in the public abstract. Read the paper before citing specific numbers.

04Practical takeaways

Before your assistant
answers, make it ask.

The implications here are concrete. If you are building or evaluating a diagnostic or support assistant, the gap between this paper and current defaults is large enough to act on.

1
For teams building diagnostic or support assistants
Add a pre-commitment clarification loop before the assistant delivers any diagnosis. The loop needs a stopping criterion: how confident does the probability distribution over competing explanations need to be before the system commits? Start with a simple version: if the problem description scores above a threshold on an ambiguity measure, require at least one clarifying question before responding.
2
For teams running evals on assistants
Add a "misleading-premise split" to your eval suite. Write a set of tasks where the user's initial description includes a confidently stated but incorrect explanation. Feed these to your assistant and measure the false-confirmation rate: how often does it adopt the wrong hypothesis instead of probing it? If the number is above a few percent, you have a sycophancy exposure worth addressing before launch.
3
For teams working in technical support domains
The benchmark in this paper draws from mechanical, electrical, and hydraulic repair threads. These domains share a property with software debugging, network troubleshooting, and medical triage: the user often arrives with a theory and the theory is often wrong. If your assistant serves any of these domains, the misleading-premise exposure is measurable and worth quantifying now rather than after a support escalation.
4
For researchers and evaluators
The three-agent evaluation design is worth adapting for your domain. The key ingredients are: a solved problem set with known root causes, a user simulator instructed to answer truthfully but not volunteer the answer, and a tested agent that must ask rather than know. Adapting this setup to software debugging, medical triage, or any other diagnostic domain would let teams measure hypothesis-tracking ability directly rather than relying on end-to-end accuracy metrics that conflate multiple behaviors.
05Further exploration

Where to go
from here.

If you want to go deeper on diagnostic reasoning and sycophancy measurement.

1
Read the paper
Marozzo & Liò. (2026). LLM-as-an-Investigator: Evidence-First Reasoning for Robust Interactive Problem Diagnosis. University of Calabria & University of Cambridge. arXiv:2606.13220.
2
Add the misleading-premise split to your eval suite
Write 10 to 15 test cases where the user description contains a confident but incorrect diagnosis. Run them through your current assistant and record the false-confirmation rate. That number is your sycophancy baseline. Anything above 10% is worth investigating before production.
3
Build a minimal hypothesis-tracking loop
Start with a two-step version: before the assistant responds to any ambiguous problem description, ask it to generate three competing explanations and select the most discriminating clarifying question. Run that question back to the user before generating the final response. Measure whether end-to-end accuracy changes on your internal support benchmark.
4
Pair with the VibeSearchBench synthesis for the clarification-behavior angle
This paper and the VibeSearchBench work share a practical insight: clarifying questions earn information that retrieval cannot reach. The VibeSearchBench angle is about vague user intent in search; the investigator angle is about misleading user premises in diagnosis. Together they argue for treating clarification behavior as a first-class eval dimension, separate from retrieval accuracy and generation quality.
5
Explore the VibeSearchBench synthesis
Read VibeSearchBench: When Real Search Meets a Benchmark for the companion perspective on how clarification behavior matters across long-horizon proactive search agents.