When queries get vague,
agents fall apart.
First surfaced in Tandemly Briefing — 2026-05-29.
Xiaohongshu researchers built a benchmark to test search agents on how people actually search: vaguely, over multiple turns, revealing what they want piece by piece. The best frontier model scored 30.3 out of 100. The gap between that number and a passing grade is the gap between benchmark performance and what users actually experience.
Benchmarks say agents search well.
Users say they don't.
There is a persistent mismatch between how search agents score on benchmarks and how well they perform for real users. VibeSearchBench is an attempt to close that gap by testing what the existing benchmarks skip.
Most AI search benchmarks share an assumption that rarely gets examined: the user already knows what they want. Queries are well-formed. The correct answer has a predefined structure. The agent's job is to retrieve the right content and produce output in the right format. That makes evaluation tractable. It also makes it artificial.
Real information-seeking doesn't work this way. A researcher starting a literature review might say "find me recent work on memory in transformer agents." Over the course of the search they realize they care specifically about retrieval-augmented approaches, they want results from the last year, and safety considerations matter too. None of that was in the original query. The full intent emerged through the search itself.
Existing benchmarks test whether an agent retrieves correctly once intent is clear. They don't test whether the agent can surface intent before retrieval. That second skill is what VibeSearchBench measures: proactive clarification across multiple turns, with no preset answer schema, and a user who reveals what they want only in response to good questions.
How well do frontier search agents perform when queries are vague, answers are schema-free, and the agent must earn information from a user through dialogue? The answer, it turns out, is not well. The best score is 30.3 Triplet F1.
200 tasks, two tracks,
no fixed answer shape.
The benchmark replaces single-turn fixed-schema queries with persona-driven conversations evaluated against knowledge graphs. Every design choice resists the shortcuts that inflate scores on simpler benchmarks.
Fixed-schema benchmarks let agents score well by matching format without recovering content. A knowledge graph has no preset shape: the agent must discover what the correct output structure is by learning what the user actually needs. That's a different skill from retrieval, and it's the one that fails when queries are vague.
30.3 Triplet F1.
Plenty of room to improve.
The best frontier model captured roughly a third of the target information. Three patterns explain where the rest was lost.
A score of 100 would mean perfect recovery of every entity and relationship in the ground-truth knowledge graph. At 30.3, the best-performing system captured roughly a third of the target triplets. The rest were missed: entities not identified, relationships not surfaced, or information the agent never thought to ask about. A single-turn retrieval benchmark would likely show a very different picture for the same models on the same underlying topics. The gap between the two scores is the cost of intent ambiguity.
The agents that scored highest were the ones that asked better questions in the early turns. Retrieval quality had less influence on the final Triplet F1 than the agent's ability to surface the user's latent intent before retrieving anything. Good clarifying questions unlock information the retriever can then find. Poor or absent questions leave entire branches of the knowledge graph unexplored.
VibeSearch-Pro tasks (literature reviews, market analysis, technical due diligence) scored lower than VibeSearch-Daily tasks across all models. Domain-specific professional searches require the agent to understand the shape of knowledge in unfamiliar fields and to ask questions that would make sense to a domain expert. Everyday search tasks involve evolving preferences, but the underlying knowledge structures are less opaque.
Chinese and English tasks tested whether agents could switch languages mid-task without losing context or continuity. Several models showed accuracy drops when the conversation crossed language boundaries in ways that monolingual benchmarks would never catch. The gap between Chinese-language and English-language subtask performance was not uniform across model families, suggesting that language-switching coherence is a distinct capability, not a side effect of general multilingual fluency.
200 tasks, manually curated, with LLM-as-judge evaluation for triplet matching. The judge's semantic equivalence decisions introduce some variance, particularly for domain-specific relation types. The benchmark is new; published results should be treated as baselines, not as ceilings. The authors note that Triplet F1 captures whether information was recovered but not how efficiently the agent's clarification strategy reached it. A high-scoring agent that asked 30 questions to recover 30 triplets is different from one that asked 5 targeted questions to reach the same result.
What this means
for search agent builders.
The gap between 30.3 and a passing score is an engineering target. Here is where to look first.
Where to go
from here.
The benchmark is public. The implementation is open. The baselines are low.