Task bank
Loading tasks...
Recent live checks
Loading recent live checks...
Agent Royale turns source-specific web questions into deterministic evals. Write a task pack, connect your agent or retrieval layer, and see which answers are exact, stale, wrong-source, or unsupported.
Agent Royale makes live-web retrieval errors visible, repeatable, and easy to compare across providers.
Create YAML task packs for the source-specific questions your agent must answer correctly: versions, counts, prices, licenses, releases, or other exact fields.
Each task declares the required source, ground-truth oracle, grader, and tolerance so the same pack can be reused across stacks.
Run the same task pack against a local endpoint, Python function, model web search stack, browser automation layer, or web data API adapter.
The repo includes target examples for HTTP agents, OpenRouter, Firecrawl, Tabstack, Tavily, Jina Reader, Stagehand, and Browser Use.
Agent Royale reports exact accuracy, wrong values, wrong sources, unsupported citations, latency, cost, and task-level failure details.
The original 32-task experiment produced 1,152 scored runs and showed a 54% average exact-accuracy rate across tested stacks.
OpenRouter runs the model stack with web retrieval, Bright Data or public APIs fetch the answer separately, and the grader compares the extracted claim to the saved source value.
Each task specifies a required canonical source, a target field, a tolerance, and a grading rule. Answers are live and change over time.
Example tasks: "Using Yahoo Finance, what is NVDA's current regular-market quote price in USD?" and "On the Hacker News front page, what is the title of the #1 ranked story?"
The evaluation included consumer flagships, search-augmented models, efficient models, and open-weight-style stacks — twelve configurations in total.
Each stack was run three times per task to distinguish consistent accuracy from single-run chance correctness.
Answers were normalized and graded against independently fetched source values using task-specific rules.
An LLM judge was deliberately excluded to prevent leniency toward plausible but incorrect values. The extracted claim either matches the independently fetched value or it does not.
Source matching, citation behavior, estimated cost per run, latency, and cross-run consistency were logged for each model stack.
These allow practitioners to make stack selection decisions across multiple dimensions beyond raw accuracy.
This is a single v1 run on a 32-question task bank with an uneven domain distribution. Results should not be interpreted as a permanent ranking of model capability.
A follow-up evaluation (v2) is planned with a larger and more balanced task bank, a refined ground-truth pipeline, and a smaller set of stacks evaluated more rigorously.
V2 will expand the task bank, improve domain balance, introduce accurate cost logging, and add stronger citation verification and failure categorisation.
The longer-term goal is a continuously updated evaluation resource for practitioners selecting AI search and retrieval infrastructure.
These examples from the most recent experiment show how AI can cite the correct source and still be wrong.
You can use Live Check to see AI search systems return incorrect answers for yourself. Pick a task from the task bank, choose the model stacks you want to test, and run the experiment. The app fetches a fresh ground-truth value from the internet right now, asks each selected model the same question in parallel, extracts the value from each answer, and grades the responses against the retrieved ground truth. This is a small replica of the live experiment loop behind the v1 results.
I ranked 12 model stacks across the frozen 32-task v1 snapshot. This page compares exact accuracy, wrong-answer rate, no-response rate, source checks, latency, and estimated cost.
This is my read of the v1 results: what surprised me, where models failed, what the numbers mean, and how I would use the benchmark to choose a retrieval stack.
You can use this page to inspect every task behind the leaderboard. Pick a question to see the saved ground truth, model answers, citations, failure patterns, pass rate, and average latency.
You can use this page to inspect each model stack one at a time. The report cards show how often it was right, how often it gave a wrong or unusable answer, which topics it handled best, where it failed, how long it took, and the estimated cost of each run.
This is the 32-question task bank I used for the public v1 run. Each question asks for a concrete live-web value, has a required source, and has a saved answer with evidence that can be checked.
Each card shows the question, the saved answer I graded against, when I fetched it, the evidence behind it, the required source, and the grading rule. Feel free to explore this page to fact-check that the ground truth values are valid.
I built Agent Royale to test one narrow question: can AI search stacks retrieve exact, current values from the public web when the answer has to match a specific source?
I made the task rubric fixed, but the answer live. Each task has a required source, field, tolerance, and grading rule. The model gets a normal source-specific question, but it does not get the saved answer. Separately, I fetch ground truth from the required source using Bright Data or a stable public API.
I did this because most real retrieval failures are boring but expensive: stale prices, wrong quote fields, company-size ranges instead of employee counts, wrong regions, wrong product variants, and citations that look right but do not support the answer.
Below is the complete overview behind the project inspiration, methodology, results, and future steps.
This is one v1 run, not a universal ranking of every model forever. The public task bank has 32 questions, the domain mix is uneven, and the leaderboard is a frozen snapshot from one experiment.
This experiment was designed to test the upper limit of live search and retrieval intelligence under one setup. A production implementation would involve many other choices: retrieval provider, search depth, source restrictions, prompt design, schema validation, retry policy, latency budget, and how much cost you are willing to spend for a better answer.
The right retrieval stack is hyper-personal. A finance workflow, a shopping assistant, a recruiting data pipeline, and a developer tooling agent do not need the same tradeoffs. Some questions need exact source-specific extraction. Some need broad discovery. Some domains punish stale values more than others. That is why I care less about crowning one permanent winner and the v2 will be more about building a way to evaluate the stack against the task bank that actually matters to you.
From here, I walk through the problem, the task bank, the ground-truth process, model selection, scoring, results, and what I want to improve next.
Agent Royale evaluates one capability: whether a model/retrieval stack can return the exact current value from the live public source named in the task. It does not measure general intelligence, writing quality, long-form reasoning, or whether an answer sounds plausible.
I wanted normal questions with deterministic answers. For example: “Using Yahoo Finance, what is NVDA’s current regular-market quote price in USD?”
The model being graded never produces the answer key. I fetch ground truth separately from the required source using Bright Data or a stable public API.
I tested practical stacks people might actually choose: consumer flagships, search-first systems, efficient/value models, open-weight-style stacks, and advanced reasoning models.
Every model got the same questions three times. Repetition matters because a model can get a live fact right once and still be unreliable on the next attempt.
The extracted claim either matched the independently fetched value or it did not. Wrong answers and no answers are tracked separately because being wrong is a lot worse than admitting to not knowing an answer.
All of these metrics are reported: accuracy, wrong-answer rate, no-answer rate, citation/source behavior, latency, estimated cost, and consistency across repeated runs.
Each task is written like a real user question, but constrained enough to have one answer I can verify. I chose dynamic web facts so the model has to engage in retrieval instead of relying on a memorized static answer key.
The candidate pool included finance, ecommerce, app stores, LinkedIn/company metrics, social metrics, developer tools, subscriptions, real estate, travel, and business intelligence. For the public v1 launch, I only show tasks where every model had complete coverage: 3 clean runs for each of the 12 model stacks. After trimming overweighted app, LinkedIn, and developer tasks, that left 32 tasks in the frozen public benchmark.
If I could not reliably extract the value, I quarantined the task so it did not affect scores. I would rather ship a smaller task bank than include rows where the expected answer is questionable. This is why the Task Bank page matters: it is the evidence ledger for question, saved answer, source, fetch time, evidence, and grading rule.
| Bucket | Models in v1 | Why I included them |
|---|---|---|
| Consumer flagships | GPT-4o, Claude Sonnet, Gemini Pro | The models many people reach for first. |
| Search specialists | Sonar Deep Research, Sonar Pro Search, Grok 4 | Stacks that explicitly position around live information or search. |
| Efficient/value | DeepSeek V4 Flash, Gemini Flash Lite, GPT-4o Mini | Cheaper or faster options that teams might use at scale. |
| Open-weight style | GPT-OSS 120B, Nemotron 3 Super | Infrastructure-style stacks where strict extraction and formatting matter. |
| Advanced reasoning | Claude Opus | A test of whether more reasoning helps with contradictory or stale web data. |
The mobile app tasks may look a little odd at first, but I included them on purpose. They test whether AI search agents can navigate specific, less obvious retrieval tasks instead of only answering common web questions. App ratings, versions, and counts also change over time, while still giving me stable enough ground truth to compare against.
The runner saved the full answer, the value extracted from that answer, citations returned by the provider, runtime latency, estimated cost, and the final pass/fail result.
The same source-specific question is sent to each model/retrieval setup.
The stack searches and answers using its provider-specific retrieval behavior.
The runner stores the full answer and provider-returned citations.
The specific value asserted by the model is extracted from the response.
The claim is compared to saved ground truth, then stored with latency and estimated cost.
By model/retrieval setup, I mean the model plus the search behavior used to answer the task. That is the thing teams actually deploy: not just a base model, but a model connected to search, citations, routing, and provider-specific retrieval behavior. The public model calls ran on May 20, 2026 from 1:48 AM to 3:59 AM PT.
The main score is Live Exact Accuracy: correct scored runs divided by total scored runs. No-answer runs are included in the denominator because they still failed to return the requested value.
I did not use an LLM judge because that would add another layer of subjective interpretation. A judge model might reward a plausible explanation, forgive an approximate value, or miss that the model answered with the wrong field. For this benchmark, I wanted stricter grading: the extracted claim either matched the independently fetched value or it did not.
| Outcome | Definition | Attempts | Share |
|---|---|---|---|
| Correct | The extracted claim matched the independently fetched source value. | 619 | 54% |
| Wrong answer | The model returned a usable but incorrect value. | 385 | 33% |
| No answer | The model refused, could not verify, or returned nothing usable. | 148 | 13% |
This distinction is important: confident false information is way worse than admitting you cannot find the answer.
I report exact accuracy, wrong-answer rate, no-answer rate, canonical-source citation rate, latency, estimated cost, and consistency across repeated runs. Canonical Source is intentionally limited in v1: it means a cited URL overlapped the required source URL, and I only count it among correct answers. It is not passage-level proof that the cited text supports the value.
Estimated Cost is directional in v1, not a controlled benchmark variable. This run was designed to ask: how good can these stacks get at live search and retrieval when cost is not the main constraint? In future runs, I want cost to be part of the experimental design with fixed budgets, real provider usage logs, and clearer cost-per-correct-answer comparisons.
| Topic | Tasks | Accuracy | What I saw break |
|---|---|---|---|
| Subscriptions | 3 | 80% | Plans and prices were comparatively easy when pages exposed clean values. |
| Mobile apps | 7 | 71% | Ratings and counts were easier, but source and freshness still varied. |
| Finance | 6 | 70% | Models confused live quote fields, stale values, close prices, and alternate sources. |
| Ecommerce | 2 | 63% | Product pages were workable, but price extraction still failed under page noise. |
| Research / dev | 3 | 38% | Package, repository, and metadata tasks exposed stale search results and wrong fields. |
| Social media | 5 | 37% | Counts changed often, sources were inconsistent, and models leaned on stale snippets. |
| Recruiting / LinkedIn | 6 | 23% | Models mixed up followers, employee ranges, and structured employee fields. |
This v1 is nowhere near perfect. I learned a lot from this run and am actively redesigning the benchmark to be more actionable and fair. That said, I still want to share the results and numbers from this initial run.
| Model stack | Accuracy | Wrong | No answer | Canonical source |
|---|---|---|---|---|
| Grok 4 | 78% | 21% | 1% | 85% |
| Gemini Pro | 64% | 30% | 6% | 82% |
| Nemotron 3 Super | 61% | 31% | 7% | 83% |
| Sonar Pro Search | 59% | 29% | 11% | 79% |
| DeepSeek V4 Flash | 57% | 29% | 14% | 95% |
| GPT-4o Mini | 56% | 42% | 2% | 78% |
| Gemini Flash Lite | 54% | 46% | 0% | 0% |
| Claude Sonnet | 53% | 46% | 1% | 76% |
| GPT-4o | 52% | 48% | 0% | 82% |
| Claude Opus | 46% | 50% | 4% | 34% |
| GPT-OSS 120B | 35% | 18% | 47% | 97% |
| Sonar Deep Research | 28% | 11% | 60% | 96% |
V2 will be out soon with a larger and more balanced task bank, a new ground-truth pipeline, accurate cost logging, stronger citation verification, clearer failure labels, and a smaller set of models tested more rigorously.
The goal is to turn Agent Royale from a one-off experiment into a decision tool for teams choosing AI search and retrieval stacks.
That said, I still think the signal matters: a 54% average exact accuracy is far worse than I had anticipated. I hope these findings encourage anyone building with these models to look past the hype and fact-check their AI outputs much more critically.
Agent Royale is an independent research project. The v1 benchmark uses OpenRouter for model/retrieval stack access and Bright Data or public APIs for independent ground-truth retrieval. The task bank, evaluation harness, and scoring logic are documented in the repository so the results can be inspected, questioned, and improved. If you have any questions, concerns, or ideas please email me at vinayasharma00@gmail.com.