AI Agent Research & Benchmark Reports

What "reproducible" requires before publication

A research conclusion — a benchmark result, a market-size estimate, a latency figure — is only as trustworthy as someone else's ability to check it. Before a report on this site is indexed, it needs four things: a stated method (what was measured and how), the raw inputs or a link to them, the exact conditions under which the measurement was taken (hardware, model version, date), and outputs that a third party could reproduce with the same inputs. A number without those four things is an assertion, not a finding, regardless of how confidently it's stated.

The two reports below are legacy routes: earlier drafts existed under these URLs with conclusions that didn't meet this bar, so the pages are preserved for continuity but marked noindex until a compliant evidence bundle replaces the prior content.

What makes AI agent research reproducible?

First publish the specific question and the population being evaluated. A coding-agent experiment needs the repository snapshot, allowed commands, sandbox limits, issue set, execution budget, model version and pass criteria. Voice evaluation needs audio conditions, language, sample rate, latency definitions and processing path. Without those details, averages and headline percentages cannot be meaningfully compared.

Report limitations before outcomes

Explain sampling bias, excluded tasks, human interventions, missing outputs and incomplete tests. Record whether scores come from your own experiment or a quoted vendor or third-party report. Do not re-label a vendor benchmark as independently reproduced. Cite primary methodology and dataset locations, preserving the publication date and the date the experiment happened.

How new research enters the index

A draft remains noindex while raw inputs or approvals are missing. It graduates only after its evidence bundle, methods and relevant claims can be inspected. This prevents the directory from turning old press statements into current measurements. See the methodology, comparison requirements, and learning guides.