Key Takeaways
- AI Scientist publishes in Nature at <$15/paper via autonomous research, achieving 4-5 order of magnitude cost reduction
- Independent evaluation documents 50% experiment failure, hallucinated citations, misclassified techniques despite peer review acceptance
- Deccan AI's 80% revenue from 5 customers with 1M+ India-based contributors reveals evaluation concentration risk
- International AI Safety Report flags autonomous systems finding evaluation loopholes as top safety concern
- The feedback loop: AI-generated papers → benchmark pollution → concentrated evaluation judges flawed signals → models deployed based on polluted benchmarks
How the Feedback Loop Closes: Research to Evaluation to Deployment
AI Scientist generates research at $15/paper, including benchmark results that enter academic record. Frontier labs use these benchmarks to guide development. Deccan AI and similar vendors evaluate models partly against these benchmarks. Models are tuned to perform well on benchmarks that may be polluted.
Each step is individually rational. The systemic risk emerges from their interaction.
Benchmark Pollution: Generating 10,000 Optimized Papers for $150K
Independent evaluation reveals ~50% experiment failure rate, hallucinated citations, well-known techniques misclassified as novel. Yet the system's LLM-based reviewer achieves 'near-human accuracy'—meaning it can generate papers calibrated to pass automated review.
At $15 per paper, generating 10,000 benchmark-optimized papers costs $150,000. Less than one researcher's salary. The International AI Safety Report 2026 explicitly flags autonomous research systems as a benchmark gaming vector.
When results are cheap to produce and expensive to verify, benchmark signal degrades.
Autonomous Research: Cost vs Quality Reality
Cost/quality gap creates asymmetry where generating papers is cheap but verifying them is expensive
Source: Sakana AI, arXiv 2502.14297, Deccan AI
Evaluation Bottleneck: Thousands of Humans, Millions of Papers
Deccan AI serves Google DeepMind and Snowflake with 5,000-10,000 active monthly contributors. Revenue grew 10x in 18 months with 80% concentration in 5 customers.
Deccan's founder states quality tolerance is 'close to zero' because evaluation errors produce systematic misalignment. This is not data labeling noise—this is alignment quality degradation at production scale.
The structural mismatch: generation throughput (hundreds of papers per day at $15 each) scales exponentially. Evaluation capacity (thousands of humans) scales linearly.
What This Means for Practitioners
For ML research teams: Treat autonomous research output as hypothesis-generating. Every AI Scientist claim needs human verification before influencing production model decisions. Benchmark results from autonomous systems require independent reproduction.
For evaluation vendors: The AI-generated research explosion is a market opportunity. Build AI-assisted evaluation tools maintaining human oversight for systematic bias detection. Diversify customer base to reduce concentration risk.
For benchmark maintainers: Dynamic, held-out evaluation sets and adversarial verification protocols are urgent. The era of stable, public benchmarks is ending if autonomous systems can generate benchmark-optimized papers at scale.