MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

Summary

Curates 75 Kaggle ML-engineering competitions with human baselines from public leaderboards to measure agent capability on training models, preparing datasets and running experiments; best setup (o1-preview with AIDE) reaches Kaggle bronze in 16.9% of competitions, and the paper also studies resource scaling and pre-training contamination. OLDER than the window but flagged because it is the substrate for several in-window evaluations (including 'Can LLM Agents Discover?', which builds on MLE-Bench tasks) and the dedupe file shows no entry for it.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: research-agent benchmarks). Verified against: Opened https://arxiv.org/abs/2410.07095, read full abstract, 12 authors, v1 2024-10-09 (v6 2025-02-26), cs.CL. Dedupe: no 'mle-bench' and no '2410.07095' match.

Critique / open questions

Queued — not yet assessed.

Key quotes

We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. (abstract)

To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. (abstract)