CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks / Pattaraphon Kenny Wongchamcharoen, Kris Gulati, Min Min Fong, Abhishek Nagaraj.

Format:

Book

Author/Creator:

Wongchamcharoen, Pattaraphon Kenny.

Contributor:

Gulati, Kris.

Fong, Min Min.

Nagaraj, Abhishek.

National Bureau of Economic Research.

Series:

Working Paper Series (National Bureau of Economic Research) no. w35663.

NBER working paper series no. w35663

Language:

English

Physical Description:

1 online resource: illustrations (black and white);

Place of Publication:

Cambridge, Mass. National Bureau of Economic Research 2026.

Summary:

Most LLM benchmarks rank models on their ability to automate work tasks. In practice, however, models are often used to assist other (human or LLM) agents. The question that drives model selection is therefore not only which model produces the best output, but which model most improves the work of another (weaker) agent. We introduce a unified framework that evaluates the capability of models to automate and augment another agent's performance. Across seven economically grounded real-world tasks, an assistant model writes assistance text for a standardized lower-capacity worker model, which produces the deliverable. In automation mode, the assistant produces the output directly. Outputs are scored through blind pairwise comparisons by an LLM judge panel with task-specific rubrics, replicated across ten runs. Rankings across the two regimes are only modestly correlated, and the automation winner loses augmentation on five of seven tasks. Assistance is not reliably positive. The unaided worker outranks every assisted condition on three tasks, and only one model's guidance beats no guidance on average. These results suggest that automation ability is an incomplete proxy for assistance quality, motivating benchmarks that evaluate models according to the roles they play in human-AI and multi-agent systems.

Notes:

August 2026.

Print version record