Human-AI Synergy

Human-AI synergy, as defined by researchers Christoph Riedl and Ben Weidmann, is a measurable quantity separate from an individual's solo problem-solving ability: the extra performance a person gets specifically from working with an AI model, as distinct from how good they are working alone. Their paper "Quantifying Human-AI Synergy" (PsyArXiv preprint, first posted September 2025, submitted to ICLR 2026) introduces a Bayesian Item Response Theory framework that estimates two separate scores for each person: an individual ability parameter and a collaborative ability parameter, while also accounting for how difficult each task is. A standard benchmark score only measures output on a fixed task. This framework instead treats a human-AI pairing as a joint process and asks how much of the pair's performance came from the human, the AI, the task, and their interaction.

The authors validated the framework on data from 667 people completing math, physics, and moral-reasoning tasks with and without AI assistance, comparing two models of different capability, GPT-4o and Llama-3.1-8B. They found statistically significant synergy: people paired with AI outperformed people working alone, and a more capable model produced more synergy overall, though the gap between the two models narrowed on harder problems. The person's individual problem-solving score did not reliably predict their collaborative score. The people who gained the most from AI assistance were not necessarily the strongest solo performers.

The paper attributes collaborative ability partly to theory of mind, or a user's capacity to infer and adapt to the AI's outputs moment to moment, which measurably affected response quality during the interaction. The authors argue this supports interactive, dynamic benchmarks that complement static single-task benchmarks, and suggest it as a consideration for training language models to adapt to varied human collaborators rather than being optimized only against fixed prompts.