Audio MultiChallenge
Benchmark measuring AI intelligence and instruction-following performance on audio input.
Audio MultiChallenge is a benchmark that measures intelligence and instruction-following performance of AI models when processing audio input rather than text. The benchmark presents models with tasks in audio form — questions, instructions, multi-step procedures — and evaluates both the accuracy of the model's response and its ability to follow complex, multi-part instructions correctly. It is among the most widely used benchmarks for tracking the intelligence of speech-enabled AI systems and serves as a key comparison point for evaluating whether real-time interaction models sacrifice intelligence for responsiveness.
The benchmark is challenging for audio-native models because it tests not just speech recognition or transcription quality, but genuine reasoning about auditory content. Tasks involve following multi-step verbal instructions (navigate to the third submenu, select the option that was mentioned most recently, confirm by saying yes), comprehending questions about verbally described scenarios, and maintaining context across long audio passages. The audio format introduces noise, variability in speaker accents and prosody, and the absence of visual context that might otherwise scaffold comprehension.
Audio MultiChallenge results are reported as APR (Ability Performance Rate), which captures both whether the model completed the task correctly and the reliability of that performance across examples. Higher APR indicates more consistent and accurate task completion on audio input. Current state-of-the-art models achieve around 43-48% APR on this benchmark, indicating that audio-based instruction following remains substantially harder than text-based instruction following — where frontier models exceed 90% on IFEval. This gap motivates the continued development of audio-native reasoning capabilities in interaction models.
The significance of Audio MultiChallenge for interaction model evaluation is that it provides a turn-based intelligence baseline. A model that scores well on FD-Bench (interactive metrics) but poorly on Audio MultiChallenge (intelligence metrics) is responsive but not smart. The core claim of the interaction model paradigm is that it is possible to achieve both simultaneously — high FD-Bench and high Audio MultiChallenge scores — and the gap between models that achieve this and those that do not defines the current frontier.