Multi-Tool Reasoning and Process-Supervision Benchmarking of Language Models
A benchmark methodology for evaluating language models on complex tasks that require chaining multiple external tool calls, scoring not only the final answer but each intermediate reasoning step through human-verified process-supervision labels. It supports comparing outcome-supervised reward models, which judge only the final result, against process-supervised reward models, which judge each step, and demonstrates that step-level supervision generalizes better when ranking tool-use trajectories.
ToolComp: A Multi-Tool Reasoning and Process Supervision Benchmark
Introduces ToolComp, a benchmark for evaluating how well language models perform complex, multi-step tasks that chain together several external tool calls. It contains 485 human-edited and verified p…