Automatic Process Supervision for LLM Reasoning via Monte Carlo Tree Search
A method that improves large language models' step-by-step reasoning by generating process-supervision data automatically instead of using human step labels. Monte Carlo Tree Search samples and expands reasoning steps with the LLM itself and scores each step's relative correctness from the search statistics; the model is then trained by minimizing a weighted log-likelihood of generating the steps, and the generate-then-train cycle repeats until convergence. It improves accuracy on mathematical-reasoning datasets and the gains transfer across datasets.
Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search Shuangtao Li1
Process supervision trains a language model on the correctness of each intermediate reasoning step rather than only the final answer, but it usually needs expensive human step-level labels. This pape…