Скачать или смотреть The Lessons of Developing Process Reward Models in Mathematical Reasoning

This research paper focuses on improving how well large language models (LLMs) can solve math problems. The researchers looked into a specific type of model called a Process Reward Model (PRM). *PRMs are designed to check each step of a math solution to see if it's correct**, unlike other models that only look at the final answer. They found that **a common way of training PRMs, called Monte Carlo (MC) estimation, isn't as good as other methods**, like using a really smart LLM to judge the steps or having humans check the work. They also discovered that **the usual way of testing these models, Best-of-N (BoN), can be misleading because it only cares about the final answer and not how the model got there.* To fix these problems, they came up with a better way to train PRMs by combining MC estimation with LLM judging. Their new PRM is really good at spotting mistakes in math solutions and is even better than some of the most advanced LLMs out there.

https://arxiv.org/pdf/2501.07301