Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models

As the foundational stage of training LLMs, effective pretraining is crucial for developing reasoning abilities. Before discussing pretraining for LLMs’ reasoning, we first outline the basic process of general LLM pretraining. Through pretraining, LLMs not only acquire core linguistic knowledge but also gain diverse world knowledge, establishing a robust groundwork for the emergence of advanced capabilities and effective value alignment [191]. Typically, LLM pretraining relies on high-quality text corpora [35, 168], including extensive collections of web content, books, codes and other types of data. Leveraging these rich textual corpora, LLMs are built on the transformer architecture that are trained with next-token prediction task. After pretraining, LLMs generally demonstrate exceptional in-context learning capabilities [14], enabling them to generate coherent text and provide accurate answers to a wide range of questions by utilizing their vast knowledge base. Notably, the pretraining stage plays a pivotal role in cultivating the reasoning abilities of LLMs. For example, research [160] has shown that datasets rich in code and mathematical content serve as a key foundation for developing robust reasoning skills. Following this observations, newly developed LLMs [1] begin to introduce carefully designed synthetic data for enhancing the reasoning abilities of LLMs. During pretraining, a critical challenge lies in balancing the proportion of code and mathematical data with general text corpora to maintain strong general linguistic abilities while unlocking the reasoning potential of LLMs.

While pretraining enables LLMs to exhibit reasoning abilities through in-context learning, fine-tuning techniques are widely employed to achieve zero-shot and improved reasoning capabilities for LLMs. Here, we first outline the basic fine-tuning process and then explore its potential for enhancing reasoning abilities. As described in [104], after the pretraining stage, LLMs enter a supervised fine-tuning phase (SFT), also referred to as the instruction tuning stage. The primary goal of this phase is to refine the model’s output style, ensuring its responses are aligned with human needs and real-life applications. This is achieved by training via diverse instruction datasets that reflect a wide range of everyday human interactions, typically created through extensive and carefully curated manual annotation and refinement [195]. With the advent of ChatGPT, new methods have emerged for generating diverse instruction datasets. These include techniques that distill data directly from powerful LLMs [153, 167] and automated approaches for large scale dataset construction from existing corpora [158, 32]. Using these well-crafted instruction-tuning datasets, the fine-tuning process continually uses the next-token prediction objective, similar to pretraining. However, unlike pretraining, fine-tuning specifically calculates the loss for the answers while generally ignoring the loss for the questions. Additionally, incorporating datasets that include chain-of-thought (CoT) [160] reasoning and mathematical problem-solving examples has been shown to significantly enhance the reasoning capabilities of LLMs, making this an area of active research. Following the practice in general aspects, most current approaches leverage data distillation from advanced large reasoning models, followed by fine-tuning to enhance the reasoning capabilities of LLMs to obtain the final large reasoning models.

Relying solely on direct data distillation from advanced large reasoning models limits the potential of new LLMs. A more promising approach is to use reinforcement learning for data construction and model training, which precisely corresponds to the final alignment stage in general LLM training. In the general training of LLM, the alignment phase typically involves methods such as Reinforcement Learning from Human Feedback (RLHF) [104] to guide the model toward generating content that meets the criteria of being helpful, harmless, and honest. The goal of this phase is to enhance the safety and controllability of LLMs in the reality. Compared to the former SFT phase, this stage usually incorporates a large amount of carefully curated, manually labeled ranking data to accurately reflect human preferences [35, 168]. This data includes not only correct demonstrations but also undesirable cases that should be avoided. Standard RLHF typically involves an SFT model, a reward model, and a aligned model, which are iteratively optimized using methods like PPO [121]. Due to the high data requirements and training costs of standard RLHF, methods like Direct Preference Optimization (DPO) [112], have been proposed to reduce reliance on explicit reward models. In DPO, preference loss is defined as a function of the policy to directly guide model optimization. Given the multi-step nature and complexity of reasoning problems, alignment-based post-training has become the final and most critical step in stimulating the reasoning capabilities of LLMs. By carefully decomposing the reasoning process and gradually feeding signals back to the model, various self-training methods [45, 64, 183] based on reinforcement learning and preference learning have achieved notable success.

Human-like reasoning is one of the most important abilities that emerge in LLMs with sufficiently large model parameters [157]. While zero-shot reasoning may remain unreliable for some tasks, researchers have discovered various prompting techniques to enhance these capabilities. These techniques can be broadly categorized into three main approaches: step-by-step reasoning, multi-path exploration, and decomposition-based methods.

The step-by-step reasoning approach, exemplified by Chain-of-Thought prompting [160], demonstrates that explicitly showing intermediate reasoning steps significantly improves problem-solving abilities. Even simple prompts like “Let’s think step by step” can effectively guide the reasoning process [62]. This approach has been further refined through Self-Consistency [153], which generates multiple reasoning paths to arrive at more reliable conclusions, and Auto-CoT [189], which automates the generation of effective reasoning chains.

Multi-path exploration approaches extend beyond linear reasoning by considering multiple potential solution paths simultaneously. Tree of Thoughts [172] organizes alternative reasoning pathways in a tree structure, enabling systematic exploration of different solution strategies. Graph-of-Thoughts [11] further generalizes this to a graph structure, allowing for more flexible reasoning patterns and backtracking capabilities. ReAct [173] enriches this paradigm by interleaving reasoning with action steps, enabling more dynamic interaction with external environments.

For complex problems, decomposition-based methods have proven particularly effective. Least-to-Most Prompting [196] and Algorithm of Thoughts [122] systematically break down complex problems into manageable components, while Plan-and-Solve [147] provides strategic guidance for tackling these subproblems. These methods are especially valuable when dealing with tasks that require multiple steps or different levels of analysis.

These extensive reasoning capabilities, enhanced through structured prompting strategies, have proven particularly effective for tasks requiring careful analysis and systematic thinking, enabling LLMs to accomplish a wide variety of complex social scientifically relevant tasks. The success of these methods demonstrates that while LLMs possess inherent reasoning abilities, their full potential can be unlocked through careful guidance and structure in the prompting process.

On top of the instruction following and in-context learning capabilities of LLMs, researchers start to design agentic workflows that program the “thinking patterns” of LLMs [137]. Such agentic workflows allow researchers to enhance LLM’s reasoning ability without any additional training, but it often requires more test-time compute. In-context learning [33, 25] is the ability to improve LLM’s task specific performance by simply providing a few in-context demonstrations, enables LLMs to efficiently generalize to unseen problems without computationally expensive trainings [14]. Although the origin of such capabilities remains a largely debatable topic, recent studies suggest in-context learning improves LLMs’ performance by allowing them to capture the label space, the distribution of input text, and the desired format of answers [97]. Such desirable features have enabled researchers to adapt general purpose LLMs to diverse task scenarios, such as simulating the perspective of certain demographic groups through in-context role play [22]. Recent studies suggest effective agentic workflow can largely improve LLMs abilities for simulating human behavior [105, 127], human-LLM interaction [89], and collaborative task solving [107]. The ability to program LLMs with agentic workflow lays the foundation of improving LLM’s reasoning capabilities with complex cognitive architecture.

The role of human annotation in constructing datasets for LLMs is indispensable. Human annotators are characterized by their meticulousness, patience, and precision, as well as their adaptability to novel scenarios and capability to handle ambiguous data effectively [98]. Zhou et al. [195] demonstrate that even with minimal human-annotated data, models can achieve strong performance, highlighting the critical role of carefully curated annotations in model effectiveness. Human-annotated data plays a pivotal role in enhancing the reasoning capabilities of large language models. In the context of Reinforcement Learning with Human Feedback (RLHF)[104], preference data from human annotators enables LLMs, initially trained on general text corpora, to align with intricate human values and complex ethical considerations. This generalizable approach of annotation helps in fine-tuning models for specific tasks. Building on this foundation, Lightman et al.[75] demonstrated the efficacy of using human annotators to evaluate the reasoning quality at each step of mathematical reasoning processes, significantly improving the accuracy of LLM reasoning. This highlights how human annotation can bridge the gap between general training data and domain-specific challenges, such as complex reasoning tasks.

Enhancing reasoning capabilities in LLMs requires process supervision, where human annotators guide each step of the reasoning process [75]. However, such supervision demands extensive human-annotated data, making it resource-intensive and unsustainable. Given that LLM training typically requires terabytes of data, the volume of which is critical for model performance, constructing datasets purely through manual annotation becomes increasingly impractical. This highlights the need for alternative approaches to improve reasoning without relying solely on human annotation. One promising approach is the collaboration between humans and LLMs for annotation, where LLMs are leveraged to accelerate the process while preserving the high quality of human-generated annotations. Specifically, the annotation process can be divided into two stages: the pre-annotation stage and the refinement stage. During the pre-annotation stage, LLMs can be employed to perform an initial round of annotations, taking advantage of a few manually provided examples for a quick and efficient setup [42, 61]. In the refinement stage, human annotators can assess the quality of LLM-generated annotations and focus on correcting only the subset of annotations with poor quality [61, 152, 96, 42]. To enable scalable annotation processes, recent works have increasingly focused on how to maximize automation while ensuring data quality, thus reducing human involvement without compromising the accuracy of the annotations.

Data annotation is a challenging and resource-intensive task, particularly in scenarios requiring complex operations such as filtering, identifying, organizing, and reconstructing textual data. These tasks are often tedious, time-consuming, and demand significant human effort, making them a costly bottleneck in large-scale data construction efforts [142, 31]. To address these challenges, leveraging LLMs for data annotation provides a cost-effective and efficient alternative. With context window lengths exceeding 100k tokens, LLMs can effortlessly process lengthy texts and large volumes of structured data [2], handling the intricate requirements of data annotation with remarkable efficiency. Their strong instruction-following capabilities [187] enable them to flexibly accommodate diverse and complex annotation scenarios, while achieving a level of quality comparable to that of human annotators. By automating these demanding tasks, LLMs significantly reduce the reliance on human labor, streamlining the annotation process and enhancing overall productivity [181].

LLMs are capable of handling a wide variety of automated annotation tasks, ranging from simple question-answer extraction [106] to the inclusion of additional target information [161]. Without human demonstrations, LLMs rely on their powerful reasoning and in-context learning abilities to independently address more complex annotation needs. For instance, Schick et al.[120] demonstrated how LLMs can be used to construct datasets for tool usage. For each candidate position that may require an API call, the LLM is able to comprehend the logical relationships within the surrounding context, generate relevant questions, and identify the appropriate tool API to address the issue. When human demonstrations are available, LLMs can further enhance their performance by mimicking the patterns and reasoning strategies illustrated in these examples. For complex tasks, human demonstrations provide high-quality trajectories—sequences of thoughts, observations, or actions—that guide the LLMs in replicating human decision-making processes. Existing studies have shown that even zero-shot LLMs, guided by task-agnostic prompts based on human demonstrations, can perform annotation tasks effectively [65]. Moreover, for tasks involving highly intricate and nuanced trajectories, LLMs can incorporate specialized agents, such as a Plan-Agent, Tool-Agent, and Reflect-Agent, to address different aspects of the annotation process, thereby further enhancing their ability to align with human-like reasoning and behavior [109]. These diverse capabilities extend naturally to reasoning outcome annotation tasks, where LLMs not only infer underlying logical structures but also systematically document intermediate reasoning steps and their associated conclusions. This enables the creation of annotated datasets that capture not just final outcomes but the full reasoning processes leading to them, offering richer insights for downstream applications.

Beyond annotation with human demonstrations, LLMs can independently enhance their annotation capabilities through search with feedback, a process that involves iterative refinement by learning from dynamic environment. Failed data points can be considered a classic form of feedback, serving as valuable feedback for the model to identify weaknesses and design targeted adjustments. By self-correcting erroneous samples and generating refined training data, LLMs engage in a cycle of self-improvement that strengthens both their understanding and reasoning [70]. Furthermore, LLMs can systematically analyze the causes of their errors, extracting key insights and encoding these as self-learned knowledge to guide future reasoning tasks [72]. This feedback-driven approach can also involve pairing failed trajectories with successful ones based on similarity, enabling contrastive learning strategies to refine the model’s parameters. Through such iterative search and refinement mechanisms, LLMs not only address errors but also develop a more robust capacity for reasoning, enabling deeper generalization and adaptability across complex tasks [135].

In complex reasoning tasks, each step of the model’s output can significantly influence the final result, making it essential to label intermediate decisions as ”correct,” ”incorrect,” or assign an intermediate reward, namely process annotation. However, manually labeling these steps is costly and time-consuming. For example, Lightman et al. [75] inputs massive manual efforts to produce a large-scale process annotation dataset, i.e. PRM800K, which satisfies the requirement in training an effective process reward model (PRM) and greatly enhances the reasoning capability of LLMs. Therefore, automated methods are increasingly needed to efficient process annotation, ensuring scalability and cost-effectiveness. Initial automated approaches hire external stronger LLMs to annotate the intermediate process generated by smaller LLMs. Furthermore, Monte Carlo based method reduces the reliance on external stronger LLMs, and can use weaker LLMs to complete data annotation and thereby train stronger LLMs via a self-reinforced manner.

Annotation with stronger LLM: As a straightforward automated labeling method, Luo et al. [84] design to utilize a more powerful external model to annotate the intermediate results of a generative model’s inference process. Rather than relying on manual annotation, the method employs a pre-trained, high-performance model, like GPT series, to evaluate each generated step. By leveraging the capabilities of a stronger external model, this approach enhances both the accuracy and scalability of the labeling process, making it more feasible for large-scale tasks. However, the major limitation of this approach is its reliance on the highly capable external model, which means the performance of the labeling process is ultimately constrained by the capabilities of the external model used.

Annotation by Monte Carlo simulation: To reduce reliance on the powerful external models, Wang et al. [148] and Wang et al. [156] propose an improved method that avoids directly scoring the intermediate steps. Instead, their approaches use an external model to continue the reasoning for several steps from the given intermediate output and randomly repeat this simulation process multiple times. The quality of the intermediate step is then assessed based on the average outcome of these extended inferences. This Monte Carlo method has shown promising results in tasks such as mathematical problem solving and code generation.

Annotation by tree search simulation: The approach of using multiple-step Monte Carlo simulation with an external model to assess the quality of intermediate steps based on the average outcomes has become one of the most widely used methods for automated process annotation. To further enhance the efficiency of this method, Luo et al. [85] proposes an improvement by replacing the repeated Monte Carlo simulations with a Monte Carlo Tree Search (MCTS) strategy. In this improved method, multiple leaf nodes representing the final inference results are generated from the intermediate step using MCTS. The quality of the intermediate step is then evaluated based on the average outcomes of these leaf nodes. Compared to random repeated inferences, MCTS leverages tree search to improve the inference quality, while also allowing leaf nodes to share high-quality parent nodes, reducing computational overhead and increasing efficiency. This method has demonstrated superior performance in mathematical problem solving, outperforming human annotations.

One step forward from the MCTS-based simulation, Zhang et al. [183] introduces a self-refining mechanism into the process annotation. They leverage the obtained process annotations to train a process reward function (PRM), which in turn improves the performance of the large language model (LLM). The refined LLM is then used to repeat the MCTS-based simulation, generating higher-quality annotations. This iterative process, involving repeated cycles of improvement, results in progressively enhanced process annotations. This method has shown excellent performance across several tasks, including mathematical mathematical problem solving, questioning and answering, and multi-domain knowledge reasoning, demonstrating its effectiveness in continuously refining and improving the quality of the annotations through iterative enhancement.

Supervised Fine-Tuning is a learning technique that refines pre-trained models’ capabilities for specific tasks or domains using labeled data, while retains model’s understanding on pre-trained knowledge. While pre-training allows models to learn broad, general-purpose features from massive amounts of unstructured data, fine-tuning specializes the model by exposing it to smaller, task-specific datasets with clear input-output mappings.

SFT is a critical step in improving the reasoning ability of LLMs, enabling their application in downstream tasks by adapting them from general-purpose systems to domain-specific tools. For example, LLMs like GPT [111], BERT [30], and T5 [113] are pre-trained on vast amounts of text data using self-supervised learning, equipping them with broad language understanding and generation capabilities. However, their outputs are not always aligned with task-specific requirements. Without fine-tuning, LLMs tend to perform poorly on certain reasoning tasks, such as object counting [182], satellite understanding [91], and engineering questions answering [154]. Through SFT, we can partially address these challenges by refining the model’s outputs based on labeled task-specific datasets.

However, the direct application of SFT may not fully explore the model’s reasoning capabilities in the desired domains, particularly in tasks that require more complex decision-making or multi-step problem-solving. The introduction of CoT techniques [160] has revolutionized the SFT process, by explicitly training the model to generate intermediate reasoning steps before arriving at an answer. With CoT-based SFT, LLMs are encouraged to generate intermediate reasoning steps explicitly, thus enhancing their reasoning ability to tackle tasks that require more structured and organized thoughts. For instance, ReasonBert [29] shows that fine-tuning models with reasoning chains significantly enhances their performance on tasks such as math word problems and logical reasoning by incorporating step-by-step reasoning processes. Another key study [80] investigates how fine-tuning models with reasoning improves their interpretability and reduces errors in complex decision-making scenarios by generating more transparent, stepwise thought processes. By fine-tuning with CoT, models not only improve their final answers but also enhance their ability to “think through” the problem, providing clearer insights into the model’s reasoning process.

Despite the diverse methods and outstanding performance of SFT, it comes with several limitations. First, SFT heavily relies on high-quality labeled datasets, which can be expensive and time-consuming to curate, especially for niche domains or tasks requiring expert annotations. Second, SFT may lead to catastrophic forgetting, where the model loses some of its pre-trained general-purpose knowledge during the fine-tuning process, reducing its utility for tasks reasoning outside the fine-tuning domain. Finally, the computational cost of fine-tuning large-scale models can still be prohibitive, even with parameter-efficient approaches, posing challenges for organizations with limited resources. Addressing these limitations requires careful dataset curation, regularization techniques, and the exploration of alternative methods, such as prompt tuning or multi-task fine-tuning, to balance task specialization and generalization.

Due to the high reliance on expensive, high-quality labeled datasets, and high computational costs of SFT, reinforcement learning has emerged as a powerful alternative framework for training models to master reasoning processes. Unlike supervised learning, RL enables models to learn through trial and error reward signals, discovering optimal strategies for achieving specific objectives. As shown in Figure 2 (a), the model takes action based on its current state and receives feedback in the form of a reward signal. This feedback guides the model to update its parameters over time, optimizing for cumulative rewards.

Classic reinforcement learning. RL has become a critical step in the development of LLMs. In RL framework, the parameters of LLMs are updated based on the rewards for their actions. Specifically, the value function or Q-function is updated based on the feedback of reward model, attributing the credit for an action’s outcome entirely to its immediate effect. This approach simplifies the framework, making it conceptually straightforward while enhancing the model’s ability to respond effectively. Two key methods currently dominate RL training for LLMs: Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF).

Ouyang et al. [104] use RLHF to align LLMs with human intent. Besides, by fine-tuning GPT-3 on human-labeled demonstrations and rank comparisons, they develop a reward model predicting human annotator’s preferences. It effectively aligns trained LLMs with human preferences, outperforming GPT-3 in reasoning and instruction-following despite being smaller. Bai et al. [8] also leverage RLHF to create helpful and harmless language models. Following a Helpful, Honest, and Harmless framework, they fine-tune a base model, train a preference model with rejection sampling, and iteratively refine it with human feedback. This process produces AI assistants that excel in NLP tasks and demonstrate strong ethical reasoning.

To reduce the reliance on large human-labeled datasets, Bai et al. [9] propose Constitutional AI, a framework for training AI assistants to be helpful and harmless using principles instead of expensive human feedback. The process includes two phases: supervised learning and RLAIF. In the supervised phase, the model critiques and refines its outputs based on constitutional principles, creating a fine-tuning dataset. In the RLAIF phase, the model generates self-assessments to guide training, bypassing the need for human-labeled data on harmfulness. Ramamurthy et al. [114] focus on using RL to align LLMs with human preferences. They introduce RL4LMs, a library for RL-based fine-tuning, and the GRUE benchmark, which evaluates models using reward functions reflecting human preferences. To address training challenges, they propose the natural language policy optimization algorithm, which stabilizes training by constraining token sampling. This work provides a strong foundation for integrating RL into LLM fine-tuning for improved alignment and performance.

Direct preference optimization Classic RL methods rely on training a reward model to score outputs based on human preferences. While DPO streamlines this process by directly leveraging preference data without requiring an explicit reward model. Instead of optimizing a complex reward function, DPO uses pairwise preference comparisons, i.e., data indicating which of two outputs is preferred by humans. This direct approach simplifies the learning pipeline while preserving the alignment benefits of RL-based methods, which is often simpler and more effective. Rafailov et al. [112] introduce DPO, a novel framework for aligning language models, which directly optimizes the policy to align with human preferences through a simple classification loss. By parameterizing the reward model to derive an optimal policy in closed form, DPO eliminates the need for sampling and extensive hyperparameter tuning during fine-tuning. Experiments show that DPO matches or surpasses RLHF methods like PPO in tasks such as sentiment control, summarization, and dialogue generation, while being more stable, computationally efficient, and effective in producing reasoning outputs. Amini et al. [4] propose Direct Preference Optimization with an Offset (ODPO), an extension of DPO for aligning language models with human preferences. ODPO improves upon DPO by considering the degree of preference between responses rather than treating all preference pairs equally. It introduces an offset in the likelihood difference between preferred and dispreferred responses, proportional to their quality difference. This approach not only improves alignment but also strengthens the model’s reasoning capability, particularly in tasks such as sentiment control, toxicity reduction, and summarization. Experiments demonstrate that ODPO achieves better alignment and responsible behavior, especially when preference data is limited.

In conclusion, the RL and DPO methods offer a straightforward and effective method for fostering reasoning ability in LLMs. By focusing on immediate rewards following each action, these methods also align models with human preferences. The emphasis on short-term feedback simplifies the learning process, avoiding the complexities of credit assignment across long sequences. This streamlined approach is particularly well-suited for real-time applications and tasks requiring clear, concise reasoning, ultimately strengthening the ability of LLMs to deliver coherent and ethical outcomes.

For complex reasoning tasks, such as mathematical problem-solving, LLMs need to perform multi-step reasoning like Chain-of-Thought to ultimately reach an accurate solution. In these tasks, the reward feedback is typically only available after all the reasoning steps are completed and the final solution is obtained. As shown in Figure 2 (b), this is known as the outcome reward model (ORM). In such cases, the key to improving the LLM’s reasoning ability lies in distinguishing the correctness and importance of intermediate reasoning steps based on the outcome rewards.

Classic reinforcement learning. ReFT [143] applies the PPO [121] method from RLHF [104] to reasoning tasks. Based on the outcome reward model, the value function in PPO is able to infer the contribution of intermediate reasoning steps. Compared to supervised fine-tuning, ReFT is capable of learning more diverse reasoning paths, exhibiting stronger generalization abilities in reasoning tasks. However, VinePPO [60] discovers that the value network trained with ORM in PPO exhibits significant bias when identifying the value of intermediate reasoning steps, a well-known challenge in RL called the credit assignment problem. To address this issue, VinePPO abandons the value network in PPO and instead employs a Monte Carlo sampling method to compute unbiased estimates of the value function. Experimental results demonstrate that VinePPO consistently outperforms typical PPO in mathematical reasoning tasks. Critical Plan Step Learning (CPL) is a method designed to enhance LLM’s generalization in reasoning tasks by searching within high-level abstract plans [150]. CPL employs Monte Carlo Tree Search (MCTS) to explore different planning steps in multi-step reasoning tasks, and utilizes Step-APO to learn critical plan steps. This approach enables models to learn more diverse reasoning paths, thereby improving generalization across various tasks. Subsequently, the model iteratively trains the policy and value models to further enhance performance. During each iteration, the policy model generates plan steps and final solutions, while the value model evaluates the quality of intermediate steps. Training data, generated by MCTS, is used to update both the policy and value models.

Direct preference optimization. In the task of mathematical reasoning, directly employing the DPO [112] method for preference optimization yields suboptimal results due to the presence of lengthy reasoning steps in the preference data. Amini et al. [4] introduced ODPO, which refines DPO by taking into account the degree of preference between responses instead of treating all preference pairs as equal. ODPO has achieved significant improvements over DPO in mathematical reasoning tasks.

In summary, the primary challenge of training based on outcome rewards lies in distinguishing the correctness and importance of intermediate reasoning steps. Current methods, primarily based on Monte Carlo sampling or Monte Carlo Tree Search, offer advantages in estimating the significance of these intermediate steps, though the computational cost during search remains high. Existing work has primarily focused on mathematical or other reasoning problems, where the final solutions can be easily verified. These methods can be extended to a wider range of reasoning tasks, including those where the solutions are difficult to validate. A potential approach is to learn a reward model based on human annotation data, and use it to judge the quality of the final solution. Based on the final score provided by the reward model, Monte Carlo sampling or search techniques can then be employed to further improve performance.

Process Reward Model (PRM) based Reinforcement Learning represents a significant advancement in LLM reasoning, emphasizing the evaluation of intermediate steps rather than solely focusing on end-state outcomes. As shown in Figure 2 (c), the reward of PRM is distributed across each reasoning step, rather than being concentrated at the final outcomes. By providing nuanced feedback throughout the reasoning trajectory, PRM enables models to optimize behavior with greater alignment to human preferences and complex task requirements. This approach is crucial for tasks that involve sequential decision-making, where intermediate steps or decisions are of significance for the final goal. We explore PRMs’ evolution and highlight their role in improving reasoning by providing step-level rewards during complex tasks.

Classic reinforcement learning A series of recent works apply PRMs for mathematical or logistic reasoning, since a seminal work from OpenAI [75] has proven the importance of process reward. SELF-EXPLORE [55] uses PRMs to enhance mathematical reasoning by identifying and addressing “first pits”, which are the initial incorrect steps in problem-solving. By rewarding steps that correct such errors, PRMs enable self-supervised fine-tuning without requiring extensive human annotations. This model achieves significant improvements in accuracy on mathematical benchmarks like GSM8K and MATH by leveraging step-level fine-grained feedback. MATH-SHEPHERD [149] introduces a PRM framework designed for step-by-step verification and reinforcement in mathematical reasoning tasks. By automating process supervision through MCTS-inspired methods, MATH-SHEPHERD eliminates the need for human annotations while ensuring high accuracy in multi-step problem-solving. PRMs are employed to reinforce logical progression and correctness, resulting in improved performance on benchmarks like GSM8K and MATH. DeepSeekMath integrates PRMs via Group Relative Policy Optimization (GRPO) [128], a RL algorithm that optimizes step-level rewards. PRMs are used to enhance mathematical reasoning and reasoning consistency across domains. By focusing on intermediate reasoning steps, DeepSeekMath achieves state-of-the-art performance on several benchmarks, showcasing the power of PRMs in mathematical domains. Scaling Automated Process Verifiers introduces Process Advantage Verifiers (PAVs), a PRM variant, to evaluate step-level progress in problem-solving [123]. PAVs use step-level supervision to improve the efficiency and accuracy of search algorithms and reinforcement learning. By focusing on steps that make meaningful progress toward a correct solution, PAVs enable substantial gains in sample efficiency, compute efficiency, and reasoning accuracy compared to outcome reward models. This demonstrates the importance of fine-grained process rewards in scaling LLM reasoning capabilities.

Interactive process reward models. PRMs are also applied to interactive tasks, such as conversation and multi-turn question answering. ArCHer employs a hierarchical RL approach using PRMs to train agents for multi-turn, long-horizon tasks [198]. It implements a dual-layer system: a high-level value function evaluates utterance-level rewards, while a low-level PRM optimizes token-by-token generation within each turn. This hierarchical structure ensures more effective credit assignment and allows for nuanced training of language models to handle multi-turn interactions and reasoning tasks. The use of PRMs enables ArCHer to scale efficiently, achieving significant gains in sample efficiency and performance across agent tasks. Multi-turn Reinforcement Learning from Preference Human Feedback [126] integrates PRMs into multi-turn reinforcement learning to optimize long-term objectives with human feedback. The Multi-turn Preference Optimization (MTPO) algorithm compares entire multi-turn interactions to generate preference signals, where PRMs are used to assign step-by-step rewards. This enables LLM agents to align behavior with long-term goals, improving overall performance in dynamic, multi-turn tasks such as conversations and strategic decision-making.

Direct preference optimization. Several recent studies leverage MCTS to enable the optimization of multi-step reasoning tasks through Direct Preference Optimization [165, 17, 183, 16]. For instance, SVPO [17] employs MCTS to automatically annotate step-level preferences for multi-step reasoning tasks. From the perspective of learning to rank, it trains an explicit value model to replicate the behavior of an implicit reward model. Furthermore, SVPO integrates the explicit value model with DPO, where the value model not only aids the policy model in navigating more efficient reasoning paths but also guides preference learning. However, these works primarily focus on first collecting preference data or training a reward model and then performing policy optimization based on static data and the pre-trained reward model. Xie et al. [165] advanced these approaches by integrating data collection and policy preference optimization into an iterative process. This method can be considered as an online version of Direct Preference Optimization, where the updated policy is iteratively utilized to collect preferences through MCTS.

The evolution of multi-step RL techniques for LLMs reflects a transition from sparse outcome-based feedback to detailed process-oriented supervision. PRMs now stand as the centerpiece of the progression of LLM’s reasoning ability, offering nuanced, step-level rewards that drive substantial improvements in reasoning tasks. Future research may focus on refining these models and expanding their applicability across diverse task domains.

Reinforcement fine-tuning (RFT) [101] is a technique recently proposed by OpenAI for customizing expert LLMs tailored to specific vertical domains. Currently, RFT remains part of a research program and the technical details have not been fully released. Available information suggests RFT utilizes a small amount of preference data provided by users along with a grader model to evaluate LLM’s output. This technique enables iterative optimization of the LLM’s multi-steps reasoning capabilities. As a result, RFT technique can enhance LLM’s strategy of reasoning through similar problems in the optimized domains.

The grader model. RFT introduces the concept of a grader model to assess the outputs of LLMs. Considering that reinforcement learning training typically requires a reward model to provide feedback, the grader is likely analogous to a reward model, transforming textual inputs (e.g., questions and answers) into scalar values of reasoning quality. This suggests that the grader could act as a reward model trained on user-provided preference data, potentially operating as either an outcome reward model or a process reward model [76].

Data efficiency. In OpenAI’s live sessions, it was mentioned that RFT can enable learning in new domains with as few as dozens of user preference data. This suggests that RFT facilitates the exploration of diverse reasoning paths to address tasks based on limited preference data. Such an approach demonstrates remarkably high sample efficiency while mitigating the risk of overfitting [56].

Training stability. The stability of reinforcement learning training is notoriously difficult problem that presents significant challenges to its broader application. Variations in random seeds or adjustments to certain hyperparameters can greatly impact the training outcomes of RL. In the context of the RFT project, OpenAI has announced plans to make this technology available to the public via APIs, enabling users to fine-tune domain-specific expert models using their own data. This claim potentially indicates that RFT has achieved a level of stability sufficient for reliably fine-tuning language models using RL techniques.

Beyond train-time optimization through techniques such as reinforcement learning, researchers have discovered that test-time prompting techniques like Chain-of-Thought and Tree-of-Thoughts can further enhance LLMs’ capabilities [160, 153]. While simply asking models for direct answers often yields suboptimal results, guiding them through explicit reasoning processes at test-time significantly improves their performance [62]. These prompting strategies have shown remarkable effectiveness across various domains, from mathematical reasoning to complex decision-making tasks [173, 196]. The emergence of structured prompting methods like ReAct and Least-to-Most Prompting has demonstrated that LLMs can benefit from explicit guidance in organizing their thought processes, leading to more reliable and interpretable outputs [189]. Although these approaches typically increase token consumption and computational overhead, they provide a compelling complement to train-time methods by enhancing LLMs’ reasoning capabilities and solution accuracy without requiring model parameter modifications [172, 11]. This suggests a promising direction for improving LLM performance through sophisticated test-time interventions rather than solely relying on model architecture or training modifications.

As previously mentioned, PRM marks a significant shift from sparse outcome-based feedback to detailed process-oriented supervision. Moreover importantly, PRM can also be utilized during the test-time phase, where it can further boosts the model’s reasoning capabilities. OpenAI o1 series models stand as a prominent example of the advanced application of PRM. The new test-time scaling laws suggest that inference capabilities can be effectively enhanced by increasing test-time compute, providing a clear direction for the future development of LLMs. We introduce some methods applied during the inference phase, as shown in Figure 3. Red hollow circles represent the reasoning paths discarded during the algorithm’s exploration process in the inference phase, green hollow circles signify the reasoning paths adopted during exploration, and green solid circles mark the endpoints of the reasoning paths once the correct answer is identified.

Majority Vote: Majority vote is one of the most straight-forward strategy to generate one final answer from dense test-time compute. During inference, each inference trace produces a prediction for a given input. The basic idea is to select the answer which most inference traces accord with. The predictions from all the models are then aggregated, and the class that appears the most (the ”majority vote”) is selected as the final output: $f*=argmax_{f}\sum_{y}\mathbb{I}_{final\_ans(y)=f}$, where $\mathbb{I}$ is the indicator function and $y$ is each evaluation trace.

Tree Search [15]: Tree Search is a classic algorithm that systematically explores different choices by recursively constructing a search tree. It is commonly used in complex decision-making problems, such as board games and planning tasks. Monte Carlo Tree Search (MCTS) is one of the most widely used tree search methods. It consists of four main steps: Selection, Expansion, Simulation, and Backpropagation. By progressively expanding the search space, MCTS incrementally improves decision-making. Tree Search has already been applied in some LLM inference tasks, achieving notable success. For instance, the Tree-of-Thoughts framework [172] enables LLMs to consider multiple reasoning paths structured as a tree. It incorporates self-evaluation to make thoughtful decisions, determining the optimal course of action for the next step. This approach significantly enhances the performance of model inference.

Beam Search [133]: Beam Search is an improved version of greedy search, commonly used in generation tasks to select the optimal output sequence. The main idea is to retain the top-K highest-scoring paths (referred to as beams) at each time step from all candidate paths for further expansion. Unlike greedy search, Beam Search maintains multiple candidate paths, thereby expanding the search space and improving generation quality. Beam Search is widely applied in LLM inference. For example, BART [71] uses Beam Search as its primary inference strategy, demonstrating its outstanding effectiveness in text generation tasks.

Lookahead Search [134]: Lookahead Search is another promising method that has the potential to significantly enhance LLM inference. It modifies the scoring mechanism at each step of Beam Search. Instead of selecting the best candidates based solely on the scores of the current step, Lookahead Search performs forward simulations by rolling out up to $k$ steps ahead. If a solution endpoint is reached during the forward simulation, the process halts early. During Lookahead Search, a pre-trained and frozen Predictive Reward Model is used to score each step of the simulation. The cumulative scores derived from the PRM over the $k$-step simulation are then used to determine whether to retain or discard a beam branch. This strategy improves decision-making by incorporating more context into each evaluation step. Compared to beam search, lookahead search increases the depth of the exploration space, allowing for the judgment of current decision-making based on the results of more distant simulated decisions. However, it also increases the demand for computational resources which also leads to poor performance when computational resources are limited.

In September 2024, OpenAI released o1, a groundbreaking language model that represents a significant advancement in AI reasoning capabilities, particularly excelling in complex tasks like mathematics, coding, and scientific problem-solving. On December 20, 2024, OpenAI opened testing applications for o3, an upgraded version of o1 [102], which is considered to have literal doctorate-level intelligence [7]. These model achieve remarkable results across various challenging benchmarks, including scoring at the gold medal level in International Mathematics Olympiad [73] and matching PhD-level performance in physics, chemistry, and biology questions [48]. Extensive evaluations show distinct reasoning patterns of o1 series through systematic analysis of its basic reasoning capabilities. We list the key findings of existing research as follows:

Open-source frameworks have also made substantial strides in developing advanced reasoning capabilities for LLMs. These frameworks serve as invaluable references for researchers and developers aiming to replicate or approximate the reasoning strengths of proprietary models like OpenAI’s o1. In this section, we introduce four significant open-source efforts, each of which employs distinct strategies to enhance LLM reasoning (summarized in Table 2). By exploring their unique implementations, we aim to provide insights into the diverse methodologies used to reinforce reasoning abilities in LLMs.

Verbal Reinforcement Search (VRS) leverages the pre-trained reasoning and semantic capabilities of LLMs to explore and optimize solution spaces. Unlike traditional reinforcement learning or training-intensive approaches, VRS operates purely through test-time inference, using iterative feedback loops to refine solutions without requiring additional training. By drawing on the semantic knowledge encoded in LLMs and their ability to follow complex instructions, VRS provides a versatile approach for navigating diverse problem spaces. This inference-driven framework finds application across individual agents, multi-agent systems, and embodied agents, supporting a wide range of tasks, including programmatic optimization, collaborative decision-making, and interactions in real-world settings. This section analyzes VRS through these three key aspects, delving into the methodologies and unique insights presented within each category.

In individual agent settings, VRS relies on iterative reasoning and feedback mechanisms to refine solutions within structured problem spaces. This approach is well-suited for tasks like mathematical optimization, symbolic reasoning, and hypothesis-driven discovery, where systematic refinement significantly improves problem-solving outcomes. Research on mathematical discovery illustrates how VRS reshapes the problem-solving process into a dynamic iterative cycle. For example, studies on combinatorial problems, including the cap set and online bin-packing, highlight how programmatic solutions evolve through feedback-driven evaluation [115]. Similarly, symbolic regression research treats equations as dynamic constructs, iteratively generating, evaluating, and optimizing mathematical expressions [130]. These approaches show how VRS navigates constrained spaces, surpassing traditional optimization techniques in efficiency and accuracy. In scientific discovery, VRS has shown its utility in integrating reasoning with empirical data and simulations. Researchers have developed systems for biomedical hypothesis refinement by synthesizing diverse data sources. For instance, applications in oncology use iterative synthesis to address the complexity of multi-scale data [162]. In physical sciences, VRS is used to refine hypotheses through simulation feedback, advancing fields like molecular design and the discovery of physical laws [88]. These findings emphasize the role of VRS in connecting abstract reasoning with real-world validation, supporting tasks that are both data-intensive and hypothesis-driven. Reflective processes in heuristic optimization further showcase the flexibility of VRS. For example, researchers have explored the iterative generation and evaluation of strategies for solving combinatorial problems [174]. This approach focuses on creating adaptive hyper-heuristics that generalize effectively across different domains by continuously refining solutions through feedback cycles. Overall, VRS applies iterative reasoning and feedback to connect abstract problem-solving with real-world applications, addressing challenges in mathematics, science, and optimization with precision and adaptability.

In multi-agent systems, VRS facilitates collaboration between LLM-based agents through natural language communication. These systems leverage shared reasoning and iterative refinement to tackle complex solution spaces, allowing agents to exchange insights and achieve common goals. Meta-structure discovery in heterogeneous information networks (HINs) exemplifies how VRS is applied in multi-agent contexts. Recent research has combined LLM reasoning with evolutionary optimization to refine meta-structures, enhancing their explainability and predictive accuracy [20]. Similarly, in socioeconomic prediction, multi-agent systems integrate knowledge graphs and meta-path reasoning to extract cross-task insights for applications like population estimation and economic activity prediction. This approach facilitates collaboration between LLM agents and improves performance in multi-task environments [199]. Causal discovery also benefits from multi-agent frameworks enabled by VRS. For example, systems using LLMs as reasoning agents collaboratively debate and propose causal relationships. By incorporating statistical methods and natural language interactions, these frameworks generate accurate causal graphs while addressing ambiguities in causal relationships [69]. In financial decision-making, VRS enhances hierarchical collaboration. The FINCON framework employs a manager-analyst system to refine financial strategies using conceptual verbal reinforcement. By minimizing redundant communication and improving strategy refinement, FINCON demonstrates the utility of VRS in optimizing financial decision-making processes [176]. With iterative refinement and shared reasoning, VRS supports multi-agent systems in tackling complex tasks such as meta-structure refinement, socioeconomic prediction, and financial decision-making.

In embodied agent settings, VRS is used to address real-world tasks by integrating reasoning with physical interactions, supporting activities such as experimental planning and execution in laboratory settings. These systems extend VRS into dynamic environments, combining semantic reasoning with practical experimentation. For example, autonomous chemical research has demonstrated the use of LLM-powered systems to independently design, execute, and refine experiments [13]. These agents integrate tools such as robotic liquid handlers, spectrometry devices, and web-based research modules to perform tasks like reaction optimization and compound synthesis. One application involves optimizing palladium-catalyzed cross-coupling reactions, where the system uses natural language prompts to determine conditions, calculate stoichiometries, and autonomously execute experiments. When faced with errors, such as incorrect module calls, the system revises its approach by referencing documentation and iterating on the task. This iterative process demonstrates how VRS supports adaptability and precision in experimental workflows. By combining reasoning and real-time feedback, embodied agents illustrate the capability of VRS to refine and optimize complex processes in dynamic environments. These systems reduce human intervention while accelerating scientific discovery, making them a valuable tool for real-world experimentation and innovation.

In general, previous studies have showcased the adaptability and effectiveness of VRS across individual agents, multi-agent systems, and embodied agents. Leveraging the semantic reasoning and iterative feedback capabilities of LLMs, VRS tackles a wide range of tasks without the need for additional training. From structured optimization in mathematical and scientific contexts to collaborative exploration in multi-agent frameworks, and dynamic experimentation in real-world applications, VRS provides a unified approach to problem-solving. VRS as a versatile framework, capable of addressing complex challenges across both computational and physical domains while driving advancements in diverse fields.

When applied to open-ended tasks such as creative writing, complex logical reasoning, and open-world gaming, the solution space tends to expand dramatically, often becoming unbounded or ill-defined. These tasks typically require continuous interaction with the environment to acquire relevant information, making simple solution space searches inefficient. To address these challenges, some studies incorporate an external memory module for LLM agents. This module stores information such as observations, successful and failed actions from past trials. Agents explore their environments iteratively, using memory as a foundation for verbal reinforcement learning. Through this process, they summarize experience, extract interpretable high-level insights of the solution space, and refine their actions in subsequent trials, thereby improving inference performance. These studies not only focus on exploring the external solution space but also emphasize the intrinsic ability of LLM agents to develop an understanding of the solution space from memory. As the agents accumulate memory through environmental exploration, their capabilities are progressively reinforced and generalized to unseen tasks. Specifically, we classify the studies in this area into the following three categories.

Experiential learning. Methods in this category encourage LLM agents to simply emulate favorable experiences stored in memory while avoiding unfavorable ones. REMEMBERER [184] introduces a semi-parametric RL-LLM agent that records past observation-action pairs in memory and uses a traditional off-policy Q-learning algorithm to dynamically maintain and update the Q-value (expected future reward) of each observation-action pair. When faced with a new task, the agent retrieves relevant actions with the highest and lowest Q-values from memory, incorporating these as encouraged and discouraged examples in the prompt. Memory Sharing [39] leverages concepts from multi-agent reinforcement learning to enhance learning efficiency. Multiple agents execute tasks concurrently in a shared environment and contribute high-quality prompt-answer pairs to a collective memory pool. Each agent can retrieve the most relevant examples from this pool to facilitate few-shot learning. Similarly, Experiential Co-Learning [108] employs a multi-agent framework in which Instructor and Assistant agents alternately provide instructions and solutions during multi-step code generation. This dynamic exchange helps extract shortcuts to reduce redundancy and prevent repetitive mistakes. When encountering new tasks, these agents retrieve relevant memories alternately to improve in-context learning.

Reflective learning. Although using memory as few-shot exemplars is straightforwardly effective, this approach does not fully exploit the semantic comprehension capabilities of LLMs. Some studies argue that LLM agents should reflect directly on the successes and failures stored in memory to summarize underlying causes explicitly, adopting these insights as guidelines. Reflexion [129] is a pioneering effort in this area, reflecting on the reasons behind success or failure semantically based on task feedback signals. It integrates reflective text and past trajectories into prompts to enhance decision-making in subsequent trials. ExpeL [190] combines imitation and reflection by retrieving the most relevant successful experiences from memory, summarizing patterns of successful trajectories, and identifying insights from comparisons of success-failure pairs. RAHL [138], inspired by hierarchical reinforcement learning, organizes memory into goal modules and sub-task modules, enabling reflection and experience summarization at different levels. For new tasks, it retrieves relevant experience to formulate high-level goals and low-level sub-tasks separately.

Concept learning. Explicit reflection significantly enhances the inference capabilities of LLMs. Building on this, some studies aim to enable LLM agents to develop generalized “concepts” that transcend specific tasks, facilitating a broader understanding of the environment and tasks. This generalization helps agents internalize cognitive abilities from memory and evolve continuously as memory grows. Agent-Pro [188], for example, enables agents to establish beliefs about themselves and their environments in card-based gaming. Instead of reflecting on individual actions, it evaluates the rationality and consistency of these beliefs, iteratively refining strategies. Similarly, Richelieu [44] equips agents with an understanding of the environment in military strategy games. It retrieves the most relevant states from memory to formulate plans and assess feasibility. By employing self-play, it collects experience autonomously, assuming the roles of all players to advance its knowledge. Self-Evolving GPT [40], inspired by human memory mechanisms, designs a memory-based autonomous learning framework for LLMs. It categorizes tasks to determine relevant memory retrievals and identifies differences between stored memories and the current task to extract shared general experience. Additionally, it generates unseen tasks for practice, consolidating its knowledge based on memory retrieval outcomes.

The design of agentic systems plays a crucial role in harnessing the power of LLMs for many downstream tasks. An important branch of test-time enhancing techniques is to leverage LLMs to search the agentic systems. Studies in this area can be classified into three levels of search: prompt level, module level, and agent level. Please note that this approach does not aim to directly search the solution space, rather it leverages the empirical data to optimize the agentic system itself, which is similar to a meta-learning problem. We summarize the related works in this area as follows.

Prompt level. The process of “verifying and correcting” improves the prompts by iteratively integrating useful feedback experience. The verification signal can come from external feedback [43], LLM’s self evaluation [90], and other sources. On the other hand, prompts themselves are also worth searching and optimizing. Automated prompt engineering, such as evolutionary prompt optimization [38] and meta prompt iterations [169], can achieve better results than manual prompts, but it also introduces more token consumption.

Module level. Agentsquare [125] proposes to use LLM to search the modular design of agentic system, where the modules are essentially blocks of prompts that have specific functions of Planning, Reasoning, Tool use, and Memory. The basic units of these agentic modules have standard IO interface that allows them to collaborate well with each other. The advantage of module level search is it allows new agents to easily reused the classic agent design, such as CoT and ToT, through recombination of modules. Besides, Aflow[186] connects the different calling nodes of LLM through edges represented by code. In addition to the search method, it is necessary to evaluate the performance of the searched agents. The function used to evaluate the performance of agents can also be driven by LLMs to improve search efficiency while closely matching their actual performance.

Agent level. ADAS proposes to leverage LLMs to search the entire agentic systems defined in python code space [53]. Besides, multi-agent systems make decisions and achieve goals in a shared environment. In multi-agent level search, key aspects include agent creation, environmental perception, action, interaction, and system evolution. The search for multi-agent systems has achieved good results in downstream tasks such as long story creation [54]. The unified search and optimization mechanism for multi-agent systems is currently being explored. GPTSwarm [200] enhances the collaborative capability of agents through graph optimization.

Agentic System Search provides agents with the ability to self-improve, enabling them to optimize themselves to enhance their reasoning abilities without the need to make changes to the LLM structure. The above three levels of search have vast search spaces. The common challenge faced by these three search levels is to improve search efficiency, reduce search costs, and ensure automation while ensuring search rationality.

The test-time enhancing techniques review in this section currently are not incorporated in the implementations of large reasoning models. However, they have huge potential to further boost the reasoning capacities of LLMs through more comprehensive test-time “thinking”, facilitating LLMs to strategically reason across the solution space, leverage past experiences and dynamically optimize agentic workflows. Therefore, training LLMs to master these test-time techniques represents a promising future research direction, with the potential to elevate LLMs from “reasoners” to fully functional “agents.”

Mathematical reasoning has become a crucial testbed for evaluating LLM’s reasoning capabilities. The landscape of mathematical reasoning benchmarks spans from elementary arithmetic to advanced university-level mathematics, providing systematic ways to assess different aspects of mathematical understanding and problem-solving abilities.

In the realm of mathematical word problems (MWP), benchmarks progress from fundamental arithmetic operations to increasingly complex problem-solving scenarios. At the basic level, datasets like MATH-401 [177] evaluate pure arithmetic capabilities through 401 carefully structured expressions, while MultiArith [116] and AddSub [51] assess the ability to translate simple word problems into mathematical operations (such as addition or subtraction). Moving to elementary and high school levels, comprehensive datasets such as GSM8K [24] and MATH [50] present more sophisticated multi-step reasoning challenges, with GSM8K offering 8.5K grade school problems and MATH providing 12.5K problems across various mathematical domains with graduated difficulty levels.

The evaluation of advanced mathematical capabilities is primarily conducted through competition and specialized test datasets. Collections like CHAMP [92] and ARB [5] present competition-level problems that require sophisticated problem-solving strategies, while MATHQA [5] incorporates standardized test questions from GRE and GMAT examinations. At the highest level, datasets such as FIMO [78] challenge models with International Mathematical Olympiad problems, testing the limits of automated mathematical reasoning.

Geometric reasoning represents a distinct category requiring spatial understanding and formal mathematical proofs. Datasets like Geometry3K [82] and GEOQA [19] provide specialized geometric problems, while UniGEO [18] offers a unified framework for geometric reasoning tasks focusing on calculation and proving. These benchmarks are particularly valuable in assessing models’ abilities to connect visual and mathematical reasoning.

The field of theorem proving and formal mathematics has evolved to include rigorous evaluation frameworks. MINIF2F [193] and LeanDojo [170] focus on formal mathematical proofs related to Lean Theorem, while THEOREMQA-MATH [23] examines understanding of mathematical theorems. Specialized datasets like TRIGO [166] and PISA [57] address specific areas of mathematical reasoning, such as trigonometry and formal proof systems.

Lastly, cross-modal mathematical reasoning has emerged as a crucial area, reflecting the diverse ways mathematical problems are presented in real-world scenarios. MATHVISTA [81] and CHARTQA [93] evaluate visual mathematical reasoning through diagrams and charts, while TABMWP [83] and MultiHiertt [192] assess the ability to reason with tabular and textual data. SciBench [151] bridges the gap between pure mathematics and scientific applications, testing mathematical reasoning in broader scientific contexts.

Building upon mathematical reasoning capabilities, the ability to engage in systematic logical reasoning stands as another fundamental criterion for evaluating LLM’s cognitive abilities. While mathematical reasoning focuses on quantitative operations and formal proofs, logical reasoning encompasses the broader capacity to draw valid conclusions, recognize patterns, and generate rational explanations across diverse contexts. According to Luo et al. [86], logical reasoning can be classified into three main types: Deductive, Inductive, and Abductive reasoning. Each type represents a distinct cognitive process essential for comprehensive logical analysis while maintaining interconnections in cognitive assessment.

Deductive reasoning also known as premise-based reasoning, involves deriving specific conclusions from general principles with absolute certainty. For example, given a set of rules about relationships between entities, a model must determine what specific relationships must be true. ProofWriter [140] exemplifies this category, requiring models to construct explicit logical derivations from given premises. Other benchmarks, such as FOLIO [46] and PrOntoQA [119] evaluates first-order logic reasoning in natural contexts, WaNLI [77] have introduced increasingly sophisticated evaluation criteria with 107,885 examples.

Inductive reasoning emphasizes pattern recognition and generalization from specific observations to broader principles [47]. This involves identifying underlying regularities and extending them to new situations, dealing with probability rather than certainty. BigBench [136] with numerous specialized components that examine advanced pattern inference capabilities. Also, CLUTTR [132] benchmark series evaluates this capability through relationship patterns of varying complexity.

Abductive reasoning, also termed explanatory reasoning, refers to the process of forming the most likely explanation for a set of observations or facts, even though the conclusion is not guaranteed to be certain [34]. This type of reasoning tests how models handle scenarios with incomplete information by generating reasonable explanations. The $\alpha$NLI [99] benchmark implements this through narrative completion tasks, where models must select the most likely explanation for given situations. The AbductionRule [175] series offers structured evaluation frameworks across different domains, with specific variants for animal-related and person-related reasoning scenarios. $\alpha$ARCT [100] specifically examines the ability to select and justify plausible explanations and argument comprehension.

Commonsense reasoning remains a significant challenge in NLP, as it aims to evaluate LLM’s ability to understand and apply everyday commonsense knowledge. There are various benchmarks targeting different dimensions of commonsense reasoning tasks. For instance, CommonsenseQA [141] requires models to answer reasoning questions grounded in commonsense knowledge bases.

SocialIQA [118] focus on social interaction commonsense reasoning, which revolves around causal reasoning in social scenarios. In contrast, datasets like SWAG [178] and HellaSwag [179] introduce adversarial text reasoning tasks, where models must predict the most plausible continuation of events based on contextual clues, thereby increasing task complexity. For physical commonsense reasoning, benchmarks such as PIQA [12] and PHYRE [10] concentrate on evaluating models’ understanding of everyday physical tasks and interactive reasoning scenarios. PIQA primarily uses question-answering tasks, while PHYRE emphasizes interactive physical simulations. Similarly, WinoGrande [117] builds upon the Winograd Schema Challenge by introducing a larger-scale dataset and mo re complex disambiguation tasks to test semantic understanding and conference resolution capabilities.

Other works, such as OBQA [95] and CConS [63], explore model performance in counter-commonsense contexts, highlighting the challenges faced by current models in implicit reasoning and background knowledge utilization. More recently, comprehensive benchmarks like MMLU [49] and critical studies such as FactCC [66] have further analyzed LLM’s commonsense reasoning and factual reasoning. These benchmarks offer valuable perspectives on the generalization abilities of language models and serve as valuable tools for evaluating and improving their performance across diverse commonsense reasoning tasks.

The development of code generation benchmarks has been instrumental in evaluating the reasoning capabilities of LLMs in programming tasks. These benchmarks assess models’ proficiency in generating accurate, efficient, and reliable code across various domains. For example, ODEX [155] introduces an execution-driven evaluation framework for open-domain code generation, emphasizing the importance of running generated code to verify its correctness and functionality.

As for the real-world scenarios, SWE-bench [58] focuses on real GitHub issues, challenging models to resolve practical software engineering problems. In the realm of data science, DS-1000 [67] presents a benchmark featuring authentic and dependable data science code generation tasks, enabling the assessment of models’ abilities to handle complex data manipulations and analyses. Besides, the APPS benchmark [49] measures coding challenge competence by evaluating models on a diverse set of programming problems, reflecting the challenges encountered in competitive programming and technical interviews.

MBPP [6] focuses on program synthesis problems, assessing models’ abilities to generate correct and efficient code based on given specifications, thereby contributing to the understanding of LLMs’ capabilities in automated code generation. The HumanEval [21] evaluates LLMs trained on code by providing a set of Python programming problems, each provided with a function definition and accompanying documentation, requiring models to generate correct and functional code solutions.

The emergence of agent-based benchmarks has revolutionized our ability to assess LLMs as independent agents within interactive environments. These sophisticated evaluation frameworks assess crucial capabilities including decision-making, reasoning, and environmental interaction across diverse scenarios.

WebArena [197] provides a practical web environment for building and testing autonomous agents, enabling the evaluation of LLMs’ web navigation and interaction skills. Similarly, Mind2Web [28] aims to develop generalist agents capable of operating across diverse web tasks, emphasizing adaptability in dynamic online environments.

In E-commerce settings, WebShop [171] introduces a platform for scalable real-world web interaction, focusing on grounded language agents that can perform tasks such as online shopping, thereby testing models’ practical application abilities. To bridge textual and embodied environments, ALFWorld [131] aligns text-based inputs with interactive learning scenarios, facilitating the assessment of models’ abilities to transfer knowledge between different modalities.

Comprehensive evaluation frameworks like AgentBench [79] and AgentGym [164] have been developed to systematically assess LLMs functioning as agents. AgentBench includes diverse environments to assess reasoning and decision-making skills, while AgentGym focuses on evolving LLM-based agents across diverse settings, emphasizing adaptability and learning efficiency. Furthermore, AgentBoard [87] offers an analytical platform for evaluating multi-turn LLM agents, providing insights into their performance over extended interactions and highlighting areas for improvement in sustained reasoning tasks.

Even if the breakthrough of OpenAI o1 series at the engineering level remains unknown, theoretically and technically, its breakthrough currently seems to mainly lie in the post-training learning of slow-thinking data. Also, the human cognitive science of “System 1 + System 2” has been repeatedly mentioned, but the ideas on how to implement it based on large models have been constantly updated, mainly still staying at the stage of drawing on the concept of slow thinking. That is, the mechanism named “System 1 + System 2” of human brains has guided the design of LLMs, but the guidance is still very limited. In other words, the imitation of the human brain is only about system-level design rather than very detailed techniques. The complex mechanisms of human slow thinking and their benefits still show high potential to support the next-level reasoning abilities of LLMs. To accomplish it, the domain knowledge of slow thinking should be used in the related designs such as reasoning data generation, reward functions, learning process, etc.

There has been no truly significant and representative work on the theoretical analysis of slow-thinking of LLMs up to now. The generative artificial intelligence is so mysterious that understanding LLMs also requires some tricks or special techniques such as new metrics for understanding hallucination of LLM [37]. To understand the slow-reasoning abilities, we may need to also step into the theoretical analysis. Taking the two different versions, OpenAI o1 Preview and OpenAI o1 Mini, as examples, the main difference lies in the cost and depth of thinking in the CoT inference stage, yet they show significant differences in tasks such as text generation, code generation, and mathematical problem-solving. The special characteristics of reasoning shown by LLMs also inspire us to design task-adaptive usage and applications. Specifically, it may support more interesting insights to link the reasoning mechanism and the performance in different tasks.

As pointed out throughout this paper, the progress of reasoning enhancement technologies is rapid. The reasoning abilities are not limited to the tasks in these popular benchmark tasks, but also in more general tasks in downstream applications. For example, the FunSearch work [115] has shown the general ability for tasks difficult to provide solutions, but verification is fast. There could be a bundle of tasks with similar features in various domains, such as urban planning, logistics scheduling, etc. An interesting question is whether there may be many complementary problems in current research, which is difficult to verify, but the reasoning process is easier. It may be possible to further verify the quality of some answers by combining LLMs and external evaluators, or we can use these answers with evaluated scores to train the reward model.