How Does Decoding Affect Reasoning in LLMs?

The quality and diversity of language models’ outputs depend on the decoding techniques. Greedy decoding gives you the word / token which has the highest probability of all possible tokens in the model’s vocabulary. The disadvantage of greedy decoding is that it always generates the same token (with top-1 probability) with the same inputs, leading to a lack of creativity for the generated text. Sampling avoids this issue by introducing some randomness to generate tokens with low probability. However, too much randomness will also affect the accuracy or fluency of models’ generations.

Despite the varying decoding techniques, how do they affect reasoning in LLMs? Let’s take a look at one math reasoning example from the paper Chain-of-Thought Reasoning without Prompting:

From this math example, we can see that for decoding step 0, using varying top-k tokens leads to different reasoning paths, while only two paths infer the final correct answer. This example shows the importance of decoding techniques in the accuracy of reasoning tasks.

Therefore, how can we ensure both the accuracy and diversity of generated reasoning paths from LLMs?

In our recent paper, Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs, we propose a simple technique, i.e., selective sampling, to switch between greedy and high-temperature sampling for each decoding position to ensure both the quality and diversity of the reasoning path generation. We assume greedy tends to generate more accurate answers, while high-temperature sampling increases the diversity of reasoning paths. Therefore, this “switch” is determined based on a sampling risk, which estimates the likelihood of output errors when applying high-temperature sampling at that position.

In the figures below, we show that applying min-p or top-p sampling techniques leads to wrong outputs (red squares) at the high-sampling-risk decoding positions.

To predict the sampling risk, we train a lightweight classifier to discriminate between safe and high-risk prefixes. The figure below shows the high-sampling-risk predictions (orange) from our classifier. As expected, high-risk tokens occur more often at integer numbers, which are essential for the math computation.

Based on our experiment results, we show how selective sampling outperforms other comparable sampling techniques, such as min-p, in both accuracy and diversity.

More interestingly, we show the percentages of adopting greedy and high-temperature sampling under different temperatures. Here, we can see that for higher temperatures or harder tasks, selective sampling chooses greedy decoding more than high-temperature sampling. This observation is expected, as higher temperatures or more challenging tasks tend to degrade the quality. Using greedy decoding more frequently in such cases results in a more reliable strategy to maintain the output quality.

In summary, decoding affects reasoning in LLMs in both accuracy and diversity. Future work can explore more directions in dynamic or task-specific decoding for LLM generation.