What is Self-Consistency prompting?
Self-Consistency prompting is a technique used to improve the accuracy and reliability of responses generated by Large Language Models (LLMs). It is categorized as a text-based technique, specifically within the family of ensembling methods.
The core idea behind Self-Consistency is the intuition that multiple different reasoning paths can lead to the same correct answer. It combines sampling and majority voting to generate diverse reasoning paths and select the most consistent answer, thereby improving accuracy and coherence. This approach contrasts with the simple 'greedy decoding' strategy often used in basic Chain of Thought (CoT) prompting.
The process for implementing Self-Consistency typically involves the following steps:
- Generating diverse reasoning paths: The LLM is provided with the same prompt multiple times. To encourage the model to generate different reasoning paths and perspectives, a high or non-zero temperature setting is crucial. For instance, benchmarking experiments used a temperature of 0.5 for Self-Consistency, while other prompts used a temperature of 0.
- Extracting the answer: The final answer is extracted from each generated response. This extraction needs to be separate from the reasoning steps.
- Choosing the most common answer: A majority vote is conducted over all the generated responses to select the final answer. This statistically increases the likelihood of the answer being correct. Universal Self-Consistency is a similar variation that inserts all outputs into a prompt template that selects the majority answer.
Self-Consistency has demonstrated improvements on various tasks, including arithmetic, commonsense, and symbolic reasoning. It is particularly valuable in domains where factual precision is imperative, such as fact-checking and information verification. While effective, Self-Consistency has higher costs compared to simpler methods like greedy decoding due to the need for multiple generations. It can be used in conjunction with other techniques, such as Zero-Shot CoT to select examples for Few-Shot CoT prompts.