While large language models (LLMs) excel in many areas, they can handle complex tasks that require precise reasoning. Recent solutions often focus on sophisticated ensemble methods or frameworks where multiple LLM agents work together. These approaches certainly improve performance, but add layers of complexity. However, what if a simpler strategy could lead to significant profits?

This paper investigates a fascinating phenomenon: the possibility of improving the performance of LLM simply by scaling the number of factors used. It introduces an extremely simple method – sampling and voting – which involves generating multiple results from LLMs and using a majority vote to decide the final answer. Let’s dive into the details.

Sampling and Voting Method

At its core, the sampling and voting method is refreshingly simple and involves two phases (see Fig. 2):

• Sampling: The task query is repeatedly fed to an LLM (or a multi-agent LLM context), generating multiple outputs (samples).
• Voting: The majority determines the final answer. For closed-ended tasks (eg, multiple choice), this involves counting the frequency of each choice. For open-ended tasks (eg code generation), similarity measures such as the BLEU score are used to classify samples. The sample with the most similarity to others wins.

This procedure (Algorithm 1) is elegantly agnostic, making it a powerful plug-in for improving existing LLM techniques.

The effectiveness of the method is extensively evaluated in the following three papers:

• Numerical Reasoning: GSM8K and the challenging MATH dataset
• General reasoning: MMLU and a chess state monitoring task
• Generate Code: HumanEval data

To explore the range of benefits, the authors tested language models of different scales, including Llama2, GPT-3.5-Turbo, and GPT-4.

To test how well the method plays with other methods, it was combined with several techniques:

• Direct engineering: Integration with Chain-of-Thought (CoT), Zero-Shot Cot and Solo Performance Prompting.
• LLM Multi-Agent Collaboration: It is used in combination with discussion methods (LLM-Debate) and self-reflection.

The results offer fascinating insights:

• Performance Scaling: Increasing the number of agents generally enhances the performance of LLM on tasks and models of different sizes. Surprisingly, smaller LLMs, when scaled up, often compete with or outperform their larger counterparts (Fig. 1).
• Compatibility: The method combines seamlessly with other techniques, leading to even greater performance gains.
• Simplicity vs. Complexity: In most cases, the proposed method alone achieves results equal to more complex approaches, indicating strength in its simple design.

Thorough experiments demonstrate the consistency of the method across hyperparameters (Fig. 4) and reveal a key point: performance gains are positively correlated with task difficulty (Table 5). To unpack this relationship, three dimensions of difficulty are isolated:

• Inherent Difficulty: Profits first increase and then decrease as problems become extremely complex.
• Number of Steps: The gains become more pronounced as the steps required to solve the task increase.
• Prior probability: Performance improves when the probability of a correct answer is greater.

These findings inspired optimizations such as incremental or hierarchical sampling and voting, maximizing profits through a nuanced understanding of task difficulty.

In conclusion, this work establishes a new benchmark, proving that sometimes, “more agents” may indeed be all you need. In many cases, scaling LLM agents with a simple sampling and voting strategy significantly improves performance without complex methods. This discovery simplifies complex LLM applications and paves the way for cost optimization of future systems, the focus of ongoing research.

check it Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us Twitter and Google news. Participation Our 38k+ ML SubReddit, 41k+ Facebook Community, Discord Channeland LinkedIn Groops.

If you like our work, you will love our work newsletter..

Don’t forget to join us Telegram channel

You might also like ours FREE AI Courses….