Multi-agent Self-improving of Large Language Models (LLMs)

点击翻译按钮获取中文摘要

In the rapidly evolving field of artificial intelligence (AI), Large Language Models (LLMs) stand out as powerful tools capable of understanding human instructions and generating helpful answers. However, the development of these models faces significant challenges. In general, improving LLMs' generation ability and aligning their generation with human values rely heavily on vast amounts of human feedback annotations. This approach, while effective, is difficult to scale and may inherently limit the models' potential. As an alternative, some researchers turn to train LLMs using self-generated data, i.e., self-learning. Self-learning also presents a set of problems, including the risk of reinforcing existing biases or inaccuracies without external correction. This dilemma sets the stage for a novel approach to advancing LLM capabilities without substantial demand for human resources or the pitfalls of self-learning. This project tries to propose an innovative self-improving framework through a multi-agent system that enables these models to learn and enhance themselves by leveraging feedback from other peer models. By integrating the strengths and diversity of various LLMs, the system is expected to refine its ability to follow instructions, align with human values, and perform across a broad spectrum of downstream tasks with minimal human supervision. The vision is to establish a scalable and efficient method for continuous improvement through inter-model interactions, sidestepping the constraints of human feedback and the limitations of self-generated data training. At the heart of this self-improving system are two pivotal questions: 1. Can the diversity of LLMs enrich the quality of self-generated training data? 2. Can collaboration among different LLMs reduce the necessity for human annotations while ensuring ongoing enhancement? Addressing these two open queries could open the door to a new paradigm in AI training/alignment methodologies. This exploration aims at fostering more efficient AI systems development with reduced reliance on human oversight and intervention. This project, therefore, is also an open-ended exploration into future AI training strategies. It seeks to contribute to the AI community by moving away from heavily human-supervision-dependent models to more data-efficient and self-improving LLM systems.