Multi-agent Self-improving of Large Language Models (LLMs)

大型语言模型 (LLM) 的多智能体自我改进

• 批准号: 2903811
• 金额: --
• 依托单位: University of Edinburgh
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2024
• 资助国家: 英国
• 起止时间: 2024 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2903811
• 关键词: Multi; agent; Self; improving; Large

In the rapidly evolving field of artificial intelligence (AI), Large Language Models (LLMs) stand out as powerful tools capable of understanding human instructions and generating helpful answers. However, the development of these models faces significant challenges. In general, improving LLMs' generation ability and aligning their generation with human values rely heavily on vast amounts of human feedback annotations. This approach, while effective, is difficult to scale and may inherently limit the models' potential. As an alternative, some researchers turn to train LLMs using self-generated data, i.e., self-learning. Self-learning also presents a set of problems, including the risk of reinforcing existing biases or inaccuracies without external correction. This dilemma sets the stage for a novel approach to advancing LLM capabilities without substantial demand for human resources or the pitfalls of self-learning. This project tries to propose an innovative self-improving framework through a multi-agent system that enables these models to learn and enhance themselves by leveraging feedback from other peer models. By integrating the strengths and diversity of various LLMs, the system is expected to refine its ability to follow instructions, align with human values, and perform across a broad spectrum of downstream tasks with minimal human supervision. The vision is to establish a scalable and efficient method for continuous improvement through inter-model interactions, sidestepping the constraints of human feedback and the limitations of self-generated data training. At the heart of this self-improving system are two pivotal questions: 1. Can the diversity of LLMs enrich the quality of self-generated training data? 2. Can collaboration among different LLMs reduce the necessity for human annotations while ensuring ongoing enhancement? Addressing these two open queries could open the door to a new paradigm in AI training/alignment methodologies. This exploration aims at fostering more efficient AI systems development with reduced reliance on human oversight and intervention. This project, therefore, is also an open-ended exploration into future AI training strategies. It seeks to contribute to the AI community by moving away from heavily human-supervision-dependent models to more data-efficient and self-improving LLM systems.

在快速发展的人工智能（AI）领域，大型语言模型（LLM）作为能够理解人类指令并生成有用答案的强大工具脱颖而出。然而，这些模式的发展面临重大挑战。一般来说，提高LLM的生成能力并使其生成与人类价值观相一致在很大程度上依赖于大量的人类反馈注释。这种方法虽然有效，但难以扩展，并且可能固有地限制模型的潜力。作为替代方案，一些研究人员转向使用自我生成的数据来训练LLM，即，自学自学也带来了一系列问题，包括在没有外部纠正的情况下加强现有偏见或不准确性的风险。这种困境为一种新的方法奠定了基础，以提高LLM的能力，而无需大量的人力资源需求或自学的陷阱。该项目试图通过多代理系统提出一个创新的自我改进框架，使这些模型能够通过利用其他对等模型的反馈来学习和增强自己。通过整合各种LLM的优势和多样性，该系统有望改进其遵循指令的能力，与人类价值观保持一致，并在最少的人为监督下执行广泛的下游任务。我们的愿景是建立一种可扩展的、高效的方法，通过模型间的交互进行持续改进，避开人类反馈的约束和自我生成数据训练的局限性。这个自我完善系统的核心是两个关键问题：1。LLM的多样性能否丰富自我生成的训练数据的质量？2.不同LLM之间的协作是否可以在确保持续增强的同时减少人工注释的必要性？解决这两个开放的问题可以为人工智能训练/对齐方法的新范式打开大门。这种探索旨在促进更有效的人工智能系统开发，减少对人类监督和干预的依赖。因此，该项目也是对未来人工智能培训策略的开放式探索。它旨在通过从严重依赖人类监督的模型转向更有效的数据和自我改进的LLM系统来为AI社区做出贡献。