Model design for multi-modal tasks

多模态任务的模型设计

• 批准号: 2894242
• 金额: --
• 依托单位: Imperial College London
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2023
• 资助国家: 英国
• 起止时间: 2023 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2894242
• 关键词: Model; design; multi; modal; tasks

In recent years, artificial intelligence has paved the way for significant advancements across various domains, particularly with the advent of generative models. Generative models, known for their remarkable ability to produce highly realistic content, have demonstrated exceptional potential in a wide range of applications, ranging from text generation [1] to image synthesis [2]. However, to further enhance the performance and adaptability of these generative models in real-world scenarios, it is imperative that we explore innovative techniques to facilitate efficient and scalable model specialization. Notably, contemporary generative models often come burdened with intensive computational requirements and lack the flexibility to specialize in specific tasks, thereby limiting their utility across diverse domains.Simultaneously, we have witnessed a remarkable growth of diverse generative models, pretrained or fine-tuned for specific tasks, scenarios, and domains. As these pretrained generative models are already equipped with certain abilities for specific tasks, the question of how to leverage them for customization to more complex tasks has become a pressing concern. To address these challenges, this research aims to render these models amenable to personalized specialization, accommodating the diverse requirements of individual researchers and practitioners.This research will be guided by the following specific objectives and potential research directions: (1) Comprehensive Exploration of Generative Models: We will conduct a thorough examination of the current landscape of generative models. Our investigation will encompass a detailed analysis of their applications, strengths, and areas where optimization can enhance their efficiency. (2) Investigation of Diverse Model Editing [3] and Fusion [4] Techniques: We will embark on an exploration of various model editing and fusion techniques with the aim of comprehending their potential in augmenting the capabilities of generative models. These techniques are intended to reduce the training costs associated with model personalization and enhance their performance through fusion with diverse models from different modalities. (3) Innovation and Evaluation of Model Specialization Strategies: With a specific focus on challenging tasks, such as those involving embodied agents, we will conceive and assess innovative strategies for model specialization derived from our proposed model editing and fusion techniques. The goal is to combine different models and employ automated search for different models [5] to tackle complex real-world tasks effectively.In summary, this research seeks to make a meaningful contribution to the evolving landscape of generative models, with a specific emphasis on their efficiency and adaptability. Our work is in alignment with the artificial intelligence and robotics thematic area in the EPSRC's remit, which aims to provide valuable tools to efficiently tailor generative models to unique needs and requirements. Thus, they can be applied to a wide spectrum of applications, including fields such as medical image generation, drug discovery, image super-resolution, and content generation for games and so on. [1] Zou, H., Kim, Z. M., & Kang, D. (2023). Diffusion models in nlp: A survey. arXiv preprint arXiv:2305.14671.[2] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).[3] Mitchell, E., Lin, C., Bosselut, A., Finn, C., & Manning, C. D. (2021). Fast model editing at scale. arXiv preprint arXiv:2110.11309.[4] Singh, S. P., & Jaggi, M. (2020). Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33, 22045-22055.[5] Significant Gravitas. AutoGPT [Computer software]. https://github.com/Significant-Gravitas/AutoGPT.

近年来，人工智能为各个领域的重大进步铺平了道路，特别是随着生成模型的出现。生成模型以其产生高度逼真内容的卓越能力而闻名，在广泛的应用中展示了非凡的潜力，从文本生成[1]到图像合成[2]。然而，为了进一步提高这些生成模型在现实场景中的性能和适应性，我们必须探索创新技术来促进高效和可扩展的模型专业化。值得注意的是，当代生成模型往往负担着密集的计算需求，缺乏专注于特定任务的灵活性，从而限制了它们在不同领域的实用性。同时，我们也见证了各种生成模型的显著增长，这些模型针对特定的任务、场景和领域进行了预训练或微调。由于这些预训练的生成模型已经具备了特定任务的某些能力，如何利用它们来定制更复杂的任务已经成为一个紧迫的问题。为了应对这些挑战，本研究旨在使这些模型能够适应个性化的专业化，适应个体研究人员和实践者的不同需求。本研究将以以下具体目标和潜在的研究方向为指导：(1)生成模型的综合探索：我们将对生成模型的现状进行全面的考察。我们的调查将包括对它们的应用、优势和优化可以提高其效率的领域的详细分析。(2)多种模型编辑[3]和融合[4]技术的研究：我们将着手探索各种模型编辑和融合技术，目的是了解它们在增强生成模型能力方面的潜力。这些技术旨在降低与模型个性化相关的训练成本，并通过融合来自不同模式的各种模型来提高其性能。(3)模型专业化策略的创新和评估：我们将特别关注具有挑战性的任务，例如涉及具身代理的任务，我们将构思和评估基于我们提出的模型编辑和融合技术的模型专业化创新策略。目标是结合不同的模型，并对不同的模型进行自动搜索，以有效地处理复杂的现实世界任务。总之，本研究旨在对生成模型的发展前景做出有意义的贡献，特别强调它们的效率和适应性。我们的工作与EPSRC职权范围内的人工智能和机器人主题领域保持一致，旨在提供有价值的工具，以有效地定制生成模型以满足独特的需求和要求。因此，它们可以应用于广泛的应用领域，包括医学图像生成、药物发现、图像超分辨率和游戏内容生成等领域。[10]邹辉，金志明，康东，（2023）。nlp中的扩散模型综述。arXiv预印本arXiv:2305.14671Rombach， R., Blattmann， A., Lorenz， D., Esser， P., & Ommer， B.（2022）。基于潜在扩散模型的高分辨率图像合成。《IEEE/CVF计算机视觉与模式识别会议论文集》（第10684-10695页）Mitchell， E., Lin， C., Bosselut， A., Finn， C., and Manning， C.（2021）。快速模型编辑的规模。arXiv预印本arXiv:2110.11309.[4]Singh， S. P, & Jaggi， M.（2020）。通过最优输运的模型融合。神经信息处理系统进展，33,2245 - 2255 .[j]重要的庄严。AutoGPT[计算机软件]。https://github.com/Significant-Gravitas/AutoGPT。