LUMINOUS - Language Augmentation for Humanverse

LUMINOUS - Humanverse 的语言增强

• 批准号: 10106564
• 金额: $ 12.29万
• 依托单位: UNIVERSITY COLLEGE LONDON
• 依托单位国家: 英国
• 项目类别: EU-Funded
• 财政年份: 2024
• 资助国家: 英国
• 起止时间: 2024 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=10106564
• 关键词: LUMINOUS; Language; Augmentation; Humanverse

LUMINOUS aims at the creation of the next generation of Language Augmented XR systems, where natural language-based communication and Multimodal Large Language Models (MLLM) enable adaptation to individual, not predefined user needs and unseen environments. This will enable future XR users to interact fluently with their environment, while having instant access to constantly updated global as well as domain-specific knowledge sourcesto accomplish novel tasks. We aim to exploit MLLMsinjected with domain specific knowledge for describing novel tasks on user demand. These are then communicated through a speech interface and/or a task adaptable avatar (e.g., coach/teacher) in terms of different visual aids and procedural steps for the accomplishment of the task. Language driven specification of the style, facial expressions, and specific attitudes of virtual avatars will facilitate generalisable and situation aware communication in multiple use cases and different sectors. LLMs will benefit in parallel in identifying new objects that were not part of their training data and then describing them in a way that they become visually recognizable. Our results will be prototyped and tested in three pilots, focussing on neurorehabilitation (support of stroke patients with language impairments), immersive industrial safety training, and 3D architectural design review. A consortium of six leading R&D institutes experts in six different disciplines (AI, Augmented Vision, NLP, Computer Graphics, Neurorehabilitation, Ethics) will follow a challenging workplan, aiming to bring about a new era at the crossroads of two of the most promising current technological developments (LLM/AI and XR), made in Europe.

LUMINOUS旨在创建下一代语言增强XR系统，其中基于自然语言的通信和多模态大型语言模型（MLLM）能够适应个人，而不是预定义的用户需求和看不见的环境。这将使未来的XR用户能够与他们的环境进行流畅的交互，同时能够即时访问不断更新的全球以及特定领域的知识源，以完成新的任务。我们的目标是利用MLLMsinjected领域特定的知识来描述新的任务，用户的需求。然后通过语音接口和/或任务自适应化身（例如，教练/教师）在不同的视觉辅助和程序步骤方面完成任务。虚拟化身的风格，面部表情和特定态度的语言驱动规范将促进多个用例和不同部门中的可概括和情景感知通信。LLM将同时受益于识别不属于其训练数据的新对象，然后以视觉上可识别的方式描述它们。我们的研究结果将在三个试点中进行原型和测试，重点是神经康复（支持有语言障碍的中风患者），沉浸式工业安全培训和3D建筑设计审查。由六个不同学科（AI，增强视觉，NLP，计算机图形学，神经康复，伦理学）的六个领先研发机构专家组成的联盟将遵循一项具有挑战性的工作计划，旨在在欧洲制造的两个最有前途的当前技术发展（LLM/AI和XR）的十字路口开创一个新时代。