LUMINOUS - Language Augmentation for Humanverse

LUMINOUS旨在创建下一代语言增强XR系统，其中基于自然语言的通信和多模态大型语言模型（MLLM）能够适应个人，而不是预定义的用户需求和看不见的环境。这将使未来的XR用户能够与他们的环境流畅地交互，同时能够即时访问不断更新的全球以及特定领域的知识来源，以完成新的任务。我们的目标是利用注入领域特定知识的mllms来描述用户需求的新任务。然后通过语音界面和/或任务适应性化身（如教练/老师）根据不同的视觉辅助和完成任务的程序步骤进行交流。语言驱动的虚拟化身的风格、面部表情和特定态度的规范将促进在多个用例和不同部门的通用和情境感知通信。法学硕士将同时受益于识别不属于他们训练数据的新对象，然后以一种视觉上可识别的方式描述它们。我们的研究结果将在三个试点中进行原型和测试，重点是神经康复（支持有语言障碍的中风患者）、沉浸式工业安全培训和3D建筑设计审查。由六个不同学科（人工智能、增强视觉、NLP、计算机图形学、神经康复、伦理学）的六个领先研发机构专家组成的联盟将遵循一项具有挑战性的工作计划，旨在在欧洲制造的两项最有前途的当前技术发展（LLM/AI和XR）的十字路口开创一个新时代。

LUMINOUS aims at the creation of the next generation of Language Augmented XR systems, where natural language-based communication and Multimodal Large Language Models (MLLM) enable adaptation to individual, not predefined user needs and unseen environments. This will enable future XR users to interact fluently with their environment, while having instant access to constantly updated global as well as domain-specific knowledge sourcesto accomplish novel tasks. We aim to exploit MLLMsinjected with domain specific knowledge for describing novel tasks on user demand. These are then communicated through a speech interface and/or a task adaptable avatar (e.g., coach/teacher) in terms of different visual aids and procedural steps for the accomplishment of the task. Language driven specification of the style, facial expressions, and specific attitudes of virtual avatars will facilitate generalisable and situation aware communication in multiple use cases and different sectors. LLMs will benefit in parallel in identifying new objects that were not part of their training data and then describing them in a way that they become visually recognizable. Our results will be prototyped and tested in three pilots, focussing on neurorehabilitation (support of stroke patients with language impairments), immersive industrial safety training, and 3D architectural design review. A consortium of six leading R&D institutes experts in six different disciplines (AI, Augmented Vision, NLP, Computer Graphics, Neurorehabilitation, Ethics) will follow a challenging workplan, aiming to bring about a new era at the crossroads of two of the most promising current technological developments (LLM/AI and XR), made in Europe.