Unified Transcription And Translation For Extended Reality

UTTER的目标是利用大型语言模型来构建下一代多模态扩展现实（XR）技术，用于转录、翻译、摘要和记录。我们将使这些技术具有可扩展性、适应性、情境化、健壮性、可解释性和情感感知性。我们将提高这些技术的上下文敏感性，这样它们就可以考虑到对话的全部历史，以及更广泛的背景。我们将引入信心感知模型，它可以考虑到自身的局限性。我们将开发可解释的模型，这样人类用户就可以知道为什么模型做出了它所做的决定。我们将改进适应性，以便能够快速推出特定于领域和特定于语言的模型。对于这些进步，我们将利用预训练的扩展现实（XR）模型，该模型最佳地结合了文本和语音信号，并通过适配器和提示进行了有效的训练。我们还将研究如何高效部署这些大型复杂模型，使它们以节能的方式投入生产。我们的用例原型将包括(i)用于会议的个人助理，可以改善在线世界的沟通；（ii）支持全球市场的高级客户服务助理。这些原型将在整个项目中开发和测试，并每年发布和评估。通过我们的级联拨款计划，以及我们发布的工具，以促进使用预训练的XR模型，将使这些技术在整个欧洲得到采用和发展。

The aim of UTTER is to leverage large language models to build the next generation of multimodal eXtended reality (XR) technologies for transcription, translation, summarisation, and minuting. We will make these technologies scalable, adaptable, contextualised, robust, explainable, and emotion-aware. We will increase the context-sensitivity of the technologies, so they can take into account the full history of the conversation, as well as its wider context. We will introduce confidence-aware models, which can take into account their own limitations. We will develop explainable models, so the human user can know why the model made the decisions it did. We will improve adaptation, so that domain-specific and language-specific models can be quickly rolled out. For these advances we will make use of pretrained eXtended reality (XR) models, which optimally combine text and speech signals, and are trained efficiently with adapters and prompting. We will also develop efficient methods to deploy such large and complex models, so that they can put into production in an energy-efficient manner. Our use-case prototypes will cover (i) A personal assistant for meetings that can improve communication in the online world and (ii) an advanced customer service assistant to support global markets. These prototypes will be developed and tested throughout the project, with annual releases and evaluations. Through our cascaded grant programme, and our release of tools to facilitate the use of pre-trained XR models, will enable the take-up and development of these technologies throughout Europe.