Enhanced AI Perception through Unified Joint Embedding of Multimodal Sensory Data

人工智能（AI）是一个旨在创造能够像人类一样思考和学习的机器的领域，它的子集深度学习使用多层神经网络来表示人类大脑，严重依赖大量标记数据来训练机器。这种依赖性通常会阻碍它们在动态的真实场景中的应用。相比之下，人类天生处理和交织多种感官-从听到城市声音的交响乐到感受物体的精细纹理和视觉上的距离。这种融合我们的感官和解释我们周围环境的自然能力可能是人工智能进化中缺失的一环。这就提出了一个核心研究问题：多感官数据的整合能否缩小人类认知和机器学习之间的差距，使机器能够更多地从自然感官体验中学习，而更少地从大量的标记数据中学习？本研究的目的如下。首先，本研究旨在开发能够从不同的感官输入中提取结构的多模态计算模型。为了解决标记的多模态数据集的稀缺性，我们利用自然发生的配对数据来区分特定于模态的信息并将其整合，从以视听线索启动的人类学习过程中汲取灵感（例如，我们将特定鸟类的声音与其照片相匹配）。其次，这项研究旨在通过整合新的感觉方式来扩展AI的感知范围，包括热数据，触觉信号和空间深度。这种整合旨在扩大AI的感知范围，超越传统的视觉和音频模式。第三，本研究旨在建立一个整体感知系统。所开发的多模态计算模型将被训练以同时处理各种各样的感觉模态，包括文本和视觉线索、听觉信号、空间深度、热感觉、IMU读数、触觉信号等。首先，本研究采用对比学习法。这种技术可以帮助模型找到来自不同模态的数据点之间的相似性和差异。因此，这些模型可以将模式关联起来，并在多个模态之间建立联系。其次，本研究探讨并关联了一些新的情态对，如视觉深度和视觉触觉，这在以前没有得到广泛的研究。此外，本研究旨在超越传统的双模态嵌入，并从多种自然共生模态中开发出一个统一的嵌入景观。在这种情况下，嵌入是简化复杂数据以供机器解释的基本数学表示。通过利用最先进的大型语言模型（LLM）和视觉语言模型（VLM），我们的目标是识别每个罕见模态的特定特征，匹配配对，并建立它们之间的联系。这得益于LLM/VLM的零触发学习功能，该功能使模型能够解释和执行以前从未遇到过的任务。因此，该模型可以更好地编码信息从各种感官形式到一个统一的，联合嵌入式空间。这个项目属于EPSRC的人工智能技术研究领域的福尔斯。我们渴望通过探索多模态感知数据的联合嵌入来改善人工智能的感知。我们改进的方法旨在开发一个更加互联和丰富的嵌入式空间，增强其对各种下游任务的适应性。例如，在机器人自动化领域，从这项研究中得出的统一的联合嵌入有可能显着提高机器人的感知能力，并在不同的场景中彻底改变操作效率。

Artificial Intelligence (AI), a field aiming to create machines that can think and learn like humans, and its subset, Deep Learning, which uses multi-layered neural networks to represent human brains, rely heavily on large collections of labeled data to train the machines. This dependency often hinders their application in dynamic, real-world scenarios. In contrast, humans natively process and intertwine multiple senses - from hearing the symphony of urban sounds to feeling the object's fine textures and interpreting distances visually. This natural ability to blend our senses and interpret our surroundings can potentially be the missing link in AI's evolution. This raises the central research question: can the integration of multisensory data close the gap between human cognition and machine learning, so that machines can learn more from natural sensory experiences and less from extensive labeled data?The objectives of this research are as follows. First, this research aims to develop multimodal computational models capable of extracting structures from diverse sensory inputs. To address the scarcity of labeled multimodal datasets, we leverage naturally occurring paired data to distinguish between modality-specific information and integrate them, drawing inspiration from human learning processes initiated with audio-visual cues (e.g., we match the sound of a specific bird to its photo). Second, this research seeks to extend AI's perception range by incorporating novel sensory modalities, including thermal data, tactile signals, and spatial depth. This integration aims to augment AI's perceptual range beyond the conventional modalities of vision and audio. Third, this research aims to establish a holistic perception system. The developed multimodal computational model will be trained to process a wide array of sensory modalities concurrently, covering text and visual cues, auditory signals, spatial depth, thermal sensations, IMU readings, tactile signals, etc.The novelty of the research methodology is as follows. First, this research uses contrastive learning. This technique helps models to find similarities and differences across data points from different modalities. Consequently, the models can correlate patterns and build connections across multiple modalities. Second, this research explores and associates novel modality pairs such as visual-depth and visual-touch, which have not been extensively researched before. Moreover, this research aims to move beyond the traditional dual-modality embeddings and develops a unified embedding landscape from diverse naturally co-occurring modalities. In this context, embeddings are essential mathematical representations that simplify complex data for machine interpretation. By leveraging state-of-the-art Large Language Models (LLMs) and Vision-Language Models (VLMs), we aim to identify features specific to each rare modality, match the pairs, and establish connections between them. This is facilitated by the zero-shot learning capabilities of LLMs/VLMs, a feature that enables models to interpret and execute tasks they have never encountered before. Consequently, the model can better encode information from various sensory modalities into a unified, joint embedded space.This project falls within the EPSRC's Artificial Intelligence Technologies research area. We aspire to improve AI's perception through the exploration of joint embedding of multimodal sensory data. Our refined approach aims to develop a more interconnected and enriched embedded space, enhancing its adaptivity to a variety of downstream tasks. For example, in the field of robotic automation, the unified, joint embedding derived from this research have the potential to significantly improve robots' perception and revolutionize operational efficiency across different scenarios.