RI: Small: Lifelong Multimodal Concept Learning

虽然近年来机器学习和人工智能取得了很大的进步，但这些系统仍然有很大的局限性。机器学习系统有不同的学习和部署阶段。如果获得了新的信息，整个系统通常会被重建，而不是只学习新的信息，否则系统会忘记大量过去的知识。系统不能自主学习，往往需要强有力的监督。该项目旨在通过创建新的多模态大脑启发算法来解决这些问题，这些算法能够立即学习而不会过度遗忘。这些算法可以用更少的计算资源进行学习，这可以促进在手机和家用机器人等设备上的学习。从多模态数据流中快速学习对于实现与人工智能体的自然交互至关重要。自主多模态学习将减少对标注数据的依赖，这是提高人工智能实用性的巨大瓶颈，并可能显著提高性能。这项研究将为其他人提供构建模块，用于创建新的算法、应用程序和认知技术。这些算法是基于互补学习系统理论来解释人脑如何快速学习的。人类大脑利用海马体立即学习新信息，然后在睡眠时将这些信息转移到新皮层。基于这一理论，深度神经网络的流学习算法将被创建，这将使从结构化数据流中快速学习而不会灾难性地忘记过去的知识。这些算法将根据其对包含数千个类别的大型图像数据库进行分类的能力进行评估。这些系统将成为视觉问答和视觉查询检测的多模态流学习的先驱，使语言能够为视觉场景的理解提供信息。这些特征将被整合在一起，使模型能够在有限的人类监督下自主地查询环境。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

While machine learning and artificial intelligence has greatly advanced in recent years, these systems still have significant limitations. Machine learning systems have distinct learning and deployment phases. If new information is acquired, the entire system is often rebuilt rather than having only the new information being learned because otherwise the system will forget a large amount of its past knowledge. Systems cannot learn autonomously and often require strong supervision. This project aims to address these issues by creating new multi-modal brain-inspired algorithms capable of learning immediately without excess forgetting. These algorithms can enable learning with fewer computational resources, which can facilitate learning on devices such as cell phones and home robots. Fast learning from multimodal data streams is critical to enabling natural interactions with artificial agents. Autonomous multimodal learning will reduce reliance on annotated data, which is a huge bottleneck in increasing the utility of artificial intelligence, and may enable significant gains in performance. This research will provide building blocks that others can use to create new algorithms, applications, and cognitive technologies.The algorithms are based on the complementary learning systems theory for how the human brain learns quickly. The human brain uses its hippocampus to immediately learn new information and then this information is transferred to the neocortex during sleep. Based on this theory, streaming learning algorithms for deep neural networks will be created, which will enable fast learning from structured data streams without catastrophic forgetting of past knowledge. The algorithms will be assessed based on their ability to classify large image databases containing thousands of categories. These systems will be leveraged to pioneer multimodal streaming learning for visual question answering and visual query detection, enabling language to inform understanding of visual scenes. These traits will be integrated to enable a model to autonomously query an environment with limited human supervision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.