EFRI BRAID: Neuroscience Inspired Visual Analytics

智能取决于机器学习和存储关于世界的知识的效率，使其能够处理意想不到的任务和新环境，在没有监督的情况下快速学习，解释决策，推断未被观察到的情况，并预测可能的结果。当前机器学习方法的一个关键限制是它们针对不同任务、环境和上下文进行的专门训练，以实现所需的准确性。然而，对于在受助者的一生中面临越来越多的任务、上下文和环境的视觉辅助系统来说，这样的方法是不可扩展的。相比之下，这个项目中设想的系统将有效地利用人类智能中体现的关键原则，例如来自不断进化的记忆的反馈，以及注意力、知识模型和决策等更高的认知过程。通过受神经科学原则启发的协调硬件-软件联合设计，该项目设想了一种新的高能效视觉分析范式，可以在便携式外形因素中为视觉障碍患者提供与人类助手相当的各种任务协助。设想的人工智能认知助手将理解整个视野，推理对象之间的关系，并推断它们如何与人的目标相关(例如，导航到特定位置，完成特定任务，或总结周围发生的事情)。这些进展将是与目前辅助系统的重大飞跃，目前辅助系统的辅助能力基本，主要限于受控环境中的对象检测和导航，不支持个性化适应或持续学习。生物大脑在进化和环境压力下生存，负担不起对环境、任务、目标和背景的每一种新组合进行长期再培训的奢侈。取而代之的是，注意力等机制已经演变为基于这些组合因素动态地对信息进行优先排序。该项目设想通过嵌入自上而下的注意机制来提高传统人工智能系统的效率和准确性，该机制选择性地处理输入的最相关的空间和时间区域。通过利用知识的分层结构，支持在不同尺度上进行选择性模型交错的连续终身学习，设想的系统将动态调整学习和推理的努力，以实现与任务和环境复杂性相称的能量比例计算。其中一些神经科学原理将嵌入基于现场可编程门阵列(现场可编程门阵列)的硬件结构和定制的基于CMOS-铁电混合的硬件结构中，以通过硬件和算法的联合设计来突出效率优势。该项目将评估我们在工程系统中的创新，该系统作为视觉障碍者的认知视觉助手。这项工作将使新一代认知视觉系统能够抽象、学习、适应和推理，类似于有视力的人类助手，改变目前可用的视觉辅助系统的格局，与人类助手相比是初级的。虽然机器学习方法被广泛应用于许多应用中，但它们并不容易适应新的环境或任务。训练新的机器学习模型所消耗的能源对全球电力消耗提出了巨大的需求。相比之下，拟议的受神经科学启发的视觉分析系统可以利用注意力和终身学习原则，以传统系统的一小部分成本适应新的环境和任务。由此产生的高能效视觉分析将增强视力障碍者的独立性，将目前的智能辅助设备转变为虚拟伴侣，并极大地扩大辅助活动的范围。教育、外展和扩大参与活动是该项目各个组成部分不可或缺的一部分。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Intelligence is determined by how efficiently a machine learns and stores knowledge about the world, enabling it to handle unanticipated tasks and new environments, learn rapidly without supervision, explain decisions, deduce the unobserved, and anticipate the likely outcomes. A key limitation of current machine learning approaches is their specialized training for different tasks, environments and contexts to achieve desirable accuracies. However, such an approach is not scalable for a visual assistant system that faces an increasing variety of tasks, contexts and environments during the lifetime of the assisted individual. In contrast, the envisioned system in this project will effectively leverage key principles embodied in human intelligence such as feedback from the continuously evolving memory, and higher cognitive processes like attention, knowledge models, and decision making. Through coordinated hardware-software codesign inspired by neuroscience principles, the project envisions a new energy-efficient visual analytics paradigm that can provide persons with visual impairments assistance in a variety of tasks comparable to that of human assistants in a portable form-factor. The envisioned AI cognitive assistant will understand the entire visual field, reason about relationships between objects, and deduce how they relate to the person’s goals (e.g., navigating to a specific location, accomplishing a specific task, or summarizing what is going on in the surroundings). These advances will be a significant leap from current assistive systems that are rudimentary in their assistive capabilities, limited primarily to object detection and navigation in controlled settings with no support for personalized adaptation or continuous learning.Biological brains, under evolutionary and environmental pressure to survive, cannot afford the luxury of lengthy retraining for every new combination of environment, task, goal, and context. Instead, mechanisms such as attention have evolved to dynamically prioritize information based on these combined factors. This project envisions enhancing the efficiency and accuracy of traditional AI systems by embedding top-down attention mechanisms that selectively process spatial and temporal regions of the input that are most relevant. Through support for continuous lifelong learning with selective model interleaving at different scales by exploiting the hierarchical structure of knowledge, the envisioned system will dynamically adapt the effort for learning and inference to enable energy-proportional computing that is commensurate with the task and environmental complexity. Some of these neuroscience principles will be embedded in Field Programmable Gate Array (FPGA) based hardware fabrics and custom hybrid CMOS-ferroelectric-based hardware fabrics to accentuate the efficiency benefits through the co-design of hardware and algorithms. The project will evaluate our innovations in an engineered system that serves as a cognitive visual assistant for persons with visual impairments. This work will enable a new generation of cognitive vision systems that can abstract, learn, adapt and reason akin to sighted human assistants, transforming the landscape of currently available visual assist systems that are rudimentary compared to human assistants. While machine learning approaches are widely used in many applications, they are not easy to adapt to new environments or tasks. The energy consumed by training new machine learning models is placing enormous demands on global power consumption. In contrast, the proposed neuroscience-inspired visual analytics system can adapt to new environments and tasks with a fraction of the cost of traditional systems by leveraging attention and life-long learning principles. The resulting energy-efficient visual analytics will enhance the independence of persons with visual impairments, transforming current smart assistive gadgets to virtual companions, and greatly enhancing the range of activities for assistance. Educational, outreach and broadening participation initiatives are integral to the various components this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.