EFRI BRAID: Neuroscience Inspired Visual Analytics

智能取决于机器学习和存储有关世界的知识的效率，使其能够处理意外的任务和新环境，在没有监督的情况下快速学习，解释决策，推断未观察到的信息，并预测可能的结果。当前机器学习方法的一个关键限制是它们针对不同的任务、环境和上下文进行专门的训练，以达到理想的准确性。然而，这样的方法对于在被辅助个体的寿命期间面临越来越多的各种任务、上下文和环境的视觉辅助系统是不可扩展的。 相比之下，该项目中设想的系统将有效地利用人类智能中体现的关键原则，例如来自不断发展的记忆的反馈，以及更高的认知过程，如注意力，知识模型和决策。通过受神经科学原理启发的协调的硬件-软件协同设计，该项目设想了一种新的节能视觉分析范式，可以在便携式外形中为视觉障碍者提供与人类助手相当的各种任务援助。设想中的人工智能认知助理将理解整个视野，推理对象之间的关系，并推断它们与人的目标的关系（例如，导航到特定位置、完成特定任务或概括周围发生的事情）。这些进步将是当前辅助系统的一个重大飞跃。目前的辅助系统在辅助功能方面还很初级，主要限于在受控环境中进行物体检测和导航，不支持个性化适应或持续学习。生物大脑在进化和环境的生存压力下，无法承受对环境、任务、目标和背景的每一个新组合进行长期再训练的奢侈。 相反，注意力等机制已经发展到基于这些组合因素动态地优先考虑信息。该项目设想通过嵌入自上而下的注意力机制来提高传统人工智能系统的效率和准确性，这些机制选择性地处理最相关的输入的空间和时间区域。通过利用知识的层次结构，在不同尺度上选择性地交织模型，支持持续的终身学习，设想的系统将动态地适应学习和推理的努力，以实现与任务和环境复杂性相称的能量比例计算。这些神经科学原理中的一些将嵌入基于现场可编程门阵列（FPGA）的硬件结构和定制的基于CMOS-铁电体的混合硬件结构中，以通过硬件和算法的协同设计来突出效率优势。该项目将评估我们在一个工程系统中的创新，该系统可作为视觉障碍者的认知视觉助手。这项工作将使新一代的认知视觉系统能够抽象，学习，适应和推理，类似于有视力的人类助手，改变目前可用的视觉辅助系统的景观，这些系统与人类助手相比是初级的。虽然机器学习方法在许多应用中得到了广泛的应用，但它们并不容易适应新的环境或任务。训练新的机器学习模型所消耗的能量对全球功耗提出了巨大的要求。相比之下，所提出的神经科学启发的视觉分析系统可以通过利用注意力和终身学习原则来适应新的环境和任务，而成本仅为传统系统的一小部分。由此产生的节能视觉分析将提高视力障碍者的独立性，将当前的智能辅助设备转变为虚拟伴侣，并大大增加援助活动的范围。教育，推广和扩大参与倡议是不可或缺的各个组成部分，这个项目。这个奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的智力价值和更广泛的影响审查标准的支持。

Intelligence is determined by how efficiently a machine learns and stores knowledge about the world, enabling it to handle unanticipated tasks and new environments, learn rapidly without supervision, explain decisions, deduce the unobserved, and anticipate the likely outcomes. A key limitation of current machine learning approaches is their specialized training for different tasks, environments and contexts to achieve desirable accuracies. However, such an approach is not scalable for a visual assistant system that faces an increasing variety of tasks, contexts and environments during the lifetime of the assisted individual. In contrast, the envisioned system in this project will effectively leverage key principles embodied in human intelligence such as feedback from the continuously evolving memory, and higher cognitive processes like attention, knowledge models, and decision making. Through coordinated hardware-software codesign inspired by neuroscience principles, the project envisions a new energy-efficient visual analytics paradigm that can provide persons with visual impairments assistance in a variety of tasks comparable to that of human assistants in a portable form-factor. The envisioned AI cognitive assistant will understand the entire visual field, reason about relationships between objects, and deduce how they relate to the person’s goals (e.g., navigating to a specific location, accomplishing a specific task, or summarizing what is going on in the surroundings). These advances will be a significant leap from current assistive systems that are rudimentary in their assistive capabilities, limited primarily to object detection and navigation in controlled settings with no support for personalized adaptation or continuous learning.Biological brains, under evolutionary and environmental pressure to survive, cannot afford the luxury of lengthy retraining for every new combination of environment, task, goal, and context. Instead, mechanisms such as attention have evolved to dynamically prioritize information based on these combined factors. This project envisions enhancing the efficiency and accuracy of traditional AI systems by embedding top-down attention mechanisms that selectively process spatial and temporal regions of the input that are most relevant. Through support for continuous lifelong learning with selective model interleaving at different scales by exploiting the hierarchical structure of knowledge, the envisioned system will dynamically adapt the effort for learning and inference to enable energy-proportional computing that is commensurate with the task and environmental complexity. Some of these neuroscience principles will be embedded in Field Programmable Gate Array (FPGA) based hardware fabrics and custom hybrid CMOS-ferroelectric-based hardware fabrics to accentuate the efficiency benefits through the co-design of hardware and algorithms. The project will evaluate our innovations in an engineered system that serves as a cognitive visual assistant for persons with visual impairments. This work will enable a new generation of cognitive vision systems that can abstract, learn, adapt and reason akin to sighted human assistants, transforming the landscape of currently available visual assist systems that are rudimentary compared to human assistants. While machine learning approaches are widely used in many applications, they are not easy to adapt to new environments or tasks. The energy consumed by training new machine learning models is placing enormous demands on global power consumption. In contrast, the proposed neuroscience-inspired visual analytics system can adapt to new environments and tasks with a fraction of the cost of traditional systems by leveraging attention and life-long learning principles. The resulting energy-efficient visual analytics will enhance the independence of persons with visual impairments, transforming current smart assistive gadgets to virtual companions, and greatly enhancing the range of activities for assistance. Educational, outreach and broadening participation initiatives are integral to the various components this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.