CAREER: Robust and Ultra-low-power Spatial Intelligence

基于机器学习的自主导航为相机等电子传感器主动探索其应用空间提供了独特的机会。例如，自动飞行摄像机可以定位农田中受感染的植物，以防止疾病传播。同样，在工业厂房中，自动飞行的气体传感器可以迅速识别气体泄漏。因此，大多数电子传感器，通过基于机器学习的自我导航，可以大大提高用例。然而，从实用性的角度来看，飞行器（即无人机）必须足够小，对用户（如办公室里的人）来说不显眼、不干扰。无人机的小尺寸对于在狭窄的空间中导航也是必要的。由于小型无人机只能携带很小的电池载荷，因此最小化功耗对于机器学习导航的机载处理至关重要。此外，飞行空间可以是高度动态的，例如，在室内应用中会有人员的移动和照明条件的变化。因此，无人机的导航必须对这些因素具有弹性。这项研究有望为基于机器学习的自主导航开发新的硬件实现，这种导航可以在一个小电池上维持。新硬件还将具有最小的占地面积，便于与小型无人机集成。该平台还将强大地处理现实世界的不确定性，例如室内照明条件的变化和人们的运动。研究者还将开展各种协同教育活动，如在当地高中组织机器学习研讨会，开发机器学习硬件的新课程，并通过这项研究指导本科生。研究人员将专门开发一个基于深度学习的无人机位置和方向连续跟踪平台。该平台将仅通过摄像头的视觉输入进行操作，以最大限度地减少必要的成本和硬件占用。将采用内存计算方法来最小化平台的功耗。具体而言，该研究将研究具有内存计算物理和操作约束的导航模型的协同设计，以显着提高平台的计算效率。为了提高预测的鲁棒性，低功耗芯片的深度学习框架还将增加概率推理。使用该程序，不仅可以提取预测本身，而且还可以提取预测置信度。因此，当深度学习模型的错误预测可能是由于飞行场景的巨大变化时，无人机将自我意识到。为了在不确定的情况下运行，无人机还将包含一个基于概率推理的计算框架。概率框架将考虑许多预测假设，并根据测量结果依次过滤掉不太可能的假设。与通过单次处理流程提取的基于深度学习的预测不同，通过推理进行的预测由于考虑了大量假设和测量，因此能耗更高。因此，深度学习和基于推理的框架也协同集成，以同时优化鲁棒性和能量效率。开发平台中的处理核心将为深度学习和推理模型重新配置，以最大限度地减少必要的资源。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning-based autonomous navigation presents a unique opportunity for electronic sensors such as cameras to proactively explore their application spaces. For example, an autonomous flying camera can locate infected plants in an agriculture field to prevent the disease spread. Similarly, in an industrial plant, self-flying gas sensors can swiftly identify gas leaks. Most electronic sensors, thus enlivened by machine learning-based self-navigation, can have dramatically elevated use-cases. For practicality, however, the flying vehicle (i.e., drone) must be small enough to be inconspicuous and non-intrusive to users, such as people in offices. The small size of the drones is also necessary to navigate through constricted spaces. Since a tiny drone can only carry a tiny battery's payload, minimizing power dissipation for onboard processing of machine learning-based navigation is quite critical. Furthermore, flying space can be highly dynamic, e.g., there will be a movement of people and changes in lighting conditions in indoor applications. Therefore, drone's navigation must be resilient against these factors. This research is expected to develop new hardware implementations for machine learning-based autonomous navigation that can sustain on a tiny battery. The new hardware will also have a minimal footprint for easy integration with tiny drones. The platform will also robustly handle real-world's uncertainties, such as changes in indoor lighting conditions and people's movement. The investigator will also pursue various synergistic educational activities such as organizing workshops on machine learning at local high schools, developing a new course on machine learning hardware, and mentoring undergraduate students through this research. The investigator will specifically develop a platform for deep learning-based continuous tracking of drone's position and orientation. The platform will operate on visual inputs alone from a camera to minimize the necessary cost and hardware footprint. A compute-in-memory approach will be employed to minimize the power dissipation of the platform. Specifically, the research will investigate the co-designing of navigational models with physical and operating constraints of compute-in-memory to dramatically improve the platform's computational efficiency. To improve the robustness of prediction, deep learning framework of the low-power chip will also be augmented with a probabilistic inference. Using the procedure, not only the prediction itself, but the prediction confidence will also be extracted. Thereby, a drone will be made self-aware of when the mispredictions from deep learning models are likely due to dramatic changes in the flying scene. To operate under uncertainties, the drone will also encompass a computing framework based on probabilistic reasoning. The probabilistic framework will operate by considering many predictive hypotheses and sequentially filtering out the unlikely ones based on measurements. Unlike deep learning-based predictions which are extracted through a single-shot processing flow, predictions from reasoning are more energy expensive by considering a multitude of hypotheses and measurements. Therefore, deep learning and reasoning-based frameworks are also synergistically integrated to concurrently optimize robustness and energy efficiency. Processing cores in the developed platform will be reconfigurable for both deep learning and reasoning models to minimize the necessary resources.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.