VEC: Medium: Large-Scale Visual Recognition: From Cloud Data Centers to Wearable Devices

计算机硬件和软件的进步有望彻底改变社会与视觉信息互动的方式。然而，视觉识别系统由于缺乏一种实用的方法来对视觉场景中出现的数百万个概念进行分类，从而有效地识别在给定场景中出现的少量概念而受到限制。此外，虽然视觉数据的实时处理可以显著扩展我们对周围环境的感知，但由于散热有限（例如，没有风扇或液体冷却），目前最先进的视觉系统无法在智能手机等可穿戴设备上实现。这项研究将通过开发人工智能（AI）系统来克服这些挑战，该系统可以有效地管理高性能可穿戴视觉识别最关键的资源，包括可穿戴设备的实时功耗和计算。这些系统将被赋予启动密集计算的能力，这些计算由可穿戴设备内的材料热管理，这些材料被设计成在高温下融化，在爆发之间凝固。此外，人工智能系统将管理设备与外部（基于云的）计算资源以及位于数据中心的大规模视觉概念数据库之间的通信，从而在可穿戴的外形因素中提供极致的性能。这项工作的核心概念将整合到本科和研究生课程中，并将向研究界提供一个示范系统，并用于高中学生的教育模块。这项工作旨在通过共同设计视觉模型和计算基础设施来推进大规模视觉识别的核心能力。其目标是通过在可穿戴设备和云上无缝集成视觉计算，实现百科全书式的实时视觉识别。pi设想了一种可穿戴视觉识别系统，该系统可以持续捕获实时视频输入，同时通过设备计算和云卸载相结合，通过自动或按需视觉识别提供智能、实时的辅助。由于一些基本的挑战，这样一个系统目前是不可行的。首先，可穿戴设备严重的能量和热限制使它们无法执行视觉识别所需的密集计算。其次，如何在可视化模型和数据中心基础设施方面支持百科全书式识别仍然是一个悬而未决的问题。特别是，目前的视觉模型虽然在识别1000个对象类别方面非常成功，但如何扩展到数百万或更多不同的视觉概念，目前还不清楚。此外，这种百科全书式的可视化模型必须通过数据中心基础设施来支持，但是在如何构建这种基础设施方面几乎没有取得进展。该项目通过跨学科的方法，将计算机视觉、硬件架构、超大规模集成电路设计和传热集成在一起，解决了这些基本挑战。pi将调查三个研究重点。在Thrust 1中，pi将开发一种新型的深度神经网络，允许资源高效地执行模块。这个新框架提供了一种统一的方式来设计、学习和运行可扩展的视觉模型，可以最大限度地利用受资源限制的识别，如延迟、能量或可穿戴设备的散热。在推力2中，pi将利用推力1中开发的新框架，设计和制造具有计算冲刺能力的视觉处理芯片（极限计算的爆发远远超过稳态散热能力）。在Thrust 3中，pi将设计数据中心基础设施，支持大规模的视觉概念分层索引，用于百科全书式识别，重点关注延迟、吞吐量和能源效率。最后，pi将建立一个演示系统来评估提议的算法、软件和硬件组件，并评估端到端系统的整体性能。该项目的网站（http://mivec.eecs.umich.edu/）将提供对这项研究结果的访问，包括技术报告、数据集和源代码。

Advances in computer hardware and software promise to revolutionize the ways in which society interacts with visual information. However, visual recognition systems are limited by the lack of a practical means to classify the millions of concepts that arise in visual scenes and thus efficiently recognize when a small number of these concepts appear in a given scene. Furthermore, while real-time processing of visual data could significantly expand our perception of our surroundings, state-of-the-art vision systems cannot currently be implemented on wearable devices such as smartphones due to the limited heat dissipation (e.g., no fans or liquid cooling) and power such devices can provide. This research will overcome these challenges by developing artificial intelligence (AI) systems that efficiently manage the resources most crucial for high-performance wearable-based visual recognition, including the wearable device's real-time power consumption and computation. These systems will be empowered to initiate bursts of intense computation that are thermally managed by materials within the wearable device which are engineered to melt during heavy heating and solidify between bursts. Moreover, the AI systems will govern the communication between the device and external (cloud-based) computation resources as well as large-scale visual concept databases housed in data centers, thus providing extreme performance in a wearable form factor. Central concepts of this work will be integrated in undergraduate and graduate coursework, and a demonstration system will be made available to the research community and used in educational modules for high school students.This effort seeks to advance the core capabilities of large-scale visual recognition by co-designing visual models and computing infrastructure. The goal is to enable encyclopedic, real-time visual recognition through seamless integration of visual computing on wearable devices and in the cloud. The PIs envision a wearable visual recognition system that continuously captures live video input while providing intelligent, real-time assistance through automatic or on-demand visual recognition by means of a combination of computation at the device and offloading to the cloud. Such a system is not currently feasible due to a number of fundamental challenges. First, the severe energy and thermal constraints of wearable devices render them incapable of performing the intensive computation necessary for visual recognition. Second, it remains an open question how to support encyclopedic recognition in terms of both visual models and data center infrastructure. In particular, it remains unclear how current visual models, although highly successful at recognizing 1,000 object categories, can scale to millions or more distinct visual concepts. Moreover, such an encyclopedic visual model must be supported through data center infrastructure, but little progress has been made on how to build such infrastructure. This project addresses these fundamental challenges through an interdisciplinary approach integrating computer vision, hardware architecture, VLSI design, and heat transfer. The PIs will investigate three research thrusts. In Thrust 1, the PIs will develop a new type of deep neural networks that allow resource-efficient execution of modules. This new framework provide a unified way to design, learn, and run scalable visual models that can maximize the utility of recognition subject to resource constraints, such as latency, energy, or thermal dissipation of a wearable device. In Thrust 2, the PIs will design and fabricate a visual processing chip capable of computational sprinting (bursts of extreme computation well above steady-state thermal dissipation capabilities), leveraging the new framework developed in Thrust 1. In Thrust 3, the PIs will design datacenter infrastructure that supports large-scale hierarchical indexing of visual concepts for encyclopedic recognition, with a focus on latency, throughput, and energy efficiency. Finally, the PIs will build a demonstration system to evaluate the proposed algorithms, software, and hardware components and to assess the overall performance of an end-to-end system. The project web site (http://mivec.eecs.umich.edu/) will provide access to the results of this research including technical reports, datasets, and source code.