SHF: Small: Bitstream Processing

嵌入式计算系统在当今世界变得非常普遍，从可穿戴设备到家用智能扬声器，再到自动电器和车辆。这些系统实现的许多计算任务需要非常高性能的硬件来完成音频语音识别、视觉对象识别、路径优化和自主控制等任务。过去，这种高性能硬件依赖于部署在低功耗微控制器或数字信号处理器上的二进制定点算法。然而，传感和控制接口本身不使用二进制数字表示，而是使用比特流，它使用一段时间内的数字密度对数字输入和输出值进行编码。传统的计算基板需要转换输入和输出，以便与利用比特流的物理系统接口。该项目正在开发新颖的、受生物学启发的方法，用于直接操作以本地比特流格式表示的数据。直接在这些比特流上运行的计算硬件可以无缝集成到感知现实世界的系统中，处理传感数据，并根据处理的数据发出控制命令，以及根据环境的奖励学习动作。这些新方法的能力将通过两个实验平台进行验证：一个是运行在比特流音频数据上的极低功耗语音识别系统，另一个是学习在环境中导航的自主机载飞行器。这项研究具有广泛的行业和经济范围的影响，因为它将导致发现和实现新的、强大的、节能的方法来实现功率和能源受限的嵌入式计算系统。这项研究提倡开发新颖的、受生物学启发的方法来处理以比特流表示的数据。比特流使用一（一元）或一加零（二进制）的密度对数值进行编码，是从环境（输入）和机器人控制（输出）中感知到的数据的自然表示，并且可以使用低成本，但精确的sigma-delta调制器低成本地生成。该研究项目的初始阶段侧重于发展视觉、听觉和惯性感觉处理的理论和算法基础，包括特征提取、带通滤波、视角和坐标变换、线性优化和记忆形成，这些都是基于语音处理、计算机视觉、峰值神经网络、强化学习和信号处理领域的原理。新的感官处理能力随后被部署在两个实验平台上：首先，用于语音识别的超低功耗声学模型可以证明比特流处理在通过长短期记忆进行特征提取和序列学习方面的适用性。接下来，比特流传感处理技术直接与控制系统耦合，使无人机能够在受控的室内环境中导航，同时提高学习效率，识别和瞄准奖励来源。这两个演示平台都依赖于生物脉冲神经网络、用于算术运算的随机计算以及用于数据表示和信号处理任务的过采样sigma-delta调制理论的概念，并在能耗、计算密度和自主操作方面提供了前所未有的效率水平。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Embedded computing systems are becoming very common in today's world, ranging from wearable devices, to in-home smart speakers, to autonomous appliances and vehicles. Many of the computational tasks that these systems implement require very high performance hardware for tasks like audio voice recognition, visual object recognition, path optimization, and autonomous control. Historically, such high-performance hardware relied on binary fixed-point algorithms deployed on low-power microcontrollers or digital signal processors. However, the sensing and control interfaces themselves do not use binary number representations, but instead use bitstreams, which encode numeric input and output values using the density of ones over time. Conventional computing substrates requires conversion of both inputs and outputs to interface with physical systems that utilize bitstreams. This project is developing novel, biologically-inspired approaches for directly operating on data represented in the native bitstream format. Compute hardware that directly operates on these bitstreams can be seamlessly integrated into systems that sense the real world, process sensory data, and issue control commands based on the processed data as well as learned actions based on rewards from the environment. The capability of these new approaches will be demonstrated through two experimental platforms: a very low power speech recognition system that operates on bitstream audio data, and an autonomous airborne vehicle that learns to navigate its environment. This research has broad industry- and economy-wide impact since it will lead to the discovery and realization of novel, powerful, and energy-efficient approaches for implementing power- and energy-constrained embedded computing systems.This research advocates development of novel, biologically-inspired approaches for processing data represented as bitstreams. Bitstreams, which encode numeric values using density of ones (unary) or ones and zeroes (binary) are a natural representation for data sensed from the environment (input) as well as robotic control (output), and can be inexpensively generated using low-cost, yet accurate, sigma-delta modulators. The initial phase of this research project focuses on developing the theoretical and algorithmic underpinnings for visual, auditory, and inertial sensory processing, including feature extraction, bandpass filtering, perspective and coordinate transforms, linear optimization, and memory formation, which are grounded in principles from the speech processing, computer vision, spiking neural networks, reinforcement learning, and signal processing domains. The novel sensory processing capabilities are then deployed in two experimental platforms: first, an ultra-low power acoustic model for speech recognition that can demonstrate the suitability of bitstream processing for feature extraction and sequence learning via long short-term memory. Next, bitstream sensory processing technology is coupled directly to a control system that enables an unmanned aerial vehicle to navigate in a controlled indoor environment while learning, with increasing efficiency, to identify and target sources of rewards. Both demonstration platforms rely on concepts from biological spiking neural networks, stochastic computing for arithmetic operations, as well as oversampled sigma-delta modulation theory for data representation and signal processing tasks, and provide unprecedented levels of efficiency in terms of energy consumption, compute density, and autonomous operation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.