SHF: Small: Enabling New Machine-Learning Usage Scenarios with Software-Defined Hardware for Symbolic Regression

尽管机器学习取得了广泛的成功，但现有技术存在限制和/或不吸引人的权衡，这些限制和/或不吸引人的权衡阻碍了重要的使用场景，特别是在嵌入式和实时系统中。例如，人工神经网络为许多应用提供了足够的精度，但对于嵌入式应用来说，计算成本太高，并且可能需要大量的训练数据集，对于某些应用来说，收集这些数据是不切实际的。即使在使用云计算执行时，神经网络通常也需要图形处理单元加速，这大大增加了电力成本，而电力成本已经主导了大型数据中心和超级计算机的总拥有成本。同样，线性回归是一种广泛使用的机器学习技术，但通常需要用户指定模型或指导，这对于难以理解的现象和/或多维问题是不可能的。这个项目表明，符号回归通过提供有吸引力的帕累托最优权衡来补充现有的机器学习技术，从而在现有技术令人望而却步的情况下实现新的机器学习使用场景。这些符号回归的好处来自三个关键优势：1)自动模型发现，2)与现有技术相比，计算效率最低，能力损失最小，3)对训练集大小的敏感度较低。尽管符号回归已经被研究了几十年，但由于搜索具有大量局部最优解的无限解空间的挑战，符号回归通常仅限于玩具示例。该项目提供的解决方案通过两个主要贡献显著推进了最先进的技术：1)符号回归探索过程的1,000,000倍加速；2)只有在如此显著的加速下才可能实现的全新探索算法。为了加快符号回归探索过程，调查人员引入了软件定义的硬件，每个周期都会重新配置，以提供特定于解决方案的管道，作为现场可编程门阵列上的虚拟硬件覆盖。虽然这种加速本身大大改进了符号回归中的最先进技术，但更重要的贡献是启用了新的探索算法，如果不大幅提高性能，这些算法是不可行的。研究人员利用这一性能改进引入了一种新的混合探索算法，该算法使用不同配置的遗传编程和确定性启发式算法执行多个并发搜索，并结合两种新的预测机制来避免局部最优：子树超前预测和算子关联。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Despite the widespread success of machine learning, existing techniques have limitations and/or unattractive trade-offs that prohibit important usage scenarios, particularly in embedded and real-time systems. For example, artificial neural nets provide sufficient accuracy for many applications, but can be too computationally expensive for embedded usage and may require large training data sets that are impractical to collect for some applications. Even when executed with cloud computing, neural nets often require graphics-processing unit acceleration, which greatly increases power costs that can already dominate the total cost of ownership in large-scale data centers and supercomputers. Similarly, linear regression is a widely used machine-learning technique, but generally requires model specification or guidance by the user, which is prohibitive for difficult-to-understand phenomena and/or many-dimensional problems. This project shows that symbolic regression complements existing machine-learning techniques by providing attractive Pareto-optimal trade-offs that enable new machine-learning usage scenarios where existing technologies are prohibitive. These symbolic-regression benefits come from three key advantages: 1) automatic model discovery, 2) computational efficiency with minimal loss in capability compared to existing techniques, and 3) lower sensitivity to training set size. Despite being studied for decades, symbolic regression is generally limited to toy examples due to the challenge of searching an infinite solution space with numerous local optima. This project presents a solution that significantly advances the state-of-the-art via two primary contributions: 1) 1,000,000x acceleration of the symbolic-regression exploration process, and 2) fundamentally new exploration algorithms that are only possible with such significant acceleration. To accelerate the symbolic-regression exploration process, the investigators introduce software-defined hardware that re-configures every cycle to provide a solution-specific pipeline implemented as a virtual hardware overlay on field-programmable gate arrays. Although this acceleration by itself improves upon the state-of-the-art in symbolic regression considerably, the more important contribution is the enabling of new exploration algorithms that are not feasible without massive increases in performance. The investigators use this performance improvement to introduce a new hybrid exploration algorithm that performs multiple concurrent searches using different configurations of genetic programming and deterministic heuristics, combined with two new prediction mechanisms to avoid local optima: sub-tree look-ahead prediction and operator correlation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.