CIF: Small: An Algebraic, Convex, and Scalable Framework for Kernel Learning with Activation Functions

近年来，公众对机器学习的兴趣显著增加，机器学习在从医疗诊断到语音识别到自动驾驶再到广告的各个领域都得到了应用。然而，保持这种兴趣的能力将取决于机器学习算法是否在可靠性、可伸缩性和可解释性方面继续进步。随着更多的数据可用，自动驾驶汽车会变得更安全吗？Siri会更好地理解你吗？医生是否能够更好地了解疾病的病因和治疗方法？虽然神经网络和深度学习在最近几年得到了广泛的采用，但低于这些方法的算法在20多年里没有实质性的变化。因此，这个项目重温了机器学习算法背后的基础数学--将经典结果与流行的基于神经网络的方法相结合。这个数学框架然后被用来提出新的方法来提高机器学习的准确性，增加处理大数据集的能力，并允许机器学习算法的结果更容易地用可测量的物理量来解释。为了达到精确度、可扩展性和可解释性的目标，该项目对学习核的经典问题进行了代数重新表述。具体地说，对于任何给定的核代数，该代数中的正核及其相关的特征映射可以用正矩阵来表示--导致凸优化问题的解产生显式特征映射，该特征映射可以用可测量的物理量来解释。基于该框架，使用激活函数来定义通用的核代数，这些核代数在所有核的集合中是稠密的，其特征映射模仿定义神经网络的神经切核的特征映射，从而提高了算法的精度。然后，使用鞍点表示和原始-对偶方法将核学习问题转化为二次规划问题，从而得到更具可伸缩性的核学习算法。最后，通过求解一个相关的偏微分方程组，得到特征映射的奇异值分解。这种分解用于识别数据中的关键特征，并进一步产生随样本数量线性扩展的简化算法-意味着可扩展到具有数万个样本的数据集。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Public interest in machine learning has increased significantly in recent years, with application in a diversity of fields, from medical diagnosis to speech recognition to autonomous driving to advertising. The ability to sustain this interest, however, will depend on whether machine learning algorithms continue to advance in terms of both reliability, scalability, and interpretability. As more data becomes available, Will self-driving cars become safer? Will Siri understand you better? Will doctors be able to better understand the causes and treatments for diseases? While neural networks and deep learning have seen widespread adoption in recent years, the algorithms which underly these methods have not changed substantially in over 20 years. This project, therefore, revisits the fundamental mathematics which underly machine learning algorithms – integrating classical results with the popular neural-network based approaches. This mathematical framework is then used to propose new methods for improving the accuracy of machine learning, for increasing the ability to process large data sets, and for allowing the results of machine learning algorithms to be more readily interpreted in terms of measurable physical quantities. To achieve the goals of accuracy, scalability and interpretability, the project poses an algebraic reformulation of the classical problem of learning the kernel. Specifically, for any given kernel algebra, the positive kernels in that algebra and their associated feature maps may be represented by positive matrices – leading to a convex optimization problem whose solution yields an explicit feature map which may be interpreted in terms of measurable physical quantities. Based on this framework, activation functions are used to define kernel algebras which are universal, yet which are dense in the set of all kernels and whose feature maps mimic those of the neural tangent kernel which defines neural networks – leading to improved accuracy of the algorithms. Next, a saddle-point representation and primal-dual approach is used to convert the kernel learning problem to quadratic programming – resulting in more scalable kernel learning algorithms. Finally, a singular value decomposition of the resulting feature map is obtained by solving an associated partial differential equation. This decomposition is used to identify key features in the data and, furthermore, yields reduced algorithms which scale linearly with the number of samples – implying scalability to datasets with tens of thousands of samples.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.