CRII: AF: Algorithms for Noise-Tolerant Function Testing with Applications to Deep Learning

机器学习已经成为计算机科学的一个重要领域，它有可能显著改变我们的生活和社会。在深度学习中，人们需要依赖于能够快速测试目标函数的性质。该项目的目标是开发用于测试高维函数解析性质的算法。更好地了解深度学习中使用的优化目标的属性将使该领域的研究人员能够在优化方法的选择方面做出更有根据的决策。它将简化并引入严格的参数调整艺术，在训练深度神经网络中发挥高性能的关键作用。由PI开发的近似算法泛函分析（Lp-testing）框架构成了本研究的起点，已在宾夕法尼亚大学和布宜诺斯艾利斯大学的大数据学习理论和算法课程中教授。与本提案的研究成果一起纳入m.s /Ph.D。由PI教授的印第安纳大学数据科学基础和大数据算法课程。PI将开发超高效的算法，以帮助人类理解深度学习中使用的高维函数和目标的分析特性。设计这些工具的三个主要目标和相关挑战是：(1)在缺乏明确的全局结构的情况下对深度学习目标的局部属性进行算法分析(2)基于噪声数据对函数的分析属性进行严格分析(3)在深度学习应用中由于性能原因导致的函数评估中引入对抽样误差的容错性。该项目将涉及开发新的数学方法，以理解噪声函数的全局性质，如单调性、凸性和lipschitz性如何受到随机低维线性子空间投影的影响。它将为生成这样的子空间提供选择分布的建议，以便最好地保留所需的属性。对数据依赖方法的基本优势的严格研究将作为项目的一个单独部分进行。

Machine learning has emerged as an important area of computer science, which has a potential significantly to change our lives and society. In deep learning, one needs to rely on being able to quickly test the properties of objective functions. The goal of this project is to develop algorithms for testing analytic properties of high-dimensional functions. Better understanding of properties of optimization objectives used in deep learning will enable researchers in the field to make more educated decisions regarding the choice of optimization methods. It will simplify and introduce rigor in the art of parameter tuning that plays key role in achieving high performance in training deep neural nets. The framework for approximate algorithmic functional analysis (Lp-testing) developed by the PI that forms the starting point for this research has been taught in courses on learning theory and algorithms for big data at the University of Pennsylvania and University of Buenos Aires. Together with the outcomes of the research in this proposal it will be included into M.S./Ph.D. classes on foundations of data science and algorithms for big data at Indiana University taught by the PI.The PI will develop ultra-efficient algorithms for assisting humans in their understanding of analytic properties of high-dimensional functions and objectives used in deep learning. Three main goals and related challenges in the design of such tools are:(1) Performing algorithmic analysis of local properties of deep learning objectives in the absence of clear global structure (2) Enabling rigorous analysis of analytic properties of functions based on noisy data (3) Introducing tolerance to sampling errors in function evaluations arising in deep learning applications for performance reasons. The project will involve development of new mathematical methods for understanding how global properties of noisy functions such as monotonicity, convexity and Lipschitzness are affected by projections onto random low-dimensional linear subspaces. It will suggest choices of distributions for generation of such subspaces in order to best preserve the desired properties. A rigorous study of fundamental advantages of data-dependent methods will be conducted as a separate part of the project.