AF: Large: Collaborative Research: Nonconvex Methods and Models for Learning: Towards Algorithms with Provable and Interpretable Guarantees

人工智能沿着机器学习也许是我们这个时代最主要的研究主题-对社会和我们目前的生活方式具有深远的影响。虽然可能性有很多，但也有人怀疑这些方法将走多远-以及可能需要什么新的理论基础才能将它们带到下一个水平，克服可能的障碍。最近，机器学习经历了范式转变，越来越依赖随机优化来训练高度非凸模型，包括但不限于深度网络。理论理解已经落后，主要是因为大多数问题在最坏的情况下都是难以解决的。此外，传统的机器学习理论主要关注分类，而许多实际成功是由无监督学习和表示学习驱动的。大多数过去的表示学习理论都集中在简单的模型上，如k均值聚类和PCA，而实际工作使用更复杂的模型，如自动编码器，受限玻尔兹曼机和深度生成模型。该提案提出了一个雄心勃勃的议程，旨在扩展理论，以拥抱和支持这些实践趋势，并希望影响实践。为下一代机器学习方法和优化算法提供理论基础。该项目最终可能会对实际的机器学习产生重大影响，甚至会引起该领域的文化变革-理论和实践-并产生长期影响。鉴于机器学习的普遍性以及经济和科学影响，这种影响将扩展到其他学科，特别是与神经科学研究人员的（持续）合作。该项目将通过积极的教学和指导计划，培养新一代的机器学习研究人员，从本科到博士后。新一代将轻松结合前沿理论和应用。今天迫切需要这样的人，高级PI在培训/指导几个现有的PI方面发挥了作用。技术贡献将包括知识表示和语义的新理论模型，以及证明非凸优化例程收敛的框架。理论将被开发来解释和利用表征学习和监督学习之间的相互作用，这种相互作用在深度学习中已被证明是如此成功，并且似乎是领域适应，迁移学习和交互式学习等新学习范式的基础。将尝试用具有更多“可解释”属性和性能曲线的模型来取代神经模型。 所有的PI都有将理论与实践相结合的记录。他们还致力于一种非正统的研究方法，借鉴了机器学习的所有过去阶段：早期阶段的可解释表示（依赖于逻辑表示或概率模型），中间阶段的可证明保证（凸优化，内核等），以及对最新深度网络阶段的非凸方法的拥抱。这种折衷主义在机器学习中并不常见，可能会产生新的范式和新的科学类型。

Artificial Intelligence along with Machine Learning are perhaps the most dominant research themes of our times - with far reaching implications for society and our current life style. While the possibilities are many, there are also doubts about how far these methods will go - and what new theoretical foundations may be required to take them to the next level overcoming possible hurdles. Recently, machine learning has undergone a paradigm shift with increasing reliance on stochastic optimization to train highly non-convex models -- including but not limited to deep nets. Theoretical understanding has lagged behind, primarily because most problems in question are provably intractable on worst-case instances. Furthermore, traditional machine learning theory is mostly concerned with classification, whereas much practical success is driven by unsupervised learning and representation learning. Most past theory of representation learning was focused on simple models such as k-means clustering and PCA, whereas practical work uses vastly more complicated models like autoencoders, restricted Boltzmann machines and deep generative models. The proposal presents an ambitious agenda for extending theory to embrace and support these practical trends, with hope of influencing practice. Theoretical foundations will be provided for the next generation of machine learning methods and optimization algorithms. The project may end up having significant impact on practical machine learning, and even cause a cultural change in the field -- theory as well as practice -- with long-term ramifications. Given the ubiquity as well as economic and scientific implications of machine learning today, such impact will extend into other disciplines, especially in (ongoing) collaborations with researchers in neuroscience. The project will train a new generation of machine learning researchers, through an active teaching and mentoring plan at all levels, from undergrad to postdoc. This new generation will be at ease combining cutting edge theory and applications. There is a pressing need for such people today, and the senior PIs played a role in training/mentoring several existing ones. Technical contributions will include new theoretical models of knowledge representation and semantics, and also frameworks for proving convergence of nonconvex optimization routines. Theory will be developed to explain and exploit the interplay between representation learning and supervised learning that has proved so empirically successful in deep learning, and seems to underlie new learning paradigms such as domain adaptation, transfer learning, and interactive learning. Attempts will be made to replace neural models with models with more "interpretable" attributes and performance curves. All PIs have a track record of combining theory with practice. They are also devoted to a heterodox research approach, borrowing from all the past phases of machine learning: interpretable representations from the earlier phases (which relied on logical representations, or probabilistic models), provable guarantees from the middle phase (convex optimization, kernels etc.), and an embrace of nonconvex methods from the latest deep net phase. Such eclecticism is uncommon in machine learning, and may give rise to new paradigms and new kinds of science.