CSR: Medium: Distributed Inference Algorithms for Machine Learning and Optimization

机器学习系统正在以越来越大的规模运行，其方式几乎惠及人类活动的所有领域，从改进的语音识别、搜索和广告、自动语言翻译，到即将到来的自动驾驶汽车等活动。在大型、可扩展的计算机集群上实现实现这些系统的“推理算法”已经非常困难，而计算机数据中心的未来趋势将进一步加剧这一困难：越来越多的节点、混合了传统微处理器、图形处理器的异类集群、更多的小型和高能效微处理器，以及硬件变化，例如引入基于闪存的固态硬盘。本提案的目标是设计、分析和实现新的推理算法，这些算法不仅利用这些趋势获得高性能，而且还能够实现未来的、甚至更大规模的系统。该方案的具体目标是：1.开发一系列新的机器学习优化算法；2.从理论和实验上分析它们的收敛性质；3.发布实现它们的开源代码。提出的这项研究是基于未来数据中心设计的四个可能的转变：1.CPU功率与能耗比大幅提高的小型、高能效微处理器将在未来的数据中心变得普遍。混合不同类型硬件的架构，从计算机图形处理器到通用多核微处理器，正在成为所有主要半导体制造商的标准。这些更改将传播到数据中心。硬盘正越来越多地被固态存储器补充和取代，固态存储器需要的访问时间减少到原来的1/10到1/10。通过软件定义的网络和专门的网络芯片，软件定义的网络和专门的网络芯片将取代传统的层次化树形结构(具有固有的瓶颈)，以实现更均衡的布局。所有这四个方面都为设计更快的机器学习算法提供了相当大的潜力。要做到这一点，需要紧密耦合的算法和系统设计，成功地创建在可以构建的系统类型上运行良好的算法，以及要构建的为机器学习算法提供正确支持的系统。为该项目开发的软件将作为开源软件分发。有关详细信息，请参阅该项目的网站：http://www.parameterserver.org

Machine learning systems are operating at increasing scale in ways that benefit nearly all areas of human activity, from improved voice recognition, search and advertising, automatic language translation, and on the horizon, to activities such as self-driving cars. It is already extremely hard to implement on large, scalable clusters of computers the "inference algorithms" that enable these systems, and future trends of computer data centers will further exacerbate this difficulty: increasingly large numbers of nodes, heterogeneous clusters that mix conventional microprocessors, graphics processors, larger numbers of small and power-efficient microprocessors, and hardware changes such as the introduction of flash-based solid-state disks. The goal of this proposal is to design, analyse, and implement novel inference algorithms that not only take advantage of these trends for high performance , but that also enable future, even-larger-scale systems to be implemented. The proposal specifically aims to achieve the following:1. Develop a broad family of novel optimization algorithms for machine learning;2. Analyse their convergence properties theoretically, as well as empirically;3. Release open-source code implementing them. The research proposed is based upon four likely shifts in the design of data centers of the future:1. Small and power efficient microprocessors with a much improved CPU power to energy consumption ratio will become common in the data centers of the future.2. Architectures mixing different types of hardware, ranging from computer graphics processors to general purpose multi-core microprocessors are becoming the norm among all major semiconductor manufacturers. These changes will propagate to the data center.3. Hard disks are increasingly being supplemented and replaced by solid state memory which requires 10,000 to 100,000 times less time to access.4. Modern network architectures that replace traditional hierarchical tree structures (with inherent bottlenecks) by more balanced layouts are being enabled by software-defined networking and specialized network chips. All four of these aspects offer considerable potential to design faster machine learning algorithms. Doing so requires tightly coupled algorithmic and systems design that successfully creates algorithms that work well on the kinds of systems that can be built, and systems to be built that provide the right support for machine learning algorithms. The software developed for this project will be distributed as open source.For further information see the project web site at: http://www.parameterserver.org