CSR: Medium: Distributed Inference Algorithms for Machine Learning and Optimization

CSR：中：用于机器学习和优化的分布式推理算法

• 批准号: 1409802
• 负责人: David Andersen
• 金额: $ 120万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-09-01 至 2018-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1409802&HistoricalAwards=false
• 关键词: CSR; Medium; Distributed; Inference; Algorithms

Machine learning systems are operating at increasing scale in ways that benefit nearly all areas of human activity, from improved voice recognition, search and advertising, automatic language translation, and on the horizon, to activities such as self-driving cars. It is already extremely hard to implement on large, scalable clusters of computers the "inference algorithms" that enable these systems, and future trends of computer data centers will further exacerbate this difficulty: increasingly large numbers of nodes, heterogeneous clusters that mix conventional microprocessors, graphics processors, larger numbers of small and power-efficient microprocessors, and hardware changes such as the introduction of flash-based solid-state disks. The goal of this proposal is to design, analyse, and implement novel inference algorithms that not only take advantage of these trends for high performance , but that also enable future, even-larger-scale systems to be implemented. The proposal specifically aims to achieve the following:1. Develop a broad family of novel optimization algorithms for machine learning;2. Analyse their convergence properties theoretically, as well as empirically;3. Release open-source code implementing them. The research proposed is based upon four likely shifts in the design of data centers of the future:1. Small and power efficient microprocessors with a much improved CPU power to energy consumption ratio will become common in the data centers of the future.2. Architectures mixing different types of hardware, ranging from computer graphics processors to general purpose multi-core microprocessors are becoming the norm among all major semiconductor manufacturers. These changes will propagate to the data center.3. Hard disks are increasingly being supplemented and replaced by solid state memory which requires 10,000 to 100,000 times less time to access.4. Modern network architectures that replace traditional hierarchical tree structures (with inherent bottlenecks) by more balanced layouts are being enabled by software-defined networking and specialized network chips. All four of these aspects offer considerable potential to design faster machine learning algorithms. Doing so requires tightly coupled algorithmic and systems design that successfully creates algorithms that work well on the kinds of systems that can be built, and systems to be built that provide the right support for machine learning algorithms. The software developed for this project will be distributed as open source.For further information see the project web site at: http://www.parameterserver.org

机器学习系统正在以越来越大的规模运行，其方式使人类活动的几乎所有领域都受益，从改进的语音识别，搜索和广告，自动语言翻译，以及即将到来的自动驾驶汽车等活动。在大型可扩展的计算机集群上实现支持这些系统的“推理算法”已经非常困难，计算机数据中心的未来趋势将进一步加剧这一困难：越来越多的节点，混合了传统微处理器、图形处理器的异构集群，大量的小型和节能微处理器，以及硬件变化，例如引入基于闪存的固态磁盘。该提案的目标是设计、分析和实现新的推理算法，这些算法不仅利用这些趋势来实现高性能，而且还能够实现未来更大规模的系统。 该提案具体旨在实现以下目标：1.为机器学习开发一系列新颖的优化算法;2.从理论上和实证上分析了它们的收敛性;3.发布实现它们的开源代码。 该研究基于未来数据中心设计的四个可能的转变：1.具有大幅提高的CPU功耗与能耗比的小型节能微处理器将在未来的数据中心中变得常见。2.混合不同类型硬件的架构，从计算机图形处理器到通用多核微处理器，正在成为所有主要半导体制造商的标准。这些更改将传播到数据中心。3.硬盘越来越多地被固态存储器所补充和取代，固态存储器需要的访问时间减少了10，000到100，000倍。4.软件定义网络和专用网络芯片正在支持以更平衡的布局取代传统分层树结构（具有固有瓶颈）的现代网络架构。 所有这四个方面都为设计更快的机器学习算法提供了相当大的潜力。这样做需要紧密耦合的算法和系统设计，成功地创建在可以构建的系统上运行良好的算法，以及为机器学习算法提供正确支持的待构建系统。 为该项目开发的软件将作为开放源代码发布。欲了解更多信息，请访问项目网站：http://www.parameterserver.org