III: Small: Cumulon: Easy and Efficient Statistical Big-Data Analysis in the Cloud

“大数据”的数量和多样性以爆炸性的速度增长，为改变科学和社会带来了巨大的潜力。在将数据转化为洞察力的愿望的推动下，分析变得越来越统计化，并且对分析大数据感兴趣的人比以往任何时候都多。 近年来，云计算的兴起为支持大数据分析提供了一种有希望的可能性。 然而，对于许多科学家和统计学家来说，使用云计算对大数据进行任何重要的统计分析仍然非常困难。 第一个挑战是开发-用户需要以底层的、特定于平台的方式进行编码和思考，并且在许多情况下，需要进行大量的手动调优以获得可接受的性能。 第二个挑战是部署--用户面临着一系列令人抓狂的选择，从硬件配置（例如，要请求的机器的类型和数量），软件配置（例如，每台机器的并行执行槽的数量），到执行参数和实现方案。 该项目旨在构建Cumulon，这是一种端到端解决方案，用于在云中更轻松，更高效地进行大数据统计计算。 对于开发，用户可以以声明式的方式思考和编码，而不必担心如何将数据和计算映射到特定的硬件和软件上。 对于部署，Cumulon为用户提供满足其要求的最佳“计划”，沿着完成时间和金钱成本，以帮助他们做出决定。 计划不仅对实现备选方案和执行参数的选择进行编码，还对群集资源和配置参数的选择进行编码。该项目开发了有效的成本建模和有效的优化技术，用于可能计划的巨大搜索空间。Cumulon解决了不确定性和可扩展性的挑战（不仅在功能方面，而且在优化方面）。Cumulon还具有性能跟踪存储库，该存储库收集来自过去部署的数据，并使用它们来改进成本建模和优化。Cumulon旨在为包括科学家和统计学家在内的广泛用户提供更轻松、更具成本效益的大数据统计计算。 除了利用云提供按需、按需付费的计算资源访问外，Cumulon还进一步简化了开发和部署，减少了对编程和调优支持的依赖，并加速了数据驱动的发现。 Cumulon不仅仅是一个一次性解决方案，它被设计为一个可发展的开源生态系统的基础，该生态系统可以跟上大数据分析的发展。它的性能跟踪存储库以独立的方式使社区受益。 随着定量、数据驱动方法的重要性日益增加，Cumulon可以影响许多领域。这个跨学科的PI团队-来自计算机科学，统计学等-正在将Cumulon应用于生物医学研究和计算新闻学的具体应用。通过合作，PI寻求吸引不同的人才，激励他们解决具有潜在社会影响的问题，并帮助他们为大数据的新挑战做好准备。http://db.cs.duke.edu/projects/cumulon

"Big data" have been growing in volume and diversity at an explosive rate, bringing enormous potential for transforming science and society. Driven by the desire to convert data into insights, analysis has become increasingly statistical, and there are more people than ever interested in analyzing big data. The rise of cloud computing in recent years offers a promising possibility for supporting big data analytics. However, it remains frustratingly difficult for many scientists and statisticians to use the cloud for any non-trivial statistical analysis of big data. The first challenge is development---users need to code and think in low-level, platform-specific ways, and, in many cases, resort to extensive manual tuning to achieve acceptable performance. The second challenge is deployment---users are faced with a maddening array of choices, ranging from hardware provisioning (e.g., type and number of machines to request), software configuration (e.g., number of parallel execution slots per machine), to execution parameters and implementation alternatives. This project aims to build Cumulon, an end-to-end solution for making statistical computing over big data easier and more efficient in the cloud. For development, users can think and code in a declarative fashion, without worrying about how to map data and computation onto specific hardware and software. For deployment, Cumulon presents users with best "plans" meeting their requirements, along with completion time and monetary cost to help them make decisions. A plan encodes choices of not only implementation alternatives and execution parameters, but also cluster resource and configuration parameters. This project develops effective cost modeling and efficient optimization techniques for the vast search space of possible plans. Cumulon addresses the challenges of uncertainty and extensibility (in terms of not only functionality but also optimizability). Cumulon also features a performance trace repository, which collects data from past deployments and uses them to improve cost modeling and optimization.Cumulon aims to make statistical computing over big data easier and more cost-effective for a wide range of users including scientists and statisticians. Besides leveraging the cloud to provide on-demand, pay-as-you-go access to computing resources, Cumulon further simplifies development and deployment, reduces reliance on programming and tuning support, and accelerates data-driven discoveries. More than a one-shot solution, Cumulon is designed as a basis for an evolvable, open-source ecosystem that keeps up with advances in big-data analytics. Its repository of performance traces benefits the community in independent ways. With the growing importance of quantitative, data-driven methods, Cumulon can impact many domains. The interdisciplinary team of PIs---from computer science, statistics, etc. ---is applying Cumulon to concrete applications in biomedical research and computational journalism. Through collaboration, the PIs seek to attract diverse talents, motivate them to work on problems with potential societal impacts, and help prepare them for the new challenges of big data.For further information see the web site at: http://db.cs.duke.edu/projects/cumulon