III: Small: RIOT: Statistical Computing with Efficient, Transparent I/O

最近的技术进步使得在科学、商业和社会中收集大量数据成为可能。这些数据集使我们比以往任何时候都更接近于解决诸如解码人类基因组和应对气候变化等重要问题。与此同时，数据量的指数级增长带来了一个紧迫的挑战。许多现有的分析工具假设数据集适合内存；当应用于大量数据集时，由于过多的磁盘输入/输出（I/O）操作，它们变得不可接受地慢。跨应用领域，许多高级数据分析是由统计人员使用自定义编程完成的。由于缺乏易于使用的统计计算环境来支持大型数据集的I/ o高效处理，因此阻碍了进展。有许多提高I/ o效率的方法，但由于效率和可操作性等问题，没有一种方法得到统计学家的关注。基于磁盘的存储引擎和I/ o效率函数库只是部分解决方案，因为程序中许多I/ o效率低下的来源仍然处于更高的互操作级别。数据库系统似乎是一个自然的解决方案，它具有高效的i /O和声明性语言（SQL），可以实现高级优化。然而，整合数据库和统计计算的许多工作仍然以数据库为中心，这迫使统计学家学习不熟悉的语言，并处理它们与宿主语言的阻抗不匹配。为了对统计计算产生实际影响，本项目假设更好的方法是使其对用户透明，从而实现I/ o效率。透明意味着不需要学习SQL或任何新语言。透明意味着现有代码不需要修改就可以运行，并自动获得I/ o效率。这个昵称为RIOT的项目旨在扩展R——一个广泛流行的开源统计计算环境——以透明地提供高效的I/O。实现透明度具有挑战性；riot通过端到端解决方案解决了所有方面的问题：I/ o高效算法、流水线执行、延迟求值、I/ o成本驱动的表达式优化、智能存储和实现，以及与解释性主机语言的无缝集成。RIOT将研究与教育相结合，并继续通过REU和独立学习让本科生参与其中的传统。作为数据库研究人员，PI致力于从编程语言和高性能计算中学习和借鉴工作。RIOT的研究结果有助于与这些社区建立协同作用，并为进一步的合作播下种子。为了确保对统计计算的实际影响，RIOT已经获得了统计学家和核心开发团队的合作，以开发，评估和传播RIOT。更多信息可在http://www.cs.duke.edu/dbgroup/Main/RIOT找到

Recent technological advances enable collection of massive amounts ofdata in science, commerce, and society. These datasets bring uscloser than ever before to solving important problems such as decodinghuman genomes and coping with climate changes. Meanwhile, theexponential growth in data volume creates an urgent challenge. Manyexisting analysis tools assume datasets fit in memory; when applied tomassive datasets, they become unacceptably slow because of excessivedisk input/output (I/O) operations.Across application domains, much of advanced data analysis is donewith custom programming by statisticians. Progress has been hinderedby the lack of easy-to-use statistical computing environments thatsupport I/O-efficient processing of large datasets. There have beenmany approaches toward I/O-efficiency, but none has gained tractionwith statisticians because of issues ranging from efficiency tousability. Disk-based storage engines and I/O-efficient functionlibraries are only a partial solution, because many sources ofI/O-inefficiency in programs remain at a higher, inter-operationlevel. Database systems seem to be a natural solution, with efficientI/O and a declarative language (SQL) enabling high-leveloptimizations. However, much work in integrating databases andstatistical computing remains database-centric, forcing statisticiansto learn unfamiliar languages and deal with their impedance mismatchwith host languages.To make a practical impact on statistical computing, this projectpostulates that a better approach is to make it transparent to usershow I/O-efficiency is achieved. Transparency means no SQL, or any newlanguage to learn. Transparency means that existing code should runwithout modification, and automatically gain I/O-efficiency. Theproject, nicknamed RIOT, aims at extending R---a widely popularopen-source statistical computing environment---to transparentlyprovide efficient I/O. Achieving transparency is challenging; RIOTdoes so with an end-to-end solution addressing issues on all fronts:I/O-efficient algorithms, pipelined execution, deferred evaluation,I/O-cost-driven expression optimization, smart storage andmaterialization, and seamless integration with an interpreted hostlanguage.RIOT integrates research and education, and continues the tradition ofinvolving undergraduates through REU and independent studies. As adatabase researcher, the PI is committed to learning and drawing fromwork from programming languages and high-performance computing.Findings from RIOT help create synergy and seed further collaborationwith these communities. To ensure practical impact on statisticalcomputing, RIOT has enlisted collaboration from statisticians and theR core development team on developing, evaluating, and disseminatingRIOT.Further information can be found at: http://www.cs.duke.edu/dbgroup/Main/RIOT