III: Small: RIOT: Statistical Computing with Efficient, Transparent I/O

点击翻译按钮获取中文摘要

Recent technological advances enable collection of massive amounts ofdata in science, commerce, and society. These datasets bring uscloser than ever before to solving important problems such as decodinghuman genomes and coping with climate changes. Meanwhile, theexponential growth in data volume creates an urgent challenge. Manyexisting analysis tools assume datasets fit in memory; when applied tomassive datasets, they become unacceptably slow because of excessivedisk input/output (I/O) operations.Across application domains, much of advanced data analysis is donewith custom programming by statisticians. Progress has been hinderedby the lack of easy-to-use statistical computing environments thatsupport I/O-efficient processing of large datasets. There have beenmany approaches toward I/O-efficiency, but none has gained tractionwith statisticians because of issues ranging from efficiency tousability. Disk-based storage engines and I/O-efficient functionlibraries are only a partial solution, because many sources ofI/O-inefficiency in programs remain at a higher, inter-operationlevel. Database systems seem to be a natural solution, with efficientI/O and a declarative language (SQL) enabling high-leveloptimizations. However, much work in integrating databases andstatistical computing remains database-centric, forcing statisticiansto learn unfamiliar languages and deal with their impedance mismatchwith host languages.To make a practical impact on statistical computing, this projectpostulates that a better approach is to make it transparent to usershow I/O-efficiency is achieved. Transparency means no SQL, or any newlanguage to learn. Transparency means that existing code should runwithout modification, and automatically gain I/O-efficiency. Theproject, nicknamed RIOT, aims at extending R---a widely popularopen-source statistical computing environment---to transparentlyprovide efficient I/O. Achieving transparency is challenging; RIOTdoes so with an end-to-end solution addressing issues on all fronts:I/O-efficient algorithms, pipelined execution, deferred evaluation,I/O-cost-driven expression optimization, smart storage andmaterialization, and seamless integration with an interpreted hostlanguage.RIOT integrates research and education, and continues the tradition ofinvolving undergraduates through REU and independent studies. As adatabase researcher, the PI is committed to learning and drawing fromwork from programming languages and high-performance computing.Findings from RIOT help create synergy and seed further collaborationwith these communities. To ensure practical impact on statisticalcomputing, RIOT has enlisted collaboration from statisticians and theR core development team on developing, evaluating, and disseminatingRIOT.Further information can be found at: http://www.cs.duke.edu/dbgroup/Main/RIOT