III: Small: RIOT: Statistical Computing with Efficient, Transparent I/O

III：小型：RIOT：具有高效、透明 I/O 的统计计算

• 批准号: 0916027
• 负责人: Jun Yang
• 金额: $ 50万
• 依托单位: Duke University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-09-01 至 2014-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0916027&HistoricalAwards=false
• 关键词: III; Small; RIOT; Statistical; Computing

Recent technological advances enable collection of massive amounts ofdata in science, commerce, and society. These datasets bring uscloser than ever before to solving important problems such as decodinghuman genomes and coping with climate changes. Meanwhile, theexponential growth in data volume creates an urgent challenge. Manyexisting analysis tools assume datasets fit in memory; when applied tomassive datasets, they become unacceptably slow because of excessivedisk input/output (I/O) operations.Across application domains, much of advanced data analysis is donewith custom programming by statisticians. Progress has been hinderedby the lack of easy-to-use statistical computing environments thatsupport I/O-efficient processing of large datasets. There have beenmany approaches toward I/O-efficiency, but none has gained tractionwith statisticians because of issues ranging from efficiency tousability. Disk-based storage engines and I/O-efficient functionlibraries are only a partial solution, because many sources ofI/O-inefficiency in programs remain at a higher, inter-operationlevel. Database systems seem to be a natural solution, with efficientI/O and a declarative language (SQL) enabling high-leveloptimizations. However, much work in integrating databases andstatistical computing remains database-centric, forcing statisticiansto learn unfamiliar languages and deal with their impedance mismatchwith host languages.To make a practical impact on statistical computing, this projectpostulates that a better approach is to make it transparent to usershow I/O-efficiency is achieved. Transparency means no SQL, or any newlanguage to learn. Transparency means that existing code should runwithout modification, and automatically gain I/O-efficiency. Theproject, nicknamed RIOT, aims at extending R---a widely popularopen-source statistical computing environment---to transparentlyprovide efficient I/O. Achieving transparency is challenging; RIOTdoes so with an end-to-end solution addressing issues on all fronts:I/O-efficient algorithms, pipelined execution, deferred evaluation,I/O-cost-driven expression optimization, smart storage andmaterialization, and seamless integration with an interpreted hostlanguage.RIOT integrates research and education, and continues the tradition ofinvolving undergraduates through REU and independent studies. As adatabase researcher, the PI is committed to learning and drawing fromwork from programming languages and high-performance computing.Findings from RIOT help create synergy and seed further collaborationwith these communities. To ensure practical impact on statisticalcomputing, RIOT has enlisted collaboration from statisticians and theR core development team on developing, evaluating, and disseminatingRIOT.Further information can be found at: http://www.cs.duke.edu/dbgroup/Main/RIOT

最近的技术进步使收集科学、商业和社会中的大量数据成为可能。这些数据集使我们比以往任何时候都更接近于解决诸如破译人类基因组和应对气候变化等重要问题。与此同时，数据量的指数增长带来了一个紧迫的挑战。许多现有的分析工具都假设数据集适合内存；当应用于海量数据集时，由于过多的磁盘输入/输出(I/O)操作，它们变得不可接受地缓慢。由于缺乏易于使用的统计计算环境来支持大数据集的I/O效率处理，这一进展受到了阻碍。关于I/O效率的方法有很多种，但由于从效率到可调性等一系列问题，没有一种方法受到统计学家的重视。基于磁盘的存储引擎和I/O效率高的函数库只是部分解决方案，因为程序中I/O效率低下的许多来源仍然处于更高的互操作级别。数据库系统似乎是一种自然的解决方案，它具有高效的I/O和支持高级开发的声明性语言(SQL)。然而，在数据库和统计计算的集成方面，许多工作仍然以数据库为中心，迫使统计学家学习不熟悉的语言，并处理它们与宿主语言的阻抗不匹配。为了对统计计算产生实际影响，该项目假设更好的方法是使其对用户透明，从而实现I/O效率。透明意味着没有SQL，也没有任何需要学习的新语言。透明性意味着现有代码无需修改即可运行，并自动获得I/O效率。该项目绰号RIOT，旨在扩展R-一个广受欢迎的开源统计计算环境-以透明地提供高效的I/O。实现透明度具有挑战性；RIOT通过端到端的解决方案解决所有方面的问题：I/O高效算法、流水线执行、延迟计算、I/O成本驱动的表达式优化、智能存储和物化，以及与解释宿主语言的无缝集成。RIOT整合了研究和教育，并延续了通过REU和独立研究培养本科生的传统。作为数据库研究人员，PI致力于从编程语言和高性能计算中学习和借鉴。Riot的发现有助于创造协同效应，并促进与这些社区的进一步合作。为了确保对统计计算的实际影响，RIOT已经征集了统计学家和核心开发团队的合作，以开发、评估和传播RIOT。有关更多信息，请访问：http://www.cs.duke.edu/dbgroup/Main/RIOT