课题基金 / 基金详情

CAREER: Towards a Big Data Application Server Stack

CAREER: Towards a Big Data Application Server Stack
职业:迈向大数据应用服务器堆栈
批准号:
1351047
负责人:
Tyson Condie
金额:
$46.47万
依托单位国家:
美国
项目类别:
Continuing Grant
财政年份:
2014
资助国家:
美国
项目状态:
已结题
起止时间:
2014-02-01 至 2019-01-31

项目摘要

项目成果

Tyson Condie的其他基金

相似基金

相关文献

中文摘要
翻译
Google的MapReduce启发了大数据分析工作,并作为Apache Hadoop等开源系统的模板。MapReduce编程模型具有广泛的适用性,但广泛的采用暴露了一些局限性,例如缺乏对迭代(这在机器学习算法中很常见),流处理,图形分析,实时和交互式查询的支持。在编程框架之外,底层实现提供了一个如何横向扩展大规模分布式计算的模板:通过划分底层数据将它们分解为可以并行执行的小任务,并保存中间状态以减轻部分故障的影响(在大型集群上运行时必须计划)。因此,挑战在于构建其他编程框架的实现(例如,SQL和机器学习),它们与MapReduce共享相同的横向扩展和容错运行时特性,而不会强加其限制。Apache Hadoop YARN、Google Omega和Berkeley Mesos等资源管理器朝着这个方向迈出了第一步,将资源分配与高级编程模型和语言的细节分离开来。资源管理器在同一个底层机器集群上多路复用多个作业,从而提高利用率并促进干净的软件堆栈。当在容器中执行的单个机器资源(CPU/GPU,内存,磁盘)切片的任务完成时,容器将返回到资源管理器,在那里它可用于其他作业。与更高级别的堆栈不同,容器是一个白板进程,旨在托管任意计算。这个项目规定了进一步的可重用软件层,以捕获诸如我应该为一项工作分配多少资源之类的问题。什么是冗余的代码路径,我能在可重用的库中提供它们吗?什么是正确的语言和运行时抽象?在MapReduce和相关的SQL实现、ML工具包、存储系统和消息传递系统等系统的上下文中探索这些问题,是我们工作的主要焦点。我们的目标是将一套大规模数据处理任务统一在一个基于现代资源管理器(云操作系统)的单一运行时层上。我们的研究结果将因素的共性,在专门的系统,并提供他们在一个单一的底层运行时系统,缩短时间?市场?下一个即用型大数据工具包,这反过来又会增加这些工具对更广泛社区的可用性。通过在下一代资源管理器上大规模实施和部署应用程序所获得的经验可以帮助为未来云计算平台的开发提供关键设计选择,从而影响广泛的科学,工程,国家安全,医疗保健和商业应用。该项目为研究生和本科生提供了更好的研究型高级培训机会,包括目前在计算机科学,数据库,机器学习和云计算方面代表性不足的团体成员。
英文摘要
Google's MapReduce inspired much of the Big Data Analytics work and has served as a template for open source systems like Apache Hadoop. The MapReduce programming model has wide applicability, but widespread adoption has exposed some limitations, such as the lack of support for iteration (which is common in machine learning algorithms), stream processing, graph analytics, real-time and interactive queries. Beyond the programming framework, the underlying implementation offers a template for how to scale-out massively distributed computations: break them up into small tasks that can be carried out in parallel by partitioning the underlying data, and save intermediate state to mitigate the impact of partial failures (which must be planned for when running on large clusters). The challenge then, is to build implementations of other programming frameworks (e.g., SQL and machine learning) that share the same scale-out and fault-tolerance runtime characteristics of MapReduce without imposing its limitations. Resource managers such as Apache Hadoop YARN, Google Omega and Berkeley Mesos take a first step in this direction by separating resource allocation from the details of higher-level programming models and languages. Resource managers multiplex several jobs on the same underlying machine cluster, thereby increasing utilization and fostering clean-slate software stacks. When the task executing in a container a slice of a single machine's resources (CPU/GPU, memory, disk) is finished, the container is returned to the resource manager, where it is made available to other jobs. Unlike in higher-level stacks, a container is a blank-slate process, designed to host arbitrary computations. This project prescribes further reusable software layers that capture issues like how many resources should I dedicate to a job?; what are the redundant code-pathways and can I provide them in a reusable library?; what are the right language and runtime abstractions? Exploring these questions in the context of systems like MapReduce and related SQL implementations, ML toolkits, storage systems, and messaging systems, on next generation resource managers, is the primary focus of our work.The goal is to unify a suite of large-scale data processing tasks on a single runtime layer, built on modern resource managers (the cloud operating systems). Our results will factor out commonalities in specialized systems and provide them in a single underlying runtime system, shortening the time to ?market? for the next ready-to-use Big Data toolkit, which in turn would increase the availability of such tools to the broader community. Experience gained by implementing and deploying applications at scale, over next generation resource managers, could help inform critical design choices in the development of future cloud computing platforms, and hence impact a broad range of scientific, engineering, national security, healthcare and business applications. The project offers enhanced opportunities for research-based advanced training of graduate and undergraduate students, including members of groups that are currently under-represented in computer science, in databases, machine learning, and cloud computing.
期刊论文(0)
专著(0)
科研奖励(0)
会议论文
Student Travel Fellowships for ACM Symposium on Cloud Computing 2017
III: Medium: Collaborative Research: Scaling Machine Learning to Massive Datasets---A Logic Based Approach
  • 批准号:
    1302698
  • 项目类别:
    Continuing Grant
  • 资助金额:
    $66.7万
  • 财政年份:
    2013
  • 负责人:
    Tyson Condie
  • 依托单位:
海外基金