COLLABORATIVE RESEARCH: SEI + II (AST): Bypass-Yield Caching for Large-Scale Scientific Database Workloads in the World-Wide Telescope

【摘要】世界范围望远镜（WWT）是一个虚拟天文台，它联合了全球范围内的天文学和天体物理学数据库，其最终目标是统一所有在线数据，并使其可供世界各地的每个人使用。它允许研究人员通过一个查询访问多个数据库，从而极大地提高了执行多光谱和时间研究的能力。在目前的形式下，WWT中站点和用户数量的增加不可避免地会导致网络危机。随着数据密集型科学应用的规模不断扩大，带宽限制了所有共享网络的应用的性能。随着越来越多的科学家和教育工作者采用和依赖WWT，增加的带宽需求将降低所有应用程序的性能。联邦政府需要集中精力做好“网络公民”，认真使用共享资源。否则，这些应用程序产生的工作负载将使它们在公共网络上不受欢迎。为了避免网络危机，该项目将开发并发布一个基于两项关键技术的开源商品缓存设备：旁路生成缓存和自组织数据库存储。Bypass-yield缓存是一种用于科学数据库工作负载的利他式缓存框架，它平衡了联邦中的并行性和缓存的好处。它采用“网络公民”作为其主要目标——缓存以最小化网络流量。数据库缓存引入了一个严重的存储管理问题，传统的管理不适合这个问题。缓存中表的动态创建和销毁需要具有低空间开销的自动化增量存储管理。自组织数据库存储实现了存储管理和数据库组织的自动化，将缓存转变为无需管理的设备。缓存设备是一种启用技术，使WWT能够在不影响共享网络性能的情况下接受大量用户。开源软件和商用硬件使得设备的购买成本低廉，缓存的自组织使其无需维护。高速缓存设备将加强天体物理学和天文学研究，使科学家能够进行实验，并以以前无法预见的速度发现异构数据集之间的相关性。WWT和缓存网关还将把望远镜研究和教育带给以前无法获得的用户社区，特别是本科生和高中生。项目计划包括以试点计划的形式进行推广，该计划将在高中、大学、科学博物馆和图书馆安装和维护WWT网关，并协助这些机构开发课程。

ABSTRACTThe World-Wide Telescope (WWT) is a virtual observatory that federates astronomy and astrophysics databases at a global scale, with the ultimate goal of unifying all on-line data and making it available to everyone from everywhere. It dramatically improves the ability to perform multi-spectral and temporal studies by allowing researchers to access many databases with a single query. In its current form, increasing the number of sites and users in the WWT leads inevitably to a network crisis. As data-intensive scientific applications increase in scale, bandwidth constrains the performance of all applications sharing a network. As more scientists and educators adopt and rely on the WWT, the increased bandwidth requirements will degrade the performance of all applications. Federations need to focus on being good "network citizens," using shared resources conscientiously. If not, the workloads generated by these applications will make them unwelcome on public networks.To avert the network crisis, this project will develop and release an open-source, commodity caching appliance based on two crucial technologies: bypass-yield caching and self-organizing database storage. Bypass-yield caching is an altruistic caching framework for scientific database workloads that balances parallelism in federations against the benefits of caching. It adopts "network citizenship" as its principal goal -- caching in order to minimize network traffic. Database caching introduces an acute storage management problem for which traditional administration is inappropriate. The dynamic creation and destruction of tables in a cache requires automated, incremental storage management with low space overhead. Self-organizing database storageautomates storage management and database organization, turning the cache into an administration-free appliance.Caching appliances are an enabling technology, making it possible for the WWT to accept a large number of users without impeding the performance of shared networks. Open-source software and commodity hardware make the acquisition of the appliance inexpensive and the self-organization of the cache makes it maintenance-free. The caching appliance will enhance astrophysical and astronomy research, making it possible for scientists to conduct experiments and find correlations across heterogeneous data sets at previously unforeseen rates. The WWT and the caching gateways will also bring telescope research and education to communities of users for which it was previously unavailable, particularly undergraduates and high school students. Project plans include outreach in the form of a pilot program that will install and maintain WWT gateways at high schools, colleges, and science museums and libraries, and assist those institutions in curriculum development.