COLLABORATIVE RESEARCH: SEI + II (AST): Bypass-Yield Caching for Large-Scale Scientific Database Workloads in the World-Wide Telescope

万维网望远镜(WWT)是一个虚拟天文台，它在全球范围内联合天文学和天体物理学数据库，最终目标是统一所有在线数据，并使其适用于世界各地的每个人。它允许研究人员通过一个查询访问多个数据库，从而极大地提高了执行多光谱和时态研究的能力。在目前的形式下，WWT网站和用户数量的增加不可避免地会导致网络危机。随着数据密集型科学应用规模的增加，带宽限制了共享网络的所有应用的性能。随着越来越多的科学家和教育工作者采用和依赖WWT，增加的带宽要求将降低所有应用程序的性能。联合会要把重点放在做好“网络公民”上，认真利用共享资源。为了避免网络危机，本项目将开发和发布一个基于两项关键技术的开源商品缓存设备：旁路-收益缓存和自组织数据库存储。BYPASS-YEILE缓存是一个无私的缓存框架，用于科学数据库工作负载，它平衡了联邦中的并行性和缓存的好处。它采用“网络公民身份”作为其主要目标--缓存，以最大限度地减少网络流量。数据库缓存带来了一个严重的存储管理问题，传统的管理方式不适合这个问题。动态创建和销毁缓存中的表需要以较低的空间开销进行自动化的增量存储管理。自组织数据库存储自动化了存储管理和数据库组织，将缓存变成了无需管理的设备。缓存设备是一种使能技术，使WWT能够接受大量用户，而不会影响共享网络的性能。开源软件和商用硬件使设备的采购成本更低，缓存的自组织使其无需维护。缓存设备将加强天体物理和天文学研究，使科学家能够进行实验，并以以前未曾预见的速度发现不同数据集之间的相关性。WWT和缓存网关还将把望远镜研究和教育带给以前无法获得的用户社区，特别是本科生和高中生。项目计划包括以试点计划的形式进行推广，在高中、大学、科学博物馆和图书馆安装和维护WWT网关，并帮助这些机构制定课程。

ABSTRACTThe World-Wide Telescope (WWT) is a virtual observatory that federates astronomy and astrophysics databases at a global scale, with the ultimate goal of unifying all on-line data and making it available to everyone from everywhere. It dramatically improves the ability to perform multi-spectral and temporal studies by allowing researchers to access many databases with a single query. In its current form, increasing the number of sites and users in the WWT leads inevitably to a network crisis. As data-intensive scientific applications increase in scale, bandwidth constrains the performance of all applications sharing a network. As more scientists and educators adopt and rely on the WWT, the increased bandwidth requirements will degrade the performance of all applications. Federations need to focus on being good "network citizens," using shared resources conscientiously. If not, the workloads generated by these applications will make them unwelcome on public networks.To avert the network crisis, this project will develop and release an open-source, commodity caching appliance based on two crucial technologies: bypass-yield caching and self-organizing database storage. Bypass-yield caching is an altruistic caching framework for scientific database workloads that balances parallelism in federations against the benefits of caching. It adopts "network citizenship" as its principal goal -- caching in order to minimize network traffic. Database caching introduces an acute storage management problem for which traditional administration is inappropriate. The dynamic creation and destruction of tables in a cache requires automated, incremental storage management with low space overhead. Self-organizing database storageautomates storage management and database organization, turning the cache into an administration-free appliance.Caching appliances are an enabling technology, making it possible for the WWT to accept a large number of users without impeding the performance of shared networks. Open-source software and commodity hardware make the acquisition of the appliance inexpensive and the self-organization of the cache makes it maintenance-free. The caching appliance will enhance astrophysical and astronomy research, making it possible for scientists to conduct experiments and find correlations across heterogeneous data sets at previously unforeseen rates. The WWT and the caching gateways will also bring telescope research and education to communities of users for which it was previously unavailable, particularly undergraduates and high school students. Project plans include outreach in the form of a pilot program that will install and maintain WWT gateways at high schools, colleges, and science museums and libraries, and assist those institutions in curriculum development.