COLLABORATIVE RESEARCH: SEI + II (AST): Bypass-Yield Caching for Large-Scale Scientific Database Workloads in the World-Wide Telescope

万维天文望远镜（WWT）是一个虚拟天文台，在全球范围内联合天文学和天体物理学数据库，其最终目标是统一所有在线数据，并使其可供任何人从任何地方。 它通过允许研究人员使用单个查询访问许多数据库，极大地提高了执行多光谱和时间研究的能力。 以目前的形式，WWT中站点和用户数量的增加不可避免地会导致网络危机。 随着数据密集型科学应用规模的增加，带宽限制了共享网络的所有应用的性能。 随着越来越多的科学家和教育工作者采用和依赖WWT，增加的带宽需求将降低所有应用程序的性能。 联盟需要专注于成为良好的“网络公民”，认真使用共享资源。 否则，这些应用程序产生的工作负载将使它们在公共网络上不受欢迎。为了避免网络危机，该项目将开发并发布一个开源的商品缓存设备，该设备基于两项关键技术：旁路收益缓存和自组织数据库存储。 缓存收益率缓存是一种用于科学数据库工作负载的利他主义缓存框架，它平衡了联邦中的并行性和缓存的好处。 它采用“网络公民”作为其主要目标-缓存，以最大限度地减少网络流量。 数据库缓存引入了一个严重的存储管理问题，传统的管理是不合适的。 在缓存中动态创建和销毁表需要自动化的增量存储管理，并且空间开销很低。 自组织数据库存储自动化了存储管理和数据库组织，将该高速缓存转变为无需管理的设备。缓存设备是一种支持技术，使WWT能够接受大量用户，而不会影响共享网络的性能。 开源软件和商用硬件使设备的购买成本低廉，而该高速缓存的自组织性使其免维护。 缓存设备将增强天体物理学和天文学研究，使科学家能够以以前无法预见的速度进行实验并发现异构数据集之间的相关性。WWT和缓存网关还将为用户社区带来望远镜研究和教育，这在以前是不可用的，特别是本科生和高中生。 项目计划包括以试点方案的形式进行推广，该方案将在高中、大学、科学博物馆和图书馆安装和维护WWT网关，并协助这些机构制定课程。

ABSTRACTThe World-Wide Telescope (WWT) is a virtual observatory that federates astronomy and astrophysics databases at a global scale, with the ultimate goal of unifying all on-line data and making it available to everyone from everywhere. It dramatically improves the ability to perform multi-spectral and temporal studies by allowing researchers to access many databases with a single query. In its current form, increasing the number of sites and users in the WWT leads inevitably to a network crisis. As data-intensive scientific applications increase in scale, bandwidth constrains the performance of all applications sharing a network. As more scientists and educators adopt and rely on the WWT, the increased bandwidth requirements will degrade the performance of all applications. Federations need to focus on being good "network citizens," using shared resources conscientiously. If not, the workloads generated by these applications will make them unwelcome on public networks.To avert the network crisis, this project will develop and release an open-source, commodity caching appliance based on two crucial technologies: bypass-yield caching and self-organizing database storage. Bypass-yield caching is an altruistic caching framework for scientific database workloads that balances parallelism in federations against the benefits of caching. It adopts "network citizenship" as its principal goal -- caching in order to minimize network traffic. Database caching introduces an acute storage management problem for which traditional administration is inappropriate. The dynamic creation and destruction of tables in a cache requires automated, incremental storage management with low space overhead. Self-organizing database storageautomates storage management and database organization, turning the cache into an administration-free appliance.Caching appliances are an enabling technology, making it possible for the WWT to accept a large number of users without impeding the performance of shared networks. Open-source software and commodity hardware make the acquisition of the appliance inexpensive and the self-organization of the cache makes it maintenance-free. The caching appliance will enhance astrophysical and astronomy research, making it possible for scientists to conduct experiments and find correlations across heterogeneous data sets at previously unforeseen rates. The WWT and the caching gateways will also bring telescope research and education to communities of users for which it was previously unavailable, particularly undergraduates and high school students. Project plans include outreach in the form of a pilot program that will install and maintain WWT gateways at high schools, colleges, and science museums and libraries, and assist those institutions in curriculum development.