ITR/IM+AP: A Conceptual Study of Multi-Nested Self-Populating Scientific Data Centers

这是一项信息技术研究提案，旨在开发一种新的“数据爬行”软件框架概念，允许自由职业者在网络上可用的各种科学数据集之间“航行”。现有的世界数据中心(WDC)系统为全世界科学界提供免费获取一些全球地球物理参数的服务。然而，这一系统仍然要求(强制或自愿)将当地收集的数据提交给数据中心，这些数据中心设在不同地理位置的某些机构或组织，主要按相应的科学学科组织。与此同时，大量的科学数字数据库目前放在万维网上，无需提交给任何数据中心，实现了世界数据中心系统免费获取数据的目标，但(最重要的)促进了对当地机构努力的认可。这就需要复杂的方法来搜索和识别数据类型，然后从这些数据库中检索一定数量的数据。将开发的“数据爬虫”将检查数据类型，并在数据主机和发出请求的“数据中心”之间建立相互同意的互连，然后将检索到的数据样本(子集)提供给这个单嵌套或多嵌套、自我填充的数据中心，从而检查它们的可访问性。该研究项目将产生一个“数据爬虫”搜索引擎的工作原型，并将其实际应用于基于Web的科学数据集。这将使用户能够提高他们在环境信息学和地球科学建模和表示方面的能力，并通过无处不在的内容基础设施的支持促进科学研究。后者将帮助组织一个“虚拟全球地磁观测站”，通过来自多个嵌套数据源点的馈送，可以在全球范围内显示世界范围内的地磁数据。这项工作将集中在日地物理和地球物理的学科上，但拟议的软件框架可以应用于任何科学(甚至商业)数据库，这些数据库在网络上没有预先设定的结构。

This is an Information Technology Research proposal to develop a new concept of "data-crawling" software framework allowing a freelance "sailing" across various scientific datasets available on the Web. The existing World Data Center (WDC) System serves the worldwide scientific community in providing free access to a number of global geophysical parameters. However, this system still requires the (mandatory or voluntary) submission of locally collected data to the data centers, housed at certain institutions or organizations at different geographic locations and mainly organized by the corresponding scientific discipline. At the same time, a huge number of scientific digital databases are currently placed on the World Wide Web without submission to any data center, fulfilling the WDC System's goal of free access to the data, but (most importantly) promoting recognition of the local institution efforts. This creates a need for sophisticated methods to search and to recognize the data type, and then to retrieve a certain amount of data from these databases. The "data-crawler" to be developed will check the data types and levels their accessibility in establishing mutually agreed interconnections between a data host and a requesting "data center" and then supplying retrieved data samples (subsets) to this single- or multi-nested, self-populating data center. The research project will yield a working prototype of the "data-crawler" search engine with practical applications to the Web-based scientific datasets. This will allow users to improve their capabilities in environmental informatics and geoscience modeling and representation, as well as facilitate scientific research via the support of ubiquitous content infrastructure. The latter will help in organizing a "Virtual Global Magnetic Observatory" (VGMO) where the worldwide geomagnetic data can be visualized on a global scale through the feed from multi-nested data source sites. The effort will focus on the disciplines of solar-terrestrial physics and geophysics, but the proposed software framework can be applied to any scientific (or even commercial) databases distributed without a pre-set structure over the Web.