ITR/IM: Methodologies Managing Large Geophysical Datasets

地球物理数据集通常非常大。例如，一台可用的气象雷达每年可以产生近0.9 tb的数据。开发大陆规模的多年数据集可能成为一项重大且可能非常昂贵的工作。地球物理卫星数据也是如此。信息技术的进步，特别是磁盘存储容量的迅速增加，减轻了一些数据访问问题，但也加剧了其他问题。在线获取tb的地球物理数据是可能的，但以其原生/分布格式对数据进行可视化和分析仍然非常困难。数据压缩或使用衍生产品等方法有时有效，但过于简单，最终会适得其反。这个问题显然是多方面的，但这个项目的重点是研究和开发在分发给用户后处理数据的方法。具体来说，首席研究员将开发一个客户端-服务器架构，向用户隐藏数据集管理的复杂性。主要组件是用于操作数据集的数据查询语言或DQL。通过DQL，用户可以轻松有效地添加、删除或修改数据集的子集，并快速搜索符合特定条件的子集。DQL客户机-服务器包括资源管理，因此用户可以控制分析对客户机CPU、网络利用率和其他计算资源的影响。该项目的一部分是完成一套多年、多站点的气象雷达数据集。这将作为研究的试验台，并将对许多其他未来的研究工作具有巨大的内在价值。当技术和数据可供原始受众以外的用户使用时，他们通常会发现令人兴奋的应用程序。全球定位系统的许多创新应用就是一个例子。另一个更贴切的例子是气象雷达数据在诸如水文学、鸟类和昆虫研究以及气候学等领域的新应用。如果这些数据集能够以本项目所解决的方式轻松使用，将会有许多新的用途。

Geophysical data sets are typically very large. For example, an operational weather radar can generate almost 0.9 terabyte of data per year. Developing continental-scale, multiyear data sets can become a major and potentially very costly undertaking. Similar statements are true for geophysical satellite data.Advances in information technology, particularly rapid increases in disk storage capacity alleviate some data access problems, but accentuate others. It may be possible to get a terabyte of geophysical data online but visualizing and analyzing the data in their native/distribution format remains extremely difficult. Approaches such as data compression or using derived products sometimes work, but are simplistic and ultimately counterproductive. The problem is clearly multifaceted, but the focus of this project is on researching and developing methods for dealing with the data after distribution to the user. Specifically the Principal Investigator will develop a client-server architecture that hides the complexities of the data set management from the user. A major component is a data query language or DQL for manipulating a data set. Through DQL a user can easily and efficiently add, delete, or modify, subsets of a data set, and rapidly search for subsets that match certain criteria. The DQL client-server includes resource management so that a user may control the impact of an analysis on the client CPU, network utilization, and other computing resources. Part of the project is the completion of a multiyear, multisite data set of weather radar data. This will serve as a test bed for the research and will have tremendous intrinsic value for many other future research efforts.When technology and data become available to users other than the original audience, they often find exciting applications. The many innovative applications of the global positioning system are one example. Another more pertinent example is the novel application of weather radar data to such fields as hydrology, bird and insect studies, and climatology, to name a few. There would be many new uses if these data sets were easily usable in the way this project address.