ITR/IM: Methodologies Managing Large Geophysical Datasets

点击翻译按钮获取中文摘要

Geophysical data sets are typically very large. For example, an operational weather radar can generate almost 0.9 terabyte of data per year. Developing continental-scale, multiyear data sets can become a major and potentially very costly undertaking. Similar statements are true for geophysical satellite data.Advances in information technology, particularly rapid increases in disk storage capacity alleviate some data access problems, but accentuate others. It may be possible to get a terabyte of geophysical data online but visualizing and analyzing the data in their native/distribution format remains extremely difficult. Approaches such as data compression or using derived products sometimes work, but are simplistic and ultimately counterproductive. The problem is clearly multifaceted, but the focus of this project is on researching and developing methods for dealing with the data after distribution to the user. Specifically the Principal Investigator will develop a client-server architecture that hides the complexities of the data set management from the user. A major component is a data query language or DQL for manipulating a data set. Through DQL a user can easily and efficiently add, delete, or modify, subsets of a data set, and rapidly search for subsets that match certain criteria. The DQL client-server includes resource management so that a user may control the impact of an analysis on the client CPU, network utilization, and other computing resources. Part of the project is the completion of a multiyear, multisite data set of weather radar data. This will serve as a test bed for the research and will have tremendous intrinsic value for many other future research efforts.When technology and data become available to users other than the original audience, they often find exciting applications. The many innovative applications of the global positioning system are one example. Another more pertinent example is the novel application of weather radar data to such fields as hydrology, bird and insect studies, and climatology, to name a few. There would be many new uses if these data sets were easily usable in the way this project address.