Autonomic Data Management for Very Large Dataset Visualization

在历史上，我们从来没有过这样的能力来生成、收集和存储数字数据。在生物信息学、医学、遥感和纳米技术等许多应用中，TB级的数据存储库正变得司空见惯。在一些应用中，如网络流量可视化和视频可视化，我们遇到了动态数据流几乎在时间上无限的场景。许多可视化任务正在演变为可视化数据挖掘过程。在这个提出的协作项目中，需要解决的主要科学挑战是如何管理涉及超大数据集的可视化任务，这些数据集必须驻留在一个或几个远程数据服务器上，原因包括空间需求过大、数据共享提供、动态数据生成和数据安全。由于数据采集和数据生成技术(如基于Web的数据采集和网格计算)的快速发展，在复杂的计算和通信基础设施上管理这种可视化正在成为交互式可视化数据挖掘的瓶颈。不可能设计一种统一的方法来满足不同的需求，也不可能以成本效益的方式将复杂数据管理的负担放在用户身上。发展复杂计算基础设施的一个新兴战略是自主计算，它在自适应的生物系统和自治的社会和经济系统中寻找灵感。在最近由国际和平研究所协调的一项调查中，自主计算的概念首次被纳入可视化的背景下，并建议以一种渐进的方式将这一概念部署在可视化基础设施中。本研究的目的是研究一种用于可视化中的超大数据集的自主管理方法，设计并构建用于获取动态系统信息和用户体验的基于知识的框架，并开发能够动态地利用这些信息来进行可视化数据管理的自适应算法。这项联合研究计划将主要由国际和平研究所陈敏教授和他的博士生David Chisnall先生与犹他大学的Charles Hansen教授及其研究团队合作进行。开展这项联合计划的主要资源已经到位，目前由Chisnall先生担任博士研究生(2004年1月至2007年12月)。对O_1、O_2和O_3进行了一些研究。因此，我们不为人员编制和设备费用寻求任何资金。为该项目寻求的资金用于支付旅费和生活费，因为面对面的会议和短期访问在涉及密集的算法设计和软件开发以及直接接触犹他州的计算基础设施、数据集和科学用户的如此密切的合作中是不可或缺的。

Never before in history have we had such capability for generating, collecting and storing digital data. Data repositories at terabyte level are becoming commonplace in many applications, including bioinformatics, medicine, remote sensing and nano-technology. In some applications, such as network traffic visualization and video visualization, we are encountering the scenario that dynamic data streams are almost temporally unbounded. Many visualization tasks are evolving into visual data mining processes.The main scientific challenge to be tackled in this proposed collaborative project is how to manage visualization tasks that involve very large datasets which must reside on one or a few remote data servers for various reasons, including excessive space requirements, data sharing provision, dynamic data generation, and data security. Because of the rapid developments of data capture and data generation technologies (e.g., web-based data collection and Grid computing), the management of such visualization over a complex computing and communication infrastructure is becoming a bottleneck to interactive visual data mining. It is not possible to device a uniform approach to suit various requirements nor cost-effective to place the burden of complex data management upon users.One emerging strategy for developing complex computing infrastructure is autonomic computing, which seeks inspiration in self-adaptive biological systems and self-governing social and economic systems. In a recent survey coordinated by the PI, the concept of autonomic computing was first brought into the context of visualization, and it was suggested that the concept can be deployed in a visualization infrastructure in an evolutionary manner. It is desirable to investigate into an autonomic approach to the management of very large datasets in visualization, to design and prototype knowledge-based framework for capturing dynamic system information and users' experiences, and to develop adaptive algorithms that can utilize such information dynamically for data management in visualization.The aim of this research is to develop an autonomic approach to the management of very large dataset visualization. The joint research programme will be carried out primarily by the PI, Professor Min Chen, and his PhD student, Mr. David Chisnall, in collaboration with Professor Charles Hansen and his research team in University of Utah.The main resources for conducting this joint programme is in place with a current PhD studentship held by Mr. Chisnall (January 2004 - December 2007). Some work on O1, O2, and O3 has been carried out. We therefore do not seek any funding for staffing and equipment costs. The funding sought for this project is for covering travelling and subsistence cost, as face-to-face meetings and short-term visits are indispensable in such close collaboration involving intensive algorithm design and software development, as well as direct access to computing infrastructure, datasets, and scientific users in Utah.