Collaborative Research: High-Performance Techniques, Designs and Implementation of software Infrastructure for Change Detection and Mining

摘要NSF 0536994，ChoudharyNSF 0536947，FOX管理、自动发现和传播信息的问题对国防、国土安全以及应急准备和响应至关重要。这些数据中的大部分来自在线传感器，这些传感器充当流数据源，提供持续的信息流。随着传感器资源的激增，数据流变得泛滥，以及时和可理解的方式提取和提供重要特征变得越来越困难。更具体地说，为数据泛滥的应用程序开发数据挖掘和同化工具面临三个基本挑战。分布式实时流的数据量如此之大，以至于即使是目前的极值计算也无法有效地处理它。其次，当今可广泛部署的网络协议和Web服务无法提供大容量实时数据流和分布式计算资源连接到具有高带宽延迟产品的网络所需的低延迟和高带宽。最后，当今绝大多数的统计和数据挖掘算法都假定所有数据都位于同一位置，并且都位于文件中。这里，实时数据流是分布式的，必须对使用它们的应用程序进行优化，以处理多个大容量实时流。目标是开发新的算法和硬件加速方案，以允许对这样的大规模流数据集进行实时统计建模和变化检测。通过使用面向服务的体系结构原则，将开发和测试一个框架，用于将高性能的变化检测软件服务集成到网格消息传递底层中，包括统计建模中常用内核的加速。将使用开放地理空间联盟标准支持地理信息系统服务，以实现地理参考。该项目有可能在几个重要领域产生近期和长期影响。在短期内，统计建模和变化检测算法的核心和模块的实施将使最终用户应用程序(如国土安全、国防)在数据驱动的决策支持方面实现一到两个数量级的性能改进。从长远来看，用于变化检测和数据挖掘算法的工具包和内核的可用性将促进国防、安全、科学等许多领域的应用程序的开发。此外，这项研究将使用可重构的结构加速对流数据的功能，包括变化检测和数据挖掘，从而开辟新的研究途径，并使更新的数据驱动应用程序能够处理复杂的数据集。研究生和本科生(通过本科生奖学金)都参与了这项研究。此外，小组成员还利用音频/视频和远程教育工具积极接触为少数群体服务的机构。

ABSTRACTNSF 0536994, ChoudharyNSF 0536947, FoxProblems in managing, automatically discovering, and disseminating information are of critical importance to national defense, homeland security, and emergency preparedness and response. Much of this data originates from on-line sensors that act as streaming data sources, providing a continuous flow of information. As sensor sources proliferate, the flow of data becomes a deluge, and the extraction and delivery of important features in a timely and comprehensible manner becomes an ever increasingly difficult problem. More specifically, developing data mining and assimilation tools for data deluged applications faces three fundamental challenges. The amount of distributed real time streaming data is so large that even current extreme scale computing cannot effectively process it. Second, today's broadly deployable network protocols and web services do not provide the low latency and high bandwidth required by high volume real time data streams and distributed computing resources connectedover networks with high bandwidth delay products. Finally, the vast majority of today's statistical and data mining algorithms assume that all the data is co-located and at rest in files. Here, the real time data streams are distributed and the applications that consume them must be optimized to process multiple high volume real time streams. The goal is to develop novel algorithms and hardware acceleration schemes to allow real-time statistical modeling and change detection on such large-scale streaming data sets. By using Service Oriented Architecture principles, a framework for integrating high -performance change detection software services, including accelerations of commonly used kernels in statistical modeling, into a Grid messaging substrate will be developed and tested. Geographical Information System (GIS) services will be supported usingOpen Geospatial Consortium standards to enable geo-referencing. This project has the potential to have near-term and long-term impact in several important areas. In the near-term, the implementation of kernels and modules of statistical modeling and change detection algorithms will allow the end-user applications (e.g., homeland security, defense) to achieve one to two orders of magnitude improvement in performance for data driven decision support. In the longer term, the availability of toolkits and kernels for the change detection and data mining algorithms will facilitate the development of applications in many areas including defense, security, science and others. Furthermore, this research will bring the use of reconfigurablearchitectural acceleration of functions on streaming data including change detection and data mining, thereby opening new avenues of research and enabling newer data-driven applications on complex datasets. Both graduate and undergraduate students (through undergraduate fellowships) are engaged in the research. In addition, team members actively engage with minority serving institutions using audio/video and distance education tools.