Effective and Efficient Data Analysis Techniques for Emerging Data Intensive Applications

从手持设备到传感器，从监控到微阵列，再到网络的广泛可用性（例如，社交媒体，社交网络，物联网，点击流等），技术的最新发展允许以电子方式捕获，收集和维护大量结构化和非结构化数据，从而形成大数据存储库。示例数据源包括与通过Web访问进行信息检索相关的日志和事务、与点击流和零售购买等电子商务相关的日志和事务、与天气和股票市场状态相关的事件、与社会媒体和社会网络相关的社会交互、与交通控制、欺诈检测和国土安全相关的监控系统等等。这些数据被认为是知识发现的宝贵资源，有助于制定有效的决策。然而，这只有在这些大量数据能够得到有效处理和分析的情况下才能实现，而传统方法越来越难以做到这一点。有时数据可能会出现偏差，因为数据实例的数量与大量特征（如拥有数百个样本和数千个基因）相比很小；这可以通过数据充实来解决。此外，真实数据通常还存在各种其他相互关联的问题，这些问题需要处理降维、缺失值和噪声、有限计算能力下的可扩展性（即，使用常用计算机）、数据建模和集成以揭示特定语义（例如，捕获和监控社交媒体数据中的行为和趋势，以便提供更好的推荐）等。不能孤立地处理这些问题，因此需要一个综合框架，能够处理数据中出现的这些问题的任何组合。

Recent developments in the technology from handheld devices to sensors to surveillance to microarrays to the wide availability of the Web (e.g., social media, social networking, internet of things, click stream, etc.) allow for electronically capturing, collecting and maintaining huge volumes of structured and unstructured data leading to big data repositories. Example data sources include logs and transactions related to information retrieval via Web access, to e-commerce such as click streams and retail purchases, on-going events such as weather and stock market status, social interactions such as social media and social networking, monitoring systems such as traffic control, fraud detection, and homeland security, and so on. Such data is recognized as a valuable resource for knowledge discovery leading to effective decision making. Yet this can happen only if these large volumes of data can be processed and analyzed effectively, which is increasingly problematic to do by conventional means. Sometimes the data could be skewed where the number of data instances is small compared to the huge number of features like having hundreds of samples and thousands of genes; this could be fixed by data enrichment. In addition, real data generally suffers from various other interrelated problems that require dealing with dimensionality reduction, missing values and noise, scalability under limited computing power (i.e., using commonly available computers), data modeling and integration to uncover specific semantics (e.g., capturing and monitoring behavior and trend in social media data to allow for better recommendations), etc. These problems cannot be treated in isolation and hence there is a need for an integrated framework capable of handling any combination of these problems when present in the data.