Mining interesting patterns from big data

在当前的大数据时代，大量的各种不同准确性的有价值的数据（例如，不精确或不确定的数据，如传感器数据和医学实验室测试结果，其中的内容由于诸如固有的测量不准确性或采样频率之类的因素而不确定）可以被容易地生成或高速收集。因此，我们淹没在数据中，却渴望知识。为了能够理解这些数据，需要大数据管理和大数据挖掘（发现可能嵌入数据中的隐含的、以前未知的和潜在有用的知识）的数据科学解决方案。在过去的几年里，我和我的HQP一起开发了使用概率方法从不确定数据中找到频繁项集的算法。这些算法通过一些优化得到了增强，包括一些用于捕获数据中重要内容的内存树结构。沿着我目前的研究计划的这个方向，我计划扩大我的研究工作，目标是建立一个更有效，用户友好，强大的数据科学系统，从大数据中挖掘有趣的模式。具体来说，我计划（1）调整开发的算法，以考虑用户的偏好，并将这些用户的约束条件内的挖掘过程，使最终的算法只输出有趣的模式，没有后处理步骤是必要的;（2）挖掘和集成异构数据集从多个相关的来源，并纳入先验和后验知识，这些数据;（3）通过探索其他优化和技术（例如，Apache Spark、Scala），并进一步降低内存需求（例如，通过调整Bigtable）;（4）探索其他定量和定性方法来捕获和分析更多的数据和信息;（5）探索现实生活中的应用（例如，挖掘社交网络、Web、电信数据、农业数据、气象数据、新闻馈送、推文、博客）并找到其他感兴趣的模式，诸如趋势、序列和子图（例如，社交网络/图）;以及（6）开发数据可视化和可视化分析工具，其使得用户能够可视化和分析数据，以基于从数据挖掘的可视化信息来改变用户指定的挖掘参数和/或偏好。这将使最终的系统更具交互性和探索性。这反过来又有助于用户丰富他们的知识，以便他们能够迅速采取适当的行动或做出最佳（商业或军事）决策，从而对改善人类生活和加拿大经济以及国防和安全产生重大的积极影响。

In the current era of big data, high volumes of a wide variety of valuable data of different veracity (e.g., imprecise or uncertain data like sensor data and medical lab test results, in which the contents are uncertain due to factors like inherited measurement inaccuracies or sampling frequency) can be easily generated or collected at a high velocity. Consequently, we are drowning in data but starving for knowledge. In order to be able to make sense of these data, data science solutions for big data management and big data mining (which discovers implicit, previously unknown & potentially useful knowledge that might be embedded in data) are in demand. Over the past few years, I--together with my HQP--have developed algorithms that use probabilistic approaches for finding frequent sets of items from uncertain data. The algorithms are enhanced with a few optimizations, including some in-memory tree structures for capturing important contents in the data. Along this direction of my current research program, I plan to broaden my research work with an objective to build a more efficient, user-friendly, and powerful data science system for mining interesting patterns from big data. Specifically, I plan to (1) adapt the developed algorithms to take into account the user preference and push these user constraints inside the mining process so that the resulting algorithms only output interesting patterns and no post-processing step is needed; (2) mine & integrate heterogeneous data sets from multiple related sources, and incorporate prior & posterior knowledge about these data; (3) further improve performance so as to provide users with real-time responses by exploring other optimizations and techniques (e.g., Apache Spark, Scala) and further reduce the memory requirements (e.g., by adapting Bigtable); (4) explore other quantitative & qualitative approaches to capture & analyze more data & information; (5) explore real-life applications (e.g., mining social networks, Web, telecommunication data, agricultural data, meteorological data, news feed, tweets, blogs) and find other interesting patterns such as trends, sequences & subgraphs (e.g., social networks/graphs); and (6) develop data visualization & visual analytics tools that enable users to visualize and analyze data, to change the user-specified mining parameters, and/or preferences based on the visualized information mined from the data. This would make the resulting system more interactive and exploratory. This, in turn, helps users to enrich their knowledge so that they could promptly take appropriate actions or to make the best (business or military) decision, which would consequently have significant positive impacts on the improvement of human life and the benefits to Canadian economy as well as national defense & security.