Web Data Analytics and Scientific Workflows

该项目的主要目标是增强Stratosphere快速分析需要迭代分析程序的问题中不断发展的大型数据集的能力。其中，我们关注两个要求很高的领域：Web数据和科学工作流（SWfs）。第一个领域涉及对非结构化、异构和分布式web内容的分析。它建立在第一阶段所执行的工作的基础上，在第一阶段中，我们的子项目与声明性信息提取的研究有关。在第二阶段，我们专注于文本网络数据的分析，包括通过集中的网络爬虫动态获取这些数据。这需要对Stratosphere进行一些增强，例如处理web数据的特定操作符，使用高级信息提取来提高聚焦爬行的准确性的方法，以及支持聚焦爬行固有迭代性质的新颖执行模型。第二个研究领域针对SWf作为数据流程序的实施、优化和有效执行，特别是生命科学中下一代测序数据的分析工作流程。生产基因组数据的成本急剧下降，很快就会成为临床实践的标准，这就产生了大量的数据，必须由大量正在运行的项目来分析。SWfs本质上是数据流程序，因此通常非常适合由系统管理，如Stratosphere。然而，特别是基因组分析具有在数据流优化和并行化过程中必须考虑的特殊属性。特别是，即使在大型中心之外，大规模测序也正在成为常规，这意味着分析必须在小集群上有效地执行，并且应该最佳地利用多核机器。这个领域的另一个特性是不存在单一的、标准化的分析管道；相反，基因组受到各种独立的分析，导致部分重叠功能的复杂工作负载，为整体工作负载优化留下了很大的空间。

The main objective of this project is to enhance Stratosphere’s abilities to quickly analyze evolving, large datasets in problems that require iterative analytical programs. Therein, we focus on two demanding areas: Web data and Scientific Workflows (SWfs). The first area deals with the analysis of unstructured, heterogeneous, and distributed web content. It builds on work performed in the first phase, in which our subproject was concerned with research in declarative information extraction. In the second phase, we focus on the analysis of textual web data including the dynamic acquisition of such data through focused web crawls. This requires a number of enhancements to Stratosphere, such as specific operators to deal with web data, methods for using advanced information extraction to improve accuracy of focused crawling, and a novel execution model supporting the inherently iterative nature of focused crawling. The second research area targets the implementation, optimization, and efficient execution of SWf as dataflow programs, in particular analysis workflows for next generation sequencing data in the Life Sciences. The cost of producing genome data has fallen so steeply that it is short before becoming standard in clinical practice, creating an avalanche of data that must be analyzed by a multitude of programs running in pipelines. SWfs essentially are dataflow programs and thus generally well suited to be managed by a system as Stratosphere. However, especially genome analysis has particular properties that must be considered during dataflow optimization and parallelization. In particular, large-scale sequencing is becoming routine even outside large centers, which means that analysis must be performed efficiently also on small clusters and should optimally exploit multi-core machines. Another property of this area is that there exists no single, standardized analysis pipeline; rather, genomes are subjected to various independent analyses, resulting in complex workloads of partly overlapping functionality leaving much room for holistic workload optimization.