Web Data Analytics and Scientific Workflows

该项目的主要目标是提高Stratosphere快速分析需要迭代分析程序的问题中不断变化的大型数据集的能力。其中，我们专注于两个要求严格的领域：Web数据和科学工作流（SWfs）。第一个领域涉及非结构化、异构和分布式Web内容的分析。它建立在第一阶段所做的工作的基础上，在第一阶段中，我们的子项目关注的是陈述性信息提取的研究。在第二阶段，我们专注于文本Web数据的分析，包括通过集中的Web抓取动态获取这些数据。这需要对Stratosphere进行一些增强，例如处理Web数据的特定操作符，使用高级信息提取来提高重点爬行的准确性的方法，以及支持重点爬行固有的迭代性质的新颖执行模型。第二个研究领域的目标是实施，优化和有效执行SWf作为一个快速程序，特别是生命科学中下一代测序数据的分析工作流程。生产基因组数据的成本急剧下降，以至于在成为临床实践的标准之前，它很短，产生了大量的数据，必须由管道中运行的大量程序进行分析。SWF本质上是一种可编程的程序，因此通常非常适合由Stratosphere这样的系统来管理。然而，特别是基因组分析具有在并行优化和并行化期间必须考虑的特定属性。特别是，即使在大型中心之外，大规模测序也正在成为常规，这意味着分析也必须在小集群上高效执行，并且应该最佳地利用多核机器。该领域的另一个特性是不存在单一的标准化分析管道;相反，基因组要进行各种独立分析，导致功能部分重叠的复杂工作负载，为整体工作负载优化留下了很大的空间。

The main objective of this project is to enhance Stratosphere’s abilities to quickly analyze evolving, large datasets in problems that require iterative analytical programs. Therein, we focus on two demanding areas: Web data and Scientific Workflows (SWfs). The first area deals with the analysis of unstructured, heterogeneous, and distributed web content. It builds on work performed in the first phase, in which our subproject was concerned with research in declarative information extraction. In the second phase, we focus on the analysis of textual web data including the dynamic acquisition of such data through focused web crawls. This requires a number of enhancements to Stratosphere, such as specific operators to deal with web data, methods for using advanced information extraction to improve accuracy of focused crawling, and a novel execution model supporting the inherently iterative nature of focused crawling. The second research area targets the implementation, optimization, and efficient execution of SWf as dataflow programs, in particular analysis workflows for next generation sequencing data in the Life Sciences. The cost of producing genome data has fallen so steeply that it is short before becoming standard in clinical practice, creating an avalanche of data that must be analyzed by a multitude of programs running in pipelines. SWfs essentially are dataflow programs and thus generally well suited to be managed by a system as Stratosphere. However, especially genome analysis has particular properties that must be considered during dataflow optimization and parallelization. In particular, large-scale sequencing is becoming routine even outside large centers, which means that analysis must be performed efficiently also on small clusters and should optimally exploit multi-core machines. Another property of this area is that there exists no single, standardized analysis pipeline; rather, genomes are subjected to various independent analyses, resulting in complex workloads of partly overlapping functionality leaving much room for holistic workload optimization.