Parrot: A Progressive Analysis System on Large Text Collections
Parrot: A Progressive Analysis System on Large Text Collections
复制标题
Parrot：大型文本集合的渐进分析系统
DOI：
10.1007/s41019-020-00144-y
复制
发表时间：
2021
影响因子：
4.2
通讯作者：
Wang X. Sean
中科院分区：
文献类型：
--
作者：
Zhang Yazhong;Zhang Hanbing;He Zhenying;Jing Yinan;Zhang Kai;Wang X. Sean
文本数据的大小随着对及时且经济高效的分析的需求而沿着增长，而计算能力的增长却无法跟上数据的增长。处理大量文本数据时的延迟可能会对用户活动和洞察力产生负面影响。这就需要从阻塞方式到渐进式处理的范式转变。在本文中，我们提出了一个基于样本的渐进式处理模型，侧重于词频计算的文本。该模型基于增量执行引擎，并将以渐进的方式为单个查询计算一系列近似结果，以提供准确性和延迟之间的平滑权衡。作为一部分，我们提出了一个新的自举技术的变种，逐步量化结果误差。我们在Apache Spark之上的名为Parrot的系统中实现了这种方法，并使用真实数据来测试其性能。实验结果表明，我们的方法是2.4×-19.7×快速得到1%的误差范围内的结果，而置信区间总是覆盖准确的结果非常好。
The size of textual data continues to grow along with the need for timely and cost-effective analysis, while the growth of computation power cannot keep up with the growth of data. The delays when processing huge textual data can negatively impact user activity and insight. This calls for a paradigm shift from blocking fashion to progressive processing. In this paper, we propose a sample-based progressive processing model that focuses on term frequency calculation on text. The model is based on an incremental execution engine and will calculate a series of approximate results for a single query in a progressive way to provide a smooth trade-off between accuracy and latency. As a part, we proposed a new variant of the bootstrap technique to quantify result error progressively. We implemented this method in our system called Parrot on top of Apache Spark and used real-world data to test its performance. Experiments demonstrate that our method is 2.4×–19.7× faster to get a result within 1% error while the confidence interval always covers the accurate results very well.