AitF: Collaborative Research: Fast, Accurate, and Practical: Adaptive Sublinear Algorithms for Scalable Visualization

随着人类活动的各个领域都产生了大量的数据，数据探索——分析、理解并从数据中提取价值——已经变得至关重要。数据可视化是迄今为止最常见的数据探索机制，新手和专家数据分析师都在使用它。然而，对越来越大的数据集进行数据可视化仍然很困难：即使是对大型数据集进行简单的可视化也可能是缓慢且非交互式的，而对数据集的采样部分进行可视化可能会误导分析师。该项目旨在开发FastViz，这是一个可扩展的可视化引擎，它不仅可以在同一时间内对大几个数量级的数据集进行可视化，而且还可以确保生成的可视化结果满足最终用户正确分析所需的关键属性。为了确保即时使用，FastViz将应用于三个现实世界的应用领域：电池科学、广告分析和基因组数据分析，并在PIs开发的开源视觉探索平台Zenvisage中实现。学生在项目中获得宝贵的经验，结合算法和系统的考虑，使数据探索。FastViz的开发是由对系统考虑因素的同时调查驱动的，例如支持各种形式在线采样的索引和存储技术，以及对(a)可视化生成的算法考虑，其目标是产生逐步改进的可视化，其中重要的特征首先显示，以及(b)可视化选择，其目标是从尚未生成的可视化集合中选择，那些满足期望标准的。在系统方面，FastViz将利用并回馈在线采样系统的最新发展，使其能够使用更强大的采样模式。在算法方面，FastViz将从测试、分布学习和次线性算法文献中汲取思想，就pi的最佳知识而言，这些思想尚未在实践中被采用。所开发的算法将遵循最优性保证，并且在可能的情况下，遵循实例最优性保证，确保它们将以最有效的方式适应数据特征。该项目将使人们更好地理解采样算法开发与系统设计之间的相互作用，一方面促进采用更现实的模型和算法，另一方面促进开发更强大的采样引擎，使算法所需的模型成为可能。

With the wealth of data being generated in every sphere of human endeavor, data exploration--analyzing, understanding, and extracting value from data--has become absolutely vital. Data visualization is by far the most common data exploration mechanism, used by novice and expert data analysts alike. Yet data visualization on increasingly larger datasets remains difficult: even simple visualizations of a large dataset can be slow and non-interactive, while visualizations of a sampled fraction of a dataset can mislead an analyst. The project aims to develop FastViz, a scalable visualization engine, that will not only enable visualization on datasets that are orders of magnitude larger in the same time, but also ensure the resulting visualizations satisfy key properties essential for correct analysis by end-users. To ensure immediate utilization, FastViz will be applied to three real-world application domains: battery science, advertising analysis, and genomic data analysis, and implemented in Zenvisage, an open-source visual exploration platform developed by the PIs. Students in the project gain invaluable experience in combining the algorithmic and systems considerations that enable data exploration. FastViz's development is driven by simultaneous investigation of systems considerations, such as indexing and storage techniques that enable various forms of online sampling, and algorithmic considerations for (a) visualization generation, where the goal is to produce incrementally improving visualizations in which the important features are displayed first, and (b) visualization selection, where the goal is to select, from a collection of as yet not generated visualizations, those that that satisfy desired criteria. On the systems front, FastViz will leverage and contribute back to recent developments on online sampling systems that enable the use of more powerful sampling modalities. On the algorithms front, FastViz will draw ideas from testing, distribution learning, and sublinear algorithms literature that, to the best knowledge of the PIs, have not been adapted in practice. The algorithms developed will obey optimality guarantees, and wherever possible, instance-optimality guarantees, ensuring that they will adapt to data characteristics in the most efficient way possible. The project will lead to a better understanding of the interplay between sampling algorithms development and systems design, facilitating the adoption of more realistic models and algorithms on the one hand, and the development of more powerful sampling engines that enable the models required within the algorithms.