Large-scale 3D Reconstruction with an R-based Analysis Workflow
Large-scale 3D Reconstruction with an R-based Analysis Workflow
复制标题
使用基于 R 的分析工作流程进行大规模 3D 重建
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Hui Zhang
中科院分区：
文献类型：
--
作者：
Riqing Chen;Hui Zhang
随着大规模分析的数据量和技术复杂性的增加，许多领域专家不再能够参与数据探索和分析工作流。所期望的是一个计算功能强大但仍然熟悉的分析界面，用于领域专家通过仅关注单个数据集来充分参与分析工作流，将大规模计算留给系统。为了实现这一目标，我们提出了VisRden，一个研究原型，结合了用户友好的可视化编程和可扩展的计算后端的大规模三维重建在龋损研究。VisRden使用R作为分析语言，通过隐藏可视化界面背后的计算复杂性，为用户提供一组核心功能，并允许高级用户提供自定义R脚本和变量，以完全嵌入到最终的分析脚本中。使用R作为分析语言，使龋学家能够继续探索数据，并以他们已经熟悉的方式提出新的分析方法。VisRden在类似MapReduce的框架中使用R和SGE（Sun Grid Engine）数组作业征服了大规模图像处理和3D重建。基于图像的操作和结果聚合被调度为并行方式的阵列作业，以加速知识发现过程。所有这些联合收割机提供了一个新的分析工作流程，用于执行类似的大规模分析循环，这些分析循环需要专家用户密切监督，提供反馈并细化子任务。
As the volume of data and technical complexity of large-scale analysis increases, many domain experts can no longer be seated in the data exploration and analysis workflow. What is desired is a computational powerful but still familiar analysis interface for domain experts to fully participate in the analysis workflow by just focusing on individual datasets, leaving the large-scale computation to the system. Towards this goal, we present VisRden, a research prototype that combines user friendly visual programming and scalable computing backend for large-scale 3D reconstruction in carious lesion research. VisRden uses R as the analysis language, making a set of core functions available to the users by hiding the computational complexity behind a visual interface, and allowing advanced users to provide custom R scripts and variables to be fully embedded into the final analysis script. Using R as the analysis language allows cariologists to continue explore data and propose new analysis methods in the way they are already familiar with. VisRden conquers large-scale image processing and 3D reconstruction in a MapReduce-like framework using R and SGE (Sun Grid Engine) array jobs. Image-based operations and result aggregation are scheduled as array jobs in a parallel means to accelerate the knowledge discovery process. All these combine to provide a new analytics workflow for performing similar large-scale analysis loops that need expert users to closely supervise, provide feedback, and refine the subtasks.