II-New: Collaborative: A Mixed Reality Environment for Enabling Everywhere Data-Centric Work

II-新：协作：支持无处不在的以数据为中心的工作的混合现实环境

• 批准号: 1629913
• 负责人: Aidong Lu
• 金额: $ 39.93万
• 依托单位: University of North Carolina at Charlotte
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-10-01 至 2021-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1629913&HistoricalAwards=false
• 关键词: II; New; Collaborative; Mixed; Reality

This infrastructure project will develop an open source software toolkit, called OpenMR, to support building "mixed reality" data analysis systems that project data into the physical world using a new class of display devices such as Microsoft Hololens and Oculus Rift. Through OpenMR, these lightweight, wearable, mobile devices will tap into data-intensive infrastructures hosted in the cloud, with the goal of developing systems that allow users to perform data-intensive tasks from anywhere, without requiring heavy dedicated large-format displays supported by dedicated local computers. To pursue this research, the investigators will acquire both dedicated cloud-computing servers (to support data analysis) and mixed reality hardware devices (to create the interfaces). They will develop OpenMR to connect this hardware, to support common analysis tasks such as selecting, filtering, and classifying data, and to create data displays in the physical world. To both demonstrate the toolkit and advance data analysis research, they will build a number of prototype mixed reality interfaces for researchers whose work requires analyzing a large amount of data in domains including weather, biology, and medical imaging. In addition to advancing those specific research areas, studying these prototypes with real users will support research around the underlying data analysis techniques, the cognitive science of how people interact with data in the physical world, and the design principles needed to build mixed reality systems. This, in turn, will make these emerging technologies more likely to succeed and spread, and increase the chance of finding potential 'killer apps' for these systems. The infrastructure will also directly support education and research at the partner universities around data visualization, computer graphics, computer vision, and machine learning, while the release of the toolkit will benefit the wider community. This research is timely and important because as smart devices, in particular virtual and mixed reality devices such as Google Glass, Microsoft Hololens, Oculus Rift and Google Cardboard, become commonplace, these devices will play an increasingly important role relative to traditional laptop and digital computers when interacting with digital information. The long-term vision of the project is to develop a mixed reality research infrastructure to support everywhere data-centric innovations, providing immersive, intuitive, location-free, advanced machine learning, data analysis, reduction, summary and storage tools. This includes advanced support for the full pipeline of data-centric work in mixed reality spaces through the OpenMR open source toolkit, including front end visualization and interaction that leverages awareness of available rendering spaces and hardware along with effective visualization patterns in 2D and 3D spaces to optimize interaction; key components of data analysis and machine learning on the middle layers including automatic, generic feature engineering and joint optimization of classification performance and effective identification of discriminating features; and high-performance computing and cost-sensitive job management on the server. The team will evaluate OpenMR's efficiency, stability, scalability, functionality, flexibility, and ease of adoption through a number of mechanisms, including self-evaluations and documentation of the design process, review from domain experts, and evaluation with both expert and novice users on data analysis tasks that cur across the specific application domains described above. The toolkit itself will be released on the GitHub open source platform during the third year of the project after it has reached an initial level of maturity and usefulness. The investigators will publicize OpenMR through a Youtube channel with a set of demonstration videos; outreach to relevant researchers interested in immersive visualization, visual analytics, multi-sensory human-computer interaction, machine learning with human-in-the-loop, and high-performance computing; and collaboration with undergraduates in the Students, Technology, Academia, Research, and Service Computing Corps consortium.

这个基础设施项目将开发一个名为OpenMR的开源软件工具包，以支持构建“混合现实”数据分析系统，该系统使用微软Hololens和Oculus Rift等新型显示设备将数据投射到物理世界。通过OpenMR，这些轻量级、可穿戴的移动设备将利用托管在云中的数据密集型基础设施，目标是开发允许用户从任何地方执行数据密集型任务的系统，而不需要由专用本地计算机支持的重型专用大屏幕显示器。为了进行这项研究，调查人员将获得专用云计算服务器(用于支持数据分析)和混合现实硬件设备(用于创建界面)。他们将开发OpenMR来连接这个硬件，以支持常见的分析任务，如选择、过滤和分类数据，并在物理世界中创建数据显示。为了演示工具包和高级数据分析研究，他们将为需要分析天气、生物和医学成像等领域的大量数据的研究人员构建一些混合现实界面原型。除了推进这些特定的研究领域外，与真实用户一起研究这些原型还将支持围绕基本数据分析技术、关于人们如何与物理世界中的数据交互的认知科学以及构建混合现实系统所需的设计原则的研究。反过来，这将使这些新兴技术更有可能成功和传播，并增加为这些系统找到潜在“杀手级应用”的机会。该基础设施还将直接支持合作大学在数据可视化、计算机图形学、计算机视觉和机器学习方面的教育和研究，而工具包的发布将使更广泛的社区受益。这项研究是及时而重要的，因为随着智能设备，特别是谷歌眼镜、微软全息眼镜、Oculus Rift和谷歌纸板等虚拟和混合现实设备变得司空见惯，这些设备在与数字信息交互时将发挥相对于传统笔记本电脑和数字计算机越来越重要的作用。该项目的长期愿景是开发混合现实研究基础设施，以支持无处不在的以数据为中心的创新，提供身临其境、直观、无位置、先进的机器学习、数据分析、简化、总结和存储工具。这包括通过OpenMR开源工具包为混合现实空间中以数据为中心的工作的全线提供高级支持，包括前端可视化和交互，利用对可用渲染空间和硬件的感知以及2D和3D空间中的有效可视化模式来优化交互；中间层的数据分析和机器学习的关键组件，包括自动、通用特征工程和分类性能的联合优化和有效识别区分特征；以及服务器上的高性能计算和成本敏感型作业管理。该团队将通过一系列机制评估OpenMR的效率、稳定性、可扩展性、功能性、灵活性和易用性，包括设计过程的自我评估和文档记录、领域专家的审查，以及与专家和新手用户就上述特定应用领域的数据分析任务进行评估。该工具包本身将在项目的第三年期间在GitHub开源平台上发布，届时它已经达到了初步的成熟度和实用性。调查人员将通过YouTube频道和一套演示视频来宣传OpenMR；接触对沉浸式可视化、视觉分析、多感官人机交互、人在环中的机器学习和高性能计算感兴趣的相关研究人员；以及与学生、技术、学术、研究和服务计算公司联盟的本科生合作。

项目成果

Improving Information Sharing and Collaborative Analysis for Remote GeoSpatial Visualization Using Mixed Reality

使用混合现实改进远程地理空间可视化的信息共享和协作分析

• DOI: 10.1109/ismar.2019.00021
• 发表时间: 2019
• 期刊: 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR
• 作者: Mahmood, Tahir;Fulmer, Willis;Mungoli, Neelesh;Huang, Jian;Lu, Aidong
• 通讯作者: Lu, Aidong

Towards mobile immersive analysis: A study of applications

迈向移动沉浸式分析：应用研究

• DOI: 10.1109/immersive.2016.7932378
• 发表时间: 2016
• 期刊: 2016 Workshop on Immersive Analytics (IA
• 作者: Lu, Aidong;Huang, Jian;Zhang, Shaoting;Wang, Chuang;Wang, Weichao
• 通讯作者: Wang, Weichao

Scalable web-embedded volume rendering

• DOI: 10.1109/ldav.2017.8231850
• 发表时间: 2017-10
• 期刊: 2017 IEEE 7th Symposium on Large Data Analysis and Visualization (LDAV)
• 作者: Mohammad Raji;Alok Hota;Jian Huang

Cross-Platform Immersive Visualization and Navigation with Augmented Reality

跨平台沉浸式可视化和增强现实导航

• 发表时间: 2021
• 期刊: VINCI 2021: The 14th International Symposium on Visual Information Communication and Interaction
• 作者: Murari, Akshay;Mahfoud, Elias;Wang, Weichao;Lu, Aidong
• 通讯作者: Lu, Aidong

Photo-Guided Exploration of Volume Data Features

• DOI: 10.2312/pgv.20171091
• 发表时间: 2017-10
• 作者: Mohammad Raji;Alok Hota;R. Sisneros;P. Messmer;Jian Huang