Design the Future 2: CrowdDesignVR

我们最初的提案CrowdDesign开始探索如何通过利用移动电话的多功能传感器功能来帮助快速制作基于移动传感器的用户界面的原型。主要目标是调查我们是否可以将这种依赖传感器的任务众包给移动设备，以帮助设计人员在现场快速评估新的交互技术。我们已经确定了一个非常强大的研究轨迹，推动CrowdDesign项目在今年之后继续下去：CrowdDesignVR。在这个后续项目中，我们建议大幅扩展CrowdDesign项目的范围，并将其从智能手机平台提升到虚拟现实平台。为了能够探索一条非常有前途的众包式人机交互的研究轨迹，我们需要投入时间和精力来实现一个高质量的VR众包平台。CrowdDesignVR将成为第一个用于虚拟现实的众包系统。它将在STeam VR分发网络上分发任务，这使得它能够接触到大量的VR用户。之前的研究无法达到这个规模，因为研究一直局限于对当地参与者进行机会抽样，然后培训他们使用特定的VR系统。相比之下，通过允许访问数千甚至数万个蒸汽用户，CrownDesignVR促进了用户交互数据的收集，其规模要大几个数量级。这提供了许多更广泛的好处：1)我们可以在项目过程中创建更准确的人类行为模型；2)我们可以收集足够的训练数据来训练机器学习模型，例如深度神经网络模型，以准确解码常见的用户界面交互模式，如打字、手势和确定用户是否有意进行操作。由于高保真VR环境中的众包任务是一种新的研究途径，有许多基本问题需要回答。我们相信这个项目可能会在理解VR中众包的设计空间方面产生潜在的开创性工作。另一个潜在的影响是数据本身。我们在构建用于解码触摸屏和物理键盘上的打字任务的深层神经网络的内部工作表明，深层神经网络(具体地说，递归神经网络)输出传统的隐马尔可夫模型解码。然而，我们也发现需要收集的数据量非常大，事实上，我们使用我们之前在报告中提到的CrowdDesign任务架构来收集数百名用户的触摸屏数据。CrowdDesignVR可以大大拓宽范围，让我们解决用户界面设计中一些以前没有解决的深层次问题，例如我们如何构建一个手势识别器，它能够学习识别2D平面和3D空间上的开环(直接回忆运动记忆)和闭合环(视觉引导运动)手势。大量的数据将使我们能够训练一个递归神经网络来学习这种分离。这种潜力是巨大的，因为用户总是处于开环和闭环交互之间的连续体中。然而，由于导致观察到的行为的潜在生成模型的根本差异，在实验室中很难收集足够的训练数据。

Our initial proposal CrowdDesign set out to explore how we can aid rapid prototyping of mobile sensor-based user interfaces by exploiting the versatile sensor capabilities of mobile phones. The primary objective was to investigate if we can crowdsource such sensor-dependent tasks to mobile devices in order to assist designers in rapidly evaluating new interaction techniques in situ. We have identified a very strong research trajectory that motivates continuing the CrowdDesign project beyond this year: CrowdDesignVR. In this follow-up project we propose to substantially extend the scope of the CrowdDesign project and elevate it from the smartphone platform and into a virtual reality platform. To enable the exploration of a very promising research trajectory for crowdsourced human-computer interaction, we need to invest time and effort into realising a high-quality crowdsourcing platform for VR.CrowdDesignVR will be the first crowdsourcing system for virtual reality. It will distribute tasks across the Steam VR distribution network, which allows it to reach a large sample of VR users. Prior research cannot reach this scale, as research has been limited to opportunity-sampling local participants and then train them to use a specific VR system. In contrast, by enabling access to thousands or even tens of thousands of Steam users, CrownDesignVR facilitates user interaction data collection at a scale that is several orders of magnitudes larger. This provides a number of wider benefits: 1) we can during the course of the project create more accurate models of human actions; 2) we can collect sufficient training data to train machine learning models, such as deep neural network models to accurately decode common user interface interaction patterns, such as typing, gesturing and determining whether an action was intended or not by the user.Since crowdsourcing tasks in a high-fidelity VR environment is a new avenue of research, there are many fundamental questions that need to be answered. We believe this project could result in potential seminal work on the understanding of the design space for crowdsourcing in VR.Another potential impact is the data itself. Our internal work on building deep neural networks for decoding typing tasks on touchscreen and physical keyboards has revealed that deep neural networks (specifically, recurrent neural networks) output traditional hidden Markov model decoding. However, we have also found that the amount of data that needs to be collected is very large, in fact, we use our CrowdDesign task architecture as mentioned previously in our report to collect touchscreen data from hundreds of users. CrowdDesignVR can substantially widen the scope and let us tackle some of deep previously unsolved questions in user interface design, such as how we can build a gesture recogniser that is capable of learning to recognise both open-loop (direct recall from motor memory) and closed-loop (visually-guided motion) gestures on both the 2D plane and in 3D space. A large amount of data would allow us to train a recurrent neural network to learn this separation. The potential is large as users are always in a continuum between open-loop and closed-loop interaction. However, due to the fundamental differences in the underlying generative models that result in the observed behaviour, it is very difficult to collect sufficient training data in lab.