Design the Future 2: CrowdDesignVR

点击翻译按钮获取中文摘要

Our initial proposal CrowdDesign set out to explore how we can aid rapid prototyping of mobile sensor-based user interfaces by exploiting the versatile sensor capabilities of mobile phones. The primary objective was to investigate if we can crowdsource such sensor-dependent tasks to mobile devices in order to assist designers in rapidly evaluating new interaction techniques in situ. We have identified a very strong research trajectory that motivates continuing the CrowdDesign project beyond this year: CrowdDesignVR. In this follow-up project we propose to substantially extend the scope of the CrowdDesign project and elevate it from the smartphone platform and into a virtual reality platform. To enable the exploration of a very promising research trajectory for crowdsourced human-computer interaction, we need to invest time and effort into realising a high-quality crowdsourcing platform for VR.CrowdDesignVR will be the first crowdsourcing system for virtual reality. It will distribute tasks across the Steam VR distribution network, which allows it to reach a large sample of VR users. Prior research cannot reach this scale, as research has been limited to opportunity-sampling local participants and then train them to use a specific VR system. In contrast, by enabling access to thousands or even tens of thousands of Steam users, CrownDesignVR facilitates user interaction data collection at a scale that is several orders of magnitudes larger. This provides a number of wider benefits: 1) we can during the course of the project create more accurate models of human actions; 2) we can collect sufficient training data to train machine learning models, such as deep neural network models to accurately decode common user interface interaction patterns, such as typing, gesturing and determining whether an action was intended or not by the user.Since crowdsourcing tasks in a high-fidelity VR environment is a new avenue of research, there are many fundamental questions that need to be answered. We believe this project could result in potential seminal work on the understanding of the design space for crowdsourcing in VR.Another potential impact is the data itself. Our internal work on building deep neural networks for decoding typing tasks on touchscreen and physical keyboards has revealed that deep neural networks (specifically, recurrent neural networks) output traditional hidden Markov model decoding. However, we have also found that the amount of data that needs to be collected is very large, in fact, we use our CrowdDesign task architecture as mentioned previously in our report to collect touchscreen data from hundreds of users. CrowdDesignVR can substantially widen the scope and let us tackle some of deep previously unsolved questions in user interface design, such as how we can build a gesture recogniser that is capable of learning to recognise both open-loop (direct recall from motor memory) and closed-loop (visually-guided motion) gestures on both the 2D plane and in 3D space. A large amount of data would allow us to train a recurrent neural network to learn this separation. The potential is large as users are always in a continuum between open-loop and closed-loop interaction. However, due to the fundamental differences in the underlying generative models that result in the observed behaviour, it is very difficult to collect sufficient training data in lab.