Mixed reality in pediatric rehabilitation

混合现实（MR）视频游戏，其中现实生活中的对象被用来与虚拟环境进行交互，可以为残疾儿童提供有吸引力的，基于家庭的机会，以练习日常功能所需的精细运动技能。准确跟踪物体的能力是MR视频游戏设计和广泛使用的关键。相机通常用于跟踪MR中的对象，因为它们成本低、结构紧凑且广泛可用。然而，基于摄像机的对象跟踪方法具有局限性，对象通常需要用特殊的标记/标识符进行标记，跟踪性能对环境（例如照明）敏感，并且对于快速移动或被遮挡的对象容易出错。该研究计划将通过结合来自许多不同来源和传感器的信息来解决MR应用中目标跟踪的当前挑战。例如，跟踪玩家的移动可以指示与目标对象的交互以及目标对象的移动。另外，由MR游戏呈现的任务可以预测何时/何处预期对象位置的变化。同样，依靠不同的传感器（例如颜色，深度，惯性）可以提供关于物体运动的更多信息内容，并能够更好地适应操作条件的变化。本研究的新贡献将包括以下方法：（1）确定何时依赖不同的信息源/传感器，（2）使系统动态地适应操作条件的变化（例如，接近传感器、遮挡），其可能影响不同信息源的可靠性，（3）检测与具有一系列能力的儿童的对象交互相关联的移动/姿势，以及（4）推广针对不同MR应用开发的方法。 在这项研究的过程中，三个新的MR治疗游戏将被设计具有商业化和翻译的高潜力。 八名研究生和四名本科生将获得计算机视觉，数字媒体和以用户为中心的设计方面的高质量跨学科培训，并有机会与数字媒体行业领导者和临床环境中的医疗保健提供者进行互动。学员将有能力在学术界，工业界和数字媒体和健康技术领域的创业中追求职业生涯，这是加拿大两个最高需求的行业。

Mixed reality (MR) video games, wherein real-life objects are used to interact with virtual environments, can provide children with disabilities with engaging, home-based opportunities to practice fine motor skills needed for everyday function. The ability to accurately track objects is key to the design and widespread use of MR video games. Cameras are often used to track objects in MR as they are low-cost, compact and widely available. However, camera-based approaches to object tracking have limitations objects often need to be tagged with special markers/identifiers, tracking performance is sensitive to the environment (e.g. lighting) and prone to errors for fast-moving or occluded objects. This research program will address current challenges in object tracking for MR applications by combining information from many different sources and sensors. For instance, tracking movements of the player can be indicative of interactions with, and movements of, the targeted object. Additionally, the tasks presented by the MR game can be predictive of when/where to expect changes in object position. Likewise, relying on diverse sensors (e.g. colour, depth, inertial) can provide greater informational content regarding an object's movements and the ability to better adapt to changes in operating conditions. Novel contributions of this research will include methods to: (1) determine when to rely on different information sources/sensors, (2) adapt the system dynamically to changes in operating conditions (e.g. proximity to sensor, occlusions) that may impact the reliability of different information sources, (3) detect movements/gestures associated with object interactions for children with a range of abilities, and (4) generalize the approaches developed for diverse MR applications. In the course of this research, three novel MR therapy games will be designed with high potential for commercialization and translation. Eight graduate and four undergraduate students will be provided with high quality, interdisciplinary training in computer vision, digital media, and user-centred design with opportunities to interact with industry leaders in digital media and with healthcare providers in clinical contexts. Trainees will be well-positioned to pursue careers in academia, industry, and entrepreneurship in the fields of digital media and health technology, two of Canada's most high demand sectors.