Mixed reality in pediatric rehabilitation

混合现实（MR）视频游戏，其中使用现实生活中的物体与虚拟环境进行交互，可以为残疾儿童提供有吸引力的，基于家庭的机会，练习日常功能所需的精细运动技能。精确追踪物体的能力是MR电子游戏设计和广泛使用的关键。在磁共振成像中，摄像机通常用于跟踪物体，因为它们成本低、结构紧凑且易于获得。然而，基于相机的对象跟踪方法有局限性，通常需要用特殊标记/标识符标记对象，跟踪性能对环境（例如照明）很敏感，并且对于快速移动或遮挡的对象容易出错。该研究计划将通过结合来自许多不同来源和传感器的信息来解决当前MR应用中目标跟踪的挑战。例如，追踪玩家的移动可以指示玩家与目标对象的互动和移动。此外，MR游戏呈现的任务可以预测物体位置的变化时间/地点。同样，依靠不同的传感器（如颜色、深度、惯性）可以提供更多关于物体运动的信息内容，以及更好地适应操作条件变化的能力。本研究的新贡献将包括以下方法：(1)确定何时依赖不同的信息源/传感器，(2)动态调整系统以适应可能影响不同信息源可靠性的操作条件的变化（例如靠近传感器，遮挡），(3)检测与各种能力的儿童的物体交互相关的运动/手势，以及(4)推广为各种MR应用开发的方法。在本研究过程中，将设计出三种具有商业化和翻译潜力的新型MR治疗游戏。八名研究生和四名本科生将在计算机视觉、数字媒体和以用户为中心的设计方面接受高质量的跨学科培训，并有机会与数字媒体领域的行业领导者和临床环境中的医疗保健提供者进行互动。学员将有能力在数字媒体和卫生技术领域从事学术界、工业界和企业家的职业，这两个领域是加拿大需求最高的行业。

Mixed reality (MR) video games, wherein real-life objects are used to interact with virtual environments, can provide children with disabilities with engaging, home-based opportunities to practice fine motor skills needed for everyday function. The ability to accurately track objects is key to the design and widespread use of MR video games. Cameras are often used to track objects in MR as they are low-cost, compact and widely available. However, camera-based approaches to object tracking have limitations objects often need to be tagged with special markers/identifiers, tracking performance is sensitive to the environment (e.g. lighting) and prone to errors for fast-moving or occluded objects. This research program will address current challenges in object tracking for MR applications by combining information from many different sources and sensors. For instance, tracking movements of the player can be indicative of interactions with, and movements of, the targeted object. Additionally, the tasks presented by the MR game can be predictive of when/where to expect changes in object position. Likewise, relying on diverse sensors (e.g. colour, depth, inertial) can provide greater informational content regarding an object's movements and the ability to better adapt to changes in operating conditions. Novel contributions of this research will include methods to: (1) determine when to rely on different information sources/sensors, (2) adapt the system dynamically to changes in operating conditions (e.g. proximity to sensor, occlusions) that may impact the reliability of different information sources, (3) detect movements/gestures associated with object interactions for children with a range of abilities, and (4) generalize the approaches developed for diverse MR applications. In the course of this research, three novel MR therapy games will be designed with high potential for commercialization and translation. Eight graduate and four undergraduate students will be provided with high quality, interdisciplinary training in computer vision, digital media, and user-centred design with opportunities to interact with industry leaders in digital media and with healthcare providers in clinical contexts. Trainees will be well-positioned to pursue careers in academia, industry, and entrepreneurship in the fields of digital media and health technology, two of Canada's most high demand sectors.