Mid-Air Touch Technology for Accelerated Digital Human Integration and Enhanced Multisensory Experiences

用于加速数字人集成和增强多感官体验的空中触摸技术

• 批准号: MR/V025511/1
• 负责人: Patricia Ivette Cornelio Martinez
• 金额: $ 101.73万
• 依托单位: Ultraleap
• 依托单位国家: 英国
• 项目类别: Fellowship
• 财政年份: 2022
• 资助国家: 英国
• 起止时间: 2022 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=MR%2FV025511%2F1
• 关键词: Mid; Air; Touch; Technology; Accelerated

In this FLF proposal, we expand the advances of mid-air haptic technology through the lens of multisensory experiences by applying the principles of human-computer interaction (HCI). Traditional user interfaces typically employ physical touch, e.g., pressing buttons and taping on touchscreens. Mid-air interaction technology replaces "taping on screens" with hand-gesture recognition algorithms and holographic touch. Cameras track your hands to activate actions on a computer system from a distance while phased ultrasonic arrays deliver tactile sensations to your palms and fingertips in mid-air. This unique combination enables novel interaction paradigms previously only seen in sci-fi movies (e.g., Minority Report). We can now interact with computers, digital objects and with each other in immersive 3D environments that we can not only see and hear, but can also touch and feel, without the need to wear or hold any additional controllers.To date, innovation in mid-air technologies has been focussed on hardware and software development to advance engineering methods related to accuracy, recognition and rendering. However, very little is known about how these technologies influence human behaviour and therefore it is unknown how to exploit human perception to improve interactivity and help society. This fellowship is at the forefront of enabling the evolution of those novel and innovative interaction paradigms through the specific lens of multisensory experiences not just taking a leap from physical to mid-air touch interactions, but also integrating it with other sensory modalities (vision, audio, smell). Through studying the cross-sensory associations and integration of mid-air touch into multisensory experiences, this fellowship aspires to create more emotionally engaging and compelling digital experiences where the human senses are as important as in the real world. Then, taking advantage of the effect of multisensory stimuli on human decision-making, we take a unique perspective on understanding the impact of such novel mid-air interfaces on societal responsibility. In particular, we focus on improving the users' sense of agency (SoA) when interacting with autonomous systems. Today, intelligent systems involve increased automation and make many decisions for us (cars, smart homes, robots), which tend to reduce the SoA i.e., the feeling of being in control of one's actions. By understanding the effects of mid-air interactions enhanced by multisensory experiences on the SoA, we aim to design and develop more responsive systems that provide the user with a feeling of being in control and strengthen responsible interactions.As a result of this project, we ambition to help create a future where the holographic display in autonomous vehicles can be touched and felt, and that this tactile dimension generated by focused ultrasound patterns is designed to promote a sense of trust towards the autonomous system, and a future where we can have a 3D video call with friends and family whilst being able to communicate our affection by sending tactile sensations generated by a beam of focused ultrasound and even perceive smells from their environment.This FLF will challenge existing interaction paradigms in HCI through a systematic understanding of the role of mid-air haptics from a multisensory perspective, aiming to design digital interactions where you can see, hear, touch and smell, and this multisensory experience helps to share agency between users and technology. Our approach will break from the conventional studies in mid-air technology by fostering a new and inclusive ecosystem of multidisciplinary research around mid-air technology involving psychology, neuroscience and HCI so that the impact of these technology is not limited to hardware and software, but also provides a positive impact on society with respect an accelerated digitisation of human experiences.

在这个FLF方案中，我们通过应用人机交互(HCI)的原理，通过多感觉体验的镜头来扩展空中触觉技术的进步。传统的用户界面通常使用物理触摸，例如按下按钮和在触摸屏上粘贴。空中交互技术用手势识别算法和全息触摸取代了“屏幕上的胶带”。摄像头跟踪你的手，从远处激活电脑系统上的动作，而相控超声波阵列在半空中向你的手掌和指尖提供触觉感觉。这种独特的组合使得以前只在科幻电影(如《少数民族报告》)中才能看到的新颖互动范式成为可能。我们现在可以在身临其境的3D环境中与计算机、数字对象和彼此交互，我们不仅可以看到和听到，而且可以触摸和感觉，而不需要佩戴或握住任何额外的控制器。到目前为止，空中技术的创新一直集中在硬件和软件开发上，以推进与精度、识别和渲染相关的工程方法。然而，人们对这些技术如何影响人类行为知之甚少，因此不知道如何利用人类的感知来改善互动和帮助社会。通过多感官体验的特定镜头，该奖学金处于推动这些新颖和创新的互动范式演变的前沿，不仅从物理触摸互动跃升到空中触摸互动，而且还将其与其他感官形式(视觉、音频、嗅觉)相结合。通过研究跨感官联系和将空中触摸融入多感官体验，该奖学金渴望创造更具情感吸引力和引人入胜的数字体验，其中人类感官与现实世界一样重要。然后，利用多感官刺激对人类决策的影响，我们以独特的视角理解这种新颖的空中界面对社会责任的影响。特别是，我们专注于提高用户在与自治系统交互时的代理感(SoA)。今天，智能系统涉及到更多的自动化，并为我们做出许多决策(汽车、智能家居、机器人)，这往往会降低SoA，即控制一个人的行动的感觉。通过了解多感官体验增强的空中交互对SoA的影响，我们的目标是设计和开发更灵敏的系统，为用户提供控制的感觉，并加强负责任的交互。作为该项目的结果，我们希望帮助创造一个自动车辆中的全息显示可以触摸和感受的未来，以及由聚焦超声模式产生的这种触觉维度旨在促进对自动系统的信任感，在未来，我们可以与朋友和家人进行3D视频通话，同时能够通过发送由聚焦超声波产生的触觉感觉来交流我们的情感，甚至能够从他们的环境中感知气味。这一FLF将通过从多感官角度系统地理解空中触觉的作用来挑战现有的人机交互模式，旨在设计你可以看到、听到、触摸和闻到的数字互动，这种多感官体验有助于在用户和技术之间共享代理。我们的方法将打破传统的空中技术研究，围绕空中技术培育一个新的、包容的多学科研究生态系统，涉及心理学、神经科学和人机交互，使这些技术的影响不仅限于硬件和软件，还将在人类体验的加速数字化方面对社会产生积极影响。

项目成果

Responsible Innovation of Touchless Haptics: A Prospective Design Exploration in Social Interaction

非接触式触觉的负责任创新：社交交互中的前瞻性设计探索

• 发表时间: 2023
• 作者: Cornelio P

It Sounds Cool: Exploring Sonification of Mid-Air Haptic Textures Exploration on Texture Judgments, Body Perception, and Motor Behaviour.

听起来很酷：探索空中触觉纹理的声音化对纹理判断、身体感知和运动行为的探索。

• DOI: 10.1109/toh.2023.3320492
• 发表时间: 2023
• 期刊: IEEE transactions on haptics
• 影响因子: 2.9
• 作者: Montano-Murillo R

The sense of agency in emerging technologies for human-computer integration: A review.

• DOI: 10.3389/fnins.2022.949138
• 发表时间: 2022
• 期刊: FRONTIERS IN NEUROSCIENCE
• 影响因子: 4.3
• 作者: Cornelio, Patricia;Haggard, Patrick;Hornbaek, Kasper;Georgiou, Orestis;Bergstroem, Joanna;Subramanian, Sriram;Obrist, Marianna
• 通讯作者: Obrist, Marianna

Ultrasound Mid-Air Haptics for Touchless Interfaces

用于非接触式界面的超声波空中触觉

• DOI: 10.1007/978-3-031-04043-6_4
• 发表时间: 2022
• 作者: Cornelio P