COG-MHEAR: Towards cognitively-inspired 5G-IoT enabled, multi-modal Hearing Aids

目前，只有40%可以从助听器(HAS)中受益的人拥有助听器，而且大多数拥有HA设备的人使用频率不够高。使用看得见的声音(害怕看起来变老)是一种社会耻辱，它们需要有意识地努力专注于不同的声音和说话者，而且只有有限的使用语音增强--使口语(对人们来说，这通常是听力最重要的方面)更容易区分。仅仅让一切变得更响亮是不够的！为了在2050年之前改变听力护理，我们的目标是彻底重新思考已经设计的方式。我们的变革性方法--第一次--借鉴了正常听力的认知原理。听众很自然地将耳朵和眼睛的信息结合在一起：我们用眼睛帮助我们听力。我们将创造“多模式”辅助设备，它不仅能放大声音，还能同时利用从一系列传感器收集的信息来提高语音清晰度。例如，关于一个人所说的话的大量信息是通过视觉信息、说话者的嘴唇、手势等动作来传达的。这被当前的商业HAS忽略，并且可以被馈送到语音增强过程中。我们还可以使用可穿戴传感器(嵌入医管局本身)来估计听力努力及其对人的影响，并利用这一点来判断语音增强过程是否真的有所帮助。创造这些多模式的“视听”带来了许多艰巨的技术挑战，需要从整体上加以解决。传统上，使用嘴唇动作需要摄像机拍摄说话者，这会带来隐私问题。一旦收集到数据，我们就可以通过对数据进行加密来克服其中的一些问题，我们将开创在视频数据保持加密的情况下处理和理解视频数据的新方法。我们的目标是永远不访问原始视频数据，但仍然将其用作有用的信息来源。为了补充这一点，我们还将研究远程唇读的方法，而不使用视频馈送，而是探索使用无线电信号进行远程监控。添加这些新的传感器和理解产生的数据所需的处理将给HA设备带来显著的额外功率和小型化负担。我们将需要使我们复杂的视觉和声音处理算法以最小的功率和最小的延迟运行，并将通过专门的硬件实现来实现这一点，加速关键的处理步骤。长远来说，我们的目标是所有处理工作都在房委会本身进行--把资料保存在个人的本地，以保障私隐。在短期内，一些处理将需要在云中完成(因为它太耗电了)，我们将创建到云基础设施的新的极低延迟(&lt；10ms)接口，以避免在说出单词时和听到单词时之间的延迟。我们还计划利用灵活的电子产品(电子皮肤)和天线设计方面的进步，使整个单元尽可能地小巧、谨慎和可用。参与设计和与医管局制造商、临床医生和最终用户共同制作将是所有上述工作的核心，指导所有在设计、优先次序和外形因素方面作出的决定。我们强大的用户群，包括Sonova、诺基亚/贝尔实验室、苏格兰聋人和听力损失行动，将有助于最大限度地发挥我们雄心勃勃的研究计划的影响。我们工作的结果将是完全集成的软件和硬件原型，将在一系列现代嘈杂的混响环境中使用听力受损志愿者的听力和可理解性测试进行临床评估。我们雄心勃勃的愿景的成功与否，将取决于我们的示范计划所假定的根本进步将如何在未来十年重塑房委会的格局。

Currently, only 40% of people who could benefit from Hearing Aids (HAs) have them, and most people who have HA devices don't use them often enough. There is social stigma around using visible HAs ('fear of looking old'), they require a lot of conscious effort to concentrate on different sounds and speakers, and only limited use is made of speech enhancement - making the spoken words (which are often the most important aspect of hearing to people) easier to distinguish. It is not enough just to make everything louder!To transform hearing care by 2050, we aim to completely re-think the way HAs are designed. Our transformative approach - for the first time - draws on the cognitive principles of normal hearing. Listeners naturally combine information from both their ears and eyes: we use our eyes to help us hear. We will create "multi-modal" aids which not only amplify sounds but contextually use simultaneously collected information from a range of sensors to improve speech intelligibility. For example, a large amount of information about the words said by a person is conveyed in visual information, in the movements of the speaker's lips, hand gestures, and similar. This is ignored by current commercial HAs and could be fed into the speech enhancement process. We can also use wearable sensors (embedded within the HA itself) to estimate listening effort and its impact on the person, and use this to tell whether the speech enhancement process is actually helping or not.Creating these multi-modal "audio-visual" HAs raises many formidable technical challenges which need to be tackled holistically. Making use of lip movements traditionally requires a video camera filming the speaker, which introduces privacy questions. We can overcome some of these questions by encrypting the data as soon as it is collected, and we will pioneer new approaches for processing and understanding the video data while it stays encrypted. We aim to never access the raw video data, but still to use it as a useful source of information. To complement this, we will also investigate methods for remote lip reading without using a video feed, instead exploring the use of radio signals for remote monitoring. Adding in these new sensors and the processing that is required to make sense of the data produced will place a significant additional power and miniaturization burden on the HA device. We will need to make our sophisticated visual and sound processing algorithms operate with minimum power and minimum delay, and will achieve this by making dedicated hardware implementations, accelerating the key processing steps. In the long term, we aim for all processing to be done in the HA itself - keeping data local to the person for privacy. In the shorter term, some processing will need to be done in the cloud (as it is too power intensive) and we will create new very low latency (<10ms) interfaces to cloud infrastructure to avoid delays between when a word is "seen" being spoken and when it is heard. We also plan to utilize advances in flexible electronics (e-skin) and antenna design to make the overall unit as small, discreet and usable as possible. Participatory design and co-production with HA manufacturers, clinicians and end-users will be central to all of the above, guiding all of the decisions made in terms of design, prioritisation and form factor. Our strong User Group, which includes Sonova, Nokia/Bell Labs, Deaf Scotland and Action on Hearing Loss will serve to maximise the impact of our ambitious research programme. The outcomes of our work will be fully integrated, software and hardware prototypes, that will be clinically evaluated using listening and intelligibility tests with hearing-impaired volunteers in a range of modern noisy reverberant environments. The success of our ambitious vision will be measured in terms of how the fundamental advancements posited by our demonstrator programme will reshape the HA landscape over the next decade.