SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users
SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users
复制标题
DOI：
10.1145/3373625.3416991
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
D. Jain;Hung Ngo;Pratyush Patel;Steven M. Goodman;Leah Findlater;Jon E. Froehlich
中科院分区：
文献类型：
--
作者：
D. Jain;Hung Ngo;Pratyush Patel;Steven M. Goodman;Leah Findlater;Jon E. Froehlich
智能手表有可能为聋人或听力困难的人提供一目了然的、始终可用的声音反馈。在本文中，我们提出了四种低资源深度学习声音分类模型的性能评估：MobileNet，Inception，ResNet-lite和VGG-lite，跨四种设备架构：watch-only，watch+phone，watch+phone+cloud和watch+cloud。虽然与之前的工作直接比较具有挑战性，但我们的结果表明，最佳模型VGG-lite的性能与非便携式设备的最新技术水平相似，在20个声音类别中的平均准确率为81.2%（SD=5.8%），在三个最高优先级的声音中的平均准确率为97.6%（SD=1.7%）。对于设备架构，我们发现手表+手机架构在CPU、内存、网络使用和分类延迟之间提供了最佳平衡。基于这些实验结果，我们构建并进行了一项基于智能手表的声音感知应用程序的定性实验室评估，该应用程序名为SoundWatch（图1），有8名DHH参与者。定性研究结果显示支持我们的声音感知应用程序，但也发现了错误分类，延迟和隐私问题。最后，我们为未来的可穿戴声音感知技术提供设计考虑。
Smartwatches have the potential to provide glanceable, always-available sound feedback to people who are deaf or hard of hearing. In this paper, we present a performance evaluation of four low-resource deep learning sound classification models: MobileNet, Inception, ResNet-lite, and VGG-lite across four device architectures: watch-only, watch+phone, watch+phone+cloud, and watch+cloud. While direct comparison with prior work is challenging, our results show that the best model, VGG-lite, performed similar to the state of the art for non-portable devices with an average accuracy of 81.2% (SD=5.8%) across 20 sound classes and 97.6% (SD=1.7%) across the three highest-priority sounds. For device architectures, we found that the watch+phone architecture provided the best balance between CPU, memory, network usage, and classification latency. Based on these experimental results, we built and conducted a qualitative lab evaluation of a smartwatch-based sound awareness app, called SoundWatch (Figure 1), with eight DHH participants. Qualitative findings show support for our sound awareness app but also uncover issues with misclassifications, latency, and privacy concerns. We close by offering design considerations for future wearable sound awareness technology.