SignQuery: A Natural User Interface and Search Engine for Sign Languages with Wearable Sensors
SignQuery: A Natural User Interface and Search Engine for Sign Languages with Wearable Sensors
复制标题
DOI：
10.1145/3570361.3613286
复制
发表时间：
2023-10
期刊：
影响因子：
--
通讯作者：
Hao Zhou;Taiting Lu;Kristina Mckinnie;Joseph Palagano;Kenneth DeHaan;Mahanth K. Gowda
中科院分区：
文献类型：
--
作者：
Hao Zhou;Taiting Lu;Kristina Mckinnie;Joseph Palagano;Kenneth DeHaan;Mahanth K. Gowda
搜索引擎，如谷歌，百度，必应已经彻底改变了我们与网络世界的互动方式，在推荐，学习，广告，医疗保健，娱乐等应用程序的数量在本文中，我们设计的搜索引擎，如美国手语（ASL）的手语。手语使用手和身体动作进行交流，具有丰富的语法，复杂性和词汇量，可与口语相媲美。这是聋人社区的主要语言，全球人口为15.5亿。然而，目前还不存在支持原生形式的手语查询的搜索引擎。虽然将手语翻译成口语并使用现有的搜索引擎可能是一种可能性，但这可能会错过关键信息，因为现有的翻译系统要么在词汇量上有限，要么局限于特定的领域。与此相反，本文提出了一个整体的方法，在本地形式的ASL查询以及ASL视频和文本信息在线转换成一个共同的表示空间。这样的联合表示空间提供了一个公共框架，用于精确地表示不同的信息源，并将查询与在线可用的相关信息准确地匹配。我们的系统使用低侵入性的可穿戴传感器捕获的标志查询。为了最大限度地减少训练开销，我们从不同主题的在线ASL视频的大型语料库中获得合成训练数据。通过对一组具有本地ASL流利性的聋人用户进行评估，其准确性与亚马逊，Netflix，Yelp等最先进的推荐系统相当，暗示了系统在真实的世界中的可用性。例如，我们的系统在10点时的重呼率为64.3%，即，在前十名搜索结果中，其中六个与搜索查询相关。此外，该系统对手势模式、方言、传感器位置等的变化是鲁棒的。
Search Engines such as Google, Baidu, and Bing have revolutionized the way we interact with the cyber world with a number of applications in recommendations, learning, advertisements, healthcare, entertainment, etc. In this paper, we design search engines for sign languages such as American Sign Language (ASL). Sign languages use hand and body motion for communication with rich grammar, complexity, and vocabulary that is comparable to spoken languages. This is the primary language for the Deaf community with a global population of ≈ 500 million. However, search engines that support sign language queries in native form do not exist currently. While translating a sign language to a spoken language and using existing search engines might be one possibility, this can miss critical information because existing translation systems are either limited in vocabulary or constrained to a specific domain. In contrast, this paper presents a holistic approach where ASL queries in native form as well as ASL videos and textual information available online are converted into a common representation space. Such a joint representation space provides a common framework for precisely representing different sources of information and accurately matching a query with relevant information that is available online. Our system uses low-intrusive wearable sensors for capturing the sign query. To minimize the training overhead, we obtain synthetic training data from a large corpus of online ASL videos across diverse topics. Evaluated over a set of Deaf users with native ASL fluency, the accuracy is comparable with state-of-the-art recommendation systems for Amazon, Netflix, Yelp, etc., suggesting the usability of the system in the real world. For example, the re-call@10 of our system is 64.3%, i.e., among the top ten search results, six of them are relevant to the search query. Moreover, the system is robust to variations in signing patterns, dialects, sensor positions, etc.