CISE-MSI: DP: HCC: Training a Virtual Guide Dog for Visually Impaired People to Learn Safe Routes Using Crowdsourcing Multimodal Data

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。美国有1200万盲人和视障人士（BVI）面临着独立户外旅行的挑战，特别是在复杂的城市环境中。导盲犬可以提供安全舒适的旅行体验;然而，只有5%的BVI人选择狗，主要是因为价格。现有的移动的导航应用程序利用GPS和数字地图来提供导航服务，但通常地图不提供足够的信息，例如人行道可达性和动态信息，以确保安全的旅行体验。为此，本研究研究了BVI人在户外独立旅行时的挑战和行为，使用机器学习方法对众包多式联运旅行数据进行研究。研究成果包括对英属维尔京群岛人的旅行行为的深入了解和两个基于混合和增强现实技术的辅助移动的应用程序：一个用于个性化旅行计划和定向和移动培训，另一个用于个性化导航和旅行辅助。这项研究对改善英属维尔京群岛人民的生活质量和帮助当地政府机构更好地维护人行道产生了更广泛的影响。从教育的角度来看，该研究为纽约的学生提供了独特的培训机会，包括从本科（2年制和4年制）到硕士和博士生的各个层次的STEM代表性不足的人群。该项目旨在回答以下研究问题：1）如何在复杂的城市环境中有效地调查全面的人行道数据？2)如何识别BVI用户在人行道上的所有出行挑战并了解他们的出行行为？3)如何开发一个电子导盲犬移动的应用程序，为BVI用户提供个性化的旅行指导，甚至没有地图服务？该项目开发了一种独特的以用户为中心的众包方法，由BVI人自己收集多式联运旅行数据，无需人工标记，并开发多式联运机器学习算法，以学习BVI人的语义人行道和旅行行为模型。本研究还开发了两个新颖的无障碍移动的应用程序来验证上述模型的有效性：一个混合的基于现实的应用程序，用于旅行规划和现实定向&移动训练模拟，用于帮助BVI人建立心理地图;以及使用语义人行道地图和行进行为模型的基于增强现实的应用，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Twelve million blind and visually impaired (BVI) people in US face challenges traveling outdoor independently, particularly in a complex urban environment. A guide dog can offer safe and comfortable travel experiences; however, only 5% of BVI people choose dogs mainly due to the cost. Existing mobile navigation apps utilizing GPS and digital maps to provide navigation services, but usually the maps do not offer enough information, such as sidewalk accessibility and dynamic information, to ensure a safe travel experience. To this end, this research studies the challenges and behaviors of BVI people when they travel outdoor independently, using machine learning approaches on crowdsourcing multimodal travel data. The outcome of the research includes a deep understanding of BVI people’s travel behaviors and two assistive mobile apps based on mixed and augmented reality technology: one for personalized trip planning and orientation and mobility training, and another for personalized navigation and travel assistance. The research has broader impact in improving the quality of life of BVI people, and helping local government agencies to better maintain sidewalks. From the educational perspective, the research provides unique training opportunities for students at the City University of New York, including underrepresented populations in STEM at various levels, from undergraduate (both 2-year and 4-year), to master and doctoral students.The project aims to answer the following research questions: 1) How to efficiently survey comprehensive sidewalk data in a complex urban environment? 2) How to identify all travel challenges of BVI users on sidewalks and learn their travel behaviors? 3) How to develop an electronic guide Dog mobile app to provide personalized travel guidance for BVI users with or even without a map service? The project develops a unique user-centric crowdsourcing approach to collect multimodal travel data by BVI people themselves and without manual labeling, and develops multimodal machine learning algorithms to learn the models of semantic sidewalks and travel behaviors of BVI people. The research also develops two novel accessible mobile apps to validate the effectiveness of the above models: a mixed reality-based app for trip planning and realistic orientation & mobility training simulation, for helping BVI people to build a mental map; and an augmented reality-based app using the semantic sidewalk map and the travel behavior models, for safe and comfortable travel assistance.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.