CISE-MSI: DP: HCC: Training a Virtual Guide Dog for Visually Impaired People to Learn Safe Routes Using Crowdsourcing Multimodal Data

CISE-MSI：DP：HCC：使用众包多模式数据为视障人士训练虚拟导盲犬以学习安全路线

• 批准号: 2131186
• 负责人: Hao Tang
• 金额: $ 49.99万
• 依托单位: CUNY Borough of Manhattan Community College
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2131186&HistoricalAwards=false
• 关键词: CISE; MSI; DP; HCC; Training

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Twelve million blind and visually impaired (BVI) people in US face challenges traveling outdoor independently, particularly in a complex urban environment. A guide dog can offer safe and comfortable travel experiences; however, only 5% of BVI people choose dogs mainly due to the cost. Existing mobile navigation apps utilizing GPS and digital maps to provide navigation services, but usually the maps do not offer enough information, such as sidewalk accessibility and dynamic information, to ensure a safe travel experience. To this end, this research studies the challenges and behaviors of BVI people when they travel outdoor independently, using machine learning approaches on crowdsourcing multimodal travel data. The outcome of the research includes a deep understanding of BVI people’s travel behaviors and two assistive mobile apps based on mixed and augmented reality technology: one for personalized trip planning and orientation and mobility training, and another for personalized navigation and travel assistance. The research has broader impact in improving the quality of life of BVI people, and helping local government agencies to better maintain sidewalks. From the educational perspective, the research provides unique training opportunities for students at the City University of New York, including underrepresented populations in STEM at various levels, from undergraduate (both 2-year and 4-year), to master and doctoral students.The project aims to answer the following research questions: 1) How to efficiently survey comprehensive sidewalk data in a complex urban environment? 2) How to identify all travel challenges of BVI users on sidewalks and learn their travel behaviors? 3) How to develop an electronic guide Dog mobile app to provide personalized travel guidance for BVI users with or even without a map service? The project develops a unique user-centric crowdsourcing approach to collect multimodal travel data by BVI people themselves and without manual labeling, and develops multimodal machine learning algorithms to learn the models of semantic sidewalks and travel behaviors of BVI people. The research also develops two novel accessible mobile apps to validate the effectiveness of the above models: a mixed reality-based app for trip planning and realistic orientation & mobility training simulation, for helping BVI people to build a mental map; and an augmented reality-based app using the semantic sidewalk map and the travel behavior models, for safe and comfortable travel assistance.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项全部或部分由《2021年美国救援计划法案》（公法117-2）资助。在美国，1200万盲人和视障人士面临着独立户外旅行的挑战，尤其是在复杂的城市环境中。导盲犬可以提供安全舒适的旅行体验；然而，只有5%的英属维尔京群岛人选择狗，主要是由于成本。现有的移动导航应用程序利用GPS和数字地图提供导航服务，但通常地图没有提供足够的信息，如人行道可达性和动态信息，以确保安全的旅行体验。为此，本研究利用机器学习方法对众包多模式旅行数据进行研究，研究BVI人在独立户外旅行时面临的挑战和行为。研究结果包括对英属维尔京群岛居民旅行行为的深入了解，以及基于混合现实和增强现实技术的两款辅助移动应用程序：一款用于个性化旅行规划和定向和移动培训，另一款用于个性化导航和旅行辅助。这项研究对提高英属维尔京群岛居民的生活质量，帮助当地政府机构更好地维护人行道具有更广泛的影响。从教育的角度来看，该研究为纽约城市大学的学生提供了独特的培训机会，包括从本科生（2年制和4年制）到硕士生和博士生等各级STEM中代表性不足的人群。该项目旨在回答以下研究问题：1)如何在复杂的城市环境中高效地调查综合人行道数据？2)如何识别BVI用户在人行道上的所有出行挑战，并了解他们的出行行为？3)如何开发电子导盲犬手机应用程序，为英属维尔京群岛用户提供个性化的旅游指导，甚至不提供地图服务？该项目开发了一种独特的以用户为中心的众包方法，通过BVI人自己收集多模式旅行数据，无需人工标记，并开发了多模式机器学习算法，以学习BVI人的语义人行道和旅行行为模型。该研究还开发了两个新颖的可访问移动应用程序来验证上述模型的有效性：一个基于混合现实的应用程序，用于旅行计划和现实定位和移动训练模拟，用于帮助英属维尔京群岛人民建立心理地图；还有一个基于增强现实的应用程序，使用语义人行道地图和旅行行为模型，为安全舒适的旅行提供帮助。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Real-Time 3D Object Detection, Recognition and Presentation Using a Mobile Device for Assistive Navigation

• DOI: 10.1007/s42979-023-01881-3
• 发表时间: 2023-07
• 期刊: SN Computer Science
• 作者: Jin Chen;Zhigang Zhu

Improving Building Energy Efficiency through Data Analysis

通过数据分析提高建筑能源效率

• DOI: 10.1145/3599733.3600244
• 发表时间: 2023
• 期刊: The 14th ACM International Conference on Future Energy Systems (e-Energy ’23 Companion
• 作者: Phillip, DiAndra;Chen, Jin;Maksakuli, Fani;Ruci, Arber;Sturdivant, E'edresha;Zhu, Zhigang
• 通讯作者: Zhu, Zhigang

MultiCLU: Multi-stage Context Learning and Utilization for Storefront Accessibility Detection and Evaluation

• DOI: 10.1145/3512527.3531361
• 发表时间: 2022-06
• 期刊: Proceedings of the 2022 International Conference on Multimedia Retrieval
• 作者: X. Wang;Jiajun Chen;Hao Tang;Zhigang Zhu

Context Understanding in Computer Vision: A Survey

• DOI: 10.1016/j.cviu.2023.103646
• 发表时间: 2023-02
• 期刊: ArXiv
• 作者: Xuan Wang;Zhigang Zhu

An Integrated Mobile Vision System for Enhancing the Interaction of Blind and Low Vision Users with Their Surroundings [An Integrated Mobile Vision System for Enhancing the Interaction of Blind and Low Vision Users with Their Surroundings]

用于增强盲人和低视力用户与其周围环境的互动的集成移动视觉系统[用于增强盲人和低视力用户与其周围环境的互动的集成移动视觉系统]

• DOI: 10.5220/0011984400003497
• 发表时间: 2023
• 期刊: Proceedings of the 3rd International Conference on Image Processing and Vision Engineering
• 作者: Chen, Jin;Ramnath, Satesh;Samaroo, Tyron;Maksakuli, Fani;Ruci, Arber;Sturdivant, E’edresha;Zhu, Zhigang
• 通讯作者: Zhu, Zhigang