Enhancing Programming and Machine Learning Education for Students with Visual Impairments through the Use of Compilers, AI and Cloud Technologies

通过使用编译器、人工智能和云技术加强对视力障碍学生的编程和机器学习教育

• 批准号: 2202632
• 负责人: Wei Wang
• 金额: $ 77.09万
• 依托单位: University of Texas at San Antonio
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-15 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2202632&HistoricalAwards=false
• 关键词: Enhancing; Programming; Machine; Learning; Education

Attractive high-paying and highly flexible Computer Science careers should be more readily accessible for people with blindness or visual impairments (BVI). Unfortunately, teaching the required computer programming and data science skills to students with BVI is extremely challenging due to two major difficulties. The first difficulty comes from the limited capability of current screen readers to properly read computer codes that are a mix of English letters, digits, and punctuation marks. The specialized set of keystrokes used in programming is also not conveniently read by screen readers (e.g., spaces and tabs). The second difficulty comes from time-consuming and frustrating code navigation, whereby students with BVI must repeatedly use screen readers to read every line to locate the desired line for editing. Partnering with San Antonio Lighthouse for the Blind and Vision Impaired, the project will develop new accessibility tools, including a program syntax- and semantics-aware screen reader and a voice-command-based code navigation framework to address the above two difficulties. These accessibility tools will be offered through cloud-based web interfaces to provide nationwide access to students and educators. The success of this project will improve the effectiveness of teaching computer programming and data science to students with BVI, which in turn will increase accessibility for more individuals with BVI to participate in Computing Science with high-paying career opportunities and could lead to a more-diverse Computer Science workforce. These accessibility tools will use compilers, Artificial Intelligence (AI), and cloud technologies to read computer code statements based on their meanings, rather than only reading one character at a time. The screen reader will articulate the necessary information that beginning coders need and help them more easily understand the lexicon and semantics used in computer programming and data science. The voice-command-based code navigation will employ speech recognition and natural language processing so that students will be able to use their voice to easily locate a specific statement (e.g., a variable declaration) within their code. These accessibility tools will be integrated into Jupyter notebook and offered through the cloud which will give nationwide access to students and educators. This cloud-based solution will also allow sophisticated AI models to be employed without requiring the students to have powerful and expensive computers to run these accessibility tools. The project will conduct a systematic evaluation of these accessibility tools using single-case research design to deepen the understanding of how technologies, including compilers, AI, and cloud computing, can be applied to teaching Computer Science skills to students with BVI. The evaluation will also provide feedback on the effectiveness of different speech styles and provide additional feedback for future improvements of these accessibility tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对于失明或视觉障碍的人（BVI），应该更容易获得有吸引力的高薪和高灵活的计算机科学职业。不幸的是，由于两个重大困难，向患有BVI的学生讲授所需的计算机编程和数据科学技能非常具有挑战性。第一个困难来自当前屏幕读取器正确读取英语字母，数字和标点符号的混合计算机代码的能力有限。屏幕读取器（例如，空格和标签）也无法方便阅读编程中使用的专门击键集。第二个困难来自耗时和令人沮丧的代码导航，因此，BVI的学生必须反复使用屏幕读取器阅读每一行以找到所需的线路进行编辑。该项目与盲人和视力受损的San Antonio Lighthouse合作，将开发新的可访问性工具，包括程序语法和语义感知的屏幕读取器和基于语音命令的代码导航框架，以解决以上两个困难。这些可访问性工具将通过基于云的Web界面提供，以便在全国范围内访问学生和教育工作者。该项目的成功将提高向BVI的学生教授计算机编程和数据科学的有效性，而BVI的学生将增加BVI患者参与具有高薪职业机会的计算科学的可访问性，并可能导致更多样化的计算机科学劳动力。 这些可访问性工具将使用编译器，人工智能（AI）和云技术根据其含义读取计算机代码语句，而不仅仅是一次阅读一个字符。屏幕读取器将阐明启动编码者需要的必要信息，并更轻松地帮助他们了解计算机编程和数据科学中使用的词典和语义。基于语音命令的代码导航将采用语音识别和自然语言处理，以便学生能够在其代码中轻松地定位特定语句（例如，变量声明）。这些可访问性工具将集成到Jupyter笔记本中，并通过云提供，这将使全国范围的学生和教育工作者访问。这种基于云的解决方案还将允许使用复杂的AI模型，而无需学生拥有强大且昂贵的计算机来运行这些可访问性工具。该项目将使用单盘研究设计对这些可访问性工具进行系统评估，以加深对包括编译器，AI和云计算在内的技术的理解，可用于向BVI学生教授计算机科学技能。该评估还将提供有关不同语音风格的有效性的反馈，并为这些可访问性工具的未来改进提供了其他反馈。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估评估值得支持的。

项目成果

Virtual Summer Camp for High School Students with Disabilities - An Experience Report

残疾高中生虚拟夏令营 - 经验报告

• DOI: 10.1145/3545945.3569818
• 发表时间: 2023
• 期刊: Proceedings of the 54th ACM Technical Symposium on Computer Science Education
• 作者: Wang, Wei;Ewoldt, Kathy B.;Xie, Mimi;Mestas-Nuñez, Alberto M.;Soderman, Sean;Wang, Jeffrey
• 通讯作者: Wang, Jeffrey