Enhancing Programming and Machine Learning Education for Students with Visual Impairments through the Use of Compilers, AI and Cloud Technologies

对于失明或视觉障碍(BVI)的人来说，有吸引力的高薪和高度灵活的计算机科学职业应该更容易获得。不幸的是，由于两个主要困难，向英属维尔京群岛的学生传授所需的计算机编程和数据科学技能极具挑战性。第一个困难来自于当前屏幕阅读器正确读取由英文字母、数字和标点符号组成的计算机代码的能力有限。编程中使用的专用击键集也不便于屏幕阅读器(例如，空格和制表符)读取。第二个困难来自耗时且令人沮丧的代码导航，BVI的学生必须重复使用屏幕阅读器来阅读每一行，以定位所需的行进行编辑。该项目将与圣安东尼奥盲人和视障人士灯塔合作，开发新的无障碍工具，包括程序语法和语义感知的屏幕阅读器和基于语音命令的代码导航框架，以解决上述两个困难。这些无障碍工具将通过基于云的网络界面提供，以便在全国范围内向学生和教育工作者提供访问。该项目的成功将提高向英属维尔京群岛学生教授计算机编程和数据科学的有效性，这反过来将增加更多英属维尔京群岛学生参与计算科学的机会，获得高薪职业机会，并可能导致计算机科学劳动力更加多样化。这些辅助工具将使用编译器、人工智能(AI)和云技术根据含义读取计算机代码语句，而不是一次只读取一个字符。屏幕阅读器将阐明初级程序员需要的必要信息，并帮助他们更容易地理解计算机编程和数据科学中使用的词汇和语义。基于语音命令的代码导航将使用语音识别和自然语言处理，以便学生能够使用他们的语音在他们的代码中轻松定位特定语句(例如，变量声明)。这些辅助工具将被集成到Jupyter笔记本电脑中，并通过云提供，这将使全国范围内的学生和教育工作者都可以访问。这个基于云的解决方案还将允许使用复杂的人工智能模型，而不需要学生拥有强大而昂贵的计算机来运行这些辅助工具。该项目将使用单案例研究设计对这些可访问性工具进行系统评估，以加深对如何将包括编译器、人工智能和云计算在内的技术应用于向BVI学生教授计算机科学技能的理解。评估还将就不同演讲风格的有效性提供反馈，并为这些无障碍工具的未来改进提供额外反馈。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Attractive high-paying and highly flexible Computer Science careers should be more readily accessible for people with blindness or visual impairments (BVI). Unfortunately, teaching the required computer programming and data science skills to students with BVI is extremely challenging due to two major difficulties. The first difficulty comes from the limited capability of current screen readers to properly read computer codes that are a mix of English letters, digits, and punctuation marks. The specialized set of keystrokes used in programming is also not conveniently read by screen readers (e.g., spaces and tabs). The second difficulty comes from time-consuming and frustrating code navigation, whereby students with BVI must repeatedly use screen readers to read every line to locate the desired line for editing. Partnering with San Antonio Lighthouse for the Blind and Vision Impaired, the project will develop new accessibility tools, including a program syntax- and semantics-aware screen reader and a voice-command-based code navigation framework to address the above two difficulties. These accessibility tools will be offered through cloud-based web interfaces to provide nationwide access to students and educators. The success of this project will improve the effectiveness of teaching computer programming and data science to students with BVI, which in turn will increase accessibility for more individuals with BVI to participate in Computing Science with high-paying career opportunities and could lead to a more-diverse Computer Science workforce. These accessibility tools will use compilers, Artificial Intelligence (AI), and cloud technologies to read computer code statements based on their meanings, rather than only reading one character at a time. The screen reader will articulate the necessary information that beginning coders need and help them more easily understand the lexicon and semantics used in computer programming and data science. The voice-command-based code navigation will employ speech recognition and natural language processing so that students will be able to use their voice to easily locate a specific statement (e.g., a variable declaration) within their code. These accessibility tools will be integrated into Jupyter notebook and offered through the cloud which will give nationwide access to students and educators. This cloud-based solution will also allow sophisticated AI models to be employed without requiring the students to have powerful and expensive computers to run these accessibility tools. The project will conduct a systematic evaluation of these accessibility tools using single-case research design to deepen the understanding of how technologies, including compilers, AI, and cloud computing, can be applied to teaching Computer Science skills to students with BVI. The evaluation will also provide feedback on the effectiveness of different speech styles and provide additional feedback for future improvements of these accessibility tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.