HCC: Small: Programming by Voice: Extending Initial Programming Environments for Children with Disabilities

PI在这项研究中的目标是赋予患有某些残疾的儿童权力，使他们能够充分参与用于教授计算机科学的初始编程环境（IPEs）。具体来说，PI将调查科学和必要的工具构建，以支持语音功能的IPEs改编，如Scratch， Lego LabVIEW for Mindstorms和Alice，这些最初是为用键盘和鼠标手动输入而设计的，以便允许肢体使用有限的儿童通过替代界面进行交互。上述ipe传统上依赖于涉及窗口、图标和其他图形小部件的用户界面，并且需要一种程序输入模式，这可能对上肢运动障碍的人构成障碍，他们可能缺乏用手控制鼠标或键盘所需的灵活性和移动性。PI的方法将是模仿常见的鼠标和键盘交互，使用为每个IPE定制的语音驱动界面。为此，PI将开发一个语音感知应用程序，该应用程序与IPE并行运行，听取来自用户的语音命令，根据受IPE概念影响的语法解释命令，并在IPE中模仿类似鼠标和键盘操作的适当动作。核心研究问题将包括如何使用逆向工程和模型驱动工程将这种辅助定制添加到自动化中。在项目的第一年，PI将扩展先前开发的概念验证，以覆盖整个Scratch界面。结果将是一个强大的工具，可以在Scratch中实现语音编程，并将作为残疾儿童目标群体的评估工具。从项目第一阶段获得的经验教训将推动为现有应用程序定制语音接口所需步骤的一般化。将研究涉及屏幕抓取和逆向工程以及模型驱动工程的技术，以使IPEs适应语音编程的过程自动化。由此产生的工具将应用于一个新的IPE， Lego LabVIEW for Mindstorms，允许残疾儿童编写机器人程序。项目的设计和评估将与伯明翰脑瘫联合组织合作进行，该组织将招募项目参与者，并为项目设计的培训、评估和反馈提供资源。更广泛的影响：该项目将人机交互与计算机科学教育的理念结合起来，为残疾儿童（针对6-12年级）的计算思维教学提供定制的辅助环境。这项工作将提高我们支持语音编程的软件开发环境自动化生成的能力，从而使上肢运动障碍的儿童能够充分参与计算机科学教育机会。PI将包括研究生参与研究；他将指导本科生荣誉项目，他还将指导来自代表性不足群体的高中生进行与本工作相关的科学展览项目。研究结果将通过一个项目网页传播，其中将包括开放来源软件、教材、录象示范和出版物。

The PI's objective in this research is to empower children with certain kinds of disabilities so they can participate fully in initial programming environments (IPEs) that are used to teach computer science. Specifically, the PI will investigate the science and necessary tool construction to support speech-enabled adaptation of IPEs such as Scratch, Lego LabVIEW for Mindstorms, and Alice, which were originally designed for manual input with a keyboard and mouse, in order to allow children with limited use of their limbs to interact via alternative interfaces. The aforementioned IPEs traditionally rely on user interfaces involving windows, icons, and other graphical widgets, and require a mode of program input that can pose a barrier to those with upper limb motor impairments, who may lack the dexterity and mobility needed to control a mouse or keyboard with their hands. The PI's approach will be to imitate the common mouse and keyboard interactions with a voice-driven interface that is customized for each IPE. To these ends, the PI will develop a speech-aware application that runs in parallel to the IPE, listens to the voice commands from the user, interprets the commands according to a grammar influenced by the IPE concepts, and imitates appropriate actions similar to mouse and keyboard operation within the IPE. Core research questions will include how such assistive customizations can be added with automation using reverse engineering and model-driven engineering.During the first year of the project, the PI will extended a previously developed proof-of-concept to cover the entire Scratch interface. The result will be a robust tool that enables Programming by Voice in Scratch, and which will serve as the evaluation instrument for a target group of children with disabilities. The lessons learned from the first phase of the project will drive a generalization of the steps needed to customize a speech interface for an existing application. Techniques involving screen scraping and reverse engineering, as well as model-driven engineering, will be investigated to automate the process of adapting IPEs to Programming by Voice. The resulting tools will be applied to a new IPE, the Lego LabVIEW for Mindstorms, to allow children with disabilities to program robots. The design and evaluation of the project will be performed in collaboration with United Cerebral Palsy of Birmingham, who will recruit participants into the project and provide resources for training, evaluation, and feedback on the project design.Broader Impacts: This project unites ideas of human-computer interaction with computer science education to provide customized assistive environments for teaching computational thinking to children with disabilities (targeting grades 6-12). The work will advance our ability to automate the generation of software development environments that support Programming by Voice, resulting in advanced capabilities to enable children with upper limb motor impairments to participate fully in computer science education opportunities. The PI will involve graduate students in the research; he will supervision undergraduate Honors projects, and he will also mentor high school students from underrepresented groups with science fair projects related to this work. The results of the research will be disseminated through a project web page that will include open source software, teaching materials, video demonstrations and publications.