HCC-Small: Displaying Prosodic Text for Reading Aloud with Expression

朗读是一项复杂的运动、感知、认知和语言技能，需要数年时间才能学会和掌握。文本对于培养读者来说是有问题的，因为标点符号不能可靠地标记短语单位或适当的停顿结构；逗号并不总是需要停顿，问号也不总是需要升调。正在学习这些惯例的年轻读者只能通过反复尝试来解码作者想要的韵律；即使是那些拥有准确解码技能的人，也经常遇到将文本块化为有意义的单元的困难。结果，他们逐字阅读，没有足够的韵律变化，这对他们理解他们大声朗读的内容的能力造成了不利的影响。传统的阅读教学和软件程序强调快速、准确的解码和单词识别；很少或根本没有强调促进富有表现力的、有韵律的口头阅读。然而，韵律线索，如基本频率F0(感知为音调/语调)、强度(感知为响度)和持续时间(感知为长度)，传达了广泛的语言和情感功能，将语音代码与潜在的语义和句法内容联系起来，这对语言理解至关重要。在这个项目中，PI将探索一些创新，以使发展读者朗读表达。她将设计一个交互式阅读界面，显示韵律变化的文本，帮助孩子们流利地朗读，表达得当。韵律目标(F0轮廓、强度包络、单词和停顿持续时间)将从流利的成人读者制作的录音中得出，并使用新颖的半自动声学到图形映射转换成文本处理。该软件将提供与模拟成人生产相对应的听觉和视觉提示；儿童自身生产的近乎实时的视觉和听觉反馈将使自我监测能够进一步支持学习。由此产生的电子媒体将类似于一本儿童书籍，与相应的韵律文本一起显示故事图像，并将包括额外的收听和录音功能。该软件将使用重复测量设计进行评估，在该设计中，32名6-8岁的儿童将阅读带有和不带有韵律文本的年龄和年级水平的适当故事。PI的假设是，提供与潜在韵律目标相关的显性视觉线索将提高口语阅读的流畅性，包括准确性、速度和表现力。附加提示还可以提供脚手架以支持对口语文本的理解。将努力将韵律文本呈现技术扩展到更大的一组口语内容。项目成果将在数字信号处理、语音声学、语音和语言发展、阅读习得、视觉排版和人机交互等领域做出贡献。广泛影响：该项目的最终目标是激励年轻读者通过他们对文本的表达实现使页面上的文字变得生动起来。国际语音学会预计，在这项工作中开发的工具和方法也将适用于改善非母语人士、言语障碍者和学习障碍者的口语韵律。

Reading aloud is a complex motor, perceptual, cognitive and linguistic feat that takes years to learn and master. Text is problematic for developing readers because punctuation does not reliably mark phrase units or appropriate pause structure; commas do not always necessitate a pause, and question marks do not always necessitate rising intonation. Young readers who are learning these conventions are left to decode the author's intended prosody by trial and error; even those who have accurate decoding skills often experience difficulty chunking text into meaningful units. As a result, they read in a word-by-word manner with insufficient prosodic variation, which adversely impacts their ability to comprehend what they have read aloud. Traditional reading instruction and software programs emphasize rapid, accurate decoding and word recognition; little or no emphasis is placed on facilitating expressive, prosodic oral reading. Yet prosodic cues such as fundamental frequency F0 (perceived as pitch/intonation), intensity (perceived as loudness), and duration (perceived as length), convey a wide range of linguistic and affective functions that link the speech code to underlying semantic and syntactic content, which is crucial for language comprehension. In this project, the PI will explore a number of innovations to enable developing readers to read aloud with expression. She will design an interactive reading interface that displays prosodically varying text to help children read aloud fluently with appropriate expression. Prosodic targets (F0 contour, intensity envelop, and word and pause duration) will be derived from recordings made by a fluent adult reader and translated into textual manipulations using novel semi-automated acoustic-to-graphic mappings. The software will provide auditory and visual cues corresponding to the model adult production; near-real time visual and auditory feedback of the child's own production will enable self-monitoring to further support learning. The resulting electronic media will resemble a children's book, displaying a story image along with the corresponding prosodic text, and will include additional listening and recording functions. The software will be assessed using a repeated measures design, in which 32 children aged 6-8 years will read age and grade-level appropriate stories with and without the prosodic text. The PI's hypothesis is that providing explicit visual cues pertaining to the underlying prosodic targets will improve oral reading fluency, including accuracy, rate, and expressiveness. The additional cues may also provide the scaffolding to support comprehension of spoken text. Efforts to scale the prosodic text rendering techniques to a larger set of spoken content will be undertaken. Project outcomes will contribute to the fields of, digital signal processing, speech acoustics, speech and language development, reading acquisition, visual typography, and human-computer interaction.Broader Impacts: The ultimate goal of this project is to inspire young readers to make the words on the page "come alive" through their expressive realization of the text. The PI expects the tools and methodologies developed in this work will also be applicable to improving spoken prosody for non-native speakers, for individuals with speech impairments, and for those with learning disabilities.