HCC-Small: Displaying Prosodic Text for Reading Aloud with Expression

点击翻译按钮获取中文摘要

Reading aloud is a complex motor, perceptual, cognitive and linguistic feat that takes years to learn and master. Text is problematic for developing readers because punctuation does not reliably mark phrase units or appropriate pause structure; commas do not always necessitate a pause, and question marks do not always necessitate rising intonation. Young readers who are learning these conventions are left to decode the author's intended prosody by trial and error; even those who have accurate decoding skills often experience difficulty chunking text into meaningful units. As a result, they read in a word-by-word manner with insufficient prosodic variation, which adversely impacts their ability to comprehend what they have read aloud. Traditional reading instruction and software programs emphasize rapid, accurate decoding and word recognition; little or no emphasis is placed on facilitating expressive, prosodic oral reading. Yet prosodic cues such as fundamental frequency F0 (perceived as pitch/intonation), intensity (perceived as loudness), and duration (perceived as length), convey a wide range of linguistic and affective functions that link the speech code to underlying semantic and syntactic content, which is crucial for language comprehension. In this project, the PI will explore a number of innovations to enable developing readers to read aloud with expression. She will design an interactive reading interface that displays prosodically varying text to help children read aloud fluently with appropriate expression. Prosodic targets (F0 contour, intensity envelop, and word and pause duration) will be derived from recordings made by a fluent adult reader and translated into textual manipulations using novel semi-automated acoustic-to-graphic mappings. The software will provide auditory and visual cues corresponding to the model adult production; near-real time visual and auditory feedback of the child's own production will enable self-monitoring to further support learning. The resulting electronic media will resemble a children's book, displaying a story image along with the corresponding prosodic text, and will include additional listening and recording functions. The software will be assessed using a repeated measures design, in which 32 children aged 6-8 years will read age and grade-level appropriate stories with and without the prosodic text. The PI's hypothesis is that providing explicit visual cues pertaining to the underlying prosodic targets will improve oral reading fluency, including accuracy, rate, and expressiveness. The additional cues may also provide the scaffolding to support comprehension of spoken text. Efforts to scale the prosodic text rendering techniques to a larger set of spoken content will be undertaken. Project outcomes will contribute to the fields of, digital signal processing, speech acoustics, speech and language development, reading acquisition, visual typography, and human-computer interaction.Broader Impacts: The ultimate goal of this project is to inspire young readers to make the words on the page "come alive" through their expressive realization of the text. The PI expects the tools and methodologies developed in this work will also be applicable to improving spoken prosody for non-native speakers, for individuals with speech impairments, and for those with learning disabilities.