权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Harnessing multimodal data to enhance machine learning of children’s vocalizations

利用多模态数据增强儿童发声的机器学习

基本信息

批准号：
10411575
负责人：
DANIEL S MESSINGER
金额：
$ 20万
依托单位：
UNIVERSITY OF MIAMI CORAL GABLES
依托单位国家：
美国
项目类别：
财政年份：
2021
资助国家：
美国
起止时间：
2021-02-01 至 2026-01-31
项目状态：
未结题

来源：
https://reporter.nih.gov/project-details/10411575
关键词：
Administrative Supplement Adult Affect African American African Caribbean Age Algorithms Archives Asians Benchmarking Cells Characteristics Child Child Development Child Language Child Support Cochlear implant procedure Code Complex Computer Models Computer Vision Systems Computer software Consent Data Data Collection Data Set Databases Detection Development Dimensions Environment Equilibrium Exhibits Exposure to Face Feedback Female Fostering Frequencies Funding Gardenal Gaussian model Goals Hearing Hispanics Hour Human Individual Language Language Development Learning Left Legal patent Life Linguistics Link Location Machine Learning Manuscripts Measurement Measures Metadata Modeling Modernization Movement Nursery Schools Output Parents Participant Pattern Persons Play Positioning Attribute Postdoctoral Fellow Predictive Factor Privacy Probability Procedures Process Production Publications Pythons Radial Radio Randomized Reporting Research Research Personnel Research Project Grants Resources Sampling Series Sex Distribution Shoulder Signal Transduction Social Development Source Speech Speech Sound Statistical Models Stream System TensorFlow Testing Time TimeLine Training United States National Institutes of Health Walkers Weight automated analysis base computerized data processing contextual factors convolutional neural network cost data analysis pipeline data de-identification data integration data pipeline deafness deep learning deep neural network demographics denoising design dyadic interaction educational atmosphere experience feeding functional outcomes hearing impairment improved interactive feedback interest light weight machine learning algorithm male markov model meter multi-ethnic multimodal data multimodality multiple datasets open data open source parent grant peer prediction algorithm prototype radio frequency recurrent neural network recursive neural network repository response sensor signal processing social socioeconomics sound speech processing supervised learning supplemental instruction teacher tool undergraduate student vocalization

项目摘要

Project Summary This Administrative Supplement proposes implementation of a multimodal data pipeline to support machine learning of child language production in complex naturalistic environments. The Supplement builds on the parent R01 (DC018542) that gathers objective, longitudinal data to capture the vocal interactions of children with hearing loss (HL). Even with cochlear implantation, HL is a life-altering condition with high social costs. Inclusion of children with HL and typically hearing (TH) peers in preschool classrooms is a national standard, but it is not clear how early vocal interaction contributes to the language development of children with HL and their TH peers. The parent R01 employs computational models of child location and orientation to indicate when children are in social contact with their peers and teachers. An additional strategy for pursuing the broad goals of the R01— identifying interactive contexts in which children produce phonemically complex vocalizations and interactive speech—is machine learning. Machine learning algorithms can determine the contextual, individual, and interactive factors that predict children’s vocalizations and vocal interactions. However, the parent R01 does not propose machine learning, nor are data disseminated in a format designed to facilitate machine learning. To facilitate machine learning in the classroom, a rigorous diarization process is required to determine speaker identity, which is operationalized as the likelihood that each vocalization was spoken by a given child or teacher. We will integrate audio processing of each target child and teacher’s first-person audio recording with processing of their interactive partners’ recordings. The influence of partner recordings will be determined by their physical distance and orientation relative to the target. This will yield a weighted speaker identification score for each vocalization. For 25% of the sample, the algorithmic score will be compared to speaker identification provided by trained coders to quantify intersystem reliability. Processed datasets will include 7,160 hours of multimodal recordings of child and teacher movement in classrooms synchronized with continuously recorded, child- and teacher-specific (first-person) audio recordings. De-identified output data will characterize vocalizations with respect to algorithmically computed speaker identification probabilities, coder-identified speaker identity (25% of sample), phonemic complexity and audio characteristics (e.g., fundamental frequency), as well as the position and relative orientation of all individuals in the classroom, and child demographics (including characterizations of HL). Over the course of the supplement, output data, Python processing code, and metadata descriptions of the processing pipeline will be disseminated in dedicated distribution portals including Github, Kaggle, and the UCI repository. Recordings will be released to certified investigators via NIH-funded repositories such as Databrary and Homebank.

项目总结