Harnessing multimodal data to enhance machine learning of children’s vocalizations

利用多模态数据增强儿童发声的机器学习

• 批准号: 10411575
• 负责人: DANIEL S MESSINGER
• 金额: $ 20万
• 依托单位: UNIVERSITY OF MIAMI CORAL GABLES
• 依托单位国家: 美国
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-02-01 至 2026-01-31
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10411575
• 关键词: Administrative Supplement; Adult; Affect; African American; African Caribbean; Age; Algorithms; Archives; Asians; Benchmarking; Cells; Characteristics; Child; Child Development; Child Language; Child Support; Cochlear implant procedure; Code; Complex; Computer Models; Computer Vision Systems; Computer software; Consent; Data; Data Collection; Data Set; Databases; Detection; Development; Dimensions; Environment; Equilibrium; Exhibits; Exposure to; Face; Feedback; Female; Fostering; Frequencies; Funding; Gardenal; Gaussian model; Goals; Hearing; Hispanics; Hour; Human; Individual; Language; Language Development; Learning; Left; Legal patent; Life; Linguistics; Link; Location; Machine Learning; Manuscripts; Measurement; Measures; Metadata; Modeling; Modernization; Movement; Nursery Schools; Output; Parents; Participant; Pattern; Persons; Play; Positioning Attribute; Postdoctoral Fellow; Predictive Factor; Privacy; Probability; Procedures; Process; Production; Publications; Pythons; Radial; Radio; Randomized; Reporting; Research; Research Personnel; Research Project Grants; Resources; Sampling; Series; Sex Distribution; Shoulder; Signal Transduction; Social Development; Source; Speech; Speech Sound; Statistical Models; Stream; System; TensorFlow; Testing; Time; TimeLine; Training; United States National Institutes of Health; Walkers; Weight; automated analysis; base; computerized data processing; contextual factors; convolutional neural network; cost; data analysis pipeline; data de-identification; data integration; data pipeline; deafness; deep learning; deep neural network; demographics; denoising; design; dyadic interaction; educational atmosphere; experience; feeding; functional outcomes; hearing impairment; improved; interactive feedback; interest; light weight; machine learning algorithm; male; markov model; meter; multi-ethnic; multimodal data; multimodality; multiple datasets; open data; open source; parent grant; peer; prediction algorithm; prototype; radio frequency; recurrent neural network; recursive neural network; repository; response; sensor; signal processing; social; socioeconomics; sound; speech processing; supervised learning; supplemental instruction; teacher; tool; undergraduate student; vocalization

Project Summary This Administrative Supplement proposes implementation of a multimodal data pipeline to support machine learning of child language production in complex naturalistic environments. The Supplement builds on the parent R01 (DC018542) that gathers objective, longitudinal data to capture the vocal interactions of children with hearing loss (HL). Even with cochlear implantation, HL is a life-altering condition with high social costs. Inclusion of children with HL and typically hearing (TH) peers in preschool classrooms is a national standard, but it is not clear how early vocal interaction contributes to the language development of children with HL and their TH peers. The parent R01 employs computational models of child location and orientation to indicate when children are in social contact with their peers and teachers. An additional strategy for pursuing the broad goals of the R01— identifying interactive contexts in which children produce phonemically complex vocalizations and interactive speech—is machine learning. Machine learning algorithms can determine the contextual, individual, and interactive factors that predict children’s vocalizations and vocal interactions. However, the parent R01 does not propose machine learning, nor are data disseminated in a format designed to facilitate machine learning. To facilitate machine learning in the classroom, a rigorous diarization process is required to determine speaker identity, which is operationalized as the likelihood that each vocalization was spoken by a given child or teacher. We will integrate audio processing of each target child and teacher’s first-person audio recording with processing of their interactive partners’ recordings. The influence of partner recordings will be determined by their physical distance and orientation relative to the target. This will yield a weighted speaker identification score for each vocalization. For 25% of the sample, the algorithmic score will be compared to speaker identification provided by trained coders to quantify intersystem reliability. Processed datasets will include 7,160 hours of multimodal recordings of child and teacher movement in classrooms synchronized with continuously recorded, child- and teacher-specific (first-person) audio recordings. De-identified output data will characterize vocalizations with respect to algorithmically computed speaker identification probabilities, coder-identified speaker identity (25% of sample), phonemic complexity and audio characteristics (e.g., fundamental frequency), as well as the position and relative orientation of all individuals in the classroom, and child demographics (including characterizations of HL). Over the course of the supplement, output data, Python processing code, and metadata descriptions of the processing pipeline will be disseminated in dedicated distribution portals including Github, Kaggle, and the UCI repository. Recordings will be released to certified investigators via NIH-funded repositories such as Databrary and Homebank.

项目概要 本行政补充建议实施多模式数据管道以支持 复杂自然环境中儿童语言产生的机器学习。补充品 以父 R01 (DC018542) 为基础，收集客观的纵向数据来捕捉声音 听力损失儿童（HL）的互动。即使植入了人工耳蜗，HL 也能改变生活 社会成本高的状况。将 HL 儿童和正常听力 (TH) 的同龄人纳入 学前班教室是国家标准，但尚不清楚早期的声音互动如何有助于 HL 儿童及其 TH 同龄人的语言发展。母公司 R01 雇用 儿童位置和方向的计算模型，以指示儿童何时进行社交接触 与他们的同龄人和老师。追求 R01 广泛目标的附加策略—— 识别儿童发出复杂音位发声的互动环境， 交互式语音——就是机器学习。机器学习算法可以确定上下文， 预测儿童发声和声音互动的个体和互动因素。然而， 母版 R01 并未提出机器学习，也没有以旨在促进机器学习的格式传播数据。 促进机器学习。为了促进课堂上的机器学习，严格的分类 需要确定说话者身份的过程，该过程可操作为每个说话者的可能性 发声是由特定的孩子或老师说出的。我们将整合每个目标的音频处理 孩子和老师的第一人称录音以及对其互动伙伴录音的处理。 伙伴录音的影响将取决于他们的物理距离和相对方向 到目标。这将为每个发声产生加权的说话人识别分数。对于 25% 的 样本中，算法分数将与训练有素的编码员提供的说话人识别进行比较，以 量化系统间的可靠性。处理后的数据集将包括 7,160 小时的多模式记录 教室中儿童和教师的活动与持续记录的同步，儿童和教师的活动 教师特定（第一人称）录音。去识别化的输出数据将表征发声 关于算法计算的说话人识别概率，编码器识别的说话人 身份（样本的 25%）、音素复杂性和音频特征（例如基频）， 以及教室中所有个人和儿童的位置和相对方向 人口统计（包括 HL 的特征）。在补充的过程中，输出数据， Python处理代码和处理管道的元数据描述将在 专用的分发门户，包括 Github、Kaggle 和 UCI 存储库。录音将是 通过 NIH 资助的存储库（例如 Databrary 和 Homebank）向经过认证的研究人员发布。