MRI: Development of Real-time 3D Social Signal Imaging System (SSIS)

这个项目代表了一个计算模型的进步，该模型能够检测自闭症谱系障碍、精神分裂症和强迫症风险儿童的早期社会行为标记。实时3D社会信号成像系统（SSIS）将利用每秒几十次产生数十亿像素的相机，精确测量社会信号。该基础设施将用于重建凝视、面部、手指、身体和物理外观的3D几何形状。该系统有望能够生成大量的多视角视觉数据，以重建高保真度的3D信号，这是实现社会智能所需的，可以解码人类表情的每一个细微差别。辨别微妙的社会信号（例如，目光跟随）的能力可以通过利用大型相机系统进行计算建模。社会信号成像系统（SSIS）以前所未有的时间和空间分辨率促进了三维社会信号的定量测量。这一发展包括以下步骤：(i)设计一个分布式视觉计算架构，以有效地处理多视图视觉数据流；（ii）建立一个新的高保真度的三维图像来表示视点不变的社会信号（凝视、面部、手指、身体、外表）；（三）创建新的社会信号三维数据集，用于发现行为标记；（iv）开发新的计算机视觉算法（识别、匹配、跟踪、重建），为社会信号成像量身定制，在保持准确性的同时最大限度地减少计算延迟。该系统提供了微观社会信号的独特特征，能够克服现有方法在高危儿童行为评估中的基本限制。这项工作影响了不同的学科，如机器人、神经科学、心理学、精神病学和医学。研究成果将通过研讨会、机器学习和技术夏令营以及其他活动，在代表性不足群体的K-12学生中传播。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project represents a step toward a computational model capable of detecting early social behavioral markers in children at risk for autism spectrum disorder, schizophrenia, and obsessive-compulsive disorder. The real-time 3D Social Signal Imaging System (SSIS) will be designed to precisely measure social signals utilizing cameras producing billions of pixels dozens of times per second. The infrastructure will be designed to enable reconstruction of the 3D geometry of gaze, face, finger, body, and physical appearance. The system is expected to be capable of generating a vast amount of multiple perspective visual data to reconstruct high fidelity 3D signals, needed to enable social intelligence that can decode every nuance of human expression.The ability to discern subtle social signals (e.g., gaze following) can be computationally modeled by leveraging a massive camera system. The Social Signal Imaging System (SSIS) facilitates quantitative measurements of the social signals in 3D at unprecedented temporal and spatial resolutions. This development involves the following steps: (i) Design a distributed visual computing architecture to efficiently process the Multiview visual data streams; (ii) Build a new high-fidelity 3D representation of the view-invariant social signals (gaze, face, finger, body, appearance); (iii) Create a novel 3D dataset of social signals for use in discovering behavioral markers; and (iv) Develop new computer vision algorithms (recognition, matching, tracking, reconstruction) tailored to social signal imaging that minimize computational latency while maintaining accuracy. The system provides a unique characterization of microscopic social signals that enable overcoming fundamental limitations of existing approaches in behavioral assessment of at-risk children. This work impacts diverse disciplines such as robotics, neuroscience, psychology, psychiatry, and medicine. The outcomes will be disseminated through K-12 students from under-represented groups via workshops, machine learning and technology summer camps, and other activities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.