MRI: Development of Real-time 3D Social Signal Imaging System (SSIS)

该项目代表着朝着能够检测自闭症谱系障碍、精神分裂症和强迫症风险儿童的早期社会行为标记物的计算模型迈出了一步。实时3D社会信号成像系统(SSIS)将被设计成利用每秒数十次产生数十亿像素的摄像头来精确测量社会信号。基础设施将被设计成能够重建凝视、面部、手指、身体和外表的3D几何图形。该系统预计将能够生成海量的多视角视觉数据，以重建高保真3D信号，这是实现社交智能所必需的，可以解码人类表情的每一个细微差别。通过利用大型相机系统，可以通过计算模拟识别细微的社交信号(例如，凝视跟踪)。社会信号成像系统(SSIS)有助于以前所未有的时间和空间分辨率对3D社会信号进行定量测量。这一发展涉及以下步骤：(I)设计分布式视觉计算体系结构以高效地处理多视点视觉数据流；(Ii)构建新的高保真的视觉不变社会信号(凝视、面部、手指、身体、外表)的3D表示；(Iii)创建用于发现行为标记的新的社会信号3D数据集；以及(Iv)开发针对社会信号成像的新的计算机视觉算法(识别、匹配、跟踪、重建)，其在保持准确性的同时最小化计算延迟。该系统提供了微观社会信号的独特特征，使其能够克服现有方法在高危儿童行为评估中的根本局限性。这项工作影响了多个学科，如机器人学、神经科学、心理学、精神病学和医学。结果将通过研讨会、机器学习和技术夏令营以及其他活动，通过来自代表性不足群体的K-12学生进行传播。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project represents a step toward a computational model capable of detecting early social behavioral markers in children at risk for autism spectrum disorder, schizophrenia, and obsessive-compulsive disorder. The real-time 3D Social Signal Imaging System (SSIS) will be designed to precisely measure social signals utilizing cameras producing billions of pixels dozens of times per second. The infrastructure will be designed to enable reconstruction of the 3D geometry of gaze, face, finger, body, and physical appearance. The system is expected to be capable of generating a vast amount of multiple perspective visual data to reconstruct high fidelity 3D signals, needed to enable social intelligence that can decode every nuance of human expression.The ability to discern subtle social signals (e.g., gaze following) can be computationally modeled by leveraging a massive camera system. The Social Signal Imaging System (SSIS) facilitates quantitative measurements of the social signals in 3D at unprecedented temporal and spatial resolutions. This development involves the following steps: (i) Design a distributed visual computing architecture to efficiently process the Multiview visual data streams; (ii) Build a new high-fidelity 3D representation of the view-invariant social signals (gaze, face, finger, body, appearance); (iii) Create a novel 3D dataset of social signals for use in discovering behavioral markers; and (iv) Develop new computer vision algorithms (recognition, matching, tracking, reconstruction) tailored to social signal imaging that minimize computational latency while maintaining accuracy. The system provides a unique characterization of microscopic social signals that enable overcoming fundamental limitations of existing approaches in behavioral assessment of at-risk children. This work impacts diverse disciplines such as robotics, neuroscience, psychology, psychiatry, and medicine. The outcomes will be disseminated through K-12 students from under-represented groups via workshops, machine learning and technology summer camps, and other activities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.