CRI II-NEW: IIS: Omniview Multi-modal Sensor Laboratory for Understanding Human Interactions in Ubiquitous Environments

CRI II-新：IIS：用于理解无处不在的环境中人类交互的 Omniview 多模式传感器实验室

• 批准号: 1730183
• 负责人: Sean Banerjee
• 金额: $ 74.69万
• 依托单位: Clarkson University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-06-01 至 2021-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1730183&HistoricalAwards=false
• 关键词: CRI; II; NEW; IIS; Omniview

Understanding how people interact with objects and with each other is an important research area in human-computer interaction, particularly in contexts where phones, cameras, sensors, voice assistants, robots, and other computing devices help people accomplish their goals. To study these interactions, this project will equip a lab with state of the art equipment for capturing human activity that doesn't require attaching markers, wires, or sensors to people, and develop software to manage the large amounts of captured data and interfaces that help researchers and designers make use of the data. Much of the envisioned fundamental research will focus on how age, gender, physical ability, and experience level affect the way people interact with objects, as well as on using social cues such as emotions, inter-person distances, and gestures to understand how people interact with each other. A number of researchers at the lead investigators' institution will use the lab for related projects around people- and object-aware technologies, including assistive robotics, self-driving vehicles, teams and collaboration, and smart environments. The lab will also provide training for a postdoctoral researcher and research opportunities for undergraduates, support a number of courses taught at the PIs' institution, and provide new opportunities for interdisciplinary research.The PIs will develop a temporally synchronized and spatially calibrated sensor system that includes force plates, microphones, RGB and infrared cameras, and Kinect sensors, and deploy it in 20x20x12 foot lab. Data will be synchronized using linear time codes (LTCs), including custom hardware previously developed by the PIs to add LTC data to Kinects, and managed by a large cluster of computers and network attached storage devices. The infrastructure will use computer vision and sound reconstruction algorithms on the raw sensor data to reconstruct 3D spatiotemporal data that provides a full range 3D visualization of human interactions. By combining data from multiple sensor modalities in 3D, the infrastructure enables research in strengthening recognition of gestures and emotions of people engaged in interactions by analysis of prosody changes, face points, body skeletons, spatiotemporal motions, linguistics, heat signatures, and emotion- or task-driven physical impact. Further, the PIs will develop large repositories of omniview multi-modal 3D spatiotemporal data, and conduct research on tying these repositories to sensors on next-generation ubiquitous devices to make them people- and object-aware.

理解人与物体以及人与人之间的交互是人机交互的一个重要研究领域，特别是在手机、相机、传感器、语音助手、机器人和其他计算设备帮助人们实现目标的情况下。 为了研究这些相互作用，该项目将为实验室配备最先进的设备，用于捕获人类活动，而不需要将标记，电线或传感器连接到人身上，并开发软件来管理大量捕获的数据和界面，帮助研究人员和设计师利用这些数据。 大部分设想的基础研究将集中在年龄，性别，身体能力和经验水平如何影响人们与物体的互动方式，以及使用情感，人际距离和手势等社交线索来了解人们如何相互交流。 首席研究员所在机构的许多研究人员将利用该实验室开展与人和物体感知技术相关的项目，包括辅助机器人、自动驾驶汽车、团队和协作以及智能环境。 该实验室还将为博士后研究人员提供培训，为本科生提供研究机会，支持在PI机构教授的一些课程，并为跨学科研究提供新的机会。PI将开发一个时间同步和空间校准的传感器系统，包括测力板，麦克风，RGB和红外摄像头，以及Kinect传感器，并将其部署在20x20x12英尺的实验室中。 数据将使用线性时间码（LTC）进行同步，包括PI先前开发的自定义硬件，以将LTC数据添加到Kinect，并由大型计算机集群和网络连接存储设备进行管理。 该基础设施将在原始传感器数据上使用计算机视觉和声音重建算法来重建3D时空数据，从而提供人类交互的全方位3D可视化。 通过将来自3D中的多个传感器模态的数据相结合，该基础设施能够通过分析韵律变化、面部点、身体骨骼、时空运动、语言学、热特征以及情绪或任务驱动的物理影响来研究加强对参与交互的人的手势和情感的识别。 此外，PI将开发omniview多模态3D时空数据的大型存储库，并研究将这些存储库与下一代无处不在设备上的传感器绑定，使其能够感知人和物体。

项目成果

User-Independent Detection of Swipe Pressure Using a Thermal Camera for Natural Surface Interaction

使用热像仪实现自然表面交互，独立于用户检测滑动压力

• DOI: 10.1109/mmsp.2018.8547052
• 发表时间: 2018
• 期刊: 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP
• 作者: Dunn, Tim;Banerjee, Sean;Banerjee, Natasha Kholgade
• 通讯作者: Banerjee, Natasha Kholgade

Combining Pairwise Feature Matches from Device Trajectories for Biometric Authentication in Virtual Reality Environments

• DOI: 10.1109/aivr46125.2019.00012
• 发表时间: 2019-12
• 期刊: 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)
• 作者: A. Ajit;N. Banerjee;Sean Banerjee

Automatic Material Classification Using Thermal Finger Impression

使用热手指印模进行自动材料分类

• DOI: 10.1007/978-3-030-37731-1_20
• 发表时间: 2020
• 期刊: Multimedia Modeling
• 作者: Gately, Jacob;Liang, Ying;Kolessar Wright, Matthew;Banerjee, Natasha Kholgade;Banerjee, Sean;Dey, Soumyabrata
• 通讯作者: Dey, Soumyabrata

When Will Breakfast be Ready: Temporal Prediction of Food Readiness Using Deep Convolutional Neural Networks on Thermal Videos

早餐何时准备好：在热视频上使用深度卷积神经网络对食物准备情况进行时间预测

• DOI: 10.1109/icmew.2018.8551500
• 发表时间: 2018
• 期刊: 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW
• 作者: Jiang, Yijun;Luo, Miao;Banerjee, Sean;Banerjee, Natasha Kholgade
• 通讯作者: Banerjee, Natasha Kholgade

Programmatic 3D Printing of a Revolving Camera Track to Automatically Capture Dense Images for 3D Scanning of Objects

• DOI: 10.1007/978-3-319-73600-6_39
• 发表时间: 2018-02
• 作者: N. Lamb;N. Banerjee;Sean Banerjee