PFI:BIC: Multimodal-Sensor-Enabled Environments with Advanced Cognitive Computing Enabling Smart Group Meeting Facilitation Services.

PFI：BIC：具有先进认知计算的多模式传感器支持环境，可实现智能团体会议便利服务。

• 批准号: 1631674
• 负责人: Richard Radke
• 金额: $ 100万
• 依托单位: Rensselaer Polytechnic Institute
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1631674&HistoricalAwards=false
• 关键词: PFI; BIC; Multimodal; Sensor; Enabled

Millions of meetings take place every day in the US, incurring a tremendous cost in terms of managers' and employees' precious time and salary. Unfortunately, group meetings suffer from serious problems that undermine productivity and collegiality, including overt or unconscious bias, "groupthink", fear of speaking, and unfocused discussion. Few automatic tools exist for keeping meetings on track, accurately recording who said what, and making group discussions more productive. The goal of this research is to design intelligent rooms that provide facilitation services by identifying meeting participants, understand their conversations, summarize discussions, and help the group efficiently get through an agenda, all without requiring the participants to wear microphones or other sensors. The research will have broader impacts in several aspects. Any steps to make group meetings for complex, long-term projects more productive and easier to control would result in immediate economic impact. The success of a service system that facilitates long-term group interactions will result in a major opportunity for technology transfer and a highly marketable hardware/software platform for collaboration in domains including business, education, and finance. The project will result in new group meeting data to be used by researchers in different fields such as organizational psychologists, and computer scientists. Finally, the project will produce highly visible infrastructure for research and education that has the potential for greater public engagement with science and technology.The research will be realized at different scales in two existing physical testbeds: the Smart Conference Room (SCR) at the Engineering Research Center for Lighting Enabled Systems and Applications (LESA) and the Collaborative Research Augmented Immersive Virtual Environment Lab (CRAIVE-Lab), a much larger space with a tall, panoramic screen. Both testbeds will be expanded as part of this project to integrate sensing and computing technologies developed by the partners. The major technology modules include: (1) advanced time-of-flight sensors for robust occupant tracking, resulting in centimeter-accurate, real-time locations of all participants in the meeting without requiring video cameras or wearable sensors; (2) custom beamforming microphone arrays for acoustical tracking and sound source separation, allowing highly-directional beams to be directed at each participant's instantaneous location, clearly isolating their speech without requiring lapel or headset microphones; (3) natural language understanding algorithms for extracting knowledge from the speech, enabling the system to learn meeting-specific terminology, link concepts, assess speaker roles, and summarize discussion; and (4) cognitive computing tools for meeting facilitation, including assessments of participation and productivity, ideation interventions for brainstorming, and active decision support. The resulting service systems will be cognitive physical environments that understand their occupants' locations, movement, speech, vocabulary, and intentions. A key aspect of the research is a multi-year study that tracks technical research groups that hold regular, unscripted meetings in the testbeds, assessing the effectiveness of the service system. The lead institution for the project is Rensselaer Polytechnic Institute, with investigators from the departments of Electrical, Computer, and Systems Engineering, Computer Science, and Architecture. The industrial partners in the effort are IBM Research (large business, Yorktown Heights, NY) and Heptagon Advanced MicroOptics (large business, Zurich, Switzerland).This award is partially supported by funds from the Directorate for Computer and Information Science and Engineering (CISE), Divisions of Information and Intelligent Systems (IIS) and Computer and Network Systems (CNS).

在美国，每天都有数百万场会议，这给管理者和员工的宝贵时间和工资带来了巨大的成本。 不幸的是，小组会议存在严重的问题，这些问题破坏了生产力和合作精神，包括公开或无意识的偏见，“群体思维”，害怕发言和没有重点的讨论。 很少有自动工具可以让会议保持正轨，准确记录谁说了什么，并使小组讨论更有成效。 这项研究的目标是设计智能会议室，通过识别会议参与者，理解他们的对话，总结讨论，并帮助小组有效地完成议程，而无需参与者佩戴麦克风或其他传感器。 该研究将在几个方面产生更广泛的影响。 任何使复杂、长期项目的小组会议更有成效、更容易控制的措施都将产生立竿见影的经济影响。 促进长期群体互动的服务系统的成功将为技术转让和商业、教育和金融等领域的合作提供高度适销对路的硬件/软件平台带来重大机会。 该项目将产生新的小组会议数据，供组织心理学家和计算机科学家等不同领域的研究人员使用。 最后，该项目将为研究和教育提供高度可见的基础设施，这些基础设施有可能使公众更多地参与科学和技术。研究将在两个现有的物理试验台上以不同的规模实现：照明系统与应用工程研究中心（LESA）的智能会议室（SCR）和协同研究增强沉浸式虚拟环境实验室（CRAIVE-Lab），一个更大的空间，一个高大的全景屏幕。 作为该项目的一部分，这两个试验台将得到扩展，以整合合作伙伴开发的传感和计算技术。 主要的技术模块包括：（1）先进的飞行时间传感器，用于强大的乘员跟踪，从而在不需要摄像机或可穿戴传感器的情况下实现会议中所有参与者的厘米级精确的实时位置;（2）用于声学跟踪和声源分离的定制波束形成麦克风阵列，允许高定向波束指向每个参与者的瞬时位置，（3）自然语言理解算法，用于从语音中提取知识，使系统能够学习会议专用术语，链接概念，评估发言者角色，并总结讨论;以及（4）用于会议便利的认知计算工具，包括对参与和生产力的评估、用于头脑风暴的构思干预以及积极的决策支持。 由此产生的服务系统将是认知物理环境，可以理解其居住者的位置，运动，语音，词汇和意图。这项研究的一个关键方面是一项为期多年的研究，该研究跟踪了在测试平台上定期举行无脚本会议的技术研究小组，评估了服务系统的有效性。 该项目的牵头机构是伦斯勒理工学院，研究人员来自电气，计算机和系统工程，计算机科学和建筑系。 这项工作的工业合作伙伴是IBM研究院（大企业，约克敦高地，纽约）和Heptagon高级微光学公司（大企业，苏黎世，瑞士）。该奖项部分由计算机和信息科学与工程局（CISE）、信息和智能系统司（IIS）和计算机和网络系统司（CNS）提供资金支持。

项目成果

Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization

• DOI: 10.18653/v1/p19-1210
• 发表时间: 2019-07
• 作者: Manling Li;Lingyu Zhang;Heng Ji;R. Radke

Privacy-Preserving Understanding of Human Body Orientation for Smart Meetings

智能会议中对人体朝向的隐私保护理解

• 发表时间: 2017
• 期刊: 13th IEEE Workshop on Perception Beyond the Visible Spectrum (in conjunction with CVPR 2017
• 作者: Bhattacharya, I;Eshed, N.;Radke, R.J.
• 通讯作者: Radke, R.J.

The unobtrusive group interaction (UGI) corpus

不显眼的群体互动（UGI）语料库

• DOI: 10.1145/3304109.3325816
• 发表时间: 2019
• 期刊: Proceedings of the 10th ACM Multimedia Systems Conference
• 作者: Bhattacharya, Indrani;Foley, Michael;Ku, Christine;Zhang, Ni;Zhang, Tongtao;Mine, Cameron;Li, Manling;Ji, Heng;Riedl, Christoph;Welles, Brooke Foucault
• 通讯作者: Welles, Brooke Foucault

Temporal Attention and Consistency Measuring for Video Question Answering

• DOI: 10.1145/3382507.3418886
• 发表时间: 2020-10
• 期刊: Proceedings of the 2020 International Conference on Multimodal Interaction
• 作者: Lingyu Zhang;R. Radke

Multiparty Visual Co-Occurrences for Estimating Personality Traits in Group Meetings

• DOI: 10.1109/wacv45572.2020.9093642
• 发表时间: 2020-03
• 期刊: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV)
• 作者: Lingyu Zhang;Indrani Bhattacharya;Mallory M. Morgan;Michael Foley;Christoph Riedl;B. F. Welles;R. Radke