Automated Collaboration Assessment Using Behavioral Analytics

使用行为分析进行自动协作评估

• 批准号: 2016849
• 负责人: Nonye Alozie
• 金额: $ 75万
• 依托单位: SRI International
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2016849&HistoricalAwards=false
• 关键词: Automated; Collaboration; Assessment; Using; Behavioral

The Automated Collaboration Assessment Using Behavioral Analytics project willmeasure and support collaboration as students engage in STEM learning activities.Collaboration promotes clarifications of misconceptions and deeper understanding of conceptsin STEM which prepares students for future employment in STEM and beyond. This projectaligns with the goal of the Cyberlearning for Work at the Human-Technology Frontier program tofund exploratory research that supports learners in working productively in technology-richSTEM environments. Collaboration is an important learning skill in K-12 STEM education, yet teachers have fewconsistent ways to measure and support students’ development in this area. This project willresult in both an improved understanding of productive collaboration and a prototypeinstructional tool that can help teachers identify nonverbal behaviors and assess overallcollaboration and engagement quality. Using nonverbal behaviors to assess engagement willdecrease dependence on discourse and content-based dialogue and increase the transferabilityof this work into different domains. This project is particularly timely as the ability to collaborateand engage in group work are growing requirements in professional and learning settings; at thesame time the very act of collaboration is being disrupted by the Coronavirus pandemic andthere is a high likelihood that much of this “new normal” (social distancing; combining in-personand remote collaboration) will be with us for some time. This project will meet the urgent needcurrently felt by educators and educational institutions to support the development ofcollaboration skills among students, even as the very act of collaboration is shifting and nontraditionalforms of education are taking hold.This project is a collaboration between the Center for Education Research andInnovation (CERI) and Center for Vision Technology (CVT) at SRI International (SRI) and willcapture multiple students’ actions as they work collaboratively face-to-face, both in-person andthrough a virtual platform. This project will use a collaboration conceptualmodel, multistage predictive and explainable machine learning models, and video analytics toassess and report on collaborative behaviors and interactions. The behavior analytics systemwill use facial expressions, body movements, and meta-information about the collaboration taskto identify interactions that show how students contribute to the collaboration, individually andcollectively. This 2-year project will use reliability and model prediction testing and sequential,correlation, and thematic analyses of video recordings, surveys, interviews, and student artifactsto answer the following research questions: Can machine learning models reliably assesscollaboration when compared to human assessments? How do individual behaviors duringcollaboration lead and relate to group level interactions and collaboration quality? and Can wevalidate and relate the assessed collaboration behaviors to student outcomes as represented bygroup-generated artifacts? The intellectual merits include contributions to the advancement oftwo fields: (1) machine learning— by developing and exploring new algorithms that generateexplainable collaboration skill assessments and teacher/student dashboards at different grainsizes of the interactions, and (2) learning sciences—by contributing a collaboration conceptualmodel that shows how specific skills, interactions, and behaviors correspond to collaborationquality at group and individual levels. Broader impacts of this work include increasing theavailability and types of feedback presented to instructors and learners from diversebackgrounds. This will expand the settings and number of individuals who can be evaluated andsupported on collaboration by making collaborative learning easier to monitor through tools thatcan be used by a wide audience of educators and professionals.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

使用行为分析的自动化协作评估项目将在学生参与STEM学习活动时衡量和支持协作。协作促进澄清误解，加深对STEM概念的理解，为学生将来在STEM及其他领域就业做好准备。该项目与Cyberlearning for Work at the Human-Technology Frontier项目的目标一致，该项目旨在资助探索性研究，支持学习者在技术丰富的STEM环境中高效工作。协作是K-12 STEM教育中一项重要的学习技能，但教师很少有一致的方法来衡量和支持学生在这一领域的发展。该项目将提高对生产性协作的理解，并提供一种原型教学工具，可以帮助教师识别非语言行为并评估整体协作和参与质量。使用非语言行为来评估参与将减少对话语和基于内容的对话的依赖，并增加这项工作到不同领域的可转移性。这个项目特别及时，因为在专业和学习环境中，合作和参与小组工作的能力越来越高;与此同时，合作的行为正在被冠状病毒大流行所破坏，这种“新常态”（社交距离;结合面对面和远程协作）很可能会持续一段时间。该项目将满足教育工作者和教育机构目前迫切需要支持学生之间的合作技能的发展，即使合作的行为正在改变，非传统的教育形式正在站稳脚跟。这个项目是教育研究和创新中心（CERI）和SRI国际（SRI）视觉技术中心（CVT）之间的合作。并将捕捉多个学生的行动，因为他们合作面对面，无论是在人或通过一个虚拟平台。该项目将使用协作概念模型、多阶段预测和可解释的机器学习模型以及视频分析来评估和报告协作行为和交互。行为分析系统将使用面部表情、身体动作和关于协作任务的元信息来识别互动，这些互动显示了学生如何单独或集体地为协作做出贡献。这个为期2年的项目将使用可靠性和模型预测测试以及对视频记录，调查，访谈和学生作品的顺序，相关性和主题分析来回答以下研究问题：与人类评估相比，机器学习模型能否可靠地评估协作？协作过程中的个人行为如何引导和关联到团队层面的互动和协作质量？以及我们能否将评估的协作行为与学生的成果（以小组生成的工件为代表）进行评估和关联？知识价值包括对两个领域的进步的贡献：（1）机器学习-通过开发和探索新的算法，以不同的交互粒度生成可解释的协作技能评估和教师/学生仪表板，以及（2）学习科学-通过贡献一个协作概念模型，显示特定的技能，交互，和行为对应于群体和个体水平上的协作质量。这项工作的更广泛的影响包括增加反馈的可用性和类型，提供给来自不同背景的教师和学习者。这将扩大环境和个人谁可以评估和支持的协作，使协作学习更容易通过工具，可以使用广泛的教育工作者和专业人士进行监测。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Towards Explainable Student Group Collaboration Assessment Models Using Temporal Representations of Individual Student Roles

使用个体学生角色的时间表示建立可解释的学生小组协作评估模型

• 发表时间: 2021
• 期刊: Proceedings of the Fourteenth International Conference on Educational Data Mining
• 作者: Som, A.

Exploring the process of group-based collaboration: a validation argument for a collaboration model and observation rubric for training explainable machine learning models.

探索基于小组的协作过程：协作模型的验证论证和用于训练可解释的机器学习模型的观察规则。

• 发表时间: 2022
• 期刊: Proceedings of the 16th International Conference of the Learning Sciences - ICLS 2022
• 作者: Alozie, N.

Investigating the relationship among solution quality, group variability in science confidence, and reciprocal participation in online science collaborative problem-solving tasks.

研究解决方案质量、科学信心的群体差异以及在线科学协作解决问题任务的相互参与之间的关系。

• 发表时间: 2022
• 期刊: Proceedings of the 16th International Conference of the Learning Sciences – ICLS 2022
• 作者: Rachmatullah, A.

Automated Student Group Collaboration Assessment and Recommendation System Using Individual Role and Behavioral Cues

使用个人角色和行为线索的自动化学生小组协作评估和推荐系统

• 发表时间: 2021
• 期刊: Frontiers of Computer Science
• 影响因子: 4.2
• 作者: Som, A.

Collaboration Conceptual Model to Inform the Development of Machine Learning Models Using Behavioral Analytics

协作概念模型为使用行为分析的机器学习模型的开发提供信息

• 发表时间: 2020
• 期刊: Annual meeting program American Educational Research Association
• 作者: Alozie, N.