The Comprehensive Assessment Of Team Member Effectiveness: A New Peer Evaluation Instrument
The Comprehensive Assessment Of Team Member Effectiveness: A New Peer Evaluation Instrument
复制标题
团队成员效能的综合评估：一种新的同行评估工具
DOI：
10.18260/1-2--751
复制
发表时间：
2006
影响因子：
1.3
通讯作者：
Harlan W. Feinstein
中科院分区：
文献类型：
--
作者：
M. Ohland;H. Pomeranz;Harlan W. Feinstein
一个由多所大学组成的研究小组设计了一种简单易用的同行评估工具。该系统通过网络界面收集学生的数据，确保了同行评分的保密性。系统分析数据以计算建议的成绩调整，以便在团队成员之间公平分配团队的成绩。该系统还为教师提供广泛的反馈，以了解学生团队的某些动态，这些动态可以从同行评估数据中辨别出来。目前正在进行可靠性和有效性研究。本文包括行为锚定评分量表的描述，电子界面，以及它提供的反馈。新仪器的设计该仪器是在广泛的团队研究基础上开发的。从团队合作文献中识别潜在项目，创建李克特量表工具，以及在大型调查样本上使用探索性因子分析和验证性因子来减少工具和识别因素结构，在其他地方也有报道。1 .我们早期发表的作品描述了在工程课堂中评估团队合作的重要性及其所代表的挑战，并制定了雄心勃勃的评估计划，该计划将有助于开发一种易于使用且对教师和学生都有意义的工具；2 .描述并展示了行为锚定评分系统的好处；3 .详细介绍了创建新的行为锚定评分量表的过程，以简化管理、数据分析、和报告，让反馈更容易理解。本文展示了如何将行为锚定评分量表纳入电子界面，如何设计数据库，以及它提供的反馈。为了使学生和教师的同行评估变得简单，数据库模式变得相当复杂，总结为20页的功能需求文档。该系统包括管理员、教师和学生的视图，并具有密码保护、同意和报告模块。该工具被称为团队成员效能综合评估，简称CATME。管理员界面：管理员在确认来自教员的请求后授予对系统的访问权限。界面显示活动和待处理的教师帐户，跟踪每个教师的最后登录日期和时间。这个界面还提供了访问原始数据的调查发布为研究目的。教员界面：参与的教员使用系统输入有关班级的信息，用学生填充班级，在班级中填充学生团队，并为团队活动设置调查。教师界面还控制给学生的指示，调查的因素，是否需要同意，以及将报告哪些数据。教师从之前公布的五个主要调查因素中进行选择（对团队工作的贡献，与队友的互动，保持团队的正轨，期望质量，以及拥有与任务相关的知识/技能/能力），此外还有可选的后续问题。任何级别的教员输入的任何内容都可以被选择以供将来使用——例如，如果团队在同行评估工具的多个管理中保持不变，则团队可以在系统中重用。系统支持上传常用的文本文件格式，方便大规模数据录入。该系统还具有自动化功能，例如当调查仪器打开时通知学生。学生界面：使用CATME同行评估的学生在他们第一次被分配到系统中时都会收到一封电子邮件，因此他们可以用他们可以记住的密码设置系统并使系统安全。学生登录后可以找到一个正在进行的调查列表，他们被要求完成调查，另一个列表是已完成的调查，学生可以在其中查看他们的结果。如果学生没有完成整个调查，界面会记住学生离开的地方以及他们已经输入的数据。当学生们回来时，他们从第一组未回答的问题开始。由于分配了团队，界面会向每个学生显示正在评估的队友列表。这张海报将展示每个界面的屏幕截图以及现场演示。正在设计一个演示模式，特别是让教师评估系统看到学生的界面。下面显示了一个示例界面截图，说明了调查管理产生的数字评分，推荐的分数调整因素（如果教师需要），以及提醒教师注意团队中可能发生的动态的特别注意事项。
A multi-university research team has designed a peer evaluation instrument that is simple to use. The system gathers data from students through a web interface, ensuring the confidentiality of the peer ratings. The system analyzes the data to calculate suggested grade adjustments for equitably distributing a team’s grade among the team’s members. The system also provides extensive feedback to faculty as to certain dynamics of student teams that can be discerned from the peer evaluation data. Reliability and validity studies are under way. This paper includes a description of the behaviorally anchored rating scales, the electronic interface, and the feedback it provides. The design of a new instrument The instrument was developed from a broad base of teamwork research. The identification of potential items from the teamwork literature, the creation of a Likert-scale instrument, and the use of exploratory factor analysis and confirmatory factor on a large survey sample to reduce the instrument and identify the factor structure are in press elsewhere. 1 Our earlier published work described the importance of assessing teamwork in the engineering classroom and the challenges it represents and laid out the ambitious assessment plan that would help develop an instrument that is easy to use and yet meaningful for both faculty and students, 2 described and demonstrated the benefit of a behaviorally anchored rating system, 3 detailed the process of creating a new behaviorally anchored rating scale to simplify administration, data analysis, and reporting, and make feedback more understandable. 4 This paper shows how the behaviorally anchored rating scales are incorporated into an electronic interface, how the database is designed, and the feedback it provides. Designing a complicated database to keep the administration and reporting simple To make the peer evaluation simple for students and faculty, the database schema became quite complicated, summarized in a 20-page functional requirements document. The system includes views for administrators, faculty, and students, and has modules for password protection and consent and reporting. The instrument is called the Comprehensive Assessment of Team Member Effectiveness, referred to as CATME for short. Administrator interface: The administrator grants access to the system after confirming that the request has come from a faculty member. The interface shows active and pending faculty accounts, keeps track of each faculty’s last login date and time. This interface also provides access to the raw data for surveys released for research purposes. Faculty interface: Participating faculty enter information about classes using the system, populate a class with students, populate teams of students within a class, and set up surveys for team activities. The faculty interface also controls the instructions given to the students, the P ge 11262.2 factors surveyed, whether consent is required, and what data will be reported. Faculty select from among five primary survey factors as published earlier (Contributing to the Team's Work, Interacting with Teammates, Keeping Team on Track, Expecting Quality, and Having TaskRelated Knowledge/Skills/Abilities) in addition to optional follow-up questions. Anything entered by a faculty member at any level can be selected for future use—for example, if teams remain constant for multiple administrations of the peer evaluation instrument, the teams may be reused within the system. The system supports uploading common text file formats to facilitate large-scale data entry. The system also has automated features, such as notifying students when a survey instrument has opened. Student interface: Students using the CATME peer evaluation each receive an email when they are first assigned in the system, so they can set up the system with a password they can remember and make the system secure. A student logs in to find a list of active surveys that they are being requested to complete and a second list of completed surveys where the students can view their results. If a student doesn't complete the entire survey, the interface remembers where the student left off and what data they’ve already entered. When the students return, they start at the first set of unanswered questions. Since the teams are assigned, the interface shows each student a list of their teammates being evaluated. Demonstrating the interface This poster presentation will feature screen shots of each interface as well as a live demo. A demonstration mode is being designed, particularly to allow faculty evaluating the system to see the student interface. A sample interface screenshot is shown below, illustrating the numerical ratings resulting from the survey administration, recommended grade adjustment factors (if desired by the faculty member), and special notes that alert the faculty member to possible dynamics occurring within a team.