Designing a Peer Evaluation Instrument that is Simple, Reliable, and Valid

一个简单、可靠的合作学习小组同伴评价方法是本项目的重点。该项目解决了一个重要的问题，有效地利用和更广泛地接受工程教育中的合作学习。正在设计和测试的仪器不会提供与更复杂和难以使用的仪器相同的反馈水平，但其可靠性和有效性应与目前可用的最佳仪器相当或更高，其简单性将鼓励那些尚未准备好采取更雄心勃勃的方法的教师广泛采用。这个项目建立在合作学习的基础上。一个管理良好的多项目仪器很有可能成为一个成功的仪器。该项目正在研究同行评价中的关键问题（分数膨胀、相同分数和偏见）和可靠性（评分者数量、管理机构数量）。因为没有对团队工作质量的真正度量，所以项目正在使用多种评估方法并寻求它们之间的一致性。仪器效度是通过口头协议分析、行为观察和与其他验证仪器获得的结果相一致来建立的。正在评估测试-重测信度和评分者间信度。该项目正在五个参与机构以分配方式进行。这些机构共有9名教员，在11门不同的课程中试验不同的评估方法。课程和讲师的广度保证了开发的方法的健壮性。这一同行评估工具有望显著改善基于团队的工程教育和评估活动。使用它的教师能够通过给学生形成性的同伴反馈来改善团队技能的教学。使用它的工程项目可以证明他们的学生有团队合作的能力。通过出版物和演讲成功地传播了原型仪器，这是有希望的。这一工具的简单性、有效性和可靠性有望加快其他机构对其的采用。来自多个工程教育联盟的学校的参与也可能加速其传播。评估工具及其相关的规则和说明也对许多学术项目有潜在的兴趣。正在开发的工具在任何采用合作小组的地方都是有用的。对该工具进行验证和可靠性评估的广泛测试也应尽量减少任何意外的性别或种族偏见。

Intellectual MeritA simple, reliable cooperative learning group peer evaluation method is the focus of this project. The project addresses an issue that is significant for the effective use of and wider acceptance of cooperative learning in engineering education. The instrument being designed and tested will not provide the same level of feedback as more complex and difficult- to-use instruments, but its reliability and validity should be comparable or higher than those of the best instruments currently available, and its simplicity will encourage widespread adoption by faculty who are not ready to make the commitment to more ambitious approaches. This project builds from what is known about cooperative learning. A multi-item instrument with good administration has good potential for becoming a successful instrument. The project is working on crucial issues in peer evaluation (inflation of scores, identical scores, and bias) and in reliability (number of raters, number of administrations). Because there is no true measure of the quality of teamwork, the project is using multiple assessment methodologies and seeking concurrence among them. Instrument validity is being established through verbal protocol analysis, behavioral observation, and concurrence with results obtained with other validated instruments. Both test-retest reliability and inter-rater reliability are being assessed. The project is being conducted in a distributive fashion in five participating institutions. There are nine faculty across these institutions, experimenting with different methods of assessment in eleven different courses. The breadth of courses and instructors promises to make the developed method robust. Broader impact This peer evaluation instrument is expected to significantly improve both team-based engineering education and assessment activities. Instructors using it are able to improve the teaching of team skills by giving students formative peer feedback. Engineering programs using it can document that their students have the ability to work in teams. The successful dissemination of the prototype instrument through publications and presentations has been promising. The simplicity, validity, and reliability of this instrument can be expected to speed its adoption at other institutions. The involvement of schools from multiple Engineering Education Coalitions is also likely to accelerate its dissemination. The assessment instrument and its associated rubric and instructions are also of potential interest to a large number of academic programs. The instrument under development is useful anywhere cooperative groups are employed. Extensive testing for validation and reliability evaluation for this tool should also minimize any unintended gender or racial bias.