When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching
When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching
复制标题
DOI：
10.1177/1098214020931941
复制
发表时间：
2021-07-20
影响因子：
1.7
通讯作者：
Laursen, Sandra L.
中科院分区：
文献类型：
--
作者：
Weston, Timothy J.;Hayward, Charles N.;Laursen, Sandra L.
在研究和评价中，观察被广泛地用来描述教学活动。因为进行观测通常是资源密集型的，所以从观测数据中自信地做出推断是很重要的。虽然人们的注意力集中在评分员之间的可靠性上，但在一个学期的课程中，单一班级测量的可靠性受到的关注较少。我们研究了观察在评估教学实践中的用途和局限性，以及在一门典型的课程中需要多少观察才能对教学实践做出自信的推断。我们根据概括性理论进行了两项研究，以计算一学期教学中不同班级之间的可信度。为了获得可靠的测量，需要对整个学期的11个课时进行观察，比文献中通常观察到的一到四个课时要多得多。研究结果表明，从业者可能需要投入比预期更多的资源来实现可靠的测量和比较。
Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.