Investigating the reliability of aggregate measurements of learning process data: From theory to practice
Investigating the reliability of aggregate measurements of learning process data: From theory to practice
复制标题
DOI：
10.1111/jcal.12951
复制
发表时间：
2024-02
期刊：
影响因子：
--
通讯作者：
Yingbin Zhang;Yafei Ye;Luc Paquette;Yibo Wang;Xiaoyong Hu
中科院分区：
文献类型：
--
作者：
Yingbin Zhang;Yafei Ye;Luc Paquette;Yibo Wang;Xiaoyong Hu
学习分析 (LA) 研究通常会汇总学习过程数据，以提取表明感兴趣结构的测量结果。然而，这种聚合将产生可靠测量的保证尚未得到明确检验。聚合测量的可靠性证据很少被报道，留下了一个隐含的假设，即此类测量没有错误。本研究通过调查聚合测量的心理测量利弊来解决这些差距。本研究提出了聚合过程数据的框架，其中包括聚合合适的条件，以及选择适当可靠性证据和计算程序的指南。我们通过分析本科生在计算机科学入门课程中的学业拖延和编程能力来支持和演示该框架。一段时间内的聚合是可以接受的，并且只有在兴趣结构在此期间稳定的情况下才可以提高测量可靠性。否则，聚合可能会掩盖有意义的行为变化，因此应该避免。在选择可靠性证据类型时，一个关键问题是过程数据是否可以被视为重复测量。另一个问题是过程的长度是否不平等以及个别事件是否不可靠。如果第二个问题的答案是否定的，则将每个过程分段为固定数量的箱有助于计算可靠性系数。所提出的框架可以作为 LA 研究中聚合过程数据的通用指南。研究人员应在随后的解释之前检查并报告汇总测量的可靠性证据。
Learning analytics (LA) research often aggregates learning process data to extract measurements indicating constructs of interest. However, the warranty that such aggregation will produce reliable measurements has not been explicitly examined. The reliability evidence of aggregate measurements has rarely been reported, leaving an implicit assumption that such measurements are free of errors.This study addresses these gaps by investigating the psychometric pros and cons of aggregate measurements.This study proposes a framework for aggregating process data, which includes the conditions where aggregation is appropriate, and a guideline for selecting the proper reliability evidence and the computing procedure. We support and demonstrate the framework by analysing undergraduates' academic procrastination and programming proficiency in an introductory computer science course.Aggregation over a period is acceptable and may improve measurement reliability only if the construct of interest is stable during the period. Otherwise, aggregation may mask meaningful changes in behaviours and should be avoided. While selecting the type of reliability evidence, a critical question is whether process data can be regarded as repeated measurements. Another question is whether the lengths of processes are unequal and individual events are unreliable. If the answer to the second question is no, segmenting each process into a fixed number of bins assists in computing the reliability coefficient.The proposed framework can be a general guideline for aggregating process data in LA research. Researchers should check and report the reliability evidence for aggregate measurements before the ensuing interpretation.