Accuracy of the Q-matrix-validation algorithm and sensitivity of model fit indices toward different types of Q-matrix misspecification for diagnostic classification models (DiaFit)

点击翻译按钮获取中文摘要

Diagnostic classification models (DCM) represent an internationally widely discussed but in Germany little known class of confirmatory probabilistic multidimensional latent-variable models with categorical latent variables providing meaningful proficiency profiles based on statistically-driven multivariate classifications. These profiles can inform students and teachers about the particular strengths and weaknesses in a given academic area and allow for the derivation of appropriate supportive interventions. For the estimation of DCM the specification of attributes involved in the solution process of items in a Q-matrix is required. The determination of the Q-matrix is of key importance, as it represents the theoretical basis of DCM. In practical applications, however, the true Q-matrix is unknown and may contain incorrect entries. This phenomenon is called Q-matrix misspecification. It is common to distinguish between three different types of Q-matrix misspecification: underspecified, overspecified, and balanced. Several algorithms have been proposed in the literature for the empirical validation of the Q-matrix. However, their accuracy has been investigated only for selected types of Q-matrix misspecification. Moreover, assessing model fit for DCM continues to be a challenge, although various absolute and relative model fit indices for DCM have been proposed in the past. The project DiaFit investigates in a complex simulation study and an application study important methodological issues related to the assessment of the model fit and the accuracy of the Q-matrix validation algorithm for different types of Q-matrix misspecification for log-linear DCM. In the simulation study the sensitivity of different absolute and relative model fit measures (e.g. AIC, BIC, MAD, RMSEA) to three different types of misspecified Q-matrices (underspecified, overspecified, balanced) will be investigated. Moreover, the accuracy of the Q-matrix validation algorithm for these three types of Q-matrix misspecification will be addressed. In the simulation the number of respondents, attributes, and items as well as the item-structure complexity in the Q-matrix will be varied. The results will essentially extend the state of the art in model fit assessment and robustness of the Q-matrix validation algorithm for log-linear DCM and enable recommendations for the assessment of the model fit of DCM in applied settings. The results of the simulation will be applied in a secondary analysis to two different datasets on basic arithmetic skills in elementary school that have already been collected. The Q-matrices are also available for these datasets. This application study will illustrate how different model fit indices and the Q-matrix validation method can be used in practical applications when the true data-generating Q-matrix is unknown.