Sensitivity of self-reported noncognitive skills to survey administration conditions
Sensitivity of self-reported noncognitive skills to survey administration conditions
复制标题
自我报告的非认知技能对调查管理条件的敏感性
DOI：
10.1073/pnas.1910731117
复制
发表时间：
2020-01-14
影响因子：
11.1
通讯作者：
Kautz, Tim
中科院分区：
文献类型：
--
作者：
Chen, Yuanyuan;Feng, Shuaizhang;Kautz, Tim
意义 最近的证据表明，非认知技能对于生活的成功至关重要，并且可以通过干预来塑造。由于这些证据，政策制定者和研究人员对衡量非认知技能越来越感兴趣，并且通常依赖于受访者对自己技能进行自我报告的衡量标准。此类自我报告已应用于项目评估以及学校问责和改进系统。我们证明自我报告对调查管理条件很敏感，包括调查管理员是否描述了正在评估的技能以及受访者是否获得与其他任务绩效相关的激励。这些发现对自我报告措施的解释具有影响。社会政策或干预措施可能会影响对自我报告的非认知技能的反应，但不会影响技能本身。非认知技能（例如坚持性和自我控制）通常使用自我报告的问卷来衡量，受访者在问卷中对自己的技能进行评分。在许多应用中——包括项目评估和学校问责系统——此类报告被假定为仅衡量感兴趣的技能。然而，自我报告还可能涵盖技能之外的其他维度，例如受访者情况的各个方面，其中可能包括激励措施和他们完成调查问卷的条件。为了探索这种可能性，本研究进行了两项实验，以估计调查管理条件对学生对非认知技能问卷的反应的影响程度。第一个实验测试了向学生提供有关非认知技能重要性的信息是否会直接影响他们的反应，第二个实验测试了与另一项任务的表现相关的激励是否会间接影响反应。这两项实验都表明，非认知技能的自我报告对调查条件很敏感。与项目评估文献中发现的条件相比，这些条件的影响相对较大，范围为 0.05 至 0.11 标准差。这些发现表明，应谨慎解释干预措施或其他社会政策对自我报告的非认知技能的影响。
Significance Recent evidence has shown that noncognitive skills matter for success in life and can be shaped through interventions. Because of this evidence, policy makers and researchers have increasingly become interested in measuring noncognitive skills and typically rely on self-reported measures in which respondents rate their own skills. Such self-reports have been applied in program evaluations, as well as school accountability and improvement systems. We demonstrate that self-reports are sensitive to survey administration conditions, including whether a survey administrator describes the skills being assessed and whether respondents receive incentives tied to performance on other tasks. These findings have implications for the interpretation of self-reported measures. Social policies or interventions might affect responses on self-reported noncognitive skills without affecting the skills themselves. Noncognitive skills (e.g., persistence and self-control) are typically measured using self-reported questionnaires in which respondents rate their own skills. In many applications—including program evaluation and school accountability systems—such reports are assumed to measure only the skill of interest. However, self-reports might also capture other dimensions aside from the skill, such as aspects of a respondent’s situation, which could include incentives and the conditions in which they complete the questionnaire. To explore this possibility, this study conducted 2 experiments to estimate the extent to which survey administration conditions can affect student responses on noncognitive skill questionnaires. The first experiment tested whether providing information about the importance of noncognitive skills to students directly affects their responses, and the second experiment tested whether incentives tied to performance on another task indirectly affect responses. Both experiments suggest that self-reports of noncognitive skills are sensitive to survey conditions. The effects of the conditions are relatively large compared with those found in the program evaluation literature, ranging from 0.05 to 0.11 SDs. These findings suggest that the effects of interventions or other social policies on self-reported noncognitive skills should be interpreted with caution.