Developing Novel Machine Learning Techniques to Improve Comparative Judgements for e-Learning and e-Assessment

开发新颖的机器学习技术以改进电子学习和电子评估的比较判断

• 批准号: 2440744
• 金额: --
• 依托单位: Swansea University
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2440744
• 关键词: Developing; Novel; Machine; Learning; Techniques

There is a wide selection of aims we could examine under this project:1) To understand how machine learning (ML) can function as a support for an educator and not a strait jacket. Key questions to ask:Are different measures relevant to different teachers? How do they interpret the data and use it to guide their teaching?2) To understand how we can gather large volumes of quantifiable data about qualitative assessments allowing machine learning (ML) approaches to work on the data.3) What design methods are suitable to use in order to engage with students and teachers in this complex design process. Key questions are:How do we express possibilities for different insights that we can gather from the data?How do we capture ideas about more nebulous elements of ML like privacy and ensure ongoing consent for data collection?4) To identify appropriate ML methods and develop a complete ML framework for analysing qualitative assessments.5) To actively query a small subset of all submissions to learn what attributes constitute excellent quality, taking an interactive approach with the educators to improve our proposedframework's performance.6) To validate and make the developed tool useable in the real-world.We aim to redress the imbalance caused by automated marking tools promoting specific approaches to assessment. We will develop a decision support tool for educators assessing qualitative work that will make such assessments more attractive by increasing the educators understanding of the student cohorts' work and, potentially, reducing the amount of time they need to spend marking it. A comparative judgement framework [1] will be developed to allow educators to understand how changes in practice between different cohorts' impact on assessments. If the comparative judgement framework can show reliable performance at the cohort level, we will explore how it can be applied to individual qualitative assignments to support educators' assessments of them. Year One - Identifying assessments: Qualitative assessments are subjective and learning about good and bad practices autonomously challenging. To address this, the student will survey teaching practitioners to identify the most important types of qualitative assessments, and key learning outcomes in those assessments. We will also work with educators and students to understand their attitudes to automated assessment of their work, their concerns about the tool and the ways in which they can be reassured of the validity of such assessments following Value Sensitive Design approaches [2]. This will narrow down thescope of the tool and be used to create a proof-of-concept.Year One and Two - Learning knowns and identifying gaps in knowledge: CDSM has awealth of data on submissions and their respective grades. At this stage, we aim to learn from this data to identify key features within the scope identified in the previous step. Here, we will first focus on finding meaningful structures in the dataset from a semantic perspective [3], potentially utilising a supervised approach (using existing grades) and some form of contextual embeddings [4]. These will help derive insight into what makes a submission address specificlearning outcomes. Given we use a probabilistic model, we can deduce which submissions our model has low confidence (or high uncertainty in predictions) about: this shows the gaps in our knowledge, which only a human practitioner can help fill.Year Three - Interactively improving models: develop an interactive visual system to display insights into cohort data and individual submissions and allow educations practitioners to respond to two key questions about a submission:Were our model predictions correct? How would they rate a given assignment?These submissions will be carefully picked via an active learning strategy where we select submission that we are unsure about or that are likely to improve our model to provide better predictions.

在这个项目下，我们可以研究的目标有很多：1）了解机器学习（ML）如何作为教育工作者的支持而不是紧身衣。要问的关键问题：不同的措施是否与不同的教师有关？他们如何解释这些数据并用它们来指导他们的教学？2)了解我们如何收集有关定性评估的大量可量化数据，从而允许机器学习（ML）方法处理数据。3）适合使用哪些设计方法来让学生和教师参与这个复杂的设计过程。关键问题是：我们如何表达从数据中收集到的不同见解的可能性？我们如何捕捉关于ML更模糊的元素（如隐私）的想法，并确保对数据收集的持续同意？4)识别适当的机器学习方法并开发一个完整的机器学习框架来分析定性评估。5）主动查询所有提交的一小部分，以了解哪些属性构成优秀的质量，采取与教育工作者互动的方法来提高我们提出的框架的性能。6）验证并使开发的工具在真实的中可用-我们的目标是纠正自动评分工具所造成的不平衡，促进具体的评估方法。我们将为评估定性作业的教育工作者开发一个决策支持工具，通过增加教育工作者对学生群体作业的理解，并可能减少他们需要花费的时间，使这种评估更具吸引力。将开发一个比较判断框架[1]，让教育工作者了解不同群体之间的实践变化对评估的影响。如果比较判断框架能够在队列层面显示可靠的表现，我们将探索如何将其应用于个人定性作业，以支持教育工作者对它们的评估。第一年-确定评估：定性评估是主观的，自主学习好的和坏的做法具有挑战性。为了解决这个问题，学生将调查教学实践者，以确定最重要的定性评估类型，以及这些评估中的关键学习成果。我们还将与教育工作者和学生合作，了解他们对工作自动化评估的态度，他们对工具的担忧，以及他们可以在价值敏感设计方法之后确保此类评估有效性的方式[2]。这将缩小工具的范围，并用于创建概念验证。第一年和第二年-学习知识并确定知识差距：CDSM拥有大量关于提交材料及其相应等级的数据。在这个阶段，我们的目标是从这些数据中学习，以确定上一步确定的范围内的关键特征。在这里，我们将首先关注从语义的角度[3]在数据集中找到有意义的结构，可能使用监督方法（使用现有的等级）和某种形式的上下文嵌入[4]。这些将有助于深入了解是什么使提交解决具体的学习成果。假设我们使用概率模型，我们可以推断出我们的模型具有低置信度的提交（或预测的高度不确定性）：这显示了我们知识的空白，只有人类实践者可以帮助填补。第三年-交互式改进模型：开发一个交互式视觉系统，以显示对队列数据和个人提交的见解，并允许教育从业者回答关于问题：我们的预测是否正确？他们会如何评价一个给定的任务？这些提交将通过主动学习策略进行仔细挑选，我们选择我们不确定或可能改进我们的模型以提供更好预测的提交。