My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
复制标题
DOI：
10.1145/3576123.3576134
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
James Finnie-Ansley;Paul Denny;Andrew Luxton-Reilly;E. Santos;J. Prather;Brett A. Becker
中科院分区：
文献类型：
--
作者：
James Finnie-Ansley;Paul Denny;Andrew Luxton-Reilly;E. Santos;J. Prather;Brett A. Becker
OpenAI法典的引入引发了对生成人工智能模型对计算教育实践影响的兴趣激增。Codex也是GitHub Copilot的底层模型，这是一个插件，通过流行的代码编辑器中的自动完成功能，学生可以访问人工智能生成的代码。这一领域的研究，特别是关于教育意义的研究，刚刚起步，并且几乎完全集中在入门编程（或CS1）问题上。最近的研究表明，Codex在典型的CS1考试问题上的表现要比大多数学生好得多。然而，尚不清楚食品法典委员会在更复杂的规划作业和考试方面的限制是什么。在本文中，我们详细介绍了Codex在过去考试中更高级的CS2（数据结构和算法）考试问题上的表现。我们将这些结果与在正常条件下参加相同考试的学生的结果进行了比较，表明Codex的得分高于大多数学生。我们考虑这些工具对未来的本科计算机教育的影响。
The introduction of OpenAI Codex sparked a surge of interest in the impact of generative AI models on computing education practices. Codex is also the underlying model for GitHub Copilot, a plugin which makes AI-generated code accessible to students through auto-completion in popular code editors. Research in this area, particularly on the educational implications, is nascent and has focused almost exclusively on introductory programming (or CS1) questions. Very recent work has shown that Codex performs considerably better on typical CS1 exam questions than most students. It is not clear, however, what Codex’s limits are with regard to more complex programming assignments and exams. In this paper, we present results detailing how Codex performs on more advanced CS2 (data structures and algorithms) exam questions taken from past exams. We compare these results to those of students who took the same exams under normal conditions, demonstrating that Codex outscores most students. We consider the implications of such tools for the future of undergraduate computing education.