Aspirations and Practice of ML Model Documentation: Moving the Needle with Nudging and Traceability
Aspirations and Practice of ML Model Documentation: Moving the Needle with Nudging and Traceability
复制标题
DOI：
10.1145/3544548.3581518
复制
发表时间：
2022-04
期刊：
影响因子：
--
通讯作者：
Avinash Bhat;Austin Coursey;Grace Hu;Sixian Li;Nadia Nahar;Shurui Zhou;Christian Kastner;Jin L. C. Guo
中科院分区：
文献类型：
--
作者：
Avinash Bhat;Austin Coursey;Grace Hu;Sixian Li;Nadia Nahar;Shurui Zhou;Christian Kastner;Jin L. C. Guo
机器学习（ML）模型的文档编制实践通常福尔斯达不到传统软件的既定实践，这阻碍了模型的可问责性，并无意中助长了模型的不当或滥用。最近，模型卡-一项关于模型文件编制的建议-引起了显著的注意，但其对实际做法的影响尚不清楚。在这项工作中，我们系统地研究了该领域的模型文档，并研究如何鼓励更负责任和更负责任的文档实践。我们对公开提供的模型卡的分析揭示了建议和实践之间的巨大差距。然后，我们设计了一个名为DocML的工具，旨在（1）推动数据科学家在模型开发过程中遵守模型卡的建议，特别是与道德相关的部分，以及（2）评估和管理文档质量。一项实验室研究揭示了我们的工具对长期文档质量和问责制的好处。
The documentation practice for machine-learned (ML) models often falls short of established practices for traditional software, which impedes model accountability and inadvertently abets inappropriate or misuse of models. Recently, model cards, a proposal for model documentation, have attracted notable attention, but their impact on the actual practice is unclear. In this work, we systematically study the model documentation in the field and investigate how to encourage more responsible and accountable documentation practice. Our analysis of publicly available model cards reveals a substantial gap between the proposal and the practice. We then design a tool named DocML aiming to (1) nudge the data scientists to comply with the model cards proposal during the model development, especially the sections related to ethics, and (2) assess and manage the documentation quality. A lab study reveals the benefit of our tool towards long-term documentation quality and accountability.