Ensuring the Trustworthiness of Data Science Systems in a Cost Effective Manner

能够从结构化或非结构化数据中提取知识和见解的管道(即过程和技术)通常被称为数据科学(DS)。集成DS流水线的大型软件系统通常称为DS系统。将DS管道集成到生产系统中带来的挑战远远超出了传统的机器学习程序(例如，ML模型的匹配质量)。当今DS系统面临的两个显著挑战是：1)可伸缩性挑战：DS系统必须处理的大量数据及其快速发布周期要求重新考虑如何在如此大的规模上进行测试过程。例如，当每周都在部署新版本时，我们如何执行长达一个月的测试！2)可解释性挑战：仔细而自信地了解特定事件发生的原因或确定DS系统是否正确运行仍然是一项重大任务。此外，DS系统决策的基本原理必须在不同版本之间保持稳定，除非有充分的理由支持其他版本。拟议的协作研究计划将开发方法，以成本效益和可扩展的方式在DS系统的背景下实现“可信计算”。业界(黑莓)和学术界(SAIL)的合作伙伴都是世界公认的“值得信赖的计算”领域的领导者，十多年来一直致力于优化和测试高度安全系统的性能和可靠性。这项研究建立在他们丰富的经验基础上，并将培训9名HQP，涉及软件性能工程、安全工程和经验软件工程，以及基本的沟通和演示技能。

The pipelines (i.e., processes and techniques) that enable the extraction of knowledge and insights from structured or unstructured data are commonly referred to as Data Science (DS). Large scale software systems that integrate DS pipelines are commonly referred to as DS systems. Integrating DS pipelines into production systems brings challenges that go well beyond the classical Machine Learning programs (e.g., quality of the fit of an ML model). Two notable challenges for DS systems today are: 1) Scalability challenges: The large amount of data that DS systems must process and their rapid release cycles require a rethinking of how testing processes are being conducted at such scale. For instance how can we perform an extended month-long test when new releases are being deployed on a weekly basis! 2) Explainability challenges: Carefully and confidently understanding why a particular event occurred or determining whether a DS system is performing correctly remains a major undertaking. Moreover the rationale for the decisions of a DS system must remain stable across versions unless there are good reasons to warrant otherwise.The proposed collaborative research program will develop approaches to enable "Trustworthy Computing" within the context of DS systems in a cost effective and scalable manner. Both the industrial (BlackBerry) and academic (SAIL) partners are world-recognized leaders in "Trustworthy Computing" with over a decade working on optimizing and testing the performance and reliability of highly secure systems. The research builds on their extensive experience and will train nine HQPs in software performance engineering, security engineering, and empirical software engineering, as well as essential communication and presentation skills.