Mining Software Repositories to Support the Development and Operation of Ultra Large Scale Software Systems

挖掘软件存储库 (MSR) 领域探索从源代码控制和错误跟踪系统等静态记录保存软件存储库中恢复有用信息的创新方法。恢复的信息用于获得对软件开发实践的基于经验的理解。这些知识可以帮助从业者预测、计划和理解项目的各个方面。在过去的十年中，国际 MSR 界探索了创新方法，从此类静态记录存储库中恢复有用且可操作的信息。 MSR 技术现已集成到世界上一些最大、最成功的组织的开发人员工具包中：Microsoft、BlackBerry、AT&T Labs、CISCO、Mozilla 和 Avaya。作为 MSR 领域的创始人之一，我相信只有将 MSR 结果和技术超越软件开发挑战，才能充分发挥 MSR 的潜力。我的最终目标是将 MSR 建立为贯穿软件系统生命周期的战略决策工具。然而，当今的研究重点仍然是传统的软件开发，其中软件的开发与其操作之间存在明显的分离，开发周期持续数月或数年。这种分离对于超大规模 (ULS) 软件服务来说是一个关键挑战，因为快速和频繁的发布是常态。显然需要利用软件服务开发和运营的知识。这种知识集成带来的好处的两个例子是：i）开发人员可以使用操作知识（例如日志）来更好地了解他们的代码在现场的使用情况，以便他们可以对其进行优化，并更有效地修复或优先修复错误； ii) 操作员可以利用源代码文档来快速理解、分类和解决现场问题。拟议计划的长期愿景是系统地揭示潜力并凭经验证明 MSR 在显着提高当今快速发展的长寿命软件的可靠性和成本效益方面的优势。未来五年的短期目标将侧重于与 BlackBerry、SAP 和 Avaya 等行业合作伙伴合作实现这一愿景。未来五年，该研究计划将沿着以下两个主题取得进展：（1）以简单的格式恢复、表示和链接开发和运营知识，以便于获取这些知识； (2) 开发方法论、工具和途径，探索和展示 MSR 在理解和支持软件开发和运营活动方面的好处。特别是，拟议的研究计划将侧重于使用系统日志作为弥合软件开发和运营之间差距的机制。日志是开发人员在源代码中插入的语句生成的消息，操作人员通常使用它来监控系统的现场操作。然而，由于日志的非结构化性质、规模庞大以及使用临时日志分析技术，日志中丰富的知识尚未得到充分利用。拟议的研究计划将培养五名高素质人才（HQP）：3名博士生和2名硕士生。所有总部将与加拿大公司密切合作，了解从业者每天面临的挑战以及软件数据在应对这些挑战方面的潜力。

The Mining Software Repositories (MSR) field explores innovative approaches to recover useful information from static record-keeping software repositories like source control and bug tracking systems. The recovered information is used to gain an empirically based understanding of software development practices. This knowledge helps practitioners predict, plan and understand various aspects of their project. Over the past decade, the international MSR community has explored innovative approaches to recover useful and actionable information from such static record-keeping repositories. MSR techniques are now integrated into developer toolkits at some of the world's largest and most successful organizations: Microsoft, BlackBerry, AT&T Labs, CISCO, Mozilla and Avaya.As one of the founders of the MSR field, I believe that the full potential of MSR can only be achieved by taking MSR results and techniques beyond software development challenges. My ultimate goal is to establish MSR as a strategic decision-making instrument throughout the lifetime of software systems. However, the focus of research today remains on traditional software development where a clear separation exists between the development of the software and its operation, with development cycles lasting months or years. This separation is a key challenge to Ultra Large Scale (ULS) software services where fast and frequent releases are the norm. There is a clear need to leverage knowledge from both the development and the operation of software services. Two examples of benefits arising from this knowledge integration are: i) developers can use operation knowledge (e.g., logs) to better understand how their code is used in the field so they can optimize it, and fix or prioritize the fixing of bugs more effectively; and ii) operators can make use of source code documentation to rapidly understand, triage and resolve field problems.The long-term vision of the proposed program is to systematically uncover the potential and to empirically demonstrate the benefits of MSR in delivering significant improvements in the reliability and cost-effectiveness of today's long-lived rapidly-evolving software. Short term goals over the next five years will focus on realizing this vision in collaboration with industry partners like BlackBerry, SAP and Avaya. Over the next five years the research program will progress along the following two themes: (1) the recovery, representation and linking of development and operation knowledge in a simple format in order to ease access to such knowledge; and (2) the development of methodologies, tools and approaches that explore and demonstrate the benefits of MSR in understanding and supporting software development and operation activities.In particular, the proposed research program will focus on the use of system logs as a mechanism to bridge the gap between software development and operation. Logs are messages generated from statements inserted by developers in the source code and are often used by operators for monitoring the field operation of a system. However, the rich knowledge in logs has not yet been fully-used because of their non-structured nature, their large scale, and the use of ad-hoc log analysis techniques.The research program proposed will train five highly qualified personnel (HQP): 3 PhD and 2 Master's students. All HQP will collaborate closely with Canadian companies, gaining an appreciation of the challenges facing practitioners on a daily basis and the potential of software data in addressing those challenges.