Performance management tools for big data applications

大数据应用的性能管理工具

• 批准号: 513200-2017
• 负责人: Krishnamurthy, Diwakar
• 金额: $ 4.95万
• 依托单位: University of Calgary
• 依托单位国家: 加拿大
• 项目类别: Collaborative Research and Development Grants
• 财政年份: 2017
• 资助国家: 加拿大
• 起止时间: 2017-01-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=618876
• 关键词: Performance; management; tools; big; data

The proposed project will develop tools that will allow users and operators of big data systems to better analyze and optimize the performance of their applications. Enterprises are increasingly relying on big data cluster systems to support their analytics requirements. Newer generation big data platforms such as Apache Spark provide a rich set of Application Programming Interfaces (APIs) that allow data scientists to quickly develop and deploy analytics applications. However, despite recent advances in big data platforms, analyzing and optimizing the performance of big data applications remains a challenge. Users and operators of big data clusters often face challenges in understanding the reasons for poor application performance and optimizing applications and clusters to meet performance requirements. To address these challenges, the proposed project will focus on two long term objectives. First, the project will devise an intuitive and intelligent interface that will support rapid performance analysis and optimization of big data applications. Using the interface, a user will be able to quickly identify bottlenecks that limit an application's performance. The interface can also be used to visualize and understand an application's performance scalability. Such features can in turn lead to increased productivity for users and more efficient usage of clusters for operators. Second, the project will develop automated service level management techniques for big data systems. Typically, big data systems use schedulers to control the allotment of resources to applications. Although production scheduling systems are sophisticated, they currently lack the ability to drive scheduling decisions so as to automatically meet user service level requirements, e.g., a specified deadline for completing an application, while simultaneously achieving efficient resource usage. Consequently, users and cluster operators often resort to tedious trial and error approaches, which can cause both missed service levels as well as inefficient resource usage. The proposed project will devise new scheduling techniques to address this limitation.

拟议的项目将开发工具，使大数据系统的用户和运营商能够更好地分析和优化其应用程序的性能。企业越来越依赖大数据集群系统来支持他们的分析需求。新一代的大数据平台，如Apache Spark，提供了丰富的应用程序编程接口（api），允许数据科学家快速开发和部署分析应用程序。然而，尽管大数据平台最近取得了进展，但分析和优化大数据应用的性能仍然是一个挑战。大数据集群的用户和运营商在了解应用性能不佳的原因，并优化应用和集群以满足性能要求方面经常面临挑战。为了应对这些挑战，拟议的项目将侧重于两个长期目标。首先，该项目将设计一个直观智能的界面，支持大数据应用的快速性能分析和优化。使用该接口，用户将能够快速识别限制应用程序性能的瓶颈。该接口还可以用于可视化和理解应用程序的性能可伸缩性。这些特性可以反过来提高用户的生产力，并使运营商更有效地使用集群。其次，该项目将为大数据系统开发自动化服务水平管理技术。通常，大数据系统使用调度器来控制应用程序的资源分配。虽然生产调度系统是复杂的，但它们目前缺乏驱动调度决策的能力，从而自动满足用户服务水平的要求，例如，完成应用程序的指定截止日期，同时实现有效的资源利用。因此，用户和集群操作员经常采用繁琐的试错方法，这可能导致错过服务级别以及低效的资源使用。拟议的项目将设计新的调度技术来解决这一限制。