Context-based Pattern Recognition for Automating Big Data Management in Clouds

我们建议在云环境产生的大量操作数据中构建用于实时、动态模式识别的数据模型。我们的模式识别技术将应用于云环境的一系列自动化管理任务，例如异常诊断和资源配置。我们还将研究我们的模式识别技术对其他类型的大数据的可扩展性和适用性，例如托管在云环境中的生物医学和生物特征数据。目标 *数据收集 * 我们将捕获各种类型的云运营数据，并在收集点为所有感兴趣的应用程序范围（例如，工作负载阶段、参数化请求或方法调用。我们计划捕获和调查的运营数据类型包括常规日志记录和监控信息，例如吞吐量、延迟、可靠性或其他应用程序QoS指标，以及日志语句和CPU、网络带宽、磁盘和内存消耗的资源消耗时间序列数据。2.模式学习和识别 * 随着数据的收集，一个或多个统计学习技术被部署，以学习数学数据模型，每个上下文。对于在统计上与学习的数据模型不同的上下文，我们将触发自动系统适配，例如，在资源分配方面和/或提供上下文丰富的问题清单以供人类检查。**方法论 * 我们的关键思想是在我们的动态模式学习和识别框架中利用应用程序上下文，通过显式地增强常规系统日志记录和监控数据收集与这些数据发生的应用程序上下文。我们期望几种互补的数据建模和分析方法将联合或独立地应用于各种背景，例如，对于相同的应用范围、对于相同粒度的不同范围、和/或对于不同粒度的嵌套范围。不同的方法可能更适合不同的数据和不同的上下文。例如，基于聚类的分类方法可以很好地用于日志模板数据，但是用于资源消耗的时间序列数据中的模式匹配的神经网络方法可以提供更好的模式提取、学习和匹配精度。这两种方法可以提供更高的精度异常检测比任何一种方法单独。为了检测应用程序上下文，我们目前对应用程序源代码使用静态分析。然而，随着项目的进展，我们计划包括更多基于二进制代码检查的通用技术。在我们对原型的实验评估中，我们将在我们的云软件栈中使用开源系统，并使用聚类、动态时间扭曲、神经网络和其他统计学习方法进行实验。

We propose to build data models for real-time, on-the-fly pattern recognition in the large volumes of operational data produced by Cloud environments. Our pattern recognition techniques will be applied to a range of automated management tasks for Cloud environments, such as, anomaly diagnosis, and resource provisioning. We will also investigate the extensibility and applicability of our pattern recognition techniques to other types of big data, such as, biomedical and biometric data hosted on Cloud environments.******Objectives***1. Data Collection***We will capture various types of Cloud operational data and tag them with the high level application scope in which they occur, at the collection points, for all the application scopes of interest e.g., workload phase, parametrized request, or method invocation. The types of operational data we plan to capture and investigate include routine logging and monitoring information, such as, throughput, latency, dependability, or other application QoS metrics, as well as log statements and resource consumption time series data for CPU, network bandwidth, disk and memory consumption.******2. Pattern Learning and Recognition***As data is collected, one or more statistical learning techniques are deployed in order to learn mathematical data models, per context. For contexts that statistically diverge from the learned data model, we will trigger an automated system adaptation e.g., in terms of resource allocation and/or provide context-rich problem digests for human inspection.******Methodology***Our key idea is to leverage application contexts within our on-the-fly pattern learning and recognition framework, by explicitly augmenting routine system logging and monitoring data collection with the application contexts where these data occur. We expect that several, complementary data modelling and analysis methods will be applied jointly, or independently, for the various contexts, e.g., for the same application scope, for different scopes of the same granularity, and/or for nested scopes of different granularities. Different methods may work better for different data and different contexts. For example, a classification method based on clustering may work well for log template data, but a Neural Network method for pattern matching in time series data for resource consumption could provide better pattern extraction, learning and matching accuracy. Both methods could provide higher accuracy for anomaly detection than either method alone. In order to detect application contexts we currently use static analysis on application source code. However, we plan on including more general techniques based on binary code inspection as our project progresses. In our experimental evaluation of our prototype, we will use open-source systems in our Cloud software stack and experiment with clustering, dynamic time warp, Neural Networks and other statistical learning methods.