Intelligent Management of Big Data Storage

数据量的持续革命性增长和数据密集型应用程序的日益多样性要求迫切研究有效的存储管理方法。在2012年夏天，全球的数据量约为10的21次方字节，每个互联网用户约为1.1TB，并且这一数量继续以约50%的复合年增长率增长。有人说：“到2013年，存储系统将不再需要手动调整性能或手动放置数据。与虚拟内存管理类似，存储阵列的算法将决定数据放置（存储管理的未来，Gartner 2010）。满足数据密集型应用程序的服务级别目标/协议（SLO/SLA）要求并不简单，而且将变得越来越具有挑战性。特别是，考虑到新设备技术的出现，越来越需要智能机制来管理底层架构的基础设施。为了科普这一挑战，我们提出了EPSRC主题为“Towards an intelligent information infrastructure（TI 3）"的主流研究计划，特别是关于“数据泛滥”和“用于低功率、高速度、高密度、低成本存储器和存储解决方案的新兴技术”的探索。如今，随着存储的广泛分布，例如在云存储解决方案中，基础设施提供商很难决定数据驻留在哪里，在什么类型的设备上，与哪个其他（可能是竞争）用户拥有的其他数据共存，甚至在哪个国家。实现能源消耗目标的需要加剧了这一问题。这些决策问题激发了本研究的建议，其目的是开发新的基于模型的技术和算法，以促进数据密集型应用程序及其底层存储设备infrastructure.We建议开发的技术和工具的定量分析和优化的多层数据存储系统的有效管理。主要目标是开发新的建模方法，以定义和促进最合适的数据放置和数据迁移策略。这些策略的共同目标是将数据放置在分层存储体系结构中最有效的目标设备上。在所提出的研究中，分配算法将能够决定放置策略和触发数据迁移，以优化适当的效用函数。我们的研究还将考虑不断发展的存储和能效技术可能产生的量化影响，方法是开发适当的模型，并将其整合到我们的分层分配方法中。本质上，我们的模型将专门用于不同的存储和电力技术（例如化石燃料，太阳能，风能）。我们生产的模型、优化器和方法将在我们的内部云（已购买）上进行试点实施测试;在Amazon EC2资源上;最后在工业受控生产环境中进行测试，作为我们与NetApp合作的一部分。这将提供反馈，使我们能够改进，增强和扩展我们的技术，从而进一步提高最大的存储系统的实用性。

The continuing revolutionary growth of data volumes and the increasing diversity of data-intensive applications demands an urgent investigation of effective means for efficient storage management. In the summer of 2012, the volume of data in the world was around 10 to the power of 21 bytes, about 1.1TB per internet user, and this volume continues to increase at about 50% Compound Annual Growth Rate. It has been said that "By 2013, storage systems will no longer be manually tunable for performance or manual data placement. Similar to virtual memory management, the storage array's algorithms will determine data placement (The Future of Storage Management, Gartner 2010). Meeting service-level objective/agreement (SLO/SLA) requirements for data-intensive applications is not straightforward and will become increasingly more challenging. In particular, there is an increasing need for intelligent mechanisms to manage the underlying architectures' infrastructure, taking into account the advent of new device technologies.To cope with this challenge, we propose a research program in the mainstream of EPSRC's theme "Towards an intelligent information infrastructure (TI3)", specifically with reference to the "deluge of data" and the exploration of "emerging technologies for low power, high speed, high density, low cost memory and storage solutions". Today, with the widespread distribution of storage, for example in cloud storage solutions, it is difficult for an infrastructure provider to decide where data resides, on what type of device, co-located with what other data owned by which other (maybe competing) user, and even in what country. The need to meet energy-consumption targets compounds this problem. These decisional problems motivate the present research proposal, which aims at developing new model-based techniques and algorithms to facilitate the effective administration of data-intensive applications and their underlying storage device infrastructure.We propose to develop techniques and tools for the quantitative analysis and optimisation of multi-tiered data storage systems. The primary objective is to develop novel modelling approaches to define and facilitate the most appropriate data placement and data migration strategies. These strategies share the common aim of placing data on the most effective target device in a tiered storage architecture. In the proposed research, the allocation algorithm will be able to decide the placement strategy and trigger data migrations to optimize an appropriate utility function. Our research will also take into account the likely quantitative impact of evolving storage and energy-efficiency technologies, by developing suitable models of these and integrating them into our tier-allocation methodologies. In essence, our models will be specialised for different storage and power technologies (e.g. fossil fuel, solar, wind). The models, optimisers and methodologies that we produce will be tested in pilot implementations on our in-house cloud (already purchased); on Amazon EC2 resources; and finally in an industrial, controlled production environment as part of our collaboration with NetApp. This will provide feedback to enable us to refine, enhance and extend our techniques, and hence to further improve the utility of the biggest of storage systems.