I-Corps: Feasibility Study for Commercializing a Domain-Specific Big Data Analytics Cloud Software Stack

在所谓的大数据时代，许多公司的数据变得太大，无法通过传统的计算平台进行处理和存储。这些行业迫切需要应对日益增长的大数据带来的挑战，特别是对于没有大规模计算基础设施的小公司和承包商。那些拥有大规模计算基础设施的大公司目前正在使用传统的高性能计算（HPC）技术来处理大数据需求。然而，传统的HPC技术被设计用于处理计算密集型工作，而不是数据密集型工作。此外，传统的HPC需要对HPC架构有深入的了解和专业知识，并使用低级并行编程模型，这通常会为普通用户和领域专家直接利用HPC基础设施造成障碍。I-Corps团队提出了云计算技术，为满足计算和存储需求提供了一种经济高效的解决方案，特别适用于小型公司和初创公司。该项目旨在探索存储、处理和分析石油地震数据、医疗图像、零售、能源等多个领域的大量数据的软件栈的商业化机会。除了软件堆栈提供的可扩展性能和容错功能外，它还提供高级编程模板，工作流程和特定于领域的语言，以简化大规模计算平台的使用。拟议的基于云的软件堆栈将通过用户友好的Web界面提供定制的计算平台服务，旨在填补大数据分析平台与客户之间的差距。的要求在各种领域。目标用户是石油，医疗和其他工业领域处理大量数据的数据科学家，分析师，研究人员和开发人员。

In the so-called big data era, the data in many companies are becoming too big to be processed and stored by traditional computing platforms. There are urgent needs to address the challenge from ever-increasing big data facing these industries, especially for small companies and contractors who do not have large scale computing infrastructure. Those big companies that have the large scale computing infrastructure, are currently using the traditional High Performance Computing (HPC) techniques to handle the big data requirements. However, the traditional HPC techniques were designed to process computation intensive works, instead of data-intensive jobs. Moreover, the traditional HPC requires deep knowledge and expertise in HPC architectures and uses low-level parallel programming models, which typically create an obstacle for the regular users and domain experts to utilize HPC infrastructure directly. This I-Corps team proposes the cloud computing technique that provides a cost-effective solution to address the computation and storage requirements, which is especially useful to small companies and startups.This project is to explore the commercialization opportunities for a software stack that stores, processes and analyzes big volume of data in a variety of domains, including petroleum seismic data, medical images, retails, energy and more. Besides the scalable performance and fault-tolerance features provided by the software stack, it also provides high-level programming templates, workflow and a domain specific language to simplify the usage of large scale computing platforms. The proposed cloud-based software stack will deliver a customized computing platform service via a user-friendly web interface that is designed to fill the gap between the big data analytics platforms and the customer?s requirements in a variety of domains. The targeted users are data scientists, analysts, researchers and developers in the petroleum, medical and other industrial domains dealing with big volume of data.