SBIR Phase I: A scale-out data transfer solution for data-intensive enterprises

此小型企业创新研究(SBIR)第一阶段项目的更广泛影响/商业潜力将是使数据密集型工程和科学组织能够提高其数据传输速度，而不必管理过早、成本高昂且具有破坏性的网络升级。此外，该项目的发现和结果为数据密集型社区(包括商业和研究)的成员提供了亟需的信息，让他们认识到，除了传统的三个维度：存储、计算和网络之外，将数据移动视为第四个关键IT维度的重要性。此外，该项目的结果将表明，第四维必须由前三个维度的平衡相互作用来定义，以达到所需的数据传输速度--这是一个重要且难以理解的概念，但对于以经济高效的方式使用高速数字通信至关重要。与目前最先进的产品相比，该项目产生的产品将更具可扩展性，节省空间和能源，具有成本效益，并且易于部署和使用。最后，该产品将证明它可以满足本十年主要科学和商业努力的数据传输需求。这将促进科学和工程领域的加速发展，从而促进整个国家的经济健康。这一小型企业创新研究(SBIR)第一阶段项目解决了现代分布式数据密集型工程和科学运营对除存储、计算和网络之外的第四个IT维度(数据移动)的需求。为了适应指数级的数据增长趋势，该项目产生的产品将提供高性能的数据传输和加密。该设计在IOPS、计算能力和网络接口方面进行了平衡和扩展。它具有面向集群的体系结构，使用点对点技术来简化部署、操作和使用。两种正在申请专利的算法帮助处理包含小文件和超大文件混合的数据集，并提供对网络延迟的不敏感性。其独特的优化使闪存能够得到有效利用。该项目将开发简单、高效且支持横向扩展的存储分层软件、横向扩展高速数据加密方法以及用于实施更高效的存储I/O密集型应用程序的横向扩展方法。预计两个现有数据传输节点(DTN)前端并行数据文件系统之间的数据速率将与来自两个常用高性能数据传输应用的数据速率相当。通过两个四节点群集，该产品预计将在16x10Gbps聚合链路上实现155Gbps内存到内存，并在5000英里100Gbps链路上实现70Gbps文件到文件加密。

The broader impact/commercial potential of this Small Business Innovation Research (SBIR) Phase I project will be to enable data-intensive engineering and science organizations to enhance their data transfer speeds without having to administer premature, costly, and disruptive network upgrades. In addition, the findings and results from this project shed a much needed light to the members of data-intensive communities (both commercial and research) regarding the importance of treating data movement as the 4th critical IT dimension, besides the traditional three: storage, compute, and networking. Furthermore, the project's outcome will show that the 4th-dimension must be defined by balanced interactions of the first three to achieve a desired data transfer speed - an important and elusive concept to grasp, but critical in using high-speed digital communications cost-effectively. Compared to the current state-of-the-art, the product resulting from this project will be far more scalable, space and energy efficient, cost-effective, and easy to deploy and use. Finally, the product will demonstrate that it can meet the data transfer needs of major scientific and commercial endeavors of this decade. This should foster accelerated progress in science and engineering, and thus also the overall national economic health.This Small Business Innovation Research (SBIR) Phase I project addresses the need in modern distributed data-intensive engineering and science operations for a 4th IT dimension - data movement - besides storage, compute, and networking. To match the exponential data-growth trend, the product resulting from this project will provide high-performance data transfer and encryption. The design balances and scales in: IOPS, computation power, and network interfaces. It has a cluster-oriented architecture, using peer-to-peer technologies to ease deployment, operation, and usage. Two patent-pending algorithms help tackle data sets containing a mix of small files and very large files, and provide insensitivity to network latency. Its unique optimizations enable effective use of flash storage. The project will develop a simple, effective, and scale-out capable storage tiering software, a scale-out high-speed data encryption approach, and a scale-out approach for implementing more efficient storage I/O intensive applications. It is anticipated that data rates between two existing Data Transfer Nodes (DTNs) front ending parallel data file systems will be comparable with those from two commonly used high performance data transfer applications. With two four-node clusters, the product is anticipated to achieve 155Gbps memory-to-memory over a 16x10Gbps aggregated link and 70Gbps file-to-file with encryption over a 5000 mile 100Gbps link.