SBIR Phase I: A scale-out data transfer solution for data-intensive enterprises

这个小企业创新研究（SBIR）第一阶段项目的更广泛的影响/商业潜力将是使数据密集型工程和科学组织能够提高他们的数据传输速度，而不必管理过早的、昂贵的和破坏性的网络升级。此外，这个项目的发现和结果向数据密集型社区（包括商业和研究）的成员揭示了将数据移动作为除传统的三个关键IT维度（存储、计算和网络）之外的第四个关键IT维度的重要性。此外，该项目的结果将表明，第四维度必须通过前三个维度的平衡交互来定义，以实现所需的数据传输速度-这是一个重要且难以捉摸的概念，但对于经济有效地使用高速数字通信至关重要。与目前最先进的技术相比，该项目产生的产品将更具可扩展性，空间和能源效率，成本效益高，易于部署和使用。最后，该产品将证明它可以满足本十年主要科学和商业努力的数据传输需求。这将促进科学和工程的加速发展，从而促进整个国民经济的健康发展。这个小型企业创新研究（SBIR）第一阶段项目解决了现代分布式数据密集型工程和科学操作中除了存储、计算和网络之外的第四个IT维度（数据移动）的需求。为了配合指数级数据增长的趋势，本项目的产品将提供高性能的数据传输和加密。该设计在IOPS、计算能力、网络接口等方面进行平衡和伸缩。它具有面向集群的体系结构，使用点对点技术来简化部署、操作和使用。两种正在申请专利的算法有助于处理包含小文件和非常大文件的混合数据集，并且对网络延迟不敏感。其独特的优化使闪存的有效使用。该项目将开发一种简单、有效且具有横向扩展能力的存储分层软件，一种横向扩展的高速数据加密方法，以及一种用于实现更高效的存储I/O密集型应用程序的横向扩展方法。预计两个现有的数据传输节点（dtn）前端并行数据文件系统之间的数据速率将与两个常用的高性能数据传输应用程序之间的数据速率相当。通过两个四节点集群，该产品预计在16x10Gbps聚合链路上实现155Gbps内存对内存，在5000英里100Gbps链路上实现70Gbps文件对文件的加密。

The broader impact/commercial potential of this Small Business Innovation Research (SBIR) Phase I project will be to enable data-intensive engineering and science organizations to enhance their data transfer speeds without having to administer premature, costly, and disruptive network upgrades. In addition, the findings and results from this project shed a much needed light to the members of data-intensive communities (both commercial and research) regarding the importance of treating data movement as the 4th critical IT dimension, besides the traditional three: storage, compute, and networking. Furthermore, the project's outcome will show that the 4th-dimension must be defined by balanced interactions of the first three to achieve a desired data transfer speed - an important and elusive concept to grasp, but critical in using high-speed digital communications cost-effectively. Compared to the current state-of-the-art, the product resulting from this project will be far more scalable, space and energy efficient, cost-effective, and easy to deploy and use. Finally, the product will demonstrate that it can meet the data transfer needs of major scientific and commercial endeavors of this decade. This should foster accelerated progress in science and engineering, and thus also the overall national economic health.This Small Business Innovation Research (SBIR) Phase I project addresses the need in modern distributed data-intensive engineering and science operations for a 4th IT dimension - data movement - besides storage, compute, and networking. To match the exponential data-growth trend, the product resulting from this project will provide high-performance data transfer and encryption. The design balances and scales in: IOPS, computation power, and network interfaces. It has a cluster-oriented architecture, using peer-to-peer technologies to ease deployment, operation, and usage. Two patent-pending algorithms help tackle data sets containing a mix of small files and very large files, and provide insensitivity to network latency. Its unique optimizations enable effective use of flash storage. The project will develop a simple, effective, and scale-out capable storage tiering software, a scale-out high-speed data encryption approach, and a scale-out approach for implementing more efficient storage I/O intensive applications. It is anticipated that data rates between two existing Data Transfer Nodes (DTNs) front ending parallel data file systems will be comparable with those from two commonly used high performance data transfer applications. With two four-node clusters, the product is anticipated to achieve 155Gbps memory-to-memory over a 16x10Gbps aggregated link and 70Gbps file-to-file with encryption over a 5000 mile 100Gbps link.