INCA: An Integrated Cluster Computing Architecture for Machine Translation

机器翻译（MT）领域的进展在很大程度上依赖于开源工具包，这使得新的研究小组更容易以更低的成本解决问题，扩大参与。 不幸的是，工具包没有跟上现代计算基础设施（例如，MapReduce框架），大多数工具包中的“原语”很难扩展到新的模型，因为它们专注于管道组件而不是算法概念，并且实验管理几乎被忽视。本项目正在开发用于翻译的集成集群计算架构（印加），通过实现可扩展的，开源工具包，可以利用MapReduce集群，灵活实现多种类型的MT系统。 MT并不适合MapReduce（它具有大量的内存占用，需要迭代算法）;新的算法正在开发，以利用该框架，而不受其限制。实验管理，评估和建议的“最佳做法”也是工具包的一部分，这一项目预计将通过向研究界提供开放源码工具包，对机器翻译研究产生广泛影响。 一个适合本科生的课程项目将使用该工具包开发和公开共享。 大规模并行机器翻译问题的技术解决方案将适用于数据密集型自然语言处理和机器学习领域，预计该工具包的元素也将有助于此类研究工作。

Progress in the field of machine translation (MT) has come to depend heavily on open-source toolkits, which make it easier for new research groups to tackle the problem at lower cost, broadening participation. Unfortunately, toolkits have not kept up with modern computing infrastructure (e.g., the MapReduce framework) required for modern "big data" approaches to MT, the "primitives" in most toolkits are hardly extensible to new models since they focus on pipeline components rather than algorithmic concepts, and experiment management has been all but ignored.This project is developing the Integrated Cluster Computing Architecture (INCA) for translation to overcome these challenges, by implementing an extensible, open-source toolkit that can leverage MapReduce clusters and flexibly implement many types of MT systems. MT is not a perfect fit for MapReduce (it has massive memory footprints and requires iterative algorithms); new algorithms are being developed to take advantage of the framework without being limited by it. Experiment management, evaluation, and advice about "best practices" are also part of the toolkit, to make it as widely accessible as possible.This project is expected to have broad impact in MT research through the open-source toolkit to be made available to the research community. A course project suitable for undergraduates will be developed and shared openly using the toolkit. Technical solutions to problems in large-scale, parallelized MT will be applicable in areas of data-intensive natural language processing and machine learning, and elements of the toolkit are expected to be useful in such research efforts as well.