Minimizing the usage of hardware counters for collective communication using triggered operations
Minimizing the usage of hardware counters for collective communication using triggered operations
复制标题
使用触发操作最大限度地减少集体通信中硬件计数器的使用
DOI：
10.1145/3343211.3343222
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
M. Garzarán
中科院分区：
文献类型：
--
作者：
Nusrat S. Islam;G. Zheng;S. Sur;Akhil Langer;M. Garzarán
触发操作和计数事件或计数器是可由通信库（诸如MPI）用来将集体操作卸载到主机结构接口（HFI）或网络接口卡（NIC）的构建块。当触发计数器达到指定阈值时，触发操作可用于调度将来发生的网络或算术操作。在操作完成时，完成计数器的值增加1。通过这种机制，可以创建一个相关操作链，以便当所有相关操作都完成执行时触发操作的执行。触发操作依赖于HFI上的硬件计数器，并且是有限的资源。因此，如果所需计数器的数量超过硬件计数器的数量，则集合需要停止，直到前一集合完成并且计数器被释放。此外，如果HFI具有计数器高速缓存，则利用大量计数器可能导致高速缓存颠簸并提供较差的性能。因此，减少计数器的数量非常重要，特别是在大型超级计算机上运行时，或者当应用程序使用非阻塞集合并且多个集合可以并发运行时。在本文中，我们提出了一种算法来优化的硬件计数器的数量时，卸载集体触发操作。通过该算法，不同的操作可以根据它们之间的依赖关系及其拓扑顺序共享和重用触发器和完成计数器。我们的实验结果表明，我们提出的算法显着减少了计数器的数量比默认的方法，不考虑操作之间的依赖关系。
Triggered operations and counting events or counters are building blocks that can be used by communication libraries, such as MPI, to offload collective operations to the Host Fabric Interface (HFI) or Network Interface Card (NIC). Triggered operations can be used to schedule a network or arithmetic operation to occur in the future, when a trigger counter reaches a specified threshold. On completion of the operation, the value of a completion counter increases by one. With this mechanism, it is possible to create a chain of dependent operations, so that the execution of an operation is triggered when all its dependent operations have completed its execution. Triggered operations rely on hardware counters on the HFI and are a limited resource. Thus, if the number of required counters exceeds the number of hardware counters, a collective needs to stall until a previous collective completes and counters are released. In addition, if the HFI has a counter cache, utilizing a large number of counters can cause cache thrashing and provide poor performance. Therefore, it is important to reduce the number of counters, specially when running on a large supercomputer or when an application uses non-blocking collectives and multiple collectives can run concurrently. In this paper, we propose an algorithm to optimize the number of hardware counters used when offloading collectives with triggered operations. With our algorithm, different operations can share and re-use trigger and completion counters based on the dependences among them and their topological orderings. Our experimental results show that our proposed algorithm significantly reduces the number of counters over a default approach that does not consider the dependences among the operations.