A study of dynamic optimization of data acquisition system using GRID technology

在传统的数据采集系统中，事件生成器和触发处理器的功能和数据流是固定的，并且取决于加速器条件，存在浪费资源或性能瓶颈的可能性。本研究的目的是通过真实的实时监控处理器负载和数据流速率来动态优化处理器的使用和数据流。在第一年，监控机制是使用一个测试台系统开发的，该系统由25台通过大型GbE交换机连接的PC服务器组成。该机制利用贝儿实验中使用的网络共享存储器（NSM）实现，并成功运行。此外，还使用名为MICH-G2的GRID软件对节点间的传输速度进行了测量，确认为60 MB/秒以上。第二年，开发了处理器利用率和数据流的优化机制。在处理器上实现了处理功能的动态变化， ...更多信息 模块化处理软件并使用与数据采集框架的动态链接来替换它们。结合使用在每个处理节点上实现的大型环形缓冲区和套接字连接，成功地改变了节点上的数据流路径和/或处理功能，而不中断数据采集。作为最后一步，我们试图开发一种算法来优化整个系统的性能，然而，我们还没有成功地开发出满足数据采集系统中真实的使用的机制。在数据流速率逐渐变化期间，观察到数据流路径和处理软件的大量切换发生在某个边界处，导致资源浪费。有时在切换期间观察到数据采集的短暂停止。因此，我们需要继续努力，以改善优化算法与一些新的技术，如神经net.The机制开发的数据流和处理器负载的监测用于真实的时间重建农场的贝儿实验，并在国际会议上报告计算在高能物理04。少

In a conventional data acquisition system, the function and data flow of event builder and trigger processor are fixed, and there are possibilities of wasting resouces or performance bottleneck depending on the accelerator condition. The purpose of this study is to optimize the usage of the processors and the data flow dynamically by a real time monitoring of the processor load and the data flow rate. In the first year, the monitoring mechanism was developed using a test bench system consisting of 25 PC servers connected via a large GbE switch. The mechanism was realized utilizing Network Shared Memory(NSM) used for the Belle experiment and it was succeeded to make it work. The transfer speed between nodes was also measured using a GRID software called MICH-G2 and was confirmed to be more than 60MB/sec.In the second year, the development of the optimization mechanism of processor utilization and data flow was done. A dynamic change of processing function on a processor was realized by … More modularizing the processing software and replacing them using a dynamic link with the data acquisition framework. Together with a use of large ring buffers and socket connections implemented on each processing node, it was succeeded to change data flow path and/or processing function on a node without interrupting the data acquisition. As the last step, an algorithms to optimize system-wide performance was trying to be developed, however, we have not yet succeeded to develop the mechanism satisfying the real usage in the data acquisition system. A massive switch of the data flow path and processing software is observed to happen at some boundary during a gradual change in the data flow rate, resulting in a waste of resources. Sometimes a short stop of the data acquisition is observed during the switching. Therefore, we need to continue the effort to improve the optimization algorithm with some new technology like the neural-net.The mechanism developed for the monitoring of data flow and processor load was used for the real time reconstruction farm of the Belle experiment and it was reported at the international conference on Computing in High Energy Physics 04. Less