Utilizing distributed parallel computation for computer intensive statistical analysis within a heterogeneous computer environment

点击翻译按钮获取中文摘要

We adopted a form of implementations of Beowulf. type PC cluster for the distributed parallel computation environment and studied the actual implementations and pros and cons of the use of such implementation. In order to install such system, we reorganized the network system and computer resource in our laboratories and established the distributed parallel computation environment with PVM and MPI(here we used "lam" as an actual implementation) with those computers.We developed a task management method "Time Prediction Model", which control tasks with their expected run time as a manager-worker model and the method was compared with "Work Pool Methods", in order to control tasks within our heterogeneous computing resource. Based on the results given here, we also considered a fundamental design of an integrated application development environment for the distributed parallel environment, which would support computationally intensive statistical investigation, such as Bootstrap and simulation based analyses.For the algorisms, we first investigated some basic calculation tools, such as matrix multiplications and inversion. Especially for multiplication of larger matrices, several methods, such as column/row wise matrix splitting and Fox algorithm with checker board arrangements are investigated regarding the memory and CPU resource configuration. Furthermore, programs for the bootstrapping and tree based methods were reviewed and their plausibility for the parallel processing, granularity issues and coherency for the task management methods were investigated.Based on these considerations, we installed a distributed parallel processing simulation platform within the heterogeneous computer resources and confirmed its efficiency for use of computer intensive analysis for real data analysis and their performance evaluation, via analyses of multinomial categorical data with over-dispersion problem.