diane guatelli

Geant4 in a Distributed Computing Environment: 

Geant4 in a Distributed Computing Environment S. Guatelli1, P. Mendez Lorenzo2, J. Moscicki2, M.G. Pia1 1. INFN Genova, Italy, 2. CERN, Geneva, Switzerland Geant4 2005 10th user conference and collaboration workshop November 3-10, 2005, Bordeaux, France                                                                                         

Vision: 

Vision Problem How to obtain a quick response from a Geant4 simulation Case 1: quick response in few minutes i.e. dosimetry, study the efficiency of detectors Case 2: reasonable time for response from G4 simulations requiring high statistics i.e. medical, space science, high energy physics applications, tests of Geant4 physics models Solution Parallelisation On dedicated pc clusters On the GRID Study a general approach, independent from the specific Geant4 application

Slide3: 

Quick response Transparent configuration in sequential or parallel mode Transparent access to the GRID through an intermediate software layer Parallelisation Access to distributed computing resources

Strategy: 

G4 IMRT application Execution time of 109 events ~ 9 days and half Goal: quick response ~ few hours Strategy Study the performance of two Geant4 applications as typical examples: Geant4 Brachytherapy application Geant4 IMRT application Parallelisation through DIANE Performance tests On a single CPU On clusters On the GRID Quantitative analysis of the results G4 Brachytherapy application Execution time of 20 M events ~ 5 hours Goal: quick response ~ few minutes Sequential mode on a Pentium IV, 3 GHz

Outline: 

Outline Diane overview How to dianize a G4 application Results of performance tests Conclusions

DIANE Overview: 

DIANE Overview DIANE R&D Project started in 2001 in CERN/IT with very limited resources collaboration with Geant 4 groups at CERN, INFN, ESA succesful prototypes running on LSF and EDG Parallel cluster processing make fine tuning and customisation easy transparently using GRID technology application independent Developed by J. Moscicki, CERN http://cern.ch/DIANE

Practical Example: 

Practical Example Example: simulation with analysis The job is divided into tasks The tasks are executed on worker components Each task produces a file with histograms Job result = sum of histograms produced by tasks Master-worker model Client starts a job Workers perform tasks and produce histograms Master integrates the results

Running in a distributed environment: 

Running in a distributed environment Not affecting the original code of the application standalone and distributed case is the same code Good separation of the subsystems the application does not need to know that it runs in a distributed environment the distributed framework (DIANE) does not need to care about what actions an application performs internally The application developer is shielded from the complexity of underlying technology via DIANE

How to dianize a G4 application: 

How to dianize a G4 application Look at the Geant4 extended example: ExDIANE in the parallel directory Completely transparent to the user: same G4 code Documentation at http://www.cern.ch/diane specific for Geant4 applications Installing and compiling DIANE Compiling and running a Geant4 application through DIANE

Test results: 

Test results Study the performance of the execution of the dianized G4Brachy: Test on a single CPU Test on a dedicated farm (60 CPUs) Test on a farm, shared with other users (LSF, CERN) Test on the GRID Tools and libraries: Simulation toolkit: Geant4 7.0.p01 Analysis tools: AIDA 3.2.1 and PI 1.3.3 DIANE: DIANE 1.4.2 CLHEP: 1.9.1.2 G4EMLOW2.3

Results – G4Brachy: 1 CPU: 

Results – G4Brachy: 1 CPU with respect to the number of events Test on a single dedicated CPU (Intel ®, Pentium IV, 3.00 GHz) Execution time with respect to the number of events of the job The overhead of DIANE is negligible in high statistics jobs

Slide12: 

Dedicated farm : 30 identical biprocessors (Pentium IV, 3 GHz) Thanks to Hurng-Chun Lee (Academia Sinica Grid Computing Center, Taiwan) Thanks to Regional Operation Centre (ROC) Team, Taiwan Results – G4Brachy: farm

Comment: 

Comment The job ends when all the tasks are executed on the workers If the job is split into a higher number of tasks, there is a higher chance that the workers finish the tasks at the same moment Example of a good job balancing Example of a job that can be improved from a performance point of view Worker number Time (seconds)

Results – G4Brachy: farm (3): 

Comparison Preliminary! Results – G4Brachy: farm (3) Test on LSF cluster of CERN: case of farm shared with other users The load of the cluster changes quickly in time The conditions of the test are not reproducible

Results – G4Brachy: GRID (1): 

Results – G4Brachy: GRID (1) The load of the GRID changes quickly in time The conditions of the test are not reproducible G4Brachy executed on the GRID on nodes located in Spain, Russia, Italy, Germany, Switzerland Execution on the GRID through DIANE, 20 M events,180 tasks, 30 workers Execution on the GRID, without DIANE Without DIANE: - 2 jobs not successfully executed due to set-up problems of the workers Through DIANE: - All the tasks are executed successfully on 22 workers - Not all the workers are initialized and used: on-going investigation Worker number Time (seconds) Worker number Time (seconds)

How the GRID load changes: 

How the GRID load changes Execution time of G4Brachy in two different conditions of the GRID DIANE used as intermediate layer Worker number Time (seconds) Worker number Time (seconds) 20 M events, 60 workers initialized, 360 tasks

Conclusions: 

Conclusions General approach to obtain quick response from Geant4 simulations Advantage of using DIANE as intermediate layer in a dedicated farm or GRID Transparency Good separation of the subsystems Good management of the CPU resources DIANE is very advantageous as an intermediate layer to the GRID from a performance point of view A quantitative analysis of the performance results is in progress Submission of this work for publication in IEEE Trans. Nucl. Sci. Acknowledgments to: M. Lamanna (CERN), Hurng-Chun Lee (ASGC, Taiwan), L. Moneta (CERN), A. Pfeiffer (CERN) Thanks to the GRID team of CERN and the Regional Operation Centre Team of Taiwan