UJ Research Cluster | User / Using Geant4 on the cluster

Geant4 is a Monte Carlo code for particle transport; it is widely used in High Energy Physics, but is also applicable to lower energies.

Using Geant4

Here we look at how to run multiple instances of a plain Geant4 simulation.
In recent versions of Geant4.9 there is some support for parallel execution of a single instance, using MPI; that is left as an exercise for the reader (in particular the compatibility with Root is doubtful).

Configure the environment

The cluster uses a network file system to use CERN's versions of software like Geant4. This is called CernVM File System. Source the following script to set up the environment:

# source this

# get the Geant4
. /cvmfs/geant4.cern.ch/etc/setup.sh
. /cvmfs/geant4.cern.ch/etc/setup-gcc.sh
#export G4PLATFORM=x86_64-slc5-gcc43-opt
. /cvmfs/geant4.cern.ch/etc/login.sh
. /cvmfs/geant4.cern.ch/releases/geant4.9.5.p01/setup.sh


# get the ROOT installation
export ROOTSYS=/cvmfs/sft.cern.ch/lcg/app/releases/ROOT/5.34.05/$G4PLATFORM/root
export PATH=$ROOTSYS/bin:$PATH
export LD_LIBRARY_PATH=$ROOTSYS/lib:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$ROOTSYS/lib:$DYLD_LIBRARY_PATH

# add in libexpat ... it couldnt find it in the system
export LD_LIBRARY_PATH=/cvmfs/sft.cern.ch/lcg/external/expat/2.0.1/$G4PLATFORM/lib:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=/cvmfs/sft.cern.ch/lcg/external/expat/2.0.1/$G4PLATFORM/lib:$DYLD_LIBRARY_PATH

# now we need cmake
export PATH=/cvmfs/sft.cern.ch/lcg/external/CMake/2.8.6/$G4PLATFORM/bin:$PATH

# not yet sure if we are configure for make or cmake
#CMAKE_INSTALL_PREFIX=/cvmfs/geant4.cern.ch/releases/geant4.9.5.p01/x86_64-slc5-gcc43-opt/

# now we need CLHEP
export CLHEP_BASE_DIR=/cvmfs/sft.cern.ch/lcg/external/clhep/2.0.4.5/x86_64-slc5-gcc43-opt

Note that every worker node will need to source this, so the easiest might be to have a shell file that first sets up the environment, then runs the executable.

Prepare your code for parallel execution

The most important thing is to make sure that for each instance you use a different seed for the number number generator; this can be done using with /random/setSeed . You must also be careful that your program saves the data in separate files, to avoid conflicts between the various instances.
To avoid having to generate the macro "on-the-fly", you can use the alias mechanism in the Geant4 macro, like

/analysis/setFilename BGOpix1+NaI_coinc_26cm-opt.{seed}.root
/random/setSeed {seed}

and change the main() of your Geant4 code so that is uses command line parameters to set the alias, like

if (argc>=3) {
for (int ia=2;ia<argc;ia++){
// split an alias=value string and call /control/alias
char* alias=argv[ia];
char* value=strchr(alias,'=');
if (value!=NULL) {
value[0]='\0';value++;
char cmd[1024];
snprintf(cmd,1024,"/control/alias %s %s",alias,value);
//printf("%s\n",cmd);
UI->ApplyCommand(cmd);
}
}
UI->ApplyCommand("/control/listAlias");
}

[$[Get Code]]

Run with PBS

Sample job definition script batchG4test.pbs:

#PBS -N batchG4test
#PBS -m ae
#PBS -q default
#PBS -l cput=01:00:00
#PBS -v	LD_LIBRARY_PATH,ROOTSYS,SEED
cd $PBS_O_WORKDIR
time ./g4test novis.g4m seed=$SEED
## end of batch script

Then you can use the script to launch multiple instances of your Geant4 job:

# SEED=1 qsub batchG4test.pbs
# SEED=2 qsub batchG4test.pbs
# SEED=3 qsub batchG4test.pbs

The analysis of the separate ntuple files can easily be done by setting up an ntuple chain.