ATLAS Computing: Getting Started On The UJ Research Cluster

1.  New UJ Research Cluster

• Please find all details on the bhubesi Google Folder.
  • May of the instructions there now supercede those below
• Grafana Reporting

2.  Athena Software Release

The UJ cluster currently has Athena release kit 14.5.2 installed. To be able to run the athena software you will need an account on the cluster. See here for details on how to obtain an account.

3.  Preparing Your Account to use the ATLAS Software

(from the ATLAS Computing Workbook, modified for osg-ce.uj.ac.za)

ATLAS software is divided into packages, and these are managed through the use of a configuration management tool, CMT. This is used to copy ("check out") code from the main ATLAS repository and handle linking and compilation.

You must prepare your account in the following way. (Note that cmthome does not have to be in $HOME, it can be in any sub directory, but if so you will need to amend all the following examples accordingly.)

> cd $HOME
> mkdir cmthome
> mkdir testarea
> mkdir testarea/14.5.2    (this is the directory in which you will work)
> cd cmthome

Then do:

> source /nfs/osg/app/atlas_app/atlas_rel/14.5.2/CMT/*/mgr/setup.sh

Now using your text editor of choice, create a file called requirements. See AtlasLogin for a full explanation of what this does. You can copy a requirements file from /nfs/osg/app/atlas_app/atlas_rel/14.5.2/cmtsite/requirements and customise it. See also UseAtlasSoftwareProjectsKit.

My requirements file on osg-ce.uj.ac.za looks like this:

#---------------------------------------------------------------------
set CMTSITE STANDALONE
set SITEROOT /nfs/osg/app/atlas_app/atlas_rel/14.5.2
macro ATLAS_TEST_AREA ${HOME}/testarea
macro ATLAS_DIST_AREA ${SITEROOT}
apply_tag projectArea
macro SITE_PROJECT_AREA ${SITEROOT}
macro EXTERNAL_PROJECT_AREA ${SITEROOT}
apply_tag opt
apply_tag setup
apply_tag simpleTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
set CMTCONFIG i686-slc4-gcc34-opt
#---------------------------------------------------------------------

Now do: > cmt config

You will only have to follow these procedures once until the version of CMT changes. Now kill the terminal window and log back in.

Note: The set CMTCONFIG line effectively hardwires you to a particular platform (in this case 32 bit optimised) which cannot be overridden each time you start work (see below). Experienced users may wish to omit it and use the tags specified in AtlasLogin explicitly. ATLAS Releases 14 onwards only work with SL4.

Note: Make sure you have set ATLAS_TEST_AREA correctly in the requirements file or you will get errors saying that the include directory is not found in your work area when you try to compile for example. You will need to modify this requirements file if you change your test area directory. Also make sure you use ${HOME} or an absolute path to refer to your home directory and not ~.

Note:: The requirements files above use one default test area directory (${HOME}/testarea or whatever you set). If you want the functionality to use any directory as a test area then you whould replace the macro ATLAS_TEST_AREA line with:
macro ATLAS_TEST_AREA "$PWD"
and then cd to the directory you want before doing source ~/cmthome/setup.sh as desribed below.

Note: By default the test directory has sub directories for each release e.g. testarea/14.2.23. If you include the line apply_tag oneTest then all releases use the same directory but you will need to clean it up if the release number changes. This might be useful if you follow the procedure above.

Some tips for using CMT can be found at SoftwareDevelopmentWorkbookCmtTips and more advanced setup instructions at WorkBookAdvancedSetup.

4.  What To Do Every Time You Start Work

You need to issue the following command to set your terminal's environment variables to the correct values, and to set up CMT. You should do this every time you start a new terminal window in which you wish to use ATLAS software. With the requirements files above, you would type:

> source ~/cmthome/setup.sh -tag=14.5.2

which would set the release number to 14.5.2 and the development area to ${HOME}/testarea/14.5.2. If you did not specify apply_tag setup in your requirements file then you have to in addition type:

> source /nfs/osg/app/atlas_app/atlas_rel/14.5.2/AtlasOffline/14.5.2/AtlasOfflineRunTime/cmt/setup.sh

You can add a tag "32" to make sure that the 32-bit version is used even on 64-bit machines, as the 64-bit version of the athena software is not yet validated. However 32 is currently the default so this is not yet strictly necessary.

> source ~/cmthome/setup.sh -tag=14.5.2,32

You might want to put these in your login script.

5.  Testing Your Setup

> echo $CMTCONFIG indicates the chosen directory name for compilation results (typically i686-slc4-gcc34-opt ).

> echo $CMTPATH should list two paths: your own and the Atlas release. If not, see comments above regarding the requirements file (or you haven't done > source ~/cmthome/setup.sh). A third path with Gaudi is appended once you use your first package, and a fourth with LCG common code. Note that it is possible to specify your own path CMTPATH by hand : export CMTPATH=_ [or setenv CMTPATH ] provided this is done after source ~/cmthome/setup.sh. This allows an easier switch between your working directories. But note that you should not try to switch releases by modifying directly CMTPATH.

> cmt should give you a list of cmt commands.

> cmt show path lists areas where packages are taken in order of priority.

6.  Likely Problems

If later on you get errors like:

mkdir: cannot create directory `/include': Permission denied

then the setup probably failed. CMT should have appended your testarea to "/" but it was probably undefined and instead of /path_to_your_testarea/include you just have /include and you obviously don't have permission to create files in the root of the file system. You need to check the cmthome/requirements file and the source ~/cmthome/setup.sh command carefully. Things to check are:

• Make sure you have set ATLAS_TEST_AREA correctly
• Use ${HOME}$ not ~
• Use ${HOME}/testarea not ${HOME}/testarea/14.5.2
• Don't source ~/cmthome/setup.sh more than once with different tags
• Make sure you are using the correct .sh or .csh for your shell
• Make sure you don't have an old .cmtrc file in your home directory
• Sometimes there are problems if you use symbolic links to get to your testarea. i.e. do cd ${HOME}/testarea/14.2.23/MyPackage rather than cd linktomypackage from somewhere else where linktomypackage is a symbolic link to ${HOME}/testarea/14.5.2/MyPackage.

If you get errors like

athena.py: No such file or directory

then CMT probably didn't set up the run time environment properly. Make sure your cmthome/requirements has apply_tag setup in it or that you have -tag=14.5.2,setup when you do source ~/cmthome/setup.sh.

If all else fails start a new shell or terminal window and try again.

7.  Running on the Grid

To run jobs on the grid you will need a grid certificate and have to be part of the ATLAS Virtual Organization (VO).

• Starting on the Grid ATLAS Twiki page
• Getting a grid certificate in South Africa