Using Swift on the cluster

Swift is a programming language developed by the University of Chicago for running applications on a cluster, taking care of input files, output files etc. Here are some steps required to get a basic example working:

Source code

We will use the following simple c++ program, saved as test.cc

#include <iostream>
#include <stdlib.h>

using namespace std;

int main(int argc, char** argv) {
        sleep(50);   // wait a while so that we can see the jobs in the queue with qstat
        if (argc > 1) {
                float num = atof(argv[1]);
                cout << "The square of " << num << " is " << num*num << endl;
        } else {
                cout << "No input supplied" << endl;
        }
}

]

Compile this with
g++ test.cc

tc.data file

Now, we need to tell swift about our application. Create a file called tc.data with the following (change cookish to your user account):

#NOTE WELL: fields in this file must be separated by tabs, not spaces; and
#there must be no trailing whitespace at the end of each line.
#
# sitename  transformation  path   INSTALLED  platform  profiles
localhost       test    /nfs/home/cookish/test/a.out    INSTALLED       INTEL64::LINUX  null
uj      test    /nfs/home/cookish/test/a.out    INSTALLED       INTEL64::LINUX  globus

This creates two sites we can submit jobs to, one called localhost which should just run jobs, and one called uj which will submit them to the cluster (in namespace globus). Now we will be able to use the application called test in Swift scripts, which is mapped to the executable /nfs/home/cookish/test/a.out

sites file

Create a file called sites-uj.xml with the following:

<config>
  <pool handle="uj">
    <gridftp  url="local://localhost" />
    <execution provider="pbs" />
    <workdirectory >/nfs/home/cookish/tmp</workdirectory>
    <profile namespace="karajan" key="jobThrottle">3</profile>
    <profile namespace="karajan" key="initialScore">500</profile>
    <profile namespace="globus" key="queue">batch</profile>
  </pool>
</config>

This tells Swift that the namespace globus should submit jobs to pbs in the batch queue.

Swift file

Now, create test.swift with the following:

type file;

app (file o) test (int x)
{
  test x stdout=@o;
}

file out[]<simple_mapper; location="./outputdir", prefix="test.",suffix=".out">;
foreach j in [1:@toint(@arg("n","10"))] {
  out[j] = test(j);
}

This script tells it to run the program n times, putting the results in a directory called "outputdir", with filenames test.0001.out, test.0002.out, etc.

Running the app

Execute the swift script with:
swift -tc.file ./tc.data -sites.file sites-uj.xml test.swift -n=2