UJ Research Cluster | SysAdm / Downtimes

For any downtimes, scheduled or not, an entry should be put in to the GOCDB
This will stop the pilot factories from sending jobs to the site.

The current web link can be found at: https://goc.egi.eu/portal/index.php?Page_Type=Site&id=285
The sysadmin will have to be registered on the GOCDB and have rights to the ZA-UJ site.

Good practice is to inform the VO's supported at your site of the downtime about a day before it will go into affect.
Do this with the operations portal broadcast mechanism.
https://operations-portal.egi.eu/broadcast - select the channel (several VO's and AAROC ) and write a touching message.

For ATLAS you can manually set the queue status if hammercloud is still sending jobs, or to restart it.
This can be done quickly with the following script which has been modified for UJ (from a UI or LXplus):

########################################
# Author: Alessandra Forti
# Date: 21/02/2012
########################################
#!/bin/bash

usage(){

cat <<EOF 

Usage: $0 status queue [comment]

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
source $ATLAS_LOCAL_ROOT_BASE/user/atlasLocalSetup.sh
localSetupEmi
voms-proxy-init -voms atlas

status values = setmanual, setonline, setoffline, setauto

queue values = UKI-NORTHGRID-MAN-HEP-ce01-long-pbs, 
               UKI-NORTHGRID-MAN-HEP-ce01-long-pbs, 
               else look at http://tinyurl.com/qrnqr3.

comment = See examples below.

Example 

setqueuestatus setoffline UKI-NORTHGRID-MAN-HEP-ce02-long-pbs elog.12345
setqueuestatus settest ZA-UJ-glite-ce-pbs HC.Test.Me
setqueuestatus settest ANALY_ZA-UJ HC.Test.Me
setqueuestatus settest ANALY_MANC Cloud.Blacklist
setqueuestatus settest ANALY_MANC_TEST Site.Test.Queue
setqueuestatus setonline UKI-NORTHGRID-MAN-HEP-ce02-long-pbs HC.Blacklist.set.online

EOF

exit

}

###########################################################################
# Main
###########################################################################

#read status queue comment

status="$1"
queue="$2"
comment="$3"

if [ -z "$status" -o -z "$queue" ];then

    usage
    exit

fi


if [ ! -z "$comment" ]; then

    comment=`echo $comment|sed 's/ /%20/g'`
fi

if [ -z $X509_USER_PROXY ]; then 
    echo "\$X509_USER_PROXY not set, using default value /tmp/x509up_u<UID>"
fi
X509_USER_PROXY=${X509_USER_PROXY:="/tmp/x509up_u`id -u`"}


sl_version=`lsb_release -r | awk '{print $2}'| cut -f1 -d.`

if [ $sl_version -eq 5 ]; then 

    echo "Executing curl -k --cert $X509_USER_PROXY \"https://panda.cern.ch:25943/server/controller/query?tpmes=$status&queue=$queue&moduser=ShifterName&comment=$comment\""
    curl -k --cert $X509_USER_PROXY "https://panda.cern.ch:25943/server/controller/query?tpmes=$status&queue=$queue&moduser=ShifterName&comment=$comment"

elif [ $sl_version -eq 6 ] ; then 

    if [ -z $X509_CERT_DIR ]; then 
	echo "\$X509_CERT_DIR not set, using default value /etc/grid-security/certificates"
    fi
    X509_CERT_DIR=${X509_CERT_DIR:="/etc/grid-security/certificates"}

    echo "Executing curl --capath $X509_CERT_DIR --cacert $X509_USER_PROXY --cert $X509_USER_PROXY \"https://panda.cern.ch:25943/server/controller/query?tpmes=$status&queue=$queue&moduser=ShifterName&comment=$comment\""
    curl --capath $X509_CERT_DIR --cacert $X509_USER_PROXY --cert $X509_USER_PROXY "https://panda.cern.ch:25943/server/controller/query?tpmes=$status&queue=$queue&moduser=ShifterName&comment=$comment"
fi