HYCOM

Using TAU with Hycom can be a bit tricky. It has .F files that need to be pre-processed. The source calls xcstop('msg') to end the program. So, we must use -optPreProcess to compile Hycom. I changed config/xt3.tau_mpi:

### Tau stub makefiles:
##  Profiling:
include /spin/proj/perc/TOOLS/tau-2.15.6b1/xt3/lib/Makefile.tau-multiplecounters-mpi-papi-pdt-pgi
## Tracing:
#include /spin/proj/perc/TOOLS/tau-2.15.6b1/xt3/lib/Makefile.tau-multiplecounters-mpi-papi-pdt-trace-pgi

OPTS_TAU  = -optPdtGnuFortranParser -optVerbose -optPreProcess -optTauSelectFile=select.tau
FCOMPILER = $(TAU_COMPILER) $(OPTS_TAU) ftn
CCOMPILER = $(TAU_COMPILER) $(OPTS_TAU) cc

on Jaguar. To compile the code, place select.tau in the source directory:

GLBA0.08/src_2.1.35_26_mpi> pwd
/lustre/scratch/sameer/hycom/GLBA0.08/src_2.1.35_26_mpi
GLBA0.08/src_2.1.35_26_mpi> cat select.tau
BEGIN_EXCLUDE_LIST
DPUDPVJ
MXKPRFBIJ
MXKPRFCIJU
MXKPRFCIJV
POFLAT
PROFILE_LAT
ROFLAT
TRIDCOF
TRIDMAT
TRIDRHS
WSCALE
END_EXCLUDE_LIST
GLBA0.08/src_2.1.35_26_mpi>

and to run the program:

% cd EXAMPLE;
% ./GLB_mpi0385.DC.TAU.com

This will create a symlink from hycom.tau in the source directory to the run directory. Remember to edit this file and set the correct paths for:

setenv V 2.1.35
setenv P hycom/GLBA0.08
setenv S /lustre/scratch/sameer/$P/expt_99.7/${OS}/data_${NMPI}x${NOMP}/DC/TAU
setenv D /lustre/scratch/sameer/$P/expt_99.7/DATA

in this script. This sets up the data properly. To run the code:

DC/TAU> pwd
/lustre/scratch/sameer/hycom/GLBA0.08/expt_99.7/XT3/data_385x0/DC/TAU
DC/TAU> cat run.jaguar
#!/bin/tcsh
#PBS -N hycom.expt_99.7.DC.TAU
#PBS -A CSC023
#PBS -m e
#PBS -j oe
#PBS -q batch
#PBS -l walltime=1:30:00
#PBS -l size=193

cd $PBS_O_WORKDIR

yod -sz 385 ./hycom.tau
DC/TAU>
> qsub run.jaguar

NOTE: Remember to remove any ovr* files from this directory prior to running. On p655-4, I have a run script for data_15x1 directory where we do:

> pwd
/mnt/netapp/home/users/sameer/apps/hycom/GLBA0.72/expt_99.8/AIX/data_15x1
> cat run
#!/bin/sh
/bin/rm -f ar* flx* hycom ovr* patch.input sum*
ln -s patch.input_0015 patch.input
cp /mnt/sonic/home/users/sameer/apps/hycom/GLBA0.72/src_2.1.27B_26_mpi/hycom .
export TAU_THROTTLE=1
poe ./hycom -procs 15

On AIX, my config/sp4_mpi file looks like:

include /home/users/sameer/TAU/tau-2.15.4/ibm64/lib/Makefile.tau-mpi-pdt

OPT           = -optVerbose -optTauSelectFile=select.tau -optNoRevert  -optKeepFiles
FC            = $(TAU_COMPILER) $(OPT) $(TAU_F90)