From Tau Wiki
Jump to: navigation, search

Link Code Version Machine Date
Sandia website git repo Keeneland May 2012

Building with TAU

To start copy over the file Makefile.keeneland to the src/MAKE directory. Then do,

configure TAU:

./configure -mpi -cuda=<cuda dir> -pdt=<pdt dir>
make install

build LAMMPS:

cd src
make yes-user-cuda
cd ../lib/cuda
make lib
cd ../../src
export TAU_MAKEFILE=<path to TAU>/x86_64/lib/Makefile.tau-icpc-mpi-pdt
export TAU_OPTIONS='-optVerbose -optShared'
make keeneland


cd examples/USER/cuda

Then choose either method depending on your choice of MPI implementation:


Running with MPI on Keeneland by doing:

qlogin -V -l nodes=2:ppn=3
mpirun --mca btl_openib_flags 1 --mca mpi_leave_pinned 0 --mca \
 btl_openib_warn_default_gid_prefix 0 -npernode 3 -np $NPROCS_3 --hostfile ${PBS_NODEFILE} \
 tau_exec -T mpi,cupti,icpc -cupti \
 /lustre/medusa/biersdor/mylammps/src/lmp_keeneland_tau_wgpu -cuda on -sf cuda < in.melt_2.5.cuda


As of this writing the USER/CUDA package requires mpich2 to run on more than one node. Here are the commands to get started:

qlogin -V -l nodes=2:ppn=3
mpdboot --totalnum=2 --ncpus=3 -f mpich_machines.txt 
cat mpich_machines.txt
mpirun -np 6 tau_exec -T icpc,mpi,pdt,cupti,mpich2 -cupti /lustre/medusa/biersdor/mylammps/src/lmp_keeneland_tau_mpich2 -sf cuda -cuda on < in.melt_2.5.cuda

Performance results

Lammps profile.png File:Lammps 3gpu.ppk

Lammps trace.png