NEK

Background

Link	Code Version	Machine	Date
SVN repo	svn February 2012	Keeneland	February 2012

Build TAU/PDT

For PDT:

./configure ; make ; make install

For TAU:

./configure  -cc=pgcc -c++=pgCC -fortran=pgi -mpi -mpilibrary=-lpgbind -pdt=/lustre/medusa/biersdor/pdtoolkit-3.17/ -pdt_c++=g++
make install 

Building NEK

use the pgi modules:

module load PE-pgi openmpi/1.5.1-pgi

As an example we will build and run for the 3D vortex example:

cd examples/vortex
cp ../../trunk/nek/makenek .
cp ../../trunk/tools/scripts/nekbmpi .

This gives us the scripts needed to build and run NEK.

First edit makenek and change the compiler to the TAU wrapper scripts:

# source path 
SOURCE_ROOT="<path to trunk/nek>"

# Fortran compiler
F77="tau_f90.sh"

# C compiler
CC="tau_cc.sh"

Setup your environment:

export TAU_MAKEFILE=<path to TAU>/x86_64/lib/Makefile.tau-mpi-pdt-pgi
export TAU_OPTIONS="-optPreProcess -optVerbose -optRevert -optTauSelectFile=`pwd`/select.tau"

select.tau

BEGIN_FILE_EXCLUDE_LIST

comm_mpi.*

END_FILE_EXCLUDE_LIST

Compile

./makenek r1854a
make -f makefile

There is a auto-generated makefile, any further changes can be made to this file. For example make sure that -r8 option is passed to the compiler.

Running NEK

If compiling was successful you should have nek5000' executable.

Use the nekbmpi script to run the example:

./nekbmpi r1854a <# of procs>

On 8 processors this example takes just less than one hour.

Performance Data

Mean breakdown:

Major MPI routines across each node:

Profile:

File:Nek vortex 8p.ppk