Home

Introduction

Prerequisites

1. Macintosh
2. GNU Compilers, Fortran optional. This should compile on other compilers, but has not yet been tested with them.
3. OpenMPI (optional)
4. Python 2.7 or 3

It should not be hard to port Everytrace to other Unixes, other compilers and other MPI libraries.

Installation

Configure and build with CMake. For example:

mkdir build; cd build
cmake \
    -DCMAKE_CXX_COMPILER=g++ \
    -DCMAKE_C_COMPILER=gcc \
    -DCMAKE_Fortran_COMPILER=gfortran \
    -DCMAKE_INSTALL_PREFIX:PATH=$MACPORTS/opt/everytrace \
    -DUSE_FORTRAN=YES \
    -DUSE_MPI=YES \
..
make install

Configure USE_FORTRAN and USE_MPI depending on whether you wish to use the resulting library with Fortran or MPI.

Building with Everytrace

1. Compile all your source code with the following flags, based on your compiler:

• GCC/gfortran: -g. (Does not affect performance of -O flags.)
• Intel ifort: -g -traceback.

1. Call everytrace_init() at the beginning of your program:

   // code
   include <everytrace.h>
   int main(int argc, char **argv) {
       everytrace_init()
       ...
   }
   

or alternatively in Fortran: program myprogram use everytrace call everytrace_init ... end program

1. Make sure signal handlers are not being installed anywhere else in your program --- or in any libraries you might be using (eg, PETSc).

2. Set up your program and libraries to call everytrace_exit(int retcode) when an error has occurred and the program must exit:

   include <everytrace.h>
   if (theres_a_problem) {
       fprintf(stderr, "There is a problem!\n");
       everytrace_exit(17);
   }
   

or in Fortran: use iso_fortran_env, only : error_unit if (theres_a_problem) then write(error_unit,*) 'There is a problem' call everytrace_exit(17) end if In many cases, this will involve installing configurable error handlers in the various libraries you use (NetCDF, PETSc, etc). In some cases, it might require modification of the libraries to enable configurable error handlers.

1. Link lib/everytrace_refaddr_gcc.o into every shared library you create.

2. Link to the appropriate Everytrace library, depending on your program:

• No Fortran (C/C++ only): libeverytrace_c.so
• No Fortran (C/C++ only) + MPI: libverytrace_c_mpi.so
• Has Fortran: libeverytrace_cf.so
• Has Fortran + MPI libeverytrace_cf_mpi.so

Note that if your program is mixed C/C++ and Fortran, then you should use the appropriate Fortran version of the library.

Running with Everytrace

1. Set the EVERYTRACE environment variable:

   export EVERYTRACE=1
   

2. Run with mpilabel, for example:

   nice mpirun -np 6 mpilabel <my-mpi-program> [my-args] >log 2>&1 &
   tail -f log
   

3. When it crashes, analyze the log file:

   everytrace log
   

Example output: ``` ------------ e0 0x107a822f6 0x1035feced 0x1035fecbc 0x1035feb32 0x7fff83eb65a9 glint2_modele_init0 (in libglint2.dylib) (glint2_modele.cpp:374) _liglint2_mod_MOD_new_glint2_modele (in libmodele.dylib) (LIGlint2.F90:148) lisnow_mod_MOD_alloc (in libmodele.dylib) (LISnow.F90:160) alloc_drv_atm (in libmodele.dylib) (ATM_DRV.f:801) giss_modele (in libmodele.dylib) (MODELE.f:423) modele_maindriver (in libmodele.dylib) (MODELE_DRV.f:39) 0x100701e79 ------------ e1 0x1075d02f6 0x103144ced 0x103144cbc 0x103144b32 0x7fff83eb65a9 glint2_modele_init0 (in libglint2.dylib) (glint2_modele.cpp:374) _liglint2_mod_MOD_new_glint2_modele (in libmodele.dylib) (LIGlint2.F90:148) lisnow_mod_MOD_alloc (in libmodele.dylib) (LISnow.F90:160) alloc_drv_atm (in libmodele.dylib) (ATM_DRV.f:801) giss_modele (in libmodele.dylib) (MODELE.f:423) modele_maindriver (in libmodele.dylib) (MODELE_DRV.f:39) 0x100249e79 ```` In this example, stacktraces were retrieved from two MPI processes, rank 0 and 1. Hexidecimal lines in the stacktraces are from libraries that were not instrumented with everytrace_refaddr_gcc.o. Note the smooth transition between traditaional Fortran, Fortran 2003 and C++ stackframes.

Misc

Notes