It should be our goal for UV-CDAT to allow the user to use any standard Python module. It is a bug if you have to use one build of UV-CDAT for one module, and another build for another module.
I want to use the multiprocessing module. This is part of the Python standard, so it is available in every build of UV-CDAT. Sometimes, but not at the same time, I want to use the py4mpi module. This isn't part of the Python standard, but is widely used, and can optionally be built into UV-CDAT
The problem is that if we build a UV-CDAT with MPI support, then MPI is always running whether you need it or not. This is not a problem on a Macintosh. But on some platforms, OpenMPI (the version we build into UV-CDAT) forbids fork operations, In Python 2.7 and below, the multiprocessing module creates processes by forking them (Python 3.x allows other, slower, methods). So it won't work!
On Rhea.ccs.ornl.gov, I built a UV-CDAT with MPI support, made sure that no Python code imports mpi4py (by deleting it from site-packages/), and then tried to run a script with the multiprocessing module. I got a warning:
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process. Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption. The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.
and when I ran anyway, my code failed sometimes, always during file open operations.
Almost surely, the explanation is that some C code someplace in UV-CDAT is calling MPI_Init. That's necessary if you are using py4mpi, but deadly (on Rhea) if you are using the multiprocessing module.
If the user never imports py4mpi, then we should not call MPI_Init. If the user imports both py4mpi and multiprocessing, then we should call MPI_Init, and issue a warning.
It should be our goal for UV-CDAT to allow the user to use any standard Python module. It is a bug if you have to use one build of UV-CDAT for one module, and another build for another module.
I want to use the multiprocessing module. This is part of the Python standard, so it is available in every build of UV-CDAT. Sometimes, but not at the same time, I want to use the py4mpi module. This isn't part of the Python standard, but is widely used, and can optionally be built into UV-CDAT
The problem is that if we build a UV-CDAT with MPI support, then MPI is always running whether you need it or not. This is not a problem on a Macintosh. But on some platforms, OpenMPI (the version we build into UV-CDAT) forbids fork operations, In Python 2.7 and below, the multiprocessing module creates processes by forking them (Python 3.x allows other, slower, methods). So it won't work!
On Rhea.ccs.ornl.gov, I built a UV-CDAT with MPI support, made sure that no Python code imports mpi4py (by deleting it from site-packages/), and then tried to run a script with the multiprocessing module. I got a warning:
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process. Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption. The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.
and when I ran anyway, my code failed sometimes, always during file open operations.
Almost surely, the explanation is that some C code someplace in UV-CDAT is calling MPI_Init. That's necessary if you are using py4mpi, but deadly (on Rhea) if you are using the multiprocessing module.
If the user never imports py4mpi, then we should not call MPI_Init. If the user imports both py4mpi and multiprocessing, then we should call MPI_Init, and issue a warning.