netcdf-C 4.6.1 build error on macOS 10.13: ocprint link fails

[mathomp4]

Environment Information

• What platform are you using? (please provide specific distribution/version in summary)
  • Linux
  • Windows
  • OSX
  • Other
  • NA
• 32 and/or 64 bit?
  • 32-bit
  • 64-bit
• What build system are you using?
  • autotools (configure)
  • cmake
• Can you provide a sample netCDF file or C code to recreate the issue?
  • Yes (please attach to this issue, thank you!)
  • No
  • Not at this time

Summary of Issue

My build of netcdf-C 4.6.1 on macOS 10.13.5 isn't quite working. I'm building with GCC 8.1.0 (built from source) and Open MPI 3.1.0. I built HDF5 with --enable-parallel so mpicc is my C compiler in this stage.

From my scan of the build log, libnetcdf.a is made, but when ocprint tries to link I get:

libtool: link: mpicc -o ocprint ocprint.o  ../liblib/.libs/libnetcdf.a -L/Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/lib /Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/lib/libhdf5_hl.a /Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/lib/libhdf5.a /Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/lib/libmfhdf.a /Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/lib/libdf.a /Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/lib/libsz.a /Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/lib/libjpeg.a /Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/lib/libcurl.a -lz -ldl -lm
Undefined symbols for architecture x86_64:
  "_ncrc_globalstate", referenced from:
      _createtempfile in libnetcdf.a(liboc_la-ocinternal.o)
      _ocset_curlproperties in libnetcdf.a(liboc_la-ocinternal.o)
      _NC_rcload in libnetcdf.a(libdispatch_la-drc.o)
      _NC_set_rcfile in libnetcdf.a(libdispatch_la-drc.o)
      _rccompile in libnetcdf.a(libdispatch_la-drc.o)
      _rclocate in libnetcdf.a(libdispatch_la-drc.o)
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:1046: ocprint] Error 1
make[4]: *** Waiting for unfinished jobs....

As a test, I took netcdf-C 4.5.0 (which had worked before on this machine) and tried building it instead (feeding in the exact same libraries that went into the build above) and ocprint, etc all built just fine.

As a second test, I built with --disable-dap instead (see original configure line below) and it will build because ocprint is avoided. But, this netCDF build is for a model and one of the requirements for the netCDF built is that it have OpenDAP capability, thus I can't disable dap for production.

Any ideas on how to fix this with autotools? I thought I'd troll around the CMakeLists.txt files to see if ocprint was being built/linked with extra libraries perhaps with CMake. When I did so I see:

# Apparently fails under cmake
#set(ocprint_FILES ocprint.c )
#ADD_EXECUTABLE(ocprint ${ocprint_FILES})
#TARGET_LINK_LIBRARIES(ocprint oc2 ${ALL_TLL_LIBS})

(which is in the oc2/CMakeLists.txt, not the ncdump/CMakeLists.txt).

Huh. Indeed, the autotools build seems to build things that CMake doesn't (e.g., nc4print which doesn't seem to appear in any CMakeLists.txt file in the tarball...and isn't installed, just built?).

Is there a way to get autotools build to skip ocprint? I don't think I need it (I've never used it and I know of no one else who has), but it would at least let me get a (partially) DAP-enabled netCDF built. (NOTE: I had thought about using a CMake build, but I'd probably need more help with that. I don't really know how to translate all of that configure line to CMake, especially the --includedir line. I didn't see in the INSTALL.md on how to translate that flag.)

Steps to reproduce the behavior

My configure line is:

  $ ./configure --prefix=/Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin --includedir=/Users/mathomp4/Baselibs/ESMA-Baselibs-5.1.3-netCDF461/x86_64-apple-darwin17.7.0/gfortran/Darwin/include/netcdf --enable-hdf4 --enable-dap --enable-parallel-tests --disable-shared --disable-examples --enable-netcdf-4 CC=mpicc FC=mpifort CXX=mpic++ F77=mpifort

And, as I said before, I can build 4.5.0 just fine using all the same infrastructure, but 4.6.1 won't.

[edhartnett]

Do you have the capability to build from the git repo?

That is, can you clone from git, and then run autoreconf -i to build the configure script and the Makefile.in files?

If so, you can comment out the following lines in ncdump/Makefile.am:

# Conditionally build the ocprint program, but do not install
if ENABLE_DAP
bin_PROGRAMS += ocprint
ocprint_SOURCES = ocprint.c
endif

If you don't know how to do this, let me know and I will do a special dist for you without these lines.

Seems like if a program is not going to be installed, there is no reason for it to be in the build. @DennisHeimbigner can we move ocprint to one of the test directories? Or build it only with a special configure flag?

[mathomp4]

@edhartnett I learned some autoreconf -i myself earlier. The HPC netCDF doesn't supply a configure in its release tarball, so I had to build it with autoreconf. :)

I can before I run configure do:
sed -i -e '/ocprint program/,+4 s/^/#/' netcdf/ncdump/Makefile.am
or the like in my main make script.

And, technically, ocprint is indeed installed (when built) because I see it in the bin directory when I build my Baselibs. There is just some bug somewhere that I'm triggering when it's built.

The program not installed (in autotools land) is nc4print, at least. But ocprint is definitely not built in CMake as far as I can see from the CMakeLists.txt.

[WardF]

Good catch, I'm wiring in ocdump to cmake right now. It will be merged in with the latest PR group.

[mathomp4]

Okay. These two lines before I run configure:

sed -i -e '/ocprint program/,+4 s/^/###/' ncdump/Makefile.am
sed -i -e '/ENABLE_DAP_TRUE/ s/^/###/' ncdump/Makefile.in

stop ocprint from building. Makefile.am alone didn't seem to do it. (And, of course, sed must be GNU sed, so I need to harden my make script to make sure sed == gsed on a Mac.)

Once I do that, netCDF builds to completion.

[DennisHeimbigner]

There are a couple of files and programs that are in the github repo
purely for purposes of helping me debug stuff. Both ocprint and nc4print
are examples (as well as cf, cf.cmake, and Make0). As a rule
those executables should be built by automake, but not by cmake
because I tend to do my debugging with automake builds.
If ocprint is being installed, then it probably should be marked as noinst_...

[WardF]

Oh, good to know, I will mark it as such in the next group of PR's.

[edhartnett]

ocprint is having a build problem, not an install problem.

Suggest that you move whatever is not part of the library to a test directory. Then they user may see a test failure, but they will still be able to do make all and get the library built.

Just marking this as noinst_ will not solve the problem, I believe.

[DennisHeimbigner]

Can someone try the build on a mac using normal gcc instead of mpicc?

[WardF]

Testing now.

[WardF]

I'm not having any problems (using the current master) when building it on my end with cmake and gcc/clang/mpich, on OSX.

[DennisHeimbigner]

So is this a false alarm?

[WardF]

I’ve fixed the issue I was seeing. Let me test against 4.6.1.

[DennisHeimbigner]

I am losing the track. I thought you said it worked fine. What issue were you
seeing?

[WardF]

The change I added so that it builds with cmake works fine. That change is in master. I see the original question was about 4.6.1, so I want to see if I can duplicate the issue in that version.