Speed up NCDEFAULT_get/put_vars code

[DennisHeimbigner]

It occurs to me that we should consider improving
the performance of NCDEFAULT_get/put_vars since they
are still being used for some dispatch tables.

The current implementations operate by reading/writing
one element at a time, which is seriously inefficient.

We should explore some alternative implementations such as this.

1. Instead of reading element at a time, we allocate some larger
   block of memory that covers multiple strided elements at a time.
2. the stride code then reads one of these blocks and extracts the
   relevant strided elements.
3. Cost is memory of the block (per-call to get_vars). - should
   we keep a blockcache
4. For writing, we would read the block, then insert the
   strided elements, and then write out the whole block.

Anyone have any other suggestions for speed up?

[wkliao]

This strategy is referred as "data sieving" in MPI-IO.
https://www.mcs.anl.gov/~thakur/papers/romio-coll.pdf

[DennisHeimbigner]

Thanks for the pointer.

[edhartnett]

I believe the only layer using the default vars is the HDF4 layer, which, as I recall, does not have a vars style function. It only uses the read, since HDF4 is read-only for netCDF.

[DennisHeimbigner]

I think the libsrc, and indirectly libdap2 dispatchers also use the default. Do not know
about pnetcdf.
[Added: also the user defined dispatch tests use it, which probably means
that many UDF implementations will also.]

[wkliao]

The PnetCDF driver in NetCDF calls PnetCDF library directly, so no.

[DennisHeimbigner]

How does pnetcdf implement get_vars? Can we profitably use that code
for our default implementation?

[wkliao]

Remember PnetCDF calls MPI-IO :)
So, it relies on MPI-IO to do data sieving.
If you like, I can point you to the codes in ROMIO that do the sieving.

[DennisHeimbigner]

Oh right. That pointer into ROMIO might be helpful, thanks.

[wkliao]

At first, you are warned the data sieving code in ROMIO is not for the faint of heart :)

In MPICH github repo, file src/mpi/romio/adio/common/ad_write_str.c implements the write request to contiguous or noncontiguous file space. Data sieving is used for the noncontiguous case. The temporary buffer used to read-modify-write is named writebuf while the user's write buffer is buf. The whole read-modify-write is from line 341 to line 455. The kernel is a C macro named ADIOI_BUFFERED_WRITE defined at line 11.

The read data sieving is in file src/mpi/romio/adio/common/ad_read_str.c

[DennisHeimbigner]

ok, maybe I will have to do it from scratch. I will see how bad it is.

[edhartnett]

I didn't realize that this was used for libsrc. That alone is good enough reason to improve it! ;-)

The pnetcdf layer calls the pnetcdf functions ncmpi_get|put_vars_*, so does not use this code.