need to update docs on chunking, also increase default buffer sizes

[edwardhartnett]

The docs here need an update: https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_perf_chunking.html

Also the default cache sizes of 10 and 64 MB are laughably small by today's ideas. We can increase them.

[dopplershift]

Sure there's lots of big iron out there, but there's also a rise in tablets and platforms like the raspberry pi. IMO, any bump in default cache sizes should both keep those platforms in mind and come with a decent set of performance benchmarks to justify the change.

[edwardhartnett]

Some points:

• HPC systems typically are not much different from desktop platforms, there are just lots of processors. So each processor will have a generous amount of memory, but not much different than they would have on a linux workstation.
• 64 MB, our current buffer size, is tiny even for rasberry pis and tablets. Tablets typically have gigabytes of memory. Rasberry Pis now have 512 MB.
• This is configure controlled, so those operating netCDF on a small platform have control over this setting. I would not hold back the rest of the netCDF user universe for rasberry pi, most of which, if they are using netCDF, will be using netCDF classic, not netCDF-4/HDF5. Linux and PC workstations are the vast majority of netCDF users. HPC netCDF users also occupy a special place since they are the source for so much data, and the reason everyone uses netCDF on the desktop. (Because the datasets they need are in netCDF, not because they love netCDF.)
• We need on-going performance testing of the netCDF C library. Any assistance in this area would be most welcome. The HDF5 team has set up a nice set of daily performance statistics which is helping them a lot. We need the same. In some sense, every change should be justifying its performance impact. especially if that impact is negative.

[dopplershift]

Some good points. All I'm saying is we shouldn't just blindly assume every platform is capable--that doesn't mean that defaults shouldn't be better optimized.

I would think that if the current defaults are "laughably small", then demonstrating performance improvements with updated defaults should be trivial. Just my $0.02.

[edwardhartnett]

Yes, I see about a 10% improvement by increasing it, at least every circumstance I've measured.

I am doing some work on some NOAA GFS history files (see #1543 ) and I will write up my results, which will include the buffers. I'm also going to contact the HDF5 team for some clarification and advice.

Another thing I need to look into is seeing what happens if I set it too high for a particular machine. Is it ignored? HDF5 errors? Seg fault? I will test and see.

When all that is sorted out I will come back with a PR with some better docs and hopefully some bigger buffers. ;-)

[dopplershift]

10% for "free" is pretty significant.

[edwardhartnett]

After studying this issue and running some benchmarks I believe we should increase the cache to 16 MB from 4 MB. The attached chart shows the improvement in a newly developed benchmarking program which does random record reads of a file. (Reading a file in one pass is not a good way to measure cache performance, as it does not give the cache a chance to do anything.)

I will have a PR up shortly with this change, plus a bunch of documentation improvements and clarifications regarding caching for netCDF-4.

[WardF]

If I wanted to recreate this on a raspberry pi or other 'small' platform, for the sake of curiosity, how would I go about that; is the newly developed bencharking program part of the PR, or completely outside of NetCDF, or somewhere inbetween? I'm also curious what happened when you set it too high, did you see the dreaded segfaults? The results look pretty good.

Thanks Ed, have a good Thanksgiving.

[edwardhartnett]

The benchmarking program is part of the build (after PR is merged). It is nc_perf/tst_bm_rando.c. It will run fine on raspberry pis.

I did not get segfaults for setting this too high. It seemed to be ignored. In any case, 16 MB will not be too high, even for raspberry pi.

[WardF]

I don't expect it will be too high, but I do like running experiments and generating numbers, and it is worth doing to satisfy my curiosity if nothing else :). I'm looking at the PR now, thanks Ed!