Cleanup the handling of cache parameters.

[DennisHeimbigner]

re: Issue #2733

When addressing the above issue, I noticed that there was a disconnect in NCZarr between nc_set_chunk_cache and nc_set_var_chunk cache. Specifically, setting nc_set_chunk_cache had no impact on the per-variable cache parameters when nc_set_var_chunk_cache was not used.

So, modified the NCZarr code so that the per-variable cache parameters are set in this order (#1 is first choice):

1. The values set by nc_set_var_chunk_cache
2. The values set by nc_set_chunk_cache
3. The defaults set by configure.ac

[edwardhartnett]

OK, I think I see the problem here.

IIRC, HDF5 has two different chunk cache settings, one for the file chuck cache (nc_set_chunk_cache) and the variable chunk cache. They are different, but seem to be conflated in this PR.

[DennisHeimbigner]

My understanding was that the file-level chunk cache is only a set of size parameters and there is no
actual file level cache object. It is only variables that have actual cache objects.
Is this incorrect for HDF5? If so, then when is the file chunk cache used?

[WardF]

I'm untangling this today as well to see where MAX_DEFAULT_CACHE_SIZE was being used prior to this. I may re-add it for the full release with a 'deprecated' tag, unless we can determine it serves a separate purpose. I'm just focusing on untying this logic knot around these similarly named variables. Regarding #3067, and the observed performance degradation, I am inclined to make the changes necessary to the default values to restore previous performance, while also making it more clear what the options represent, as need be.

[DennisHeimbigner]

I agree. The first priority is getting performance back to the way it was.