OpenDAP access with Authentication

[kisabaev]

I have been trying to access NC file which requires authentication using the URL option suggested here.

import netCDF4
username = "username" # use your own for testing
password = "password" # use your own for testing

nc_prec_url = "http://{}:{}@dap.ceda.ac.uk/badc/ukcp18/data/land-cpm/uk/2.2km/rcp85/12/pr/1hr/latest/pr_rcp85_land-cpm_uk_2.2km_12_1hr_20671101-20671130.nc".format(username, password)

ds = netCDF4.Dataset(nc_prec_url)

However, I'm getting:

OSError: [Errno -72] NetCDF: Malformed or inaccessible DAP DDS

If I'm doing something wrong, what is the correct way to access NC file which requires Authentication please? Many Thanks!

[jswhit]

This is a question for the netcdf-c developers (https://github.com/Unidata/netcdf-c/issues or support-netcdf@unidata.ucar.edu)

[jswhit]

To verify that it's a netcdf-c issue, can you try it with ncdump? You could also try putting your username and password in the .daprc file (see https://www.unidata.ucar.edu/software/netcdf/docs/auth.html) - but be sure to make it readable only by you.

[kisabaev]

I will try .daprc route and let you know how I get on.

I have seen ncdump being mentioned on other posts - but still haven't found any use cases. Apologies I'm not a regular nc user! It doesn't seem to be a direct method of netCDF4.ncdump and it's not a python library (so example or a document would be much appreciated!). This post is also showing a 404 page (seems to be the closest thing I found on the subject). Thank you so much for engaging with my question!

[jswhit]

ncdump is a utility that comes with the c library
https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_utilities_guide.html#ncdump_guide

Just try ncdump -h <url>

[kisabaev]

It's the same error message with ncdump as well.

Apparently I had .dodsrc file already set up in HOME directory as part of manual script configuration (function tried to set up a certificate) - the content of .dodsrc is:

    HTTP.COOKIEJAR=./dods_cookies
    HTTP.SSL.CERTIFICATE=C:\Users\myusername/.certs\credentials.pem
    HTTP.SSL.KEY=C:\Users\myusername/.certs\credentials.pem
    HTTP.SSL.CAPATH=C:\Users\myusername/.certs\ca-trustroots

Do I have everything needed in .dodsrc file as this is also not working whilst trying
ds = netCDF4.Dataset(nc_prec_url) # url has no username and pass this time

The only way I could make it work was by downloading the NC file using the above certificate via HTTP GET request which is good news as certificate is working, but the whole dataset is almost 1 Tb to download locally (which is doable) - but the worst part is that the download just hangs after 5-6 files which must be due to their server (and there are over 1000 files to download) - hence accessing the data with netCDF4.Dataset would be best without downloading NC files. Any ideas why HTTP GET request download works (so certificate probably works?), but netcdf library is failing to access the data?

CREDENTIALS_FILE_PATH = r"C:\Users\myusername/.certs\credentials.pem"
nc_prec_url = "http://dap.ceda.ac.uk/badc/ukcp18/data/land-cpm/uk/2.2km/rcp85/12/pr/1hr/latest/pr_rcp85_land-cpm_uk_2.2km_12_1hr_20671101-20671130.nc"
response = requests.get(nc_prec_url, cert=(CREDENTIALS_FILE_PATH), verify=False)
nc_prec_file = nc_prec_url.rsplit('/', 1)[-1]
with open(nc_prec_file, 'wb') as file_object:
    file_object.write(response.content) 

[jswhit]

I'm not really familiar with the DAP authentication stuff in netcdf-c. If you can't find it in the docs, I would suggest asking over at netcdf-c