OpenDAP access with Authentication

[kisabaev]

The original issue is raised here as we access nc files using netcdf4-python, but python community suggested that I should contact netcdf-c. In short, we have been having issues with accessing NC files which require authentication. Both URL and certificate options failed as well as the ncdump. The detailed description is given in the original issue (please also check my last message re being able to download with HTTP GET request using the certificate option, but it still doesn't work with netcdf library). Any suggestions please (URL option seems hassle free)?

[DennisHeimbigner]

After doing this:

nc_prec_url = "http://{}:{}@dap.ceda.ac.uk/badc/ukcp18/data/land-cpm/uk/2.2km/rcp85/12/pr/1hr/latest/pr_rcp85_land-cpm_uk_2.2km_12_1hr_20671101-20671130.nc".format(username, password)
Could you print out the value of nc_prec_url? (no need to use real user+pwd)
just so I can that it formatted correctly?
Also, do the same with the real user+pwd and put that url into
a web browser with ".dds" appended.
Something like:
http://user:pwd@dap.ceda.ac.uk/badc/ukcp18/data/land-cpm/uk/2.2km/rcp85/12/pr/1hr/latest/pr_rcp85_land-cpm_uk_2.2km_12_1hr_20671101-20671130.nc.dds
Tell me if the browser can read it.

[kisabaev]

The .dds appended URL is returning 404 page and the URL format is correct with username and password. I have created a temporary account which I will delete later to test stuff on. Could you please have a look with the following account and NC url above? Many Thanks!

u = "tmpuser"
p = "tmppass"

[DennisHeimbigner]

So when I attempt to use the url with those user+pwd, I get this error:

Error:curl error: Peer certificate cannot be authenticated with given CA certificates
Are you using a self-signed certificate or something?

p.s. The command I used was this:

ncdump -h 'http://tmpuser:tmppwd@dap.ceda.ac.uk/badc/ukcp18/data/land-cpm/uk/2.2km/rcp85/12/pr/1hr/latest/pr_rcp85_land-cpm_uk_2.2km_12_1hr_20671101-20671130.nc#log&show=fetch'

[kisabaev]

Ok I have tried the same command without single quotes around the URL (and no log and show at the end of the URL), which produces this error:

NetCDF: Malformed or inaccessible DAP DDS

using this command:
ncdump -h http://tmpuser:tmppwd@dap.ceda.ac.uk/badc/ukcp18/data/land-cpm/uk/2.2km/rcp85/12/pr/1hr/latest/pr_rcp85_land-cpm_uk_2.2km_12_1hr_20671101-20671130.nc

Either way, what does this mean? username and password are correct, how should I go about fixing the error? Thanks!

[DennisHeimbigner]

The #log&show-fetch is important because it can give more information about
the details of the error. So are you seeing the certificate error or something else?
ps. I think the user and pwd are not the problem. the tmpuser+tmppwd worked
ok for me.

[kisabaev]

The command with single quotes is returning this error:

(Invalid argument 'show' is not recognized as an internal or external command,
operable program or batch file.

After removing the single quotes - the error message is like before (but still complains that show is not recognized). Do I need to install any additional libraries for show to work?

[DennisHeimbigner]

Ok, I need to know what operating system your are using: Windows, Linux, Mac?
Next, try the command again with double quotes (") instead of single quotes (').
When you get the output, make sure it includes the following:

1. the exact command you typed
2. the first, say, 15 lines of output after you type the command - this is important
   because this is where the important error informaton will appear

[kisabaev]

double quotes seem to have worked with show parameter (the OS is Windows by the way)...

ncdump -h "http://tmpuser:tmppwd@dap.ceda.ac.uk/badc/ukcp18/data/land-cpm/uk/2.2km/rcp85/12/pr/1hr/latest/pr_rcp85_land-cpm_uk_2.2km_12_1hr_20671101-20671130.nc#log&show=fetch"

but the error message is familiar - here is the full output:

[DennisHeimbigner]

I turns out you cannot use that url to get the data you want directly.
Rather it leads to a login page that you need to manually enter a user+pwd.
and then attempts to go to the full specified web page.
You can see this by putting the URL into a web-browser.
The netcdf code is assuming that the URL goes directly to the data, but it does not
so that is why you get all that HTML returned. If you read it, you can see that it
is a login page. Netcdf does not know how to handle such a page.
You can walk the URL step by step until you get to the page named "latest"
and then if you click on, say,

pr_rcp85_land-cpm_uk_2.2km_12_1hr_20670101-2067..> 25-Feb-2020 19:02 849676562

then the web browser will read the .nc file ask if you want to down load it.

If you want to do the equivalent using ncdump, then you will need to find
some other URL that does not go thru the login page.
In any case, these are not OPeNDAP files, but rather raw netcdf files.

[kisabaev]

Hi, yes apparently you were right - those were direct download URLs! I found out the actual OpenDAP URL in this page. however it's generating the same error message:

ncdump -h "http://tmpuser:tmppwd@dap.ceda.ac.uk/thredds/dodsC/badc/ukcp18/data/land-cpm/uk/2.2km/rcp85/12/pr/1hr/latest/pr_rcp85_land-cpm_uk_2.2km_12_1hr_20671101-20671130.nc#log&show=fetch"

Although this is a different URL - it's taking to the same login page whilst tried in browser, ncdump is producing the same error - do you think it's OpenDAP URL? I'm just confused why OpenDAP URL is also not working...

Here is the link which actually lists various URLs for the same dataset including OpenDAP. For a logged in user, the new OpenDAP URL is returning 400 error message (JSON) whereas previous URL (which you have spotted) was downloading NC file (for logged in users) - so definitely these URLs are different.

[DennisHeimbigner]

As you note, that DodsC (opendap) URL is still going thru the login page
hence the same error.
Until you can figure out an alternative URL that avoids that page, there is not
much I can do to help. Sorry.

[DennisHeimbigner]

One more thought See if that python script
creates a file /Users//.dodsrc or /Users//.daprc.
If so, please send it to me with any sensitive passwords XXXX'd out.

[kisabaev]

Thanks for not giving up just yet! I'm attaching .dodsrc file together with .certs folder (the certificate is encrypted - so passwords are fine)
opendap_credentials.zip

[DennisHeimbigner]

I have contacted the CEDA help desk to see if they can provide any info.
Also, since this started as python, did you try modifying that python script
example to see if it worked for your data?

[kisabaev]

Thanks so much! Yes the script generates the credentials certificate I sent in the earlier message, but it doesn't work with netcdf library, but it seemed to work with simple HTTP GET request - so certificate itself seems to work... It's the same error message with the new OpenDAP URL - I will send the full script tomorrow.