unable to use thredds access on Windows, problem with cookies

[jmbeckers]

I try accessing a netcdf file using NCDatasets under Julia, with libraries compiled with mingw compiler (x86_64-w64-mingw32-gcc (GCC) 4.8.5 and NetCDF 4.7.4 (for windows).

It seems there is a problem with a cookiejar that cannot be created (see error messages below).

A look at the function ocinternal.c seems to suggest the directory or file cannot be created

Maybe it is due to a hard coded "/" in the path name which is not working under Windows ?

netcdf-c/oc2/ocinternal.c

Line 565 in b2097a7

strlcat(path,"/",len);

Any suggestion how to make it work would be welcomed.

NCDataset("https://erddap.ifremer.fr/erddap/griddap/SDC_GLO_CLIM_TS_V2_1")
state->auth.curlflags.cookiejar != NULL

Assertion failed: ocpanic(("state->auth.curlflags.cookiejar != NULL")), file ocinternal.c, line 566

signal (22): SIGABRT
in expression starting at REPL[2]:1
crt_sig_handler at /cygdrive/c/buildbot/worker/package_win64/build/src\signals-win.c:92
raise at C:\Windows\System32\msvcrt.dll (unknown line)
abort at C:\Windows\System32\msvcrt.dll (unknown line)
assert at C:\Windows\System32\msvcrt.dll (unknown line)
ocset_curlproperties at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line)
ocopen at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line)
.text at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line)
NCD2_open at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line)
NC_open at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line)
nc_open at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line)
nc_open at C:\Users\jmbeckers.julia\packages\NCDatasets\XVX8L\src\netcdf_c.jl:266
unknown function (ip: 000000005f6f6b8d)
#NCDataset#12 at C:\Users\jmbeckers.julia\packages\NCDatasets\XVX8L\src\dataset.jl:187
NCDataset at C:\Users\jmbeckers.julia\packages\NCDatasets\XVX8L\src\dataset.jl:157 [inlined]
NCDataset at C:\Users\jmbeckers.julia\packages\NCDatasets\XVX8L\src\dataset.jl:157
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
do_call at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:126
eval_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:215
eval_stmt_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:166 [inlined]
eval_body at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:583
jl_interpret_toplevel_thunk at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:731
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:885
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:830
jl_toplevel_eval at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:894 [inlined]
jl_toplevel_eval_in at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:944
eval at .\boot.jl:373 [inlined]
eval_user_input at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:150
repl_backend_loop at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:244
start_repl_backend at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:229
#run_repl#47 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:362
run_repl at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:349
#930 at .\client.jl:394
jfptr_YY.930_35191.clone_1 at C:\Users\jmbeckers\AppData\Local\Programs\Julia-1.7.1\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757
#invokelatest#2 at .\essentials.jl:716 [inlined]
invokelatest at .\essentials.jl:714 [inlined]
run_main_repl at .\client.jl:379
exec_options at .\client.jl:309
_start at .\client.jl:495
jfptr__start_43221.clone_1 at C:\Users\jmbeckers\AppData\Local\Programs\Julia-1.7.1\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
true_main at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:559
jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:701
mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42
BaseThreadInitThunk at C:\Windows\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
Allocations: 722840 (Pool: 722461; Big: 379); GC: 1

[DennisHeimbigner]

The / should get converted to \ by NC_mktmp.

   tmppath = NC_mktmp(path);

I did quite a lot of work on mingw support that will not show up until
the next release. Are you in a position to pull and build the current netcdf-c main
on github and try that?

[DennisHeimbigner]

In any case, one workaround should be this:

1. create a file called .dodsrc in you home directory
2. Insert this line in that .dodsrc file. HTTP>COOKIEJAR=<abs path to some temp file>

So e.g

HTTP.COOKIEJAR=/tmp/netcdf-cookies

[jmbeckers]

In any case, one workaround should be this:

1. create a file called .dodsrc in you home directory

2. Insert this line in that .dodsrc file. `HTTP>COOKIEJAR=<abs path to some temp file>`

So e.g

HTTP.COOKIEJAR=/tmp/netcdf-cookies

Thanks, I tried to translate that into Windows jargon and put a file .dodsrc into
c:\Users\jmbeckers

with content
HTTP.COOKIEJAR=C:\Users\jmbeckers\AppData\Local\Temp\netcdf-cookies

no joy. Same when using / instead of \ in the path to the file, including or not an empty file netcdf-cookies

As for pulling your new version and compiling myself, I would be far out of my comfort zone, as you might guess from my unsuccessful attempt right now.

[DennisHeimbigner]

ok. try putting the .dodsrc in the same directory in which you are working.

[jmbeckers]

yep, that worked ! I have put it in the C:\Users\jmbeckers\AppData\Local\Programs\Julia-1.7.1\bin directory where I launched julia.exe by hand from a cmd window in order to be able to see the error message. This also works if I run it directly with the Julia "icon", but not when I run in Jupyter (probably I need to copy the file also into a directory where Jupyter starts IJulia).
Anyway, thanks

PS: edit, just copied the .dodsrc also into the place where Jupyter starts by default and it works now too.

[DennisHeimbigner]

It is probably that you do not have the environment variable HOME set on windows.

[jmbeckers]

That does not seem to be the case; at least when I search for environmental variable in "Settings", the panel which defines the variables contains no HOME. BTW I tried to include HTTP.COOKIEJAR=C:\Users\jmbeckers\AppData\Local\Temp\netcdf-cookies
as an environmental variable there, but that did not work out either.
The secrets of Windows ...

[Alexander-Barth]

Maybe this is helpful to get the home directory on Windows:
https://stackoverflow.com/a/42700844

I am wondering if NetCDF can also look for USERPROFILE if HOME is undefined (on Windows)?

netcdf-c/libdispatch/ddispatch.c

Line 79 in 93742f8

char* home = getenv("HOME");

(I can make a PR if this is helpful)

[jmbeckers]

%HOMEPATH% points me to \Users\jmbeckers where I first tried to place the .dodsrc file without success.

[DennisHeimbigner]

Netcdf-c uses "HOME" not "HOMEPATH",
But apparently, I should be using the environment variable USERPROFILE instead of HOME.

[CyprienBosserelle]

I had to tweak this workaround to make it work for me. It may not help with isolating the ultimate issue but hopefully that helps other in the same situation as me.

This didn't work form me (with any folder explicitly written)

HTTP.COOKIEJAR=C:\Users\USERNAME\AppData\Local\Temp

I kept getting an error saying i do not have read write access to the folder .

However, somehow, this worked for me

HTTP.COOKIEJAR=$USERPROFILE