Issue creating zarr from datasets with string variables

[malmans2]

I'm having troubles creating a NCZarr from datasets with string variables.

Here is the test dataset: test_string_dim.nc.zip
Here are the commands that illustrate the problem:

>> nc-config --version
netCDF 4.8.1

>> ncdump -h test_string_dim.nc
netcdf test_string_dim {
dimensions:
	dim2 = 9 ;
	dim3 = 10 ;
	time = 20 ;
	dim1 = 8 ;
variables:
	double dim2(dim2) ;
		dim2:_FillValue = NaN ;
	string dim3(dim3) ;
	int64 time(time) ;
		time:units = "days since 2000-01-01 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
	double var1(dim1, dim2) ;
		var1:_FillValue = NaN ;
		var1:foo = "variable" ;
	double var2(dim1, dim2) ;
		var2:_FillValue = NaN ;
		var2:foo = "variable" ;
	double var3(dim3, dim1) ;
		var3:_FillValue = NaN ;
		var3:foo = "variable" ;
		var3:coordinates = "numbers" ;
	int64 numbers(dim3) ;
}

# Error
>> nccopy test_string_dim.nc file://test_string_dim.zarr#mode=nczarr
NetCDF: Invalid argument
Location: file ; line 2121

# Bypass error dropping dim3 variable
>> nccopy -V var1,var2,var3,time,dim2 test_string_dim.nc file://test_string_dim_no_dim3.zarr#mode=nczarr
>> ncdump -h file://test_string_dim_no_dim3.zarr#mode=nczarr
netcdf test_string_dim_no_dim3 {
dimensions:
	dim2 = 9 ;
	dim3 = 10 ;
	time = 20 ;
	dim1 = 8 ;
variables:
	double dim2(dim2) ;
		dim2:_FillValue = NaN ;
	int64 time(time) ;
		time:units = "days since 2000-01-01 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
	double var1(dim1, dim2) ;
		var1:_FillValue = NaN ;
		var1:foo = "variable" ;
	double var2(dim1, dim2) ;
		var2:_FillValue = NaN ;
		var2:foo = "variable" ;
	double var3(dim3, dim1) ;
		var3:_FillValue = NaN ;
		var3:foo = "variable" ;
		var3:coordinates = "numbers" ;
	int64 numbers(dim3) ;
}

[DennisHeimbigner]

Unfortunately, the Zarr spec does not support variable length strings.
I am been looking at what can be done with fixed length strings, but
it is difficult.
Can you get by using characters, something like this?

char dim3(dim3,maxchars) ;

[malmans2]

Thanks! I didn't realise that.

I don't have a real use case at the moment, I ran into this while playing with xarray's test data.
This is the original dim3 generated by xarray:

array(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'], dtype='<U1')

I looked into netcdf4-python documentation, and I think the problem with this dtype is indeed explained in section Dealing with strings.

If it's of use, I'm attaching the zarr data generated by xarray: test_string_dim.xrzarr.zip

[DennisHeimbigner]

As you know, the U type refers to the NumPy 32 bit unicode encoding; that is each character is
stored as a 32 bit value. It is very unlikely that we would support that encoding any time soon.
I have a branch in which I am trying to simulate the netcdf-4 string type using fixed length strings.
If you are interested, I will let you know when it is ready for testing.

[malmans2]

Sure, I'm happy to give it a go when it's ready for testing!