Problems writing netcdf data - _FillValue error

[durack1]

@doutriaux1 this error has appeared, which doesn't seem obviously wrong (user error) - I'm curious if this is related to #1470:

#!/bin/env python

# -*- coding: utf-8 -*-
"""
Created on Tue Aug  4 11:24:35 2015

@author: durack1
"""
import os
import cdms2 as cdm

# Set outfile criteria
on = 0
cdm.setNetcdfShuffleFlag(on)
cdm.setNetcdfDeflateFlag(on)
cdm.setNetcdfDeflateLevelFlag(on)

# Read data
path = '/work/cmip5/historical/atm/mo/ts/'
inFile = 'cmip5.CCSM4.historical.r1i1p1.mo.atm.Amon.ts.ver-v20120604.latestX.xml'
inFile = os.path.join(path,inFile)
fIn = cdm.open(inFile)
time = fIn.getAxis('time')
slab = fIn('ts',time=('1850','1900'))
print 'slab _FillValue:',slab._FillValue,type(slab._FillValue)
print 'slab type:',slab.dtype

# Write data
outFile = './testGemma.nc'
if os.path.exists(outfile):
  os.remove(outFile)
  print 'outFile removed..'
fOut = cdm.open(outFile,'w')
fOut.write(slab.astype('float32'))
fOut.close()
fIn.close()

Yields the following output:

[durack1@crunchy ~/Desktop]$ testGemma.py
slab _FillValue: 1e+20 <type 'float'>
slab type: float32
cdunif: NetCDF: Not a valid data type or _FillValue type mismatch
If you are trying to write a NetCDF4 type (e.g. int64) please do not turn off cdms_shuffle and cdms_deflate
Traceback (most recent call last):
  File ".//testGemma.py", line 32, in <module>
    fOut.write(slab.astype('float32'))
  File "/usr/local/uvcdat/2015-07-09/lib/python2.7/site-packages/cdms2/dataset.py", line 1796, in write
    id=varid, extend=extend, fill_value=fill_value, index=index)
  File "/usr/local/uvcdat/2015-07-09/lib/python2.7/site-packages/cdms2/dataset.py", line 1721, in createVariableCopy
    setattr(newvar, attname, attval)
  File "/usr/local/uvcdat/2015-07-09/lib/python2.7/site-packages/cdms2/fvariable.py", line 145, in __setattr__
    setattr(self._obj_, name, value)
IOError: cdunif: NetCDF: Not a valid data type or _FillValue type mismatch
If you are trying to write a NetCDF4 type (e.g. int64) please do not turn off cdms_shuffle and cdms_deflate

[doutriaux1]

@durack1 it worked for me 2 minutes ago on crunchy with the same version... Odd.
Although I didn't use the setNeCDFDeflate flags... I think you're right it is the same issue as #1470

[durack1]

Ok @doutriaux1 @dnadeau4 it seems like the type of the _FillValue is getting confused during read/write:

    float ts(time, lat, lon) ;
        ts:standard_name = "surface_temperature" ;
        ts:long_name = "Surface Temperature" ;
        ts:comment = "TS no change, CMIP5_table_comment: \"\"skin\"\" temperature (i.e., SST for open ocean)" ;
        ts:units = "K" ;
        ts:original_name = "TS" ;
        ts:cell_methods = "time: mean (interval: 30 days)" ;
        ts:cell_measures = "area: areacella" ;
        ts:history = "2011-10-21T17:19:45Z altered by CMOR: replaced missing value flag (-1e+32) with standard missing value (1e+20)." ;
        ts:missing_value = 1.e+20f ;
        ts:_FillValue = 1.e+20f ;
        ts:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_CCSM4_historical_r0i0p0.nc areacella: areacella_fx_CCSM4_historical_r0i0p0.nc" ;

So this kludge gets it synced back to a consistent type with slab (float32):

slab._FillValue = np.array(slab._FillValue).astype('f')
fOut.write(slab)

[doutriaux1]

@dnadeau4 does #1524 fix this?

[dnadeau4]

working on it. I have to create a new xml file on my machine. These nc files are big!

[dnadeau4]

yes it does! 👍 😄

[doutriaux1]

awesome! Want to post a PR with a test for this before I close?

[dnadeau4]

I have a test with a 395Mb file and 1 xml file. Not sure how to post this..