weird cdscan error

[doutriaux1]

for @mcenerney1 and travis systems cdscan chokes on missing_value attribute. No idea why. The same cdscan version on my mac and linux systems work on the file that fails for @mcenerney1

see: https://travis-ci.org/UV-CDAT/uvcmetrics/builds/185660014

it's got the log details.

cdscan itself is easy to fix as shown in @dnadeau4 PR

but we need to understand why this error is triggered only on SOME systems.

in case the travis log disappear here the gist of the error:

2: XML NAME: /home/travis/test_data/cam_output/c_t_b30.009.cam2.h0_csad836dce7d9e4045bcb5c184366d8bc0.xml
2: RUNNNIG CDSCAN
2: CDSCAN RUN ERROR len() of unsized object
2: END ERROR LOG
2:   File "/home/travis/miniconda/envs/travis/lib/python2.7/site-packages/metrics/computation/reductions.py", line 2864, in run_cdscan
2:     cdscan.main(cdscan_line)
2:   File "/home/travis/miniconda/envs/travis/lib/python2.7/site-packages/cdms2/cdscan.py", line 1635, in main
2:     cleanupAttrs(attrs)
2:   File "/home/travis/miniconda/envs/travis/lib/python2.7/site-packages/cdms2/cdscan.py", line 464, in cleanupAttrs
2:     if len(attval)==1:
2: cdscan_line= ['cdscan', '-q', '-x', '/tmp/travis/uvcmetrics/c_t_b30.009.cam2.h0_csad836dce7d9e4045bcb5c184366d8bc0.xml', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-01.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-02.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-03.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-04.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-05.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-06.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-07.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-08.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-09.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-10.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-11.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0600-12.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-01.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-02.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-03.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-04.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-05.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-06.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-07.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-08.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-09.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-10.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-11.nc', '/home/travis/test_data/cam_output/c_t_b30.009.cam2.h0.0601-12.nc']

[mcenerney1]

It looks like your travis_fix almost worked. diags_test_06 & meta_diags failed. the rest passed.

[mcenerney1]

Is it possible that there is some cache file or build directory that needs to be deleted?

[doutriaux1]

no it's bad... really really bad... scary bad...

[doutriaux1]

in short the regridder and masking appears to produce different results on your system and travis systems than on any other systems we tested things on. I'm suspecting NC4. But why? I even used travis' won VMs and couldn't reproduce your behavior. IT has to be env variable related...

[mcenerney1]

YIKES!

[williams13]

What Happened?!

…

-Dean From: James McEnerney <notifications@github.com<mailto:notifications@github.com>> Reply-To: UV-CDAT/cdms <reply@reply.github.com<mailto:reply@reply.github.com>> Date: Thursday, December 22, 2016 at 10:09 AM To: UV-CDAT/cdms <cdms@noreply.github.com<mailto:cdms@noreply.github.com>> Cc: Subscribed <subscribed@noreply.github.com<mailto:subscribed@noreply.github.com>> Subject: Re: [UV-CDAT/cdms] weird cdscan error (#70) YIKES! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#70 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACrXKRNkKl4e-z4faVGGmV8nwvrOFhs0ks5rKrzbgaJpZM4LSiZN>.

[mcenerney1]

I've been plagued with a cdscan type problem and Travis's cite has had problems running the metrics test. Charles has been tracking this one and it "appears" they are the same problem. The crazy thing is that it works fine everywhere else. My system has been scrubbed and it still has issues.

[doutriaux1]

we have no idea, that's what's scary. somehow suddenly on travis (and on Jim's machine) cdscan is following a different path. Something with missing_value. It was easy to fix but now the regridder ends up being a bit different more masking. I'm suspecting this is due to missing value being handled a bit differently with the cdscan generated. I tried using travis VMs via docker but I can't reproduce the error there either. I'm suspecting a env issue. We have to figure out why cdscan behaves differently on these two systems...

[doutriaux1]

@mcenerney1 could you yank your anaconda all together and try installing 2.6 not 2.8 and see if the problem persists.

[mcenerney1]

reinstalled anaconda and created new env 2.6_1 with
conda create -n 2.6_1 -c conda-forge -c uvcdat uvcdat=2.6.1
getting
ImportError: No module named vtkGeovisCorePython
What's missing? vtk is
vtk 7.1.0.2.6 uvcdat_master uvcdat

[zshaheen]

@mcenerney1 Dont use the conda-forge channel when installing 2.6.1.

[mcenerney1]

With
conda create -n 2.6_1 -c uvcdat uvcdat=2.6.1
I get import error

import cdms2
Traceback (most recent call last):
File "", line 1, in
File "/Users/mcenerney1/anaconda/envs/2.6_1/lib/python2.7/site-packages/cdms2/init.py", line 17, in
from cdmsobj import CdArray, CdChar, CdByte, CdDouble, CdFloat, CdFromObject, CdInt, CdLong, CdScalar, CdShort, CdString
File "/Users/mcenerney1/anaconda/envs/2.6_1/lib/python2.7/site-packages/cdms2/cdmsobj.py", line 5, in
import cdmsNode
File "/Users/mcenerney1/anaconda/envs/2.6_1/lib/python2.7/site-packages/cdms2/cdmsNode.py", line 10, in
import cdtime
ImportError: dlopen(/Users/mcenerney1/anaconda/envs/2.6_1/lib/python2.7/site-packages/cdtime.so, 2): Library not loaded: @rpath/libjpeg.9.dylib
Referenced from: /Users/mcenerney1/anaconda/envs/2.6_1/lib/libjasper.1.0.0.dylib
Reason: image not found

[durack1]

@doutriaux1 @dnadeau4 it also seems a similar issue is now happening with the systematic xml generation on crunchy - it would appear to be a 2.8 specific issue, as 2.6 seems to work:

So 2.8:

(uvcdat) bash-4.1$ /usr/local/anaconda2/envs/2.8/bin/cdscan -x /export/durack1/Desktop/test2p8.xml /cmip5_css02/data/cmip5/output1/CSIRO-BOM/ACCESS1-0/1pctCO2/fx/ocean/fx/r0i0p0/sftof/1/*.nc
Finding common directory ...
Common directory: /cmip5_css02/data/cmip5/output1/CSIRO-BOM/ACCESS1-0/1pctCO2/fx/ocean/fx/r0i0p0/sftof/1/
Scanning files ...
/cmip5_css02/data/cmip5/output1/CSIRO-BOM/ACCESS1-0/1pctCO2/fx/ocean/fx/r0i0p0/sftof/1/sftof_fx_ACCESS1-0_1pctCO2_r0i0p0.nc
Traceback (most recent call last):
  File "/usr/local/anaconda2/envs/2.8/bin/cdscan", line 1681, in <module>
    main(sys.argv)
  File "/usr/local/anaconda2/envs/2.8/bin/cdscan", line 1635, in main
    cleanupAttrs(attrs)
  File "/usr/local/anaconda2/envs/2.8/bin/cdscan", line 464, in cleanupAttrs
    if len(attval)==1:
TypeError: len() of unsized object

And 2.6:

(uvcdat) bash-4.1$ /usr/local/anaconda2/envs/2.6/bin/cdscan -x /export/durack1/Desktop/test2p6.xml /cmip5_css02/data/cmip5/output1/CSIRO-BOM/ACCESS1-0/1pctCO2/fx/ocean/fx/r0i0p0/sftof/1/*.nc
Finding common directory ...
Common directory: /cmip5_css02/data/cmip5/output1/CSIRO-BOM/ACCESS1-0/1pctCO2/fx/ocean/fx/r0i0p0/sftof/1/
Scanning files ...
/cmip5_css02/data/cmip5/output1/CSIRO-BOM/ACCESS1-0/1pctCO2/fx/ocean/fx/r0i0p0/sftof/1/sftof_fx_ACCESS1-0_1pctCO2_r0i0p0.nc
/export/durack1/Desktop/test2p6.xml written