-32767 value for status code

[ampernito]

I got some values of -32767 in status_code. The particles having this status also don't have the right coordinate values, mass etc. It shows this value 9.9692e+36. Interestingly, I only got this problem from the NetCDF certain solution. The corresponding uncertain solution seems fine.

I ran my model online (WebGNOME https://gnome.orr.noaa.gov/#config ) and I exported my particles as NetCDF. Then processed it in MATLAB.

[ChrisBarker-NOAA]

hmm -- this is very curious.

Did it look OK in the visulaization?

Can you share your input files and configuration?

Ideally, a .gnome file, as you get if you go to WebGNOME: and Download you model setup:

Usually, the coordinate values being off is due to a bad value in your drivers -- winds or currents.

But with so much off -- particularly status_codes, that's very odd.

Thinking now, I'm suspecting something with the netcdf file being corrupted -- but why ???

Is it possible that your model ran out of data before the run was finished? we only recently added the ability to output the netcdf in that case -- and it's possible there's an edge-case we missed.

Though I just tested it, and it worked fine for me, so ???

[ampernito]

Yes it went well in the map visualization.

You may access the files through this link

https://drive.google.com/drive/folders/12WOpB_RuP_gb8gaDeH3ROWrrlrC_mLFJ?usp=sharing

I also did 2 separate runs with the same inputs but using one mover only [i.e. (1) wind only (2) currents only] but I did not encounter the problem.

[ChrisBarker-NOAA]

Ive downloaded your files, and found:

1. I see the problem in your netCDF files
2. I can replicate the problem running your save file (that's good news, if we can replicate it, we may be able to fix it

What I've figured out:

The strange values you are seeing are the netCDF "_FillValue" -- that's the value that the netCDF library fill a variable with when its created. So having those values in the output means that those particular data were not set -- the question is why???

And it's particularly odd that it doesn't affect the uncertainty elements.

In the Python NEtCDF library, the values set to _FillValue are "Masked" in a masked array -- so it's a bit more subtle when you have this issue.

I noticed something else a bit strange: the total number of data points is different in the certain and uncertain output -- that could be explained by particles going off map -- but I'd expect the uncertain particles to be more likely to go off the map.

Now that I can replicate it -- I"ll dig deeper.

But any more you can figure out about exactly which data are "bad" would be helpful.

I wrote a little script to report the issue for various files -- what you provided, whet I got running your save file, and the what I got running your save file, but outputting only every hour.

**********

For file: MTPE_010_netcdf_from_ampernito/MTPE_010_netcdf.nc
number of timesteps:  2977
<class 'netCDF4.Variable'>
int16 status_codes(data)
    long_name: particle status code
    flag_values: [ 0  2  3  7 10 12 32]
    flag_meanings: 0:not_released 2:in_water 3:on_land 7:off_maps 10:evaporated 12:to_be_removed 32:on_tideflat
unlimited dimensions: data
current shape = (7431271,)
filling on, default _FillValue of -32767 used
Number of masked elements: 3452

**********

For file: MTPE_010_netcdf_from_ampernito/MTPE_010_netcdf_uncertain.nc
number of timesteps:  2977
<class 'netCDF4.Variable'>
int16 status_codes(data)
    long_name: particle status code
    flag_values: [ 0  2  3  7 10 12 32]
    flag_meanings: 0:not_released 2:in_water 3:on_land 7:off_maps 10:evaporated 12:to_be_removed 32:on_tideflat
unlimited dimensions: data
current shape = (7431275,)
filling on, default _FillValue of -32767 used
Number of masked elements: 0

**********

For file: MTPE_010_netcdf_local/MTPE_010_netcdf.nc
number of timesteps:  2977
<class 'netCDF4.Variable'>
int16 status_codes(data)
    long_name: particle status code
    flag_values: [ 0  2  3  7 10 12 32]
    flag_meanings: 0:not_released 2:in_water 3:on_land 7:off_maps 10:evaporated 12:to_be_removed 32:on_tideflat
unlimited dimensions: data
current shape = (7431271,)
filling on, default _FillValue of -32767 used
Number of masked elements: 3452

**********

For file: MTPE_010_netcdf_local/MTPE_010_netcdf_uncertain.nc
number of timesteps:  2977
<class 'netCDF4.Variable'>
int16 status_codes(data)
    long_name: particle status code
    flag_values: [ 0  2  3  7 10 12 32]
    flag_meanings: 0:not_released 2:in_water 3:on_land 7:off_maps 10:evaporated 12:to_be_removed 32:on_tideflat
unlimited dimensions: data
current shape = (7431275,)
filling on, default _FillValue of -32767 used
Number of masked elements: 0

**********

For file: MTPE_010_netcdf_hour_timestep/MTPE_010_netcdf.nc
number of timesteps:  745
<class 'netCDF4.Variable'>
int16 status_codes(data)
    long_name: particle status code
    flag_values: [ 0  2  3  7 10 12 32]
    flag_meanings: 0:not_released 2:in_water 3:on_land 7:off_maps 10:evaporated 12:to_be_removed 32:on_tideflat
unlimited dimensions: data
current shape = (1859688,)
filling on, default _FillValue of -32767 used
Number of masked elements: 861

**********

For file: MTPE_010_netcdf_hour_timestep/MTPE_010_netcdf_uncertain.nc
number of timesteps:  745
<class 'netCDF4.Variable'>
int16 status_codes(data)
    long_name: particle status code
    flag_values: [ 0  2  3  7 10 12 32]
    flag_meanings: 0:not_released 2:in_water 3:on_land 7:off_maps 10:evaporated 12:to_be_removed 32:on_tideflat
unlimited dimensions: data
current shape = (1859692,)
filling on, default _FillValue of -32767 used
Number of masked elements: 0

[ampernito]

Thank you for taking the time to investigate this case. The following are my other observations.

1. I also noticed the difference in the total number of data points. I set the number of particles to 5000 but the certain output has a total of 5001 unique IDs.

2. Also, from what I've observed not all particles where released by the end of the run. The last row of the p_count variable shows that only 4979 particles where released at the last time step.

3. I'm not sure if I got the timeline right because I re-arranged the ragged array based on particle count and time, which results to fewer values than the actual data points. But the "_FillValue" are occurring at the end of the simulation, probably the last 7 days of the run.

[ampernito]

Update and latest observations

I tried to export multiple netCDF outputs at single time step for every 12:00 PM of 10 random dates (i.e. "Mar05","Mar06","Mar09","Mar12","Mar19","Mar20","Mar21","Mar23","Mar26","Mar28").

1. 6 out of 10 netCDF files (from March 12 to March 28) show an unequal number of particle counts between the certain and uncertain outputs. In some instances, the certain output contains more elements than the uncertain one, and in others, the reverse is true.

2. No "_FillValue = -32767" was observed from the status values.

3. No strange latitude and longitude values.

[ampernito]

Hi just an update. I exported best guess particles only and found no errors in status codes.

[ChrisBarker-NOAA]

Thanks! sorry we haven't had a chance to dig deeper into this -- but that's a good hint. Somehow the uncertainty elements may be messing something up ...

Still pretty mysterious though!