FITS File Upload: no longer extracting metadata

[jggautier]

Harvard Dataverse and Demo Dataverse, both on v4.14, aren't extracting metadata from FITS files anymore.

On Harvard Dataverse, it looks like the last published file where metadata was extracted was on June 19, 2018. Since then, no uploaded file whose file type is FITS has had its metadata extracted.

#2904 is related

[jggautier]

In this dataset on Demo Dataverse I tested publishing files whose metadata Harvard Dataverse has and has not extracted. When I open the three files in a text editor, I can see the metadata in the headers.

[meghangoodchild]

In this dataset, I tried uploading a FITS file and the metadata was not extracted. It is also not working on our instance at Scholars Portal Dataverse (currently on 4.17). Would anyone be able to look into this? Thanks!

[djbrooke]

@meghangoodchild - we'll bring this into a sprint and fix this in a release soon.

[qqmyers]

FYI : https://github.com/IQSS/dataverse/blob/develop/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java#L169 looks suspicious. The next few lines take the storage identifier and strip any tmp:// prefix beofre turning it into a file path. Line 169 doesn't strip the prefix before prepending the tempfile dir and then sends that to the metadata extractor.

I just saw this while working on other things and haven't tested to see if this is really the issue.

[qqmyers]

Probably not the real issue - just saw that at least single file uploads don't have tmp:// at the start of their storageidentifiers in this method, so this may be a red herring or only a secondary problem (presumably some files have tmp:// in their storageidentifiers when they go through this method?).

[djbrooke]

• In the short term we'd like to restore the functionality to the way it was before
• Longer term, once we have better support for file level metadata, we can remove the hacky extraction in place right now (stuffing it into the description, storing it in dedicated database tables and rolling it up to the dataset metadata) and investigate moving that to the representations that we want
• Is this possibly related to when we moved to S3?