Add support for dates and times to the CSV ingest

[eaquigley]

---

Author Name: Leonid Andreev (@landreev)
Original Redmine Issue: 3885, https://redmine.hmdc.harvard.edu/issues/3885
Original Date: 2014-04-24
Original Assignee: Kevin Condon

---

From the parent master ticket:

Last, request to add support for date type.

[eaquigley]

---

Original Redmine Comment
Author Name: Leonid Andreev (@landreev)
Original Date: 2014-04-24T22:10:39Z

---

Support for dates and times has been added.
An entire column must be made of uniformly parseable date or time values (and, possibly, missing values).
Variables recognized as times and dates will be marked as such in the "variable metadata" DDI.
Plus, following the DVN 2-3.* convention, the actual values will not be quoted in the tab file.

The only recognized formats are:

for dates:

yyyy-MM-dd

for times:

yyyy-MM-dd HH:mm:ss z (with time zone)
yyyy-MM-dd HH:mm:ss (without)

[eaquigley]

---

Original Redmine Comment
Author Name: Leonid Andreev (@landreev)
Original Date: 2014-04-25T15:34:03Z

---

attached a couple of simple files with parseable dates and times.

[eaquigley]

---

Original Redmine Comment
Author Name: Leonid Andreev (@landreev)
Original Date: 2014-04-29T21:59:11Z

---

taking back the ticket

[eaquigley]

---

Original Redmine Comment
Author Name: Leonid Andreev (@landreev)
Original Date: 2014-05-01T00:32:48Z

---

Kevin,
I took this, and other CSV tickets back yesterday, after you told me CSV ingest wasn't working.
I can't reproduce any issues with it today (tested on my own build and dvn-build).
I'm guessing what you saw may have been another problem caused by the issue I fixed yesterday. It was the result of switching from the datafile object to the filemetadata, for storing some file information, such as file names.
I really can't think of anything else.
I'm moving this back into QA; and I'm giving it to you, since you were already looking into it.
please let me know if there are any other issues you are seeing.

Do note that unlike all the other supported file formats, a CSV file needs the ".csv" extension to be recognized as ingestable as tabular data. I did mention this in the file type recognition ticket; and it needs to be documented somewhere too.
(there's no good way of recognizing a CSV file really, except for reading and parsing every text file in its entirety and counting the fields).

[eaquigley]

---

Original Redmine Comment
Author Name: Kevin Condon (@kcondon)
Original Date: 2014-05-01T20:22:27Z

---

Tested on 5/1

Basic dates and times work, along with missing values.

I need to check the z designation since Z in iso 8601 means UTC time or zulu time. The way that spec, which UNF uses, specifies timezones is to convert to UTC and then use a numeric offset:

http://en.wikipedia.org/wiki/ISO_8601

Combined date and time in UTC:
2014-04-29T13:16:30+00:00
2014-04-29T13:16:30Z

I will also check the DDI

[eaquigley]

---

Original Redmine Comment
Author Name: Leonid Andreev (@landreev)
Original Date: 2014-05-01T20:55:38Z

---

Sorry about the confusion; this was meant to be "z" as in, Java time format specification; which is just short for "zone" and means, "zone goes here".
It is not the same as literal "Z" attached to the time string, indicating that it's a UTC time...
this would be an example of a recognized format with the timezone:
2013-09-19 16:23:12 EST
also, attaching a simple file that worked for me.

[eaquigley]

---

Original Redmine Comment
Author Name: Kevin Condon (@kcondon)
Original Date: 2014-05-01T22:40:46Z

---

Tested on 5/1

Works as described.

Closing ticket