Skip to content

Feature: Upload external datasets#112

Merged
dandelany merged 5 commits intodevelopfrom
feature/upload-external-datasets
Oct 23, 2024
Merged

Feature: Upload external datasets#112
dandelany merged 5 commits intodevelopfrom
feature/upload-external-datasets

Conversation

@duranb
Copy link
Copy Markdown
Contributor

@duranb duranb commented Oct 7, 2024

Partially resolves NASA-AMMOS/plandev-ui#1424

Refer to UI PR for testing procedure.

@duranb duranb requested review from a team as code owners October 7, 2024 21:49
Comment thread src/packages/plan/plan.ts Fixed
Comment thread src/packages/plan/plan.ts Fixed
Comment thread src/packages/plan/plan.ts Fixed
@dandelany dandelany added the publish Tells GH to publish docker images for this PR label Oct 14, 2024
Copy link
Copy Markdown
Collaborator

@dandelany dandelany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good - a few non-blocking notes below, but I'm going to approve and merge this for today's release. I've tested locally with the examples and all seems to be working correctly. We still need to document this feature and potentially add some better tests & coverage for some edge cases/handling for other data types, but this is a first pass at an experimental feature that users are asking for, so I think it's worth releasing v1 as is. Thanks @duranb !

Comment thread src/packages/plan/plan.ts
// Create a lookup for the profile name's index in each CSV row
const headerIndexMap: Record<string, number> = parsedCSV[0].reduce(
(prevHeaderIndexMap: Record<string, number>, header: string, headerIndex: number) => {
if (new RegExp(timeColumnKey).test(header)) {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to note here is that this will loosely match any header that contains the substring time_utc, eg. event_time_utc will match. I think this is OK here since there shouldn't be any ambiguity, since there should only be one timestamp column & the others should be numeric, but worth calling out & considering if it should be a strict equality match

Comment thread src/util/time.test.ts
expect(getTimeDifference('2024-245T00:01:00.0', '2024-245T12:02:00.0', 6)).toEqual(43260000000);
expect(getTimeDifference('2024-243T00:01:00.0', '2024-245T12:02:00.0', 6)).toEqual(216060000000);
});
});
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could use some more unit tests here for better coverage of the parsing functions in time.ts

@dandelany dandelany merged commit dbcac54 into develop Oct 23, 2024
@dandelany dandelany deleted the feature/upload-external-datasets branch October 23, 2024 01:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

publish Tells GH to publish docker images for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Load data into Aerie from UI

4 participants