Refactor csv export by chrwm · Pull Request #401 · OpenEnergyPlatform/open-MaStR

chrwm · 2022-12-15T22:13:29Z

Closes #400

The feature has been depreciated from Markstammdatenregister

* rename self.--- objects * move constants to constants.py

* was not tested

* rename self.--- objects * move constants to constants.py

Can not be improved as e.g. `orm.BasicUnit.EinheitMastrNummer == orm_tables["unit_data"].EinheitMastrNummer` EinheitenMastrNummer has to be called with .

* all csv tables are exported via one export function * all db queries are created in one function `create_db_query`. The structure of the function should enable easy tables joins in the future for csv export within on session. * additional tables can be now also exported with a limit * in constants.py ORM_MAP was extended. ORM_MAP maps the parameters that the user inserts to the tables classes in orm.py. This means additional tables for export can be easily added here, if desired. * metadata saving was moved to its own function

* generalize csv-export logging info

…xport # Conflicts: # open_mastr/soap_api/mirror.py

chrwm · 2023-01-05T14:51:37Z

A short summary of the most important changes:
3 main functions exist:

csv-export is triggered via MaStR.to_csv(): defines and collects tables to export
open_mastr.utils.helpers.create_db_query: creates db-queries for TECHNOLOGIES and ADDITIONAL_TABLES
open_mastr.utils.helpers.db_query_to_csv: exports db-query to csv for TECHNOLOGIES and ADDITIONAL_TABLES

functions relevant to the csv_export are now located in helpers.py

Pytest
Only open_mastr.utils.helpers.db_query_to_csv is tested for:

The test checks for 2 random tech and 2 random additional tables:
1. If csv's have been created and encoded in 'utf-8' and are not empty
2. For techs, if limited (limit=60) EinheitMastrNummer in basic_units are included in CSV file
3. For additional_tables, if csv is not empty

It is due to the Markstammdatenregister has unstable columns (columns are renamed, new columns are added, columns are deleted). This makes testing with static columns impractical in my opinion.
@FlorianK13 @deniztepe if you have ideas for better testing, please make suggestions.

chrwm · 2023-01-05T16:38:36Z

Volkswagen was not necessary ;)

deniztepe

Overall it works well. I guess, the commented three points are small issues.

Currently metadata is only and can only be created for technologies. Will be enabled for additional_tables in #386

* _db_path was getting longer if the folder has more files * black applied

chrwm · 2023-01-16T16:01:18Z

@deniztepe you can approve, if I met your suggestions.

nesnoj · 2023-01-20T07:27:21Z

Hey @chwrwm, I'd like to reference the package prior to the CSV refactoring. However, the latest version 0.12.1 does not include all recent fixes.
It would be very nice if you could prepare an in-between-release (at 1ba8c7c) without the CSV stuff.
Is that possible for you? If not, I could take take of this..

nesnoj · 2023-01-20T09:03:14Z

Hey @chwrwm, I'd like to reference the package prior to the CSV refactoring. However, the latest version 0.12.1 does not include all recent fixes. It would be very nice if you could prepare an in-between-release without the CSV stuff. Is that possible for you? If not, I could take take of this..

Hmm, maybe it is not necessary. Did the CSV format change with the refactoring?

chrwm added 8 commits December 15, 2022 19:41

Remove StatistikFlag from orm.py #400

4c216c7

Delete parameter statistik_flag #400

f5b12bb

The feature has been depreciated from Markstammdatenregister

Move csv export related functions to helpers.py #400

6ed2679

* rename self.--- objects * move constants to constants.py

Move constants to constants.py #400

39e8e8d

Move csv export to helpers.py #400

72f7c83

Adjust mastr.py to new csv export #400

639fa2d

Update script #400

7a8c016

* was not tested

Update imports #400

3757c34

* rename self.--- objects * move constants to constants.py

chrwm added the ♻️ refactor Refactoring of existing code for better maintainability label Dec 15, 2022

chrwm self-assigned this Dec 15, 2022

chrwm changed the title ~~Refactor 400 csv export~~ Refactor csv export Dec 15, 2022

chrwm mentioned this pull request Dec 15, 2022

Allow custom output path specification #402

Merged

chrwm added 16 commits December 16, 2022 17:55

Refactor orm table and subtables selection #400

32faee7

Add parameter validation for csv_export #400

4667b95

Rename metadata creation function #400

752a154

Refactor table join #400

b7350a3

Can not be improved as e.g. `orm.BasicUnit.EinheitMastrNummer == orm_tables["unit_data"].EinheitMastrNummer` EinheitenMastrNummer has to be called with .

Specify index_label for csv export #400

371dd64

Add table naming for additional_tables #400

20f9d5d

* generalize csv-export logging info

Clean imports #400

7263ec2

Update docstring of db_query_to_csv #400

0b35f1d

Update docstring of save_metadata #400

f3a7015

Update docstring of create_db_query #400

3b61a6e

Update docstring of to_csv #400

2f497d9

Add conftest.py to collect fixtures #400

145fbe5

Refactor for easier unit testing #400

e0c9cf0

Update test_db_query_to_csv #400

08071c3

Merge remote-tracking branch 'origin/develop' into refactor-400-csv-e…

f783470

…xport # Conflicts: # open_mastr/soap_api/mirror.py

chrwm marked this pull request as ready for review January 5, 2023 14:25

Update CHANGELOG.md #400

aa76050

Skip test_db_query_to_csv if db is smaller than 1MB #400

46f5105

chrwm requested review from FlorianK13 and deniztepe January 5, 2023 16:38

chrwm mentioned this pull request Jan 6, 2023

Feature 403 SQLA 2 compatibility #405

Merged

deniztepe requested changes Jan 12, 2023

View reviewed changes

Comment thread open_mastr/utils/config.py

Comment thread open_mastr/utils/helpers.py

Comment thread tests/test_helpers.py Outdated

chrwm added 2 commits January 12, 2023 16:03

Disable save_metadata #400

dc88399

Currently metadata is only and can only be created for technologies. Will be enabled for additional_tables in #386

Update db empty check #400

fe25dff

deniztepe mentioned this pull request Jan 13, 2023

Update config files if necessary #410

Closed

2 tasks

Edit db empty check to scan folder #400

092383c

* _db_path was getting longer if the folder has more files * black applied

deniztepe self-requested a review January 17, 2023 07:07

deniztepe approved these changes Jan 17, 2023

View reviewed changes

chrwm merged commit 95e32a0 into develop Jan 19, 2023

chrwm deleted the refactor-400-csv-export branch January 19, 2023 21:54

FlorianK13 removed their request for review January 20, 2023 09:04

nesnoj mentioned this pull request Jan 20, 2023

CSV export fails: KeyError: 'additional_table' #414

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor csv export#401

Refactor csv export#401
chrwm merged 29 commits intodevelopfrom
refactor-400-csv-export

chrwm commented Dec 15, 2022

Uh oh!

chrwm commented Jan 5, 2023

Uh oh!

chrwm commented Jan 5, 2023

Uh oh!

deniztepe left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrwm commented Jan 16, 2023

Uh oh!

nesnoj commented Jan 20, 2023 •

edited

Loading

Uh oh!

nesnoj commented Jan 20, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

chrwm commented Dec 15, 2022

Uh oh!

chrwm commented Jan 5, 2023

Uh oh!

chrwm commented Jan 5, 2023

Uh oh!

deniztepe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrwm commented Jan 16, 2023

Uh oh!

nesnoj commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nesnoj commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nesnoj commented Jan 20, 2023 •

edited

Loading

nesnoj commented Jan 20, 2023 •

edited

Loading