Structure of open-MaStR

[FlorianK13]

We came up with two possible structures for the open-MaStR:
Version 1:

Data from the MaStR can either be downloaded by API or by the full data XML dump. Data from both sources should be formatted the same as early as possible, in the socalled extract file. From there on, the same process happens with both downloads: the data is cleansed, meaning that obvious errors are removed. This results in the raw file, which is in sqlite format. This is then read into the docker MaStR database. Different modular functions take the database from docker and do different kinds of preprocessing.

Version 2:

Here, the download from the API / the XML files are written into the docker database as soon as possible into tables with the ending _raw. Different cleansing functions start automatically to change this raw data. The user can then decide whether to run different postprocessing functions, which will generate new tables in the database with the ending _cleaned.

Right now, we tend to implement version 2.

[henhuy]

Thank you for your update.
I also tend to version 2.
My idea was to make the processing and storage of data database-agnostic. Thus, let the user decide if he wants to use postgres or sqlite (this would be very simple - only sqlalchemy connection URL had to be changed), or to use Docker (in the end this is an URL too, but docker has to be set up first). As default, I would recommend Sqlite, as this does not need any dependency and is shipped with python (nice!) and I think all processing steps would be possible with SQlite as well. If this should become an API or something I would prefer postgres for sure (don't use sqlite in production)...

Your code seems to be nicely parsing the XML into DB. Next step would be to have the same table structure for both XML and API download. Right now, I don't know API download structure yet and cannot tell the difference to the simple tables we get from current XML Parser? Maybe the structure should be considered while parsing the XML.

[henhuy]

Just took a look into API output.
Only difference I think, is that Anlagen and Einheiten are merged into one table.
If we are implementing this within the XML Parser, we would meet same structure, I guess.

Another thing we should integrate from my parser is replacing FKs to Katalogwerte/einheiten with the actual value. This is a simple function.

[FlorianK13]

So you would argue for a Version 3:

Both the API and xml download write an sqlite databse. The user can then decide to output the sqlite database or maybe a csv file, or the sqlite databse is written to a docker postgres. Here, the postprocessing can take place (which needs postgis).

[FlorianK13]

I would imagine the following code structure for the usage of the package:

from open_mastr.mastr import Mastr

Database = Mastr()
Database.download(method = {'API','bulk'})
Database.to_docker()
Database.postprocess()

I added a skeleton version of the Mastr class to #202 feature branch in the folder open_mastr. Maybe it is good if you check the structure there and add your thoughts on what those functions should do and if additional functions are needed?

[FlorianK13]

The new structure is implemented and will be released in 0.11.0