Implement the DbImporter for the MPDS API (Pauling file)#1238
Merged
sphuber merged 2 commits intoMar 7, 2018
Conversation
nmounet
previously approved these changes
Mar 7, 2018
sphuber
force-pushed
the
fix_1237_implement_mpds_dbimporter
branch
from
2c4fd55 to
67c9a12
Compare
The MPDS API only supports a single output format for a query. Either the result of the query is returned as a 'json' object or the result is a concatenation of string formatted cif files. To import CifData nodes, however, we need both the json object to retrieve the required source information, but we also need the raw cif file, as we do not want to reconstruct the structure or cif ourselves from the basic structural data that is provided in the json. As a workaround, we fire the exact same query twice, once asking the result in the json format and the other in the cif format. We add the cif string to the json result entries by cross referencing the source id that is present in both the json entry and the raw cif string. A special MpdsCifEntry will then use that cif string to directly set the contents attribute. This will then prevent a separate HTTP request to the source uri to retrieve the cif content, which would also result in a 429 HTTP error due to too many requests being fired
sphuber
force-pushed
the
fix_1237_implement_mpds_dbimporter
branch
from
67c9a12 to
fd2034b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1237
The MPDS API only supports a single output format for a query. Either
the result of the query is returned as a 'json' object or the result
is a concatenation of string formatted cif files. To import CifData
nodes, however, we need both the json object to retrieve the required
source information, but we also need the raw cif file, as we do not
want to reconstruct the structure or cif ourselves from the basic
structural data that is provided in the json. As a workaround, we
fire the exact same query twice, once asking the result in the json
format and the other in the cif format. We add the cif string to the
json result entries by cross referencing the source id that is present
in both the json entry and the raw cif string. A special
MpdsCifEntrywill then use that cif string to directly set the contents attribute.
This will then prevent a separate HTTP request to the source uri to
retrieve the cif content, which would also result in a 429 HTTP error
due to too many requests being fired