Breaking API changes in 4.12: RSpace (and other?) integrations broken

[richarda23]

Hi
Since 4th April our acceptance tests for the Java Dataverse SDK are failing running against demo dataverse, I expect this is due to updating to 4.12 . I'm wondering if there are any known changes to the returned data structure from a call to https://demo.dataverse.org/api/v1/dataverses/<dataverseAlias>? From the error messages we are getting it may be something to do with cardinality of dataverseContacts field.

Just more broadly, is there a published changelog of API changes when a new version of Dataverse is released? Is it possible to access a demo version of the previous version (4.11 in this case) for a short time after a new release for comparison?

Thanks Richard

[pdurbin]

@richarda23 hi! I'm sorry to hear we may have broken something. I guess is that pull request #5655 could have changed something you're relying on. I haven't looked deeply at it. If you spot something, please let us know.

When you say Java SDK I assume you mean https://github.com/IQSS/dataverse-client-java . It would be great to have some public continuous integration set up for it. It looks like we talked about this a bit at IQSS/dataverse-client-java#5 . (I have some news I'll share on that issue.)

We don't publish any changelogs (we probably should, especially if you ask the https://keepachangelog.com people 😄 ). We do tag all issues and pull requests with GitHub milestones, if that helps. It's sometimes a lot to wade through.

IQSS generally only has the latest release in production and on our demo site. For public APIs like this I will admit that sometimes I poke around installations that are out there in the community. You can get a list from the bottom of https://dataverse.org/metrics

I hope this helps. Thanks for the bug report!

[poikilotherm]

It should be fairly easy to get a v4.11 instance running with IQSS/dataverse-kubernetes.

[richarda23]

Hi Phil, thanks for quick reply We have a lot of integrations with other software now and aren't really able to follow every single issue for all projects on the offchance that a change might affect the API. An API changelog can be really simple/low effort, we just use a public GoogleDoc for ours to enumerate changes, e.g. https://docs.google.com/document/d/1HjZBnhPF3SQr3_Mt9wCMl625fH5hydifLp_Quo-3bTo/edit <https://docs.google.com/document/d/1HjZBnhPF3SQr3_Mt9wCMl625fH5hydifLp_Quo-3bTo/edit>. Since our software (RSpace ELN) may be connected to different versions of Dataverse at various customer sites, updating to fix to work with 4.12 might then break interactions with 4.11. We'll look into your Kubernetes project, that would be an idea way to test against different versions. Feel free to clone and run integration tests from https://github.com/IQSS/dataverse-client-java <https://github.com/IQSS/dataverse-client-java>, it would be good to have some forewarning if they break against a dev build of future Dataverse releases before it is released so we get some chance to modify our API client code. Cheers Richard

…

On 8 Apr 2019, at 12:19, Philip Durbin ***@***.***> wrote: @richarda23 <https://github.com/richarda23> hi! I'm sorry to hear we may have broken something. I guess is that pull request #5655 <#5655> could have changed something you're relying on. I haven't looked deeply at it. If you spot something, please let us know. When you say Java SDK I assume you mean https://github.com/IQSS/dataverse-client-java <https://github.com/IQSS/dataverse-client-java> . It would be great to have some public continuous integration set up for it. It looks like we talked about this a bit at IQSS/dataverse-client-java#5 <IQSS/dataverse-client-java#5> . (I have some news I'll share on that issue.) We don't publish any changelogs (we probably should, especially if you ask the https://keepachangelog.com <https://keepachangelog.com/> people 😄 ). We do tag all issues and pull requests with GitHub milestones, if that helps. It's sometimes a lot to wade through. IQSS generally only has the latest release in production and on our demo site. For public APIs like this I will admit that sometimes I poke around installations that are out there in the community. You can get a list from the bottom of https://dataverse.org/metrics <https://dataverse.org/metrics> I hope this helps. Thanks for the bug report! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5724 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVoDr7UQdEry7YB_VPJgzluhNM-kBcL-ks5veyXIgaJpZM4ch54D>.

[pdurbin]

@richarda23 thanks, I like your Google doc idea for a change log. Seems lightweight.

Is DataverseOperationsTest.java the one with the failure? Can you tell which test is failing?

[pdurbin]

Wow, RSpace really can't integrate with Dataverse 4.12. I just clicked the "Test" button over at https://community.researchspace.com/apps and got this crazy error:

Here's the text of the error:

Testing connection could not complete:
Status: 500 - server error
There was an error responding to this request, which has been logged.

Something went wrong: JSON parse error: Can not construct instance of com.researchspace.dataverse.entities.DataverseContacts: no String-argument constructor/factory method to deserialize from String value ('philip_durbin@harvard.edu'); nested exception is com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of com.researchspace.dataverse.entities.DataverseContacts: no String-argument constructor/factory method to deserialize from String value ('philip_durbin@harvard.edu') at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@632ff0f7; line: 1, column: 141] (through reference chain: com.researchspace.dataverse.entities.DataverseResponse["data"]->com.researchspace.dataverse.entities.Dataverse["dataverseContacts"]->java.util.ArrayList[0])

Ref: skAqKbtZlN

(Don't worry, I created my API token afterward since I posted it above.)

The next step would be to get a stack trace from a Dataverse server (server.log).

[pdurbin]

Hmm, actually, the 500 error must be on the RSpace side. I don't see anything in server.log on http://phoenix.dataverse.org which is currently running edf1f6d. Here's a screenshot:

[richarda23]

Yes, it could well be some fragility in our JSON parsing which is failing when it shouldn't do - have you made any changes to the model since 4.11?

…

On 8 Apr 2019, at 13:28, Philip Durbin ***@***.***> wrote: Hmm, actually, the 500 error must be on the RSpace side. I don't see anything in server.log on http://phoenix.dataverse.org <http://phoenix.dataverse.org/> which is currently running edf1f6d <edf1f6d>. Here's a screenshot: <https://user-images.githubusercontent.com/21006/55723916-3b0c9880-59d8-11e9-867c-9eaaac5d8f24.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5724 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVoDr-sNhYKcHqK3jNotRhO2pVzehnuiks5vezXngaJpZM4ch54D>.

[pdurbin]

@richarda23 quite possibly but I haven't looked much yet. If you have a moment you can compare these:

Dataverse 4.11: https://dataverse.no/api/dataverses/root

Dataverse 4:12: https://demo.dataverse.org/api/dataverses/demo

I hope I'm giving you the right API endpoint to compare...

[richarda23]

Thanks for the 2 models to compare Looks like dataverseContacts is populated in 4.12. Looking in our Java code, we have DataverseContact as having a field 'contactEmail': public class DataverseContacts { private String contactEmail; } Maybe this was a mistake on our part, and it is just exposed now that the dataverseContacts[] is populated, if in 4.11 it was always empty?

…

On 8 Apr 2019, at 15:07, Philip Durbin ***@***.***> wrote: @richarda23 <https://github.com/richarda23> quite possibly but I haven't looked much yet. If you have a moment you can compare these: Dataverse 4.11: https://dataverse.no/api/dataverses/root <https://dataverse.no/api/dataverses/root> <https://user-images.githubusercontent.com/21006/55730318-e5d78380-59e5-11e9-8dbf-ce34679aa25c.png> Dataverse 4:12: https://demo.dataverse.org/api/dataverses/demo <https://demo.dataverse.org/api/dataverses/demo> <https://user-images.githubusercontent.com/21006/55730328-ea9c3780-59e5-11e9-93da-3ff562a723e2.png> I hope I'm giving you the right API endpoint to compare... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5724 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVoDr3AvoVOWk8iPrA1-KEosnCo4xvXKks5ve00EgaJpZM4ch54D>.

[pdurbin]

@richarda23 I just looked at pull request #5655 again but it's not immediately obvious to me what the intended change was. I also don't see any details in #5583. Standup is in 15 minutes so I'll ask then. I appreciate you digging in on this!

[pdurbin]

it's not immediately obvious to me what the intended change was

... which is exactly what an API changelog would help with, as @richarda23 mentioned above. Here's a screenshot from the API changelog for RSpace from https://docs.google.com/document/d/1HjZBnhPF3SQr3_Mt9wCMl625fH5hydifLp_Quo-3bTo/edit?usp=sharing