API endpoint for retrieving guestbook responses times out when used to get responses from certain collections

[jggautier]

What steps does it take to reproduce the issue?

• When does this issue occur?
  When using the Retrieve Guestbook Responses for a Dataverse Collection endpoint described at https://guides.dataverse.org/en/6.9/api/native-api.html#retrieve-guestbook-responses-for-a-dataverse-collection to get the responses of guestbooks of certain collections

• Which page(s) does it occurs on?
  The results of the endpoint

• What happens?
  The endpoint doesn't produce the guestbook responses of certain collections, like IFPRI, atlas, and restat collections, and instead times out.

  Endpoint runs for about a minute...
  

  ... and times out
  

  The endpoint does produce guestbook responses of other collections, like the speed collection.

• To whom does it occur (all users, curators, superusers)?
  Superusers and managers of collections

• What did you expect to happen?
  The endpoint returns the guestbook responses

Which version of Dataverse are you using?
v6.9

Any related open or closed issues to this bug report?

• Unable to download certain guestbooks, possibly related to number of responses (~150,000 responses) #7767

[jggautier]

For collections with too many guestbook responses, their managers aren't able to get the collections' guestbook responses through their browsers, because the Download All Responses button on the Manage Dataset Guestbooks page doesn't work.

So we created the Retrieve Guestbook Responses for a Dataverse Collection endpoint so that they could use the API to get their collections' guestbook responses. See #7767 and its PR at #7931.

The endpoint times out when trying to get guestbook responses of collections in Harvard Dataverse with the most guestbook responses:

• https://dataverse.harvard.edu/dataverse/pkgw
• https://dataverse.harvard.edu/dataverse/Kencorpus
• https://dataverse.harvard.edu/dataverse/held-extracted-data
• https://dataverse.harvard.edu/dataverse/the_review
• https://dataverse.harvard.edu/dataverse/qje
• https://dataverse.harvard.edu/dataverse/data_ncov
• https://dataverse.harvard.edu/dataverse/JPE
• https://dataverse.harvard.edu/dataverse/IFPRI
• https://dataverse.harvard.edu/dataverse/restat

The endpoint does work for collections that have fewer responses. It seems like if the collection has two million or more responses, the endpoint times out, but I can't be sure about this or in general about how related the number of responses is to the time outs.

[ekraffmiller]

While we are working on this, it would also be good to support paging. We will need it to display guestbook responses in the SPA.

[jggautier]

I was wondering how this might affect the SPA

[jggautier]

Just also noting that for at least a few months now it's also felt like this endpoint takes longer to run in general, even when trying to get the responses of guestbooks with way fewer than 2 million responses.