Skip to content

fix: add retry and rate limiting for arXiv API 429 errors#250

Merged
TideDra merged 2 commits into
TideDra:mainfrom
Maxine-1520:fix/arxiv-429-retry
May 16, 2026
Merged

fix: add retry and rate limiting for arXiv API 429 errors#250
TideDra merged 2 commits into
TideDra:mainfrom
Maxine-1520:fix/arxiv-429-retry

Conversation

@Maxine-1520

Copy link
Copy Markdown
Contributor

Closes #248

Problem

When running the GitHub Actions test workflow, the arXiv retriever fetches paper details in batches of 20 via the arXiv API. Consecutive batch requests with no delay trigger HTTP 429 (Too Many Requests) errors, crashing the entire pipeline.

Changes

Modified \src/zotero_arxiv_daily/retriever/arxiv_retriever.py:

  1. Batch-level 429 retry: Each batch (20 papers) retries up to 5 times with linear backoff (30s, 60s, 90s, 120s, 150s) when receiving HTTP 429.
  2. Inter-batch delay: Added 3-second sleep between consecutive batches to reduce rate limit triggering.
  3. Precise exception handling: Only retry on \�rxiv.HTTPError\ with \status == 429; all other errors propagate immediately.

Testing

  • Core test \ est_arxiv_retriever\ passes
  • Successfully deployed and tested via GitHub Actions test workflow
image

@TideDra TideDra merged commit 2e37aad into TideDra:main May 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: arXiv API batch requests trigger HTTP 429 (Too Many Requests)

2 participants