Skip to content

Best practice for handling rate limits #34

@iserialize

Description

@iserialize

I have bit of code which i have added aiometer. What iam trying to understand is how best to optimize it to get the maximum performance. The API iam calling has a rate limit of 300 requests per minute hence setting the max_per_second to 5 (if i understand this correctly) should be within the limits however, i hit 429's even then. My code is below:

async def async_iterate_paginated_api(
    function: Callable[..., Awaitable[Any]], **kwargs: Any
) -> AsyncGenerator[List[Any], None]:
    """Return an async iterator over the results of any API."""
    page_number = 1
    response = await function(**kwargs, pageNumber=page_number)
    yield response.get("entities") 
    total_pages = response.get('pageCount')
    pages = [
        page
        for page in range(2, total_pages + 1)
    ]
    async def process(function: Callable[..., Awaitable[Any]], kwarg: Any, page: Any):
        response = await function(**kwarg,  pageNumber=page)
        return response.get('entities')

    async with aiometer.amap(functools.partial(process, function, kwargs), pages, max_per_second=5) as results:
        async for result in results:
            yield result

Any feedback would be helpful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions