I have bit of code which i have added aiometer. What iam trying to understand is how best to optimize it to get the maximum performance. The API iam calling has a rate limit of 300 requests per minute hence setting the max_per_second to 5 (if i understand this correctly) should be within the limits however, i hit 429's even then. My code is below:
async def async_iterate_paginated_api(
function: Callable[..., Awaitable[Any]], **kwargs: Any
) -> AsyncGenerator[List[Any], None]:
"""Return an async iterator over the results of any API."""
page_number = 1
response = await function(**kwargs, pageNumber=page_number)
yield response.get("entities")
total_pages = response.get('pageCount')
pages = [
page
for page in range(2, total_pages + 1)
]
async def process(function: Callable[..., Awaitable[Any]], kwarg: Any, page: Any):
response = await function(**kwarg, pageNumber=page)
return response.get('entities')
async with aiometer.amap(functools.partial(process, function, kwargs), pages, max_per_second=5) as results:
async for result in results:
yield result
Any feedback would be helpful.
I have bit of code which i have added aiometer. What iam trying to understand is how best to optimize it to get the maximum performance. The API iam calling has a rate limit of 300 requests per minute hence setting the max_per_second to 5 (if i understand this correctly) should be within the limits however, i hit 429's even then. My code is below:
Any feedback would be helpful.