Amazon Ads API Report Polling Suddenly Failing: Reports Not Completing

[akum441]

Issue: Amazon Ads API Report Polling Was Working, But Now Reports Stay in PENDING

We are experiencing an issue while polling the Amazon Ads API to check the status of reports.
The polling mechanism was working fine before, but for the last 3 days, reports are not completing on time, and our retry logic stops before fetching the data.

API & Retry Mechanism Details:
API Endpoint: https://advertising-api.amazon.com/reporting/reports/:reportId
Expected Behavior: Reports should eventually move from PENDING to COMPLETED, allowing us to retrieve the data.
Actual Behavior: Since the last 3 days, reports remain stuck in PENDING for an unusually long time.

Error Log:

Amazon Ads API Logs 

Timestamp: 2025-01-31T14:29:22.724+05:30
{
    "timestamp": "2025-01-31 08:59:22,724",
    "level": "INFO",
    "message": "Attempting to execute process to get reports.",
    "logger": "ingestion",
    "func_name": "execute",
    "line_no": 256,
    "file_name": "amazon_ads.py",
    "thread_name": "MainThread",
    "process": 1
}

Timestamp: 2025-01-31T14:29:22.724+05:30
{
    "timestamp": "2025-01-31 08:59:22,724",
    "level": "INFO",
    "message": "Attempting to generate access token.",
    "logger": "ingestion",
    "func_name": "get_access_token",
    "line_no": 83,
    "file_name": "amazon_ads.py",
    "thread_name": "MainThread",
    "process": 1
}

Timestamp: 2025-01-31T14:29:22.812+05:30
{
    "timestamp": "2025-01-31 08:59:22,812",
    "level": "INFO",
    "message": "Iteration started",
    "logger": "ingestion",
    "func_name": "retry_until_condition_satisfied_reportVersion",
    "line_no": 203,
    "file_name": "request_util.py",
    "thread_name": "MainThread",
    "process": 1
}

Timestamp: 2025-01-31T14:29:22.947+05:30
{
    "timestamp": "2025-01-31 08:59:22,946",
    "level": "INFO",
    "message": "Reports are not yet completed at server end.",
    "logger": "ingestion",
    "func_name": "retry_until_condition_satisfied_reportVersion",
    "line_no": 209,
    "file_name": "request_util.py",
    "thread_name": "MainThread",
    "process": 1
}

Retry Logic Using tenacity:

 @classmethod
    @tenacity.retry(
        retry=tenacity.retry_if_exception_type(RetryConditionUnsatisfiedError)
        or tenacity.retry_if_exception_type(RetryException),
        stop=tenacity.stop_after_attempt(MAX_RETRY_ATTEMPT),
        wait=tenacity.wait_exponential(multiplier=MULTIPLIER, max=MAX_WAIT_TIME),
        reraise=False,
        retry_error_callback=on_retry_error,
    )
    def retry_until_condition_satisfied(
        cls,
        method: str,
        url: str,
        condition: t.Callable[..., bool],
        params: t.Dict[str, t.Any] = {},
        retry_error_codes: t.List[int] = [],
    ) -> t.Any:
        logger.info("Iteration started")
        response = requests.request(method, url, **params)

        if response.status_code in API_SUCCESS_STATUS_CODES:
            if condition(response=response):
                return response
            logger.info("Reports are not yet completed at server end.")
            raise RetryConditionUnsatisfiedError(
                {
                    "errorMessage": "Reports are not yet completed at server end.",
                    "errorCode": response.status_code,
                    "response": response,
                },
            )

        if len(retry_error_codes) > 0:
            RETRY_ERRORS.extend(retry_error_codes)

        if response.status_code in RETRY_ERRORS:
            raise RetryException(
                {
                    "errorCode": response.status_code,
                    "response": response,
                    "errorMessage": f"API failed with {response.status_code} :: Error Text: {response.text}",
                },
            )

        raise RequestError({"errorCode": response.status_code, "response": response})

Issues Faced in the Last 3 Days:

1. Reports No Longer Move to COMPLETED:
   Previously, the function retried until the report was ready.
   Now, the report remains in PENDING, and retries fail.
2. Possible Amazon API Changes?
   We haven't changed anything in our implementation.
   Has Amazon changed report processing times or introduced new limitations?
3. Unclear Wait Time Impact on Retries:
   Our settings should allow retries with an exponential backoff up to 244 seconds max.
   Could something in tenacity’s retry mechanism be preventing it from retrying long enough?

Questions:

1. Has Amazon Ads API changed its report processing times in the last few days?
2. Why do reports now stay in PENDING indefinitely, despite successful retries?
3. What is the best way to handle long-running reports in Amazon Ads API?
4. Any insights on API changes, debugging retry behavior, or potential workarounds would be greatly appreciated!