Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AsyncRetriever] - Partition router is not working when using the AsyncRetriever #51533

Open
MuriloBelar opened this issue Jan 14, 2025 · 0 comments

Comments

@MuriloBelar
Copy link

Topic

The request parameters defined in the partition router are not being sent in creation_requester

Relevant information

I've started using the AsyncRetriever to create a low code connector in order to integrate a data source.

I have 2 inputs in my configuration file and I'm also using the incremental sync + start_time_option to insert a date as parameter, none of the these variables are being sent to my first request.

Although, when I hard code this parameters in my base request the connector works, the ideal scenario is to use the partitions.

This is my retriever code:

retriever:
  type: AsyncRetriever
  status_mapping:
    failed:
      - '201'
    running:
      - '202'
    timeout:
      - '203'
    completed:
      - '200'
  urls_extractor:
    type: DpathExtractor
    field_path:
      - url
  record_selector:
    type: RecordSelector
    extractor:
      type: DpathExtractor
      field_path:
        - data
  status_extractor:
    type: DpathExtractor
    field_path:
      - status
  creation_requester:
    $ref: "#/definitions/base_requester"
    path: userAdRevenueReport
    http_method: GET
  polling_requester:
    $ref: "#/definitions/base_requester"
    path: userAdRevenueReport
    http_method: GET
  download_requester:
    type: HttpRequester
    path: "{{stream_slice['url']}}"
    url_base: ""
    http_method: GET
  partition_router:
    - type: ListPartitionRouter
      values: '{{ config[''application''] }}'
      cursor_field: application
      request_option:
        type: RequestOption
        field_name: application
        inject_into: request_parameter
    - type: ListPartitionRouter
      values: '{{ config[''platform''] }}'
      cursor_field: platform
      request_option:
        type: RequestOption
        field_name: platform
        inject_into: request_parameter

incremental_sync:
  type: DatetimeBasedCursor
  cursor_field: date_reference
  lookback_window: P{{config['lookback_window__days_']}}D
  cursor_datetime_formats:
    - '%Y-%m-%d'
  datetime_format: '%Y-%m-%d'
  start_datetime:
    type: MinMaxDatetime
    datetime: '{{ config[''start_date''] }}'
    datetime_format: '%Y-%m-%dT%H:%M:%SZ'
  start_time_option:
    type: RequestOption
    field_name: date
    inject_into: request_parameter
  end_datetime:
    type: MinMaxDatetime
    datetime: '{{ day_delta(-1, format=''%Y-%m-%d'') }}'
    datetime_format: '%Y-%m-%d'
  step: P1D
  cursor_granularity: P1D
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants