Cron sync schedules more frequent than once per hour are not allowed when using Helm chart for OSS/Community edition on EKS

[dkravetz]

Topic

Cron sync schedules more frequent than once per hour are not allowed

Relevant information

Hello!

We are currently hosting the OSS/Community version of Airbyte using Helm on an EKS cluster. The initially deployed version of the helm chart was 1.1.0, which was over time upgraded to 1.2.0 and lastly to 1.3.0. I could not find any relevant information about cron schedules on the changelogs for these versions.

At this point in time we decided to use the Terraform provider to have Airbyte configured as code instead of manually doing so on the web UI, and we run into the following error:

unknown status code returned: Status 400
{"status":400,"type":"error:cron-validation/under-one-hour-not-allowed","title":"Cron sync schedules more frequent than once per hour are not allowed","detail":"Syncs","documentationUrl":null,"data":{"cronExpression":"0 0/15 * * * ?","validationErrorMessage":"Cron
executions must be more than 1 hour apart"}}

The exact same error message appears when modifying an existing cron schedule that runs every 15 minutes from the web UI, forcing us to fall back to a 1 hour schedule:

However, schedules of 15 minutes were working with no issue whatsoever, and these are the only useful links I was able to find about this topic:

• Added info about cron scheduling #16835
• https://docs.airbyte.com/using-airbyte/core-concepts/sync-schedules#sync-considerations

In particular, the pull request linked above states the following here:

On the OSS side, I'm comfortable with letting users set any valid cron schedule they want.

and the second link clearly mentions Airbyte Cloud (which is not what we are currently running):

Syncs can run at most every 60 minutes in Airbyte Cloud. Reach out to Sales if you require replication more frequently than once per hour.

It is my understanding from the information linked above that the open source/community edition should not have this restriction. The informative chart listing the differences between editions under the pricing section of Airbyte's website also seems to indicate a clear distinction between self-hosted and cloud-hosted (the latter having the 1 hour sync period restriction enabled):

Any information and help on this topic would be much appreciated! Please let me know if there's any further information that I can provide in order to help debug this problem. Thank you!

[dkravetz]

This is our values.yaml with all the overrides to the upstream chart:

postgresql:
  enabled: false # We are using our own postgres

global:
  storage:
    type: "S3"
    secretName: airbyte-config-secrets
    bucket: 
      log: our-s3-bucket
      state: our-s3-bucket
      workloadOutput: our-s3-bucket
    s3:
      region: "eu-central-1"
      authenticationType: "credentials"
      accessKeyIdSecretKey: s3-access-key-id
      secretAccessKeySecretKey: s3-secret-access-key
  database:
    type: external
    host: "our-postgres"
    port: "5432"
    database: postgres
    secretName: airbyte-config-secrets
    userSecretKey: database-user
    passwordSecretKey: database-password

[dkravetz]

Further supported by the max sync frequency listed below the chart

[marcosmarxm]

Hello @dkravetz sorry about that. This is an issue will be fixed in 1.4.0
@airbytehq/platform-compose fyi.