Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow path to environment file to be passed into prefect.yaml #17203

Open
robfreedy opened this issue Feb 19, 2025 · 0 comments
Open

Allow path to environment file to be passed into prefect.yaml #17203

robfreedy opened this issue Feb 19, 2025 · 0 comments
Labels
enhancement An improvement of an existing feature

Comments

@robfreedy
Copy link
Contributor

Describe the current behavior

With the deprecation of JSON blocks, users who have deployment specific configuration are advised to pass this configuration in using variables, which have a 5000 characters limit. For deployment specific configuration that is longer than that, they have to specify multiple variables, which is tedious.

An example of this behavior would be when a Kubernetes config map/secrets configuration needs to get passed in on a per deployment basis:

[
  {
    "name": "SOME_CONFIGMAP_KEY",
    "valueFrom": {
      "configMapKeyRef": {
        "name": "some.configmap",
        "key": "key"
      }
    }
  },
  {
    "name": "VERY_IMPORTANT_SECRET_KEY",
    "valueFrom": {
      "SecretKeyRef": {
        "name": "very.important",
        "key": "secret_key"
      }
    }
  }
]

prefect.yaml with json block

- name: test_deployment
  version:
  tags: []
  description:
  flow_name:
  entrypoint: src/main/test_flow.py:run
  work_pool:
    name: kubernetes-work-pool
    work_queue_name: general
    job_variables:
      image: fiverr/data_events_3.8_crons:latest
      job_watch_timeout_seconds: 3600
      pod_watch_timeout_seconds: 3600
      env: '{{ prefect.blocks.json.env-json }}'

Describe the proposed behavior

Being able to reference environment file to the prefect.yaml, something like the below example

Example Use

- name: test_deployment
  version:
  tags: []
  description:
  flow_name:
  entrypoint: src/main/test_flow.py:run
  work_pool:
    name: kubernetes-work-pool
    work_queue_name: general
    job_variables:
      image: fiverr/data_events_3.8_crons:latest
      job_watch_timeout_seconds: 3600
      pod_watch_timeout_seconds: 3600
      env: {% include env.yaml %}

Additional context

No response

@robfreedy robfreedy added the enhancement An improvement of an existing feature label Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

No branches or pull requests

1 participant