Using Coiled for executing prefect flows?

[gryBox]

Hi - There is some interest by a couple prefect users to use coil to run flows. The benefits being:

1. Simplified configuration for agents and executors
2. No specific cloud provider
3. Insight into a flow's execution

Proposed Approaches

... something like the current Dask Cloud Provider Environment would be desirable, but using coiled as the environment. Saving lots of configuration for the user

@jose-moralez gist

.. creating the coiled cluster in a resource manager and giving the scheduler ip to a dask executor.

Matthew Rocklin

Rather than use a scheduler IP I wonder if it would make more sense to use a named coiled.Cluster

import coiled
cluster = coiled.Cluster(
    configuration=...,
    name="prefect-production",
    shutdown_on_close=False,
)

Anyone at prefect have thoughts on this?
cc: @cicdw

[cicdw]

Hey @gryBox - you can tap into a Coiled cluster quite easily using the coiled.Cluster object:

from prefect.engine.executors import DaskExecutor
import coiled

coiled_executor = DaskExecutor(cluster_class=coiled.Cluster, 
                               cluster_kwargs=dict(name="prefect-demo", 
                                                   configuration="my-acct/demo-cluster-config"))
flow_state = flow.run(executor=coiled_executor)

This is probably the most flexible approach, and if you use a name (which I like to do) it will re-use the same cluster if it already exists if you’re running a lot of flows in succession / in parallel

[gryBox]

@cicdw This is very clean!! What about the agent? Does this mean that one can sidestep configuring Environments and or Agents? Leaving the only decision where to store the flow i.e. Non-Docker Storage for Containerized Environments?

Sorry for all the questions but this changes deployment and monitoring for the better.

[cicdw]

No worries! You’d want to use a local environment with this executor configuration (as in this doc: https://docs.prefect.io/orchestration/execution/local_environment.html#examples) but otherwise as long as the agent submits the run in a place that has access to your coiled credentials, it won’t matter what type of agent you use