DaskExecutor: ModuleNotFoundError

[MrJack91]

Hi there :)

Have a question about the use of the dask executor with own python modules.

If I have to use own python modules, i got this exception:
[2020-11-16 09:26:55+0100] ERROR - prefect.Load | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'xxx'")

I think I have to upload all my necessary files via: client.upload_file('xxx.py') (see https://distributed.dask.org/en/latest/api.html#distributed.Client.upload_file)

But how can i access the dask client?

Here the client would be available while the run: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/executors/dask.py#L257

But directly after, it's cleared again. Does anybody have a hint for me?

Regards Michael

[nbalakumaran]

Hello @MrJack91 ,

You can access the client like:

from distributed import Client

client = Client('127.0.0.1:8786')

I faced similar issue despite running worker in module dir. Using the absolute path of the file fixed it. If you want to access client and upload from another file within module, below should suffice

from os.path import join, dirname

client.upload_file(join(dirname(__file__), 'xxx.py'))

[zanieb]

I would recommend using something like Docker storage so you can package your module alongside your flow!