The main event loop of a worker vs a worker's task executor pool

[YarShev]

Hello,

I would be appreciated to know what the difference is between the main event loop of a worker vs a worker's task executor pool. Given the docstring for func parameter of Client.submit, there is probably two places where a task can be executed, i.e. the main event loop of a worker and a worker's task executor pool (e.g., ThreadPoolExecutor).

distributed/distributed/client.py

Line 1678 in 84cbb09

Callable to be scheduled as ``func(*args **kwargs)``. If ``func`` returns a

Do I correctly understand that the main event loop of a worker is the core of asyncio, which can be get via get_running_loop? If that is the case, how a function to be run on that loop would look like on the user side?

Thanks in advance!

[mrocklin]

You're correct that the main thread of a Dask worker is running an asyncio event loop. It is mostly used for network communication and control of everything else. Dask workers also contain a thread pool in which user code runs. In general we rarely run user code in the event loop (we don't trust it as much, and are afraid that it might block control systems). It may be that if your function is an async def function that we might run it on the event loop (I would have to go look to be sure) but this is very rare. Does this answer your question?

…

On Wed, Apr 27, 2022 at 2:48 AM Yaroslav Igoshev ***@***.***> wrote: Hello, I would be appreciated to know what the difference is between the main event loop of a worker vs a worker's task executor pool. Given the docstring for func parameter of Client.submit, there is probably two places where a task can be executed, i.e. the main event loop of a worker and a worker's task executor pool (e.g., ThreadPoolExecutor). https://github.com/dask/distributed/blob/84cbb099e0d0f1e1947d51d957f0b65e9b0635dd/distributed/client.py#L1678 Do I correctly understand that the main event loop of a worker is the core of asyncio, which can be get via get_running_loop <https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_running_loop>? If that is the case, how a function to be run on that loop would look like on the user side? Thanks in advance! — Reply to this email directly, view it on GitHub <#6214>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTAUU7DOK2T7AZIGHRLVHDWLJANCNFSM5UOGXUTQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[YarShev]

@mrocklin, thank you for the response. It almost answers my question. I'd like to be sure that Dask runs foo in the event loop of the main thread of a worker in the following example.

from distributed.client import Client

async def foo():
    return 1

c = Client()
f = c.submit(foo)
print(f.result())
1

Could you let me know if that is the case, when you check this?