How to capture diagonostic through python api or http endpoint instead of html

[zhqu1148980644]

In which object I can get this information or if it's possible to write plugins to access these data collected by dask.distributed.

[jrbourbeau]

Thanks for asking @zhqu1148980644. Could you be more specific about what information you'd like to capture?

[zhqu1148980644]

@jrbourbeau all information on the dashboard, like cpu, load balance per worker, etc, is there any programmatic way to fetch them?

[zhqu1148980644]

@jrbourbeau Hi, do you have any ideas, sincerely appreciate your help.

[fjetter]

Accessing through HTTP (prometheus metrics)

There exists a /metrics endpoint which is used to expose some of this information as prometheus metrics. The exposed metrics for scheduler and workers differ slightly.
Unfortunately, this is not, yet, part of the official documentation but if you are looking for the code, see here and here.

There are also some JSON endpoints available, e.g. /json/counts.json

All available endpoints are listed here

Python API

Afaik, there is not one dedicated API to extract statistics. Depending on what you are looking for the distributed Client might offer some, if not all information. For instance, there is the Client.scheduler_info.

There is of course always the possibility to write your own plugin to extract all the custom information you require.

[zhqu1148980644]

@fjetter Thanks, I choose to write my own monitoring plugin. By the way, can I use another thread to collect stats and then send it back to the calling side of client.submit? how can I achieve this, can I attach it to the submit-returned future?

[fjetter]

IIUC, you want to have custom, task/computation specific metadata attached to the futures to pass along your metrics? Afaik, there is no mechanism for this available.

[zhqu1148980644]

@fjetter How about wrapping my computation function/task into a function that automatically starts a new thread for collecting resource usages and sends it back as return value? if it's doable. And are there any better ways to achieve this?

[fjetter]

If you want to instrument your code this should be a possibility. If you want to get metrics from dask itself you won't have access to the worker instance this way and there won't be a way to get this information, I am afraid.

[mrocklin]

There are some JSON routes. Unfortunately these aren't well documented. Rather than client.submit you might want to look at client.run.

…

On Tue, Apr 6, 2021 at 8:03 AM BAKEZQ ***@***.***> wrote: Thanks, I choose to write my own monitoring plugin. By the way, can I use another thread to collect stats and then send it back to the calling side of client.submit? how can I achieve this, can I attach it to the submit-returned future? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4669 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTA42XJUNJFIGSWK25DTHMBDZANCNFSM42JC5VEA> .

[zhqu1148980644]

I'd like to collect diagnostics per job.