Consumer: ram & swap overeating when prometheus tries to collect data from karton-system with 20k+ tasks in run

[rakovskij-stanislav]

Hello,

I met a problem with karton-dashboard when I try to collect /vars using prometheus every 5 secs with more than 10k tasks.

Karton-dashboard exceeds 5s limit (context deadline exceeded prometheus error) and tries to collect data again - and karton-dashboard prepares two reports at the same time, using two cpu cores, but the second request exceeds 5s limit too, the third request will be born - and it looks like deadlock of requests.

CPU exhaust ia a bad news, but it is bearable. Every such request consumes a lot of ram, so soon the server will have no ram, this way I have no possibility even to kill karton-dashboard)

---

This issue wrote just for acknolegement about the situation with a lot of tasks - I think it is a architecure issue and probably it is fixable if we will use less resouce-consuming stats obtaining.

[rakovskij-stanislav]

The fix for someone who faces the same problem is to inscrease scrape_timeout

[rakovskij-stanislav]

Identified: ~10 of tasks had 500+ mb in redis, it will cause such problems