Increasing CPU when using thread pool statistics

[danielwidmann]

This might be related to #6, but I'm not sure as .net core 3.1 is used.

We run prometheus-net.DotNetRuntime in an alpine linux docker container with dotnet core 3.1. We noticed the CPU usage is slowly increasing over time. The situation significantly improves when thread pool stats are not collected (see plot below), but even then there still is a measurable increase in CPU usage. There doesn't seem to be a measurable increase in memory consumption.

Initialization with thread pool stats:

collector = DotNetRuntimeStatsBuilder
                .Customize()
                .WithContentionStats()
                .WithJitStats()
                .WithThreadPoolSchedulingStats()
                .WithThreadPoolStats()
                .WithGcStats()
                .WithExceptionStats()
                .StartCollecting();

Initialization without thread pool stats:

collector = DotNetRuntimeStatsBuilder
                .Customize()
                .WithJitStats()
                .WithGcStats()
                .WithExceptionStats()
                .StartCollecting();

We also did some CPU profiling on the affected system, but unfortunately 99% of the CPU time is spend in unmanaged code.

Any ideas how the increase in CPU could be avoided while still collecting metrics? Or any idea how to further investigate this issue?

[djluck]

What version of .NET 3.1 is the container using? Version < 3.1.5 has a memory leak that can cause this CPU consumption you're experiencing (see this MR dotnet/runtime#35924).

[danielwidmann]

We are using 3.1.10.

# dotnet --info

Host (useful for support):
  Version: 3.1.10
  Commit:  1721e39439

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.10 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.10 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

[djluck]

Would it be possible to share the full trace you collected with me?

[danielwidmann]

I sent you an email.

[djluck]

So that trace was collected with dotnet-trace and doesn't include CPU time. To analyze cpu-bound issues, you should use the perfcollect script- https://docs.microsoft.com/en-us/dotnet/core/diagnostics/trace-perfcollect-lttng.

[djluck]

This does look similar to the CPU overconsumption issue that we saw in earlier .NET core versions, not sure why it's popping back up. I need to dedicate some time to look at this properly at some point.

[linys2333]

I found the same problem!Don't know how to solve it.

[djluck]

This issue duplicates #6, so closing it. V4 should resolve the issues described here.