Getting stats from GPU

[kbekus]

Description

Referencing #375

Running binary agent 0.9.1 installed from the install.sh script on a VM fails to see a GPU. I get the error
beszel-agent[3404832]: 2025/02/06 21:19:17 DEBUG GPU err="no GPU found - install nvidia-smi or rocm-smi"

The OS is Debian 11, the GPU is an nVidia 3090. Running nvidia-smi returns useful data:

# nvidia-smi -l 4 --query-gpu=index,name,temperature.gpu,memory.used,memory.total,utilization.gpu,power.draw --format=csv,noheader,nounits
0, NVIDIA GeForce RTX 3090, 41, 4256, 24576, 0, 28.69
0, NVIDIA GeForce RTX 3090, 41, 4256, 24576, 0, 28.58

Trying echo "exit code: $?" yields
exit code: 0

OS / Architecture

Debian 11

Beszel version

0.9.1

Installation method

Binary

Configuration

Environment="GPU=true"

Hub Logs

No response

Agent Logs

beszel-agent[3404832]: 2025/02/06 21:19:17 DEBUG GPU err="no GPU found - install nvidia-smi or rocm-smi"

[henrygd]

Thanks, please try running the command below. You don't need to include the nvidia-smi output here, but let me know if it still says the exit code is 0.

su -s /bin/sh beszel -c "nvidia-smi" && echo "exit code: $?"

[kbekus]

su -s /bin/sh beszel -c "nvidia-smi" && echo "exit code: $?"

Sat Feb 8 13:41:50 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.02 Driver Version: 555.42.02 CUDA Version: 12.5 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:01:00.0 Off | N/A |
| 50% 35C P8 26W / 250W | 4256MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1474 G /usr/lib/xorg/Xorg 10MiB |
| 0 N/A N/A 4082536 C python 2374MiB |
| 0 N/A N/A 4082892 C /app/.venv/bin/python 1852MiB |
+-----------------------------------------------------------------------------------------+
exit code: 0

just for sake of fullness, yeah exit code 0.

[henrygd]

Thanks, I have absolutely no idea what's happening here since you should only get the no GPU found message if there was a non-zero exit code when attempting to run nvidia-smi.

I figured it was an issue running it as the beszel user, but looks like that's not the case.

For the next release I changed the check from exit code status to just whether nvidia-smi exists in the user's path, so maybe that will work.

If you want to test, you can compile the agent with these instructions. Remove --branch v0.9.1 to get the latest changes. You can build on a different machine if you want and use the same binary.

Otherwise the next release should be out next week.

[shaoyaoqian]

beszel-agent[317788]: 2025/02/11 14:41:35 WARN nvidia-smi found no valid GPU data, stopping

Tue Feb 11 14:46:05 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
|  0%   37C    P8             18W /  450W |      12MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2903      G   /usr/lib/xorg/Xorg                              4MiB |
+-----------------------------------------------------------------------------------------+

[henrygd]

1. Please leave the command below running for a minute and provide a couple lines of output.
2. Does the found no valid GPU data, stopping warning happen immediately after starting the agent, or does the agent provide GPU stats for a while then stop?

nvidia-smi -l 4 --query-gpu=index,name,temperature.gpu,memory.used,memory.total,utilization.gpu,power.draw --format=csv,noheader,nounits

Thanks

[shaoyaoqian]

Thank you!

1. The outputs:

~$ nvidia-smi -l 4 --query-gpu=index,name,temperature.gpu,memory.used,memory.total,utilization.gpu,power.draw --format=csv,noheader,nounits
0, NVIDIA GeForce RTX 4090, 39, 12, 24564, 0, 18.00
0, NVIDIA GeForce RTX 4090, 39, 12, 24564, 0, 17.49
0, NVIDIA GeForce RTX 4090, 39, 12, 24564, 0, 17.37
0, NVIDIA GeForce RTX 4090, 39, 12, 24564, 0, 17.36
0, NVIDIA GeForce RTX 4090, 39, 12, 24564, 0, 17.54

2. Immediately.

● beszel-agent.service - Beszel Agent Service
     Loaded: loaded (/etc/systemd/system/beszel-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2025-02-12 00:56:09 CST; 2s ago
   Main PID: 1710839 (beszel-agent)
      Tasks: 12 (limit: 463907)
     Memory: 8.7M
     CGroup: /system.slice/beszel-agent.service
             └─1710839 /opt/beszel-agent/beszel-agent

2月 12 00:56:09 epyc-9654 systemd[1]: Started Beszel Agent Service.
2月 12 00:56:09 epyc-9654 beszel-agent[1710839]: 2025/02/12 00:56:09 INFO Detected root device name=nvme0n1p4
2月 12 00:56:09 epyc-9654 beszel-agent[1710839]: 2025/02/12 00:56:09 INFO Detected network interface name=enp2s0f0np0 sent=>
2月 12 00:56:09 epyc-9654 beszel-agent[1710839]: 2025/02/12 00:56:09 INFO Starting SSH server address=:45876
2月 12 00:56:09 epyc-9654 beszel-agent[1710839]: 2025/02/12 00:56:09 WARN nvidia-smi found no valid GPU data, stopping

The beszel-agent installed using the script did not provide GPU stats and showed no warning. Then, as you mentioned, I removed ‘--branch v0.9.1’ and replaced /opt/beszel-agent/beszel-agent with the compiled version. However, the agent still does not provide GPU stats.