feat(nvidia): configurable nvidia-smi binary, ibstat binary, infiniband class dir paths for mock testing

[gyuho]

/tmp/gpud scan \
--nvidia-smi-command 'curl -fsSL https://pkg.gpud.dev/test-nvidia-smi-h100-no-issue' \
--nvidia-smi-query-command 'curl -fsSL https://pkg.gpud.dev/test-nvidia-smi-query-rtx4090-no-issue' \
--ibstat-command 'curl -fsSL https://pkg.gpud.dev/test-ibstat-no-issue' \
--infiniband-class-directory '/tmp/gpud-test-ib-class'

/tmp/gpud scan \
--nvidia-smi-command 'curl -fsSL https://pkg.gpud.dev/test-nvidia-smi-rtx4090-error' \
--nvidia-smi-query-command 'curl -fsSL https://pkg.gpud.dev/test-nvidia-smi-query-hw-slowdown' \
--ibstat-command 'curl -fsSL https://pkg.gpud.dev/test-ibstat-some-down' \
--infiniband-class-directory '/tmp/gpud-test-ib-class'

NVML GPU-e610277c-a7ac-52b5-0d0a-326b15d9ae86

✘ NVML GSP firmware mode is disabled (supported: false)
✔ NVML persistence mode is enabled
✔ NVML found no hw slowdown error
✔ NVML found no ecc volatile uncorrected error
✔ NVML found no running process
✘ scanned nvidia-smi -- found 1 error(s)
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                    0 |
|ERR!   38C    P5    49W / 450W |   2021MiB / 23028MiB |      0%   E. Process |
✘ scanned nvidia-smi -- found 8 hardware slowdown error(s)
GPU 00000000:19:00.0: ClockEventReasons.HWSlowdown.PowerBrakeSlowdown Active
GPU 00000000:3B:00.0: ClockEventReasons.HWSlowdown.PowerBrakeSlowdown Active
GPU 00000000:4C:00.0: ClockEventReasons.HWSlowdown.PowerBrakeSlowdown Active
GPU 00000000:5D:00.0: ClockEventReasons.HWSlowdown.PowerBrakeSlowdown Active
GPU 00000000:9B:00.0: ClockEventReasons.HWSlowdown.PowerBrakeSlowdown Active
GPU 00000000:BB:00.0: ClockEventReasons.HWSlowdown.PowerBrakeSlowdown Active
GPU 00000000:CB:00.0: ClockEventReasons.HWSlowdown.PowerBrakeSlowdown Active
GPU 00000000:DB:00.0: ClockEventReasons.HWSlowdown.PowerBrakeSlowdown Active

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 10.62124% with 446 lines in your changes missing coverage. Please review.

Project coverage is 21.01%. Comparing base (8e9d625) to head (3d02e9a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/gpud/command/command.go	0.00%	51 Missing ⚠️
internal/server/server.go	0.00%	48 Missing ⚠️
components/accelerator/nvidia/query/options.go	0.00%	45 Missing ⚠️
components/accelerator/nvidia/query/query.go	0.00%	44 Missing ⚠️
components/diagnose/options.go	42.85%	16 Missing ⚠️
config/op_options.go	0.00%	16 Missing ⚠️
...ents/accelerator/nvidia/query/infiniband/ibstat.go	59.25%	8 Missing and 3 partials ⚠️
components/diagnose/scan.go	0.00%	10 Missing ⚠️
...omponents/accelerator/nvidia/bad-envs/component.go	0.00%	9 Missing ⚠️
...onents/accelerator/nvidia/clock-speed/component.go	0.00%	9 Missing ⚠️
... and 30 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #310      +/-   ##
==========================================
+ Coverage   20.71%   21.01%   +0.29%     
==========================================
  Files         318      300      -18     
  Lines       26900    26926      +26     
==========================================
+ Hits         5572     5658      +86     
+ Misses      20677    20604      -73     
- Partials      651      664      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ccding]

we are passing these everywhere. can we use an embedded struct

[gyuho]

@ccding Addressed, PTAL. Thanks.

[gyuho]

Will merge after end-to-end tests

[gyuho]

Tested with mock.