suss doesn't yield lease, sometimes

[hwo-wd]

So yesterday was patch day and we obtained a customer complaint:

Checking the nodes we can see that k8s-ibk-worker2 is still cordoned and that k8s-ibk-worker[13] were not patched at all:

• previous logs of daemon set pod on k8s-ibk-worker2:

  I0510 03:42:33.300276       1 main.go:145] using InClusterConfig
  2024-05-10T03:42:33.300746721Z I0510 03:42:33.300707       1 main.go:121] using namespace wd-suss for the Lease
  E0510 03:42:43.302185       1 main.go:43] Get "https://10.43.0.1:443/api/v1/nodes": net/http: TLS handshake timeout
  

• current logs of daemon set pod on k8s-ibk-worker2:

  I0510 03:47:57.129368       1 main.go:145] using InClusterConfig
  2024-05-10T03:47:57.129846622Z I0510 03:47:57.129805       1 main.go:121] using namespace wd-suss for the Lease
  2024-05-10T03:47:57.152218409Z I0510 03:47:57.152112       1 suss.go:82] "node k8s-ibk-worker2 found\n"
  2024-05-10T03:47:57.152240565Z I0510 03:47:57.152142       1 main.go:58] listen on localhost:9993
  

• current logs of daemon set pod on k8s-ibk-worker1:
  suss-kcjc9_suss.log

• current logs of daemon set pod on k8s-ibk-worker3:
  suss-64qd4_suss.log

@gprossliner any idea what could be going on?

I only mitigated it via:

• uncordoning the node (10.5.24, 10:18)

[hwo-wd]

Interesting side fact:
CRM can't see any historic data before 05:30, and also no traffic came to online-aufladen:

Incoming transactions:

Monitoring: