Harbor jobservice starts before core is fully up

[swasthibp]

Hi Team,

We have encountered the failure for jobservice container before the core container is fully up:
Jobservice already contacts the core URL (2024-05-31T11:19:32.670754088+03:00 stdout F 2024-05-31T08:19:32Z [ERROR] [/pkg/config/rest/rest.go:50]: Failed on load rest config err:Get "https://core:8443/api/v2.0/internalconfig": dial tcp 10.89.0.94:8443: connect: connection refused, url:https://core:8443/api/v2.0/internalconfig)before core is fully started(2024-05-31T11:19:47.499064491+03:00 stdout F 2024/05/31 08:19:47.499 �[1;34m[I]�[0m [server.go:249] https server Running on https://:8443):

Instance 1:
Core1:
2024-05-31T11:19:16.880233280+03:00 stdout F 2024-05-31T08:19:16Z [INFO] [/core/main.go:258]: Version: v2.10.0, Git commit: 6abb4ea
2024-05-31T11:19:16.880233280+03:00 stdout F 2024-05-31T08:19:16Z [INFO] [/core/main.go:260]: Fix empty subiss for meta info data.
2024-05-31T11:19:16.881403333+03:00 stdout F 2024-05-31T08:19:16Z [INFO] [/pkg/oidc/fix.go:37]: Not found any records with empty subiss, good to go.
2024-05-31T11:19:36.991543260+03:00 stderr F redis: 2024/05/31 08:19:36 sentinel.go:514: sentinel: GetMasterAddrByName master="netactMaster" failed: dial tcp 192.168.1.21:26379: i/o timeout
2024-05-31T11:19:36.993692235+03:00 stderr F panic: redis: all sentinels specified in configuration are unreachable
2024-05-31T11:19:36.993692235+03:00 stderr F
2024-05-31T11:19:36.993692235+03:00 stderr F goroutine 1 [running]:
2024-05-31T11:19:36.993702799+03:00 stderr F github.com/beego/beego/v2/server/web.initBeforeHTTPRun.func1()
2024-05-31T11:19:36.993702799+03:00 stderr F /go/pkg/mod/github.com/beego/beego/[email protected]/server/web/beego.go:80 +0x178
2024-05-31T11:19:36.993702799+03:00 stderr F sync.(*Once).doSlow(0x1b9a2d8?, 0x3beab60?)
2024-05-31T11:19:36.993702799+03:00 stderr F /usr/local/go/src/sync/once.go:74 +0xbf

2024-05-31T11:19:47.499064491+03:00 stdout F 2024/05/31 08:19:47.499 �[1;34m[I]�[0m [server.go:249] https server Running on https://:8443

jobservice1:
2024-05-31T11:19:32.670459102+03:00 stdout F 2024-05-31T08:19:32Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://core:8443/api/v2.0/internalconfig
2024-05-31T11:19:32.670754088+03:00 stdout F 2024-05-31T08:19:32Z [ERROR] [/pkg/config/rest/rest.go:50]: Failed on load rest config err:Get "https://core:8443/api/v2.0/internalconfig": dial tcp 10.89.0.94:8443: connect: connection refused, url:https://core:8443/api/v2.0/internalconfig
2024-05-31T11:19:32.672884592+03:00 stderr F panic: failed to load configuration, error: failed to load rest config
2024-05-31T11:19:32.672884592+03:00 stderr F
2024-05-31T11:19:32.672884592+03:00 stderr F goroutine 1 [running]:
2024-05-31T11:19:32.672893530+03:00 stderr F main.main()
2024-05-31T11:19:32.672893530+03:00 stderr F /harbor/src/jobservice/main.go:44 +0x3ae
2024-05-31T11:21:54.229663341+03:00 stdout F Appending internal tls trust CA to ca-bundle ...
2024-05-31T11:21:54.231320174+03:00 stdout F Internal tls trust CA appending is Done.
2024-05-31T11:21:54.232847306+03:00 stdout F Appending trust CA to ca-bundle ...

Instance2:
core2:
2024-05-31T11:21:38.468594745+03:00 stdout F 2024-05-31T08:21:38Z [INFO] [/core/main.go:258]: Version: v2.10.0, Git commit: 6abb4ea
2024-05-31T11:21:38.468594745+03:00 stdout F 2024-05-31T08:21:38Z [INFO] [/core/main.go:260]: Fix empty subiss for meta info data.
2024-05-31T11:21:38.469170765+03:00 stdout F 2024-05-31T08:21:38Z [INFO] [/pkg/oidc/fix.go:37]: Not found any records with empty subiss, good to go.
2024-05-31T11:21:58.579140352+03:00 stderr F redis: 2024/05/31 08:21:58 sentinel.go:514: sentinel: GetMasterAddrByName master="netactMaster" failed: dial tcp 192.168.1.21:26379: i/o timeout
2024-05-31T11:21:58.581313071+03:00 stderr F panic: redis: all sentinels specified in configuration are unreachable
2024-05-31T11:21:58.581313071+03:00 stderr F
2024-05-31T11:21:58.581313071+03:00 stderr F goroutine 1 [running]:
2024-05-31T11:21:58.581313071+03:00 stderr F github.com/beego/beego/v2/server/web.initBeforeHTTPRun.func1()
2024-05-31T11:21:58.581313071+03:00 stderr F /go/pkg/mod/github.com/beego/beego/[email protected]/server/web/beego.go:80 +0x178
2024-05-31T11:21:58.581326197+03:00 stderr F sync.(*Once).doSlow(0x1b9a2d8?, 0x3beab60?)
2024-05-31T11:21:58.581326197+03:00 stderr F /usr/local/go/src/sync/once.go:74 +0xbf
2024-05-31T11:21:58.581330956+03:00 stderr F sync.(*Once).Do(...)
2024-05-31T11:21:58.581330956+03:00 stderr P /usr/local/go/src/sync/once.go:65
2024-05-31T11:21:58.581335572+03:00 stderr F

2024-05-31T11:23:35.628415950+03:00 stdout F 2024/05/31 08:23:35.625 �[1;34m[I]�[0m [server.go:249] https server Running on https://:8443

jobservice2:
2024-05-31T11:21:54.258000664+03:00 stdout F 2024-05-31T08:21:54Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://core:8443/api/v2.0/internalconfig
2024-05-31T11:21:54.258336353+03:00 stdout F 2024-05-31T08:21:54Z [ERROR] [/pkg/config/rest/rest.go:50]: Failed on load rest config err:Get "https://core:8443/api/v2.0/internalconfig": dial tcp 10.89.0.105:8443: connect: connection refused, url:https://core:8443/api/v2.0/internalconfig
2024-05-31T11:21:54.260456206+03:00 stderr F panic: failed to load configuration, error: failed to load rest config
2024-05-31T11:21:54.260456206+03:00 stderr F
2024-05-31T11:21:54.260456206+03:00 stderr F goroutine 1 [running]:
2024-05-31T11:21:54.260464699+03:00 stderr F main.main()
2024-05-31T11:21:54.260464699+03:00 stderr F /harbor/src/jobservice/main.go:44 +0x3ae
2024-05-31T11:23:56.250945720+03:00 stdout F Appending internal tls trust CA to ca-bundle ...
2024-05-31T11:23:56.252473925+03:00 stdout F Internal tls trust CA appending is Done.
2024-05-31T11:23:56.254002989+03:00 stdout F Appending trust CA to ca-bundle ...

Instance3:
ore3:
2024-05-31T11:33:17.063998265+03:00 stderr F redis: 2024/05/31 08:33:17 sentinel.go:514: sentinel: GetMasterAddrByName master="netactMaster" failed: dial tcp 192.168.1.21:26379: i/o timeout
2024-05-31T11:33:17.066116760+03:00 stderr F panic: redis: all sentinels specified in configuration are unreachable
2024-05-31T11:33:17.066116760+03:00 stderr F
2024-05-31T11:33:17.066116760+03:00 stderr F goroutine 1 [running]:
2024-05-31T11:33:17.066116760+03:00 stderr F github.com/beego/beego/v2/server/web.initBeforeHTTPRun.func1()
2024-05-31T11:33:17.066116760+03:00 stderr F /go/pkg/mod/github.com/beego/beego/[email protected]/server/web/beego.go:80 +0x178
2024-05-31T11:33:17.066116760+03:00 stderr F sync.(*Once).doSlow(0x1b9a2d8?, 0x3beab60?)
2024-05-31T11:33:17.066116760+03:00 stderr P /usr/local/go/src/sync/once.go:74 +0xbf
2024-05-31T11:33:17.066133357+03:00 stderr F

2024-05-31T11:35:26.956263980+03:00 stdout F 2024/05/31 08:35:26.956 �[1;34m[I]�[0m [server.go:249] https server Running on https://:8443

jobservice3:
2024-05-31T11:33:12.688460777+03:00 stdout F 2024-05-31T08:33:12Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://core:8443/api/v2.0/internalconfig
2024-05-31T11:33:12.690323776+03:00 stdout F 2024-05-31T08:33:12Z [ERROR] [/pkg/config/rest/rest.go:50]: Failed on load rest config err:Get "https://core:8443/api/v2.0/internalconfig": dial tcp 10.89.0.137:8443: connect: connection refused, url:https://core:8443/api/v2.0/internalconfig
2024-05-31T11:33:12.693522511+03:00 stderr F panic: failed to load configuration, error: failed to load rest config
2024-05-31T11:33:12.693522511+03:00 stderr F
2024-05-31T11:33:12.693522511+03:00 stderr F goroutine 1 [running]:
2024-05-31T11:33:12.693522511+03:00 stderr F main.main()
2024-05-31T11:33:12.693522511+03:00 stderr F /harbor/src/jobservice/main.go:44 +0x3ae
2024-05-31T11:38:38.554093478+03:00 stdout F Appending internal tls trust CA to ca-bundle ...
2024-05-31T11:38:38.555506995+03:00 stdout F Internal tls trust CA appending is Done.
2024-05-31T11:38:38.557040303+03:00 stdout F Appending trust CA to ca-bundle ...
2024-05-31T11:38:38.558368198+03:00 stdout F /harbor_cust_cert/core.crt Appended ...

The above failure is impacting us as jobservice fails frequently. Please get back on how this can be mitigated at the earliest

[chlins]

The jobservice depends on the core, and from the core logs it seems that your redis cluster it not reachable, maybe you should prioritize this problem.

[swasthibp]

redis cluster may not be reachable for few moments sporadically, but why does jobservice already start contacting core url before core states that http server is running?