Handle watch errors #15
Conversation
jsoriano
force-pushed
the
handle-watch-error
branch
from
c80dc74 to
65e21b7
Compare
kubernetes.go
Outdated
| @@ -320,13 +330,13 @@ func (c *KubernetesClient) Watch() error { | |||
| c.eventForwarder(e) | |||
| } | |||
|
|
|||
| if !more { | |||
| if !more || e.Type == watch.Error { | |||
There was a problem hiding this comment.
If a watch error is received, is the connection with the API still up?
If that is the case, should we disconnect before trying to connect again to the API ? is it managed by connect method ?
There was a problem hiding this comment.
What I have seen is that if we receive an error, the channel is closed, so for next iteration more will be false and then we'd reconnect anyway.
We could just wait for the next iteration, but we are reading from 3 channels, if after a reconnection we are requesting an incorrect version, we'll receive an error from the 3 channels and then we could be interating on errors up to three times, depending on what channel is selected on each iteration. The (not so big) problem with that is that then we'll be handling the error from one to three times when we know that we have to reconnect at the end. So I decided to reconnect directly on the first error.
This Watch method is getting quite big and complex, we should probably refactor it, we could for example handle each watcher separatedly.
There was a problem hiding this comment.
Ok, as you mentioned, it seems better trying to reconnect ASAP.
If a watch error is received we might have lost events, so reset local storages and reconnect to receive everything. Watch errors have been seen when reconnecting to a restarted API server.
jsoriano
force-pushed
the
handle-watch-error
branch
from
65e21b7 to
6854982
Compare
If a watch error is received we might have lost events, so reset local storages and reconnect to receive everything.
One of the possible errors is that the version we are requesting doesn't exist anymore, watch is then interrupted and kube2lb reconnects, and keeps reconnecting with the same error till timeout. I have seen these watch errors when reconnecting to a restarted API server in a testing scenario, I have never seen it in a production scenario with multiple API servers.
Seen while investigating #13.