[EEESUPPORT-11240] Add logging and retries to pull_cmd

[guyboltonking]

The retry logic required patching containerd; see guyboltonking/containerd#2 to review the patch. Commits can be reviewed in order.

[joneshf-dd]

I think this change will work to address the specific issue, so 👍🏾 from me. But I'm curious if we considered pulling in one of the retry packages and wrapping the call to image.Dispatch instead (so we don't have to maintain this patch or care which part of it is failing). Some (but not all) of the retry packages:

• https://pkg.go.dev/github.com/avast/retry-go/v4
• https://pkg.go.dev/github.com/cenkalti/backoff/v4
• https://pkg.go.dev/github.com/matryer/try
• https://pkg.go.dev/github.com/sethvargo/go-retry

We should also be able to use a retry package for other calls we find to be flaky that don't already have a retry mechanism.

Similar to the 💭, this isn't blocking. Something to think about, and do whatever you think is best.

[guyboltonking]

But I'm curious if we considered pulling in one of the retry packages and wrapping the call to image.Dispatch instead (so we don't have to maintain this patch or care which part of it is failing).

I'm not sure I want to wrap all of image.Dispatch, since I only want to log and retry at the point where we know we're seeing a failure, rather than retrying higher up and risking retrying on failures that we shouldn't retry on.

I briefly looked at avast, decided it was too much to add for this one case, and was also put off by the fact I couldn't go get it:

❯ go get github.com/avast/[email protected]
go: github.com/avast/[email protected]: invalid version: go.mod has post-v4 module path "github.com/avast/retry-go/v4" at revision v4.6.0

...at which point I read about how go get behaves with respect to semver, go.mod presence, and whether the moon is gibbous, and decided to give up.

[joneshf-dd]

That's also fair (re: not potentially introducing other failures by retrying the whole process). No need to spin your wheels on it!

Re: avast/retry-go, you have to include the /v4 part at the end of the package:

$ go get github.com/avast/[email protected]
go: github.com/avast/[email protected]: invalid version: go.mod has post-v4 module path "github.com/avast/retry-go/v4" at revision v4.6.0

$ go get github.com/avast/retry-go/[email protected]
go: downloading github.com/avast/retry-go/v4 v4.6.0
go: added github.com/avast/retry-go/v4 v4.6.0