Follow up for 3691 [TargetGroupBindings can now manipulate target groups from different aws accounts]

[zac-nixon]

Issue

Previous PR, #3691

Issue, #3634

Description

The original PR (#3691) was a very good start to implementing cross account Target Group management and I used the same logic. This PR mostly fixes up bugs in the approach, as 3691 didn't work using SDKv2. Things I've added / changed:

• Use a singleton STS client.
• Use a garbage collected cache.
• Do not hold on to assumed role elbv2 clients indefinitely, these clients eventually expire and need to be recreated.
• Remove annotation based configuration for the Target Group Binding, instead rely only on the TGB spec to hold configuration data.
• Fix TG ARN resolution (the previous PR didn't apply the same logic between ARN / name specification for the TG). Also refactored the logic to remove code duplication.
• Add webhook validation to prevent specifying instance target groups + assume role. ELB API doesn't allow cross account instance registration.
• Added documentation and examples.

Test Cases:

• Same account TGB interactions work the same.
• Cross account with valid role + external id works.
• Cross account with invalid external id gracefully fails in the webhook validation.
• Long running controller (5 days) has no significant memory increase nor any errors related to the AssumeRole management.

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the docs directory)
• Manually tested
• Made sure the title of the PR is a good description that can go into the release notes

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

• Backfilled missing tests for code in same general area 🎉
• Refactored something and made the world a better place 🌟

[zac-nixon]

/ok-to-test

[zac-nixon]

Removed this. Trying to get the tool fixed to generate this properly, but it will make the CR huge

[zac-nixon]

/retest

[shraddhabang]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: shraddhabang, zac-nixon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [shraddhabang,zac-nixon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment