repeated asmcli deployment in the same project exhausts project membership quota

[jeremybramwell]

I have been testing a script to automate deployment of a cluster and anthos service mesh. This involved multiple deployments in the same project using the same cluster name. Today the register_cluster function failed. when I ran the command myself I found that I had exceeded the number of memberships allowed per project:

11:54 $ gcloud container hub memberships register development-cluster-1-06kg5s6i --project=istio-deploy-test-2 --enable-workload-identity --gke-uri=https://container.googleapis.com/v1/projects/istio-deploy-test-2/zones/us-east1-b/clusters/development-cluster-1
kubeconfig entry generated for development-cluster-1.
ERROR: (gcloud.container.hub.memberships.register) HttpError accessing <https://gkehub.googleapis.com/v1/projects/istio-deploy-test-2/locations/global/memberships?alt=json&membershipId=development-cluster-1-06kg5s6i>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'content-encoding': 'gzip', 'date': 'Tue, 19 Jul 2022 15:55:28 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'transfer-encoding': 'chunked', 'status': 429}>, content <{
  "error": {
    "code": 429,
    "message": "Quota limit 'GlobalPerProjectMemberships' has been exceeded. Limit: 15 in global.",
    "status": "RESOURCE_EXHAUSTED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.QuotaFailure",
        "violations": [
          {
            "subject": "project:620350027694",
            "description": "Quota 'GlobalPerProjectMemberships' exhausted. Limit 15 in global"
          }
        ]
      }
    ]
  }
}
>

All 15 memberships were for the same cluster name, but the cluster had been deleted and recreated:

11:55 $ gcloud container hub memberships list --project=istio-deploy-test-2
NAME                                                  EXTERNAL_ID
development-cluster-1-utodbp5n                        3432ecf1-18be-402f-9084-78575db54380
development-cluster-1                                 d5020dd4-7a65-4e53-b81b-34bb52273ec3
development-cluster-1-x1oq160k                        63d5d5a9-3c59-4ac6-94c4-f56a145492ba
development-cluster-1-wdz12isg                        43ea26e3-eca7-4148-be14-79f062777d71
development-cluster-1-xvtma3ip                        a0576c6d-6a45-4a4d-bc3e-4a4421b25c40
development-cluster-1-kry8d69a                        10efb3d2-17ae-4266-a95d-651d7a096c6b
development-cluster-1-xzalfr3a                        e1c319fb-b68c-4b80-87a8-4b82983d0ad0
development-cluster-1-t8hrf3b7                        ed5093fd-4039-494b-88bc-ed5a008b8ac6
development-cluster-1-istio-deploy-test-2-us-east1-b  d167037f-bba6-41dd-837b-4c31d3a1c6ca
development-cluster-1-bwbgj5pn                        ff017383-9238-4b12-9056-719f8e07626b
development-cluster-1-e9btfa5r                        e6e1aeca-1b96-4629-b129-8797ea8c05cd
development-cluster-1-s7v5eaoy                        15479ddc-5f93-4f0f-8e9e-ed0bc99576b5
development-cluster-1-v6ttmw37                        77305f33-ecce-402b-93ca-f673400cf309
development-cluster-1-3hwxzprt                        3359b7c6-b876-4fed-b9af-187ba155ead6
development-cluster-1-vquy1ge7                        2bde3eeb-bf66-4361-8c8e-90ce0722a515

1. It looks like the script does check if the cluster is registered before creating a new membership, so I guess that since I was deleting and reusing the same cluster name the existing memberships became invalid.

   anthos-service-mesh-packages/asmcli/asmcli

   Line 3576 in 4753ebb

   if is_cluster_registered; then return; fi

   Could this script detect this situation and delete the old membership?
2. Could the asmcli surface this error for the user so that they don't keep retrying in vain? issue Number of retries in the register_cluster() function should be more than 2 #1215 suggests that the registration command often fails since it can take a while.