fix(deploy): ensure build_sa iams are set before it can be used (#394)

* fix(deploy): ensure build_sa iams are set before it can be used Fixes possible race condition in cloud functions deployment where the function build started before the build_sa had all its IAMs bound. * fix: Updates following further investigation/testing --------- Co-authored-by: henrybell <[email protected]>
cloudspannerecosystem · Oct 9, 2024 · aa48048 · aa48048
1 parent 0551694
commit aa48048
Show file tree

Hide file tree

Showing 16 changed files with 146 additions and 96 deletions.
diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md
@@ -210,35 +210,6 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives.
     [known issue][provider-issue] in the Terraform Google provider, please retry
     with -parallelism=1.
 
-    If you are using a recently created project, or one that does not have permissions
-    automatically assigned to default service accounts, you may see an error message
-    similar to the following:
-
-    ```sh
-    Error: Error waiting for Creating CloudFunctions Function: Error code 3, message: Build failed: failed to Fetch: failed to download archive gs://gcf-sources-[PROJECT_NUMBER]-us-central1/tf-poller-function-[UID]/version-1/function-source.zip:
-    Access to bucket gcf-sources-[PROJECT_NUMBER]-us-central1 denied. You must grant Storage Object Viewer permission to [PROJECT_NUMBER][email protected].
-    ```
-
-    A workaround for this issue is to run the following commands, which assign additional
-    permissions to the default compute service account, which is used by Cloud Build:
-
-    ```sh
-    PROJECT_ID=$(gcloud config get-value project)
-    PROJECT_NUMBER=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_NUMBER)")
-    gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"[email protected] --role='roles/storage.objectViewer'
-    gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"[email protected] --role='roles/logging.logWriter'
-    gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"[email protected] --role='roles/artifactregistry.writer'
-    ```
-
-    You can then re-run the `terraform apply` command listed above.
-
-    Please note that the updates to the service account may take several
-    minutes to propagate. The requirement for this workaround will be removed
-    in an upcoming release.
-
-    We recommend that you remove any unneeded permissions from the default compute
-    service account when your use of the Spanner Autoscaler is complete.
-
 ## Preparing the Application Project
 
 In this section you prepare the deployment of the Cloud Scheduler, Forwarder
@@ -417,18 +388,6 @@ topic and function in the project where the Spanner instances live.
     [known issue][provider-issue] in the Terraform Google provider, please retry
     with -parallelism=1
 
-    If you are using a recently created project, or one that does not have permissions
-    automatically assigned to default service accounts, you may see an error message
-    similar to the following:
-
-    ```sh
-    Error: Error waiting for Creating CloudFunctions Function: Error code 3, message: Build failed: failed to Fetch: failed to download archive gs://gcf-sources-[PROJECT_NUMBER]-us-central1/tf-poller-function-[UID]/version-1/function-source.zip:
-    Access to bucket gcf-sources-[PROJECT_NUMBER]-us-central1 denied. You must grant Storage Object Viewer permission to [PROJECT_NUMBER][email protected].
-    ```
-
-    If you see this issue, please follow the instructions in the section
-    [Deploying the Autoscaler](#deploying-the-autoscaler) above.
-
 ### Authorize the Forwarder function to publish to the Poller topic
 
 1.  Switch back to the Autoscaler project and ensure that Terraform variables

diff --git a/terraform/cloud-functions/distributed/app-project/.terraform.lock.hcl b/terraform/cloud-functions/distributed/app-project/.terraform.lock.hcl
diff --git a/terraform/cloud-functions/distributed/app-project/main.tf b/terraform/cloud-functions/distributed/app-project/main.tf
@@ -18,7 +18,7 @@ terraform {
   required_providers {
     google = {
       source  = "hashicorp/google"
-      version = ">= 6.3.0"
+      version = ">= 6.1.0"
     }
   }
 }

diff --git a/terraform/cloud-functions/distributed/autoscaler-project/.terraform.lock.hcl b/terraform/cloud-functions/distributed/autoscaler-project/.terraform.lock.hcl
diff --git a/terraform/cloud-functions/distributed/autoscaler-project/main.tf b/terraform/cloud-functions/distributed/autoscaler-project/main.tf
@@ -18,7 +18,7 @@ terraform {
   required_providers {
     google = {
       source  = "hashicorp/google"
-      version = ">= 6.3.0"
+      version = ">= 6.1.0"
     }
   }
 }

diff --git a/terraform/cloud-functions/per-project/.terraform.lock.hcl b/terraform/cloud-functions/per-project/.terraform.lock.hcl
diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md
@@ -294,35 +294,6 @@ In this section you prepare your project for deployment.
     [known issue][provider-issue] in the Terraform Google provider, please retry
     the command above and include the flag `-parallelism=1`.
 
-    If you are using a recently created project, or one that does not have permissions
-    automatically assigned to default service accounts, you may see an error message
-    similar to the following:
-
-    ```sh
-    Error: Error waiting for Creating CloudFunctions Function: Error code 3, message: Build failed: failed to Fetch: failed to download archive gs://gcf-sources-[PROJECT_NUMBER]-us-central1/tf-poller-function-[UID]/version-1/function-source.zip:
-    Access to bucket gcf-sources-[PROJECT_NUMBER]-us-central1 denied. You must grant Storage Object Viewer permission to [PROJECT_NUMBER][email protected].
-    ```
-
-    A workaround for this issue is to run the following commands, which assign additional
-    permissions to the default compute service account, which is used by Cloud Build:
-
-    ```sh
-    PROJECT_ID=$(gcloud config get-value project)
-    PROJECT_NUMBER=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_NUMBER)")
-    gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"[email protected] --role='roles/storage.objectViewer'
-    gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"[email protected] --role='roles/logging.logWriter'
-    gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"[email protected] --role='roles/artifactregistry.writer'
-    ```
-
-    You can then re-run the `terraform apply` command listed above.
-
-    Please note that the updates to the service account may take several
-    minutes to propagate. The requirement for this workaround will be removed
-    in an upcoming release.
-
-    We recommend that you remove any unneeded permissions from the default compute
-    service account when your use of the Spanner Autoscaler is complete.
-
 ## Importing your Spanner instances
 
 If you have existing Spanner instances that you want to

diff --git a/terraform/cloud-functions/per-project/main.tf b/terraform/cloud-functions/per-project/main.tf
@@ -18,7 +18,7 @@ terraform {
   required_providers {
     google = {
       source  = "hashicorp/google"
-      version = ">= 6.3.0"
+      version = ">= 6.1.0"
     }
   }
 }

diff --git a/terraform/gke/decoupled/.terraform.lock.hcl b/terraform/gke/decoupled/.terraform.lock.hcl
diff --git a/terraform/gke/decoupled/main.tf b/terraform/gke/decoupled/main.tf
@@ -18,7 +18,7 @@ terraform {
   required_providers {
     google = {
       source  = "hashicorp/google"
-      version = ">= 6.3.0"
+      version = ">= 6.1.0"
     }
     kubernetes = {
       source  = "hashicorp/kubernetes"

diff --git a/terraform/gke/unified/.terraform.lock.hcl b/terraform/gke/unified/.terraform.lock.hcl
diff --git a/terraform/gke/unified/main.tf b/terraform/gke/unified/main.tf
@@ -18,7 +18,7 @@ terraform {
   required_providers {
     google = {
       source  = "hashicorp/google"
-      version = ">= 6.3.0"
+      version = ">= 6.1.0"
     }
     kubernetes = {
       source  = "hashicorp/kubernetes"

diff --git a/terraform/modules/autoscaler-base/main.tf b/terraform/modules/autoscaler-base/main.tf
@@ -68,9 +68,18 @@ resource "google_service_account" "build_sa" {
   display_name = "Autoscaler - Cloud Build Builder Service Account"
 }
 
-resource "google_project_iam_binding" "build_iam" {
-  for_each = toset(["roles/storage.objectViewer", "roles/logging.logWriter", "roles/artifactregistry.writer"])
-  project  = var.project_id
-  role     = each.value
-  members  = ["serviceAccount:${google_service_account.build_sa.email}"]
+resource "google_project_iam_member" "build_iam" {
+  for_each = toset([
+    "roles/artifactregistry.writer",
+    "roles/logging.logWriter",
+    "roles/storage.objectViewer",
+  ])
+  project = var.project_id
+  role    = each.value
+  member  = "serviceAccount:${google_service_account.build_sa.email}"
+}
+
+resource "time_sleep" "wait_for_iam" {
+  depends_on      = [google_project_iam_member.build_iam]
+  create_duration = "90s"
 }
diff --git a/terraform/modules/autoscaler-base/outputs.tf b/terraform/modules/autoscaler-base/outputs.tf
@@ -17,10 +17,11 @@
 output "build_sa_id" {
   value       = google_service_account.build_sa.id
   description = "Service account ID for Builder SA"
-  depends_on  = [google_project_iam_binding.build_iam]
+  depends_on  = [time_sleep.wait_for_iam]
 }
+
 output "build_sa_email" {
   value       = google_service_account.build_sa.email
   description = "Service account email for Builder SA"
-  depends_on  = [google_project_iam_binding.build_iam]
+  depends_on  = [time_sleep.wait_for_iam]
 }
diff --git a/terraform/modules/forwarder/main.tf b/terraform/modules/forwarder/main.tf
@@ -22,13 +22,21 @@ resource "google_service_account" "build_sa" {
   display_name = "Autoscaler - Cloud Build Builder Service Account"
 }
 
-resource "google_project_iam_binding" "build_iam" {
-  for_each = toset(["roles/storage.objectViewer", "roles/logging.logWriter", "roles/artifactregistry.writer"])
-  project  = var.project_id
-  role     = each.value
-  members  = ["serviceAccount:${google_service_account.build_sa.email}"]
+resource "google_project_iam_member" "build_iam" {
+  for_each = toset([
+    "roles/artifactregistry.writer",
+    "roles/logging.logWriter",
+    "roles/storage.objectViewer",
+  ])
+  project = var.project_id
+  role    = each.value
+  member  = "serviceAccount:${google_service_account.build_sa.email}"
 }
 
+resource "time_sleep" "wait_for_iam" {
+  depends_on      = [google_project_iam_member.build_iam]
+  create_duration = "90s"
+}
 
 resource "google_service_account" "forwarder_sa" {
   account_id   = "forwarder-sa"
@@ -98,7 +106,7 @@ resource "google_cloudfunctions_function" "forwarder_function" {
   service_account_email = google_service_account.forwarder_sa.email
   build_service_account = google_service_account.build_sa.id
   depends_on = [
-    google_project_iam_binding.build_iam,
+    time_sleep.wait_for_iam,
     google_pubsub_topic_iam_member.forwader_pubsub_sub_binding
   ]
 }