Remote module from private repos

[oleksandr-polishchuk1]

What happened?

I am trying to setup provider terraform to use remote modules. It works fine for public git repos, but it doesn't for private repos.

How can we reproduce it?

provider.yaml

---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: terraform-config
  labels:
    app: crossplane-provider-terraform
spec:
  args: ["-d"]
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: crossplane-provider-terraform
spec:
  package: xpkg.upbound.io/upbound/provider-terraform:v0.7.0
  controllerConfigRef:
    name: terraform-config

providerconfig.yaml

apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  # Note that unlike most provider configs this one supports an array of
  # credentials. This is because each Terraform workspace uses a single
  # Crossplane provider config, but could use multiple Terraform providers each
  # with their own credentials.
  credentials:
    - filename: .git-credentials # use exactly this filename
      source: Secret
      secretRef:
        namespace: crossplane
        name: git-credentials
        key: git-credentials
  # This optional configuration block can be used to inject HCL into any
  # workspace that uses this provider config, for example to setup Terraform
  # providers.
  configuration: |
    terraform {
      required_providers {
        hcloud = {
          source  = "hetznercloud/hcloud"
          version = "1.40.0"
        }
      }
    }

    // Modules _must_ use remote state. The provider does not persist state.
    terraform {
      backend "kubernetes" {
        secret_suffix     = "providerconfig-default"
        namespace         = "crossplane-system"
        in_cluster_config = true
      }
    }

workspace.yaml

apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  name: sample-remote
spec:
  forProvider:
    module: git::https://github.com/USERNAME/REPO-NAME.git
    source: Remote

git-credentials secret was created from file git-credentials

git-credentials file

https://USERNAME:[email protected]

logs from provider's pod:

2023-06-27T07:51:41.281Z	DEBUG	provider-terraform	Cannot connect to provider	{"controller": "managed/workspace.tf.upbound.io", "request": "/sample-remote", "uid": "95f66631-1c89-4d6d-b9ef-1b6fee7ca71c", "version": "3324838", "external-name": "sample-remote", "error": "cannot get remote Terraform module: error downloading 'https://github.com/USERNAME/REPO-NAME.git': /usr/bin/git exited with 128: Cloning into '/tf/95f66631-1c89-4d6d-b9ef-1b6fee7ca71c'...\nfatal: could not read Username for 'https://github.com/': No such device or address\n", "errorVerbose": "error downloading 'https://github.com/USERNAME/REPO-NAME.git': /usr/bin/git exited with 128: Cloning into '/tf/95f66631-1c89-4d6d-b9ef-1b6fee7ca71c'...\nfatal: could not read Username for 'https://github.com/': No such device or address\n\ncannot get remote Terraform module\ngithub.com/upbound/provider-terraform/internal/controller/workspace.(*connector).Connect\n\tgithub.com/upbound/provider-terraform/internal/controller/workspace/workspace.go:229\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1598"}
2023-06-27T07:51:41.282Z	DEBUG	events	cannot get remote Terraform module: error downloading 'https://github.com/USERNAME/REPO-NAME.git': /usr/bin/git exited with 128: Cloning into '/tf/95f66631-1c89-4d6d-b9ef-1b6fee7ca71c'...
fatal: could not read Username for 'https://github.com/': No such device or address
	{"type": "Warning", "object": {"kind":"Workspace","name":"sample-remote","uid":"95f66631-1c89-4d6d-b9ef-1b6fee7ca71c","apiVersion":"tf.upbound.io/v1beta1","resourceVersion":"3324838"}, "reason": "CannotConnectToProvider"}

What environment did it happen in?

• Crossplane Version: universal-crossplane-1.9.2-up.1
• Provider Version: v0.6.0 - v0.8.0
• Kubernetes Version: v1.25.2

[oleksandr-polishchuk1]

UPD: it works locally with .gitconfig and .git-credentials from the pod

[ytsarev]

I've double-checked on my side. The private repository is getting checked out properly. The same worked for the Upbound customers I worked with this week, no issues.

I think you should double-check credentials/repo only on your side. There might be a culprit like expired PAT, typo in the repo URL or similar... https://www.positioniseverything.net/fatal-could-not-read-username-for-https_github_com-terminal-prompts-disabled/

[zbblanton-built]

Having the exact same problems.

The only workaround I've found so far is shelling into the pod and running:

cp .gitconfig /root/

Once I do that, the workspaces is able to pull down the private repo successfully. This is not really feasible since we need to do this every time the pod restart.

Setup:

• This is running v0.10.0
• The user is an org enabled user and all repos being pulled are under an organization

Update:

We had the securityContext set to root in the controller config. Removed that and everything started working!

  securityContext:
    runAsUser: 0