How Can I use Flux from multiple computers?

[budimanjojo]

Hello, I have managed to install flux and everything is working until I decided to control the same cluster from my another computer. I do the same flux bootstrap command in my other computer and it's stuck in reconciliation in progress. Here's the log of my kustomize-controller:

{"level":"error","ts":"2021-06-27T16:28:52.705Z","logger":"controller.kustomization","msg":"Reconciliation failed after 8m35.052389546s, next try in 10m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"flux-system","namespace":"flux-system","revision":"main/8c6a6fc59c347db2dbdf93fd9d218cb10d50c868","error":"failed to download artifact, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/8c6a6fc59c347db2dbdf93fd9d218cb10d50c868.tar.gz giving up after 10 attempt(s): Get \"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/8c6a6fc59c347db2dbdf93fd9d218cb10d50c868.tar.gz\": dial tcp 10.109.178.92:80: i/o timeout"}
{"level":"info","ts":"2021-06-27T16:29:12.624Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:29:26.891Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:29:42.642Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:30:12.681Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:30:42.698Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:31:12.721Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:31:14.545Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:31:42.755Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:32:12.776Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:32:42.793Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:33:12.812Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:33:42.840Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:34:12.863Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:34:42.883Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:35:12.899Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:35:42.920Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"info","ts":"2021-06-27T16:36:12.936Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}
{"level":"error","ts":"2021-06-27T16:36:18.084Z","logger":"controller.kustomization","msg":"Reconciliation failed after 8m35.041246698s, next try in 1m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"crds","namespace":"flux-system","revision":"main/204ae502c5d0d3ffdc924724cd24cf1fdd3829e2","error":"failed to download artifact, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/204ae502c5d0d3ffdc924724cd24cf1fdd3829e2.tar.gz giving up after 10 attempt(s): Get \"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/204ae502c5d0d3ffdc924724cd24cf1fdd3829e2.tar.gz\": dial tcp 10.109.178.92:80: i/o timeout"}
{"level":"error","ts":"2021-06-27T16:36:22.231Z","logger":"controller.kustomization","msg":"Reconciliation failed after 8m35.043269594s, next try in 1m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"apps","namespace":"flux-system","revision":"main/204ae502c5d0d3ffdc924724cd24cf1fdd3829e2","error":"failed to download artifact, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/204ae502c5d0d3ffdc924724cd24cf1fdd3829e2.tar.gz giving up after 10 attempt(s): Get \"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/204ae502c5d0d3ffdc924724cd24cf1fdd3829e2.tar.gz\": dial tcp 10.109.178.92:80: i/o timeout"}
{"level":"info","ts":"2021-06-27T16:36:42.956Z","logger":"controller.kustomization","msg":"Dependencies do not meet ready condition, retrying in 30s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"core","namespace":"flux-system"}

Also worth noting that I use 2 different github token for both system (because apparently I can't copy the token from github anymore). I can't seem to find any other useful logs though, the flux logs --verbose shows no error at all:

2021-06-27T16:33:02.354Z info GitRepository/flux-system.flux-system - Reconciliation finished in 2.824140932s, next run in 1m0s
2021-06-27T16:34:04.905Z info GitRepository/flux-system.flux-system - Reconciliation finished in 2.550550786s, next run in 1m0s
2021-06-27T16:35:07.691Z info GitRepository/flux-system.flux-system - Reconciliation finished in 2.785068572s, next run in 1m0s
2021-06-27T16:35:25.521Z info HelmRepository/jetstack-charts.flux-system - Reconciliation finished in 900.276393ms, next run in 10m0s
2021-06-27T16:36:10.336Z info GitRepository/flux-system.flux-system - Reconciliation finished in 2.644516058s, next run in 1m0s
2021-06-27T16:37:12.917Z info GitRepository/flux-system.flux-system - Reconciliation finished in 2.58128277s, next run in 1m0s
2021-06-27T16:38:15.519Z info GitRepository/flux-system.flux-system - Reconciliation finished in 2.601278826s, next run in 1m0s

I can see that in flux-system namespace there is a secret called flux-system and looks like there's identity, identity.pub, known_hosts fields. Maybe I should add my other system ssh keys there? Thanks before.

[stefanprodan]

You run bootstrap once per cluster, from which local machine you run it has no importance. Then you would drive operations on your cluster using git from any machine.

[stefanprodan]

Delete the clusters dir after flux uninstall, this will cleanup the yamls pushed by flux on your repo.

[budimanjojo]

Sorry, I still cant get it working with github even after deleting the entire cluster repository. And now I think I know why. My github is configured to authenticate with ssh public key and the flux bootstrap command seems to use the access token to sync, maybe that's why? How do I ask flux to bootstrap using ssh key instead? Thank you

[budimanjojo]

Flux created the manifests with ssh url though:

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: ssh://[email protected]/budimanjojo/home-cluster

[budimanjojo]

The ssh url is wrong though, it should be:
ssh://[email protected]:budimanjojo/home-cluster
instead of:
ssh://[email protected]/budimanjojo/home-cluster

I tried committing the right url to github and editing the manifest directly in the cluster and still not working. Any idea? spec.url was created by flux bootstrap.

[budimanjojo]

I found the problem, it was coredns can't resolve the kustomize controller pod to github, I don't know why but after restarting coredns pod then restart flux pods it works.
And I also figure out I can use --auth-token option to force flux to use token instead of ssh key. Although now it looks like the whole problem was because of coredns. But the weird thing was coredns can resolve to other domains except github.com