external dns with internal network load balancer

[L040639]

I have exposed 9094 port using internal Network load balancers and the my mirror maker which is another cluster pointing out to the bootstrap load balancer dns which is working fine.

But now i want to replace the load balancers dns to my external dns i.e using aws route53 , so configured below configuration and annotations to my strimzi cluster mentioned below but this was not added any route 53 host and records and also still my services host pointing out to the load balancer dns only .

@scholzj can you please help me here, not sure whether it is config issue ??

        configuration:
          bootstrap:
            alternativeNames:
            - kafka-bootstrap.internal.apps.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.internal.apps.com.
              external-dns.alpha.kubernetes.io/access: "private"
          brokers:
          - broker: 0
            advertisedHost: kafka-broker-0.internal.apps.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-0..internal.apps.com.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 1
            advertisedHost: kafka-broker-1.internal.apps.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-1.internal.apps.com.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 2
            advertisedHost: kafka-broker-1.internal.apps.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-2..internal.apps.com.
              external-dns.alpha.kubernetes.io/access: "private"

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: platform-cluster
  namespace: <XXXX>
spec:
  kafka:
    authorization:
      type: opa
      url: https://<XXX>/v1/data/kafka/allow
      allowOnError: false
      tlsTrustedCertificates:
        - secretName: lk-ca-root-bundle
          certificate: ca.crt
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.7'
    storage:
      type: persistent-claim
      size: 100Gi
      class: efs-sc
    listeners:
      - authentication:
          checkAccessTokenType: false
          clientId: platform-cluster
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: 'https://login.microsoftonline.com/<XXX>/discovery/v2.0/keys'
          maxSecondsWithoutReauthentication: 3600
          type: oauth
          validIssuerUri: 'https://login.microsoftonline.com/<XXX>/v2.0'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: platform-cluster-cert-secret
        name: oauth
        port: 9093
        tls: true
        type: internal
      - authentication:  ## newly added route for the mirror maker.
          checkAccessTokenType: false
          clientId: platform-cluster
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: 'https://login.microsoftonline.com/<XXX>/v2.0/keys'
          maxSecondsWithoutReauthentication: 3600
          type: oauth
          validIssuerUri: 'https://login.microsoftonline.com/<XXX>/v2.0'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: platform-cluster-cert-secret
          bootstrap:
            alternativeNames:
            - kafka-bootstrap.internal.apps.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.internal.apps.com.
              external-dns.alpha.kubernetes.io/access: "private"
          brokers:
          - broker: 0
            advertisedHost: kafka-broker-0.internal.apps.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-0.internal.apps.com.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 1
            advertisedHost: kafka-broker-1.internal.apps.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-1.internal.apps.com.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 2
            advertisedHost: kafka-broker-1.internal.apps.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-2.internal.apps.com.
              external-dns.alpha.kubernetes.io/access: "private"
        name: external
        port: 9094
        tls: true
        type: loadbalancer 
    version: 3.7.0
    replicas: 3
    # resources:
    #   requests:
    #     cpu: 250m
    #     memory: 4Gi
    #   limits:
    #     cpu: 500m
    #     memory: 8Gi
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-metrics
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - platform-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - kafka
                topologyKey: kubernetes.io/hostname
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: platform-cluster
                strimzi.io/component-type: kafka
  entityOperator:
    topicOperator: {}
    userOperator: {}
  zookeeper:
    storage:
      type: persistent-claim
      class: efs-sc
      size: 10Gi
    replicas: 3
    # resources:
    #   requests:
    #     cpu: 125m
    #     memory: 400Mi
    #   limits:
    #     cpu: 250m
    #     memory: 1Gi
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - platform-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - zookeeper
                topologyKey: kubernetes.io/hostname
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: platform-cluster
                strimzi.io/component-type: zookeeper
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

[scholzj]

The External DNS annotations are consumed by the External DNS tool. Strimzi only passes them to the Service resources and from there ExternalDNS has to pick them up. So if you do not have the DNS records, you should check:

• If Strimzi passed these annotations to the Service resources
• And if it did, the rest is up to External DNS -> so you would need to investigate its own logs.

[L040639]

@scholzj , question here do i need to maintain a separate server for the external dns or the kubernetes service will take care of that based on the anotations we provided ?

[scholzj]

ExternalDNS is a separate Kubernetes project that you need to deploy and run for these annotations to have any effect: https://github.com/kubernetes-sigs/external-dns

[L040639]

Make sense, i though it was strimzi inbuilt. Thanks for the clarification .

[scholzj]

No, sorry for the confusion. Its just an integration with other tool from the wider Kubernetes ecosystem.