Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

client/metadata got error from broker -1 while fetching metadata #9

Open
Hubbitus opened this issue Sep 22, 2020 · 11 comments
Open

client/metadata got error from broker -1 while fetching metadata #9

Hubbitus opened this issue Sep 22, 2020 · 11 comments

Comments

@Hubbitus
Copy link

Hubbitus commented Sep 22, 2020 •
edited
Loading

Hello.

I am newbie and try to use kafkactl:

podman run -it --rm --name kafkactl -v ~/.config/kafkactl/config.yml:/etc/kafkactl/config.yml \
    docker.io/deviceinsight/kafkactl:v1.11.1 \
        kafkactl --verbose get topics
[kafkactl] 2020/09/22 09:02:18 Using config file: /etc/kafkactl/config.yml
[kafkactl] 2020/09/22 09:02:18 Assuming kafkaVersion: 2.0.0
[kafkactl] 2020/09/22 09:02:18 Assuming kafkaVersion: 2.0.0
[sarama  ] 2020/09/22 09:02:18 Initializing new client
[sarama  ] 2020/09/22 09:02:18 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:18 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:02:48 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44214->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:02:48 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:48 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:02:48 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:02:48 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama  ] 2020/09/22 09:02:48 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:48 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:03:18 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44222->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:03:18 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:18 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:03:18 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:03:18 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama  ] 2020/09/22 09:03:18 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:18 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:03:48 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44228->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:03:48 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:48 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:03:48 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:03:48 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama  ] 2020/09/22 09:03:49 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:49 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:04:19 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44244->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:04:19 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:04:19 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:04:19 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:04:19 Closing Client
failed to create cluster admin: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Config is very simple (sed to show just non-comments and non-empty lines):

$ sed 's/#.*$//;/^$/d' .kafkactl.yaml
contexts:
  sandbox:
    name: sandbox
    brokers:
      - ecsc00a060af.epam.com:9092

current-context: sandbox

And according to wireshark dump TCP session kafkactl send metadata request and can connect to the server, but newer receive response:
image

Where I am wrong?
@jbvmio
Copy link
Owner

jbvmio commented Sep 23, 2020

@Hubbitus - you're using the wrong kafkactl. In your output, deviceinsight/kafkactl, if you want to use this implementation, please try jbvmio/kafkactl.

@Hubbitus
Copy link
Author

Hubbitus commented Sep 23, 2020 •
edited
Loading

I'm very sorry for the mistake. And thank you for the response.

Meantime it changes almost nothing. Your image requires config in single place versus search in some common, but then failed with the same error without details:

$ podman run -it --rm --name kafkactl -v ~/.config/kafkactl/config.yml:/root/.kafkactl.yaml \
        docker.io/jbvmio/kafkactl:v1.0.25 \
                --verbose get topics
INFO[2020-09-23T09:03:21Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:21Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:21Z] Initializing new client                      
INFO[2020-09-23T09:03:21Z] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:22Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:27Z] client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:54082->10.6.194.60:9092: i/o timeout 
INFO[2020-09-23T09:03:27Z] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:27Z] client/metadata no available broker to send metadata request to 
INFO[2020-09-23T09:03:27Z] client/brokers resurrecting 1 dead seed brokers 
INFO[2020-09-23T09:03:27Z] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2020-09-23T09:03:27Z] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:27Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:32Z] client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:54088->10.6.194.60:9092: i/o timeout 
INFO[2020-09-23T09:03:32Z] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:32Z] client/metadata no available broker to send metadata request to 
INFO[2020-09-23T09:03:32Z] client/brokers resurrecting 1 dead seed brokers 
INFO[2020-09-23T09:03:32Z] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

@Hubbitus
Copy link
Author

And it looks like kafkactl request and receive brokers metadata, but stops on that:
image
Please compare with successful session list topic from kafkacat:
image

@jbvmio
Copy link
Owner

jbvmio commented Sep 24, 2020

What version of Kafka are your brokers running @Hubbitus ? Can you try adding the --version 1.0.0 flag? Curious have you tried running the direct binary instead of Docker?

@Hubbitus
Copy link
Author

I'm unsure about the remote broker's version (waiting answer).
Use binary of kafkactl compiled locally changes nothin as I see. --version 1.0.0 also does not helps.

Still, first metadata comes and then tons of TCP retransmissions only:
image

@zotona
Copy link

zotona commented Oct 5, 2020 •
edited
Loading

i have the same issue

⇒  kafkactl -B 15.247.193.149:32764 --verbose
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Initializing new client
INFO[2020-10-05T19:01:53+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:53+03:00] Closed connection to broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:53+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:53+03:00] client/metadata retrying after 250ms... (1 attempts remaining)
INFO[2020-10-05T19:01:54+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:54+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:54+03:00] Closed connection to broker 185.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:54+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:54+03:00] Closing Client
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

we have a kafka at k8s cluster and exposed via ingress
and in documentation i could not find how setup sasl/ssl

@Hubbitus
Copy link
Author

Sorry, maybe you may at least extend logging what happened and what wrong?

@jbvmio
Copy link
Owner

jbvmio commented Nov 27, 2020 •
edited
Loading

i have the same issue

⇒  kafkactl -B 15.247.193.149:32764 --verbose
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Initializing new client
INFO[2020-10-05T19:01:53+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:53+03:00] Closed connection to broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:53+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:53+03:00] client/metadata retrying after 250ms... (1 attempts remaining)
INFO[2020-10-05T19:01:54+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:54+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:54+03:00] Closed connection to broker 185.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:54+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:54+03:00] Closing Client
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

we have a kafka at k8s cluster and exposed via ingress
and in documentation i could not find how setup sasl/ssl

@zotona You can configure sasl/ssl using the following example in your config file:

  devel-cluster:
    name: devel-cluster
    brokers:
    - devel01.example.com:9092
    - devel02.example.com:9092
    - devel03.example.com:9092
    zookeeper:
    - devel01.example.com:2181
    - devel02.example.com:2181
    - devel03.example.com:2181
    clientVersion: 2.1.1
    ssl:
      insecure: true
      tlscert: /path/to/cert.pem
      tlsca: /path/to/ca.pem
      tlskey: /path/to/key.pem
    sasl:
      username: USER
      password: PASS

@jbvmio
Copy link
Owner

jbvmio commented Nov 27, 2020

@Hubbitus

It seems there is either a connectivity issue or more likely the version of Kafka your brokers are running are not compatible. Did you ever confirm the version of Kafka your brokers are running?

@jbvmio
Copy link
Owner

jbvmio commented Nov 27, 2020

@Hubbitus - As a test, try the forcing the following client versions:

--version 0.10.1.0
--version 0.10.2.0
--version 0.11.0.0

@Hubbitus
Copy link
Author

Hubbitus commented Mar 26, 2021 •
edited
Loading 

$ kafkactl --verbose get topics                                                                                                                   
INFO[2021-03-26T17:31:36+03:00] Failed to connect to broker ecsc00a060af.epam.com:9092: dial tcp: i/o timeout 
Error connecting to Kafka: No Connectivity to Provided Brokers
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.10.1.0 get topics
INFO[2021-03-26T17:31:50+03:00] Initializing new client                      
INFO[2021-03-26T17:31:50+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:51+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:31:56+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42192->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:31:56+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:56+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:31:56+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:31:56+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:31:56+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:56+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:01+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42242->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:01+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:01+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:01+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:01+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.10.2.0 get topics
INFO[2021-03-26T17:32:15+03:00] Initializing new client                      
INFO[2021-03-26T17:32:15+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:15+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:20+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42404->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:20+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:20+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:20+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:20+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:20+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:20+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:25+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42446->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:25+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:25+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:25+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:25+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.11.0.0 get topics
INFO[2021-03-26T17:32:35+03:00] Initializing new client                      
INFO[2021-03-26T17:32:35+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:35+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:40+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42570->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:40+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:40+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:40+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:40+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:41+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:41+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:46+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42612->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:46+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:46+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:46+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:46+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 1.0.0 get topics
INFO[2021-03-26T17:32:51+03:00] Initializing new client                      
INFO[2021-03-26T17:32:51+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:51+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:56+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42694->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:56+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:56+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:56+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:56+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:56+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:56+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:33:01+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42736->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:33:01+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:33:01+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:33:01+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:33:01+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

On server: Confluent 5.3.2, Kafka 2.3.x

And off course such cluster reachable from kafkacat and confluent utilities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Hubbitus @zotona @jbvmio