Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FE Proxy fails to route the requests correctly if the non-default dns setup is utilized. #49984

Closed
zakhalex opened this issue Aug 19, 2024 · 5 comments
Closed

FE Proxy fails to route the requests correctly if the non-default dns setup is utilized. #49984

zakhalex opened this issue Aug 19, 2024 · 5 comments
Assignees
@kevincai @yandongxiao
Labels
type/bug Something isn't working

Comments

@zakhalex
Copy link

FE Proxy fails to route the requests correctly if the non-default dns setup is utilized. When I modify values.yaml setting up
resolver: "kube-dns.kube-system.svc.mycustomizations.com"
in line 1074 of https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml
the proxy starts up, but upon receiving streming load request to upload a csv file fails with errors like
"PUT /api/my_db/my_jobs/_stream_load HTTP/1.1" 502 182 "-" "curl/8.2.1" ││ 2024/08/14 15:00:03 [error] 20#20: *18 kube-starrocks-fe-service.starrocks.svc.cluster.local could not be resolved (3: Host not found) indicating that custom setting does not get fully propagated.
Starrocks version is 3.3.2 setup in Kubernetes cluster. The deployment is done through kubernetes operator helm chart (https://docs.starrocks.io/docs/deployment/helm/).

The issue appears both on default version of nginx and on v1.27.

At the same time regular FE has no issues with resolution (as seen on the attached screenshot) - only FE Proxy has.

image

Steps to reproduce the behavior (Required)

  1. Create a .csv file that looks like

"col1","col2"
1,"ABC"
2,"DEF"
3,"GHI"

  1. Deploy starrocks on kubernetes cluster with fe proxy. Let's assume root username and password are root and blank password correspondently.
  2. Create database and add a table that matches your CSV file. Let's say you go with
CREATE DATABASE control_db;
USE control_db;
CREATE TABLE test_table(
    Col1 INT,
    Col2 VARCHAR(255)
)
  1. Make sure curl is installed on your machine and execute:
    curl --location-trusted -u root: -H "label:custom_label1" -H "Expect:100-continue" -H "column_separator:," -H "columns: col1, col2" -T Tab1.csv -XPUT http://<fe_proxy_ip>:<fe_proxy_port>/api/control_db/test_table/_stream_load

Expected behavior (Required)

{
    "TxnId": 12345,
    "Label": "custom_label1",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 3,
    "NumberLoadedRows": 3,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1024,
    "LoadTimeMs": 100,
    "BeginTxnTimeMs": 1,
    "StreamLoadPlanTimeMs": 15,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 99,
    "CommitAndPublishTimeMs": 101
}

Real behavior (Required)

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.27.0</center>
</body>
</html>

StarRocks version (Required)

3.3.2 (but initially observed on 3.3.1)
@zakhalex zakhalex added the type/bug Something isn't working label Aug 19, 2024
@kevincai
Copy link
Contributor

kevincai commented Nov 26, 2024 •
edited
Loading

kube-starrocks-fe-service.starrocks.svc.cluster.local could not be resolved (3: Host not found)

did you have a customized domain name setup for the kubernetes cluster instead of default .cluster.local?

@zakhalex what's the canonical service dns name in your kubernetes cluster setup?

@zakhalex
Copy link
Author

@kevincai correct - in the environment where this issue happens, it is a customized domain name instead of the default .cluster.local (In other words - it is NOT cluster.local)

@kevincai
Copy link
Contributor

@zakhalex it is caused by the hardcode domain name of the kubernetes cluster in line https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/pkg/subcontrollers/feproxy/feproxy_configmap.go#L30

You can modify the line, change the svc.cluster.local to your customized domain name svc.mycustomization.com.

We will take a look how to support customized domain name in operator.

@yandongxiao yandongxiao mentioned this issue Nov 27, 2024
Merged
4 tasks
@kevincai
Copy link
Contributor

@kevincai correct - in the environment where this issue happens, it is a customized domain name instead of the default .cluster.local (In other words - it is NOT cluster.local)

@zakhalex wait for the next release of the operator to suppport customized domain setup. refer to StarRocks/starrocks-kubernetes-operator#604

@zakhalex
Copy link
Author

Thank you, @kevincai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kevincai kevincai

@yandongxiao yandongxiao

Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kevincai @yandongxiao @zakhalex