Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Doc] add the ports that need to be opened in connectors docs (backport #44154) #44171

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/en/loading/Flink-connector-starrocks.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,10 @@ DISTRIBUTED BY HASH(`id`);
./bin/start-cluster.sh
```

#### Network configuration

Ensure that the machine where Flink is located can access the machine where the FE or FEs in a StarRocks cluster are located through the [`http_port`](../administration/management/FE_configuration.md#http_port) and [`query_port`](../administration/management/FE_configuration.md#query_port), and access the machines where the BEs in a StarRocks cluster are located through the [`http_port`](../administration/management/BE_configuration.md#be_http_port).

### Run with Flink SQL

- Run the following command to start a Flink SQL client.
Expand Down
4 changes: 4 additions & 0 deletions docs/en/loading/Kafka-connector-starrocks.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ Submit the Kafka connector into Kafka Connect:

Currently, the Kafka connector is not uploaded to Confluent Hub. You need to download and extract [starrocks-kafka-connector-xxx.tar.gz](https://github.com/StarRocks/starrocks-connector-for-kafka/releases), package it into a ZIP file and upload the ZIP file to Confluent Cloud.

### Network configuration

Ensure that the machine where Kafka is located can access the machine where the FE or FEs in a StarRocks cluster are located through the [`http_port`](../administration/management/FE_configuration.md#http_port) and [`query_port`](../administration/management/FE_configuration.md#query_port), and access the machines where the BEs in a StarRocks cluster are located through the [`http_port`](../administration/management/BE_configuration.md#be_http_port).

## Usage

This section uses a self-managed Kafka cluster as an example to explain how to configure the Kafka connector and the Kafka Connect, and then run the Kafka Connect to load data into StarRocks.
Expand Down
4 changes: 4 additions & 0 deletions docs/en/loading/Spark-connector-starrocks.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,10 @@ COMMENT "OLAP"
DISTRIBUTED BY HASH(`id`);
```

#### Network configuration

Ensure that the machine where Spark is located can access the machine where the FE or FEs in a StarRocks cluster are located through the [`http_port`](../administration/management/FE_configuration.md#http_port) and [`query_port`](../administration/management/FE_configuration.md#query_port), and access the machines where the BEs in a StarRocks cluster are located through the [`http_port`](../administration/management/BE_configuration.md#be_http_port).

#### Set up your Spark environment

Note that the following examples are run in Spark 3.2.4 and use `spark-shell`, `pyspark` and `spark-sql`. Before running the examples, make sure to place the Spark connector JAR file in the `$SPARK_HOME/jars` directory.
Expand Down
12 changes: 8 additions & 4 deletions docs/en/unloading/Flink_connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,19 +85,23 @@ You can also deploy Flink by following the instructions provided in [Flink docum

## Before you begin

### Deploy the Flink connector

Follow these steps to deploy the Flink connector:

1. Select and download the [flink-connector-starrocks](https://github.com/StarRocks/flink-connector-starrocks/releases) JAR package matching the Flink version that you are using.
1. Select and download the [flink-connector-starrocks](https://github.com/StarRocks/flink-connector-starrocks/releases) JAR package matching the Flink version that you are using. If code debugging is needed, compile the Flink connector package to suit your business requirements.

> **NOTICE**
>
> We recommend that you download the Flink connector package whose version is 1.2.x or later and whose matching Flink version has the same first two digits as the Flink version that you are using. For example, if you use Flink v1.14.x, you can download `flink-connector-starrocks-1.2.4_flink-1.14_x.yy.jar`.

2. If code debugging is needed, compile the Flink connector package to suit your business requirements.
2. Place the Flink connector package you downloaded or compiled into the `lib` directory of Flink.

3. Restart your Flink cluster.

3. Place the Flink connector package you downloaded or compiled into the `lib` directory of Flink.
### Network configuration

4. Restart your Flink cluster.
Ensure that the machine where Flink is located can access the machine where the FE or FEs in a StarRocks cluster are located through the [`http_port`](../administration/management/FE_configuration.md#http_port) and [`query_port`](../administration/management/FE_configuration.md#query_port), and access the machines where the BEs in a StarRocks cluster are located through the [`http_port`](../administration/management/BE_configuration.md#be_http_port).

## Parameters

Expand Down
4 changes: 4 additions & 0 deletions docs/en/unloading/Spark_connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,10 @@ The processing logic of the underlying storage engine used by StarRocks cannot c

The following examples assume you have created a database named `test` in your StarRocks cluster and you have the permissions of user `root`. The parameter settings in the examples are based on Spark Connector 1.1.0.

### Network configuration

Ensure that the machine where Spark is located can access the machine where the FE or FEs in a StarRocks cluster are located through the [`http_port`](../administration/management/FE_configuration.md#http_port) and [`query_port`](../administration/management/FE_configuration.md#query_port), and access the machines where the BEs in a StarRocks cluster are located through the [`http_port`](../administration/management/BE_configuration.md#be_http_port).

### Data example

Do as follows to prepare a sample table:
Expand Down
4 changes: 4 additions & 0 deletions docs/zh/loading/Flink-connector-starrocks.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,10 @@ DISTRIBUTED BY HASH(id);
./bin/start-cluster.sh
```

#### 网络配置

确保 Flink 所在机器能够访问 StarRocks 集群中 FE 所在机器的 [`http_port`](../administration/management/FE_configuration.md#http_port) 和 [`query_port`](../administration/management/FE_configuration.md#query_port) 端口,以及 BE 所在机器的 [`http_port`](../administration/management/BE_configuration.md#be_http_port) 端口。

### 使用 Flink SQL 写入数据

- 运行以下命令以启动 Flink SQL 客户端。
Expand Down
4 changes: 4 additions & 0 deletions docs/zh/loading/Kafka-connector-starrocks.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ StarRocks 提供 Apache Kafka® 连接器 (StarRocks Connector for Apache Kafka

Kafka connector 目前尚未上传到 Confluent Hub,您需要下载并解压 [starrocks-kafka-connector-xxx.tar.gz](https://github.com/StarRocks/starrocks-connector-for-kafka/releases) ,打包成 ZIP 文件并上传到 Confluent Cloud。

### 网络配置

确保 Kafka 所在机器能够访问 StarRocks 集群中 FE 所在机器的 [`http_port`](../administration/management/FE_configuration.md#http_port) 和 [`query_port`](../administration/management/FE_configuration.md#query_port) 端口,以及 BE 所在机器的 [`http_port`](../administration/management/BE_configuration.md#be_http_port) 端口。

## 使用示例

本文以自建 Kafka 集群为例,介绍如何配置 Kafka connector 和 Kafka connect,然后启动 Kafka Connect 导入数据至 StarRocks。
Expand Down
4 changes: 4 additions & 0 deletions docs/zh/loading/Spark-connector-starrocks.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,10 @@ DISTRIBUTED BY HASH(`id`)

示例基于 Spark 3.2.4,使用 `spark-shell`,`pyspark` 和 `spark-sql` 进行演示,运行前请将 connector jar放置在 `$SPARK_HOME/jars` 目录下。

#### 网络配置

确保 Spark 所在机器能够访问 StarRocks 集群中 FE 所在机器的 [`http_port`](../administration/management/FE_configuration.md#http_port) 和 [`query_port`](../administration/management/FE_configuration.md#query_port) 端口,以及 BE 所在机器的 [`http_port`](../administration/management/BE_configuration.md#be_http_port) 端口。

### 使用 Spark DataFrame 写入数据

下面分别介绍在 Batch 和 Structured Streaming 下如何写入数据。
Expand Down
12 changes: 8 additions & 4 deletions docs/zh/unloading/Flink_connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,19 +85,23 @@ Flink Connector 支持两种数据读取方式:Flink SQL 和 Flink DataStream

## 准备工作

### 部署 Flink Connector

通过如下步骤完成 Flink Connector 的部署:

1. 根据 Flink 的版本,选择和下载对应版本的 [flink-connector-starrocks](https://github.com/StarRocks/flink-connector-starrocks/releases) JAR 包。
1. 根据 Flink 的版本,选择和下载对应版本的 [flink-connector-starrocks](https://github.com/StarRocks/flink-connector-starrocks/releases) JAR 包。如需调试代码,可选择对应分支代码自行编译。

> **注意**
>
> 推荐您下载 Flink Connector 版本在 1.2.x 及以上、并且配套的 Flink 版本与您的业务环境中安装的 Flink 版本前两位一致的 JAR 包。例如,如果您的业务环境中安装的 Flink 版本为 1.14.x,可以下载 `flink-connector-starrocks-1.2.4_flink-1.14_x.yy.jar`。

2. 如需调试代码,可选择对应分支代码自行编译。
2. 将下载或者编译的 JAR 包放在 Flink 的 `lib` 目录中。

3. 重启 Flink。

3. 将下载或者编译的 JAR 包放在 Flink 的 `lib` 目录中。
### 网络设置

4. 重启 Flink
确保 Flink 所在机器能够访问 StarRocks 集群中 FE 所在机器的 [`http_port`](../administration/management/FE_configuration.md#http_port) 和 [`query_port`](../administration/management/FE_configuration.md#query_port) 端口,以及 BE 所在机器的 [`http_port`](../administration/management/BE_configuration.md#be_http_port) 端口

## 参数说明

Expand Down
4 changes: 4 additions & 0 deletions docs/zh/unloading/Spark_connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,10 @@ Spark Connector 中,将 DATE 和 DATETIME 数据类型映射为 STRING 数据

假设您的 StarRocks 集群中已创建数据库 `test`,并且您拥有 `root` 账号权限。示例的参数配置基于 Spark Connector 1.1.0 版本。

### 网络设置

确保 Spark 所在机器能够访问 StarRocks 集群中 FE 所在机器的 [`http_port`](../administration/management/FE_configuration.md#http_port) 和 [`query_port`](../administration/management/FE_configuration.md#query_port) 端口,以及 BE 所在机器的 [`http_port`](../administration/management/BE_configuration.md#be_http_port) 端口。

### 数据样例

执行如下步骤,准备数据样例:
Expand Down
Loading