diff --git a/connectors/google-bigquery-sink/v2.10.5.9/google-bigquery-sink.md b/connectors/google-bigquery-sink/v2.10.5.9/google-bigquery-sink.md index ecdac7f65..760c889ac 100644 --- a/connectors/google-bigquery-sink/v2.10.5.9/google-bigquery-sink.md +++ b/connectors/google-bigquery-sink/v2.10.5.9/google-bigquery-sink.md @@ -49,28 +49,40 @@ The prerequisites for connecting an Google BigQuery sink connector to external s ### 1. Create a connector -Depending on the environment, there are several ways to create an Google BigQuery sink connector: +The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector, +you need to replace `--sink-type bigquery` with `--archive /path/to/pulsar-io-bigquery.jar`. You can find the button to download the `jar` package at the beginning of the document. -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **JAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-bigquery/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-bigquery/tags) - -No matter how you create an Google BigQuery sink connector, the minimum configuration contains the following parameters. +{% callout title="For StreamNative Cloud User" type="note" %} +If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first. +{% /callout %} -```yaml - configs: - projectId: {{Your BigQuery project id}} - datasetName: {{Your Bigquery DataSet name}} - tableName: {{The name of the table you want to write data to is automatically created by default}} - credentialJsonString: {{Public key certificate you created above}} -``` +```bash +pulsarctl sinks create \ + --sink-type bigquery \ + --name bigquery-sink \ + --tenant public \ + --namespace default \ + --inputs "Your topic name" \ + --parallelism 1 \ + --sink-config \ + '{ + "projectId": "Your BigQuery project Id", + "datasetName": "Your Bigquery DataSet name", + "tableName": "The name of the table you want to write data to is automatically created by default", + "credentialJsonString": "Public key certificate you created above" + }' +``` + +The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own. +If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. -> * The configuration structure varies depending on how you create the Google BigQuery sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. +{% callout title="Note" type="note" %} +You can also choose to use a variety of other tools to create a connector: +- [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector ). +- [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document. + {% /callout %} ### 2. Send messages to the topic diff --git a/connectors/google-bigquery-sink/v2.11.2.13/google-bigquery-sink.md b/connectors/google-bigquery-sink/v2.11.2.13/google-bigquery-sink.md index 1926fc007..24e62cf43 100644 --- a/connectors/google-bigquery-sink/v2.11.2.13/google-bigquery-sink.md +++ b/connectors/google-bigquery-sink/v2.11.2.13/google-bigquery-sink.md @@ -49,28 +49,40 @@ The prerequisites for connecting an Google BigQuery sink connector to external s ### 1. Create a connector -Depending on the environment, there are several ways to create an Google BigQuery sink connector: +The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector, +you need to replace `--sink-type bigquery` with `--archive /path/to/pulsar-io-bigquery.jar`. You can find the button to download the `jar` package at the beginning of the document. -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **JAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-bigquery/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-bigquery/tags) - -No matter how you create an Google BigQuery sink connector, the minimum configuration contains the following parameters. +{% callout title="For StreamNative Cloud User" type="note" %} +If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first. +{% /callout %} -```yaml - configs: - projectId: {{Your BigQuery project id}} - datasetName: {{Your Bigquery DataSet name}} - tableName: {{The name of the table you want to write data to is automatically created by default}} - credentialJsonString: {{Public key certificate you created above}} -``` +```bash +pulsarctl sinks create \ + --sink-type bigquery \ + --name bigquery-sink \ + --tenant public \ + --namespace default \ + --inputs "Your topic name" \ + --parallelism 1 \ + --sink-config \ + '{ + "projectId": "Your BigQuery project Id", + "datasetName": "Your Bigquery DataSet name", + "tableName": "The name of the table you want to write data to is automatically created by default", + "credentialJsonString": "Public key certificate you created above" + }' +``` + +The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own. +If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. -> * The configuration structure varies depending on how you create the Google BigQuery sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. +{% callout title="Note" type="note" %} +You can also choose to use a variety of other tools to create a connector: +- [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector ). +- [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document. + {% /callout %} ### 2. Send messages to the topic diff --git a/connectors/google-bigquery-sink/v3.0.2.1/google-bigquery-sink.md b/connectors/google-bigquery-sink/v3.0.2.1/google-bigquery-sink.md index 693d00b0d..43bb5cae1 100644 --- a/connectors/google-bigquery-sink/v3.0.2.1/google-bigquery-sink.md +++ b/connectors/google-bigquery-sink/v3.0.2.1/google-bigquery-sink.md @@ -49,28 +49,40 @@ The prerequisites for connecting an Google BigQuery sink connector to external s ### 1. Create a connector -Depending on the environment, there are several ways to create an Google BigQuery sink connector: +The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector, +you need to replace `--sink-type bigquery` with `--archive /path/to/pulsar-io-bigquery.jar`. You can find the button to download the `jar` package at the beginning of the document. -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **JAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-bigquery/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-bigquery/tags) - -No matter how you create an Google BigQuery sink connector, the minimum configuration contains the following parameters. +{% callout title="For StreamNative Cloud User" type="note" %} +If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first. +{% /callout %} -```yaml - configs: - projectId: {{Your BigQuery project id}} - datasetName: {{Your Bigquery DataSet name}} - tableName: {{The name of the table you want to write data to is automatically created by default}} - credentialJsonString: {{Public key certificate you created above}} -``` +```bash +pulsarctl sinks create \ + --sink-type bigquery \ + --name bigquery-sink \ + --tenant public \ + --namespace default \ + --inputs "Your topic name" \ + --parallelism 1 \ + --sink-config \ + '{ + "projectId": "Your BigQuery project Id", + "datasetName": "Your Bigquery DataSet name", + "tableName": "The name of the table you want to write data to is automatically created by default", + "credentialJsonString": "Public key certificate you created above" + }' +``` + +The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own. +If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. -> * The configuration structure varies depending on how you create the Google BigQuery sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. +{% callout title="Note" type="note" %} +You can also choose to use a variety of other tools to create a connector: +- [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector ). +- [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document. + {% /callout %} ### 2. Send messages to the topic diff --git a/connectors/google-bigquery-sink/v3.1.1.1/google-bigquery-sink.md b/connectors/google-bigquery-sink/v3.1.1.1/google-bigquery-sink.md index 8c45f4f55..e0fbca259 100644 --- a/connectors/google-bigquery-sink/v3.1.1.1/google-bigquery-sink.md +++ b/connectors/google-bigquery-sink/v3.1.1.1/google-bigquery-sink.md @@ -49,28 +49,40 @@ The prerequisites for connecting an Google BigQuery sink connector to external s ### 1. Create a connector -Depending on the environment, there are several ways to create an Google BigQuery sink connector: +The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector, +you need to replace `--sink-type bigquery` with `--archive /path/to/pulsar-io-bigquery.jar`. You can find the button to download the `jar` package at the beginning of the document. -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **JAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-bigquery/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-bigquery/tags) - -No matter how you create an Google BigQuery sink connector, the minimum configuration contains the following parameters. +{% callout title="For StreamNative Cloud User" type="note" %} +If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first. +{% /callout %} -```yaml - configs: - projectId: {{Your BigQuery project id}} - datasetName: {{Your Bigquery DataSet name}} - tableName: {{The name of the table you want to write data to is automatically created by default}} - credentialJsonString: {{Public key certificate you created above}} -``` +```bash +pulsarctl sinks create \ + --sink-type bigquery \ + --name bigquery-sink \ + --tenant public \ + --namespace default \ + --inputs "Your topic name" \ + --parallelism 1 \ + --sink-config \ + '{ + "projectId": "Your BigQuery project Id", + "datasetName": "Your Bigquery DataSet name", + "tableName": "The name of the table you want to write data to is automatically created by default", + "credentialJsonString": "Public key certificate you created above" + }' +``` + +The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own. +If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. -> * The configuration structure varies depending on how you create the Google BigQuery sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. +{% callout title="Note" type="note" %} +You can also choose to use a variety of other tools to create a connector: +- [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector ). +- [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document. + {% /callout %} ### 2. Send messages to the topic diff --git a/connectors/google-bigquery-source/v2.10.5.9/google-bigquery-source.md b/connectors/google-bigquery-source/v2.10.5.9/google-bigquery-source.md index 801805a6f..1917d7280 100644 --- a/connectors/google-bigquery-source/v2.10.5.9/google-bigquery-source.md +++ b/connectors/google-bigquery-source/v2.10.5.9/google-bigquery-source.md @@ -77,32 +77,41 @@ In other words, it will only synchronize the data before the start-up, and once ### 2. Create a connector -Depending on the environment, there are several ways to create an Google BigQuery source connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **JAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-bigquery/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-bigquery/tags) - -No matter how you create an Google BigQuery source connector, the minimum configuration contains the following parameters. - -```yaml - batchSourceConfig: - discoveryTriggererClassName: org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger - configs: - projectId: {{Your BigQuery project id}} - datasetName: {{Your Bigquery DataSet name}} - tableName: {{Your Bigquery Table name}} - credentialJsonString: {{Public key certificate you created above}} -``` +The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector, +you need to replace `--source-type bigquery` with `--archive /path/to/pulsar-io-bigquery.jar`. You can find the button to download the `jar` package at the beginning of the document. -> * The configuration structure varies depending on how you create the Google BigQuery source connector. - > For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. +{% callout title="For StreamNative Cloud User" type="note" %} +If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first. +{% /callout %} +```bash +pulsarctl sources create \ + --source-type bigquery \ + --name bigquery-source \ + --tenant public \ + --namespace default \ + --destination-topic-name "Your topic name" \ + --parallelism 1 \ + --batch-source-config '{"discoveryTriggererClassName": "org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger"}' \ + --source-config \ + '{ + "projectId": "Your BigQuery project Id", + "datasetName": "Your Bigquery DataSet name", + "tableName": "Your Bigquery Table name", + "credentialJsonString": "Public key certificate you created above" + }' +``` +The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own. +If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. + +{% callout title="Note" type="note" %} +You can also choose to use a variety of other tools to create a connector: +- [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector ). +- [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document. + {% /callout %} ### 3. Show data by Pulsar Consumer {% callout title="Note" type="note" %} diff --git a/connectors/google-bigquery-source/v2.11.2.13/google-bigquery-source.md b/connectors/google-bigquery-source/v2.11.2.13/google-bigquery-source.md index f0265adb0..84b72405f 100644 --- a/connectors/google-bigquery-source/v2.11.2.13/google-bigquery-source.md +++ b/connectors/google-bigquery-source/v2.11.2.13/google-bigquery-source.md @@ -77,30 +77,41 @@ In other words, it will only synchronize the data before the start-up, and once ### 2. Create a connector -Depending on the environment, there are several ways to create an Google BigQuery source connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **JAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-bigquery/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-bigquery/tags) - -No matter how you create an Google BigQuery source connector, the minimum configuration contains the following parameters. - -```yaml - batchSourceConfig: - discoveryTriggererClassName: org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger - configs: - projectId: {{Your BigQuery project id}} - datasetName: {{Your Bigquery DataSet name}} - tableName: {{Your Bigquery Table name}} - credentialJsonString: {{Public key certificate you created above}} +The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector, +you need to replace `--source-type bigquery` with `--archive /path/to/pulsar-io-bigquery.jar`. You can find the button to download the `jar` package at the beginning of the document. + +{% callout title="For StreamNative Cloud User" type="note" %} +If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first. +{% /callout %} + +```bash +pulsarctl sources create \ + --source-type bigquery \ + --name bigquery-source \ + --tenant public \ + --namespace default \ + --destination-topic-name "Your topic name" \ + --parallelism 1 \ + --batch-source-config '{"discoveryTriggererClassName": "org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger"}' \ + --source-config \ + '{ + "projectId": "Your BigQuery project Id", + "datasetName": "Your Bigquery DataSet name", + "tableName": "Your Bigquery Table name", + "credentialJsonString": "Public key certificate you created above" + }' ``` -> * The configuration structure varies depending on how you create the Google BigQuery source connector. - > For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. +The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own. +If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. + +{% callout title="Note" type="note" %} +You can also choose to use a variety of other tools to create a connector: +- [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector ). +- [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document. + {% /callout %} ### 3. Show data by Pulsar Consumer diff --git a/connectors/google-bigquery-source/v3.0.2.1/google-bigquery-source.md b/connectors/google-bigquery-source/v3.0.2.1/google-bigquery-source.md index a9c9575f4..9f9505cea 100644 --- a/connectors/google-bigquery-source/v3.0.2.1/google-bigquery-source.md +++ b/connectors/google-bigquery-source/v3.0.2.1/google-bigquery-source.md @@ -77,30 +77,41 @@ In other words, it will only synchronize the data before the start-up, and once ### 2. Create a connector -Depending on the environment, there are several ways to create an Google BigQuery source connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **JAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-bigquery/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-bigquery/tags) - -No matter how you create an Google BigQuery source connector, the minimum configuration contains the following parameters. - -```yaml - batchSourceConfig: - discoveryTriggererClassName: org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger - configs: - projectId: {{Your BigQuery project id}} - datasetName: {{Your Bigquery DataSet name}} - tableName: {{Your Bigquery Table name}} - credentialJsonString: {{Public key certificate you created above}} +The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector, +you need to replace `--source-type bigquery` with `--archive /path/to/pulsar-io-bigquery.jar`. You can find the button to download the `jar` package at the beginning of the document. + +{% callout title="For StreamNative Cloud User" type="note" %} +If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first. +{% /callout %} + +```bash +pulsarctl sources create \ + --source-type bigquery \ + --name bigquery-source \ + --tenant public \ + --namespace default \ + --destination-topic-name "Your topic name" \ + --parallelism 1 \ + --batch-source-config '{"discoveryTriggererClassName": "org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger"}' \ + --source-config \ + '{ + "projectId": "Your BigQuery project Id", + "datasetName": "Your Bigquery DataSet name", + "tableName": "Your Bigquery Table name", + "credentialJsonString": "Public key certificate you created above" + }' ``` -> * The configuration structure varies depending on how you create the Google BigQuery source connector. - > For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. +The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own. +If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. + +{% callout title="Note" type="note" %} +You can also choose to use a variety of other tools to create a connector: +- [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector ). +- [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document. + {% /callout %} ### 3. Show data by Pulsar Consumer diff --git a/connectors/google-bigquery-source/v3.1.1.1/google-bigquery-source.md b/connectors/google-bigquery-source/v3.1.1.1/google-bigquery-source.md index c490a0f0c..9369c9a7f 100644 --- a/connectors/google-bigquery-source/v3.1.1.1/google-bigquery-source.md +++ b/connectors/google-bigquery-source/v3.1.1.1/google-bigquery-source.md @@ -77,30 +77,41 @@ In other words, it will only synchronize the data before the start-up, and once ### 2. Create a connector -Depending on the environment, there are several ways to create an Google BigQuery source connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **JAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-bigquery/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-bigquery/tags) - -No matter how you create an Google BigQuery source connector, the minimum configuration contains the following parameters. - -```yaml - batchSourceConfig: - discoveryTriggererClassName: org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger - configs: - projectId: {{Your BigQuery project id}} - datasetName: {{Your Bigquery DataSet name}} - tableName: {{Your Bigquery Table name}} - credentialJsonString: {{Public key certificate you created above}} +The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector, +you need to replace `--source-type bigquery` with `--archive /path/to/pulsar-io-bigquery.jar`. You can find the button to download the `jar` package at the beginning of the document. + +{% callout title="For StreamNative Cloud User" type="note" %} +If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first. +{% /callout %} + +```bash +pulsarctl sources create \ + --source-type bigquery \ + --name bigquery-source \ + --tenant public \ + --namespace default \ + --destination-topic-name "Your topic name" \ + --parallelism 1 \ + --batch-source-config '{"discoveryTriggererClassName": "org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger"}' \ + --source-config \ + '{ + "projectId": "Your BigQuery project Id", + "datasetName": "Your Bigquery DataSet name", + "tableName": "Your Bigquery Table name", + "credentialJsonString": "Public key certificate you created above" + }' ``` -> * The configuration structure varies depending on how you create the Google BigQuery source connector. - > For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. +The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own. +If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. + +{% callout title="Note" type="note" %} +You can also choose to use a variety of other tools to create a connector: +- [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector ). +- [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector). +- [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document. + {% /callout %} ### 3. Show data by Pulsar Consumer