Skip to content

Commit

Permalink
refactor: Use separate settings for database connection
Browse files Browse the repository at this point in the history
  • Loading branch information
@edgarrmondragon
edgarrmondragon committed Dec 30, 2024
1 parent f3b1cd4 commit f8f5f0f
Show file tree
Hide file tree
Showing 7 changed files with 125 additions and 143 deletions.
7 changes: 5 additions & 2 deletions .github/workflows/test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
- uses: actions/checkout@v4
- run: |
docker compose up --quiet-pull --wait
docker compose exec starrocks mysql -P 9030 -h 127.0.0.1 -u root -e 'create database if not exists ${{ env.STARROCKS_DATABASE }}'
docker compose exec starrocks mysql -P ${{ env.STARROCKS_PORT }} -h 127.0.0.1 -u root -e 'create database if not exists ${{ env.STARROCKS_DATABASE }}'
docker compose ps
- uses: actions/setup-python@v5
with:
Expand All @@ -59,6 +59,9 @@ jobs:
version: ">=0.5.0"
- name: Run Tox
env:
TARGET_STARROCKS_SQLALCHEMY_URL: "starrocks://root:@localhost:${{ env.STARROCKS_PORT }}/${{ env.STARROCKS_DATABASE }}"
TARGET_STARROCKS_USER: root
TARGET_STARROCKS_HOST: localhost
TARGET_STARROCKS_PORT: ${{ env.STARROCKS_PORT }}
TARGET_STARROCKS_DATABASE: ${{ env.STARROCKS_DATABASE }}
run: |
uvx --with tox-uv tox -e $(echo py${{ matrix.python-version }} | tr -d .)
69 changes: 47 additions & 22 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# target-starrocks

`target-starrocks` is a Singer target for Starrocks.
`target-starrocks` is a Singer target for [Starrocks].

Build with the [Meltano Target SDK](https://sdk.meltano.com).
Build with the [Meltano Target SDK][Singer SDK].

<!--
Expand All @@ -24,26 +24,48 @@ pipx install git+https://github.com/ORG_NAME/target-starrocks.git@main
-->

## Configuration

### Accepted Config Options
## Supported Python Versions

<!--
Developer TODO: Provide a list of config options accepted by the target.
* 3.9
* 3.10
* 3.11
* 3.12
* 3.13

This section can be created by copy-pasting the CLI output from:
```
target-starrocks --about --format=markdown
```
-->
## Configuration

A full list of supported settings and capabilities for this
target is available by running:
### Accepted Config Options

```bash
target-starrocks --about
```
| Setting | Required | Default | Description |
| :-------------------------------- | :------- | :---------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user | False | None | User for the Starrocks database |
| password | False | None | Password for the Starrocks database |
| host | False | None | Host for the Starrocks database |
| port | False | 9030 | Port for the Starrocks database |
| database | True | None | Database name for the Starrocks database |

#### Additional Config Options

The following built-in configuration options are also supported:

| Setting | Required | Default | Description |
| :-------------------------------- | :------- | :---------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| default_target_schema | False | None | The default target database schema name to use for all streams. |
| hard_delete | False | 0 | Hard delete records. |
| add_record_metadata | False | None | Whether to add metadata fields to records. |
| load_method | False | TargetLoadMethods.APPEND_ONLY | The method to use when loading data into the destination. `append-only` will always write all input records whether that records already exists or not. `upsert` will update existing records and insert new records. `overwrite` will delete all existing records and insert all input records. |
| batch_size_rows | False | None | Maximum number of rows in each batch. |
| process_activate_version_messages | False | 1 | Whether to process `ACTIVATE_VERSION` messages. |
| validate_records | False | 1 | Whether to validate the schema of the incoming streams. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| faker_config | False | None | Config for the [`Faker`](https://faker.readthedocs.io/en/master/) instance variable `fake` used within map expressions. Only applicable if the plugin specifies `faker` as an addtional dependency (through the `singer-sdk` `faker` extra or directly). |
| faker_config.seed | False | None | Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator |
| faker_config.locale | False | None | One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |

A full list of supported settings and capabilities is available by running: `target-starrocks --about`

### Configure using environment variables

Expand All @@ -66,8 +88,8 @@ You can easily run `target-starrocks` by itself or in a pipeline using [Meltano]
```bash
target-starrocks --version
target-starrocks --help
# Test using the "Carbon Intensity" sample:
tap-carbon-intensity | target-starrocks --config /path/to/target-starrocks-config.json
# Test using the "Smoke Test" tap:
tap-smoke-test | target-starrocks --config /path/to/target-starrocks-config.json
```

## Developer Resources
Expand Down Expand Up @@ -122,11 +144,14 @@ Now you can test and orchestrate using Meltano:
```bash
# Test invocation:
meltano invoke target-starrocks --version
# OR run a test `elt` pipeline with the Carbon Intensity sample tap:
meltano run tap-carbon-intensity target-starrocks
# OR run a test `elt` pipeline with the Smoke Test sample tap:
meltano run tap-smoke-test target-starrocks
```

### SDK Dev Guide

See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the Meltano Singer SDK to
develop your own Singer taps and targets.

[Starrocks]: https://starrocks.io
[Singer SDK]: https://sdk.meltano.com
36 changes: 31 additions & 5 deletions meltano.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@ plugins:
config:
streams:
- stream_name: animals
input_filename:
https://raw.githubusercontent.com/meltano/tap-smoke-test/main/demo-data/animals-data.jsonl
input_filename: https://raw.githubusercontent.com/meltano/tap-smoke-test/main/demo-data/animals-data.jsonl
metadata:
animals:
key-properties: [id]

- name: tap-fedidb
variant: edgarrmondragon
Expand All @@ -27,12 +29,36 @@ plugins:
- stream-maps
- schema-flattening

settings_group_validation:
- [sqlalchemy_url]
settings:
- name: sqlalchemy_url
- name: user
kind: string
label: User
description: User for the Starrocks database

- name: password
kind: string
label: Password
sensitive: true
description: Password for the Starrocks database

- name: host
kind: string
label: Host
description: Host for the Starrocks database

- name: port
kind: integer
label: Port
value: 9030
description: Port for the Starrocks database

- name: database
kind: string
label: Database
description: Database for the Starrocks database

settings_group_validation:
- [database]

venv:
backend: uv
105 changes: 2 additions & 103 deletions poetry.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 1 addition & 6 deletions pyproject.toml
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[tool.poetry]
name = "edgarrmondragon-target-starrocks"
name = "target-starrocks"
version = "0.0.1"
description = "Singer target for Starrocks, built with the Meltano Singer SDK."
readme = "README.md"
Expand All @@ -18,13 +18,9 @@ classifiers = [
"Programming Language :: Python :: 3.13",
]
license = "Apache-2.0"
packages = [
{ include = "target_starrocks" },
]

[tool.poetry.dependencies]
python = ">=3.9"
fs-s3fs = { version = "~=1.1.1", optional = true }
singer-sdk = { version="~=0.43.1", extras = ["faker"] }
starrocks = "~=1.2.0"

Expand Down Expand Up @@ -66,5 +62,4 @@ requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
# CLI declaration
target-starrocks = 'target_starrocks.target:TargetStarrocks.cli'
10 changes: 9 additions & 1 deletion target_starrocks/sinks.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,15 @@ def get_sqlalchemy_url(self, config: dict) -> str:
Args:
config: The configuration for the connector.
"""
return super().get_sqlalchemy_url(config)
return sa.URL(
drivername="starrocks",
username=config.get("user"),
password=config.get("password"),
host=config.get("host"),
port=config.get("port"),
database=config["database"],
query={},
).render_as_string(hide_password=False)


class StarrocksSink(SQLSink):
Expand Down
Loading

0 comments on commit f8f5f0f

Please sign in to comment.