Skip to content

Commit

Permalink
Merge pull request #243 from george0st/change
Browse files Browse the repository at this point in the history
Extend kafka documentation
  • Loading branch information
george0st authored Mar 30, 2024
2 parents 6d0dba8 + 27b9fb8 commit 5ee17ab
Show file tree
Hide file tree
Showing 4 changed files with 23 additions and 3 deletions.
20 changes: 20 additions & 0 deletions docs/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,33 @@
- interactive access to the container
- `docker exec -it kafka1 /bin/bash`
- get kafka version `kafka-topics --version`
- list kafka topics
- `kafka-topics --bootstrap-server localhost:9092 --list`
- or `docker exec -t kafka1 /usr/bin/kafka-topics --bootstrap-server localhost:9092 --list`
- create (define) new kafka topic 'aa'
- `kafka-topics --bootstrap-server localhost:9092 --topic aa --create --partitions 3 --replication-factor 1`
- or `docker exec -t kafka1 /usr/bin/kafka-topics --bootstrap-server localhost:9092 --topic aa --create --partitions 3 --replication-factor 1`
- fire (produce) new kafka topic 'aa' (CTRL+C for finish new topic)
- `kafka-console-producer --bootstrap-server localhost:9092 --topic aa`
- or `docker exec -t kafka1 /usr/bin/kafka-console-producer --bootstrap-server localhost:9092 --topic aa`
- get (consume) kafka topics 'aa'
- `kafka-console-consumer --bootstrap-server localhost:9092 --topic aa --from-beginning`
- or `docker exec -t kafka1 /usr/bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic aa --from-beginning`


## 3. Use Kafka for tests

- Update `qgate-sln-mlrun.env`, change setting for `QGATE_KAFKA`
- format `QGATE_KAFKA = <url>, <topic name>`
- see `QGATE_KAFKA = localhost:9092, testtopic`

## 4. Install these python packages

- SQLAlchemy based on MLRun extras see `pip install mlrun[kafka]` or [dependencies.py in MLRun](https://github.com/mlrun/mlrun/blob/development/dependencies.py)
- it required `pip install kafka-python~=2.0`
- it required `pip install avro~=1.11`
- NOTE
- these are versions valid for MLRun 1.6.2

## Useful sources for kafka

Expand Down
2 changes: 1 addition & 1 deletion docs/mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,5 +51,5 @@
- Cryptography support
- it required `pip install cryptography~=42.0`
- NOTE
- these are versions valid for MLRun 1.6.1
- these are versions valid for MLRun 1.6.2

2 changes: 1 addition & 1 deletion docs/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,4 @@
- Cryptography support
- it required `pip install cryptography~=42.0`
- NOTE
- these are versions valid for MLRun 1.6.1
- these are versions valid for MLRun 1.6.2
2 changes: 1 addition & 1 deletion qgate-sln-mlrun-private.env
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ QGATE_DEFINITION = ../qgate-model
QGATE_DATASET = 01-size-100

# List of projects for testing e.g. agate-1, agate-2, etc. Default is empty list (all projects will be tested)
QGATE_FILTER_PROJECTS = agate-2
QGATE_FILTER_PROJECTS = agate-1

# Path to the output directory (as off-line storage, valid for target 'parquet' and 'csv')
# sample value e.g. ./output
Expand Down

0 comments on commit 5ee17ab

Please sign in to comment.