Skip to content

Commit

Permalink
add documents for each connector
Browse files Browse the repository at this point in the history
  • Loading branch information
nianliuu committed Nov 7, 2024
1 parent 8aa6c9c commit 67a340c
Show file tree
Hide file tree
Showing 8 changed files with 92 additions and 2 deletions.
15 changes: 13 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,15 @@ Built on top of Apache Seatunnel, Vector-Transport-Service offers:
4. High performance, low latency, and scalability
5. Real-time monitoring and visual management

![migration.png](docs/zilliz/vts.png)
![migration.png](docs/zilliz/images/vts.png)

Additionally, Vector-Transport-Service introduces vector-specific capabilities such as multiple data source support, schema matching, and basic data validation.


## Roadmap
Future roadmaps include incremental sync, combined one-time migration and change data capture, and more advanced data transformation capabilities.

![roadmap.png](docs/zilliz/roadmap.png)
![roadmap.png](docs/zilliz/images/roadmap.png)

To learn more details about VTS used in action, read our blog:
- [**Introducing Migration Services: Efficiently Move Unstructured Data Across Platforms.**](https://zilliz.com/blog/zilliz-introduces-migration-services)
Expand Down Expand Up @@ -103,6 +103,17 @@ In addition to the quick start guide, vts has much more powerful features like

For detailed information, please refer to [Tutorial.md](./Tutorial.md)

### Connectors
VTS supports a variety of connectors to move data between different systems.

Find Detailed documentation for each connector:
- [Milvus](docs/zilliz/Milvus.md)
- [Elasitc Search](docs/zilliz/Elastic%20Search.md)
- [Pinecone](docs/zilliz/Pinecone.md)
- [Qdrant](docs/zilliz/Qdrant.md)
- [Postgres Vector](docs/zilliz/Postgres%20Vector.md)
- [Tencent VectorDB](docs/zilliz/Tencent%20VectorDB.md)

## Support
If you require any assistance or have questions regarding VTS, please feel free to reach out to our support team: Email: [email protected]

Expand Down
79 changes: 79 additions & 0 deletions docs/zilliz/Elastic Search.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Tutorial: Migrating Data from Elasticsearch to Milvus Using VTS
This guide explains how to use Vector Transport Service (VTS) to migrate data from Elasticsearch to Milvus vector database.

**Prerequisites**

- Elasticsearch instance
- Milvus instance
- JAVA 8

### Step 1: Download and Extract VTS

- Navigate to the VTS releases page: https://github.com/zilliztech/vts/releases
- Download the latest binary package
- Extract the package using:
```shell
tar -xzvf "vector-transport-service-bin-xxx.tar.gz"
```
### Step 2: Configure the Migration
Create a configuration file named es_to_milvus.conf with the following structure:
```yaml
# Environment Configuration
env {
# Number of parallel tasks
parallelism = 1
# Job execution mode
job.mode = "BATCH"
}

# Source Configuration (Elasticsearch)
source {
Elasticsearch {
# SSL/TLS Settings
tls_verify_hostname = false
tls_verify_certificate = false

# Connection Settings
hosts = ["https://your-elasticsearch-host:9200"]
username = "your-username" # Leave empty if no authentication
password = "your-password" # Leave empty if no authentication

# Data Source
index = "my_index"
}
}

# Destination Configuration (Milvus)
sink {
Milvus {
# Connection Settings
url = "https://your-milvus-instance.vectordb.zillizcloud.com:19531"
token = "your-milvus-token"

# Target Database
database = "default"

# Performance Tuning
batch_size = 10 # Adjust based on your needs
}
}
```
Configuration Notes:

Replace placeholder values (marked with your-*) with your actual credentials
Adjust batch_size based on your data volume and system resources
The parallelism value can be increased for better performance on larger datasets

### Step 3: Run the Migration
Execute the migration using the SeaTunnel shell script:
```shell
./bin/seatunnel.sh --config ./es_to_milvus.conf -m local
```

#### Monitoring and Troubleshooting

Check the console output for progress and any error messages

### Next Steps

Verify data integrity in Milvus after migration. If Need help with specific configuration options or running into issues? Feel free to ask for clarification!
Empty file added docs/zilliz/Milvus.md
Empty file.
Empty file added docs/zilliz/Pinecone.md
Empty file.
Empty file added docs/zilliz/Qdrant.md
Empty file.
Empty file added docs/zilliz/Tencent VectorDB.md
Empty file.
File renamed without changes
File renamed without changes

0 comments on commit 67a340c

Please sign in to comment.