-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
8 changed files
with
92 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,15 +19,15 @@ Built on top of Apache Seatunnel, Vector-Transport-Service offers: | |
4. High performance, low latency, and scalability | ||
5. Real-time monitoring and visual management | ||
|
||
 | ||
 | ||
|
||
Additionally, Vector-Transport-Service introduces vector-specific capabilities such as multiple data source support, schema matching, and basic data validation. | ||
|
||
|
||
## Roadmap | ||
Future roadmaps include incremental sync, combined one-time migration and change data capture, and more advanced data transformation capabilities. | ||
|
||
 | ||
 | ||
|
||
To learn more details about VTS used in action, read our blog: | ||
- [**Introducing Migration Services: Efficiently Move Unstructured Data Across Platforms.**](https://zilliz.com/blog/zilliz-introduces-migration-services) | ||
|
@@ -103,6 +103,17 @@ In addition to the quick start guide, vts has much more powerful features like | |
|
||
For detailed information, please refer to [Tutorial.md](./Tutorial.md) | ||
|
||
### Connectors | ||
VTS supports a variety of connectors to move data between different systems. | ||
|
||
Find Detailed documentation for each connector: | ||
- [Milvus](docs/zilliz/Milvus.md) | ||
- [Elasitc Search](docs/zilliz/Elastic%20Search.md) | ||
- [Pinecone](docs/zilliz/Pinecone.md) | ||
- [Qdrant](docs/zilliz/Qdrant.md) | ||
- [Postgres Vector](docs/zilliz/Postgres%20Vector.md) | ||
- [Tencent VectorDB](docs/zilliz/Tencent%20VectorDB.md) | ||
|
||
## Support | ||
If you require any assistance or have questions regarding VTS, please feel free to reach out to our support team: Email: [email protected] | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
# Tutorial: Migrating Data from Elasticsearch to Milvus Using VTS | ||
This guide explains how to use Vector Transport Service (VTS) to migrate data from Elasticsearch to Milvus vector database. | ||
|
||
**Prerequisites** | ||
|
||
- Elasticsearch instance | ||
- Milvus instance | ||
- JAVA 8 | ||
|
||
### Step 1: Download and Extract VTS | ||
|
||
- Navigate to the VTS releases page: https://github.com/zilliztech/vts/releases | ||
- Download the latest binary package | ||
- Extract the package using: | ||
```shell | ||
tar -xzvf "vector-transport-service-bin-xxx.tar.gz" | ||
``` | ||
### Step 2: Configure the Migration | ||
Create a configuration file named es_to_milvus.conf with the following structure: | ||
```yaml | ||
# Environment Configuration | ||
env { | ||
# Number of parallel tasks | ||
parallelism = 1 | ||
# Job execution mode | ||
job.mode = "BATCH" | ||
} | ||
|
||
# Source Configuration (Elasticsearch) | ||
source { | ||
Elasticsearch { | ||
# SSL/TLS Settings | ||
tls_verify_hostname = false | ||
tls_verify_certificate = false | ||
|
||
# Connection Settings | ||
hosts = ["https://your-elasticsearch-host:9200"] | ||
username = "your-username" # Leave empty if no authentication | ||
password = "your-password" # Leave empty if no authentication | ||
|
||
# Data Source | ||
index = "my_index" | ||
} | ||
} | ||
|
||
# Destination Configuration (Milvus) | ||
sink { | ||
Milvus { | ||
# Connection Settings | ||
url = "https://your-milvus-instance.vectordb.zillizcloud.com:19531" | ||
token = "your-milvus-token" | ||
|
||
# Target Database | ||
database = "default" | ||
|
||
# Performance Tuning | ||
batch_size = 10 # Adjust based on your needs | ||
} | ||
} | ||
``` | ||
Configuration Notes: | ||
|
||
Replace placeholder values (marked with your-*) with your actual credentials | ||
Adjust batch_size based on your data volume and system resources | ||
The parallelism value can be increased for better performance on larger datasets | ||
|
||
### Step 3: Run the Migration | ||
Execute the migration using the SeaTunnel shell script: | ||
```shell | ||
./bin/seatunnel.sh --config ./es_to_milvus.conf -m local | ||
``` | ||
|
||
#### Monitoring and Troubleshooting | ||
|
||
Check the console output for progress and any error messages | ||
|
||
### Next Steps | ||
|
||
Verify data integrity in Milvus after migration. If Need help with specific configuration options or running into issues? Feel free to ask for clarification! |
Empty file.
Empty file.
Empty file.
Empty file.
File renamed without changes
File renamed without changes