Skip to content

Commit

Permalink
Release new docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Milvus-doc-bot authored and Milvus-doc-bot committed Jan 16, 2025
1 parent 5d6225a commit 41e9449
Show file tree
Hide file tree
Showing 11 changed files with 406 additions and 35 deletions.
6 changes: 1 addition & 5 deletions v2.4.x/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,11 +86,7 @@ Reference to cite when you use Milvus in a research paper:

## Get in touch

Join the Milvus community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ) to share your suggestions, advice, or questions with our engineering team.

<a href="https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ">
<img src="https://zilliz-cms.s3.us-west-2.amazonaws.com/readme_slack_4a07c4c92f.png" alt="Miluvs Slack Channel" height="150" width="500">
</a>
Join the Milvus community on [Discord](https://discord.com/invite/8uyFbECzPX) to share your suggestions, advice, or questions with our engineering team.

Subscribe to Milvus's mailing lists:

Expand Down
2 changes: 1 addition & 1 deletion v2.4.x/site/en/adminGuide/monitor/alert.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ The following metrics are available for alerting configuration:
| Process Opened Fds | The current number of used file descriptors. | / |

## Set up alerts
This guide takes the example of creating an alert for the memory usage of Milvus components. To create other types of alerts, please adjust your commands accordingly. If you encounter any problems during the process, feel free to ask in the [Milvus forum](https://discuss.milvus.io/) or initiate a discussion on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
This guide takes the example of creating an alert for the memory usage of Milvus components. To create other types of alerts, please adjust your commands accordingly. If you encounter any problems during the process, feel free to ask in [Github discussions](https://github.com/milvus-io/milvus/discussions) or start a thread on [Discord](https://discord.com/invite/8uyFbECzPX).

### Prerequisites
This tutorial assumes that you have Grafana installed and configured. If not, we recommend reading the [monitoring guide](monitor.md).
Expand Down
2 changes: 1 addition & 1 deletion v2.4.x/site/en/faq/operational_faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,4 +177,4 @@ Example in Python:
You can:

- Check out [Milvus](https://github.com/milvus-io/milvus/issues) on GitHub. Feel free to ask questions, share ideas, and help others.
- Join our [Milvus Forum](https://discuss.milvus.io/) or [Slack Channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk) to find support and engage with our open-source community.
- Join our [Discord Server](https://discord.com/invite/8uyFbECzPX) to find support and engage with our open-source community.
2 changes: 1 addition & 1 deletion v2.4.x/site/en/faq/performance_faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,4 @@ However, indexing a VARCHAR field does not speed up:
You can:

- Check out [Milvus](https://github.com/milvus-io/milvus/issues) on GitHub. Feel free to ask questions, share ideas, and help others.
- Join our [Slack Channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk) to find support and engage with our open-source community.
- Join our [Discord Server](https://discord.com/invite/8uyFbECzPX) to find support and engage with our open-source community.
2 changes: 1 addition & 1 deletion v2.4.x/site/en/faq/product_faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,5 +170,5 @@ Currently, Milvus 2.4.x does not support specifying default values for scalar or
You can:

- Check out [Milvus](https://github.com/milvus-io/milvus/issues) on GitHub. You're welcome to raise questions, share ideas, and help others.
- Join our [Slack community](https://slack.milvus.io/) to find support and engage with our open-source community.
- Join our [Discord Server](https://discord.com/invite/8uyFbECzPX) to find support and engage with our open-source community.

2 changes: 1 addition & 1 deletion v2.4.x/site/en/faq/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,6 @@ This page lists common issues that may occur when running Milvus, as well as pos

If you need help solving a problem, feel free to:

- Join our [Slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk) and reach out for support from the Milvus team.
- Join our [Discord Server](https://discord.com/invite/8uyFbECzPX) and reach out for support from the Milvus team.
- [File an Issue](https://github.com/milvus-io/milvus/issues/new/choose) on GitHub that includes details about your problem.

26 changes: 13 additions & 13 deletions v2.4.x/site/en/reference/architecture/data_processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,51 +4,51 @@ summary: Learn about the data processing procedure in Milvus.
title: Data Processing
---

# Data Processing
# Data processing

This article provides a detailed description of the implementation of data insertion, index building, and data query in Milvus.
This article provides a detailed description of the implementation of data insertion, index building, and data querying in Milvus.

## Data insertion

You can specify a number of shards for each collection in Milvus, each shard corresponding to a virtual channel (*vchannel*). As the following figure shows, Milvus assigns each vchannel in the log broker a physical channel (*pchannel*). Any incoming insert/delete request is routed to shards based on the hash value of primary key.

Validation of DML requests is moved forward to proxy because Milvus does not have complicated transactions. Proxy would request a timestamp for each insert/delete request from TSO (Timestamp Oracle), which is the timing module that colocates with the root coordinator. With the older timestamp being overwritten by the newer one, timestamps are used to determine the sequence of data requests being processed. Proxy retrieves information in batches from data coord including entities' segments and primary keys to increase overall throughput and avoid overburdening the central node.
Validation of DML requests is moved forward to the proxy because Milvus does not have complicated transactions. The proxy requests a timestamp for each insert/delete request from the TSO (Timestamp Oracle), which is the timing module that colocates with the root coordinator. With the older timestamp being overwritten by the newer one, timestamps are used to determine the sequence of data requests being processed. The proxy retrieves information in batches from data coord including entities' segments and primary keys to increase overall throughput and avoid overburdening the central node.

![Channels 1](../../../../assets/channels_1.jpg "Each shard corresponds to a vchannel.")

Both DML (data manipulation language) operations and DDL (data definition language) operations are written to the log sequence, but DDL operations are only assigned one channel because of their low frequency of occurrence.

![Channels 2](../../../../assets/channels_2.jpg "Log broker nodes.")

*Vchannels* are maintained in the underlying log broker nodes. Each channel is physically indivisible and available for any but only one node. When data ingestion rate reaches bottleneck, consider two things: Whether the log broker node is overloaded and needs to be scaled, and whether there are sufficient shards to ensure load balance for each node.
*Vchannels* are maintained in the underlying log broker nodes. Each channel is physically indivisible and available for any but only one node. When data ingestion rate reaches a bottleneck, consider two things: whether the log broker node is overloaded and needs to be scaled; and, whether there are sufficient shards to ensure load balancing for each node.

![Write log sequence](../../../../assets/write_log_sequence.jpg "The process of writing log sequence.")

The above diagram encapsulates four components involved in the process of writing log sequence: proxy, log broker, data node, and object storage. The process involves four tasks: validation of DML requests, publication-subscription of log sequence, conversion from streaming log to log snapshots, and persistence of log snapshots. The four tasks are decoupled from each other to make sure each task is handled by its corresponding node type. Nodes of the same type are made equal and can be scaled elastically and independently to accommodate various data loads, massive and highly fluctuating streaming data in particular.
The above diagram encapsulates four components involved in the process of writing log sequence: the proxy, log broker, data node, and object storage. The process involves four tasks: validation of DML requests, publication-subscription of the log sequence, conversion from a streaming log to log snapshots, and persistence of log snapshots. The four tasks are decoupled from each other to make sure each task is handled by its corresponding node type. Nodes of the same type are made equal and can be scaled elastically and independently to accommodate various data loads, massive and highly fluctuating streaming data in particular.

## Index building

Index building is performed by index node. To avoid frequent index building for data updates, a collection in Milvus is divided further into segments, each with its own index.
Index building is performed by index nodes. To avoid frequent index building for data updates, a collection in Milvus is divided further into segments, each with its own index.

![Index building](../../../../assets/index_building.jpg "Index building in Milvus.")

Milvus supports building index for each vector field, scalar field and primary field. Both the input and output of index building engage with object storage: The index node loads the log snapshots to index from a segment (which is in object storage) to memory, deserializes the corresponding data and metadata to build index, serializes the index when index building completes, and writes it back to object storage.
Milvus supports building indices for each vector field, scalar field and primary field. Both the input and output of index building engage with object storage: The index node loads the log snapshots to index from a segment (which is in object storage) to memory, deserializes the corresponding data and metadata to build index, serializes the index when index building completes, and writes it back to object storage.

Index building mainly involves vector and matrix operations and hence is computation- and memory-intensive. Vectors cannot be efficiently indexed with traditional tree-based indexes due to their high-dimensional nature, but can be indexed with techniques that are more mature in this subject, such as cluster- or graph-based indexes. Regardless its type, building index involves massive iterative calculations for large-scale vectors, such as Kmeans or graph traverse.
Index building mainly involves vector and matrix operations and hence is compute- and memory-intensive. Vectors cannot be efficiently indexed with traditional tree-based indexes due to their high-dimensional nature, but can be indexed with techniques especially designed for this task, such as cluster- or graph-based indices. Regardless its type, building an index involves massive iterative calculations for large-scale vectors, such as K-means or graph traversal.

Unlike indexing for scalar data, building vector index has to take full advantage of SIMD (single instruction, multiple data) acceleration. Milvus has innate support for SIMD instruction sets, e.g., SSE, AVX2, and AVX512. Given the "hiccup" and resource-intensive nature of vector index building, elasticity becomes crucially important to Milvus in economical terms. Future Milvus releases will further explorations in heterogeneous computing and serverless computation to bring down the related costs.
Unlike indexing for scalar data, building a vector index benefits greatly from SIMD (single instruction, multiple data) acceleration. Milvus has innate support for SIMD instruction sets, e.g., SSE, AVX2, and AVX512. Given the "hiccup" and resource-intensive nature of vector index building, elasticity becomes crucially important to Milvus in economical terms. Future Milvus releases will further explorations in heterogeneous computing and serverless computation to bring down the related costs.

Besides, Milvus also supports scalar filtering and primary field query. It has inbuilt indexes to improve query efficiency, e.g., Bloom filter indexes, hash indexes, tree-based indexes, and inverted indexes, and plans to introduce more external indexes, e.g., bitmap indexes and rough indexes.
Milvus also supports scalar filtering and primary field query. It has inbuilt indices to improve query efficiency, e.g., Bloom filter indices, hash indices, tree-based indices, and inverted indices, and plans to introduce more external indexes, e.g., bitmap indexes and rough indexes.

## Data query

Data query refers to the process of searching a specified collection for *k* number of vectors nearest to a target vector or for *all* vectors within a specified distance range to the vector. Vectors are returned together with their corresponding primary key and fields.
The term "data query" refers to the process of searching a specified collection for *k* number of vectors nearest to a target vector or for *all* vectors within a specified distance range to the vector. Vectors are returned together with their corresponding primary key and fields.

![Data query](../../../../assets/data_query.jpg "Data query in Milvus.")

A collection in Milvus is split into multiple segments, and the query nodes loads indexes by segment. When a search request arrives, it is broadcast to all query nodes for a concurrent search. Each node then prunes the local segments, searches for vectors meeting the criteria, and reduces and returns the search results.
A collection in Milvus is split into multiple segments, and the query nodes load indices by segment. When a search request arrives, it is broadcast to all query nodes for a concurrent search. Each node then prunes the local segments, searches for vectors meeting the criteria, and reduces and returns the search results.

Query nodes are independent from each other in a data query. Each node is responsible only for two tasks: Load or release segments following the instructions from query coord; conduct a search within the local segments. And proxy is responsible for reducing search results from each query node and returning the final results to the client.
Query nodes are independent from each other in a data query. Each node is responsible only for two tasks: Loading or releasing segments following the instructions from query coord; conducting a search within the local segments. And the proxy is responsible for reducing search results from each query node and returning the final results to the client.

![Handoff](../../../../assets/handoff.jpg "Handoff in Milvus.")

Expand Down
18 changes: 9 additions & 9 deletions v2.4.x/site/en/reference/architecture/four_layers.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Following the principle of data plane and control plane disaggregation, Milvus c

Composed of a group of stateless proxies, the access layer is the front layer of the system and endpoint to users. It validates client requests and reduces the returned results:

- Proxy is in itself stateless. It provides a unified service address using load balancing components such as Nginx, Kubernetes Ingress, NodePort, and LVS.
- The proxy is in itself stateless. It provides a unified service address using load balancing components such as Nginx, Kubernetes Ingress, NodePort, and LVS.
- As Milvus employs a massively parallel processing (MPP) architecture, the proxy aggregates and post-process the intermediate results before returning the final results to the client.

## Coordinator service
Expand All @@ -37,37 +37,37 @@ Data coord manages topology of the data nodes and index nodes, maintains metadat

## Worker nodes

The arms and legs. Worker nodes are dumb executors that follow instructions from the coordinator service and execute data manipulation language (DML) commands from the proxy. Worker nodes are stateless thanks to separation of storage and computation, and can facilitate system scale-out and disaster recovery when deployed on Kubernetes. There are three types of worker nodes:
Worker nodes are "dumb" executors that follow instructions from the coordinator service and execute data manipulation language (DML) commands from the proxy. Worker nodes are stateless thanks to separation of storage and computation, and can facilitate system scale-out and disaster recovery when deployed on Kubernetes. There are three types of worker nodes:

### Query node

Query node retrieves incremental log data and turn them into growing segments by subscribing to the log broker, loads historical data from the object storage, and runs hybrid search between vector and scalar data.
Query nodes retrieve incremental log data and turn them into growing segments by subscribing to the log broker, load historical data from the object storage, and run hybrid search between vector and scalar data.

### Data node

Data node retrieves incremental log data by subscribing to the log broker, processes mutation requests, and packs log data into log snapshots and stores them in the object storage.
Data nodes retrieve incremental log data by subscribing to the log broker, process mutation requests, and pack log data into log snapshots and stores them in the object storage.

### Index node

Index node builds indexes. Index nodes do not need to be memory resident, and can be implemented with the serverless framework.
Index nodes builds indices. They do not need to be memory resident, and can be implemented with the serverless framework.

## Storage

Storage is the bone of the system, responsible for data persistence. It comprises meta storage, log broker, and object storage.

### Meta storage

Meta storage stores snapshots of metadata such as collection schema, and message consumption checkpoints. Storing metadata demands extremely high availability, strong consistency, and transaction support, so Milvus chose etcd for meta store. Milvus also uses etcd for service registration and health check.
Meta storage stores snapshots of metadata such as collection schema, and message consumption checkpoints. Storing metadata demands extremely high availability, strong consistency, and transaction support, so Milvus chose etcd for this purpose. Milvus also uses etcd for service registration and health checks.

### Object storage

Object storage stores snapshot files of logs, index files for scalar and vector data, and intermediate query results. Milvus uses MinIO as object storage and can be readily deployed on AWS S3 and Azure Blob, two of the world's most popular, cost-effective storage services. However, object storage has high access latency and charges by the number of queries. To improve its performance and lower the costs, Milvus plans to implement cold-hot data separation on a memory- or SSD-based cache pool.
Object storage stores snapshot files of logs, index files for scalar and vector data, and intermediate query results. Milvus uses MinIO as object storage and can be readily deployed on AWS S3 and Azure Blob, two of the world's most popular, cost-effective storage services. However, object storage has high access latency and charges by the number of queries. To improve its performance and lower costs, Milvus plans to implement cold-hot data separation on a memory- or SSD-based cache pool.

### Log broker

The log broker is a pub-sub system that supports playback. It is responsible for streaming data persistence and event notification. It also ensures integrity of the incremental data when the worker nodes recover from system breakdown. Milvus cluster uses Pulsar as log broker; Milvus standalone uses RocksDB as log broker. Besides, the log broker can be readily replaced with streaming data storage platforms such as Kafka.
The log broker is a pub-sub system that supports playback. It is responsible for streaming data persistence and event notification. It also ensures integrity of the incremental data when the worker nodes recover from system breakdown. Milvus Distributed uses Pulsar as the log broker while Milvus Standalone uses RocksDB. The log broker can be readily replaced with streaming data storage platforms such as Kafka.

Milvus is built around log broker and follows the "log as data" principle, so Milvus does not maintain a physical table but guarantees data reliability through logging persistence and snapshot logs.
Milvus follows the "log as data" principle, so Milvus does not maintain a physical table but guarantees data reliability through logging persistence and snapshot logs.

![Log_mechanism](../../../../assets/log_mechanism.png "Log mechanism.")

Expand Down
Loading

0 comments on commit 41e9449

Please sign in to comment.