Skip to content

Commit

Permalink
Merge branch 'main' into lucene_snapshot
Browse files Browse the repository at this point in the history
  • Loading branch information
javanna committed Oct 1, 2024
2 parents ef5d73f + 644fd04 commit 552e935
Show file tree
Hide file tree
Showing 73 changed files with 917 additions and 465 deletions.
6 changes: 6 additions & 0 deletions docs/changelog/112258.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
pr: 112258
summary: Updated Date Range to Follow Documentation When Assuming Missing Values
area: Search
type: bug
issues:
- 111484
5 changes: 5 additions & 0 deletions docs/changelog/113588.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 113588
summary: Add asset criticality indices for `kibana_system_user`
area: Security
type: enhancement
issues: []
22 changes: 14 additions & 8 deletions docs/reference/how-to/knn-search.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,12 @@ results contains the full document `_source`. When the documents contain
high-dimensional `dense_vector` fields, the `_source` can be quite large and
expensive to load. This could significantly slow down the speed of kNN search.

NOTE: <<docs-reindex, reindex>>, <<docs-update, update>>,
and <<docs-update-by-query, update by query>> operations generally
require the `_source` field. Disabling `_source` for a field might result in
expected behavior for these operations. For example, reindex might not actually
contain the `dense_vector` field in the new index.

You can disable storing `dense_vector` fields in the `_source` through the
<<include-exclude, `excludes`>> mapping parameter. This prevents loading and
returning large vectors during search, and also cuts down on the index size.
Expand Down Expand Up @@ -102,14 +108,14 @@ merges smaller segments into larger ones through a background
explicit steps to reduce the number of index segments.

[discrete]
==== Force merge to one segment

The <<indices-forcemerge,force merge>> operation forces an index merge. If you
force merge to one segment, the kNN search only need to check a single,
all-inclusive HNSW graph. Force merging `dense_vector` fields is an expensive
operation that can take significant time to complete.

include::{es-ref-dir}/indices/forcemerge.asciidoc[tag=force-merge-read-only-warn]
==== Increase maximum segment size

{es} provides many tunable settings for controlling the merge process. One
important setting is `index.merge.policy.max_merged_segment`. This controls
the maximum size of the segments that are created during the merge process.
By increasing the value, you can reduce the number of segments in the index.
The default value is `5GB`, but that might be too small for larger dimensional vectors.
Consider increasing this value to `10GB` or `20GB` can help reduce the number of segments.

[discrete]
==== Create large segments during bulk indexing
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/inference/put-inference.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ The following services are available through the {infer} API.
You can find the available task types next to the service name.
Click the links to review the configuration details of the services:

* <<infer-service-alibabacloud-ai-search,AlibabaCloud AI Search>> (`rerank`, `sparse_embedding`, `text_embedding`)
* <<infer-service-alibabacloud-ai-search,AlibabaCloud AI Search>> (`completion`, `rerank`, `sparse_embedding`, `text_embedding`)
* <<infer-service-amazon-bedrock,Amazon Bedrock>> (`completion`, `text_embedding`)
* <<infer-service-anthropic,Anthropic>> (`completion`)
* <<infer-service-azure-ai-studio,Azure AI Studio>> (`completion`, `text_embedding`)
Expand Down
76 changes: 55 additions & 21 deletions docs/reference/inference/service-alibabacloud-ai-search.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,10 @@ include::inference-shared.asciidoc[tag=task-type]
--
Available task types:

* `text_embedding`,
* `completion`,
* `rerank`
* `sparse_embedding`,
* `rerank`.
* `text_embedding`.
--

[discrete]
Expand Down Expand Up @@ -55,31 +56,43 @@ A valid API key for the AlibabaCloud AI Search API.
The name of the model service to use for the {infer} task.
+
--
Available service_ids for the `text_embedding` task:
Available service_ids for the `completion` task:

* `ops-text-embedding-001`
* `ops-text-embedding-zh-001`
* `ops-text-embedding-en-001`
* `ops-text-embedding-002`
* `ops-qwen-turbo`
* `qwen-turbo`
* `qwen-plus`
* `qwen-max`
÷ `qwen-max-longcontext`

For the supported `text_embedding` service_ids, refer to the https://help.aliyun.com/zh/open-search/search-platform/developer-reference/text-embedding-api-details[documentation].
For the supported `completion` service_ids, refer to the https://help.aliyun.com/zh/open-search/search-platform/developer-reference/text-generation-api-details[documentation].

Available service_id for the `rerank` task is:

* `ops-bge-reranker-larger`

For the supported `rerank` service_id, refer to the https://help.aliyun.com/zh/open-search/search-platform/developer-reference/ranker-api-details[documentation].

Available service_id for the `sparse_embedding` task:

* `ops-text-sparse-embedding-001`

For the supported `sparse_embedding` service_id, refer to the https://help.aliyun.com/zh/open-search/search-platform/developer-reference/text-sparse-embedding-api-details[documentation].

Available service_id for the `rerank` task is:
Available service_ids for the `text_embedding` task:

* `ops-bge-reranker-larger`
* `ops-text-embedding-001`
* `ops-text-embedding-zh-001`
* `ops-text-embedding-en-001`
* `ops-text-embedding-002`

For the supported `text_embedding` service_ids, refer to the https://help.aliyun.com/zh/open-search/search-platform/developer-reference/text-embedding-api-details[documentation].

For the supported `rerank` service_id, refer to the https://help.aliyun.com/zh/open-search/search-platform/developer-reference/ranker-api-details[documentation].
--

`host`:::
(Required, string)
The name of the host address used for the {infer} task. You can find the host address at https://opensearch.console.aliyun.com/cn-shanghai/rag/api-key[ the API keys section] of the documentation.
The name of the host address used for the {infer} task.
You can find the host address at https://opensearch.console.aliyun.com/cn-shanghai/rag/api-key[the API keys section] of the documentation.

`workspace`:::
(Required, string)
Expand Down Expand Up @@ -130,25 +143,43 @@ If `true`, the token name will be returned in the response. Defaults to `false`
[[inference-example-alibabacloud-ai-search]]
==== AlibabaCloud AI Search service examples

The following example shows how to create an {infer} endpoint called `alibabacloud_ai_search_embeddings` to perform a `text_embedding` task type.
The following example shows how to create an {infer} endpoint called `alibabacloud_ai_search_completion` to perform a `completion` task type.

[source,console]
------------------------------------------------------------
PUT _inference/text_embedding/alibabacloud_ai_search_embeddings
PUT _inference/completion/alibabacloud_ai_search_completion
{
"service": "alibabacloud-ai-search",
"service_settings": {
"host" : "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
"api_key": "{{API_KEY}}",
"service_id": "ops-qwen-turbo",
"workspace" : "default"
}
}
------------------------------------------------------------
// TEST[skip:TBD]


The next example shows how to create an {infer} endpoint called `alibabacloud_ai_search_rerank` to perform a `rerank` task type.

[source,console]
------------------------------------------------------------
PUT _inference/rerank/alibabacloud_ai_search_rerank
{
"service": "alibabacloud-ai-search",
"service_settings": {
"api_key": "<api_key>",
"service_id": "ops-text-embedding-001",
"service_id": "ops-bge-reranker-larger",
"host": "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
"workspace": "default"
}
}
------------------------------------------------------------
// TEST[skip:TBD]

The following example shows how to create an {infer} endpoint called
`alibabacloud_ai_search_sparse` to perform a `sparse_embedding` task type.

The following example shows how to create an {infer} endpoint called `alibabacloud_ai_search_sparse` to perform a `sparse_embedding` task type.

[source,console]
------------------------------------------------------------
Expand All @@ -165,20 +196,23 @@ PUT _inference/sparse_embedding/alibabacloud_ai_search_sparse
------------------------------------------------------------
// TEST[skip:TBD]

The next example shows how to create an {infer} endpoint called
`alibabacloud_ai_search_rerank` to perform a `rerank` task type.

The following example shows how to create an {infer} endpoint called `alibabacloud_ai_search_embeddings` to perform a `text_embedding` task type.

[source,console]
------------------------------------------------------------
PUT _inference/rerank/alibabacloud_ai_search_rerank
PUT _inference/text_embedding/alibabacloud_ai_search_embeddings
{
"service": "alibabacloud-ai-search",
"service_settings": {
"api_key": "<api_key>",
"service_id": "ops-bge-reranker-larger",
"service_id": "ops-text-embedding-001",
"host": "default-j01.platform-cn-shanghai.opensearch.aliyuncs.com",
"workspace": "default"
}
}
------------------------------------------------------------
// TEST[skip:TBD]



Loading

0 comments on commit 552e935

Please sign in to comment.