Implement Command Manager settings on opensearch.yml #213

QU3B1M · 2025-01-13T10:50:58Z

Description

Implement PluginSettings class to load from the opensearch.yml file all the configuration used across the Command Manager plugin.

Note

Configurations related to index/templates names and API URI were kept as constants rather than settings preventing runtime changes, which could lead to inconsistencies within plugin components and external interactions.

The resulting configuration YAML is structured as follows:

command_manager:
  client:
    timeout: int            # Default: 20
  job:
    schedule: int         # Default: 1
    page_size: int        # Default: 100
    pit_keep_alive: int   # Default: 30

Note: All fields have default values, making none of the settings mandatory.

Proposed documentation

Command Manager Plugin Configuration

The Command Manager plugin allows you to configure various settings through the opensearch.yml file. Below is an overview of the available configuration options and their default values.

Configuration Structure

The configuration settings for the Command Manager plugin are structured in the following hierarchy within the opensearch.yml file:

command_manager:
  timeout: int
  job:
    schedule: int
    page_size: int
    pit_keep_alive: int
    index:
      template: str
  api:
    prefix: str
    endpoint: str
  index:
    name: str
    template: str

Configuration Parameters

Timeout
- Key: command_manager.timeout
- Type: Integer
- Default: 20
- Description: Specifies the API consulting timeout value in seconds.
Job Schedule
- Key: command_manager.job.schedule
- Type: Integer
- Default: 1
- Description: Defines the schedule interval in minutes for job execution.
Job Page Size
- Key: command_manager.job.page_size
- Type: Integer
- Default: 100
- Description: Sets the number of records to process per page during job execution.
Job PIT Keep Alive
- Key: command_manager.job.pit_keep_alive
- Type: Integer
- Default: 30
- Description: Determines the keep-alive time in seconds for the Point-In-Time (PIT) context.
Job Index Template
- Key: command_manager.job.index.template
- Type: String
- Default: index-template-scheduled-commands
- Description: Specifies the template name for the job scheduled commands index.
API Prefix
- Key: command_manager.api.prefix
- Type: String
- Default: /_command_manager
- Description: Sets the prefix for the command manager API endpoints.
API Endpoint
- Key: command_manager.api.endpoint
- Type: String
- Default: /commands
- Description: Defines the endpoint for command manager APIs.
Index Name
- Key: command_manager.index.name
- Type: String
- Default: .commands
- Description: Specifies the name of the index used for storing commands.
Index Template
- Key: command_manager.index.template
- Type: String
- Default: index-template-commands
- Description: Defines the template name for the commands index.

Notes

All configuration fields have default values and are not mandatory.
Adjust these settings according to your needs by modifying the opensearch.yml file.

Issues Resolved

Closes #180

Add logs to track settings values

Set Job index as constant inside main CommandManager plugin class

Replace constants usage with getters

f-galland · 2025-01-14T12:53:36Z

Testing changing the configuration options:

Test config:

root@config-test:~# tail -13 /etc/wazuh-indexer/opensearch.yml 
command_manager:
  timeout: 30
  job:
    schedule: 1
    page_size: 2
    pit_keep_alive: 30
    index.template: "index-template-scheduled-commands"
  api:
    prefix: "/_test_prefix"
    endpoint: "/test_endpoint"
  index:
    name: ".test-index"
    template: "index-template-commands"

Writing commands works:

root@config-test:~# curl -sku admin:admin https://localhost:9200/_plugins/_test_prefix/test_endpoint -H 'Content-Type: application/json' -d "$(cat command.json)"
{"_index":".test-index","_documents":[{"_id":"lRffZJQBm0txxyk-F3on"},{"_id":"lhffZJQBm0txxyk-F3on"},{"_id":"lxffZJQBm0txxyk-F3on"}],"result":"OK"}

The new index is created and used:

root@config-test:~# curl -sku admin:admin https://localhost:9200/_cat/indices/.test-index?v
health status index       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .test-index 9KCAY4yeRI27gDEmsfnYmQ   1   1          6            3     57.7kb         57.7kb

f-galland · 2025-01-14T14:03:02Z

I was trying to test the integer variables when I noticed the CPU usage spiked.
This error log is repeated in a loop:

[2025-01-14T14:01:20,670][WARN ][c.w.c.j.SearchThread     ] [node-1] Empty hits page, not getting searchAfter values

AlexRuiz7

Commands are not being indexed:

[2025-01-14T16:35:39,730][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [2] from host [127.0.0.1:9200]
[2025-01-14T16:35:39,733][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [LFR0ZZQB9G2sMpT7xx3U] to the bulk request
[2025-01-14T16:35:39,735][ERROR][c.w.c.i.CommandIndex     ] [integTest-0] Error indexing commands with bulk due to Cannot invoke "java.util.Map.size()" because "source" is null

On my testing, I added this to the build.gradle file of the command manager plugin.

testClusters.integTest {
    setting "command_manager.index.name", "custom-index-name"
    setting "command_manager.job.index.template", "pepe"
    setting "command_manager.index.template", "wololo"
...
}

More importantly, during testing, I've noticed severe issues about the configuration being exposed.

The index names cannot be changed for several reasons:
- Custom index names may not match the index pattern defined in their companion index template.
- For the commands stream index, this index is accessed by several external actors, as for now, the Management API, but more will come.
The endpoint URL to receive commands cannot be changed, as it is accessed by several external actors, as for now, the Management API, but more will come (Content Manager).

On another note, the timeout setting could use a longer but more comprehensive name, for example client.timeout or opensearch.client.timeout.

…to constants

AlexRuiz7

Note

Update in #213 (comment)

I have detected a weird behavior when changing the job's search page size. It looks like in enters an infinite loop.

[2025-01-16T14:09:55,188][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=10}
[2025-01-16T14:09:56,158][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [2] from host [127.0.0.1:9200]
[2025-01-16T14:09:56,159][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [83Y8b5QB45_abGJWFfI_] to the bulk request
[2025-01-16T14:09:56,159][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [9HY8b5QB45_abGJWFfI_] to the bulk request
[2025-01-16T14:09:56,160][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
[2025-01-16T14:09:57,214][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [3] from host [127.0.0.1:9200]
[2025-01-16T14:09:57,215][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [-XY8b5QB45_abGJWGfJe] to the bulk request
[2025-01-16T14:09:57,215][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [-nY8b5QB45_abGJWGfJe] to the bulk request
[2025-01-16T14:09:57,215][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
[2025-01-16T14:09:57,995][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [4] from host [127.0.0.1:9200]
[2025-01-16T14:09:57,996][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [_3Y8b5QB45_abGJWHPJs] to the bulk request
[2025-01-16T14:09:57,996][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [AHY8b5QB45_abGJWHPNs] to the bulk request
[2025-01-16T14:09:57,997][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
[2025-01-16T14:09:58,689][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [5] from host [127.0.0.1:9200]
[2025-01-16T14:09:58,690][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [BXY8b5QB45_abGJWH_Mi] to the bulk request
[2025-01-16T14:09:58,690][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [BnY8b5QB45_abGJWH_Mi] to the bulk request
[2025-01-16T14:09:58,690][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
[2025-01-16T14:09:59,609][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [6] from host [127.0.0.1:9200]
[2025-01-16T14:09:59,610][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [C3Y8b5QB45_abGJWIvO6] to the bulk request
[2025-01-16T14:09:59,610][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [DHY8b5QB45_abGJWIvO6] to the bulk request
[2025-01-16T14:09:59,610][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
[2025-01-16T14:10:00,406][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [7] from host [127.0.0.1:9200]
[2025-01-16T14:10:00,407][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [EXY8b5QB45_abGJWJfPX] to the bulk request
[2025-01-16T14:10:00,407][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [EnY8b5QB45_abGJWJfPX] to the bulk request
[2025-01-16T14:10:00,407][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
[2025-01-16T14:10:42,423][INFO ][c.w.c.j.SearchThread     ] [integTest-0] Query returned 14 hits.
[2025-01-16T14:10:42,478][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,479][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,480][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,482][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,483][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,484][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,485][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,486][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,487][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,489][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,491][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,492][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,493][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,494][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,495][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,497][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,498][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,499][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,501][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,502][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,503][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,504][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,506][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,507][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,508][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,510][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,511][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,513][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,515][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,516][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values
[2025-01-16T14:10:42,518][WARN ][c.w.c.j.SearchThread     ] [integTest-0] Empty hits page, not getting searchAfter values

To reproduce, add a custom setting in build.gradle. I used

setting 'command_manager.job.page_size', "1"

Then, send some commands to the API.

This does not happen when using the default value.

2025-01-16T14:06:44.290+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :run
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,666][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [1] from host [127.0.0.1:9200]
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,672][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [Q5A5b5QBzFOclsIFuc7G] to the bulk request
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,672][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [RJA5b5QBzFOclsIFuc7I] to the bulk request
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,678][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[SndWjcxMRPSqwNy_Mv76iQ/13sOWSCjSgScRXx2Jq1-Kg]
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,691][INFO ][o.o.c.m.MetadataIndexTemplateService] [integTest-0] adding template [index-template-commands] for index patterns [.commands*]
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,692][DEBUG][o.o.c.c.Coordinator      ] [integTest-0] initialized PublicationContext using class: class org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,692][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] sending cluster state over transport to node: integTest-0
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,693][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [6] with uuid [VonHqC2YQDuNTXtqeYZ2ZQ], diff size [612]
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,724][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=6}
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,724][INFO ][c.w.c.u.IndexTemplateUtils] [integTest-0] Index template [index-template-commands] created successfully
2025-01-16T14:07:21.730+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,729][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[.commands/k-JC7dBjR3SiyPN5THY95Q]
2025-01-16T14:07:21.830+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,742][INFO ][o.o.c.m.MetadataCreateIndexService] [integTest-0] [.commands] creating index, cause [auto(bulk api)], templates [index-template-commands], shards [1]/[0]
2025-01-16T14:07:21.831+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,743][WARN ][o.o.c.r.a.AllocationService] [integTest-0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-01-16T14:07:21.831+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,746][DEBUG][o.o.c.c.Coordinator      ] [integTest-0] initialized PublicationContext using class: class org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext
2025-01-16T14:07:21.831+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,747][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] sending cluster state over transport to node: integTest-0
2025-01-16T14:07:21.831+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,748][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [7] with uuid [WzbB-T1_SPK9XQW35V_xVg], diff size [780]
2025-01-16T14:07:21.831+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,788][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[.commands/k-JC7dBjR3SiyPN5THY95Q]
2025-01-16T14:07:21.831+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,814][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=7}
2025-01-16T14:07:21.931+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,879][INFO ][o.o.c.r.a.AllocationService] [integTest-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.commands][0]]]).
2025-01-16T14:07:21.931+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,879][DEBUG][o.o.c.c.Coordinator      ] [integTest-0] initialized PublicationContext using class: class org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext
2025-01-16T14:07:21.931+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,880][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] sending cluster state over transport to node: integTest-0
2025-01-16T14:07:21.931+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,880][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [8] with uuid [c5IeMq_-Riu_Rg-XjqhabQ], diff size [437]
2025-01-16T14:07:21.931+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,915][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=8}
2025-01-16T14:07:21.931+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,916][WARN ][o.o.c.r.a.AllocationService] [integTest-0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-01-16T14:07:21.931+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,930][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[.commands/k-JC7dBjR3SiyPN5THY95Q]
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,939][INFO ][o.o.c.m.MetadataMappingService] [integTest-0] [.commands/k-JC7dBjR3SiyPN5THY95Q] update_mapping [_doc]
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,940][DEBUG][o.o.c.c.Coordinator      ] [integTest-0] initialized PublicationContext using class: class org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,940][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] sending cluster state over transport to node: integTest-0
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,941][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [9] with uuid [uJ5OYnsgT4mh9bQlJ1fJMw], diff size [764]
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,980][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=9}
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:21,989][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[.commands/k-JC7dBjR3SiyPN5THY95Q]
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,005][INFO ][o.o.c.m.MetadataMappingService] [integTest-0] [.commands/k-JC7dBjR3SiyPN5THY95Q] update_mapping [_doc]
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,007][DEBUG][o.o.c.c.Coordinator      ] [integTest-0] initialized PublicationContext using class: class org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,007][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] sending cluster state over transport to node: integTest-0
2025-01-16T14:07:22.031+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,008][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [10] with uuid [ToCzOhHSTwqXMszmIk1tTA], diff size [794]
2025-01-16T14:07:22.132+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,054][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=10}
2025-01-16T14:07:22.732+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,674][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [2] from host [127.0.0.1:9200]
2025-01-16T14:07:22.732+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,675][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [SZA5b5QBzFOclsIFvc6z] to the bulk request
2025-01-16T14:07:22.732+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,675][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [SpA5b5QBzFOclsIFvc6z] to the bulk request
2025-01-16T14:07:22.732+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:22,676][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
2025-01-16T14:07:23.433+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:23,411][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [3] from host [127.0.0.1:9200]
2025-01-16T14:07:23.433+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:23,412][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [T5A5b5QBzFOclsIFwM6U] to the bulk request
2025-01-16T14:07:23.433+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:23,412][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [UJA5b5QBzFOclsIFwM6U] to the bulk request
2025-01-16T14:07:23.433+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:23,413][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
2025-01-16T14:07:24.034+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:24,010][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [4] from host [127.0.0.1:9200]
2025-01-16T14:07:24.034+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:24,011][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [VZA5b5QBzFOclsIFws7q] to the bulk request
2025-01-16T14:07:24.034+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:24,011][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [VpA5b5QBzFOclsIFws7q] to the bulk request
2025-01-16T14:07:24.034+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:24,011][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
2025-01-16T14:07:24.636+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:24,563][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [5] from host [127.0.0.1:9200]
2025-01-16T14:07:24.636+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:24,564][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [W5A5b5QBzFOclsIFxc4U] to the bulk request
2025-01-16T14:07:24.636+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:24,564][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [XJA5b5QBzFOclsIFxc4U] to the bulk request
2025-01-16T14:07:24.636+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:24,565][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
2025-01-16T14:07:25.236+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:25,178][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [6] from host [127.0.0.1:9200]
2025-01-16T14:07:25.237+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:25,179][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [YZA5b5QBzFOclsIFx857] to the bulk request
2025-01-16T14:07:25.237+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:25,179][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [YpA5b5QBzFOclsIFx857] to the bulk request
2025-01-16T14:07:25.237+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:25,179][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
2025-01-16T14:07:26.038+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:25,958][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [7] from host [127.0.0.1:9200]
2025-01-16T14:07:26.038+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:25,959][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [Z5A5b5QBzFOclsIFys6H] to the bulk request
2025-01-16T14:07:26.038+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:25,959][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [aJA5b5QBzFOclsIFys6H] to the bulk request
2025-01-16T14:07:26.038+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:25,959][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Index template index-template-commands already exists. Skipping creation.
2025-01-16T14:07:31.226+0100 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager] 
2025-01-16T14:07:31.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:07:31.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:07:31.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:07:31.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:07:31.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:07:31.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:07:37.506+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Daemon worker Thread 4: acquired lock on worker lease
2025-01-16T14:07:37.506+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Daemon worker Thread 4: released lock on worker lease
2025-01-16T14:07:37.506+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 7: acquired lock on worker lease
2025-01-16T14:07:37.506+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 7: released lock on worker lease
2025-01-16T14:07:37.506+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 6: acquired lock on worker lease
2025-01-16T14:07:37.506+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 6: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 11: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 11: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 12: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 12: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 13: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 13: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 14: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 14: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 4: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 4: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 10: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 10: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 9: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 9: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 8: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 8: released lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 15: acquired lock on worker lease
2025-01-16T14:07:37.507+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 15: released lock on worker lease
2025-01-16T14:07:37.508+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 2: acquired lock on worker lease
2025-01-16T14:07:37.508+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 2: released lock on worker lease
2025-01-16T14:07:37.508+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] included builds: acquired lock on worker lease
2025-01-16T14:07:37.508+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] included builds: released lock on worker lease
2025-01-16T14:07:37.508+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 3: acquired lock on worker lease
2025-01-16T14:07:37.508+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 3: released lock on worker lease
2025-01-16T14:07:37.508+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 5: acquired lock on worker lease
2025-01-16T14:07:37.508+0100 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 5: released lock on worker lease
2025-01-16T14:07:41.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:07:41.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:07:41.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:07:41.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:07:41.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:07:41.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:07:25.989+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] 
2025-01-16T14:07:25.989+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :run
2025-01-16T14:07:44.259+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:07:44,233][INFO ][c.w.c.j.SearchThread     ] [integTest-0] Query returned 14 hits.
2025-01-16T14:07:51.226+0100 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager] 
2025-01-16T14:07:51.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:07:51.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:07:51.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:07:51.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:07:51.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:07:51.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:08:01.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:08:01.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:08:01.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:08:01.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:08:01.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:08:01.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.

AlexRuiz7 · 2025-01-16T13:27:25Z

setting 'command_manager.client.timeout', "5" works without raising the issue above.

2025-01-16T14:19:11.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:18:18.748+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] 
2025-01-16T14:18:18.748+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :run
2025-01-16T14:19:14.442+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:19:14,438][INFO ][c.w.c.j.SearchThread     ] [integTest-0] Query returned 18 hits.
2025-01-16T14:19:21.226+0100 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager] 
2025-01-16T14:19:21.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:19:21.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:19:21.226+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
2025-01-16T14:19:21.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
2025-01-16T14:19:21.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
2025-01-16T14:19:21.227+0100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.

setting 'command_manager.client.timeout', "-1" is allowed, and raises a TimeoutException, as expected. However, this kind of exception is handled as a generic exception. We need to improve this.

2025-01-16T14:25:52.927+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:25:52,876][ERROR][c.w.c.j.SearchThread     ] [integTest-0] Generic exception retrieving page: java.util.concurrent.TimeoutException: Timeout waiting for task.

AlexRuiz7 · 2025-01-16T13:35:34Z

Using setting 'command_manager.job.pit_keep_alive', "1" and setting 'command_manager.job.pit_keep_alive', "0" works

2025-01-16T14:33:18.492+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T14:33:18,466][INFO ][c.w.c.j.SearchThread     ] [integTest-0] Query returned 12 hits.

However, if we set a negative value, the search job is never executed.
setting 'command_manager.job.pit_keep_alive', "-10"

AlexRuiz7 · 2025-01-16T15:44:18Z

Using setting 'command_manager.job.schedule', "-10" makes the command manager enters another infinite loop.

[2025-01-16T16:37:15,614][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=4}
[2025-01-16T16:37:16,120][INFO ][o.o.c.r.a.AllocationService] [integTest-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.scheduled-commands][0]]]).
[2025-01-16T16:37:16,122][DEBUG][o.o.c.c.Coordinator      ] [integTest-0] initialized PublicationContext using class: class org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext
[2025-01-16T16:37:16,122][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] sending cluster state over transport to node: integTest-0
[2025-01-16T16:37:16,124][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [5] with uuid [UYD4LyHlSz-BRwZqOZGdkw], diff size [395]
[2025-01-16T16:37:16,335][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=5}
[2025-01-16T16:37:16,337][WARN ][o.o.c.r.a.AllocationService] [integTest-0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-01-16T16:37:16,357][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] Scheduling job id ReLCb5QBuFuLdquy89cj for index .scheduled-commands .
[2025-01-16T16:37:16,358][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836358, setting next execute time to current
[2025-01-16T16:37:16,359][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836359, setting next execute time to current
[2025-01-16T16:37:16,360][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,360][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836360, setting next execute time to current
[2025-01-16T16:37:16,360][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,360][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836360, setting next execute time to current
[2025-01-16T16:37:16,361][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,361][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836361, setting next execute time to current
[2025-01-16T16:37:16,361][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,361][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836361, setting next execute time to current
[2025-01-16T16:37:16,362][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,362][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836362, setting next execute time to current
[2025-01-16T16:37:16,362][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,362][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836362, setting next execute time to current
[2025-01-16T16:37:16,363][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,363][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836363, setting next execute time to current
[2025-01-16T16:37:16,363][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,363][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836363, setting next execute time to current
[2025-01-16T16:37:16,364][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,364][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836364, setting next execute time to current
[2025-01-16T16:37:16,364][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs
[2025-01-16T16:37:16,364][INFO ][o.o.j.s.JobScheduler     ] [integTest-0] job command_manager_scheduler_extension expected time: 1737041234788 < current time: 1737041836364, setting next execute time to current
[2025-01-16T16:37:16,364][INFO ][c.w.c.j.CommandManagerJobRunner] [integTest-0] .commands index not yet created, not running command manager jobs

Using another positive value works as expected. However, during this test, I noticed that the query returns more results than the default page length (100).

2025-01-16T16:41:25.416+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] 
2025-01-16T16:41:25.416+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :run
2025-01-16T16:42:06.218+0100 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2025-01-16T16:42:06,118][INFO ][c.w.c.j.SearchThread     ] [integTest-0] Query returned 102 hits.
2025-01-16T16:42:06.360+0100 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager]

Remove plugin-scurity.policy file as it is no longer needed

AlexRuiz7 · 2025-01-16T16:01:32Z

Looks like the CM's search thread enter an infinite loop whenever the number of commands is larger than the page size (pagination seems to be broken).

In order to test this behavior, I indexed 120 commands through the CM's API. Default page length is 100, so I'm indexing more commands that what it can handle in a single page.

As a result, the CM enters the infinite loop.

I'm comparing this behavior on master.

Update: I was able to reproduce it on master. I'll open an issue to fix that.

AlexRuiz7

Summary of issues detected

[BUG] Infinite loop during the pagination of search results #226
[BUG] Unhandled TimeoutException #227
[BUG] The Command Manager accepts invalid settings #222
- [BUG] Infinite loop when using a negative value for job.schedule #223
- [BUG] Search job is not executed when using a negative value for job.pit_keep_alive #224
[Feature request] Add logging when custom settings are used #225

* Set min and max values for settings and validate their coherence * Roll back to Setting.Property.NodeScope to fix issues * Make pit keepalive equal to job schedule in seconds * Make pit keepalive equal to job schedule in seconds * Fix tests * Refactor validateSettings() --------- Signed-off-by: Fede Galland <[email protected]> Co-authored-by: Alex Ruiz <[email protected]>

Signed-off-by: Álex Ruiz <[email protected]>

f-galland

LGTM!

Implement plugin settings class to handle command manager configuration

d2082cf

QU3B1M self-assigned this Jan 13, 2025

QU3B1M added 11 commits January 13, 2025 08:18

Initialize settings on main Command Manager class

a18a76c

Add logs to track settings values

Make PluginSetting singleton thread-safe adding syncrhonization

f9fbfea

Replace constants usage with PluginSettings corresponding getters

dc8ddc9

Register settings in CommandManager main class

c94c182

Add logs to validate settings registering

747d3ee

Implement RealoadablePlugin interface to CommandManager plugin

d959c8d

Apply spotless formatting

d4c14d4

Set Job index as constant inside main CommandManager plugin class

Replace setting job.index.name for a constant

b06a775

Apply getter functions for job index name and job type

0136b8a

Replace constants usage with getters

Add unit tests for PluginSettings class

1947e26

Add docstrings to PluginSettings functions

6cb00b7

QU3B1M marked this pull request as ready for review January 14, 2025 11:01

QU3B1M requested a review from a team as a code owner January 14, 2025 11:01

Merge branch 'master' into enhancement/180-command-manager-configuration

6bcb6b9

QU3B1M mentioned this pull request Jan 14, 2025

Command Manager configuration file #180

Closed

7 tasks

Remove unnecesary debug logs

7635e26

AlexRuiz7 requested changes Jan 14, 2025

View reviewed changes

QU3B1M added 5 commits January 15, 2025 09:10

Convert indexes, templates, and API URI related configuration fields …

2bbf3a8

…to constants

Update tests with new constant values

f240284

Merge branch 'master' into enhancement/180-command-manager-configuration

853aa18

Rename timeout setting to client.timeout for clearer naming

ec0e85a

Update docstrings

14f83dc

AlexRuiz7 reviewed Jan 16, 2025

View reviewed changes

Make constant settings static

364c4bb

Remove plugin-scurity.policy file as it is no longer needed

AlexRuiz7 requested changes Jan 16, 2025

View reviewed changes

AlexRuiz7 and others added 4 commits January 21, 2025 12:02

Log settings on plugin's start (#235)

6b76309

Merge branch 'master' into enhancement/180-command-manager-configuration

f4d0712

Signed-off-by: Álex Ruiz <[email protected]>

Fix import

d501d72

f-galland approved these changes Jan 22, 2025

View reviewed changes

QU3B1M and others added 2 commits January 23, 2025 08:48

Fix old configuration references

4c4b8eb

Remove volatile from PluginSettings instance

c5e601f

AlexRuiz7 approved these changes Jan 23, 2025

View reviewed changes

AlexRuiz7 merged commit e35033e into master Jan 23, 2025
1 check passed

AlexRuiz7 deleted the enhancement/180-command-manager-configuration branch January 23, 2025 16:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Command Manager settings on opensearch.yml #213

Implement Command Manager settings on opensearch.yml #213

QU3B1M commented Jan 13, 2025 •

edited

Loading

f-galland commented Jan 14, 2025

f-galland commented Jan 14, 2025

AlexRuiz7 left a comment

AlexRuiz7 left a comment •

edited

Loading

AlexRuiz7 commented Jan 16, 2025 •

edited

Loading

AlexRuiz7 commented Jan 16, 2025 •

edited

Loading

AlexRuiz7 commented Jan 16, 2025 •

edited

Loading

AlexRuiz7 commented Jan 16, 2025 •

edited

Loading

AlexRuiz7 left a comment •

edited

Loading

f-galland left a comment

Implement Command Manager settings on opensearch.yml #213

Implement Command Manager settings on opensearch.yml #213

Conversation

QU3B1M commented Jan 13, 2025 • edited Loading

Description

Command Manager Plugin Configuration

Configuration Structure

Configuration Parameters

Notes

Issues Resolved

f-galland commented Jan 14, 2025

Testing changing the configuration options:

Test config:

Writing commands works:

The new index is created and used:

f-galland commented Jan 14, 2025

AlexRuiz7 left a comment

Choose a reason for hiding this comment

AlexRuiz7 left a comment • edited Loading

Choose a reason for hiding this comment

AlexRuiz7 commented Jan 16, 2025 • edited Loading

AlexRuiz7 commented Jan 16, 2025 • edited Loading

AlexRuiz7 commented Jan 16, 2025 • edited Loading

AlexRuiz7 commented Jan 16, 2025 • edited Loading

AlexRuiz7 left a comment • edited Loading

Choose a reason for hiding this comment

Summary of issues detected

f-galland left a comment

Choose a reason for hiding this comment

QU3B1M commented Jan 13, 2025 •

edited

Loading

AlexRuiz7 left a comment •

edited

Loading

AlexRuiz7 commented Jan 16, 2025 •

edited

Loading

AlexRuiz7 commented Jan 16, 2025 •

edited

Loading

AlexRuiz7 commented Jan 16, 2025 •

edited

Loading

AlexRuiz7 commented Jan 16, 2025 •

edited

Loading

AlexRuiz7 left a comment •

edited

Loading