Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[th2-5267] Shutdown processor thread in EventPersistor.close method #107

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/build-dev-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,14 @@ jobs:
build:
uses: th2-net/.github/.github/workflows/compound-java.yml@main
with:
build-target: 'Docker'
build-target: 'Docker,Sonatype'
devRelease: true
createTag: true
docker-username: ${{ github.actor }}
secrets:
docker-password: ${{ secrets.GITHUB_TOKEN }}
sonatypeUsername: ${{ secrets.SONATYPE_NEXUS_USERNAME }}
sonatypePassword: ${{ secrets.SONATYPE_NEXUS_PASSWORD }}
sonatypeSigningKey: ${{ secrets.SONATYPE_GPG_ARMORED_KEY }}
sonatypeSigningPassword: ${{ secrets.SONATYPE_SIGNING_PASSWORD }}
nvd-api-key: ${{ secrets.NVD_APIKEY }}
6 changes: 5 additions & 1 deletion .github/workflows/build-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,14 @@ jobs:
build:
uses: th2-net/.github/.github/workflows/compound-java.yml@main
with:
build-target: 'Docker'
build-target: 'Docker,Sonatype'
devRelease: false
createTag: true
docker-username: ${{ github.actor }}
secrets:
docker-password: ${{ secrets.GITHUB_TOKEN }}
sonatypeUsername: ${{ secrets.SONATYPE_NEXUS_USERNAME }}
sonatypePassword: ${{ secrets.SONATYPE_NEXUS_PASSWORD }}
sonatypeSigningKey: ${{ secrets.SONATYPE_GPG_ARMORED_KEY }}
sonatypeSigningPassword: ${{ secrets.SONATYPE_SIGNING_PASSWORD }}
nvd-api-key: ${{ secrets.NVD_APIKEY }}
7 changes: 6 additions & 1 deletion .github/workflows/build-sanpshot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,13 @@ jobs:
build-job:
uses: th2-net/.github/.github/workflows/compound-java-dev.yml@main
with:
build-target: 'Docker'
build-target: 'Docker,Sonatype'
docker-username: ${{ github.actor }}
scanner-enabled: false # FIXME: remove this parameter
secrets:
docker-password: ${{ secrets.GITHUB_TOKEN }}
sonatypeUsername: ${{ secrets.SONATYPE_NEXUS_USERNAME }}
sonatypePassword: ${{ secrets.SONATYPE_NEXUS_PASSWORD }}
sonatypeSigningKey: ${{ secrets.SONATYPE_GPG_ARMORED_KEY }}
sonatypeSigningPassword: ${{ secrets.SONATYPE_SIGNING_PASSWORD }}
nvd-api-key: ${{ secrets.NVD_APIKEY }}
16 changes: 14 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Overview (5.8.0)
# Overview (5.9.0)

Event store (estore) is an important th2 component responsible for storing events into Cradle. Please refer to [Cradle repository] (https://github.com/th2-net/cradleapi/blob/master/README.md) for more details. This component has a pin for listening events via MQ.

Expand Down Expand Up @@ -31,6 +31,7 @@ spec:
maxTaskDataSize: 536870912
maxRetryCount: 3
processingThreads: 1
persisotrTerminationTimeout: 5000
mqRouter:
prefetchCount: 100
extendedSettings:
Expand Down Expand Up @@ -65,7 +66,8 @@ Configuration is provided as `custom.json` file
"maxTaskDataSize": 133169152,
"maxRetryCount": 1000000,
"retryDelayBase": 5000,
"processingThreads": 1
"processingThreads": 1,
"persisotrTerminationTimeout": 5000
}
```

Expand All @@ -76,6 +78,7 @@ Configuration is provided as `custom.json` file
+ _retryDelayBase_ - constant that will be used to calculate next retry time(ms) (default: 5000):
retryDelayBase * retryNumber
+ _processingThreads_ - number of task processing threads (default: 1 thread)
+ _persisotrTerminationTimeout_ - the timeout in milliseconds to await for the persisotr thread complete (default: 5000)

If some of these parameters are not provided, estore will use default(undocumented) value.
If _maxTaskCount_ or _maxTaskDataSize_ limits are reached during processing, estore will pause processing new events
Expand All @@ -101,6 +104,15 @@ More details [here](doc/perftest-5.7.0-dev/perftest.md).

# Changes

## 5.9.0

* Publish estore as moven artifact
* added `persisotrTerminationTimeout` option
* Updated
* th2 gradle plugin `0.2.1` based on bom: `4.10.0`
* Added suppressions:
* CVE-2025-25193, CVE-2025-24970 - These vulnerabilities does not affect us because this component don’t use netty.

## 5.8.0

* Updated
Expand Down
4 changes: 2 additions & 2 deletions build.gradle
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
plugins {
id "application"
id "com.exactpro.th2.gradle.component" version "0.1.6"
id "com.exactpro.th2.gradle.component" version "0.2.1"
}

group = 'com.exactpro.th2'
Expand Down Expand Up @@ -43,7 +43,7 @@ dependencies {
implementation("com.exactpro.th2:common-utils:2.3.0-dev") {
because("executor service utils is used")
}
implementation 'com.exactpro.th2:task-utils:0.1.2'
implementation 'com.exactpro.th2:task-utils:0.1.3-TH2-5267-13392619539-6f14721-SNAPSHOT'
implementation "com.exactpro.th2:cradle-core:$cradleVersion"
implementation "com.exactpro.th2:cradle-cassandra:$cradleVersion"

Expand Down
2 changes: 1 addition & 1 deletion gradle.properties
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
release_version=5.8.0
release_version=5.9.0
description='th2 estore component'
vcs_url=https://github.com/th2-net/th2-estore

16 changes: 13 additions & 3 deletions src/main/java/com/exactpro/th2/estore/Configuration.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright 2020-2024 Exactpro (Exactpro Systems Limited)
* Copyright 2020-2025 Exactpro (Exactpro Systems Limited)
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
Expand All @@ -23,16 +23,18 @@ public class Configuration {
private static final int DEFAULT_MAX_TASK_COUNT = 128;
private static final long DEFAULT_RETRY_DELAY_BASEM_S = 5000;
private static final int DEFAULT_PROCESSING_THREADS = 1;
private static final long DEFAULT_WAIT_TIMEOUT = 5000L;

private final int maxTaskCount;
private final long maxTaskDataSize;
private final int maxRetryCount;
private final long retryDelayBase;
private final int processingThreads;
private final long persisotrTerminationTimeout;

public Configuration() {
this(DEFAULT_MAX_TASK_COUNT, defaultMaxDataSize(), DEFAULT_MAX_TASK_RETRIES,
DEFAULT_RETRY_DELAY_BASEM_S, DEFAULT_PROCESSING_THREADS);
DEFAULT_RETRY_DELAY_BASEM_S, DEFAULT_PROCESSING_THREADS, DEFAULT_WAIT_TIMEOUT);
}

@JsonCreator
Expand All @@ -41,19 +43,22 @@ public Configuration(
@JsonProperty("maxTaskDataSize") Long maxTaskDataSize,
@JsonProperty("maxRetryCount") Integer maxRetryCount,
@JsonProperty ("retryDelayBase") Long retryDelayBase,
@JsonProperty("processingThreads") Integer processingThreads
@JsonProperty("processingThreads") Integer processingThreads,
@JsonProperty("persisotrTerminationTimeout") Long persisotrTerminationTimeout
) {
this.maxTaskCount = maxTaskCount == null ? DEFAULT_MAX_TASK_COUNT : maxTaskCount;
this.maxTaskDataSize = maxTaskDataSize == null ? defaultMaxDataSize() : maxTaskDataSize;
this.maxRetryCount = maxRetryCount == null ? DEFAULT_MAX_TASK_RETRIES : maxRetryCount;
this.retryDelayBase = retryDelayBase == null ? DEFAULT_RETRY_DELAY_BASEM_S : retryDelayBase;
this.processingThreads = processingThreads == null ? DEFAULT_PROCESSING_THREADS : processingThreads;
this.persisotrTerminationTimeout = persisotrTerminationTimeout == null ? DEFAULT_WAIT_TIMEOUT : persisotrTerminationTimeout;

if (this.maxTaskCount < 1) throw new IllegalArgumentException("'maxTaskCount' should be >=1. Actual: " + maxTaskCount);
if (this.maxTaskDataSize < 1) throw new IllegalArgumentException("'maxTaskDataSize' should be >=1. Actual: " + maxTaskDataSize);
if (this.maxRetryCount < 0) throw new IllegalArgumentException("'maxRetryCount' should be >=0. Actual: " + maxRetryCount);
if (this.retryDelayBase < 1) throw new IllegalArgumentException("'retryDelayBase' should be >=1. Actual: " + retryDelayBase);
if (this.processingThreads < 1) throw new IllegalArgumentException("'processingThreads' should be >=1. Actual: " + processingThreads);
if (this.persisotrTerminationTimeout < 0) throw new IllegalArgumentException("'persisotrTerminationTimeout' should be >=0. Actual: " + persisotrTerminationTimeout);
}

public int getMaxTaskCount() {
Expand All @@ -76,6 +81,10 @@ public int getProcessingThreads() {
return processingThreads;
}

public long getPersisotrTerminationTimeout() {
return persisotrTerminationTimeout;
}

private static long defaultMaxDataSize() {
return Runtime.getRuntime().totalMemory() / 2;
}
Expand All @@ -92,6 +101,7 @@ public String toString() {
PREFIX + "maxRetryCount" + SEP + getMaxRetryCount() + SUFFIX +
PREFIX + "retryDelayBase" + SEP + getRetryDelayBase() + SUFFIX +
PREFIX + "processingThreads" + SEP + getProcessingThreads() + SUFFIX_LAST +
PREFIX + "persisotrTerminationTimeout" + SEP + getPersisotrTerminationTimeout() + SUFFIX_LAST +
"}";
}
}
Loading
Loading