[DRAFT] Tiered-Storage pause/resume

[Lazin]

This PR changes behaviour in situation when the TS is put on pause. Previously, Redpanda could potentially produce a gap in the partition if TS was paused for a long time. The local retention was allowed to delete segments which are not uploaded yet if the TS was paused. This may may be problematic for applications which expect to "see" all offsets without gaps.

The new behaviour is safe by default in this regard. The TS will block local retention even if it's paused. This creates another problem. The partition with stopped TS may consume all available disk space. To avoid this there is a way to enable old behaviour. The new topic property called redpanda.remote.allow_gaps is introduced by this PR. It is set to false by default. It only affects the way local retention behaves when TS is paused. If it's set to false local retention is blocked to avoid offset gaps. If it's set to true the local retention is unblocked but gaps could be created.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

Features

• Topic property redpanda.remote.write can be set to False safely without the risk of creating gaps in the offset range.
• New topic property redpanda.remote.allow_gaps is added. If it's set to True Redpanda is allowed to create gaps in the offset range when TS is paused.

[vbotbuildovich]

Retry command for Build#61005

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/describe_topics_test.py::DescribeTopicsTest.test_describe_topics_with_documentation_and_types
tests/rptest/tests/alter_topic_configuration_test.py::AlterTopicConfiguration.test_set_config_from_describe

[vbotbuildovich]

CI test results

test results on build#61005

test_id	test_kind	job_url	test_status	passed
rptest.tests.alter_topic_configuration_test.AlterTopicConfiguration.test_set_config_from_describe	ducktape	https://buildkite.com/redpanda/redpanda/builds/61005#01948a9a-6abf-4123-a661-c2759d178193	FAIL	0/20
rptest.tests.alter_topic_configuration_test.AlterTopicConfiguration.test_set_config_from_describe	ducktape	https://buildkite.com/redpanda/redpanda/builds/61005#01948aad-2bd3-47bf-9b88-bcfa28686146	FAIL	0/20
rptest.tests.controller_log_limiting_test.ControllerLogLimitMirrorMakerTests.test_mirror_maker_with_limits	ducktape	https://buildkite.com/redpanda/redpanda/builds/61005#01948a9a-6ac0-4ccb-a47c-302913565c2d	FLAKY	1/2
rptest.tests.describe_topics_test.DescribeTopicsTest.test_describe_topics_with_documentation_and_types	ducktape	https://buildkite.com/redpanda/redpanda/builds/61005#01948a9a-6abf-495e-a73e-f0eb68b06304	FAIL	0/20
rptest.tests.describe_topics_test.DescribeTopicsTest.test_describe_topics_with_documentation_and_types	ducktape	https://buildkite.com/redpanda/redpanda/builds/61005#01948aad-2bd3-4012-9514-156c5e6a9c58	FAIL	0/20
test_compat_rpunit.test_compat_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61005#01948a51-7484-44b6-972e-64109a8053c8	FAIL	0/2
test_compat_rpunit.test_compat_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61005#01948a51-7484-49a3-ba43-9bdf53f0a33c	FAIL	0/2