ducktape: handle SIGABORT in Redpanda

[travisdowns]

Currently on raise_on_crash we detect if Redpanda has crashed, in order to surface that as the failure reason instead of something like timeout or connection refused which invariably occurs when Redpanda dies during a test.

However we didn't include "Aborting on shard" in the crash regex, which what occurs when abort is triggered, which happening in various scenarios including OOM.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

• none

[StephanDollberg]

How is this stuff different from the one in utils.py which already has this?

[travisdowns]

How is this stuff different from the one in utils.py which already has this?

I guess that's used for "log search" which finds problematic lines in the log (e.g., errors), the thing I changed is for raise_on_crash which is trying to detect if Redpanda crashed.

[StephanDollberg]

Right so we would did already detect it but just in a different path? What does raise_on_crash do differently? I guess it gets tagged more directly?

[vbotbuildovich]

CI test results

test results on build#61399

test_id	test_kind	job_url	test_status	passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery	ducktape	https://buildkite.com/redpanda/redpanda/builds/61399#0194b7f2-34f8-4c39-aafa-c4cc82fa2bf8	FLAKY	1/2
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61399#0194b7f2-34f5-40cf-812a-6802a19ed033	FLAKY	1/4
rptest.tests.partition_balancer_test.PartitionBalancerTest.test_unavailable_nodes	ducktape	https://buildkite.com/redpanda/redpanda/builds/61399#0194b7f2-34f5-40cf-812a-6802a19ed033	FLAKY	1/2
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic	ducktape	https://buildkite.com/redpanda/redpanda/builds/61399#0194b7f2-34f7-4e93-957b-8881aae82ba1	FLAKY	1/2
rptest.transactions.consumer_offsets_test.VerifyConsumerOffsets.test_consumer_group_offsets	ducktape	https://buildkite.com/redpanda/redpanda/builds/61399#0194b7ed-4b09-4bcd-bfd5-742756bd8170	FLAKY	1/2

[travisdowns]

What does raise_on_crash do differently? I guess it gets tagged more directly?

raise_on_crash is used in the context that the test is already failing (i.e., it threw an exception), but we want to do a check to see if Redpanda crashed, so we can swap the test exception which is often "timeout" or "connection refused", with "hey, redpanda crashed".

The log search path OTOH is executed when the test succeeds, in order to possibly fail it if we find a bad log line.

So the log search path having a large set of detected problems make sense: we don't want to replace a test failure with an "error detected in logs" in general, but we do want to do that for a crash.

[travisdowns]

Right so we would did already detect it but just in a different path?

Yes, I think so, but in practice that almost never happens because when Redpanda crashes the test fails in some way first so the log search doesn't happen.

So yes as far as I know this is about better tagging.