Update to hyper 1.0 #3371

jarhodes314 · 2025-01-30T15:28:34Z

Proposed changes

This updates nearly all our usage of hyper to 1.0, including through axum. It also updates rustls to 0.23.

I've not tested it properly, although I'm not expecting massive breaking changes. There is currently 1 failing unit test that I will look into further. I believe the functionality is working as intended, but the check the test performs to clarify the error is of the expected type isn't working. Additionally there is some new flakiness in the tedge_mqtt_bridge tests that I would like to understand better (but I think if the issue is a real bug rather than a test issue, it boils down to "when we put the bridge under a frankly ridiculous amount of load, things go a bit wonky").

Currently we do still have a dependency on hyper 0.14 through rumqttd, although this only used for testing. I have updated rumqttc to the latest version using a git dependency (pinned to a commit hash to ensure we don't receive unexpected breaking changes).

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
Documentation Update (if none of the other choices apply)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Upgrade hyper, hyper-rustls to latest versions #3360

Checklist

I have read the CONTRIBUTING doc
I have signed the CLA (in all commits with git commit -s)
I ran cargo fmt as mentioned in CODING_GUIDELINES
I used cargo clippy as mentioned in CODING_GUIDELINES
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

Further comments

reubenmiller · 2025-01-30T15:52:23Z

@jarhodes314 FYI: I think we're going to have to stick to the ring crypto backend as aws-lc-rs does not support all of our targets.

jarhodes314 · 2025-01-30T17:05:00Z

@jarhodes314 FYI: I think we're going to have to stick to the ring crypto backend as aws-lc-rs does not support all of our targets.

Annoyingly, part of the dependency on aws-lc-rs is through rumqttc. I've got it working locally, I'm just trying to patch it on a fork so I can create a PR against rumqttc to fix the issue properly. Once I've got a fork, I'll point the git dependency at that so this is at least unblocked for testing.

jarhodes314 · 2025-01-30T17:25:18Z

Now created bytebeamio/rumqtt#941, I'll update this PR with the forked dependency to unblock the build.

github-actions · 2025-01-30T17:50:28Z

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
560	0	3	560	100	1h33m54.914218s

didier-wenzek

So many breaking API changes here and there :-( Thank you for tackling this.

About upgrading rumqttc, this PR is the way to go compared to #3359. Indeed, even if currently depending on fork of the main branch (bytebeamio/rumqtt#941), the changes are more future proof.

didier-wenzek · 2025-01-31T08:21:51Z

crates/common/mqtt_channel/src/config.rs


 impl zeroize::Zeroize for PrivateKey {
    fn zeroize(&mut self) {
-        self.0 .0.zeroize()
+        // FIXME I can't be implemented with rustls 0.23 due to immutability


One can submit a PR on rustls_pki_types

Since this is related to security and a removal of functionality, we should create a ticket to track it (e.g. and create a PR as @didier-wenzek suggested) before merging so we don't forget about this.

codecov · 2025-01-31T13:42:20Z

Codecov Report

Attention: Patch coverage is 76.97842% with 64 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/extensions/c8y_auth_proxy/src/server.rs	76.25%	18 Missing and 1 partial ⚠️
crates/common/axum_tls/src/acceptor.rs	78.04%	8 Missing and 1 partial ⚠️
crates/common/axum_tls/src/files.rs	69.56%	0 Missing and 7 partials ⚠️
...ates/extensions/tedge_http_ext/src/test_helpers.rs	25.00%	6 Missing ⚠️
.../core/tedge_agent/src/http_server/file_transfer.rs	76.47%	2 Missing and 2 partials ⚠️
...s/common/certificate/src/parse_root_certificate.rs	78.57%	2 Missing and 1 partial ⚠️
crates/common/mqtt_channel/src/config.rs	0.00%	3 Missing ⚠️
crates/common/axum_tls/src/error_matching.rs	0.00%	0 Missing and 2 partials ⚠️
...s/common/certificate/src/cloud_root_certificate.rs	33.33%	0 Missing and 2 partials ⚠️
crates/core/tedge/src/cli/mqtt/subscribe.rs	0.00%	2 Missing ⚠️
... and 6 more

Additional details and impacted files

📢 Thoughts on this report? Let us know!

Bravo555

The changes look good, but 3 things are left:

hyper 1.5 -> 1.6
can CryptoProvider::install_default() be removed?
most important: handling Zeroize, which we can no longer guarantee because CertificateDer and PrivateKeyDer are immutable. Making a PR to upstream is an option, but it could take some time. In the meantime, is zeroizing necessary so that we can't replace it with a noop? We're not really testing for it anywhere, so I'm not sure what exactly it provides us.

crates/common/certificate/src/parse_root_certificate.rs

Bravo555 · 2025-01-31T13:54:47Z

crates/extensions/tedge_http_ext/src/tests.rs

+    use rustls::crypto;
+    let _ = crypto::CryptoProvider::install_default(crypto::ring::default_provider());


question: why do we need this snippet in the tests?

When I remove it, the tests still work. It's a bit of a pain to always have to call this in tests (also because cryptography should be somewhat abstracted and caller shouldn't have to care about it, so we don't always necessarily know when we'll have to call into rustls).

So because it still works without the snippet, I reckon it either doesn't call into ServerConfig/ClientConfig builders or some other default is used?

Huh, having a closer look at this test, it seems like we're not using HTTPS at all (mockito doesn't support HTTPS as far as I can tell, and we certainly don't do anything to it that looks like it would enable HTTPS). @didier-wenzek can you elaborate on whether this lack of HTTPS in the test is an error or whether I'm just misunderstanding something?

I confirm that there is no more test over HTTPS, as you said because mockito doesn't support HTTPS. There was a test against https://httpbin.org, but this has been removed because subject to flackiness. See #1985

I also don't get any test failures when commenting out crates/extensions/c8y_auth_proxy/src/server.rs:1250, so I doubt it's because HTTPS isn't used.

I suspect because we now only have ring enabled and not aws-lc-rs, rustls does this for us automatically, which would explain why I previously ran into errors and now don't.

Bravo555 · 2025-01-31T13:58:58Z

Cargo.toml

-hyper-rustls = { version = "0.24", default-features = false, features = [
-    "tokio-runtime",
+hyper = { version = "1.5", default-features = false }
+hyper-rustls = { version = "0.27.5", default-features = false, features = [


question: why 1.5? cargo check works for me with hyper 1.6

Suggested change

hyper-rustls = { version = "0.27.5", default-features = false, features = [

hyper = { version = "1.6", default-features = false }

I don't quite know, but we're already using 1.6 anyway according to the lockfile

Ah, my bad, indeed there's 1.6 in the lockfile, just my Dependi extension took the older 0.14 version used for compat with rumqttd and displayed me a "not up to date mark", so we're in the clear.

Bravo555 · 2025-01-31T14:02:00Z

Cargo.toml

suggestion: http-body and hyper-util can be upgraded to latest patch version using cargo upgrade

Bravo555 · 2025-01-31T14:05:11Z

Cargo.lock

 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c546c80d6be4bc6a00c0f01730c08df82eaa7a7a61f11d656526506112cc1709"
 dependencies = [
- "thiserror-impl",
+ "thiserror-impl 1.0.61",
+]
+
+[[package]]
+name = "thiserror"
+version = "2.0.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d452f284b73e6d76dd36758a0c8684b1d5be31f92b89d07fd5822175732206fc"
+dependencies = [
+ "thiserror-impl 2.0.11",


note: Seems that hyper & friends are using a new major release of thiserror. Our crates and some other dependencies still use 1.0, so 2 versions are pulled, but because thiserror basically does macros, it shouldn't have major codegen implications.

didier-wenzek

I'm okay to merge this PR despite there are still a failing tests (acceptor_rejects_untrusted_client_certificates and bridge_disconnect_while_sending), because:

This PR is going in the correct direction, i.e. leveraging the HEAD of rumqttc to upgrade to rustls 0.23 despite conflicting constraints. Reverting to the next released version of rumqttc will be feasible with no major difficulties.
This PR spreads a lot of small but non-trivial changes across all the creates depending on TLS. It would be good to avoid conflicts.

Follow-up tasks:

Fix the two tests that have been ignored here.
Create a PR on rumqttc for chore(rumqttc): disable default-features for tokio-rustls bytebeamio/rumqtt#941
Consider to create a PR on rustls_pki_types to zeroize private keys.

@reubenmiller are you okay merging this PR soon?

didier-wenzek · 2025-01-31T14:17:19Z

crates/extensions/c8y_auth_proxy/src/body.rs

        match self {
-            Self::Small(bytes) => (bytes.clone().into(), Some(bytes)),
+            Self::Small(bytes) => (
+                SyncBody::new(Full::new(bytes.clone()).map_err(|i| match i {})),


I don't get how matching the Infallible error can help to make an impl http_body::Body() from a Result.

What am I missing?

Because Infallible is an uninhabited type (i.e. the enum has no variants, so a value of the type can never exist), match i {} is valid and evaluates to !, so this can be cast to whatever implmentation is necessary (in this case axum::Error).

match i {} evaluates to !

This is what I was missing.

didier-wenzek · 2025-01-31T14:24:34Z

.config/nextest.toml

@@ -4,7 +4,7 @@ test-threads = "num-cpus"

 [[profile.default.overrides]]
 # Remove once https://github.com/thin-edge/thin-edge.io/issues/3021 is resolved
-filter = 'test(bridge_reconnects_successfully_after_local_connection_interrupted) or test(bridge_reconnects_successfully_after_cloud_connection_interrupted)'


When has been these two flaky tests resolved? If so it would be good to close #3021.

I'm not entirely sure, but I suspect it was with the reliability improvements you made in #3122. I've not seen the flakiness locally myself in cargo-nextest and I have seen quite a lot of flaky tests in recent weeks, I believe due to the strain it puts my work laptop under.

jarhodes314 · 2025-01-31T15:19:13Z

Create a PR on rumqttc for chore(rumqttc): disable default-features for tokio-rustls bytebeamio/rumqtt#941

@didier-wenzek I don't quite understand this, you've linked to a PR against rumqttc but suggested this doesn't exist?

Fix the two tests that have been ignored here.

Yep, the bridge_disconnect one is the main new source of flakiness in the bridge tests. I will reduce the number of messages it sends because I think the issue is to do with rumqttd getting overloaded. I don't quite understand how (I presume a change in rumqttc may have changed how it sends messages:

Here's a passing example:

running 1 test
[2025-01-31T14:57:50Z INFO  tedge_mqtt_bridge::health] MQTT bridge connected to cloud broker
[2025-01-31T14:57:50Z INFO  tedge_mqtt_bridge] Bridge cloud connection subscribing to [Filter = s/ds, Qos = AtLeastOnce]
[2025-01-31T14:57:50Z INFO  tedge_mqtt_bridge::health] MQTT bridge connected to local broker
[2025-01-31T14:57:50Z INFO  tedge_mqtt_bridge] Bridge local connection subscribing to [Filter = c8y/s/us, Qos = AtLeastOnce]
[2025-01-31T14:57:50Z INFO  tedge_mqtt_bridge] Bridge local connection received ack for unknown pkid=2
[2025-01-31T14:57:51Z ERROR tedge_mqtt_bridge::health] MQTT bridge failed to connect to cloud broker: Mqtt state: Connection closed by peer abruptly
[2025-01-31T14:57:51Z INFO  tedge_mqtt_bridge::health] MQTT bridge connected to cloud broker
[2025-01-31T14:57:51Z INFO  tedge_mqtt_bridge] Bridge cloud connection subscribing to [Filter = s/ds, Qos = AtLeastOnce]
[2025-01-31T14:57:51Z INFO  tedge_mqtt_bridge] Bridge local connection received ack for unknown pkid=3
[2025-01-31T14:57:51Z INFO  tedge_mqtt_bridge] Bridge local connection received ack for unknown pkid=4
test bridge_disconnect_while_sending ... ok

And a failing example:

running 1 test
test bridge_disconnect_while_sending ... FAILED

failures:

---- bridge_disconnect_while_sending stdout ----
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge::health] MQTT bridge connected to cloud broker
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge] Bridge cloud connection subscribing to [Filter = s/ds, Qos = AtLeastOnce]
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge::health] MQTT bridge connected to local broker
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge] Bridge local connection subscribing to [Filter = c8y/s/us, Qos = AtLeastOnce]
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge] Bridge local connection received ack for unknown pkid=2
[2025-01-31T14:56:40Z ERROR tedge_mqtt_bridge::health] MQTT bridge failed to connect to cloud broker: Mqtt state: Connection closed by peer abruptly
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge::health] MQTT bridge connected to cloud broker
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge] Bridge cloud connection subscribing to [Filter = s/ds, Qos = AtLeastOnce]
[2025-01-31T14:56:40Z ERROR tedge_mqtt_bridge::health] MQTT bridge failed to connect to local broker: Mqtt state: Connection closed by peer abruptly
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge::health] MQTT bridge connected to local broker
[2025-01-31T14:56:40Z INFO  tedge_mqtt_bridge] Bridge local connection subscribing to [Filter = c8y/s/us, Qos = AtLeastOnce]
thread 'bridge_disconnect_while_sending' panicked at crates/extensions/tedge_mqtt_bridge/tests/bridge.rs:232:45:
called `Result::unwrap()` on an `Err` value: timed-out waiting for received message

Caused by:
    deadline has elapsed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The thing that sticks out to me is MQTT bridge failed to connect to local broker: Mqtt state: Connection closed by peer abruptly. The "cloud" version of this is intended, and is part of the test. The fact it happens for the "local" broker too indicates to me that rumqttd is the cause of this issue as the local broker should remain connected, and I'm fairly confident in that assertion.

This does lead to the question of why do we have a new issue with rumqttd, a dependency that hasn't updated. My suspicions are that the updated rumqttc has changed how it publishes the messages, and rumqttd isn't coping with this. It definitely seems to be related to the amount of load the system is under. If the test manages to publish the message and then the broker restarts/does something else that drops queued messages, this would explain the non-forwarding of messages. I'm trying to understand this further, but it's proving a bit difficult as enabling more logs changes the performance characteristics sufficiently to not cause the flakiness to appear.

The fact that it's (mostly, but not entirely) that test may also prove useful in understanding what's going on here. I believe I've seen extremely occasional issues (1 in some thousands of runs) with bridge_many_messages too, although that sends half the number of messages and the bridge connection remains in-tact.

didier-wenzek · 2025-01-31T15:44:07Z

Create a PR on rumqttc for chore(rumqttc): disable default-features for tokio-rustls bytebeamio/rumqtt#941

@didier-wenzek I don't quite understand this, you've linked to a PR against rumqttc but suggested this doesn't exist?

Oops, I failed to noticed that is was not a link to your fork.

jarhodes314 · 2025-01-31T17:32:59Z

crates/core/tedge/src/main.rs

@@ -31,7 +31,6 @@ use tracing::log;
 static ALLOCATOR: Cap<alloc::System> = Cap::new(alloc::System, usize::MAX);

 fn main() -> anyhow::Result<()> {
-    clap_complete::CompleteEnv::with_factory(TEdgeCli::command).complete();


Oops, must've accidentally deleted this when removing the CryptoProvider stuff.

I've created #3373 to address the fact this went undetected.

reubenmiller · 2025-02-01T07:56:34Z

I'm okay to merge this PR despite there are still a failing tests (acceptor_rejects_untrusted_client_certificates and bridge_disconnect_while_sending), because:

This PR is going in the correct direction, i.e. leveraging the HEAD of rumqttc to upgrade to rustls 0.23 despite conflicting constraints. Reverting to the next released version of rumqttc will be feasible with no major difficulties.

This PR spreads a lot of small but non-trivial changes across all the creates depending on TLS. It would be good to avoid conflicts.

Follow-up tasks:

Fix the two tests that have been ignored here.

Create a PR on rumqttc for chore(rumqttc): disable default-features for tokio-rustls bytebeamio/rumqtt#941

Consider to create a PR on rustls_pki_types to zeroize private keys.

@reubenmiller are you okay merging this PR soon?

In principle, yes I'm ok with this PR merging soon), we just need to make sure we create all of the required follow up tickets prior to merging (so we don't forget about it).

jarhodes314 · 2025-02-03T10:27:15Z

Now created the follow up tickets and linked to the relevant one from the FIXME I created in the code re zeroize. I'll set to work on creating that PR today.

didier-wenzek

Approved. Thank you for tackling this.

Upgrade axum to latest version

e1092d3

jarhodes314 requested review from didier-wenzek, albinsuresh, rina23q and reubenmiller as code owners January 30, 2025 15:28

jarhodes314 had a problem deploying to Test Pull Request January 30, 2025 15:28 — with GitHub Actions Failure

jarhodes314 requested a review from Bravo555 January 30, 2025 15:28

jarhodes314 had a problem deploying to Test Pull Request January 30, 2025 16:04 — with GitHub Actions Failure

jarhodes314 temporarily deployed to Test Pull Request January 30, 2025 17:26 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto January 30, 2025 17:35 — with GitHub Actions Failure

didier-wenzek reviewed Jan 31, 2025

View reviewed changes

jarhodes314 had a problem deploying to Test Pull Request January 31, 2025 12:50 — with GitHub Actions Failure

jarhodes314 force-pushed the feat/hyper-1 branch from 8e70689 to 6eacb6b Compare January 31, 2025 13:30

jarhodes314 temporarily deployed to Test Pull Request January 31, 2025 13:30 — with GitHub Actions Inactive

jarhodes314 temporarily deployed to Test Auto January 31, 2025 13:37 — with GitHub Actions Inactive

Bravo555 reviewed Jan 31, 2025

View reviewed changes

Bravo555 mentioned this pull request Jan 31, 2025

deps: upgrade to rumqttc-0.24, rustls-0.22 #3359

Closed

12 tasks

didier-wenzek reviewed Jan 31, 2025

View reviewed changes

jarhodes314 temporarily deployed to Test Pull Request January 31, 2025 15:26 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Pull Request January 31, 2025 15:28 — with GitHub Actions Failure

jarhodes314 temporarily deployed to Test Pull Request January 31, 2025 15:30 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto January 31, 2025 15:38 — with GitHub Actions Failure

jarhodes314 mentioned this pull request Jan 31, 2025

Add tests for shell completions #3373

Open

jarhodes314 commented Jan 31, 2025

View reviewed changes

jarhodes314 temporarily deployed to Test Pull Request January 31, 2025 18:18 — with GitHub Actions Inactive

jarhodes314 temporarily deployed to Test Auto January 31, 2025 18:25 — with GitHub Actions Inactive

reubenmiller added refactoring Developer value dependencies Pull requests that update a dependency file and removed refactoring Developer value labels Feb 3, 2025

This was referenced Feb 3, 2025

Fix acceptor_rejects_untrusted_client_certificates following hyper upgrade #3375

Open

Reinstate zeroisation of private keys #3376

Open

jarhodes314 added 6 commits February 3, 2025 10:24

Finish upgrading axum/hyper

024d12e

Update list of flaky tests

33d4759

Remove dependency on aws-lc-sys

0af1ab4

Print backtrace properly for operationerrors

3080a1d

Reduce the load of the bridge_disconnect test to alleviate flakiness

e6a48e4

Add ticket no to zeroize fixme

a2c8f02

jarhodes314 force-pushed the feat/hyper-1 branch from 30d38b2 to a2c8f02 Compare February 3, 2025 10:26

jarhodes314 temporarily deployed to Test Pull Request February 3, 2025 10:26 — with GitHub Actions Inactive

jarhodes314 temporarily deployed to Test Auto February 3, 2025 10:32 — with GitHub Actions Inactive

jarhodes314 requested review from didier-wenzek and removed request for rina23q February 3, 2025 11:46

Remove retries now flakiness is avoided through less load

3f61992

jarhodes314 temporarily deployed to Test Pull Request February 3, 2025 13:04 — with GitHub Actions Inactive

jarhodes314 temporarily deployed to Test Auto February 3, 2025 13:18 — with GitHub Actions Inactive

didier-wenzek approved these changes Feb 3, 2025

View reviewed changes

jarhodes314 added this pull request to the merge queue Feb 4, 2025

Merged via the queue into thin-edge:main with commit 073d9fb Feb 4, 2025
33 checks passed

Bravo555 mentioned this pull request Feb 5, 2025

Upgrade hyper, hyper-rustls to latest versions #3360

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to hyper 1.0 #3371

Update to hyper 1.0 #3371

jarhodes314 commented Jan 30, 2025

reubenmiller commented Jan 30, 2025

jarhodes314 commented Jan 30, 2025

jarhodes314 commented Jan 30, 2025

github-actions bot commented Jan 30, 2025 •

edited

Loading

didier-wenzek left a comment

didier-wenzek Jan 31, 2025

reubenmiller Feb 1, 2025

codecov bot commented Jan 31, 2025 •

edited

Loading

Bravo555 left a comment •

edited

Loading

Bravo555 Jan 31, 2025

jarhodes314 Jan 31, 2025

didier-wenzek Jan 31, 2025

Bravo555 Jan 31, 2025

jarhodes314 Jan 31, 2025

Bravo555 Jan 31, 2025

jarhodes314 Jan 31, 2025

Bravo555 Jan 31, 2025

Bravo555 Jan 31, 2025

jarhodes314 Jan 31, 2025

Bravo555 Jan 31, 2025

didier-wenzek left a comment

didier-wenzek Jan 31, 2025

jarhodes314 Jan 31, 2025

didier-wenzek Jan 31, 2025

didier-wenzek Jan 31, 2025

jarhodes314 Jan 31, 2025

jarhodes314 commented Jan 31, 2025

didier-wenzek commented Jan 31, 2025

jarhodes314 Jan 31, 2025

reubenmiller commented Feb 1, 2025

jarhodes314 commented Feb 3, 2025

didier-wenzek left a comment

		use rustls::crypto;
		let _ = crypto::CryptoProvider::install_default(crypto::ring::default_provider());

	hyper-rustls = { version = "0.27.5", default-features = false, features = [
	hyper = { version = "1.6", default-features = false }

Update to hyper 1.0 #3371

Update to hyper 1.0 #3371

Conversation

jarhodes314 commented Jan 30, 2025

Proposed changes

Types of changes

Paste Link to the issue

Checklist

Further comments

reubenmiller commented Jan 30, 2025

jarhodes314 commented Jan 30, 2025

jarhodes314 commented Jan 30, 2025

github-actions bot commented Jan 30, 2025 • edited Loading

Robot Results

didier-wenzek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 31, 2025 • edited Loading

Codecov Report

Bravo555 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

didier-wenzek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jarhodes314 commented Jan 31, 2025

didier-wenzek commented Jan 31, 2025

Choose a reason for hiding this comment

reubenmiller commented Feb 1, 2025

jarhodes314 commented Feb 3, 2025

didier-wenzek left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 30, 2025 •

edited

Loading

codecov bot commented Jan 31, 2025 •

edited

Loading

Bravo555 left a comment •

edited

Loading