[CORE-8525] - Iceberg schema evolution #24676

oleiman · 2025-01-03T05:50:09Z

This PR implements the Iceberg side of Schema Evolution. In broad strokes, it adds:

A visitor class for traversing (and annotating) two input schemas in lockstep to determine whether one is backward compatible with the other (i.e. A is evolvable into B), structurally.
A small, optional, mutable, variant metadata field in iceberg::nested_field, intended to support (1), both in progress and for post-traversal validation.
A visitor class for iceberg::nested_field::evolution_metadata, which validates the result of (1).
Lots of unit tests.

The visitor pattern in (1) is largely inspired by Apache Iceberg: CheckCompatibility.java and TypeUtil.java, though our implementation is somewhat simpler in the sense that we have std::variant w/ proper (enough) dynamic dispatch to arbitrary callables.

Microbenchmark

The benchmark included with this PR seeks to compare two mechanisms for traversing a pair of input schemas and updating field IDs:

"Basic" assignment which performs a simultaneous, non-recursive DFS over both schemas (copied from current catalog_schema_manager::fill_field_ids)
Annotate + Verify, which performs a single recursive DFs over both schemas, in lockstep, followed by a non-recursive DFS over the destination schema to validate & apply changes.

Notably this benchmark does not cover cases where the two schemas differ. This is primarily to test the computational cost and memory pressure of the two traversal mechanisms.

Conclusions

As expected, (2) takes more CPU time than (1), but generally keeps w/in the same order of magnitude.
- Unsurprisingly, the gap increases with both nesting level and total number of fields. It is simply more work.
The annotation step does not allocate any dynamic memory.
Since there is an increased cost, the onus is on us to perform these traversals only when absolutely necessary.

test	iterations	median	mad	min	max	allocs
StructVisitation.BasicFieldAssign	928951	368.580ns	0.675ns	367.234ns	369.255ns	6.000
StructVisitation.DescribeTransform	1018128	279.476ns	0.636ns	278.840ns	281.513ns	0.000
StructVisitation.ApplyTransform	755588	358.204ns	0.368ns	357.526ns	360.669ns	14.000
StructVisitation.BasicFieldAssignNested_5_1	1503843	191.487ns	0.060ns	191.142ns	191.900ns	12.000
StructVisitation.DescribeTransformNested_5_1	1714150	110.832ns	0.250ns	110.476ns	111.108ns	0.000
StructVisitation.ApplyTransformNested_5_1	1411552	150.257ns	0.158ns	150.099ns	150.801ns	6.000
StructVisitation.BasicFieldAssignNested_10_1	774890	348.358ns	0.029ns	348.328ns	349.848ns	22.000
StructVisitation.DescribeTransformNested_10_1	866611	204.199ns	0.106ns	204.029ns	204.343ns	0.000
StructVisitation.ApplyTransformNested_10_1	706771	252.356ns	0.004ns	252.346ns	252.441ns	11.000
StructVisitation.BasicFieldAssignNested_20_1	380216	665.858ns	0.078ns	665.654ns	665.936ns	42.000
StructVisitation.DescribeTransformNested_20_1	414367	415.410ns	0.078ns	415.288ns	415.489ns	0.000
StructVisitation.ApplyTransformNested_20_1	353225	464.909ns	0.052ns	464.836ns	465.052ns	21.000
StructVisitation.BasicFieldAssignNested_5_5	514812	536.770ns	0.104ns	536.666ns	536.968ns	10.000
StructVisitation.DescribeTransformNested_5_5	511058	559.132ns	0.696ns	555.391ns	559.829ns	0.000
StructVisitation.ApplyTransformNested_5_5	420292	447.914ns	0.271ns	447.088ns	448.384ns	10.000
StructVisitation.BasicFieldAssignNested_10_5	237045	1.006us	0.227ns	1.006us	1.007us	12.000
StructVisitation.DescribeTransformNested_10_5	228472	1.188us	1.240ns	1.179us	1.190us	0.000
StructVisitation.ApplyTransformNested_10_5	193793	764.561ns	0.106ns	763.729ns	764.784ns	11.000
StructVisitation.BasicFieldAssignNested_20_5	120174	1.895us	0.113ns	1.894us	1.897us	14.000
StructVisitation.DescribeTransformNested_20_5	112604	2.465us	0.226ns	2.441us	2.465us	0.000
StructVisitation.ApplyTransformNested_20_5	97074	1.381us	0.059ns	1.380us	1.381us	12.000
StructVisitation.BasicFieldAssignNested_100_5	23741	8.744us	3.199ns	8.738us	8.790us	18.000
StructVisitation.DescribeTransformNested_100_5	22006	12.011us	1.029ns	12.008us	12.013us	0.000
StructVisitation.ApplyTransformNested_100_5	19258	6.220us	1.268ns	6.218us	6.221us	14.000
StructVisitation.BasicFieldAssignNested_100_20	6426	37.448us	325.334ns	36.492us	37.839us	18.000
StructVisitation.DescribeTransformNested_100_20	4653	94.026us	54.727ns	93.971us	94.241us	0.000
StructVisitation.ApplyTransformNested_100_20	4144	26.626us	186.807ns	26.090us	28.064us	31.000

Backports Required

Release Notes

none

Signed-off-by: Oren Leiman <[email protected]>

oleiman · 2025-01-07T00:21:13Z

CI Failures:

bazel-build failed to checkout vtools?

oleiman · 2025-01-07T02:25:05Z

force push - refactored interfaces to indicate at each step whether anything changed (fields added/removed, types promoted, requiredness changed). Might want more granularity at some point, but basically as a first step want to signal to calling code whether a metadata update is required.

oleiman · 2025-01-07T04:25:51Z

CI Failures:

build-debug - failed to remove builder permission denied...infra

vbotbuildovich · 2025-01-07T07:33:06Z

CI test results

test results on build#60330

test_id	test_kind	job_url	test_status	passed
partition_balancer_simulator_test_rpunit.partition_balancer_simulator_test_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/60330#01943e94-cbb4-4e99-9963-4c53628a6f41	FLAKY	1/2

test results on build#60369

test_id	test_kind	job_url	test_status	passed
rptest.tests.consumer_offsets_consistency_test.ConsumerOffsetsConsistencyTest.test_flipping_leadership	ducktape	https://buildkite.com/redpanda/redpanda/builds/60369#01944294-7ef3-468e-8fe2-616c23b81086	FAIL	0/1
rptest.tests.maintenance_test.MaintenanceTest.test_maintenance_sticky.use_rpk=True	ducktape	https://buildkite.com/redpanda/redpanda/builds/60369#01944294-7ef1-4efc-9263-0be02b3b0b70	FLAKY	5/6

test results on build#60379

test_id	test_kind	job_url	test_status	passed
rm_stm_tests_rpunit.rm_stm_tests_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/60379#01944325-d6fd-4784-96eb-98185fdd5ebd	FLAKY	1/2
rm_stm_tests_rpunit.rm_stm_tests_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/60379#01944325-d6fd-4c2d-83ad-69cfb60097bb	FLAKY	1/2
rptest.tests.partition_reassignments_test.PartitionReassignmentsTest.test_reassignments_kafka_cli	ducktape	https://buildkite.com/redpanda/redpanda/builds/60379#01944383-4eaa-4edf-9a42-07985b65aaa1	FLAKY	3/6

rockwotj

Can you lay out the next steps from this piece? Here we have the ability to validate and have all the delta information between the two schemas.

I'm assuming we need to:

apply transformations in the data path from old data to new schemas based on the metadata here.
alter tables as needed in iceberg (which there is already some code for right?)

src/v/iceberg/datatypes.h

src/v/iceberg/compatibility_types.h

src/v/iceberg/datatypes.h

src/v/iceberg/tests/compatibility_test.cc

rockwotj · 2025-01-07T15:51:08Z

src/v/iceberg/tests/compatibility_test.cc

+      },
+  },
+  struct_evolution_test_case{
+    .description = "removing a required field works",


I thought we just make these null - this isn't backwards compatible... I guess the Iceberg spec allows this so it's fine?

I thought we just make these null

Ah, that's the other thing we talked about last week. There's a product preference for "clean" removal of columns. I don't feel strongly about it either way, considering 👇

this isn't backwards compatible

I think it is. As a parquet reader, I assume a column whose ID doesn't appear in the schema is just skipped?

Iceberg spec allows this

☝️

As a parquet reader, I assume a column whose ID doesn't appear in the schema is just skipped?

I'm talking if you have a query that is SELECT foo, bar FROM table and you delete column bar then the query fails.

Some automation will list all of the column names in a SELECT statement, but maybe this isn't a big deal because someone will have to explicitly remove said column from source schema and this is a case of we recommend you turn on iceberg compat checks in SR first

I'm talking if you have a query that is SELECT foo, bar FROM table

Interesting. I think we'll need to revisit this with @mattschumpert , as the whole motivation for dropping the column "for real" was something like "you dropped a field from your schema, it's confusing if it still appears there, even if always null".

I don't have a good sense of which invariant is more important from a product perspective. Doesn't seem like a blocker on this PR though.

I think it might be OK if we drop something for real to mirror that. Connector based approaches don't do that AFAIK, but that might be something that can be done and we just recommend enabling the SR compat checks so you don't break queries

Iceberg spec allows this

For my own understanding, can you elaborate on how this behaves with various query engines? Does this make the entire table unusable for certain query engines? Or does this only affect as-of queries? Or is it something that's inconsistent across engines at the moment?

Presumably if the current schema has a required field removed, subsequent queries should correctly not be able to use that field, though I guess that's only unsurprising if we think of "required" as "non-nullable"

From a SQL statement/validation side of things removing columns is only an issue if you explicitly use that column. If you're just doing SELECT * or never referencing it explicitly then it's fine to remove.

Presumably if the current schema has a required field removed, subsequent queries should correctly not be able to use that field, though I guess that's only unsurprising if we think of "required" as "non-nullable"

If you're saying the case of making a column nullable from NOT NULL, then yes nothing "breaks" in terms of the query runs, but predicates/transforms on that column may have unintended side effects, but they should not "break" from my understanding of ANSI SQL. If a query engine has different semantics we can't really do anything about it.

Anyways, I think it's fine to remove columns that are removed from the source schema because we can add compat checks later to SR that would catch these (and likely this would be caught in the existing SR validation).

rockwotj · 2025-01-07T15:53:36Z

src/v/iceberg/tests/compatibility_test.cc

+    // TODO(oren): do we need to detect field reordering? or just support it?
+    // should reordered fields in a schema correspond to some parquet layout
+    // change on disk?
+    .any_change = schema_changed::no,


I think schema registry canonicalization should save us from some of this (we should probably check that we're traversing the protobuf schema correctly), but I don't think we should care about ordering of fields, since parquet is columar it should not matter the order of columns (where in row oriented formats you might be able to stop parsing early if you don't need all the fields?)

schema registry canonicalization should save us

great point

parquet is columnar it should not matter the order of columns

That's my read of the correctness rule - i.e. "it doesn't matter what order fields appear in the schema" vs "you can change the column order by schema evolution".

rockwotj · 2025-01-07T16:07:21Z

Can you lay out the next steps from this piece? Here we have the ability to validate and have all the delta information between the two schemas.

I'm assuming we need to:
* apply transformations in the data path from old data to new schemas based on the metadata here.

* alter tables as needed in iceberg (which there is already some code for right?)

I think the first one is covered by CORE-8530 right?

oleiman · 2025-01-07T16:49:34Z

In my mind, both of these are covered by 8530.

* apply transformations in the data path from old data to new schemas based on the metadata here.

Or something. @andrwng had suggested another approach that I understood as (paraphrasing) "keep writing 'old data' to the original Parquet file with the understanding that it can be read by the new schema". So we need to agree on implementation detail, but generally keeping forward compatible records off the DLQ is the goal.

* alter tables as needed in iceberg (which there is already some code for right?)

Yup, though I think most of the existing code is fit for purpose. There's a bunch of stuff in place already for updating schemas, so we're mostly adding support for "interesting" updates.

rockwotj · 2025-01-07T16:55:33Z

but generally keeping forward compatible records off the DLQ is the goal.

Agreed on the end goal - I'm not going to fight too hard for one or the other, but note that you can have producers writing both, so you might get small files during a translation interval. It does seem simpler tho so I'm happy to go that route if that's true.

oleiman · 2025-01-07T17:05:51Z

but generally keeping forward compatible records off the DLQ is the goal.

Agreed on the end goal - I'm not going to fight too hard for one or the other, but note that you can have producers writing both, so you might get small files during a translation interval. It does seem simpler tho so I'm happy to go that route if that's true.

Yeah no strong opinion on my end either, though my initial instinct was to upcast.

rockwotj · 2025-01-07T17:15:35Z

In general this all LGTM outside the nits and small cleanups around the variant, I just want to make sure I have a good understanding of the current standing of things and the roadmap here. Thanks for this!

oleiman · 2025-01-07T19:05:04Z

force push CR comments (comments & adjusting the shape of evolution_metadata variant)

oleiman · 2025-01-07T19:44:04Z

CI Failure:

build-release I think was infra?

rockwotj · 2025-01-07T19:46:00Z

Yeah I think a bunch of agents got nuked or something 🤷

oleiman · 2025-01-07T19:47:14Z

/ci-repeat 1

andrwng

Nice! Overall this looks pretty solid.

"keep writing 'old data' to the original Parquet file with the understanding that it can be read by the new schema". So we need to agree on implementation detail

Yea the thought here was that it's hopefully uncommon for many schemas to be in use at once, and that with out current data primitives it's probably a good amount of non-trivial code to cast all the different kinds of schema changes to the actual data (especially thinking about field removal, nullability, and such).

My hunch is that the onus would generally be on query engines to reconcile the data schema with the current Iceberg schema. If that is the case, the "write old data" approach seems pretty reasonable. Wondering if either of you have had experience with that yet?

src/v/iceberg/datatypes.h

src/v/iceberg/compatibility_types.h

src/v/iceberg/compatibility_utils.h

andrwng · 2025-01-07T20:28:02Z

src/v/iceberg/datatypes.h

+    using evolution_metadata
+      = std::variant<std::nullopt_t, src_info, is_new, removed>;
+    mutable evolution_metadata meta{std::nullopt};


I think I roughly understand why it is the case, but it's interesting that this field isn't used in the comparison operator of nested_field. Just want to call that out to make sure that's intentional.

It also makes me wonder whether we should do schema evolution on some struct field_and_evo_meta { field, evolution_metadata } rather than on the fields themselves, though I guess that'd also require use to duplicate a ton of code for "evolvable complex types"..

Yeah, it's intentional. The general idea is to distinguish this little annotation field from those which carry Iceberg semantics.

As far as wrapping in an aggregate, I thought about that. I also have a previous iteration in the can that does the traversal and builds up a flat vector of "actions" (w/ field references inside) that can be evaluated post facto.

I think these alternatives mostly work fine (if they require more code), but it's a bit cumbersome to reason about pushing that information down towards table metadata w/o more API shenanigans.

I agree it looks sort of wacky, but my intuition is that sticking this extra couple bytes on each field is the simplest thing.

andrwng · 2025-01-07T20:50:27Z

src/v/iceberg/tests/compatibility_test.cc

+      },
+  },
+  struct_evolution_test_case{
+    .description = "removing a required field works",


Iceberg spec allows this

For my own understanding, can you elaborate on how this behaves with various query engines? Does this make the entire table unusable for certain query engines? Or does this only affect as-of queries? Or is it something that's inconsistent across engines at the moment?

Presumably if the current schema has a required field removed, subsequent queries should correctly not be able to use that field, though I guess that's only unsurprising if we think of "required" as "non-nullable"

src/v/iceberg/tests/compatibility_test.cc

andrwng · 2025-01-07T21:05:21Z

src/v/iceberg/compatibility.h

+schema_transform_result
+annotate_schema_transform(const struct_type& source, const struct_type& dest);


Just a thought, this seems like a thing we could fuzz test

yeah i agree. probably should. noted for a follow up 🙂

andrwng · 2025-01-07T21:07:05Z

src/v/iceberg/tests/compatibility_test.cc

+    return os << tc.description;
+}
+
+static const std::vector<struct_evolution_test_case> valid_cases{


Wondering whether the current code supports multiple updates in the same evolution?

you mean prior to this diff? I think so

I meant with the new compatibility checks -- for instance, I was wondering if it made sense to test an add and remove in the same evolution (unless i missed cases where multiple operations are happening)

oh, yeah definitely. i don't think there's a specific test geared towards that, but the one whose description begins "grouping multiple fields into a struct..." has that effect

src/v/iceberg/tests/BUILD

rockwotj · 2025-01-07T21:41:52Z

Nice! Overall this looks pretty solid.

"keep writing 'old data' to the original Parquet file with the understanding that it can be read by the new schema". So we need to agree on implementation detail

Yea the thought here was that it's hopefully uncommon for many schemas to be in use at once, and that with out current data primitives it's probably a good amount of non-trivial code to cast all the different kinds of schema changes to the actual data (especially thinking about field removal, nullability, and such).

My hunch is that the onus would generally be on query engines to reconcile the data schema with the current Iceberg schema. If that is the case, the "write old data" approach seems pretty reasonable. Wondering if either of you have had experience with that yet?

No I actually agree here that this is the simpler path forward and we should start with this. With compaction and the assumption that schema evolution is not a frequent operation, I think we may not need the more complicated thing.

vbotbuildovich · 2025-01-07T22:34:46Z

Retry command for Build#60369

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/consumer_offsets_consistency_test.py::ConsumerOffsetsConsistencyTest.test_flipping_leadership

variant: - bool is_new - bool removed - struct src_info - id of the source field - requiredness of the source field - optionally source field type (if primitive) Used in subsequent commits to: - cheaply identify struct fields that have changed shape - defer type & requiredness invariant checks to the end Only track primitive types because a) they are cheap to copy and b) structured types are effectively checked during traversal Signed-off-by: Oren Leiman <[email protected]>

Signed-off-by: Oren Leiman <[email protected]>

Templatize the visitor to add a const nested_field* version. Signed-off-by: Oren Leiman <[email protected]>

Some functions to support recursive iteration and transformation of the fields under a field or struct. Signed-off-by: Oren Leiman <[email protected]>

And lightly refactor test suite base class for reuse Signed-off-by: Oren Leiman <[email protected]>

Mix of compatible & incompatible, structural & type oriented Does not include any real tests Signed-off-by: Oren Leiman <[email protected]>

Signed-off-by: Oren Leiman <[email protected]>

Make a copy of the destination schema. Generate a schema transform from source to dest. Apply the transform to the copy and return the result (or an error code) Signed-off-by: Oren Leiman <[email protected]>

Generates increasingly nested structs and measures the time it takes to: - collect all fields of two copies w/ field_collecting_visitor - generate a schema transform from one copy to another - visit a generated schema transform Serves as a decent point of comparison between the limited schema verification already present in the tree and the utilities introduced in this commit stack. Signed-off-by: Oren Leiman <[email protected]>

Signed-off-by: Oren Leiman <[email protected]>

oleiman · 2025-01-07T23:40:04Z

force push contents (CR):

removed an unnecessary errc case
fixed up some doc comments
added a test case

andrwng · 2025-01-07T23:50:48Z

src/v/iceberg/compatibility_utils.h

+  S& s, detail::FieldOp<T*> auto&& fn, detail::FieldPredicate auto&& filter) {
+    static_assert(std::is_const_v<S> == std::is_const_v<T>);
+    chunked_vector<T*> stk;
+    for (const auto& f : std::ranges::reverse_view(s.fields)) {


Interesting that this was reversed between revisions without too much recourse. Curious what the implication is/was?

Nothing functional. Just an accidental omission - I intended to duplicate the control flow of the field collector visitor, though it shouldn't have any effect here.

andrwng · 2025-01-07T23:53:18Z

src/v/iceberg/tests/compatibility_test.cc

+    return os << tc.description;
+}
+
+static const std::vector<struct_evolution_test_case> valid_cases{


I meant with the new compatibility checks -- for instance, I was wondering if it made sense to test an add and remove in the same evolution (unless i missed cases where multiple operations are happening)

oleiman self-assigned this Jan 3, 2025

github-actions bot added area/build area/redpanda labels Jan 3, 2025

oleiman force-pushed the dlib/core-8525/compat-visitor branch 9 times, most recently from eef6479 to 9d607f2 Compare January 6, 2025 23:22

iceberg: Introduce compatibility_types module

4f334fc

Signed-off-by: Oren Leiman <[email protected]>

oleiman force-pushed the dlib/core-8525/compat-visitor branch 3 times, most recently from 0773533 to e0afd1f Compare January 6, 2025 23:41

oleiman marked this pull request as ready for review January 6, 2025 23:41

oleiman force-pushed the dlib/core-8525/compat-visitor branch from e0afd1f to 02b5d2c Compare January 7, 2025 02:23

oleiman changed the title ~~Schema Evolution & Compatability & Whatnot~~ [CORE-8525] - Iceberg schema evolution Jan 7, 2025

oleiman requested review from andrwng and rockwotj January 7, 2025 07:00

rockwotj reviewed Jan 7, 2025

View reviewed changes

oleiman force-pushed the dlib/core-8525/compat-visitor branch from 02b5d2c to 3094cfe Compare January 7, 2025 19:04

oleiman requested a review from rockwotj January 7, 2025 19:24

rockwotj previously approved these changes Jan 7, 2025

View reviewed changes

andrwng reviewed Jan 7, 2025

View reviewed changes

oleiman added 11 commits January 7, 2025 15:38

iceberg: Supporting types for schema evolution

945ced5

Signed-off-by: Oren Leiman <[email protected]>

iceberg: Introduce reverse_const_field_collecting_visitor

fd3ca80

Templatize the visitor to add a const nested_field* version. Signed-off-by: Oren Leiman <[email protected]>

iceberg: Introduce compatibility_utils

f46b4d8

Some functions to support recursive iteration and transformation of the fields under a field or struct. Signed-off-by: Oren Leiman <[email protected]>

iceberg/tests: Some helpers for compatibility_test

a8ffd0b

And lightly refactor test suite base class for reuse Signed-off-by: Oren Leiman <[email protected]>

iceberg/tests: Test cases for struct evolution.

9140b2f

Mix of compatible & incompatible, structural & type oriented Does not include any real tests Signed-off-by: Oren Leiman <[email protected]>

iceberg/compat: Introduce annotate_schema_transform

e993c0c

Signed-off-by: Oren Leiman <[email protected]>

iceberg/compat: Introduce validate_schema_transform

b142f2b

Signed-off-by: Oren Leiman <[email protected]>

iceberg/compat: Introduce evolve_schema

a6cb187

Make a copy of the destination schema. Generate a schema transform from source to dest. Apply the transform to the copy and return the result (or an error code) Signed-off-by: Oren Leiman <[email protected]>

iceberg/compat: Drop unused dependencies & includes

1bf54b7

Signed-off-by: Oren Leiman <[email protected]>

oleiman dismissed rockwotj’s stale review via 1bf54b7 January 7, 2025 23:39

oleiman force-pushed the dlib/core-8525/compat-visitor branch from 3094cfe to 1bf54b7 Compare January 7, 2025 23:39

oleiman requested review from andrwng and rockwotj January 7, 2025 23:41

andrwng approved these changes Jan 7, 2025

View reviewed changes

oleiman merged commit cb5efec into redpanda-data:dev Jan 8, 2025
17 checks passed

		schema_transform_result
		annotate_schema_transform(const struct_type& source, const struct_type& dest);

[CORE-8525] - Iceberg schema evolution #24676

[CORE-8525] - Iceberg schema evolution #24676

Conversation

oleiman commented Jan 3, 2025 • edited Loading

Microbenchmark

Conclusions

Backports Required

Release Notes

oleiman commented Jan 7, 2025

oleiman commented Jan 7, 2025

oleiman commented Jan 7, 2025

vbotbuildovich commented Jan 7, 2025 • edited Loading

CI test results

rockwotj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rockwotj commented Jan 7, 2025

oleiman commented Jan 7, 2025

rockwotj commented Jan 7, 2025

oleiman commented Jan 7, 2025

rockwotj commented Jan 7, 2025

oleiman commented Jan 7, 2025

oleiman commented Jan 7, 2025

rockwotj commented Jan 7, 2025

oleiman commented Jan 7, 2025

andrwng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rockwotj commented Jan 7, 2025

vbotbuildovich commented Jan 7, 2025

Retry command for Build#60369

oleiman commented Jan 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleiman commented Jan 3, 2025 •

edited

Loading

vbotbuildovich commented Jan 7, 2025 •

edited

Loading