API, Core: Support default values in UpdateSchema

[rdblue]

This adds default value support to UpdateSchema and the implementation of it in core.

The API has new methods that accept a default value:

• addColumn(String name, Type type, String doc, Object defaultValue)
• addRequiredColumn(String name, Type type, String doc, Object defaultValue)
• updateColumnDefault(String name, Object defaultValue)
• updateColumn(String name, Type type, String doc, Object defaultValue)

The new methods always require supplying a doc string, which can be null. However, callers should not need to know to pass null when there is no doc string. Because the type of a default value is Object, adding additional method signatures that omit the doc string would conflict (for instance addColumn("a", StringType.get(), "default or description?")).

To make the API easier to use without adding conflicting signatures or new method names, this updates the implementation to allow combining addRequiredColumn with updateColumnDefault. Previously, modifications to added columns were not supported, but adding support makes the API behave like a builder:

table.updateSchema()
    .addRequiredColumn("a", StringType.get())
    .updateColumnDefault("a", "unknown")
    .commit();

The API allows setting a field's default and does not expose the internal concepts of initial default and write default. When a column is added, its initial default can be set. Subsequent updates will only modify the write default. The only exception to this is that updateColumnDefault will change the initial default for a newly added required column if it is not set to enable the builder-like pattern.

[rdblue]

These changes enable the builder to handle all updates, even to whether the field is optional and the field name. That way the builder is always used to copy instead of needing to support specific fields when changing the name.

[rdblue]

Added fields are now tracked in updates so that the added fields can be updated in the same set of schema changes. Now this multi-map tracks the fields added to a parent struct by ID rather than keeping a list of unchangeable fields.

[rdblue]

Moved into the interface as a default implementation.

[rdblue]

Moved into the interface as a default implementation.

[rdblue]

Now, updates always use the builder to preserve existing metadata.

[rdblue]

A multi-map returns an empty list when the ID is not found.

[rdblue]

To preserve both the initial and write defaults, this creates the field with the initial default (which sets both) and then updates the field, which sets only the write default because the initial default cannot change once it is set.

[rdblue]

I think it's better to remove these assertions than to update them. These cases are covered by core tests for schema updates, which is the canonical place to test behavior. Flink and Spark tests should validate the integration (that the ADD COLUMN is passed) rather than the behavior (that NOT NULL is rejected).

[Fokko]

Adding a required column, without a default value, should affect writers. Because the field is missing the write operation will be rejected.

[rdblue]

It will cause writes without the required column to fail (as would adding an optional column), but it is a compatible change because you always know that the data can be read correctly. The incompatibility is when the NOT NULL constraint is violated because existing data has no value.

[Fokko]

LGTM

Now I need to do this on the Python side as well 👍

[aokolnychyi]

Have we ever considered accepting NestedField in methods that add columns and relying on its builder?

[rdblue]

Yes, but I don't think it is a good idea. This API is user-facing and we don't want users to need to understand the distinction between initial default and write default. This API is supposed to match SQL capabilities and is higher level.

[danielcweeks]

+1 (pending checks) Literal is a huge improvement over passing an opaque object around.

[rdblue]

Merged. Thanks for reviewing, @Fokko, @danielcweeks, and @aokolnychyi!