Iceberg Tables

[josephniblo]

Hi, Iceberg Tables are now in public preview and would make a really valuable addition to SnowDDL. Is there any plan to add support?

[littleK0i]

(edited 18 Dec 2024)

SnowDDL has a limited support for unmanaged Iceberg tables starting with 0.38.0.

Link to guide: https://docs.snowddl.com/guides/other-guides/iceberg-tables

It should cover common use cases when using Iceberg tables with external catalogs.

[leopasta-enable]

Hi littleK0i, wonder what are your current plans about adding support for Iceberg tables?
It seems snowflake is doubling down on Iceberg as a long term strategy, and now it is generally available.

For us, we currently use as an integration mechanism, reading data produced by databricks. So they function very much like an EXTERNAL TABLE for us now.
We're creating the iceberg tables outside snowddl into their own "sandbox" schema, it would be great to have it managed in SnowDDL when the support comes.

A smaller change, that would help us would be to be able to have ICEBERG_TABLES as a permission model object type, this way we could start by having snowDDL managing the schema permissions, instead of doing it in external scripts.
What do you think?

[littleK0i]

Added ICEBERG_TABLE object type in 0.34.3, just to make it available for grants and permission models.

I'll look into possibility to implement actual resolver in the next month or two. The main problem is requirement to set up external storage and all these additional catalog types. Guess we need a separate AWS account and separate testing infrastructure to make it work.

Still not sure if Iceberg brings real value for ~99% of businesses with data volumes up to 1Pb. For very big guys ... maybe, but they probably did their homework and have it in place already, along with custom automation.

[leopasta-enable]

Awesome, thanks for that!

In terms of value, I guess the main thing we would be looking for from iceberg is simplifying integration: with the iceberg layer, we can move logic from databricks to snowflake more gradually (and who knows if in future we want to move systems out of snowflake to other stacks :) )

[littleK0i]

@leopasta-enable , please note.

SnowDDL now supports unmanaged Iceberg tables starting from version 0.38.0.

Guide: https://docs.snowddl.com/guides/other-guides/iceberg-tables

Normally it should not affect anything. Iceberg tables start to resolve only if at least one relevant config is present. But I suggest to pay attention in case of updates, just in case.

[leopasta-enable]

Awesome Christmas gift, thanks littleK0i and merry Christmas! :)