Iceberg MERGE, UPDATE and DELETE operations support

[alberto-lucini]

Hi,
I'm testing spline-spark-agent (v. 1.0.2) with Iceberg tables. As I can see from this release note it should be supported since v. 0.7.8.

If I try standard operations, like appending data, lineage events are produced as expected; however, if I try with MERGE, UPDATE and DELETE operations, no event is produced.

Example of query that works as expected:

insert into test1.tab_ic_31 select * from test1.tab_ic_32;

Example of query without lineage events:

delete from test1.tab_ic_31 where i = 1;

update test1.tab_ic_31 set t = 'e' where i = 2;

merge into test1.tab_ic_31 t 
   using test1.tab_ic_32 u on t.i = u.i 
   when matched then update set t.t = u.t 
   when not matched then insert *;

The tables are defined in this way:

create table test1.tab_ic_31 (i int, t string) using iceberg;
create table test1.tab_ic_32 (i int, t string) using iceberg;

I'm working on AWS EMR 6.9.0, with spark 3.3.0 and iceberg 0.14.1.

I can see that delta tables have been introduced in v. 0.7.11 and merge, delete, update support for them in v 1.0.0.

Is there a way to capture MERGE, DELETE and UPDATE lineage also for iceberg tables? Or if not available, is there a plan to add it in a future version?

Thank you,
Alberto

[cerveada]

This is not supported by spline yet. I created an issue for it #594

[alberto-lucini]

Thank you Adam; I will follow the created issue for updates