Column-level lineage access coverage

[Karsonnel]

Hi，I'm a beginner of spark and I‘m trying to extract my column level lineage using  ‘sparkPlan’ in QueryExecution。After reading ‘Spark features coverage’ in this project's README.md, one question confused me:

• why some commands have yet to be implemented, is there any problem to do that? Because I am trying to extract my column level lineage of spark sql, i want to know if there has some problem that i can't get all commands column level lineage.

Thanks.

[wajda]

Time / man power, or a lack of it to be precise. We implement features that are most demanded first, and save some time by leaving the rest of them behind. We always welcome and appreciate any contribution though :)

[Karsonnel]

thanks for the reply. Is there a situation that we can't extract column level lineage of spark sql? If we always can extract lineage information by the way which spline-agent uses, i'm very happy to make some contribution.

[Karsonnel]

Is there a situation that we can't extract column level lineage of spark sql?
I'm trying to find this situation by debugging spline-agent using some test-sqls，but still not find a single one now. so i suppose to ask you to get some guides.

[wajda]

I would say that the agent currently supports the majority of real-world use-cases. As the non-implemented commands list suggests the situations where the lineage would not be captured is mostly limited to DDL kind of operations, like drop, truncate the table, or rename table or a column in place etc. But Read-Transform-Write scenarios (which is a primary ETL use-case) is covered pretty well.

[Karsonnel]

your answer resolve my question perfectly, I'm very thankful honestly. Thank you!