Notifications for lineage change

[abhineet13]

Background [Optional]

Hi, we would like to be alerted for any changes to the lineage of an ETL job like changes in data source path/attributes or transformations.

Question

Is there a way to get alerts/notifications for changes to a Lineage?

[wajda]

Thanks for the suggestion.
In theory, yes. But in practice it needs more precise definition of what is considered a change in lineage. In Spline, lineage is represented by a combination of execution plans and execution events. Currently, any minor logically irrelevant change in the execution plan (e.g. an extra user metadata, or a timestamp somewhere) would lead to creating another instance of the execution plan in the Spline database. Spline server doesn't make any efforts to compare them node by node as it cannot decide with certain what difference is considered meaningful and what is not. The answer heavily depends on the context (what data framework you are using, of what version, any other extensions, what is semantics of every operator and its parameters etc.) and the use-case (e.g. a change in the username can be considered important or irrelevant in different situations). So depending on the use-case it could be easy or tricky to implement.

We have similar feature requests for other use-cases, like #735 or #123
In the future versions we'll add an admin API through which it will possible to configure alerts on certain conditions. This is not a priority for us however at the moment, so don't expect it to be done in the next few months at least.

[Mjlgh]

abhineet13,

We're interested in this feature as well. We recognize there's a fair amount of complexity in this as it requires some choices to be made about what constitutes a change. Today we're writing Spline's generated lineage to our enterprise data catalog and plan to create a hash as part of the identifier for that lineage on the attribute paths created, ignoring literals and anything else that might be nondeterministic.

[abhineet13]

Thanks @wajda and @Mjlgh for your response.