Lineage is not captured

[wajda]

My Spark job runs OK, the log output says Lineage tracking is ENABLED, but I don't see any lineage generated,

This is one of the most frequently asked question about Spline Spark Agent. See below for the answer.

[wajda]

Spline agent listens to events generated by Spark when a Spark job is executed. The type of events and metadata they carry varies depending on what Spark action is called; whether you are using SQL, DataFrame or RDD API; the type of a data source you read form or write to and so on. Spline agent supports many of them, but not all of them, as there are potentially infinite number of different data sources, formats and technologies Spark can integrate with via 3rd-party extensions. Sometimes, it might happen that Spline agent receives an event that it doesn't know how to process, or intentionally ignores it as it "thinks" there is no useful (from the lineage tracking perspective) information associated with it. In such cases the lineage might not be captured as expected. If it's a right or wrong behavior depends on a particular situation.

So, if the Spline agent initialized correctly, your job completed without errors, but the lineage or one of its parts is missing, I recommend the following troubleshooting steps:

1. Make sure you save the data into a persistent source. Spline only captures lineage from the persistent writes. Memory only actions like collect() or show() are currently ignored (see Optionally capture memory only actions like collect(), count() etc #361). Likewise, if you are using save mode Ignore, or the Spline agent detects an empty result that was not persisted, the lineage would not be captured from such job run.
2. Make sure you're using DataFrame and not RDD API. RDD transformations aren't automatically tracked and the captured lineage could be incomplete (see RDD lineage support #33).
3. Make sure the Spline agent is properly initialized, and there is no associated errors in logs. Look for the phrase "Spark Lineage tracking is ENABLED".
4. Enable DEBUG log level and check if the logs contain any of the following:
   • "was not recognized as a write-command"
   • "Ignored write detected"

The support for unrecognized write-commands can be added via a custom plugin. Knowing the internal structure of a command class is crucial. If it comes form a closed-source library or a Spark distribution Spline can try to inspect it reflectively and damp into the logs. Enable TRACE log level and Spline will automatically reflect the unrecognized command classes. Find the phrase "OBJECT DUMP BEGIN" in the logs, create an issue here on GitHub and paste the relevant text block there.