How to attach extra metadata to the data lineage using a code-less configuration?

[muaragon]

Version

spark: [email protected]
spline-agent:za.co.absa.spline.agent.spark:spark-3.1-spline-agent-bundle_2.12:0.6.1

Background

After following the steps in the following blog https://www.capitalone.com/tech/software-engineering/spline-spark-data-lineage/, I've been able to extract data lineage from spark and upload it to the spline server. The next thing I want to do is inject metadata to the lineage using a code-less configuration. Based on the spline agent's default properties https://github.com/AbsaOSS/spline-spark-agent/blob/develop/core/src/main/resources/spline.default.properties, I tried to configured spark.spline.postProcessingFilter.userExtraMeta.rules but I have not been successful.

Question

What is the correct approach to configure spark.spline.postProcessingFilter.userExtraMeta.rule using a code-less configuration?

Setup

  spark-submit \
  --packages za.co.absa.spline.agent.spark:spark-3.1-spline-agent-bundle_2.12:0.6.1 \
  --conf spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener \
  --conf spark.spline.producer.url=http://localhost:8080/producer \
  --conf spark.spline.lineageDispatcher=console \
  --conf spark.spline.postProcessingFilter.userExtraMeta.rules"={\"executionPlan\":{\"foo\":{\"$js\":\"executionPlan.name()\"}\,\"bar\":{\"$env\":\"JAVA_HOME\"}\,\"baz\":{\"$jvm\":\"some.jvm.prop\"}\,\"qux\":42\,\"tags\":[\"aaa\"\,\"bbb\"\,\"ccc\"]}}" \
  my_spark_job.py

Output [Extra metadata has not been added]

:: loading settings :: url = jar:file:/usr/local/Cellar/apache-spark/3.1.2/libexec/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
+---+-----------------+
|id |nickname         |
+---+-----------------+
|1  |blue alaska      |
|2  |red colorado     |
|3  |orange arizona   |
|4  |yellow california|
|5  |purple oregon    |
+---+-----------------+

["plan",{"id":"8b8a2d06-74fb-4234-9b0c-d1cae9521fee","name":"MySparkJob","operations":{"write":{"outputSource":"file:/Users/muniker/Desktop/spline-sandbox/my-spark-job/data/output/nicknames","append":false,"id":"0","name":"InsertIntoHadoopFsRelationCommand","childIds":["1"],"params":{"path":"data/output/nicknames"},"extra":{"destinationType":"csv"}},"reads":[{"inputSources":["file:/Users/muniker/Desktop/spline-sandbox/my-spark-job/data/input/user_favorites.csv"],"id":"4","name":"LogicalRelation","output":["16","17","18"],"params":{"header":"true","inferschema":"true","path":"data/input/user_favorites.csv"},"extra":{"sourceType":"csv"}},{"inputSources":["file:/Users/muniker/Desktop/spline-sandbox/my-spark-job/data/input/locations.csv"],"id":"6","name":"LogicalRelation","output":["38","39","40"],"params":{"header":"true","inferschema":"true","path":"data/input/locations.csv"},"extra":{"sourceType":"csv"}}],"other":[{"id":"3","name":"SubqueryAlias","childIds":["4"],"output":["16","17","18"],"params":{"identifier":"user_favorites"}},{"id":"5","name":"SubqueryAlias","childIds":["6"],"output":["38","39","40"],"params":{"identifier":"locations"}},{"id":"2","name":"Join","childIds":["3","5"],"output":["16","17","18","38","39","40"],"params":{"condition":{"__exprId":"28bb4bf8-f897-420d-af53-ca976e04fa28"},"hint":"","joinType":"INNER"}},{"id":"1","name":"Project","childIds":["2"],"output":["16","44"],"params":{"projectList":[{"__attrId":"16"},{"__exprId":"c17dc983-ed33-4472-83c4-c8b3de9e0e84"}]}}]},"attributes":[{"id":"16","dataType":"c665be52-e037-46f6-b132-77ded848883b","name":"id"},{"id":"17","dataType":"b485f280-2dfd-489b-afd5-9ed7d2fa64e6","name":"favorite_color"},{"id":"18","dataType":"b485f280-2dfd-489b-afd5-9ed7d2fa64e6","name":"favorite_city"},{"id":"38","dataType":"c665be52-e037-46f6-b132-77ded848883b","name":"id"},{"id":"39","dataType":"b485f280-2dfd-489b-afd5-9ed7d2fa64e6","name":"city"},{"id":"40","dataType":"b485f280-2dfd-489b-afd5-9ed7d2fa64e6","name":"state"},{"id":"44","dataType":"b485f280-2dfd-489b-afd5-9ed7d2fa64e6","childRefs":[{"__exprId":"c17dc983-ed33-4472-83c4-c8b3de9e0e84"}],"name":"nickname"}],"expressions":{"functions":[{"id":"28bb4bf8-f897-420d-af53-ca976e04fa28","dataType":"7d49f844-9813-4aa7-a12d-13401bae49ab","childRefs":[{"__attrId":"18"},{"__attrId":"39"}],"extra":{"simpleClassName":"EqualTo","_typeHint":"expr.Binary","symbol":"="},"name":"equalto"},{"id":"5630d4e4-c566-47c6-b952-94ac8c55a5c3","dataType":"b485f280-2dfd-489b-afd5-9ed7d2fa64e6","childRefs":[{"__attrId":"17"},{"__exprId":"47180daa-3076-444b-be82-85826b205273"},{"__attrId":"40"}],"extra":{"simpleClassName":"Concat","_typeHint":"expr.Generic"},"name":"concat"},{"id":"c17dc983-ed33-4472-83c4-c8b3de9e0e84","dataType":"b485f280-2dfd-489b-afd5-9ed7d2fa64e6","childRefs":[{"__exprId":"5630d4e4-c566-47c6-b952-94ac8c55a5c3"}],"extra":{"simpleClassName":"Alias","_typeHint":"expr.Alias"},"name":"nickname","params":{"name":"nickname","explicitMetadata":"{}","exprId":"ExprId(44,bb926c07-b67b-4df3-901e-53655d3043fb)"}}],"constants":[{"id":"47180daa-3076-444b-be82-85826b205273","dataType":"0d4948b9-70a5-48f6-ab45-52ece599aaf8","extra":{"simpleClassName":"Literal","_typeHint":"expr.Literal"},"value":" "}]},"systemInfo":{"name":"spark","version":"3.1.2"},"agentInfo":{"name":"spline","version":"0.6.1"},"extraInfo":{"appName":"MySparkJob","dataTypes":[{"_typeHint":"dt.Simple","id":"c665be52-e037-46f6-b132-77ded848883b","name":"integer","nullable":true},{"_typeHint":"dt.Simple","id":"b485f280-2dfd-489b-afd5-9ed7d2fa64e6","name":"string","nullable":true},{"_typeHint":"dt.Simple","id":"7d49f844-9813-4aa7-a12d-13401bae49ab","name":"boolean","nullable":true},{"_typeHint":"dt.Simple","id":"0d4948b9-70a5-48f6-ab45-52ece599aaf8","name":"string","nullable":false}]}}]
["event",{"planId":"8b8a2d06-74fb-4234-9b0c-d1cae9521fee","timestamp":1645118580263,"durationNs":1048116874,"extra":{"appId":"local-1645118568882","readMetrics":{"numOutputRows":10,"numFiles":2,"metadataTime":0,"filesSize":222},"writeMetrics":{"numFiles":1,"numOutputBytes":82,"numOutputRows":5,"numParts":0}}}]

[wajda]

The filter needs to be enabled in order to take effect.

--conf spark.spline.postProcessingFilter=userExtraMeta

or, if you want to enable other filters at the same time

--conf spark.spline.postProcessingFilter=composite
--conf spark.spline.postProcessingFilter.composite.filters=userExtraMeta,myOtherFilter1,myOtherFilter2,etc