Send UnsupportedSparkCommandException and others to Sentry? (Plugin Maybe?)

[diegoeche]

First, thanks for answering all my questions. Our metadata filter is working like a charm

I'm now doing some exploratory work trying to capture exceptions from translations. From reading the code it seems that some commands might fail but not throw: UnsupportedSparkCommandException (this is only if the command is in the list of unsupported commands)

I thought I could write a plugin to wrap the translation execution and capture exceptions to send them to Sentry.io. But from the looks of it, it isn't possible. I'm thinking maybe the dispatcher would be the way to go to do this.

In our setup, reading/aggregating the logs (within a bunch of different teams) would be hard to do.

[wajda]

If I understand the use-case correctly it can be done on other tiers outside Spline agent, e.g.:

1. Logging
   From a quick look Sentry.io seems to provide a JUL bridge. On the other hand Spline agent uses Spark internal logging which in turn uses Log4j over SLF4J. There is also SLF4J-to-JUL adapter. I didn't try it myself but if I got this puzzle right, you could have a logging chain like this in the end with no source code change:

Spline Agent -> Spark Logging -> SLF4J -> JUL -> Sentry.io

Direct SLF4J-to-Sentry or Log4J-to-Sentry bridges might also exist, it might be worth checking.

2. Operating system
   You could use sentry-cli (together with a couple custom bash scripts maybe) to pipe Spark or Spline log files to Sentry.

This is just what jumped into my mind right away. I'm sure there might be more elegant solutions to the problem. But all in all, I'm not convinced that implementing any specific error reporting mechanism (even via plugins) directly in the Agent core is worth it. Doing it on other levels would probably be more practical anyway. Does it make sense?

[diegoeche]

Hey @wajda, thanks for the answer!

So, a bit more context. We are in Databricks and we (afaik) have different clusters/teams(even accounts) we are trying to support to run Spline. That's why we would love if we could just give them the jar, they initialize it in the notebook, we get metadata, and we get notified of exceptions (including command translation errors).

I gotta read on what tools databricks provides for log-forwarding. The whole env is a black box, so the tools are a bit limited. Maybe gotta RTFM :)

I'll tinker with maybe adding a handler within the spline-agent-initialization and maybe with a JUL filter I can get a stream of ERROR.level logging associated to spline.

Can't thank you enough, will update!

[wajda]

I see your point. Yes, a JUL bridge - that's what I would start with in that case. The JAR can be added into the agent bundle and if any extra programmatic initialization is needed it can be done, say in the custom lineage dispatcher constructor.

I think you're right, some sort of agent life cycle hooks would be handy. I'll think about it.