Performance implications/numbers of Skywalking Agents in production

[andrewgkew]

@wu-sheng - questions that came out of the conference I spoke at around Skywalking:

1. Do you have any numbers around the performance hit of running SkyWalking in production?
2. In an example of using the Java Agent its not a sidecar but it part of starting up your JVM what exactly is the performance hit?
3. I guess if you were running an agent as a sidecar it will have less of or no impact?
4. Whats the impact of running a profile in production with SkyWalking?
5. Running a profile is forward looking, has there been any thought about auto backward looking to show rather than an issue with a service, but maybe more info around where in the service there is an issue? I am told AppDynamics does something like this.

[wu-sheng]

Do you have any numbers around the performance hit of running SkyWalking in production?

I assume you are asking about Java agent from the whole context. We set up the benchmark and run the testing in very aggressive scenarios. It works well in all cases.
https://skyapmtest.github.io/Agent-Benchmarks/

It is a long history, we don't have concern about the performance anymore. Actually, the major impact is whether users are going to activate some optional features, such as tagging the value of HTTP parameters, collecting SQL parameters. Serialization and transport this huge size of data

In an example of using the Java Agent its not a sidecar but it part of starting up your JVM what exactly is the performance hit?

What do you expect sidecar to do? Auto-instrumentation is injecting codes, intercepting invocation and collecting information. From this point, you have to be inside JVM.
If you just want sidecar monitoring, no Java agent required.
We have worked with Istio/Envoy communities for years. We could use Envoy Access Logs to get the whole topology and metrics around services. As the same, you only have metrics observed from outside.

I guess if you were running an agent as a sidecar it will have less of or no impact?

Of course, collecting less information means less impact. Such as collecting parameters of method is impossible from outside sidecar.

Whats the impact of running a profile in production with SkyWalking?

We don't have documents for this, but as we have set up a very limited concurrency threshold for agent, the impact is very small. And we always recommend users to set up minimal duration as pre-condition for a profiling task.

Running a profile is forward looking, has there been any thought about auto backward looking to show rather than an issue with a service, but maybe more info around where in the service there is an issue? I am told AppDynamics does something like this.

This is possible from tech perspective. Typically, we call that auto-profiling. Once the agent detected a slow trace segment, the profiling begins running w/o any user's command.
This sometimes is useful, sometimes is costing many unnecessary resources.
Such as how to define slow, different systems and different URI(HTTP)s have different definitions. Only SRE/Ops team(sometimes only Dev team) know which service's latency is lower than expectation.
Auto-profiling will require many parameter settings to avoid the above side effect.
As an open source project, we welcome some one joining us to say contributing this feature, we could guide. This is how we work as always. Like the current profiling, it was contributed by Didi's SRE team, even we knew this would be useful from 5 years. :)

[andrewgkew]

Thanks @wu-sheng this is really helpful

Totally understand between the agent (inside JVM) and the sidecar and the limited info you can acquire, I now understand why when I setup Istio and linked that to SW the data that came back was a lot less than from the actual agent.

That benchmark blog is really good and has some great info and numbers there which I think should answer most peoples concerns around using the SW agents - great job

Agree on the open source project side and if I get time I might look and see if we can contribute this auto profiling at some stage. I think it would be a great addition to the project.

Thanks again for all your help and support, you are always so responsive and helpful.