Unreasonable increase in latency between localhost and remote

[yourarj]

I'm benchmarking my web server code for JSON deserialization against Golang. Code can be found at https://github.com/yourarj/rust-vs-go-performance-benchmark

We benchmarked the server response in TWO setups

1. localhost (server and K6 on same server)
2. remote (server and K6 are on independent machines)

Note : for both go and rust same setup has been used for localhost and remote cases.

Expected Behavior

When we move our K6 from localhost setup to remote setup go and rust should see the same network overhead.

Current Behavior

E.g. If we consider http_req_waiting for go the average request overhead is 151.3678 but in case of actix-web overhead is whopping 471.5906.

Possible Solution

Tried to play with worker count, backlog values, connection timeouts. It didn't workout. So no clear idea.

Steps to Reproduce (for bugs)

1. checkout code from https://github.com/yourarj/rust-vs-go-performance-benchmark
2. build docker images for rust and go from respective folder
3. create deployments in K8S with deployment and service file in k8s folder
4. benchmark with K6 with given scirpt.js file in test folder (k6 run script.js)

Context

We are working on adoption of rust for web on massive scale. If we can mitigate this issue that would be great selling point for rust.
Why I see this as huge potential is if Rust can beat go on localhost by 4x (8k rps vs 45k rps) if we can replicate same over network, It would mean heavens for dev and product owner in terms of performance and cost.

Your Environment

Testing environment for both.

MicroK8S Kubernetes cluster.
8cpu, 16gb, single pod exposed via node port.

Rust : version: 1.80.1, actix-web: 4.9, serde 1.0
Go: version 1.23

Load testing: Grafana K6 (vu : 10,000 (test/script.js)

[yourarj]

[UPDATE]
In localhost setup rust gets 0% failed requests and almost 90-93% cpu consumption. Memory is ample 15g free in both cases of rust.

But in remote client setup cpu consumption don't go beyond 70% and roughly 0.5 - 1% request fail with any of following reason. Request timeout, dial timeout, EOF or connection reset.