Kubernetes Envoy Example

A sample application using Envoy running in Kubernetes.

Motivation

I wanted to learn more about Envoy, so I decided to do it "the hard way." I wrote the contrived example application and pieced together the Envoy configurations from the documentation and examples.

Sample Application Summary

The sample application is fairly straight forward. It has four components:

• frontend - HTTP service written in Go. It accepts the request from the client and makes calls to the other services.
• user - gRPC service written in Go. Manages user information.
• order - gRPC service written in Go. Tracks orders. Makes calls to the item service for details about each ordered item.
• item - gRPC service written in PHP (more on this below). Inventory of items.

The basic flow of data through the service is

Services do not communicate directly with one another. All communication is via Envoy. An Envoy instance - named ingress - acts as the entrypoint into the cluster. This is similar to the "Service to service plus front proxy" example.

The network flow of data is

Services Detail

Each instance of each service runs in a Kubernetes pod. Each pod has multiple containers:

• the application itself
• Envoy proxy
• statsd_exporter - Envoy emits metrics using statsd. statsd_exporter is a statsd server that exposes these metrics fo consumption by Prometheus.

The ingress contains only Envoy and statsd exporter.

The Envoy container in each pod listens on one port (8080) for incoming requests and routes these to the local application running within the pod. Envoy listens on a different port (7070) for outbound requests. Applications make requests to http://127.0.0.1:7070 for other services rather than connecting direct. See order.json for example.

All the applications expose metrics for Prometheus.

The frontend is the user facing service. It accepts HTTP/1.1 requests on the front and makes gRPC requests to the other services.

The item service uses grpc-fastcgi-proxy to expose a simple PHP applciation. This service could represent a legacy system written in PHP that we wish to expose.

The other services - user and order - are simple gRPC services written in Go.

Tracing

Jaeger is used for tracing. Envoy is configured to send tracing information to the Zipkin compatible endpoint of Jaeger.

The applications do not send any data to Jaeger. They only ensure the tracing headers are propogated.

Service Discovery

Envoy handles all service discovery - the applications just contact Envoy on lcoal host. Kubernetes headless services are used. This means a DNS request for the service will return a record for each running Pod. This is a simple service discovery mechanism that does not require additional helper services.

Usage

This example is designed to be ran on minikube. A working Go environment is required. Once minikube is installed and running, one should clone this repository into the GOPATH:

mkdir -p $HOME/go/src/github.com/bakins
git clone https://github.com/bakins/kubernetes-envoy-example.git
cd kubernetes-envoy-example

Now to build and deploy the example applications:

• ./script/build will build the applications and push Docker images into the minikube Docker environment.
• ./script/deploy will deploy the Kubernetes manifests.

After a few seconds, you should be able to access the application by running minikube service ingress - you should see a simple json body.

To access jaeger, run minikube service jaeger-query. You may need to reload in the browser a few times.

Prometheus is available via minikube service prometheus. You should be able to run adhoc queries for metrics.

TODO: grafana dashboards

Prometheus Queries

histogram_quantile(0.95, sum(rate(envoy_upstream_rq_time_bucket{cluster!="local_service"}[5m])) by (le,app, cluster))

sum(rate(envoy_upstream_rq_time_count{cluster!="local_service"}[5m])) by (app, cluster)

sum(envoy_health_check_healthy) by (app,cluster)

sum(envoy_upstream_cx_total) by (app,cluster)

LICENSE

See LICENSE