Advanced Kubernetes YAML configurations & templates, based on my experiences running Kubernetes in production at different companies.
The top-level directory contains standard Kubernetes object templates with many Best Practices, Tips & Tricks learned over time across production environments.
The sub-directories contain ready-to-run real world apps that I've run across environments.
Start with deployment.yaml / statefulset.yaml, for advanced users see kustomization.yaml.
The service.yaml and ingress.yaml configs contain settings for using static public IP addresses and locking down your cloud load balancer's firewall rules eg. to private IP addresses, and patches for Cloudflare Proxied or VPN IPs. You may need to extend those IP lists to your office / VPN / public addresses if really want to permit direct internet access to your ingresses and aren't proxying them through a WAF in proxied mode etc.
See also the Dockerfile template in the HariSekhon/Templates repo to containerize your custom apps for deploying on to Kubernetes.
Real-world app deployments are found in the more specific <app>/ directories.
These follow the standard Kustomize <app>/base/ and <app>/overlay/ layout to make it easy to use as-is by just tweaking a couple settings in the overlay to your specific environment.
Advanced auto-scaling production-grade CI/CD on Kubernetes:
- ArgoCD - deployment, configs and optimizations. Start here: argocd/base/kustomization.yaml
- Jenkins - jenkins server and dynamically scaling agents on kubernetes. Start here: jenkins/base/kustomization.yaml
- see also: Jenkins repo with advanced Jenkinsfile & Jenkins Shared Library
- TeamCity - teamcity server and dynamically scaling agents on kubernetes. Start here: teamcity/base/kustomization.yaml
- Selenium Grid - simple and distributed auto-scaling deployments. Start here: selenium-grid/base/kustomization.yaml / selenium-grid-distributed/base/kustomization.yaml
Helm repos you should probably have installed:
| Repo Label | URL |
|---|---|
| stable | https://charts.helm.sh/stable |
| bitnami | https://charts.bitnami.com/bitnami |
| fairwinds-stable | https://charts.fairwinds.com/stable |
Table generated from helm-repos.txt by script generate_repos_markdown_table.sh
These repos can quickly installed in one command using script install_repos.sh:
./install_repos.shSee kustomization.yaml for 2 methods provided:
- template the Helm chart using a
values.yamlto Git and serve from there (see DevOps Bash Tools for thehelm_template.shconvenience script) - dynamically load the Helm chart from upstream with a
values.yaml
...then patch override anything the chart doesn't directly support using the standard Kustomize patching examples given in the kustomization.yaml.
This section has been moved and enhanced in my amazing Knowledge-Base repo:
Kubernetes Production Ready Checklist
The best documentation links are provided at the top of each yaml for fast referencing (my advanced .vimrc can open these URLs from the current file via a hotkey!)
See also HariSekhon/Knowledge-Base, especially:
Datree Kubernetes ArgoCD best practices
.envrc - use with direnv to auto-load correct Kubernetes context isolated to current shell to avoid race conditions between shells and scripts caused by naively changing the global ~/.kube/config context
Shortcut symlinks are for faster instantiation from these configs using the standard kubernetes shortcuts such as new pvc.yaml - see the Templates repo for more details on the new command to fast create new files from templates.
For more amazing diagrams see HariSekhon/Diagrams-as-Code
- GitHub repo: HariSekhon/HAProxy-configs
with MetalLB:
Is it just me or do MetaLB think they're Starfleet? (compare their logos)
A Traefik deployment I did for a client using:
alternative diagram:
A Kong API Gateway deployment I did for a client using:
A production Jenkins on Kubernetes I built for a client with auto-spawning agents for horizontal scaling and integration with Docker, SonarQube, Clair, Grype and Trivy for code & container scanning.
- jenkins/base/*.yaml - core config that is inherited by all environments
- jenkins/overlay/*.yaml - environment specific settings like ingress address, NFS server mount on agents
- claire/base/*.yaml
- sonarqube/base/*.yaml
- trivy/base/*.yaml
- GitHub repo: HariSekhon/Jenkins
- Advanced Jenkinsfile
- Groovy Shared Library with the code & container scanning functions
- clair.groovy
- grype.groovy
- trivy.groovy, trivyFS.groovy, trivyImages
- gcrDockerAuth.groovy, garDockerAuth.groovy
- among others in vars/ and don't forget about the epic Jenkinsfile
HariSekhon/Knowledge-Base - Jenkins
HariSekhon/Knowledge-Base - Jenkins-on-Kubernetes
screenshot:
A high scale production OpenTSDB replatform I did to Kubernetes for a client, ingesting 9 billion data points per day and serving 3 million queries per day.
I also had to do advanced performance tuning of their production HBase cluster which was suffering from frequent outages at this scale due to being set up by a non-SME on the wrong hardware (I had to make do with the existing hardware of course).
This was the second client I did in-depth performance tuning of HBase for - I've published a selection of useful HBase tools - see hbase_*.py and opentsdb_*.py in my DevOps Python tools repo.
Forked from the Templates repo.
The rest of my original source repos are here.
Pre-built Docker images are available on my DockerHub.