With AKS, you can use Azure Monitor Workspace for Prometheus
and Azure Managed Grafana
to collect, query and visualize the metrics from AKS.
And to collect logs, you can use Azure Log Analytics
.
It is not common to use Grafana
to visualize logs, but, technically it is possible !
And it is also useful for two reasons:
- Use the same tool to look for metrics and logs, even in the same dashboard
Azure Managed Grafana
have by default access toAzure Monitor
which contains the logs
This lab will guide you to create a Grafana dashboard to view AKS
logs from Azure Monitor
and thus Log Analytics
.
You will perform the following steps:
You can do this either using using the following commands Azure cli
$RG="rg-aks-swc-85"
az group create -n $RG -l swedencentral
$log_analytics_id=$(az monitor log-analytics workspace create -g $RG --workspace-name log-analytics --query id -o tsv)
$prometheus_id=$(az monitor account create -n azure-prometheus -g $RG --query id -o tsv)
$grafana_id=$(az grafana create -n azure-grafana-15 -g $RG --query id -o tsv)
az aks create -n aks-cluster -g $RG --network-plugin azure --network-plugin-mode overlay -k 1.29.0 --enable-azure-monitor-metrics --enable-addons monitoring --azure-monitor-workspace-resource-id $prometheus_id --grafana-resource-id $grafana_id --workspace-resource-id $log_analytics_id
az aks get-credentials -n aks-cluster -g $RG --overwrite-existing
The following resources will be created.
By default, logging from kube-system
namespace is excluded. To enable it, you will need to use the custom ConfigMap
provided by AKS
.
It is called container-azm-ms-agentconfig.yaml
.
More details in the official documentation.
Edit the file to make sure exclude_namespaces
is empty and to enable ContainerLogV2
by:
2.1. Replacing exclude_namespaces = ["kube-system","gatekeeper-system"]
with exclude_namespaces = []
2.2. Setting the version to v2: containerlog_schema_version = "v2"
Note you can also enable multi-line logging by setting it to enabled.
[log_collection_settings.enable_multiline_logs]
enabled = "true"
Then you can deploy the custom ConfigMap
.
kubectl apply -f container-azm-ms-agentconfig.yaml
Run simple query like the following:
ContainerLogV2
| limit 10
az grafana dashboard import -n azure-grafana-15 --definition 20811
To delete the creates resources, run the following command:
terraform destroy