Configure Alert Manager for Prometheus in Kubernetes (AWS EKS)

A client wished to setup and configure an Alert Manager in Kubernetes (AWS EKS) which sends alerts from Prometheus to Microsoft (MS) Teams channel through AWS SNS (Simple Notification Service). The Alert Manager supports and provides configuration for AWS SNS; however, we chose to use prometheus-msteams helm chart.

Prometheus is installed in the EKS cluster using the helm chart kube prometheus stack.

What is Helm chart?

Helm chart is a package manager for Kubernetes that bundles or packages the application into charts. A chart is a collection of files that can be described as a set of Kubernetes resources.

What is Prometheus MS Teams?

Prometheus msteams is a server that receives an alert message as a POST message from the Prometheus Alert Manager and sends it to the Microsoft Teams channel using the incoming webhook URL.

Create incoming webhook URL in MS Teams channel

click on more options and then connectors as shown below from within your respective MS Teams channel where you need to get alert messages

Search and select Incoming Webhook and click on Add. Then click on Configure, enter a name and upload an image for your webhook if needed.

Copy and save the webhook URL shown in the window.

Install kube-prometheus-stack Helm chart in the EKS cluster

(You can skip this step if you have installed this helm chart or any other prometheus and alert manager helm chart)

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
helm repo add prometheus-community https: //prometheus-community.github.io/helm-charts
helm repo update
helm install [ RELEASE_NAME ] prometheus-community/kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack

(Note – In our case we have used RELEASE_NAME as prometheus-stack)

If you want to install helm chart using custom values, then edit the values.yaml file of the helm chart and then use the command,

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
helm install –f values. yaml [ RELEASE_NAME ] prometheus-community/kube-prometheus-stack
OR
helm upgrade --install -f values. yaml [ RELEASE_NAME ] prometheus-community/kube-prometheus-stack
helm install –f values.yaml [RELEASE_NAME] prometheus-community/kube-prometheus-stack OR helm upgrade --install -f values.yaml [RELEASE_NAME] prometheus-community/kube-prometheus-stack
helm install –f values.yaml [RELEASE_NAME] prometheus-community/kube-prometheus-stack

OR

helm upgrade --install -f values.yaml [RELEASE_NAME] prometheus-community/kube-prometheus-stack

This helm chart includes Prometheus operator, Prometheus, alert manager, Prometheus node-exporter, Kube-state-metrics, Prometheus adapter for Kubernetes Metrics APIs and Grafana.

Install prometheus-ms-teams Helm chart in the EKS cluster

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
helm repo add prometheus-msteams
helm repo add prometheus-msteams
helm repo add prometheus-msteams 

Create a values.yaml file and pass the following values to the file.

(Note: Mention the value of the alert as your unique webhook URL and also the correct kube-prometheus-stack helm chart release name. In our case we use prometheus-stack)

Then install the helm chart using,

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
helm upgrade --install prometheus-msteams \
--namespace default -f values. yaml
prometheus-msteams/prometheus-msteams
helm upgrade --install prometheus-msteams \ --namespace default -f values.yaml prometheus-msteams/prometheus-msteams
helm upgrade --install prometheus-msteams \
  --namespace default -f values.yaml
  prometheus-msteams/prometheus-msteams

Create a values.yaml file and pass the following values to the file.

Create an alert manager configuration file and add that configuration file in Kubernetes secrets

Where prometheus-msteams from webhook_configs url is the Kubernetes service name of Prometheus MS Teams. Name the configuration file alertmanager.yaml.

Create a Kubernetes secret that contains this alert manager configuration,

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
kubectl create secret generic myalertmanager --from-file=alertmanager. yaml =alertmanager. yaml
kubectl create secret generic myalertmanager --from-file=alertmanager.yaml=alertmanager.yaml
kubectl create secret generic myalertmanager --from-file=alertmanager.yaml=alertmanager.yaml

You can check the secret content using,

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
kubectl get secret myalertmanager -o yaml
kubectl get secret myalertmanager -o yaml
kubectl get secret myalertmanager -o yaml

Copy the value of alertmanager.yaml as shown above in the image and enter the following command to check whether correct alert manager configuration is passed to the Kubernetes secret,

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
echo " < value- of -alertmanager. yaml > | base64 -d
echo "<value-of-alertmanager.yaml>” | base64 -d
echo "<value-of-alertmanager.yaml>” | base64 -d

Mention this secret in the alertmanager section of the kube prometheus stack helm chart values.yaml

Upgrade the helm chart kube-prometheus-stack

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
helm upgrade -f values. yaml prometheus-stack kube-prometheus-stack
helm upgrade -f values.yaml prometheus-stack kube-prometheus-stack
helm upgrade -f values.yaml prometheus-stack kube-prometheus-stack

Once updated, check if alerts are as shown (as seen in the images below)

About the author

Shubham Lonkar

Shubham is a DevOps engineer at Excellarate for 4 years. He has expertise in tools and technologies such as Docker, Kubernetes, terraform, Jenkins, Gitlab CICD, Bitbucket CICD. Shubham holds a bachelor degree in Computer Engineering from the Savitribai Phule Pune University.

DevOps is the single source of delays and missed market deadlines. Talk to us to see how we can help you make DevOps the sharpest arrow in your quiver.

Share this post

Table of Contents