Email Notifications and GitHub Webhooks with Argo CD

Argo CD Notifications and Github

Argo CD is a Kubernetes-native continuous deployment (CD) tool. It follows the GitOps pattern – Git repositories as the source of truth for desired application state. Configuration files placed in Git repositories are used to create environments needed for a CD process. Argo CD provides a notification system that allows users to receive alerts and notifications when certain events occur within the tool.

Argo CD Notifications is a standalone project that continuously monitors Argo CD and notifies users about important changes in Argo CD applications.

The mechanism of triggers and templates enables defining when the notification will be sent as well as the notification content. 

Argo CD Notifications already contain a catalog of predefined triggers and templates.

Argo CD – the process of syncing

Since Argo CD is a tool for syncing resources from Git to Kubernetes, we want to receive notifications on how the process of syncing goes. To accomplish this, Argo CD Notifications project was announced as support for notifications functionality requirements. It looks out for Application resources, composes a notification, and delivers it to configured receivers.

Triggers define when a notification should be sent. Its definition includes name, condition, and notification templates reference and it is configured in argocd-notifications-cm ConfigMap. 

Templates define the content of notifications. Templates are also defined in argocd-notifications-cm ConfigMap. The idea behind templates is reusability by multiple triggers. The following yaml configuration block represents the definition of the trigger which sends a notification when the application sync status changes to Succeeded using app-sync-status a template. 

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  trigger.on-sync-succeeded: |
    - when: app.status.sync.status == "Succeeded" # trigger condition
      send: [app-sync-status] # template names

In this blog post, I’ve introduced the Argo CD Notification services and its basic concepts, emphasizing Email service and Webhooks. Here you can find configurations of email and webhook integrations and an explanation of managing subscriptions to application notifications.

Setup

First of all, we need a Kubernetes cluster with Argo CD. We will install Argo CD on minikube.

Follow the next steps for the installation:

• create argocd namespace and install Argo CD:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

•  access Argo CD API server:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

• port forward (Argo CD available on port 8080): 

kubectl port-forward svc/argocd-server -n argocd 8080:443

• obtain admin password

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Install Argo CD notifications

• Install Argo CD notifications controller that is responsible for the processing notifications

kubectl 
apply -n argocd -f 
https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/release-1.0/manifests/install.yaml

• Install triggers and templates from the catalog 

kubectl 
apply -n argocd -f 
https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/release-1.0/catalog/install.yaml

Notification services

Notification services offer integration with services such as Email, GitHub, Slack, Teams, Grafana, custom webhooks.

Services are configured in ConfigMap argocd-notifications-cm with key service. <type>.<custom-name>.

Let’s configure plain email integration and custom webhook.

Email

The Email notification service sends emails using SMTP protocol. The following parameters should be set: host, port, username, password, from.

• Store email account username and password in argocd-notifications-secret:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
stringData:
  email-username: <EMAIL_USER>
  email-password: <PASSWORD>
type: Opaque

To enable sending emails from a Gmail account, the App Password should be configured. An App Password permits a less secure app or device to access your Google Account. To configure one, go to: https://myaccount.google.com/apppasswords.

• Register for email service in argo-notifications-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.email.gmail: |
    username: $email-username
    password: $email-password
    host: smtp.gmail.com
    port: 465
    from: $email-username

• Define a notification template that specifies the content of email notifications and trigger which emails should be sent:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.email.gmail: |
    username: $email-username
    password: $email-password
    host: smtp.gmail.com
    port: 465
    from: $email-username
  template.app-sync-succeeded: |
    email:
        subject: Application {{.app.metadata.name}} has been successfully synced.
    message: |
        {{if eq .serviceType "slack"}}:white_check_mark:{{end}} Application {{.app.metadata.name}} has been successfully synced at {{.app.status.operationState.finishedAt}}.
        Sync operation details are available at: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true .
  trigger.on-sync-succeeded: |
    - description: Application syncing has succeeded
      send:
      - app-sync-succeeded
      when: app.status.operationState.phase in ['Succeeded']

Notification templates support specifying subjects for email notifications.

Each template has access to the following fields:

• app – holds the application object.
• context – user-defined string map and might include any string keys and values.
• serviceType – holds the notification service type name. The field can be used to conditionally render service-specific fields.
• recipient – holds the recipient’s name.

You can set the custom context in argocd-notifications-cm ConfigMap. Here is an example of setting a custom argocdUrl.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  context: |
    argocdUrl: <YOUR_ARGOCD_URL>
  ...

Sometimes the application health status might alternately switch to Progressing and then back to Healthy so the trigger might generate multiple notifications. To avoid generating unnecessary notifications, the oncePer field configures triggers to generate the notification only when the corresponding application field changes.

In the following example, oncePer the property ensures that notification should be sent once per commit SHA.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  trigger.on-sync-succeeded: |
    when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
    oncePer: app.status.sync.revision
    send: [app-sync-succeeded]

Subscriptions to Email Notifications

ArgoCD applications can subscribe to events using the following annotation:

notifications.argoproj.io/subscribe.<trigger>.<service>: <recipient>

Let’s look in detail at the following annotation: 

notifications.argoproj.io/subscribe.on-sync-succeeded.gmail: test.user@gmail.com

The annotation consists of the following parts:

• notifications.argoproj.io/subscribe – annotation prefix that indicates subscription
• on-sync-succeeded – trigger name that indicates successful synchronization 
• gmail – the name of the service we have already configured
• test.user@gmail.com – recipient of notifications

Let’s define Argo CD Application yaml file application.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  annotations:
    notifications.argoproj.io/subscribe.on-sync-succeeded.gmail: test.user@gmail.com
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: .
    repoURL: https://github.com/ivona13/guestbook
    targetRevision: HEAD
  syncPolicy:
    automated: {}

and apply it to the cluster kubectl apply -n argocd -f application.yaml.

The subscription to application events in Argo CD UI:

Webhook

The Webhook notification service allows sending a generic HTTP request using a templatized request body and URL. Webhooks enable use cases such as integration with GitHub, triggering Jenkins jobs, and others.

We will configure a webhook for updating GitHub commit status.

• Store GitHub token in argocd-notifications-secret

apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
stringData:
  token: <YOUR_TOKEN>
type: Opaque

• Define argocd-notifications-cm ConfigMap that configures webhook for updating GitHub commit status:

kind: ConfigMap
apiVersion: v1
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  context: |
    argocdUrl: https://localhost:8080
  service.webhook.github-webhook: |
    url: https://api.github.com
    headers:
    - name: Authorization
      value: token $TOKEN
    subscriptions: |
      - recipients
        - github-webhook
        triggers:
        - on-sync-succeeded
  template.app-sync-succeeded: |
    webhook:
      github-webhook:
        method: POST
        path: /repos/{{call .repo.FullNameByRepoURL .app.spec.source.repoURL}}/statuses/{{.app.status.operationState.operation.sync.revision}}
        body: |
          {
            {{if eq .app.status.operationState.phase "Running"}} "state": "pending"{{end}}
            {{if eq .app.status.operationState.phase "Succeeded"}} "state": "success"{{end}}
            {{if eq .app.status.operationState.phase "Error"}} "state": "error"{{end}}
            {{if eq .app.status.operationState.phase "Failed"}} "state": "error"{{end}},
            "description": "ArgoCD",
            "target_url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
            "context": "continuous-delivery/{{.app.metadata.name}}"
          }
  trigger.on-sync-succeeded: |
    - description: Application syncing has succeeded
      send:
      - app-sync-succeeded
      when: app.status.operationState.phase in ['Succeeded']

There is the usage of the function repo.FullNameByRepoURL which is a part of built-in repo functions that templates have access to. The set of repo functions provides additional information about Application source repositories.

Global subscriptions to Webhook service

What can be seen in the application.yaml file is a definition of section subscriptions under service.webhook.github-webhook. This field enables the global configuration of subscriptions and it is applied to all applications. Triggers and recipients might be configured using the triggers and recipients fields.

Argo CD Notification CLI

For the purpose of demonstrating how the Argo CD Notifications project works, I have created a simple CLI application. 

Use the following link to the GitHub code and follow the prerequisites described in README.md file to start up the application. 

The application gives users the ability to apply prepared ConfigMap to the cluster. The user gets the list of deployed Argo CD applications that can be chosen to be monitored. After choosing the application for monitoring, the user determines the service and triggers that should be patched to the application yaml file. 

This demo application uses Argo CD API to fetch and update the Argo CD applications.

Conclusion

Argo CD Notifications provides a flexible way to interact with users and other services. It enables users to be notified when important changes happen in the applications.  

Notifications are also supported by Argo Rollouts.