Moving from traditional CD pipelines to GitOps

A software engineering team needs to maintain velocity to deliver continuously. They need a faster feedback loop to iterate quickly. Continuous Integration (CI), and Continuous Deployment (CD) is the mechanism to achieve.

If we think about the traditional CD pipeline and it’s challenges, some of them are:

Gitops is an alternative to the traditional CD pipeline and solves the above problems. This blog will start with idea of Gitops, its principles, discuss the challenges with CD pipeline, the need, why or why not an engineering team should adopt Gitops practise. We will also discuss how we adopted Gitops for one of our client’s project.

What is Gitops?

Gitops is the process to deploy, configure, monitor, update and manage infrastructure as a code. Git in Gitops refers to a deployment repository, which is a single source of truth.

Weaveworks (the company which coined the term Gitops) describes it as:

“GitOps is a way to do Kubernetes cluster management and application delivery. GitOps works by using Git as a single source of truth for declarative infrastructure and applications. With GitOps, the use of software agents can alert on any divergence between Git with what’s running in a cluster, and if there’s a difference, Kubernetes automatically updates or rollback the cluster depending on the case. With Git at the center of your delivery pipelines, developers use familiar tools to make pull requests to accelerate and simplify both application deployments and operations tasks to Kubernetes.”

In other words, Gitops is the ability to sync the repository state and reflect the changes in deployment environment automatically. It is derivative of the operation model which focuses on Pull Requests/Merge Requests. Approved changes that are desired get deployed automatically. All these changes are observable in the git history.

To understand better what Gitops is, let’s ask a question and try to answer them.

Q: What do we have in the deployment repository?

A: Deployment repository contains infrastructure code and runtime configurations to perform deployments on an environment. It can have kubernetes manifests and helm-charts.

Gitops as concept was introduced with Kubernetes ecosystem.But other tools like ansible_tower and terraform with gitlab have adopted. We will limit ourselves to Kubernetes and Weaveworks Flux

The principles of Gitops:

Challenges and Solutions

We discussed earlier, about the challenges of the traditional CD pipeline. Let us discuss in details, how Gitops solve these problems. This should also answer “Why Gitops is useful”.

Security credentials shared across the boundaries.

Any CI/CD tools need access to the Kubernetes cluster API to deploy, which implies sharing credentials across boundaries. If we see in the diagram, the credentials boundaries are crossed where the CI system has access to Kubernetes Cluster. Push Pipeline

We worked with a fintech client, where the security team did not encourage sharing credentials across environments. Hence, a pull-based pipeline was useful. Devops team need to make changes to the configuration repository. Kubernetes operators (discussed below) look for changes and apply on the cluster. If you look at the diagram, the Kubernetes cluster has access to the Github repo and docker registry. No need to share kubeconfig outside of the environment. Pull Pipeline

Monitoring systems after the deployment.

With a CD pipeline, we can deploy the changes but monitoring the deployment remains a challenge. Example: Jenkins does not have first class support for Kubernetes deployment. We need to write custom scripts to validate if a deployment was successful on the K8S cluster.

Gitops would solve the problem. It also provides retries and alert mechanisms to take actions.

Cross environment access.

Generally, CI/CD systems reside in different infrastructure to deployment environments, and the challenge is network reachability. One has to peer VPCs or establish site to site VPNs. This can become overwhelming at times and require expertise in the team. With Gitops, the software agents requires a read-only Git token, which pull the code from git repository and apply the changes.

Treating Configuration as Code

From our experience, managing configuration across different environments is a complicated problem. In the past, we have used Consul as key/value(KV) store for storing configuration. But running a consul cluster for KV store can be overkill. Periodic backups of Consul data is another overhead.

Gitops as a solution fulfils the following characteristic for configurations:

Other benefits of Gitops

Hope, by now we understand why Gitops is useful. Some other benefits are:

FluxCD

Gitops is the process, and we used Flux CD to implement the strategy. Other tools to implement Gitops are Tekton, ArgoCD and Jenkins X, to name a few. Here, we will discuss FluxCD only.

Flux installs certain Controllers. Listing down these controllers with their functionality:

How we implemented GitOps?

GitOps Workflow

Gitops Workflow

Challenges applying Gitops

We talked about all the Jazz of Gitops. Not necessary, it fits the bill for engineering teams and are ready to adopt it. The reason could be the lack of essential devops practises like a test-release-deploy pipeline. Other challenges to adopt Gitops are:

Summary

Discussion and feedback