In these recent years, technology marketing has taken to a “word mashup” approach to many new terms. We got MicroService, DevOps, DevSecOps, MLOps, AIOps, ChatOps, and even GitOps. Combining words for marketing terms doesn’t really tell us much about what is meant by these new concepts. Often the concepts are very abstract and yet to be fully defined. Yet, similar to Agile, they often require a paradigm shift in how people work together to accomplish goals similar to those they have always strived to complete successfully.
So as technologists what are we to make of all these terms. Some are marketing hype while others have more meaningful concepts underlying the terms. Often we ignore the new terms, but sometimes they warrant a look to see if they really mean anything or if they are just rebranding old concepts. For this article, I want to take a quick look (not in-depth) at GitOps.
Like many other marketing terms, it can be rather abstract. Some consider it to simply mean that you use git as a part of your development and deployment process. While others have applied a slightly more granular meaning to the term. Here are some definitions:
GitOps is a paradigm or a set of practices that empowers developers to perform tasks which typically fall under the purview of IT operations. GitOps requires us to describe and observe systems with declarative specifications that eventually form the basis of continuous everything.
— CloudBees
GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, including Git and Continuous Deployment tools.
— gitops.tech
GitOps: versioned CI/CD on top of declarative infrastructure. Stop scripting and start shipping.
GitOps in short is a set of practices to use Git pull requests to manage infrastructure and application configurations. Git repository in GitOps is considered the only source of truth and contains the entire state of the system so that the trail of changes to the system state are visible and auditable.
— Mario Vázquez, Redhat Openshift
Git as a source of truth for desired state of whole system yes really the whole system.
…
The goal is to describe everything: policies, code, configuration, and even monitored events and version control it all. Keeping everything under version control enforces convergence where changes can be reapplied if at first they didn't succeed.
The basic concept with GitOps is that you use git and regular git practices (such as merge/pull requests) to manage and approve changes to the live system. This is a practice known by developers in how they manage their source code. For this to be possible you must be able to describe your “desired” state for the system then have a mechanism for applying that state to the live system and ensure it doesn’t diverge from that “desired” state (a desired/declarative state engine).
Kubernetes Only?
One of the goals of GitOps is to ensure the desired state is always represented in the live system. Kubernetes by default provides us with a way to describe our workloads/services/configurations in addition to a control plane that works to ensure the running state matches the described state. This feature makes it possible to realize the idea of ensuring no divergence from a described state in git.
So is Kubernetes required for implementing GitOps? No, but it greatly helps facilitate its implementation since the desired/declarative state engine is already implemented as a core concept. Kubernetes can even be used to control the state of workloads that are not even running in Kubernetes. Kelsey Hightower presented a demo for DevOps Days 2020 where he created a Kubernetes controller for a serverless runtime outside of the cluster. His controller then managed the state of the target serverless platform ensuring it represented the state he described in his git repo.
There are other projects that are building on the same concepts such as Crossplane which provides a Kubernetes controller for managing cloud infrastructure outside of Kubernetes. This allows you to describe MySql instances or cloud storage like GCS or S3 using a similar domain-specific language (DSL) as managing application deployments.
So the target resource doesn’t need to be in Kubernetes but its control plane can be used as a way of managing your application workloads and possibly infrastructure.
Benefits of GitOps
Well, the benefits primarily lie in how it is implemented, so first, we will look at the basic implementation.
The first part of the implementation is to have a way to describe your “desired” state, or what you want to make a reality in your target environment, the DSL. This is primarily being accomplished with YAML and JSON. It's used to declare/provide the state, (or description of), the resources you want that is committed to a git repository. Next, the state must be applied to the environment. This is done with a combination of Kubernetes Custom Resouces Definitions (CRD’s) with Controllers and/or Operators. The operator is the actor in applying the described state to the runtime environment by pulling the DSL from git and “deploying” it to the environment.
This process can now provide a single source of truth for the desired state of all infrastructure and apps along with a controlled process that continually reconciles that desired state with the actual runtime. It also provides:
what can be described can be validated and automated
observability
single source of truth, allowing you to see in git what should be in place.
collaborate via merge requests and reviews on all changes.
using MR’s to control an approval process for changes to an environment.
auditable compliance of all changes to the cluster.
seeing in git what should be live and the history of changes.
consistent rollback process
disaster recovery, through simple cluster application state restoration/deployment with new cluster creation.
developer-centric process
a fit for the Kubernetes declarative manifest model and “desired state” engine
secure deployment of workload to the cluster using a “pull” model rather than a “push” model. Meaning that git and CI don’t need to have access to the cluster. Rather the cluster pulls in its state from git
Wrap-up
GitOps has a set of principles that require people processes and tools to fully implement. The core principles of GitOps are:
Build declarative configurations for defining all workloads.
Require all modifications to use the git review process - i.e. “kubectl” should never be used directly
An operator in the cluster should drive the observed cluster state to the desired state declared in git.
Can you rely 100% on just this form of GitOps? I would have to say no. When you start from scratch there is some bootstrapping necessary for the environment before GitOps can take over. For Kubernetes, you need to have the CRD’s applied and the controller/operator installed into a cluster that needed to be created as well. This minimal bootstrapping is necessary before GitOps for application pulls can be implemented. Tools like Terraform fulfill portions of the GitOps principles by having the declared configuration managed in git along with its ability for managing state. I believe this is where you can have a great match between Terraform and the various GitOps solutions.
This however begins another entire topic so I will end here. I may produce follow-ups to this is if there is interest in further exploring what GitOps is and how to implement it. So please provide comments/questions and feedback.
