How to run your DevOps using Kubernetes

How to run your DevOps using Kubernetes

In previous posts of this series we explained the life-cycle of DevOps for containers. And we talked about:

Best Practices

Consider the following best practices during your implementation:

  • Use Kubernetes namespaces for different projects/environments. For example: projectname-dev, projectname-test, projectname-staging. This allows the required separation between pods/services inside each namespace, allowing to create the same service name in multiple environments. Also allowing to delete a complete environment by deleting its namespace.
  • Use same docker image for deploying in multiple environments and use Kubernetes ConfigMaps (will explain shortly below) to change the configuration files (database connections or urls).
  • Name external resources like what you named Kubernetes namespaces. For example: database names, urls routes like http://projectname-dev.k8s.com or file system folders for content, environment names in release pipeline. This gives the ability to just replace all these in configurations with the environment name to deploy and allow you to associate all these objects together.
  • Don’t use latest tag to get your latest docker image. Version your images properly with a new version in each build. You can use build number to increment last part of semantic versions like (Major.Minor.Patch.Build). This will let Kubernetes know this is a new Pod (by changing the tag in yaml) to deploy (otherwise you will have to delete old Pods manually so Kubernetes can create new ones and fetch new images) and will allow deployment to be versioned later and can be rolled back to a specific build when things go wrong.
  • Keep yaml files inside source control (near Dockerfile or docker-compose files), so developers adding new configurations can apply the changes to the yaml file (as he add them in his code branch) and when this reaches the point to be deployed (merged with develop for example) it gets deployed in the pipeline without the need to change the release pipeline. Also this allows tracking changes to yaml happening in different versions.

Before we start implementing it, let’s explain some Kubernetes concepts that we will use:

Volumes

Kubernetes supports mounting volumes inside container file system. Which means taking an external drive/folder/network share/cloud storage and mount it inside a path in the container so it’s seen like a regular folder from within the container.

Volumes allows scenarios like:

  • Mounting web files (html/js/css/images) from a network shared folder into the “www” root folder of a web server that is hosted into multiple Pods. Allowing a file to be uploaded once to this file share to be seen in all servers. Also allowing Pods to be deleted and created without losing these files as they are not actually inside the Pod file system.
  • Mounting an empty folder from host machine to inside the “logs” folder of a container allowing logs to be written to host machine file system and not be lost when the Pod is deleted.

More information about Volumes

ConfigMaps

ConfigMaps are a special type of volumes that allow us to specify in yaml, a folder with 1 or more configuration files (for example a web.config file that we will write its content in yaml). Then this allow us to mount this file to replace existing web.config in the container file system. Using ConfigMaps we can provide different configurations during deployment to each environment without changing the Docker image we are using.

More information about ConfigMaps

Step 1: Create the yaml file

Now let’s create a yaml file with all the objects we want to deploy on Kubernetes. Here is what we need to create:

  • Service
  • ConfigMap
  • Deployment
  • Ingress Route

Create a new file named helloworld.yaml with the following content:

apiVersion: v1
kind: Service
metadata:
  name: helloworld-service
spec:
  selector:
    app: helloworld
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: helloworld-config
data:
  appsettings.json: |- # this will replace the original appsettings file
    {
      "ConnectionStrings": {
        "Default": "Server=localhost; Database=helloworld-dev; Trusted_Connection=True;"
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment
  labels:
    app: helloworld
spec:
  replicas: 2
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: bishoymly/helloworld:1.0.0.1023 # this version should change with every build
        ports:
        - containerPort: 80
        volumeMounts:
          - name: config
            mountPath: /app/appsettings.json # here we are replacing the appsettings file from ConfigMap
            subPath: appsettings.json
      volumes:
        - name: config
          configMap:
            name: helloworld-config
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: helloworld-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: localhost # we can specify different hosts or paths for different namespaces
    http:
      paths:
      - path: /
        backend:
          serviceName: helloworld-service
          servicePort: 80

Note the order of objects always creating the Service and ConfigMap before Deployment.

Now let’s apply this on our cluster:

kubectl create namespace helloworld-dev
kubectl apply -f helloworld.yaml -n helloworld-dev

The above commands creates a namespace named helloworld-dev then apply our yaml deployment to this namespace. We can do the same for multiple namespaces/environments without changing the yaml file.

Step 2: Apply DevOps pipeline using TFS

Here I’m giving the example using TFS Build/Release pipeline but the same can be done using any other tool.

Assuming we have a build that created a version tagged docker image and pushed it to a registry. Also I’m assuming the build published the yaml file from source code to Drops in order to use it in the release.

We will create a release definition that is triggered with continuous delivery from this build. The definition will contain multiple environments named after the target namespaces in Kubernetes like helloworld-dev, helloworld-test.

In order to change specific values in the yaml file We will use a Replace Tokens task which will look for tokens in the yaml file (with pre- and postfix __) and replace them with corresponding environment variables that TFS generates from build and release.

For example here are the environment variables that we will use in our sample:

  • Release.EnvironmentName: this will be the namespace we will use in Kubernetes and the database name in connection string.
  • Build.BuildNumber: this will be the version we use for the docker image tag.

To know what variables are available for use, when the release runs you can see all environment variables written in logs.

Then we will use these tasks in the release definition:

  1. Replace Tokens
    Source Path: browse to the folder that contains the yaml file(s)
    Target File Pattern: “*.yaml”. This performs token replacement on any yaml file
  2. Kubernetes Task (apply)
    Set the k8s connection to our cluster.
    Command: “apply”
    Check the “Use Configuration Files” option and set the file to the helloworld.yaml file using the file picker
    Arguments: “-n $(Release.EnvironmentName)”

Now we will change our yaml file to have tokens (highlighted in bold) that will be replaced in release:

apiVersion: v1
kind: Service
metadata:
  name: helloworld-service
spec:
  selector:
    app: helloworld
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: helloworld-config
data:
  appsettings.json: |- 
    {
      "ConnectionStrings": {
        "Default": "Server=localhost; Database=__RELEASE_ENVIRONMENTNAME__; Trusted_Connection=True;"
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment
  labels:
    app: helloworld
spec:
  replicas: 2
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: bishoymly/helloworld:__BUILD_BUILDNUMBER__
        ports:
        - containerPort: 80
        volumeMounts:
          - name: config
            mountPath: /app/appsettings.json # here we are replacing the appsettings file from ConfigMap
            subPath: appsettings.json
      volumes:
        - name: config
          configMap:
            name: helloworld-config
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: helloworld-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: localhost # we can specify different hosts or paths for different namespaces
    http:
      paths:
      - path: /
        backend:
          serviceName: helloworld-service
          servicePort: 80

This definition now uses environment names in the pipeline to deploy same yaml file to each environment while changing the version to build number and the database name to environment name.