Continuous Deployment with Azure Kubernetes Service, Azure Container Registry & Jenkins

"It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change." - Charles Darwin

Introduction

Containerization has taken the application development world by storm. Kubernetes has become the standard way of deploying new containerized distributed applications used by the largest enterprises in a wide range of industries for mission-critical tasks, it has become one of the biggest open-source success stories.

Although Google Cloud has been providing Kubernetes as a service since November 2014 (Note it started with a beta project), Microsoft with AKS (Azure Kubernetes Service) and Amazon with EKS (Elastic Kubernetes Service)  have jumped on to the scene in the second half of 2017.

Example:

AWS had KOPS https://kubernetes.io/docs/getting-started-guides/kops/

Azure had Azure Container Service https://docs.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-walkthrough

However, they were wrapper tools available prior to these services which would help a user create a Kubernetes cluster, but the management and the maintenance (like monitoring and upgrades) needed efforts.

Azure Container Registry:

With container demand growing, there is always a need in the market for storing and protecting the container images. Microsoft provides a Geo Replica featured private repository as a service named Azure Container Registry.

Azure Container Registry is a private registry for hosting container images. It integrates well with orchestrators like Azure Container Service, including Docker Swarm, DC/OS, and the new Azure Kubernetes service. Moreover, ACR  provides capabilities such as Azure Active Directory-based authentication, webhook support, and delete operations.

The coolest feature provided is Geo-Replication. This will create multiple copies of your image and distribute it across the globe and the container when spawned will have access to the image which is nearest.

Although Microsoft has good documentation on how to set up ACR  in your Azure Subscription, we did encounter some issues and hence decided to write a blog on the precautions and steps required to configure the Registry in the correct manner.

Note: We tried this using a free trial account. You can setup it up by referring the following link: https://azure.microsoft.com/en-in/offers/ms-azr-0044p/

Prerequisites:

  • Make sure you have resource groups created in the supported region.
    Supported Regions: eastus, westeurope, centralus, canadacentral, canadaeast
     
  • If you are using Azure CLI for operations please make sure you use the version: 2.0.23 or 2.0.25 (This was the latest version at the time of writing this blog)

Note: If version 2.0.24 is used it has the following bugs:
https://stackoverflow.com/questions/48201850/login-to-the-azure-container-ser    vice-fails-with-error-bool-object-has-no-attri/48204384#48204384
- https://github.com/Azure/azure-cli/issues/5300

Steps to install Azure CLI 2.0.23 or 2.0.25 (ubuntu 16.04 workstation):

echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ wheezy main" | \
           sudo tee /etc/apt/sources.list.d/azure-cli.list

sudo apt-key adv --keyserver packages.microsoft.com --recv-keys 52E16F86FEE04B979B07E28DB02C46DF417A0893

sudo apt-get install apt-transport-https
sudo apt-get update && sudo apt-get install azure-cli

Install a specific version:

sudo apt install azure-cli=2.0.23-1
sudo apt install azure-cli=2.0.25.1

Steps for Container Registry Setup:

Login to your Azure Account:

az login --username --password

  • Create a resource group:
az group create --name <RESOURCE-GROUP-NAME>  --location eastus
Example : az group create --name acr-rg  --location eastus
  • Create a Container Registry:
az acr create --resource-group <RESOURCE-GROUP-NAME> --name <CONTAINER-REGISTRY-NAME> --sku Basic --admin-enabled true
Example : az acr create --resource-group acr-rg --name testacr --sku Basic --admin-enabled true

Note: SKU defines the storage available for the registry for type Basic the storage available is 10GB, 1 WebHook and the billing amount is 11 Rs/day.

For detailed information on the different SKU available visit the following link: https://azure.microsoft.com/en-us/pricing/details/container-registry/

  • Login to the registry :
az acr login --name <CONTAINER-REGISTRY-NAME>
Example :az acr login --name testacr
  • Sample docker file for a node application :
FROM node:carbon
# Create app directory
WORKDIR /usr/src/app
COPY package*.json ./
# RUN npm install
EXPOSE 8080
CMD [ "npm", "start" ]
  • Build the docker image :
docker build -t <image-tag>:<software>
Example :docker build -t base:node8
  • Get the login server value for your ACR :
az acr list --resource-group acr-rg --query "[].{acrLoginServer:loginServer}" --output table
Output  :testacr.azurecr.io
  • Tag the image with the Login Server Value:
    Note: Get the image ID from docker images command
Example :docker tag image-id  testacr.azurecr.io/base:node8

Push the image to the Azure Container Registry:
Example: docker push testacr.azurecr.io/base:node8

Microsoft does provide a GUI option to create the ACR: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal

  • List Images in the Registry:

Example: az acr repository list --name testacr --output table
  • List tags for the Images:

Example: az acr repository show-tags --name testacr --repository <name> --output  table
  • How to use the ACR image in Kubernetes deployment: Use the login Server Name + the image name

Example :

containers:
- name: demo
image: testacr.azurecr.io/base:node8

Azure Kubernetes Service

Microsoft released the public preview of Managed Kubernetes for Azure Container Service (AKS) on October 24, 2017. This service simplifies the deployment, management, and operations of Kubernetes. It features an Azure-hosted control plane, automated upgrades, self-healing, easy scaling.

Similarly to Google AKE and Amazon EKS, this new service will allow access to the nodes only and the master will be managed by Cloud Provider. For more information visit the following link: https://azure.microsoft.com/en-us/blog/introducing-azure-container-service-aks-managed-kubernetes-and-azure-container-registry-geo-replication/

Let's now get our hands dirty and deploy an AKS infrastructure to play with:

  • Enable AKS preview for your Azure Subscription: While AKS is in preview, creating new clusters requires a feature flag on your subscription.

az provider register -n Microsoft.ContainerService
  • Kubernetes Cluster Creation Command: Note: A new separate resource group should be created for the Kubernetes service.Since the service is in preview, it is available only to certain regions.

Make sure you create a resource group under the following regions.

eastus, westeurope, centralus, canadacentral, canadaeast
az  group create  --name  <RESOURCE-GROUP>   --location eastus

Example : az group create --name aks-rg --location eastus
az aks create --resource-group <RESOURCE-GROUP-NAME> --name <CLUSTER-NAME>   --node-count 2 --generate-ssh-keys

Example : az aks create --resource-group aks-rg --name akscluster  --node-count 2 --generate-ssh-keys

Example with different arguments :

Create a Kubernetes cluster with a specific version.

az aks create -g MyResourceGroup -n MyManagedCluster --kubernetes-version 1.8.1

Create a Kubernetes cluster with a larger node pool.

az aks create -g MyResourceGroup -n MyManagedCluster --node-count 7
  • Install the Kubectl CLI :

To connect to the kubernetes cluster from the client computer Kubectl command line client is required.

sudo az aks install-cli

Note: If you're using Azure CloudShell, kubectl is already installed. If you want to install it locally, run the above  command:

  • To configure kubectl to connect to your Kubernetes cluster :

az aks get-credentials --resource-group=<RESOURCE-GROUP-NAME> --name=<CLUSTER-NAME>

Example :

 az aks get-credentials --resource-group=aks-rg --name=akscluster
  • Verify the connection to the cluster :

kubectl get nodes -o wide 

We had encountered a few issues while setting up the AKS cluster at the time of writing this blog. Listing them along with the workaround/fix:

Command: az aks create --resource-group aks-rg --name akscluster  --node-count 2 --generate-ssh-keys

Error: Operation failed with status: 'Bad Request'.

Details: Required resource provider registrations Microsoft.Compute, Microsoft.Storage, Microsoft.Network are missing.

Fix: If you are using the trial account, click on subscriptions and check whether the following providers are registered or not :

  • Microsoft.Compute

  • Microsoft.Storage

  • Microsoft.Network

  • Microsoft.ContainerRegistry

  • Microsoft.ContainerService

Error: We had encountered the following mentioned open issues at the time of writing this blog.

https://github.com/Azure/AKS/issues/144

https://github.com/Azure/AKS/issues/141

https://github.com/Azure/AKS/issues/145


Jenkins setup for CI/CD with ACR, AKS

Microsoft provides a solution template which will install the latest stable Jenkins version on a Linux (Ubuntu 14.04 LTS) VM along with tools and plugins configured to work with Azure. This includes:

  • git for source control

  • Azure Credentials plugin for connecting securely

  • Azure VM Agents plugin for elastic build, test and continuous integration

  • Azure Storage plugin for storing artifacts

  • Azure CLI to deploy apps using scripts

Refer the below link to bring up the Instance: https://docs.microsoft.com/en-us/azure/jenkins/install-jenkins-solution-template

Pipeline plan for Spinning up a Nodejs Application using ACR - AKS - Jenkins

image 1.png

What the pipeline accomplishes :

Stage 1:

The code gets pushed in the Github. The Jenkins job gets triggered automatically. The Dockerfile is checked out from Github.

Stage 2:

Docker builds an image from the Dockerfile and then the image is tagged with the build number.Additionally, the latest tag is also attached to the image for the containers to use.

Stage 3:

We have default deployment and service YAML files stored on the Jenkins server. Jenkins makes a copy of the default YAML files, make the necessary changes according to the build and put them in a separate folder.

Stage 4:

kubectl was initially configured at the time of setting up AKS on the Jenkins server. The YAML files are fed to the kubectl util which in turn creates pods and services.

Sample Jenkins pipeline code :

node {
      // Mark the code checkout 'stage'....
        stage('Checkout the dockefile from GitHub') {

            git branch: 'docker-file', credentialsId: 'git_credentials', url: 'https://gitlab.com/demo.git'

        }

        // Build and Deploy to ACR 'stage'...
        stage('Build the Image and Push to Azure Container Registry')          {
                app = docker.build('testacr.azurecr.io/demo')
                withDockerRegistry([credentialsId: 'acr_credentials', url: 'https://testacr.azurecr.io']) {
                app.push("${env.BUILD_NUMBER}")
                app.push('latest')
                }
        }

        stage('Build the Kubernetes YAML Files for New App') {

            <The code here will differ depending on the YAMLs used for the application>

        }

        stage('Delpoying the App on Azure Kubernetes Service') {
            app = docker.image('testacr.azurecr.io/demo:latest')
            withDockerRegistry([credentialsId: 'acr_credentials', url: 'https://testacr.azurecr.io']) {
            app.pull()
            sh "kubectl create -f ."
            }

        }
    }

What we achieved:

  • We managed to create a private Docker registry on Azure using the ACR feature using az-cli 2.0.25.

  • Secondly, we were able to spin up a private Kubernetes cluster on Azure with 2 nodes.

  • Setup Up Jenkins using a pre-cooked template which had all the plugins necessary for communication with ACR and AKS.

  • Implement a small Continuous Deployment pipeline in Jenkins which used all the Docker features.


Rohan photo.jpg

Rohan began his career with Database and Linux Administration and later switched to performance testing and automation. He has worked extensively in setting up DevOps processes and CI/CD using Docker, Kubernetes, Ansible and Chef on various cloud providers like Amazon, IBM, Microsoft and Oracle. Rohan works as Technical Lead-DevOps at Velotio.