Since beginning this project, I have been slowly building up my (private) Ansible repository on Github. This repository is pulled into AWX where there are a series of jobs that run daily. The idea behind those daily jobs is to automate configuration and administration tasks such as installing the latest software and OS updates.

I have also extended this capability to enhance my Kubernetes knowledge from building my workloads using the Rancher web interface to building manifest templates and deploying them using Ansible. At some point, I will exploring turning this into a full CI/CD pipeline.

As an example, I found a web-based IRC client called Convos which can be run as a container which I wanted to deploy on my Kubernetes cluster.

The relevant file structure for deploying Convos in the Ansible repository looks like this:

~/ansible
  /config
    kube-encrypted
  /inventory
    /group_vars
    /host_vars
  /roles
    /convos
      /manifests
        deployment.j2
        ingress.j2
        namespace.j2
        service.j2
        volume.j2
      /tasks
        main.yml
 /vars
   main.yml
 k8s-workloads.yml

Configuration and Playbook

In the config directory is an encrypted version of my kubeconfig file, which tells the kubectl command how to access my kubernetes cluster. For Rancher, this file can be downloaded from the web interface. There is a pre-run task in the k8s-workloads playbook which decrypts this file using Ansible vault to create the kubeconfig file that will be used. Noter that kubeconfig is in .gitignore so that the unencrypted version doesn’t accidentally get commited.

The k8s-workloads.yml plabook 

- name: Deploy Workloads on Kubernetes
  hosts: 
    - localhost
  gather_facts: no
  ignore_errors: yes
  vars_files:
     - vars/main.yml
  pre_tasks:  
    - name: Decrypt kubeconfig
      copy:
        src: config/kube-encrypted
        dest: config/kubeconfig
        decrypt: yes
  roles:
     - convos

As a side note, this playbook is running kubectl on the localhost and not logging into another host via ssh as Ansible usually does. This requires additional python libraries to be installed in the python environment in which Ansible is executing. To do this, I had to install AWX with a custom virtual environment:

ansible-playbook -i inventory install.yml --extra-vars "@venv_vars.yaml"

# venv_vars.yaml
---
custom_venvs:
  - name: k8s
    python: python3 # Defaults to python2
    python_ansible_version: 2.10.7
    python_modules:
      - setuptools
      - setuptools_rust
      - openshift
      - kubernetes-validate

Back to our k8s-workloads playbook, it is executing the pre-task to decrypt the kubconfig file and will subsequently invoke each role. It should be obvious that creating a new role for each workload to be deployed scales nicely.

The Convos Role

The Convos role consists of tasks which deploys manifests for the different Kubernetes configurations: namespace, volume, deployment, service, and ingress. I’ve used variables stored in vars/main.yml to customize for my specific environment.

Here is the entirety of the role’s task.yml:

 ---
- name: Deploy Convos Namespace Manifest
  k8s:
    state: present
    kubeconfig: "{{ kube_config }}"
    context: "{{ kube_context }}"
    definition: "{{ lookup('template', 'manifests/namespace.j2') }}"
    validate:
       fail_on_error: yes
    wait: true
- name: Deploy Convos Volume Manifest
  k8s:
      state: present
      kubeconfig: "{{ kube_config }}"
      context: "{{ kube_context }}"
      definition: "{{ lookup('template', 'manifests/volume.j2') }}"
      validate:
         fail_on_error: yes
      wait: true
- name: Deploy Convos Deployment Manifest
  k8s:
   state: present
   kubeconfig: "{{ kube_config }}"
   context: "{{ kube_context }}"
   definition: "{{ lookup('template', 'manifests/deployment.j2') }}"
   validate:
      fail_on_error: yes
   wait: true
- name: Deploy Convos Service Manifest
  k8s:
   kubeconfig: "{{ kube_config }}"
   context: "{{ kube_context }}"
   state: present
   definition: "{{ lookup('template', 'manifests/service.j2') }}"
   validate:
      fail_on_error: yes
   wait: true
- name: Deploy Convos Ingress Manifest
  k8s:
   kubeconfig: "{{ kube_config }}"
   context: "{{ kube_context }}"
   state: present
   definition: "{{ lookup('template', 'manifests/ingress.j2') }}"
   validate:
      fail_on_error: yes
   wait: true

If I were to change state from present to absent and run this role again, the resources would each be removed. Each manifest is a template in which the variables are exanded before being deployed to the cluster defined by kubeconfig.

Here is the full list of variables defined in vars/main.yml:

convos_namespace: convos
convos_image: nordaaker/convos:release-5.06
convos_container_port: 3000
convos_port: 3000
convos_cert_issuer: letsencrypt
convos_name: convos-chat
convos_url_host: chat.domain.tld
convos_storage_class: nfs-client
convos_storage_size: 10Gi

### Namespace

This namespace template creates a namespace specifically for this workload:

apiVersion: v1 kind: Namespace metadata: name: {{ convos_namespace }}


### Persistent Volume

The persistent volume is needed to allow data to be stored on the filesystem and accessible outside of the containers. I've used an NFS mount on the QNAP with the nfs-client storage class so that a directory is automatically created with a name of <namespace>-<workload>-data-pvc-<persistent volume claim ID>. When the volume is de-provisioned, the directory is pre-pending with "archived" so it can be deleted or backed up as appropriate. This could any other storage provisioner class available.

In the case of Convos, what  is stored on the persistent volume is the user history and configuration. Inside the container it will be mounted as /data.

piVersion: v1 kind: PersistentVolumeClaim metadata: name: {{ convos_name }}-data namespace: {{ convos_namespace }} spec: accessModes:

  • ReadWriteOnce resources: requests: storage: {{ convos_storage_size }} storageClassName: {{ convos_storage_class }}

### Deployment

The deployment is the container definition itself, including the environment variables to configure it, which will become a running pod:

apiVersion: apps/v1 kind: Deployment metadata: name: {{ convos_name }} namespace: {{ convos_namespace }} labels: app: blog spec: replicas: 1 selector: matchLabels: app: {{ convos_name }} template: metadata: labels: app: {{ convos_name }} spec: containers: - name: {{convos_name}} image: {{ convos_image }} imagePullPolicy: Always ports: - containerPort: {{ convos_container_port }} env: - name: CONVOS_REQUEST_BASE value: https://{{ convos_url_host}}/ volumeMounts: - mountPath: /data name: {{ convos_name }}-data-vol volumes: - name: {{ convos_name }}-data-vol persistentVolumeClaim: claimName: {{ convos_name }}-data


### Service and Ingress

The service defines how it can be accessed within the cluster. This could be something only other services access, such as a database, but in this case we're also going to define an ingress, including an SSL certificate from [Let's Encrypt](https://letsencrypt.org/), to allow clients outside of the cluster to access the application in their web browser:

apiVersion: v1 kind: Service metadata: labels: name: {{ convos_name }}-svc name: {{ convos_name}}-svc namespace: {{ convos_namespace }} spec: ports:

  • name: http port: {{ convos_port }} protocol: TCP targetPort: {{ convos_port }} selector: app: {{ convos_name }} type: NodePort status: loadBalancer: {}

apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: {{ convos_name}}-svc namespace: {{ convos_namespace }} annotations: cert-manager.io/cluster-issuer: {{ convos_cert_issuer }} spec: rules:

  • host: {{ convos_url_host }} http: paths:
    • backend: serviceName: {{ convos_name }}-svc servicePort: {{ convos_port }} path: / tls:
  • hosts:
    • {{ convos_url_host }} secretName: {{ convos_name }}-cert

That's it, running "ansible-playbook k8s-workloads.yml" from my local machine or as a scheduled  job from within AWX will deploy Convos to the cluster. It can be run as often as you like and, because of the [idempotency](https://en.wikipedia.org/wiki/Idempotence), nothing will change. If a new version of Convos is released, for example, updating the value of the "convos_image" variable with the new image tag (it's bad practice to use "latest") and running the playbook will deploy the new version while preserving everything else.

ansible-playbook 2.10.5 config file = /etc/ansible/ansible.cfg configured module search path = [’/var/lib/awx/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’] ansible python module location = /opt/custom-venvs/k8s/lib/python3.6/site-packages/ansible executable location = /opt/custom-venvs/k8s/bin/ansible-playbook python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] Using /etc/ansible/ansible.cfg as config file Vault password: redirecting (type: modules) ansible.builtin.k8s to community.kubernetes.k8s Skipping callback ‘awx_display’, as we already have a stdout callback. Skipping callback ‘default’, as we already have a stdout callback. Skipping callback ‘minimal’, as we already have a stdout callback. Skipping callback ‘oneline’, as we already have a stdout callback.

PLAYBOOK: k8s-workloads.yml **************************************************** 1 plays in k8s-workloads.yml

PLAY [Deploy Workloads on Kubernetes] ******************************************

TASK [Decrypt kubeconfig] ****************************************************** task path: /tmp/awx_2512_8isyjehe/project/k8s-workloads.yml:9 changed: [127.0.0.1] => {“changed”: true} META: ran handlers

TASK [convos : Deploy Convos Namespace Manifest] ******************************* task path: /tmp/awx_2512_8isyjehe/project/roles/convos/tasks/main.yml:2 redirecting (type: modules) ansible.builtin.k8s to community.kubernetes.k8s ok: [127.0.0.1] => {“changed”: false}

TASK [convos : Deploy Convos Volume Manifest] ********************************** task path: /tmp/awx_2512_8isyjehe/project/roles/convos/tasks/main.yml:11 redirecting (type: modules) ansible.builtin.k8s to community.kubernetes.k8s ok: [127.0.0.1] => {“changed”: false}

TASK [convos : Deploy Convos Deployment Manifest] ****************************** task path: /tmp/awx_2512_8isyjehe/project/roles/convos/tasks/main.yml:20 redirecting (type: modules) ansible.builtin.k8s to community.kubernetes.k8s ok: [127.0.0.1] => {“changed”: false}

TASK [convos : Deploy Convos Service Manifest] ********************************* task path: /tmp/awx_2512_8isyjehe/project/roles/convos/tasks/main.yml:29 redirecting (type: modules) ansible.builtin.k8s to community.kubernetes.k8s ok: [127.0.0.1] => {“changed”: false}

TASK [convos : Deploy Convos Ingress Manifest] ********************************* task path: /tmp/awx_2512_8isyjehe/project/roles/convos/tasks/main.yml:38 redirecting (type: modules) ansible.builtin.k8s to community.kubernetes.k8s ok: [127.0.0.1] => {“changed”: false}

META: ran handlers

PLAY RECAP ********************************************************************* 127.0.0.1 : ok=6 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0