Crowdsec is a collaborative IPS (Intrusion Prevention System) which can process data sources such as logs and act locally while utilizing community intelligence through scenarios and blocklists. In addition, it can actively block, using bouncers, based on the outcome of a local decision.

The Crowdsec agent parses local data sources such as logs to detect suspicious behaviors using parsers and scenarios. It passes the bad behaviors detected to the Local API which is the main hub for all of the Crowdsec components to communicate. The Local API will share decisions with and receive community blocklists from the Crowdsec Central API. Bouncers will communicate with the Local API to actively block IP address ranges which have been determined to be malicious.

The Crowdsec CLI is the command-line interface to the Local API and can be used to change configurations, view alerts and decisions, and remove blocks. The Crowdsec console is the shared, cloud-based component which lets you see all of your Crowdsec instances in one place.

Before I share how I deployed Crowdsec, it may be worthwhile to review the official blog post from Crowdsec, which I used. Specifically, they deployed a HelloWorld application and used Nikto to attack it to demonstrate that it works. I initially skipped this step, but I later found it helpful for troubleshooting the Traefik bouncer.

First, I had to create a Crowdsec console login because it’s needed to enroll the instance. Once logged into the console, I just clicked Add Instance to get an enrollment key.

K3S Crowdsec Ansible Role

The Crowdsec role will deploy the Crowdsec agent, local API, and the Traefik bouncer using the official Helm Charts. The Crowdsec agent will ingest the Traefik logs, but it can ingest the logs of any container if configured. The Traefik bouncer is installed as a global middleware and will return a 403 Forbidden response for any Local API decision. This effectively means that any application running on the cluster with an ingress can be protected.

As I’ve done with other deployments, I have a k3s_cluster inventory file with associated group variables stored in my Ansible repository. So I’ll add the Crowdsec-related variables:

inventory/k3s_cluster/group_vars/k3s_cluster:

crowdsec_namespace: crowdsec
crowdsec_image_tag: v1.4.6
crowdsec_dashboard_host: crowdsec-dashboard.domain.tld
crowdsec_enroll_key: console_enrollment_key
crowdsec_traefik_bouncer_key: local_api_key # To be retrieved once the agent is deployed

This Ansible role probably should have been split into a role for deploying the main chart and a separate role for the bouncer. The reason is, in order to deploy the bouncer, a key from the Local API is needed. Instead I just deployed the main chart first and then added a task to deploy the bouncer chart once I had the API key.

roles/k3s_cluster/tasks/main.yml:

- name: Crowdsec namespace
  kubernetes.core.k8s:
    kubeconfig: "/var/lib/rancher/k3s/server/cred/admin.kubeconfig"
    state: present
    definition: "{{ lookup('template', 'manifests/namespace.j2') }}"
  delegate_to: "{{ ansible_host }}"
  run_once: true
- name: Add Crowdsec chart repo
  kubernetes.core.helm_repository:
    name: crowdsec
    repo_url: "https://crowdsecurity.github.io/helm-charts"
  delegate_to: "{{ ansible_host }}"
  run_once: true
- name: Install Crowdsec Agent Chart
  kubernetes.core.helm:
    kubeconfig: "/var/lib/rancher/k3s/server/cred/admin.kubeconfig"
    release_name: crowdsec
    chart_ref: crowdsec/crowdsec
    release_namespace: "{{ crowdsec_namespace }}"
    update_repo_cache: yes
    values:
      container_runtime: containerd
      image:
        tag: "{{ crowdsec_image_tag }}"
      config:
        postoverflows: # https://docs.crowdsec.net/docs/next/whitelist/create/#whitelist-in-postoverflows
          s01-whitelist:
            localips.yaml: |
              name: "crowdsecurity/whitelists"
              description: "Whitelist events from my ip addresses"
              whitelist:  
                reason: "my ip ranges"
                ip: 
                  - "192.168.0.x" # Replace first three octets with actual local IP subnet              
      agent:
        # To specify each pod you want to process logs (pods present in the node)
        acquisition:
          # The namespace where the pod is located
          - namespace: kube-system
            # The pod name
            podName: traefik-*
            # we need to specify the program name so the parser will match and parse logs
            program: traefik
        # Those are ENV variables
        env:
          # As we are running Traefik, we want to install the traefik collection
          - name: PARSERS
            value: "crowdsecurity/cri-logs"
          - name: COLLECTIONS
            value: "crowdsecurity/traefik"
          - name: DISABLE_PARSERS
            value: "crowdsecurity/whitelists"
        persistentVolume:
          config:
            enabled: false
      lapi:
        dashboard:
          enabled: false
          ingress:
            host: "{{ crowdsec_dashboard_host }}"
            enabled: false
        env:
          - name: ENROLL_KEY
            value: "{{ crowdsec_enroll_key }}"
          - name: ENROLL_INSTANCE_NAME
            value: "my_k3s"
          - name: ENROLL_TAGS
            value: "k3s kubernetes"
          # If it's a test, we don't want to share signals with CrowdSec so disable the Online API.
          #- name: DISABLE_ONLINE_API
          #  value: "true"

Only one additional manifest, for the namespace, is needed for this role.

roles/k3s_cluster/manifests/namespace.yml:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ crowdsec_namespace }}

The Ansible playbook to execute this role will be needed to deploy the charts.

k3s-crowdsec.yml:

---
- hosts: master[0]
  become: yes
  vars:
    ansible_python_interpreter: /usr/bin/python3
  remote_user: ansible
  pre_tasks:
    - name: Install Kubernetes Python module
      pip:
        name: kubernetes
    - name: Install Kubernetes-validate Python module
      pip:
        name: kubernetes-validate
  roles:
    - role: k3s_cluster/crowdsec

Once I’ve run the playbook and the Local API has been deployed, I need to to log back into the console to accept the instance enrollment. Once accepted, I had to kill the pod so that it restarts and starts sharing signals:

$ kubectl -n crowdsec get pods
NAME                               READY   STATUS    RESTARTS   AGE
crowdsec-agent-5pzwr               1/1     Running   0          99m
crowdsec-agent-7fm2b               1/1     Running   0          99m
crowdsec-agent-c57pq               1/1     Running   0          99m
crowdsec-agent-jcl5z               1/1     Running   0          99m
crowdsec-agent-whhvt               1/1     Running   0          99m
crowdsec-agent-xz6b8               1/1     Running   0          99m
crowdsec-lapi-5cddd77b48-smz4b     1/1     Running   0          98m

$ kubectl -n crowdsec delete pod crowdsec-lapi-5cddd77b48-smz4b

Note: There will be one agent pod per k3s node.

Traefik Bouncer

In order to add a bouncer, I need to get an API key from the Local API. To do this, I execute a shell in the lapi pod:

$ kubectl -n crowdsec get pods
NAME                               READY   STATUS    RESTARTS   AGE
crowdsec-agent-5pzwr               1/1     Running   0          99m
crowdsec-agent-7fm2b               1/1     Running   0          99m
crowdsec-agent-c57pq               1/1     Running   0          99m
crowdsec-agent-jcl5z               1/1     Running   0          99m
crowdsec-agent-whhvt               1/1     Running   0          99m
crowdsec-agent-xz6b8               1/1     Running   0          99m
crowdsec-lapi-3bdde7ln63-eab6p     1/1     Running   0          98m

$ kubectl -n crowdsec exec -it crowdsec-lapi-3bdde7ln63-eab6p  -- sh

Once I have a shell in the lapi pod, I can use the Crowdsec CLI command, cscli, to get an API key:

$ cscli bouncers add traefik-ingress

Now I update the crowdsec_traefik_bouncer_key variable in inventory/k3s_cluster/group_vars/k3s_cluster with the new API key and add the task to deploy the Traefik bouncer chart.

Add an additional task to roles/k3s_cluster/tasks/main.yml and execute the k3s-crowdsec playbook again:

- name: Install Crowdsec Traefik Bouncer Chart
  kubernetes.core.helm:
    kubeconfig: "/var/lib/rancher/k3s/server/cred/admin.kubeconfig"
    release_name: traefik-bouncer
    chart_ref: crowdsec/crowdsec-traefik-bouncer
    release_namespace: "{{ crowdsec_namespace }}"
    update_repo_cache: yes
    values:
      bouncer:
        crowdsec_bouncer_api_key: "{{ crowdsec_traefik_bouncer_key }}"
        crowdsec_agent_host: "crowdsec-service.crowdsec.svc.cluster.local:8080"

Traefik Middleware

With the bouncer deployed, I can add the new middleware to any existing ingress, or I can update the Traefik config to deploy it as a global middleware. The way that k3s deploys Traefik is a bit murky so I wouldn’t blame anyone who finds this task daunting.

The best explanation I can give is that k3s uses a HelmChartConfig manifest which should redeploy the helm chart for Traefik, which ships with k3s, when the server is restarted. However, I ran into some difficulty getting the changes to take effect and accidentally uninstalled Traefik completely. In the heat of battle, I neglected to document the exact steps so feel free to contact me if you need some help working through it.

The name of the middleware follows the pattern of namespace-applabel@kubernetescrd. So, in my deployment it’s called “crowdsec-traefik-bouncer@kubernetescrd”. You can see in this excerpt of the deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: traefik-bouncer
    meta.helm.sh/release-namespace: crowdsec
  generation: 1
  labels:
    app: traefik-bouncer # applabel
    app.kubernetes.io/managed-by: Helm
  name: traefik-bouncer
  namespace: crowdsec # namespace

I mention this because the Traefik CRD can have a difficult time parsing namespaces with hyphens in them. Initially I deployed the Traefik bouncer in the same namespace as Traefik, kube-system, and it wasn’t working. Once I deployed it into the crowdsec namespace, it worked fine.

Global Crowdsec Bouncer

Rather than recreate my entire Traefik Ansible role here, I’ll just refer to the changes made since my previous post covering the Traefik role.

I added entries to the additionalArguments section of the Traefik HelmChartConfig for Crowdsec:

      # Crowdsec bouncer global middleware
      - --accesslog=true
      - --entrypoints.web.http.middlewares=crowdsec-traefik-bouncer@kubernetescrd
      - --entrypoints.websecure.http.middlewares=crowdsec-traefik-bouncer@kubernetescrd

Crowdsec Bouncer As Middleware

Alternatively, the bouncer middleware can be added on individual Ingress or IngressRoute:

Kubernetes Ingress:

ingress:
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: crowdsec-traefik-bouncer@kubernetescrd

Traefik IngressRoute:

spec:
  routes:
    middlewares:
      - name: crowdsec-traefik-bouncer@kubernetescrd

Put it all together

On the Traefik dashboard, I can verify that the middleware is active. Prior to adding my IP to the whitelist, I followed the official blog post I mentioned earlier to attack the HelloWorld application and used the cscli command to view and delete decisions. Once I was confident that it was working properly, I added my IP to the whitelist to avoid locking myself out.

Summary

Crowdsec is a fantastic project with many possible deployment options. The tutorials section of the blog is a treasure trove of How To’s for everything from securing Windows servers to firewall appliances. I’ve already deployed Crowdsec to protect my mail server using the Docker collection with the Linux firewall bouncer. I’m working on an Ansible role for this so I can easily secure as much of my infrastructure as possible.