Kubernetes CrowdSec Integration – Part 1: Detection
In this article, we will see how to install CrowdSec in a Kubernetes (K8s) cluster, configure it to monitor the applications of our choice, and detect attacks on those applications.
The microservice architecture is the most significant security challenge in a Kubernetes (K8s) cluster. Every application you deploy opens a new potential entry for attackers, increasing the attack surface.
As deployed applications generate logs and CrowdSec can run in a container… You see where I am going with this.
In this blog post, we will see how to install CrowdSec in a K8s cluster, configure it to monitor the applications we want, and see the magic happen by detecting attacks on those applications.
The first part will cover the detection and the second, which is still a work in progress, will tackle the prevention.
Here’s an architecture overview of CrowdSec inside a K8s cluster.
Before starting this guide, ensure you have:
- (Configured AWS account + eksctl) or an existing K8s cluster (ensure your cluster has enough resources available, at least 1 CPU + 1GB of RAM).
Ready to start? Let’s play!
Setup the test environment
Deploying K8s cluster
For people who aren’t lucky enough to have a K8s cluster, we will deploy one easily using AWS EKS, Eksctl.
Here is a simple configuration that will deploy a cluster with 1 t2.small node.
If you want more nodes, you just need to change desiredCapacity.
We can run our command to create the cluster:
When deploying your cluster, you will be able to collect some information.
Deploying Nginx Ingress Controller
First, we need to install nginx helm repo
We can now install Nginx ingress controller using this chart with those following values to support proxy-protocol and have the Source IP in the logs. So you need to create new file ingress-nginx-values.yaml where you will post those values:
You can install the Nginx ingress controller in using this command:
A new LoadBalancer will be created in your AWS account, you need to enable manually "Proxy protocol v2".
Once installed, you can check ingress controller pods in the new namespace:
Install HelloWorld application
To install an application example, we released a HelloWorld application that can be deployed using the Nginx ingress controller. This helm chart is hosted in our charts repository.
We first need to install the helm repo, secondly the HelloWorld application, and then CrowdSec.
Then update the repositories to get the new charts
Now, we can install the HelloWorld chart with default values in the default namespace.
To access this URL, you need to retrieve the public IP and modify your hosts’ file:
This command might take some time to get a result when using the EKS AWS service.
We can modify the hosts’ file and add one of the public IP addresses:
Now our application is reachable:
We can also look at the Nginx ingress controller logs to see HTTP logs:
Our environment is ready. Let’s deep dive into the exciting part.
Crowdsec chart is also available in our charts repositories.
First, we need to create a new namespace so that CrowdSec will be isolated:
We want to monitor Nginx ingress controller logs because our application is deployed behind the Nginx ingress controller.
We can create a new file crowdsec-values.yaml, containing the CrowdSec chart configuration.
If you want to modify the Docker image environment variables, you can follow this guide.
Now we can install CrowdSec using our config file in the CrowdSec namespace we created previously.
And we can see that our LAPI (CrowdSec’s local API) and agent are running:
To test whether CrowdSec detects attacks, we will simulate an attack on the HelloWorld application using Nikto and see CrowdSec metrics and alerts.
We will launch the attack with this command once Nikto is installed.
(launch the attack and cancel it after 20 seconds just to generate some attacks on the application)
Now we can get a shell into the CrowdSec agent pod and see metrics and alerts:
Metrics show files read by CrowdSec (in the acquisition table) and how many are parsed/unparsed, all scenarios triggered by the logs (in the bucket table). As we installed the collection crowdsecurity/Nginx, it comes with multiple scenarios that detect HTTP attacks.
Now let’s see if the CrowdSec agent detects something:
Those are alerts raised by the CrowdSec agent following our Nikto scan. We can see that several scenarios were triggered and the agent sent ban decisions to the LAPI. It means that it will be stored and shared with the bouncers to block this IP.
If you want to clean up your cluster, follow those steps:
If you deployed your K8s cluster following this tutorial, you just need to remove it with eksctl
If you used an existing cluster, you need to delete the helm charts we installed:
Uninstall the Nginx ingress controller if you installed it using this tutorial:
This tutorial looked at the basics of installing CrowdSec in Kubernetes and how it can easily detect attacks. But detection without prevention is not a complete process. We are currently working on a bouncer integration in Kubernetes. We will start with integration in the Nginx ingress controller using the Lua bouncer. It will need some core changes to adapt it, but nothing complicated.
So, it means that we will meet again in a future article dealing with the bouncer part.
About the author
Coming from a sys admin then pentester/secops background, Hamza is now DevSecOps at CrowdSec. He is also a member of the core team.