Oops! Something went wrong while submitting the form.
Tutorial
November 2, 2021
9
min. read
Kubernetes CrowdSec Integration – Part 1: Detection
In this article, we will see how to install CrowdSec in a Kubernetes (K8s) cluster, configure it to monitor the applications of our choice, and detect attacks on those applications.
Introduction
The microservice architecture is the most significant security challenge in a Kubernetes (K8s) cluster. Every application you deploy opens a new potential entry for attackers, increasing the attack surface.
As deployed applications generate logs and CrowdSec can run in a container… You see where I am going with this.
In this blog post, we will see how to install CrowdSec in a K8s cluster, configure it to monitor the applications we want, and see the magic happen by detecting attacks on those applications.
The first part will cover the detection and the second, which is still a work in progress, will tackle the prevention.
Architecture
Here’s an architecture overview of CrowdSec inside a K8s cluster.
CrowdSec architecture inside a K8s cluster
Prerequisites
Before starting this guide, ensure you have:
(Configured AWS account + eksctl) or an existing K8s cluster (ensure your cluster has enough resources available, at least 1 CPU + 1GB of RAM).
If you want more nodes, you just need to change desiredCapacity.
We can run our command to create the cluster:
$ eksctl create cluster -f cluster.yaml
2021-09-17 17:39:41 [ℹ] eksctl version 0.62.0
2021-09-17 17:39:41 [ℹ] using region eu-west-1
2021-09-17 17:39:41 [ℹ] setting availability zones to [eu-west-1a eu-west-1c eu-west-1b]
2021-09-17 17:39:41 [ℹ] subnets for eu-west-1a - public:192.168.0.0/19 private:192.168.96.0/19
2021-09-17 17:39:41 [ℹ] subnets for eu-west-1c - public:192.168.32.0/19 private:192.168.128.0/19
2021-09-17 17:39:41 [ℹ] subnets for eu-west-1b - public:192.168.64.0/19 private:192.168.160.0/19
2021-09-17 17:39:41 [ℹ] nodegroup "ng-1" will use "ami-044dfe22e0788d8ed" [AmazonLinux2/1.21]
2021-09-17 17:39:41 [ℹ] using SSH public key "/home/.ssh/id_rsa.pub" as "eksctl-alpaga-nodegroup-ng-1-7f:94:de:2d:82:df:52:b1:e1:56:4b:a6:7f:2e:91:72"
2021-09-17 17:39:42 [ℹ] using Kubernetes version 1.21
2021-09-17 17:39:42 [ℹ] creating EKS cluster "alpaga" in "eu-west-1" region with un-managed nodes
2021-09-17 17:39:42 [ℹ] 1 nodegroup (ng-1) was included (based on the include/exclude rules)
2021-09-17 17:39:42 [ℹ] will create a CloudFormation stack for cluster itself and 1 nodegroup stack(s)
...
2021-09-17 18:00:57 [ℹ] nodegroup "ng-1" has 0 node(s)
2021-09-17 18:00:57 [ℹ] waiting for at least 1 node(s) to become ready in "ng-1"
2021-09-17 18:01:49 [ℹ] nodegroup "ng-1" has 1 node(s)
2021-09-17 18:01:49 [ℹ] node "ip-192-168-68-125.eu-west-1.compute.internal" is ready
2021-09-17 18:03:51 [ℹ] kubectl command should work with "/home/.kube/config", try 'kubectl get nodes'
2021-09-17 18:03:51 [✔] EKS cluster "alpaga" in "eu-west-1" region is ready
When deploying your cluster, you will be able to collect some information.
$ kubectl cluster-info
Kubernetes control plane is running at https://XXXXXXXXXXXXXXXXXXXXXXXXXXXX.gr7.eu-west-1.eks.amazonaws.com
CoreDNS is running at https://XXXXXXXXXXXXXXXXXXXXXXXXXXXX.gr7.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
We can now install Nginx ingress controller using this chart with those following values to support proxy-protocol and have the Source IP in the logs. So you need to create new file ingress-nginx-values.yaml where you will post those values:
A new LoadBalancer will be created in your AWS account, you need to enable manually "Proxy protocol v2".
Once installed, you can check ingress controller pods in the new namespace:
$ kubectl -n ingress-nginx get pods
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-54bfb9bb-rmwqh 1/1 Running 0 6m31s
Install HelloWorld application
To install an application example, we released a HelloWorld application that can be deployed using the Nginx ingress controller. This helm chart is hosted in our charts repository.
We first need to install the helm repo, secondly the HelloWorld application, and then CrowdSec.
$ helm repo add crowdsec https://crowdsecurity.github.io/helm-charts
"crowdsec" has been added to your repositories
Then update the repositories to get the new charts
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "crowdsec" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Now, we can install the HelloWorld chart with default values in the default namespace.
$ helm install helloworld crowdsec/helloworld
W0920 12:22:22.434028 298463 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0920 12:22:22.792661 298463 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME: helloworld
LAST DEPLOYED: Mon Sep 20 12:22:21 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
http://helloworld.local/
To access this URL, you need to retrieve the public IP and modify your hosts’ file:
$ kubectl get ingresses.networking.k8s.io
NAME CLASS HOSTS ADDRESS PORTS AGE
helloworld helloworld.local a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com 80 2m18s
$ host a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com
a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com has address 52.31.225.95
a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com has address 54.73.240.30
a8523ec3bb4024bc3a7f8b26294013f3-c7ea4e370eaf1195.elb.eu-west-1.amazonaws.com has address 176.34.92.134
This command might take some time to get a result when using the EKS AWS service.
We can modify the hosts’ file and add one of the public IP addresses:
echo "52.31.225.95 helloworld.local" | sudo tee -a /etc/hosts
Now our application is reachable:
$ curl -v http://helloworld.local
* Trying 52.31.225.95:80...
* TCP_NODELAY set
* Connected to helloworld.local (52.31.225.95) port 80 (#0)
> GET / HTTP/1.1
> Host: helloworld.local
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 20 Sep 2021 10:38:21 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 13
< Connection: keep-alive
< X-App-Name: http-echo
< X-App-Version: 0.2.3
<
helloworld!
* Connection #0 to host helloworld.local left intact
We can also look at the Nginx ingress controller logs to see HTTP logs:
First, we need to create a new namespace so that CrowdSec will be isolated:
$ kubectl create ns crowdsec
namespace/crowdsec created
We want to monitor Nginx ingress controller logs because our application is deployed behind the Nginx ingress controller.
We can create a new file crowdsec-values.yaml, containing the CrowdSec chart configuration.
agent:
# To specify each pod you want to process it logs (pods present in the node)
acquisition:
# The namespace where the pod is located
- namespace: ingress-nginx
# The pod name
podName: ingress-nginx-controller-*
# as in crowdsec configuration, we need to specify the program name so the parser will match and parse logs
program: nginx
# Those are ENV variables
env:
# As it's a test, we don't want to share signals with CrowdSec so disable the Online API.
- name: DISABLE_ONLINE_API
value: "true"
# As we are running Nginx, we want to install the Nginx collection
- name: COLLECTIONS
value: "crowdsecurity/nginx"
lapi:
env:
# As it's a test, we don't want to share signals with CrowdSec, so disable the Online API.
- name: DISABLE_ONLINE_API
value: "true"
If you want to modify the Docker image environment variables, you can follow this guide.
Now we can install CrowdSec using our config file in the CrowdSec namespace we created previously.
Metrics show files read by CrowdSec (in the acquisition table) and how many are parsed/unparsed, all scenarios triggered by the logs (in the bucket table). As we installed the collection crowdsecurity/Nginx, it comes with multiple scenarios that detect HTTP attacks.
Now let’s see if the CrowdSec agent detects something:
Those are alerts raised by the CrowdSec agent following our Nikto scan. We can see that several scenarios were triggered and the agent sent ban decisions to the LAPI. It means that it will be stored and shared with the bouncers to block this IP.
Clean up
If you want to clean up your cluster, follow those steps:
If you deployed your K8s cluster following this tutorial, you just need to remove it with eksctl
eksctl delete cluster -f cluster.yaml
If you used an existing cluster, you need to delete the helm charts we installed:
This tutorial looked at the basics of installing CrowdSec in Kubernetes and how it can easily detect attacks. But detection without prevention is not a complete process. You can head over to part 2 to follow the guide and set up the remediation part.
About the author
Coming from a sys admin then pentester/secops background, Hamza is now DevSecOps at CrowdSec. He is also a member of the core team.