Secure Docker Compose stacks with CrowdSec
This article explains how to make CrowdSec and Docker Compose work together to protect applications exposed in containers. It should allow us to:
- Automatically ban malevolent IPs from accessing our container services
- Manually add/remove and inspect ban decisions
- Monitor CrowdSec’s behavior (via cli and dashboards)
The chart below shows a glimpse of how our target architecture will look:
Let’s create a Docker Compose file that will setup the following:
- A reverse-proxy that uses Nginx
- A sample application that exposes an Apache2 “hello world”
- A CrowdSec container that reads the reverse-proxy’s logs to detect attacks on the HTTP service
- A Metabase container that will generate fancy dashboards showing what has been happening
We have chosen the simplest way to collect logs: by sharing volumes between containers. If you are in production, you are probably using a logging-driver to centralize logs with rsyslog or another driver, so don't forget to adapt the CrowdSec Docker Compose configuration to read your logs properly.
The docker-compose.yml file is as follow:
The reverse-proxy (nginx) container writes its logs to a logs volume mounted by the crowdsec container.
CrowdSec's SQLite database is in a crowdsec-db volume mounted by the dashboard (metabase) container
We have put the configuration files altogether on this repository, so that you can simply clone it to deploy.
From the Docker Compose directory, you can deploy with docker-compose up -d and then check that everything is running with docker-compose ps.
Let's check and make sure everything is working!
Demo app check
With the following command, we can check whether access to our demo app is working properly.
Demo app check
We need to validate whether our CrowdSec setup reads logs as expected.
What happened and what is relevant here?
The command cscli metrics queries the Prometheus metrics exposed locally by CrowdSec and presents them in a fancy terminal output:
- The “acquisition metrics” show us that our requests are indeed generating logs that are being read (“LINES READ”), parsed (“LINES PARSED”) and even matched with installed scenarios (“LINES POURED TO BUCKET”)
- The “buckets metrics” and “parser metrics” allow us to see which parsers and scenarios are being triggered
CrowdSec configuration check
The command cscli hub list allows us to see which parsers and scenarios are deployed.
Metabase is one of the components that has been deployed, which helps us generate dashboards for better observability. You can hop onto http://127.0.0.1:3000/ and log in with firstname.lastname@example.org and password !!Cr0wdS3c_M3t4b4s3??
Metabase comes with a default password because of how it is deployed. Remember to change the default password and restrict the metabase's access to relevant IP addresses or network ranges.
At first, dashboards will be empty since no attacks were detected yet. The main one should look like this:
If any of those checks have failed for you, take a look at container logs with docker-compose logs crowdsec (for example).
Note: In real-world setups, whitelists are deployed to prevent banning private IPs.
After checking to make sure everything is ready to go, let's try some detection features. As we work with an exposed HTTP service, let's fire a Nikto from another machine in the LAN nikto -host http://192.168.2.227:8000
Note: The IP is dependent on your LAN setup and addressing plan.
On the other hand, we kept an eye on CrowdSec's logs through the following command.
docker-compose logs -f crowdsec
Here we can spot our client’s IP (192.168.2.211) being flagged for triggering various scenarios:
- crowdsecurity/http-bad-user-agent: the IP comes from a known bad user agent
- crowdsecurity/http-probing: tried to access a lot of distinct non-existent files
- crowdsecurity/http-crawl-non_statics: attempted to access a lot of distinct non-static ressources
- crowdsecurity/http-sensitive-files: IP tried to access a lot of sensitive files
- crowdsecurity/http-path-traversal-probing: IP tried to perform a path traversal attack
We can see my IP is banned with docker-compose exec crowdsec cscli decisions list.
We can review and inspect alerts with docker-compose exec crowdsec cscli alerts list and docker-compose exec crowdsec cscli alerts inspect -d XX:
Note: cscli alerts list returns the list of all alerts triggered.
Note: cscli alerts inspect -d <ID> allows you to generate more details about a given alert.
Monitoring activities with dashboards
Now that we have triggered several scenarios, we can go back to our Metabase dashboards (http://127.0.0.1:3000 with the default setup) and check the activity.
If the traffic came from a public IP (rather than a private one, as in this example), crowdsecurity/geoip-enrich would have enriched events with geo-localization data and AS/range information.
Block attacks with bouncers
Now that we have a fully functional CrowdSec service, we can detect incoming attacks on our service based on installed collection scenarios.
After detecting these attacks, our objective is to block them. You will use the cs-firewall-bouncer to achieve this. Start by installing it on the host, and block malevolent traffic directly in the DOCKER-USER chain, the default chain created by Docker to filter traffic targeting the containers.
You can find the firewall bouncer on the CrowdSec Hub. As of today, the most up-to-date version is v0.10.
Since July, custom and firewall bouncers are packaged with Debian, Ubuntu, CentOS, RHEL and Amazon Linux. Please check our GitHub repo for more information. The install is straightforward. It will deploy a systemd unit for the service and make sure you meet the requirements. Here, I didn't have ipset running, and it installed it for me.
Here, we installed the bouncer on a host where CrowdSec isn't running. As a result, the service isn't happy.
Now, let's configure the bouncer to speak with the Local API running on our CrowdSec container. We start by creating an API token for our bouncer with cscli.
docker-compose exec crowdsec cscli bouncers add HostFirewallBouncer
Then, you need to configure the bouncer to use this token to authenticate with CrowdSec's Local API. In the /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml, edit api_url, api_key, and iptables_chains. In this case, IPv6 was also disabled with disable_ipv6:
Note: We edited the chains to only have DOCKER-USER, and we set api_url accordingly to our docker-compose.yml file, along with the newly generated api token.
You shall not pass: CrowdSec in action
Now, we can get our freshly configured bouncer started with sudo systemctl start cs-firewall-bouncer.service and take a look at our new firewall configuration:
We can see that our DOCKER-USER chain has been populated with a rule to match incoming traffic against our ipset, and our ipset is filled with relevant information.
As you might have noticed by now, our ipset isn't only filled with our local decisions but also with ones from the community. However, we can see that our "local" decisions made their way to the ipset list.
Now, we can check the attacker machine and see we are blocked and cannot access the application.
The attacker is prevented from accessing all Docker Compose applications. We limited the decision to the DOCKER-USER chain on purpose, which means local applications exposed by the host would still be accessible. If we wish to extend the ban to all incoming traffic, we could have added the INPUT chain to the list, as is the case with the default setup.
Throughout this tutorial, we deployed a minimal yet complete applicative stack using Docker Compose. Then, we covered how to secure it with CrowdSec. While most people will use CrowdSec as a host-based defense mechanism, we can see that it's also suitable for Docker environments. If you want to meet with the team to share your feedback, you can find us on our Gitter channel.
About the author
Thibault Koechlin graduated from EPITECH, specializing in the security of IT systems & networks. He started his career at NBS in 2004, as an expert in penetration testing before being appointed Head of the offensive security team. He then became CISO by expanding his skills around defensive security before initiating the development of several open-source products and building teams with rare skills. He completed his ascent within the company through an operational partner role, leading the creation of the company's flagship product: Cerberhost. In December 2019, he co-founded CrowdSec with Philippe Humeau and Laurent Soubrevilla. He is the CTO of the company.