How to set up a CrowdSec multi-server installation
A few months ago, we added some interesting features to CrowdSec when releasing v1.0.x. One of the most exciting ones, is the ability of the CrowdSec agent to act as an HTTP rest API to collect signals from other CrowdSec agents. Thus, it is the responsibility of this special agent to store and share the collected signals. We will call this special agent the LAPI server from now on. Another worth noting feature, is that mitigation no longer has to take place on the same server as detection. Mitigation is done using bouncers. Bouncers rely on the HTTP REST API served by the LAPI server.
This article has been published by Linux Journal. Its source can be found here.
In this article we will describe how to deploy CrowdSec in a multi-server setup with one server sharing signal.
Both server-2 and server-3 are meant to host services. You can take a look on our Hub to know which services CrowdSec can help you secure. Last but not least, server-1 is meant to host the following local services:
- the local API needed by bouncers
- the database fed by both the three local CrowdSec agents and the online CrowdSec blocklist service. As server-1 is serving the local API, we will call it the LAPI server
We choose to use a postgresql backend for CrowdSec database in order to allow high availability. This topic will be covered in future posts. If you are ok with no high availability, you can skip step 2.
Furthermore this post will cover attack mitigation for hosted services on server-2 and server-3 using CrowdSec bouncers.
This article is the first of a series.
- Two Internet-facing preinstalled Ubuntu 20.04 servers hosting services. From now on, we will refer to these servers by server-2 and server-3 .
- One non Internet facing preinstalled Ubuntu 20.04 server. From now on we will refer to this server by server-1. Let’s assume that server-1's IP is 10.0.0.1. (no internet connection on this server is not a strict requirement).
- A local network connecting all three servers
Step 1: CrowdSec installation
Let’s install CrowdSec on every single server, following the CrowdSec installation guide.
We now have three standard CrowdSec installations running.
Step 2 (Optional): Switch the database backend to postgresql on server-1
First we have to connect to the database as a postgres user.
Thanks to the Postgresql CrowdSec documentation, we can now initialize the database.
Now let’s make CrowdSec know about this new database backend. To achieve this, we will have to update the db_config section of the /etc/crowdsec/config.yaml file.
After registering the local machine again in the database, we are able to restart CrowdSec:
Step 3: Make server-2 and server-3 report to LAPI server
First we have to configure CrowdSec on server-1 to accept connections from server-2 and server-3. Please ensure that your firewall allows connections from server-2 and server-3 on server-1's port 8080.
Let’s configure the API server on server-1 side. For this to happen, we will have to modify both /etc/crowdsec/config.yaml and /etc/crowdsec/local_api_credentials.yaml.
For /etc/crowdsec/config.yaml, it is now the API section that needs to be amended. It's only a matter of adding a # in front of listen_uri to make it a yaml comment.
For /etc/crowdsec/local_api_credentials.yaml we only have to change the configured ip accordingly:
And we can restart CrowdSec:
Now we will configure the connections on server-2 and server-3.
First we register to the LAPI server on both server-2 and server-3:
By default, the local api server is active on every CrowdSec agent installation. In this setup, we want to disable it on server-2 and server-3. To achieve this, we need to tweak the CrowdSec agent systemd service file.
Now edit /etc/systemd/system/crowdsec.service and add the -no-api parameter to CrowdSec agent invocation on both server-2 and server-3.
We can now acknowledge the changes and restart CrowdSec once again.
Last thing to do is to allow server-2 and server-3 connections on server-1.
In this output, we can see two machines that are not yet validated. Let’s validate them now.
server-2 and server-3 are now allowed to push data to server-1 CrowdSec agent. It may be needed to restart CrowdSec on server-2 and server-3.
On server-1, the command sudo cscli machines list should now show three validated machines.
Step 4: Set up Mitigation
Now we want to install mitigation on our internet-facing servers. First we have to generate two API tokens for server-2 and server-3 on server-1.
For now, there is no package available for the firewall bouncer. This is on our high priority to-do list.
Hence on both server-2 and server-3:
If iptables and nftables are not installed, the install script will ask for your permission to install it. The same will happen for ipset installation.
It is now time to use the token we generated at the beginning of this step. Both api_key and api_url have to be updated in /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml:
We can now restart the firewall bouncer.
Conclusion and perspectives
We described how to setup a CrowdSec multi-server installation. The resource overhead on server-2 and server-3 is quite limited as most of the tasks are deported to server-1. This allows to grow the setup by only:
- register and validate the CrowdSec agent on the LAPI server
- add and validate new bouncers
It is worth noting that bouncers and CrowdSec agents don’t have to be installed on the same server. Therefore, the CrowdSec agent has to be installed where logs are generated, but the mitigation can be deported where it is meaningful.
Obviously, there are caveats in this setup:
- Communications between agents are over clear http. This is acceptable on a local network, but not possible over the Internet. CrowdSec allows the use of https for those communications, a next post will cover this topic.
- Monitoring or alerting is not covered in this article either. CrowdSec allows very powerful monitoring through Prometheus scraper. A post will cover this topic too.
- The CrowdSec database is not highly available. Furthermore, the CrowdSec agent on server-1 is a single point of failure.
Now you may be wondering: how to build a highly available multi-machine CrowdSec setup? Stay tuned for our next article.
We are always more than happy to receive feedback about the solution and its usage. If you would like to meet with us and have a chat, drop by our Gitter channel or visit our GitHub repo.