How to set up a CrowdSec multi-server installation

A few months ago, we added some interesting features to CrowdSec when releasing v1.0.x. One of the most exciting ones, is the ability of the CrowdSec agent to act as an HTTP rest API to collect signals from other CrowdSec agents. Thus, it is the responsibility of this special agent to store and share the collected signals. We will call this special agent the LAPI server from now on. Another worth noting feature, is that mitigation no longer has to take place on the same server as detection. Mitigation is done using bouncers. Bouncers rely on the HTTP REST API served by the LAPI server.

This article has been published by Linux Journal. Its source can be found here.

Goals

In this article we will describe how to deploy CrowdSec in a multi-server setup with one server sharing signal.

Both server-2 and server-3 are meant to host services. You can take a look on our Hub to know which services CrowdSec can help you secure. Last but not least, server-1 is meant to host the following local services:

• the local API needed by bouncers
• the database fed by both the three local CrowdSec agents and the online CrowdSec blocklist service. As server-1 is serving the local API, we will call it the LAPI server

We choose to use a postgresql backend for CrowdSec database in order to allow high availability. This topic will be covered in future posts. If you are ok with no high availability, you can skip step 2.

Furthermore this post will cover attack mitigation for hosted services on server-2 and server-3 using CrowdSec bouncers.

This article is the first of a series.

Prerequisites

• Two Internet-facing preinstalled Ubuntu 20.04 servers hosting services. From now on, we will refer to these servers by server-2 and server-3 .
• One non Internet facing preinstalled Ubuntu 20.04 server. From now on we will refer to this server by server-1. Let’s assume that server-1's IP is 10.0.0.1. (no internet connection on this server is not a strict requirement).
• A local network connecting all three servers

Step 1: CrowdSec installation

Let’s install CrowdSec on every single server, following the CrowdSec installation guide.

wget -qO - https://s3-eu-west-1.amazonaws.com/crowdsec.debian.pragmatic/crowdsec.asc |sudo apt-key add - && echo "deb https://s3-eu-west-1.amazonaws.com/crowdsec.debian.pragmatic/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/crowdsec.list > /dev/null
sudo apt update

sudo apt install crowdsec

We now have three standard CrowdSec installations running.

Step 2 (Optional): Switch the database backend to postgresql on server-1

sudo apt install postgresql

First we have to connect to the database as a postgres user.

sudo -i -u postgres
psql

Thanks to the Postgresql CrowdSec documentation, we can now initialize the database.

postgres=# CREATE DATABASE crowdsec;
CREATE DATABASE
postgres=# CREATE USER crowdsec WITH PASSWORD 'CREATE USER crowdsec WITH PASSWORD '';
CREATE ROLE
postgres=# GRANT ALL PRIVILEGES ON DATABASE crowdsec TO crowdsec;
GRANT

Now let’s make CrowdSec know about this new database backend. To achieve this, we will have to update the db_config section of the /etc/crowdsec/config.yaml file.

db_config:
  log_level: info
  type:	postgres
  user: crowdsec
  password: ""
  db_name: crowdsec
  host: 127.0.0.1
  port: 5432

After registering the local machine again in the database, we are able to restart CrowdSec:

sudo cscli machines add -a
sudo systemctl restart crowdsec

Step 3: Make server-2 and server-3 report to LAPI server

First we have to configure CrowdSec on server-1 to accept connections from server-2 and server-3. Please ensure that your firewall allows connections from server-2 and server-3 on server-1's port 8080.

Let’s configure the API server on server-1 side. For this to happen, we will have to modify both /etc/crowdsec/config.yaml and /etc/crowdsec/local_api_credentials.yaml.

For /etc/crowdsec/config.yaml, it is now the API section that needs to be amended. It's only a matter of adding a # in front of listen_uri to make it a yaml comment.

api:
  client:
    insecure_skip_verify: false
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  server:
    log_level: info
    #listen_uri: 127.0.0.1:8080
    profiles_path: /etc/crowdsec/profiles.yaml
    online_client: # Crowdsec API credentials (to push signals and receive bad IPs)
      credentials_path: /etc/crowdsec/online_api_credentials.yaml

For /etc/crowdsec/local_api_credentials.yaml we only have to change the configured ip accordingly:

url: http://10.0.0.1:8080/
login: 
password: 

And we can restart CrowdSec:

sudo systemctl restart crowdsec

Now we will configure the connections on server-2 and server-3.

First we register to the LAPI server on both server-2 and server-3:

sudo cscli lapi register -u http://10.0.0.1:8080

By default, the local api server is active on every CrowdSec agent installation. In this setup, we want to disable it on server-2 and server-3. To achieve this, we need to tweak the CrowdSec agent systemd service file.

sudo cp /lib/systemd/system/crowdsec.service /etc/systemd/system/crowdsec.service

Now edit /etc/systemd/system/crowdsec.service and add the -no-api parameter to CrowdSec agent invocation on both server-2 and server-3.

[Unit]
Description=Crowdsec agent
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
Environment=LC_ALL=C LANG=C
PIDFile=/var/run/crowdsec.pid
ExecStartPre=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -t
ExecStart=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -no-api
#ExecStartPost=/bin/sleep 0.1
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

We can now acknowledge the changes and restart CrowdSec once again.

sudo systemctl daemon-reload
sudo systemctl restart crowdsec

Last thing to do is to allow server-2 and server-3 connections on server-1.

sudo cscli machines list
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
  NAME                                              IP ADDRESS      LAST UPDATE           STATUS  VERSION                                                            
  --------------------------------------------------------------------------------------------------------------------------------------------------------------------
  dc6f34b3a4994700a2e333df43728701D0iARTSQ6dxiwyMR  10.0.0.1  2021-04-13T12:16:11Z  ✔️  v1.0.9-4-debian-pragmatic-a8b16a66b110ebe03bb330cda2600226a3a862d7
  9f3602d1c9244f02b0d6fd2e92933e75zLVg8zSRkyANxHbC  10.0.0.3   2021-04-13T12:24:12Z  🚫
  ac86209e6f9c4d7d8de43e2ea31fe28ebvde0vWDr46Mpd3L  10.0.0.2   2021-04-13T12:22:28Z  🚫
--------------------------------------------------------------------------------------------------------------------------------------------------------------------

In this output, we can see two machines that are not yet validated. Let’s validate them now.

sudo cscli machines validate 9f3602d1c9244f02b0d6fd2e92933e75zLVg8zSRkyANxHbC
sudo cscli machines validate ac86209e6f9c4d7d8de43e2ea31fe28ebvde0vWDr46Mpd3L

server-2 and server-3 are now allowed to push data to server-1 CrowdSec agent. It may be needed to restart CrowdSec on server-2 and server-3.

sudo systemctl restart crowdsec

On server-1, the command sudo cscli machines list should now show three validated machines.

Step 4: Set up Mitigation

Now we want to install mitigation on our internet-facing servers. First we have to generate two API tokens for server-2 and server-3 on server-1.

sudo cscli bouncers add server-2
Api key for 'server-2':

    02954e85c72cf442a4dee357f0ca5a7c
    
Please keep this key since you will not be able to retrieve it!

sudo cscli bouncers add server-3
Api key for 'server-3':

    3b1030ce0840c343eecd387ac5a3a614

Please keep this key since you will not be able to retrieve it!

For now, there is no package available for the firewall bouncer. This is on our high priority to-do list.
Hence on both server-2 and server-3:

wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.10/cs-firewall-bouncer.tgz
tar zxvf cs-firewall-bouncer.tgz
cd cs-firewall-bouncer-v0.0.10/
sudo ./install.sh

If iptables and nftables are not installed, the install script will ask for your permission to install it. The same will happen for ipset installation.

It is now time to use the token we generated at the beginning of this step. Both api_key and api_url have to be updated in /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml:

mode: iptables
piddir: /var/run/update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
api_url: http://10.0.0.1:8080/
api_key: 02954e85c72cf442a4dee357f0ca5a7c
disable_ipv6: false
#if present, insert rule in those chains
iptables_chains:
- INPUT
#  - FORWARD
#  - DOCKER-USER

We can now restart the firewall bouncer.

sudo systemctl restart cs-firewall-bouncer

Conclusion and perspectives

We described how to setup a CrowdSec multi-server installation. The resource overhead on server-2 and server-3 is quite limited as most of the tasks are deported to server-1. This allows to grow the setup by only:

• register and validate the CrowdSec agent on the LAPI server
• add and validate new bouncers

It is worth noting that bouncers and CrowdSec agents don’t have to be installed on the same server. Therefore, the CrowdSec agent has to be installed where logs are generated, but the mitigation can be deported where it is meaningful.

Obviously, there are caveats in this setup:

• Communications between agents are over clear http. This is acceptable on a local network, but not possible over the Internet. CrowdSec allows the use of https for those communications, a next post will cover this topic.
• Monitoring or alerting is not covered in this article either. CrowdSec allows very powerful monitoring through Prometheus scraper. A post will cover this topic too.
• The CrowdSec database is not highly available. Furthermore, the CrowdSec agent on server-1 is a single point of failure.

Now you may be wondering: how to build a highly available multi-machine CrowdSec setup? Stay tuned for our next article.

We are always more than happy to receive feedback about the solution and its usage. If you would like to meet with us and have a chat, drop by our Gitter channel or visit our GitHub repo.