The HAProxy Bouncer is out!

This major version brings new features:

• Support of Stream mode
• Support of captcha remediations: reCaptcha V2
• Support of ban template: you can choose the HTML page that will be displayed to IP addresses that triggered a ban
• Support of redirect location: you can choose to redirect banned IP to a custom location
• Exclude location feature: you can specify a location where you don’t want to bounce

Introduction

HAProxy is an open-source software TCP/HTTP Load Balancer and proxying solution which runs on Linux, macOS and FreeBSD. It is used by a lot of our users, including some in big infrastructures protecting e-commerce websites.

Before we get started, here is a quick reminder of what a bouncer is in the CrowdSec ecosystem: pieces of standalone software in charge of acting upon blocked IPs.

They can either be within the applicative stack, or work out of band. While the CrowdSec agents detect attacks, bouncers block them. You can find more about this in the CrowdSec documentation.

In this article, we’re going to cover:

• Bouncer installation
• Captcha remediation configuration
• Benchmarks and load testing

Install the bouncer

First, add the CrowdSec repository: https://packagecloud.io/crowdsec/crowdsec/install

Then, install the bouncer:

sudo apt install crowdsec-haproxy-bouncer

Now the bouncer is installed.

Before restarting HAProxy you can review the default configuration:

/etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf
 

ENABLED=true
API_KEY=9857d25eb3da863f83a1071f0ec4bb08
# haproxy
# path to community_blocklist.map
MAP_PATH=/var/lib/crowdsec/lua/haproxy/community_blocklist.map
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=stream
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
# path to ban template
BAN_TEMPLATE_PATH=
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
# ReCaptcha Secret Key
SECRET_KEY=
# Recaptcha Site key
SITE_KEY=
# path to captcha template
CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/captcha.html
CAPTCHA_EXPIRATION=3600  

You’ll have to create or update the haproxy.cfg configuration file. The sections are identified with: 

# CrowdSec bouncer >>> 

# CrowdSec bouncer <<<

The following is required:

/etc/haproxy/haproxy.cfg 

#HA Proxy Config
global
 daemon
 maxconn 256

 # Crowdsec bouncer >>>
 lua-prepend-path /usr/local/crowdsec/lua/haproxy/?.lua
 lua-load /usr/local/crowdsec/lua/haproxy/crowdsec.lua # path to crowdsec.lua
 setenv CROWDSEC_CONFIG /etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf # path to crowdsec bouncer configuration file
 # Crowdsec bouncer <<<

defaults
 mode http
 timeout connect 5000ms
 timeout client 50000ms
 timeout server 50000ms

frontend myApp
 bind *:80

 # Crowdsec bouncer >>>
 stick-table type ip size 10k expire 30m # declare a stick table to cache captcha verifications
 http-request lua.crowdsec_allow # action to identify crowdsec remediation
 http-request track-sc0 src if { var(req.remediation) -m str "captcha-allow" } # cache captcha allow decision 
 http-request redirect location %[var(req.redirect_uri)] if { var(req.remediation) -m str "captcha-allow" } # redirect to initial url
 http-request use-service lua.reply_captcha if { var(req.remediation) -m str "captcha" } # serve captcha template if remediation is captcha
 http-request use-service lua.reply_ban if { var(req.remediation) -m str "ban" } # serve ban template if remediation is ban
 # Crowdsec bouncer <<<

 default_backend myAppBackEnd

backend myAppBackEnd
 balance roundrobin
 server myAppServer1 nginx:80 check

# Crowdsec bouncer >>>
# define a backend for google to allow DNS resolution if using reCAPTCHA
backend captcha_verifier
 server captcha_verifier www.google.com:443 check

# define a backend for crowdsec to allow DNS resolution
backend crowdsec
 server crowdsec localhost:8080 check
# Crowdsec bouncer <<<

You can refer to this documentation.

Once you have configured the bouncer and HAProxy, you can restart the HAProxy:

sudo systemctl restart haproxy  

Captcha remediations

Today, the main security problems for e-commerce websites are bots (scalping, vulnerability detection, crawling, etc). CrowdSec is able to catch them, but false positives might be triggered and we don’t want legitimate users to be blocked from the website, resulting in business loss.

If there is a doubt on users, the best way is to enforce a captcha to block bots only.

Since we are going to set up the captcha remediation, we need to edit the `SECRET_KEY` and `SITE_KEY` options with the one from Google reCaptcha (https://www.google.com/recaptcha/admin/create). 

Those values need to be put in the /etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf file, where the SECRET_KEY and SITE_KEY lines are. 

Note: Only reCaptcha v2 is supported for the time being.

Once done, we can restart the HAProxy:

sudo systemctl restart haproxy 

Now that we have seen how to configure the captcha remediation at the bouncer level, how will the bouncer know if it must return a captcha or block the IP address? CrowdSec profiles are the answer! Indeed, with CrowdSec profiles, you can choose what type of remediation you want to apply depending on the context.

The default profiles embedded with CrowdSec are the following ones:

/etc/crowdsec/profiles.yaml

name: default_ip_remediation
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
 - type: ban
   duration: 4h
on_success: break  

This profile means that for each alert with the flag Remediation set to True and where the scope is  Ip, a 4-hour decision is triggered with a  ban type. The last on_success: break  line means that the following profiles will not be applied.

So what we want is to return a captcha for all IP addresses that have triggered HTTP scenarios (if an IP did a SSH brute force for example and try to access the website, we will block it).

To do this, we have to create a profile before the default one:

/etc/crowdsec/profiles.yaml

name: captcha_remediation
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() startsWith "crowdsecurity/http-"
decisions:
 - type: captcha
   duration: 4h
on_success: break
---
name: default_ip_remediation
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
 - type: ban
   duration: 4h
on_success: break  

This new profile will emit a decision with a “captcha” for 4 hours, only for IP addresses that triggered an HTTP scenario.

Before restarting CrowdSec check that you have at least the base HTTP scenarios installed.

And then, restart CrowdSec:

sudo systemctl restart crowdsec  

We can now launch Nikto to trigger a HTTP scenario and then check the behavior of the HAProxy bouncer.

nikto -h helloworld.local

We can see that the generated decisions contain the action “captcha”:

sudo cscli decisions list

And when visiting the website, we are being asked to solve it.

Since we released various new features for this bouncer, we are now going to benchmark it to ensure its performance is not impacted.

Benchmark

Stream mode vs live mode

In live mode, whenever the bouncer sees a “new” IP, it queries CrowdSec’s LAPI to check whether there is a decision about this IP, and keeps the result in cache. In architectures with a very heavy load, this can lead to an overhead in the response time for the first request from a “new” IP.

The “stream” mode, on the other hand, allows the bouncer (HAProxy in our case) to simply query the LAPI on a regular basis (in the background) to know the “state” of the blocklist and current decisions. It will then simply store this in memory, so that “new” IPs coming will be checked against an in-memory data structure for blazing fast lookup.

Actual Benchmark

The tool used to do the benchmark is “Apache Bench” (ab). 

While running the benchmark, the CrowdSec Local API had around 10k IPs addresses stored in its database, from the community blocklists.

Here are the tested benchmarks:

• In stream mode:

• With no bouncer installed
• With a bouncer installed but without decisions for the remote IP
• With a decisions of type ‘ban’ for the remote IP
• With a decisions of type ‘captcha’ for the remote IP

• In live mode:

• With no bouncer installed
• With a bouncer installed but without decisions for the remote IP
• With a decisions of type ‘ban’ for the remote IP
• With a decisions of type ‘captcha’ for the remote IP

All the tests have been made with:

• 50 concurrent connection for 1000 requests
• 250 concurrent connection for 5000 requests

Stream mode

50 concurrent connections / 1000 requests

250 concurrent connections / 5000 requests

Live mode

50 concurrent connections / 1000 requests

250 concurrent connections / 5000 requests

Benchmark Results

All the tests are run using a AWS EC2 c5.xlarge (4vCPU/8Go RAM). On the system, we deploy:

- HAProxy 2.5

- CrowdSec v1.4.1

- CrowdSec HAProxy Bouncer v0.3

HAProxy is actually used in production by a lot of our users, sometimes in big infrastructures protecting e-commerce websites. With all those new functionalities, we wanted to prove that this bouncer is still usable in a production environment, even with a lot of traffic.

In this benchmark, we can see that the HAProxy bouncer doesn’t impact too much the performance of the web server. Either in stream or live mode, the performance remains nearly flat, with or without the bouncer. Indeed, we can see that for these modes, for 50 concurrent connections or 250, the difference is about 0 to 5ms.

With this new bouncer fresh off the lab, you now have plenty of new features to test out. As always, your feedback matters more than anything else so please feel free to share it on our Discord.