The NGINX Bouncer v1.0 is out!

This new major version brings new features:

• Support of Stream mode
• Support of captcha remediations: reCaptcha V2
• Support of ban template: you can choose the HTML page that will be displayed to IP addresses that triggered a ban
• Support of redirect location: you can choose to redirect banned IP to a custom location
• Exclude location feature: you can specify some location where you don’t want to bounce

Introduction

Before we get started, here is a quick reminder of what a bouncer is in the CrowdSec ecosystem: pieces of standalone software in charge of acting upon blocked IPs. They can either be within the applicative stack or work out of the band. While the CrowdSec agents detect attacks, bouncers block them. You can find more about this in the CrowdSec documentation.

In this article, we’re going to cover:

• Bouncer installation
• Captcha remediation configuration
• Benchmarks and load testing

All the tests will be run using an AWS EC2 c5.xlarge (4vCPU/8Go RAM). On the system, we will deploy :

• NGINX
• CrowdSec v1.3.1
• CrowdSec NGINX Bouncer v1.0

Install the bouncer

If you are already using the v0 of the bouncer, pick up the maintainer configuration while upgrading the bouncer to avoid anything breaking.

First, add the Crowdsec repository: https://packagecloud.io/crowdsec/crowdsec/install

Then install the bouncer:

sudo apt install crowdsec-nginx-bouncer

Now the bouncer is installed.

Before restarting NGINX you can review the default configuration:

/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf

API_URL=http://127.0.0.1:8080
API_KEY=9857d25eb3da863f83a1071f0ec4bb08
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=1000
UPDATE_FREQUENCY=10
MODE=stream
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
#REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION has priority
BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
# ReCaptcha Secret Key
SECRET_KEY=
# Recaptcha Site key
SITE_KEY=
CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html
CAPTCHA_EXPIRATION=3600

You can refer to the documentation.

Once you have configured the bouncer, you can restart NGINX:

sudo systemctl restart nginx

Captcha remediations

Today, the main security problems for e-commerce websites are bots (scalping, vulnerability detection, crawling, etc). CrowdSec is able to detect them, but false positives might be triggered and we don’t want legitimate users to be blocked from the website. So that legitimate users can keep browsing, the best way is to enforce a captcha to block bots only.

Since we are going to set up the captcha remediation, we need to edit the `SECRET_KEY` and `SITE_KEY` options with the one from google reCaptcha.

Those values need to be put in the /etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf file, where the SECRET_KEY and SITE_KEY lines are.

We also need to set a resolver and a trusted certificate in order to query Google for the Captcha verification.

Note: Only reCaptcha v2 is supported for the time being. 

Once done, we can restart NGINX:

sudo systemctl restart nginx

Now that we have seen how to configure the captcha remediation at the bouncer level, how will the bouncer know if it must return a captcha or block the IP address? CrowdSec profiles are the answer! Indeed, with CrowdSec profiles, you can choose what type of remediation you want to apply depending on context.

The default profiles embedded with CrowdSec are the following ones:

/etc/crowdsec/profiles.yaml

name: default_ip_remediation
filters: - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions: 
- type: ban
  duration: 4h
on_success: break

This profile means that for each alert with the flag Remediation set to True and where the scope is  Ip, a 4-hour decision is triggered with a ban type. The last on_success: break line means that the following profiles will not be applied.

So what we want is to return a captcha for all IP addresses that have triggered HTTP scenarios (if an IP did an SSH brute force for example and try to access the website, we will block it).

To do this, we have to create a profile before the default one:

/etc/crowdsec/profiles.yaml

name: captcha_remediation
filters: 
- Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() startsWith "crowdsecurity/http-"
decisions: 
- type: captcha
  duration: 4h
on_success: break
---
name: default_ip_remediation
filters: - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions: 
- type: ban
  duration: 4h
on_success: break

This new profile will emit a decision with a “captcha” during 4 hours, only for IP addresses that triggered a HTTP scenario.

Before restarting CrowdSec check that you have at least  base HTTP scenarios installed.

And restart CrowdSec:

sudo systemctl restart nginx

We can now launch Nikto to trigger a HTTP scenario and then check the behavior of the NGINX bouncer.

nikto -h helloworld.local

We can see that the generated decisions contain the action “captcha”:

sudo cscli decisions list

And when visiting the website, we are being asked to solve it.

Since we released various new features for this bouncer, we are now going to benchmark it to ensure its performance is not impacted.

Benchmark

Stream mode vs Live mode

One of the additions of this release is the support for what we call “stream mode”. In the past, the NGINX bouncer only supported live (or “query”) mode: whenever the bouncer sees a “new” IP, it queries CrowdSec’s LAPI to check whether there is a decision about this IP, and keeps the result in cache. In architectures with a very heavy load, this can lead to an overhead in the response time for the first request from a “new” IP.

The “stream” mode, on the other hand, allows the bouncer ( NGINX in our case) to simply query the LAPI on a regular basis (in the background) to know the “state” of the blocklist and current decisions. It will then simply store this in memory, so that “new” IPs coming will be checked against an in-memory data structure for blazing fast lookup :)

Actual Benchmark

The tool used to do the benchmark is “Apache Bench” (ab). 

While running the benchmark, the CrowdSec Local API had around 10k IPs addresses stored in its database, from the community blocklists.

Here are the tested benchmarks:

• In stream mode:
• With no bouncer installed
• With a bouncer installed but without decisions for the remote IP
• With a decisions of type ‘ban’ for the remote IP
• With a decisions of type ‘captcha’ for the remote IP
• In live mode:
• With no bouncer installed
• With a bouncer installed but without decisions for the remote IP
• With a decisions of type ‘ban’ for the remote IP
• With a decisions of type ‘captcha’ for the remote IP

All the tests have been made with:

• 50 concurrent connection for 1000 requests
• 250 concurrent connection for 5000 requests

Stream mode

50 concurrent connections / 1000 requests

250 concurrent connections / 5000 requests

Live mode

50 concurrent connections / 1000 requests

250 concurrent connections / 5000 requests

Benchmark Results

The NGINX is actually used in production by a lot of our users, sometimes in big infrastructures protecting e-commerce websites. With all those new functionalities, we wanted to prove that this bouncer is still usable in a production environment, even with a lot of traffic.

In this benchmark, we can see that the NGINX bouncer doesn’t impact too much the performance of the web server. Either in stream or live mode, the performance remains nearly flat, with or without the bouncer. Indeed, we can see that for these modes, for 50 concurrent connections or 250, the difference is about 0 to 5ms.

With this new bouncer fresh off the lab, you now have plenty of new features to test out. As always, your feedback matters more than anything else so please feel free to share it on our Discord.