How to stop a 7k machine botnet in 1 minute with CrowdSec

In 2020, our ways of living and working turned completely upside down in a matter of days. We all brought our companies home and staying connected to our colleagues, friends and family became a critical necessity. This opened the door for hackers to cause disruption and we saw a huge increase of DDoS attacks all around the world (+151% in H1’20 according to BusinessWire). This is the story of how CrowdSec protected one of its users' client and therefore the whole CrowdSec community, from massive DDoS attacks in a blink of an eye, and how it could protect you too.

Sorf Networks

Sorf Networks is a Turkey-based technology company that provides high-configuration managed Servers and DDoS protection solutions for their clients. They have been an IBM-Softlayer partner for 10 years.

The challenge

One of their customers had to cope with daily DDoS attacks from 10K+ machine botnets. He was struggling to find a solution that would meet technical requirements to deal with them in a timely manner.

While they took general precautions to mitigate those attacks such as introducing JS challenges, rate-limit and so on, the technical context didn’t make it viable on the whole attack surface. Some of the URLs needed to be consumed directly by very basic software that could not support JS challenges. Hackers being hackers, this was exactly what they were targeting every single day: the weakest chain point.

Sorf Networks had set up a DDoS mitigation strategy on this specific scope relying on Fail2ban, and while it was working decently, it was still too slow and thus not effective enough. When suffering a DDoS from 7-10K machines, Fail2ban would have needed 50 minutes process logs and deal with the attack. Besides, before IPs could be banned, logs would continue to stack. Finally, Fail2ban would have needed to process several thousand logs per second, which was clearly impossible.

The solution

Although CrowdSec technology was able to cope with attacks of such magnitude, it needed a tailor-made configuration to deal with such a huge traffic on one single machine. When performing “DDoS tests” from a rented botnet, the attack reached around 6700 req/s from 8600 uniques IPs. Below is a capture of one of the servers’ traffic.

CrowdSec default setup was only able to process around 1k EP/s, far from what was required for this very specific job. The solution needed to significantly improve its throughput so it could absorb the log volumetry.

Subsequently, changes were implemented within the configuration. First, the team removed the expensive and non-crucial enrichment parsers, such as the geoip enrichment. They also increased the default number of allowed go-routines from “1” to “5”  This led to another live test, again with 8.000 to 9.000 hosts, averaging between 6.000 and 7.000 requests per second.

This came at a cost, as CrowdSec was eating 600% CPU during the operation but its memory consumption stayed around 270 Mb only.

Results

The results, however, showed remarkable success: 

• in 1 minute, CrowdSec was able to ingest all the logs
• 95% of the botnet was banned and the attack efficiently mitigated
• 15 domains are now protected from DDoS attacks

“The CrowdSec platform made it possible for my team to deliver a world-class and efficient defense system to my customer in an incredibly short timeframe.”

Cagdas Aydogdu, Director of Sorf Networks

More CrowdSec stories soon! If you are interested in testing the software or would like to submit your use case so we can publish it, we'd be delighted to hear from you.