Through Smoke and Fire

“The alpaca trekked through the smoke and flame, its fleece protected by the fire’s embrace. Its journey was long, but its spirit remained strong, as it traversed the scorched earth in search of a new place.” (a two-sentence ballad about an alpaca travelling through fire and smoke made with ChatGPT)

While working on some new features for CrowdSec, we have also been experimenting with scenarios focused on post-exploitation behaviors, relying on auditd for Linux (incoming in 1.5!).

And while we wanted to test those scenarios in the wild, it led to a different experiment: how quickly can a machine get compromised? How fast can the CrowdSec network spot this compromised machine? And – even more importantly – how fast will it make its way to the crowdsourced blocklist? This is what this blog post is about! 

We came up with the following scenarios that seemed unlikely to trigger false positives but very likely to trigger post-exploitation behaviors:

  • Detecting a process that is downloading a binary from the internet (ie. invoking curl, wget) and quickly executing it.
  • Detecting binaries executed from /tmp/ or any directory starting with a ‘.’ (files and directories starting with a dot are not displayed by default, and baddies tend to take advantage of this).
  • Detecting a process that does a burst of calls to ‘rm’ or ‘pkill’.

The journey starts

Here we are, deploying a vulnerable machine on OVH. Funnily enough, we started by deploying a machine with a weak WordPress username and password. To our disappointment, after 72 hours, the box was still intact.

We then switched strategies and went for the weak ssh username and password. (note: in our next iteration, we’re going to go for applicative weakness and see if the 2nd part of the story differs).

After a few hours minutes, the machine was breached, and our slack notification let us know:

auditd postexploit pkill

Here we go! Behind the scene, here is what is happening:

We have a simple scenario that detects execution from suspicious places:

simple scenario with CrowdSec

And another simple scenario that identifies a process that calls pkill/pgrep/rm in bursts:

However, something was off: while we spotted some of the attacker’s post-exploitation behavior (trying to kill other miners and existing malware), we didn’t spot the attacker downloading and executing their backdoor. As mentioned above, we have a scenario that spots a combination of curl/wget and exec.

After a quick look at auditd logs, it appears this malware uses a different approach:

echo | base64 –decode | perl

To sound fancy, one might say it’s file-less malware 😂. We can come up with this solution:

Note: the scenario looks for an invocation of base64 shortly followed by an interpreter invocation, such as perl, python, or bash.

This is a good occasion to showcase the “replay” feature of CrowdSec : When you create a new scenario, you can then re-process old logs, to ensure that the newly created scenarios would have behaved as intended, without false positives or negatives:

crowdsec -dsn file:///var/log/audit/audit.log.1 -type auditd -no-api

We’re now happy with our post-exploitation behavior detection. But instead of killing and burning our server down to ashes, we wanted to see what the CrowdSec network would say about it, and how fast.

Through the smoke and fire

One of the critical targets for us is how fast a malevolent IP can end up in the community blocklist without taking any risk of it being a false positive or poisoning. So, after our machine was compromised, we just sat and watched:

  • 2023-01-18 05:24 – Machine gets breached.

Note: after exploitation, scripts try to kill any potential competition or detection mechanism.

  • 2023-01-18 06:45 – Machine is reported for the first time by a CrowdSec user (shoutout to 7a3e7b6… from Germany 🥳)
  • 2023-01-18 08:00 – Machine appears in “smoke”: it means that a meaningful amount of users has reported the machine with a good enough reputation. The IP starts to be enriched and is available in CTI. It also means that it can appear in the community blocklist as soon as it matches the criteria.
  • 2023-01-19 09:14 – Machine is in “fire” and is redistributed to the community: At least 100 distinct watchers from 95 distinct ranges and 49 different autonomous systems reported it. 
  • 2023-01-20 08:48 – Machine is seen for the first time by a honey pot. By the time it has been seen by a single honeypot, 226 users from 183 distinct ranges and 80 autonomous systems have reported our baddie!

Note: The red circle is when the baddie IP enters the community blocklist, leading to a significant decrease in reported numbers as users preemptively block the IP.


While this was an exciting experiment, it shows that we still have ground to cover when adding bad guys to the community blocklist. While 24 hours is “ok”, it is something we need to continue to work on and improve. The faster, the better!

You may also like

Upgrading the CrowdSec Infrastructure to Support IPv6-Only Users
Inside CrowdSec

Upgrading the CrowdSec Infrastructure to Support IPv6-Only Users

Follow our journey as we upgrade the CrowdSec infrastructure to allow our IPv6-only users to set up CrowdSec without any hiccups.

Network Effect x AI: Transforming CTI into Tactical Threat Intelligence
Inside CrowdSec

Network Effect x AI: Transforming CTI into Tactical Threat Intelligence

Delve into IP-based cyber defense and explore how the Network Effect and AI are transforming CTI into crowd-powered Tactical Threat Intelligence.

The CrowdSec FOSS Business Model: Open Source as the Digital Twin of Fair Trade
Inside CrowdSec

The CrowdSec FOSS Business Model: Open Source as the Digital Twin of Fair Trade

For the CrowdSec business model, FOSS means we offer something openly and transparently, but we also clearly express what we expect in return.