Through Smoke and Fire

“The alpaca trekked through the smoke and flame, its fleece protected by the fire's embrace. Its journey was long, but its spirit remained strong, as it traversed the scorched earth in search of a new place.” (a two-sentence ballad about an alpaca travelling through fire and smoke made with ChatGPT)

While working on some new features for CrowdSec, we have also been experimenting with scenarios focused on post-exploitation behaviors, relying on auditd for Linux (incoming in 1.5!).

And while we wanted to test those scenarios in the wild, it led to a different experiment: how quickly can a machine get compromised? How fast can the CrowdSec network spot this compromised machine? And - even more importantly - how fast will it make its way to the crowdsourced blocklist? This is what this blog post is about! 

We came up with the following scenarios that seemed unlikely to trigger false positives but very likely to trigger post-exploitation behaviors:

• Detecting a process that is downloading a binary from the internet (ie. invoking curl, wget) and quickly executing it.
• Detecting binaries executed from /tmp/ or any directory starting with a ‘.’ (files and directories starting with a dot are not displayed by default, and baddies tend to take advantage of this).
• Detecting a process that does a burst of calls to ‘rm’ or ‘pkill’.

The journey starts

Here we are, deploying a vulnerable machine on OVH. Funnily enough, we started by deploying a machine with a weak WordPress username and password. To our disappointment, after 72 hours, the box was still intact.

We then switched strategies and went for the weak ssh username and password. (note: in our next iteration, we’re going to go for applicative weakness and see if the 2nd part of the story differs).

After a few hours minutes, the machine was breached, and our slack notification let us know:

Here we go! Behind the scene, here is what is happening:

We have a simple scenario that detects execution from suspicious places:

And another simple scenario that identifies a process that calls pkill/pgrep/rm in bursts:

However, something was off: while we spotted some of the attacker's post-exploitation behavior (trying to kill other miners and existing malware), we didn’t spot the attacker downloading and executing their backdoor. As mentioned above, we have a scenario that spots a combination of curl/wget and exec.

After a quick look at auditd logs, it appears this malware uses a different approach:

echo <payload> | base64 –decode | perl

To sound fancy, one might say it’s file-less malware 😂. We can come up with this solution:

Note: the scenario looks for an invocation of base64 shortly followed by an interpreter invocation, such as perl, python, or bash.

This is a good occasion to showcase the “replay” feature of CrowdSec : When you create a new scenario, you can then re-process old logs, to ensure that the newly created scenarios would have behaved as intended, without false positives or negatives:

crowdsec -dsn file:///var/log/audit/audit.log.1 -type auditd -no-api

‍

‍

We’re now happy with our post-exploitation behavior detection. But instead of killing and burning our server down to ashes, we wanted to see what the CrowdSec network would say about it, and how fast.

Through the smoke and fire

One of the critical targets for us is how fast a malevolent IP can end up in the community blocklist without taking any risk of it being a false positive or poisoning. So, after our machine was compromised, we just sat and watched:

• 2023-01-18 05:24 - Machine gets breached.

Note: after exploitation, scripts try to kill any potential competition or detection mechanism.

• 2023-01-18 06:45 - Machine is reported for the first time by a CrowdSec user (shoutout to 7a3e7b6… from Germany 🥳)

• 2023-01-18 08:00 - Machine appears in “smoke”: it means that a meaningful amount of users has reported the machine with a good enough reputation. The IP starts to be enriched and is available in CTI. It also means that it can appear in the community blocklist as soon as it matches the criteria.

• 2023-01-19 09:14 - Machine is in “fire” and is redistributed to the community: At least 100 distinct watchers from 95 distinct ranges and 49 different autonomous systems reported it. 

• 2023-01-20 08:48 - Machine is seen for the first time by a honey pot. By the time it has been seen by a single honeypot, 226 users from 183 distinct ranges and 80 autonomous systems have reported our baddie!

Note: The red circle is when the baddie IP enters the community blocklist, leading to a significant decrease in reported numbers as users preemptively block the IP.

Conclusion

While this was an exciting experiment, it shows that we still have ground to cover when adding bad guys to the community blocklist. While 24 hours is “ok”, it is something we need to continue to work on and improve. The faster, the better!