Exploring the CrowdSec and Suricata Integration

This article was originally published on the vlaicu.io Blog by CrowdSec Ambassador, Flaviu Vlaicu. 

---

In this post I will walk you through the steps of integrating Suricata with Crowdsec, having the alerting and detection made in Suricata and the decision making in CrowdSec. 

Since CrowdSec also has a nice Console, we can visualize the alerts/decisions made there. I will also show you how to integrate CrowdSec with Pushover so you get notified each time the CrowdSec Security Engine makes a decision.

Let’s dive into the tutorial.

Prerequisites

The OPNSense CrowdSec plugin that you install is observing a few default logs from OPNSense, but it does not come configured for any Suricata log listening. 

A CrowdSec acquis.yaml file must be created or modified to get the feature we are adding to work. Without this, the fast.log file will go unobserved. You will also need the CrowdSec Hub elements, in this case CrowdSec Suricata Collection, to enable the parsing/alerting and decision making for suricata.ssh into the OPNSense and enter the commands you see below. 

Assuming you already have the CrowdSec plugin and the Security Engine up and running, you can skip the first command.

Make sure you have:

• CrowdSec plugin installed on OPNSense.
• Suricata configured with fast logs.

Commands

1. Enroll your CrowdSec engine (if not already done):

sudo cscli console enroll  

2. Install the necessary collections and whitelist:

The following commands will add the CrowdSec Suricata Collection and whitelists, preventing you from getting yourself blocked.

cscli collections install crowdsecurity/suricata

cscli collections install crowdsecurity/whitelist-good-actors

cscli parsers install crowdsecurity/whitelists

Configuring the CrowdSec Security Engine for Suricata

If the following file does not exist, create it and add the following config to it:

/usr/local/etc/crowdsec/acquis.d/suricata.yaml

---
filenames:
  - /var/log/suricata/fast.log
labels:
  type: suricata-fastlogs
---

Configuring fast logs and log rotation

Now that the Security Engine is aware and listening, we will want to create and rotate those fast.log logs. Let’s set up the rotation of /var/log/suricata/fast.log as a custom config:

# logfilename [owner:group] mode count size when flags [/pid_file] [sig_num]
/var/log/suricata/fast.log root:wheel 640 3 * $D0 BZ /var/run/suricata.pid 1

In order for fast.logs to work, I added the below config to the following two locations in custom.yaml.

1. /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml
2. /usr/local/etc/suricata/custom.yaml

%YAML 1.1
---
default-log-dir: /var/log/suricata/

outputs:
  - fast:
      enabled: yes
      filename: fast.log
      append: yes

  - eve-log:
      enabled: yes
      filetype: regular
      filename: eve.json
      types:
        - alert:
            tagged-packets: yes
        - anomaly:
            enabled: yes
        - drop:
            alerts: yes
            flows: start
        - dns:
            enabled: yes
        - tls:
            enabled: yes
        - http:
            enabled: yes

  - stats:
      enabled: yes
      filename: stats.log
      append: yes
      totals: yes

  - syslog:
      enabled: no
      facility: local5
      level: Notice

I am still testing the workflow, but this is the configuration that has worked for me to be persistent after reboot and have both fast.log and eve.json at the same time. I still see the alerts in OPNsense in the Suricata alerts tab. Others have tested and specified that you need to add the whole output of the Suricata.yaml to the custom.yaml, but for me, for some reason, this did not work.

Edit: The above config will lead to a very noisy eve.json file. A user from the Suricata forum would recommend you not set those to enabled, and instead enable them in the app-layer protocols area.

%YAML 1.1
---
outputs:
  - fast:
      enabled: yes
      filename: fast.log
      append: yes
  - eve-log:
      enabled: yes
      filetype: regular
      filename: eve.json
      metadata: yes
      pcap-file: false
      community-id: true
      community-id-seed: 0
      types:
        - alert:
            payload: no
            payload-printable: no
            packet: no
            http-body: no
            http-body-printable: no
            tagged-packets: yes
            metadata:
              app-layer: true
              flow: true
              rule:
                metadata: true
                raw: true
            xff:
              enabled: yes
              mode: overwrite
              deployment: reverse
              header: X-Forwarded-For
        - frame:
            enabled: no
        - anomaly:
            enabled: no
            types:
              applayer: no
  - stats:
      enabled: yes
      filename: stats.log
      append: yes
      totals: yes
app-layer:
  protocols:
    telnet:
      enabled: yes
    rfb:
      enabled: yes
      detection-ports:
        dp: 5900, 5901, 5902, 5903, 5904, 5905, 5906, 5907, 5908, 5909
    mqtt:
      enabled: yes
    krb5:
      enabled: yes
    bittorrent-dht:
      enabled: yes
    snmp:
      enabled: yes
    ike:
      enabled: yes
    tls:
      enabled: yes
      detection-ports:
        dp: 443
      ja3-fingerprints: auto
    pgsql:
      enabled: yes
      stream-depth: 0
      max-tx: 1024
    dcerpc:
      enabled: yes
    ftp:
      enabled: yes
    rdp:
    ssh:
      enabled: yes
    http2:
      enabled: yes
    smtp:
      enabled: yes
      raw-extraction: no
      mime:
        decode-mime: yes
        decode-base64: yes
        decode-quoted-printable: yes
        header-value-depth: 2000
        extract-urls: yes
        body-md5: no
      inspected-tracker:
        content-limit: 100000
        content-inspect-min-size: 32768
        content-inspect-window: 4096
    imap:
      enabled: detection-only
    smb:
      enabled: yes
      detection-ports:
        dp: 139, 445
    nfs:
      enabled: yes
    tftp:
      enabled: yes
    dns:
      tcp:
        enabled: yes
        detection-ports:
          dp: 53
      udp:
        enabled: yes
        detection-ports:
          dp: 53
    http:
      enabled: yes
      libhtp:
         default-config:
           personality: IDS
           request-body-limit: 100kb
           response-body-limit: 100kb
           request-body-minimal-inspect-size: 32kb
           request-body-inspect-window: 4kb
           response-body-minimal-inspect-size: 40kb
           response-body-inspect-window: 16kb
           response-body-decompress-layer-limit: 2
           http-body-inline: auto
           swf-decompression:
             enabled: no
             type: both
             compress-depth: 100kb
             decompress-depth: 100kb
           double-decode-path: no
           double-decode-query: no
         server-config:
    modbus:
      enabled: yes
      detection-ports:
        dp: 502
      stream-depth: 0
    dnp3:
      enabled: yes
      detection-ports:
        dp: 20000
    enip:
      enabled: yes
      detection-ports:
        dp: 44818
        sp: 44818
    ntp:
      enabled: yes
    quic:
      enabled: yes
    dhcp:
      enabled: yes
    sip:
      enabled: yes

Monitoring the fast.log for activity

To monitor the fast.log file, you can use either of the following commands: 

• cat /var/log/suricata/fast.log
• tail -f /var/log/suricata/fast.log  

Testing the configuration

You can test the implementation with EICAR.

The fast.logs should now appear and they should be filled with friendly, readable alerts:

This is how the alerts look in fast log format:

The alerts from the fast.log are also getting pushed into the Crowdsec Console:

Setting up Pushover notifications

Let’s have the alerts and decisions pushed via Pushover so we know what is going on without always looking for alerts. 

To achieve this, first create a free account at Pushover, install the application on your mobile device, and enroll the device. Then, create a new application for which you will receive the application token.

Updating CrowdSec profiles

You will have two tokens: the application token, which you have just create,d and the user token. You will need them both for the integration. 

Add the following configuration to the /user/local/etc/crowdsec/profiles.yaml file.

name: default_ip_remediation
#debug: true
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
 - type: ban
   duration: 4h
#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
notifications:
 - http_default   # Set the required http parameters in /etc/crowdsec/notifications/http.yaml before enabling this.
on_success: break
---
name: default_range_remediation
#debug: true
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Range"
decisions:
 - type: ban
   duration: 4h
#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
notifications:
 - http_default   # Set the required http parameters in /etc/crowdsec/notifications/http.yaml before enabling this.
on_success: break

Configuring HTTP notifications

Add the following configuration to the /usr/local/etc/crowdsec/notifications/http.yaml file.

#########################################################################
# Title:            CrowdSec : Pushover Notification (API)              #
# Author(s):        JigSawFr                                            #
# URL:              https://github.com/crowdsecurity/crowdsec           #
#########################################################################
#                             MIT License                               #
#########################################################################

type: http          # Don't change
name: http_default  # Must match the registered plugin in the profile

# One of "trace", "debug", "info", "warn", "error", "off"
log_level: info

# group_wait:         # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"
# group_threshold:    # Amount of alerts that triggers a message before  has expired, eg "10"
# max_retry:          # Number of attempts to relay messages to plugins in case of error
# timeout:            # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------
# plugin-specific options

# The following template receives a list of models.Alert objects
# The output goes in the http request body
format: |
  {
   "token": "replace with application token",
   "user": "replace with user token",
   "message": "{{range . -}}{{$alert := . -}}{{range .Decisions -}}{{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}}.\r\n https://www.shodan.io/host/{{.Value}}{{end -}}{{end -}}",
   "html": "1",
   "title": "Scenario triggered on IDS/IPS !"
  }  

url: https://api.pushover.net/1/messages.json

method: POST
headers:
  Content-Type: "application/json"

Time for final testing

Restart the service with sudo service crowdsec restart, and you are good to go! 

Any new notification should be received on your mobile device. The alerts from what I have tested are instantaneous, and this is how the notifications will look on your mobile device.

The Shodan.io lets you do further manual checks if needed. You can also add other websites for easy and fast checks, like Virustotal.com. The line below needs to be modified if you want to add other websites to check the status of the IP that has been blocked:

"message": "{{range . -}}{{$alert := . -}}{{range .Decisions -}}{{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}}.\r\n https://www.shodan.io/host/{{.Value}} \r\n https://www.virustotal.com/gui/ip-address/{{.Value}}{{end -}}{{end -}}"

Hope you found this quick tutorial handy! 

Feel free to reach out to the CrowdSec team on Discord or Discourse to let us know if this setup has worked for you, or if you have any questions or feedback, you can reach out to the CrowdSec team on Discord or Discourse.