Detect Security Incidents with AWS CloudTrail & CrowdSec Security Engine

At CrowdSec, we have a dedicated AWS serverless architecture to process the 20 million signals we receive on a daily basis. While we were working on improving the security of our data lake, we faced some limitations in the existing security tooling that looked like a great use case for CrowdSec!

CrowdSec already effectively protects workloads on AWS — it supports Kinesis as a data source and AWS WAF as a remediation component — we wanted to be able to protect our AWS infrastructure itself. 

Spoiler alert: We’re doing the same thing with Kubernetes and the support for Kubernetes audit logs. 

AWS CloudTrail has been a recurrent requested feature, so we decided to give it a shot. While this project started simply because we were not satisfied with the AWS chatbot (it cannot be described using IaC tools, it is noisy, offers no exceptions etc.), we went a lot further than the original scope of the project.

In this article, I will be covering the following:

• Using CrowdSec on top of CloudTrail to cover CIS benchmark compliance, with the addition of exceptions and fine-tuning to limit false positives.
• Presenting a couple examples of security (rather than pure compliance) scenarios: Detecting brute force on the AWS Console, and detecting suspicious logins (i.e. non-working days, non-working hours, etc.)
• Creating useful notifications.

Requirements: Retrieving and parsing CloudTrail logs

Note: This is quite an advanced topic, so I am assuming a certain level of knowledge on the side of the reader, namely on AWS CloudTrail and on the some CrowdSec concepts, such as data sources, parsers, scenarios, exceptions and profiles.

CrowdSec already supports AWS Kinesis stream, but we added support for AWS S3 bucket, alongside SQS S3 notifications for convenience, as it’s the most common way to log AWS CloudTrail. 

As a bonus, AWS S3 makes the usage of CrowdSec replay mode (which we’ll be using in this article to test our scenarios) a lot easier! 

Note: Some events are made global on the AWS side and will be logged in us-east-1 CloudTrail region, so we suggest that you use a multi-region trail.

#/etc/crowdsec/acquis.yaml
---
source: s3
polling_method: sqs
sqs_name: cloudtrail-queue
sqs_format: s3notification
polling_interval: 30
aws_region: eu-west-1
transform: map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#))
max_buffer_size: 10000000
use_time_machine: true
labels:
  type: aws-cloudtrail
  

One interesting thing is that when CloudTrail events reach S3, one line contains many records. The transform directive is here for that, to flatten the array into as many individual events as possible.

Parsing CloudTrail with the existing JSON Helpers is a formality and is mostly a matter of extracting the relevant fields. The whole document itself will still be available in evt.Unmarshaled.cloudtrail for future use, thanks to the call to UnmarshalJSON.

onsuccess: next_stage
#debug: true
filter: "evt.Parsed.program == 'aws-cloudtrail'"
name: crowdsecurity/aws-cloudtrail
description: "Parse AWS Cloudtrail logs"
statics:
  - parsed: cloudtrail_parsed
    expression: UnmarshalJSON(evt.Line.Raw, evt.Unmarshaled, 'cloudtrail')
  - target: evt.StrTime
    expression: evt.Unmarshaled.cloudtrail.eventTime
  - meta: user_type
    expression: evt.Unmarshaled.cloudtrail.userIdentity.type
 # see : https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md#nil-coalescing
  - meta: user_arn
    expression: |
      evt.Unmarshaled.cloudtrail.userIdentity?.arn ?? evt.Unmarshaled.cloudtrail.userIdentity.userName
  - meta: event_name
    expression: evt.Unmarshaled.cloudtrail.eventName
  - meta: event_source
    expression: evt.Unmarshaled.cloudtrail.eventSource
  - meta: region
    expression: evt.Unmarshaled.cloudtrail.awsRegion
  - meta: source_ip
    expression: |
      IsIP(evt.Unmarshaled.cloudtrail.sourceIPAddress) ? evt.Unmarshaled.cloudtrail.sourceIPAddress : ""
  - meta: user_agent
    expression: evt.Unmarshaled.cloudtrail.userAgent
  - meta: error_code
    expression: evt.Unmarshaled.cloudtrail.errorCode
  - meta: event_id
    expression: evt.Unmarshaled.cloudtrail.eventID
  - meta: account_id
    expression: evt.Unmarshaled.cloudtrail.userIdentity.accountId
  - meta: log_type
    value: aws-cloudtrail

This file is available in the AWS collection installable with CSCLI.

Step 1: CIS benchmark with better notifications

As I mentioned earlier, when we implemented the support for CIS benchmark internally, the lack of flexibility we faced with the AWS chatbot was the initial reason behind our decision to add support for CloudTrail.

So one of the first things we did was to reimplement the CIS benchmark, this time using CloudTrail:

• Detect CloudTrail config change
• Detect configuration change
• Detect failed console authentication
• Detect change of IAM policies
• Detect deletion of KMS
• Detect login without MFA
• Detect changes of network ACL
• Detect Network Gateway changes
• Detect usage of root account
• Detect changes in routing tables
• Detect changes of S3 policy
• Detect changes in security groups
• Detect unauthorized calls
• Detect changes in VPC configuration

As an example, the implementation of the rule Detect AWS CloudTrail configuration change looks like this:

type: trigger
name: crowdsecurity/aws-cis-benchmark-config-config-change
description: "Detect AWS Config configuration change"
filter: |
	evt.Meta.log_type == 'aws-cloudtrail' &&
	evt.Unmarshaled.cloudtrail.eventSource == "config.amazonaws.com" &&
	(
 	evt.Meta.event_name == "StopConfigurationRecorder" ||
 	evt.Meta.event_name == "DeleteDeliveryChannel" ||
 	evt.Meta.event_name == "PutDeliveryChannel" ||
 	evt.Meta.event_name == "PutConfigurationRecorder"
	)
labels:
  type: compliance

You can find all the other rules in the AWS CIS Benchmark collection on the CrowdSec Hub.

Setting up our notifications system

The next thing we needed was to set up our alerts. However, we were not fully satisfied with the AWS’s default Slack chatbot notifications:

‍

The default notification format of the AWS chatbot requires you to go to the Console to get any of the relevant information.

We needed something like this:

‍

This notification system gives you context and relevant information: 

• The failed event name
• The name of the user
• The source IP
• The account ID
• Bonus: As it’s generated by CrowdSec, you can easily set up exceptions or customize your notifications.

To get the notifications right, you need to configure the profiles configuration file alongside the HTTP notifications plugin. As we use Slack internally, we used a format compatible with Slack. This can be easily adapted for other HTTP-compliant tools (e.g. Discord, Mattermost, Telegram, etc.).

Here are the contents of the /etc/crowdsec/profiles.yaml file which defines a profile named aws_cloudtrail_notif.

name: aws_cloudtrail_notif
debug: true
filters:
 - Alert.GetScenario() startsWith "crowdsecurity/aws-"
notifications:
# - slack_cloudtrail_notif
  - http_default
on_success: break

Keeping in mind that profiles are in charge of dispatching alerts to notification plugins and deciding which scenarios trigger active remediations, the profile configuration that we described above will specifically match all the alerts generated by our AWS specific scenarios, and ensure the notifications are send to the correct place (here, our http_default notification plugin). 

Note: You can find all configuration files, templates, etc. in the documentation of the AWS CIS Benchmark collection.

The Hub page of the collection includes a custom notification template that, thanks to Sprig and the go template library, allows cool notifications:

You can also expand the notification template with further details.

Step 2: Security isn’t compliance

Now that we have working parsers, scenarios, and notifications, let’s try to make more useful stuff than ticking boxes. The scenarios covered here are just examples, but they are a great way to demonstrate some new CrowdSec features (surprise!):

• Alerting on logins in non-working hours and non-working days
• Detecting brute force attacks on the AWS Console

Non-working hours/days scenario

Here’s what we want to detect:

• AWS Console login (event name can be ConsoleLogin, GetSessionToken or GetFederationToken)
• That login was successful (responseElements->ConsoleLogin = ‘Success’)
• That login happened outside of office hours (after 8:00 PM or before 6:00 AM), or office days (Day of the week is Saturday or Sunday)

So we end up with a scenario like this:

# NWD/NWH AWS console login
type: trigger
name: crowdsecurity/aws-cloudtrail-nwo-nwd-console-login
description: "Detect console login outside of office hours"
filter: |
 evt.Meta.log_type == 'aws-cloudtrail' &&
 (evt.Meta.event_name == 'ConsoleLogin' || evt.Meta.event_name == 'GetSessionToken' || evt.Meta.event_name == 'GetFederationToken') &&
 evt.Unmarshaled.cloudtrail.responseElements?.ConsoleLogin == 'Success' &&
 (
  (evt.Time.Hour() >= 18 || evt.Time.Hour() < 6) ||
  (evt.Time.Weekday().String() == 'Saturday' || evt.Time.Weekday().String() == 'Sunday')
 )
groupby: evt.Meta.source_ip
scope:
 type: AwsARN
 expression: evt.Meta.user_arn

All the timestamps are in UTC, so adapting the proper time is left as an exercise to the reader 🙂

Detecting Console brute force

What is interesting when dealing with brute force on the AWS Console is that it will only be triggered with failed logins targeting your org and an existing username, so the risk of false positive is very low. The scenario itself is quite straightforward:

type: leaky
capacity: 5
leakspeed: 30s
name: crowdsecurity/aws-cloudtrail-bf-console-login
description: "Detect console login bruteforce"
filter: |
	evt.Meta.log_type == 'aws-cloudtrail' && (
  	(evt.Meta.event_name == 'ConsoleLogin' && evt.Unmarshaled.cloudtrail.responseElements.ConsoleLogin == 'Failure') ||
  	(evt.Meta.event_name == 'GetSessionToken' && evt.Meta.error_code=='AccessDenied') ||
  	(evt.Meta.event_name == 'GetFederationToken' && evt.Meta.error_code=='AccessDenied')
	)
groupby: evt.Meta.source_ip
blackhole: 1m
reprocess: true
scope:
  type: Ip

Although AWS documentation is not that clear about this, it seems that Console login failures are logged in us-east-1, whereas successful attempts are logged in the user's region.

Step 3: Exceptions

One more thing we can do with CrowdSec that we couldn’t do with the native AWS solution is effectively implementing exceptions. For example, because of how the AWS Console natively behaves, false positives on aws-cis-benchmark-unauthorized-call are not uncommon. However, thanks to CrowdSec we can avoid this:

name: crowdsecurity/usual-suspects
description: "Whitelist CDN providers"
debug: true
whitelist:
  #debug: true
  reason: "whitelist legit users"
#LogInfo('%s == %s // %s == ip', Split(#, ',')[0], evt.GetMeta('user_arn'), Split(#, ',')[1]) &&
  expression:
	- |
    	any(File('allowed_users.txt'), {
          	Split(#, ',')[0] == evt.Overflow.Alert.GetScenario() &&
          	Split(#, ',')[1] == evt.GetMeta('user_arn') &&
          	Split(#, ',')[2] == evt.Overflow.Alert.Source.IP
    	})
data:
	- dest_file: allowed_users.txt
  	type: string

With the associated file containing the list of scenarios/users and IPs we add the following users to the default exceptions file (/var/lib/crowdsec/data/allowed_users.txt):

crowdsecurity/aws-cis-benchmark-unauthorized-call,arn:aws:iam::xxx:user/xx,x.x.x.x
crowdsecurity/aws-cis-benchmark-unauthorized-call,arn:aws:iam::yyy:user/yy,z.z.z.z

Closing word

Detecting security incidents like brute force and suspisious non-working-hours/days logins becomes a piece of cake with the CrowdSec Security Engine and AWS CloudTrail, wouldn’t you agree? Hopefully, those security improvements and better-currated notifications will save you some time and frustration in the future! 

If you want to further improve security for your AWS Console, stay tuned for our upcoming article on impossible travel and learn how to detect users that jump from one IP to another located in different countries within a short timeframe.