Monitoring Linux Server Activity with Falco

Falco is a security tool that allows you to record security events on Linux servers based on rules. It was previously developed by Sysdig and later handed over to Cloud Native Computing Foundation.

This guide shows how to install Falco on Linux servers, write rules to detect malicious events executed by processes or users and eventually compares it with Linux Auditd.

Prerequisites

To follow this guide, you'll need access to a Debian Linux or CentOS Stream 9 server. Alternatively, you could spin up a virtual server. If you’re looking for a reliable, high-performance, and budget-friendly solution for your workflows, Hostman has you covered with Linux VPS Hosting options, including Debian VPS, Ubuntu VPS, and VPS CentOS. Instructions on how to launch a virtual server are already included.

Brief Overview of Linux System Calls 

In Linux, the user-space is reserved for user-facing services like web browsers, text editors, etc, whilst the kernel space is reserved for the privileged services. Services provided within the kernel space include memory management, process scheduling, file system management, etc.

In the context of system calls, when a user executes the cd command, the “chdir system call’’ is invoked via the chdir() wrapper function within the glibc library to change the current working directory and returns the result to the user-space program. Usually, the name of the wrapper function is the same as the invoked system call.

The GNU C Library, also known as glibc, contains system functions, acting as a wrapper around the actual function provided by the Linux kernel, allowing applications to access system functionality or make system calls through a standardized C interface.

For detailed information on how Linux systems calls work and roles/tasks of glibc wrapper functions, check Linux man page.

What is Falco?

Falco provides runtime security across hosts, containers, Kubernetes, and other cloud native environments.

It relies on both default and custom rules to detect events as malicious on Linux hosts, Kubernetes applications, etc. and associates event data with contextual metadata to deliver meaningful real-time alerts to the SIEM team.

Falco relies on different sources to gather events data. It natively supports Linux system call source by default. However, it’s possible to extend Falco capabilities to support other event sources like Kubernetes audit logs, AWS Cloudtrail, KeyCloak Admin/User events via the plugin system.

The plugin system consists of shared libraries that allows Falco to include or add new event sources, include new fields that extract information from events, etc.

As at the time of writing this guide, some of the following plugins are:

• K8saudit: Monitors and detects Kubernetes cluster events.
• Cloudtrail: Tracks events from Cloudtrail logs.
• Kafka: Records events from Kafka topics.
• Keycloak: Detects Keycloak user/admin events.

Check their website for a complete list of currently supported plugins.

In order to consume events at the kernel source, the following drivers are currently supported:

• eBPF probe
• modern eBPF probe
• kernel module

Using Modern eBPF Probe

eBPF means “extended Berkeley Packet Filter”. It enables us to run isolated programs within the Linux kernel space in order to extend the capabilities of the kernel without loading additional kernel modules.

They are programs that execute when specific hook points are triggered or an event takes place.

eBPF probe is embedded into the userspace application and works out of the box, regardless of the kernel release.

To use the modern eBPF probe, set the engine.kind parameter inside the /etc/falco/falco.yaml file to modern_ebpf to activate this feature.

There is no need to install other dependencies such as clang or llvm if you want to use modern eBPF.

Installing Falco

This section shows how to install Falco on Linux Debian and CentOS servers.

Running Falco on Debian

Step 1: Import Falco GPG key.

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg

Step 2: Setup the apt repository.

sudo bash -c 'cat << EOF > /etc/apt/sources.list.d/falcosecurity.list
deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main
EOF'

Step 3: Install the apt-transport-https package.

sudo apt install apt-transport-https

Step 4: Update the apt repository.

sudo apt update -y

Step 5: Install Falco.

sudo apt install -y falco

Running Falco on CentOS Stream 9

Step 1: Import the Falco GPG key.

rpm --import https://falco.org/repo/falcosecurity-packages.asc

Step 2: Set up the yum repository.

curl -s -o /etc/yum.repos.d/falcosecurity.repo https://falco.org/repo/falcosecurity-rpm.repo

Step 3: Update the yum repository.

yum update -y

Step 4: Install Falco.

yum install -y falco

Step 5: Execute the command to test whether Falco is successfully installed.

falco

Managing Falco with systemd

In production, it's recommended to manage Falco using Systemd because it provides a centralized way to control and automate service restart instead of manually managing Falco.

Systemd is the init process that starts required system services at boot time.

Use the following instructions to manually configure Systemd with Falco.

Step 1: Execute the following command to search for Falco services.

systemctl list-units "falco*"

Step 2: Use these commands to enable, start and check the status of falco-modern-bpf.service.

The systemctl enable command ensures Falco starts at boot time

systemctl enable falco-modern-bpf.service

This command starts the service:

systemctl start falco-modern-bpf.service

And this is how you check if the service is running:

systemctl status falco-modern-bpf.service

Step 3: Execute the command systemctl list-units | grep falco to search for active related services

The screenshot shows that both services are active. The latter is responsible for performing rules updates.

If you don't want falcoctl to perform automatic rules update, use the command below to mask it.

systemctl mask falcoctl-artifact-follow.service

It prevents falcoctl service from being enabled automatically once an aliased falco service is enabled.

Check this page for further information on using Systemd to manage Falco.

Configuring Falco Settings

This section shows how to configure some settings in the Falco configuration file located at /etc/falco/falco.yaml.

watch_config_files: This key can be assigned true or false values. The true value ensures that anytime changes are made to the rules or configuration file, it automatically reloads itself to apply the updated configuration settings.

rules_files: This key determines which rule files or directories are loaded first based on the values assigned to it. The example below ensures that rules in the /etc/falco/rules.d folder are checked first.

rules_files:
  - /etc/falco/rules.d
  - /etc/falco/falco_rules.yaml
  - /etc/falco/falco_rules.local.yaml

output_channel: Falco supports the following output channels.

• Syslog
• standard output
• http endpoint or webhook
• file output
• grpc service

You can enable one of these channels to determine where alerts and log messages are sent to.

Writing Falco Rules

Basically, a rule is made up of an event and specific condition.

Example of an event is a filesystem activity such as when a user accesses a file in the etc directory. Another example of an event is when someone or a service decides to connect or transfer a file to a remote host.

Conditions are pragmatic expressions that define the exact details Falco should look for. It involves inspecting process arguments, network addresses, etc.

Rules are written in YAML, and have a variety of required and optional keys. They are loaded at startup.

Following is the structure of a rule in Falco.

• rule: This key defines the name of the rule, e.g. rule: Unauthorised File Access.
• desc: The key desc means description. It describes the purpose of the rule, e.g. Detecting unauthorized access to files in the /etc folder by regular users.
• condition: This key informs Falco to trigger an alert when a specific event takes place, e.g. condition: open_read and fd.name startswith /etc. 
• output: The message that will be shown in the notification.
• priority: This key defines the priority level of the rule. Priority levels include WARNING, ERROR, DEBUG, NOTICE, EMERGENCY, INFORMATIONAL, CRITICAL, and ALERT.
• tags: This key is used to categorize rules, e.g. ["Sensitive_Files", and "Unauthorized_Users"].

For detailed information on Falco rules, check Falco’s website.

The following are rules to detect specific filesystem access and outbound network connection.

Creating a Rule for Filesystem Activity

Use the following steps to create a custom Falco rule.

Navigate to the path /etc/falco/rules.d using the cd command.

cd /etc/falco/rules.d

Create a custom rule file using the following command.

touch custom_rules.yaml

Open and edit the custom_rules.yaml file using vim or any other text editor.

vim custom_rules.yaml

Then copy and paste the following into the file custom_rules.yaml.

- rule: reading sensitive file
  desc: Detects when a user reads /etc/ folder
  condition: open_read and fd.name startswith /etc/
  output: “suspicious file read detected file=%fd.name accessed by user=%user.name”
  priority: WARNING
  tags: [network, filesystem]

Start Falco in the background.

falco &

To stop the background process falco from running forever, use the following command to search for process ID.

pgrep falco

Then use the kill command to terminate it by specifying the pid.

kill -9 process-pid

Now test the rule we just created to check whether Falco would alert us when a user opens or accesses the file /etc/passwd.

cat /etc/passwd

Creating a Rule for Detecting Outbound Connection

Use the following to create a rule to monitor network connection.

Navigate to the folder /etc/falco/rules.d using the command:

cd /etc/falco/rules.d

Use a text editor like vim to create a new file for custom rules.

vim custom.yaml

Copy and paste the following rule into the file custom.yaml to flag outbound connections to other hosts.

- rule: "Suspicious outbound connection"
  desc:  detect outbound connection to other hosts   
  condition: outbound and evt.type = connect and fd.sip != 8.8.8.8 
  output: "Suspicious outbound connection detected destination=%fd.sip"
  priority: WARNING
  tags: [network, exfiltration]

Make sure you execute the falco command before testing the preceding rule via the command:

ping -c 1 blacklisted_IPaddress

We'll receive a warning:

Comparison Between Falco and Linux Audit Framework.

Auditd is a part of the Linux auditing framework. It is responsible for writing audit records to the disk.

Both tools are useful in detecting events registered as malicious via rules. In addition, both tools rely on system calls as their native event source.

However, there are differences between these tools:

• Auditd does not have multiple event sources as compared to Falco.
• Auditd does not allow users to customize event output but Falco allows.

Conclusion 

Falco is useful in detecting events defined as malicious via rules. These define whether events are malicious or not.

However, it's worth noting that the folder /etc/falco/ should be restricted to privileged users and also be monitored by Falco otherwise anyone can tweak rules in the file to avoid detection.