Cloud servers from $4/mo - Grab the Deal! Let’s start ->
English
Community
Sign In
Sign In

How to Use Grep and Regular Expressions in Linux

How to Use Grep and Regular Expressions in Linux
Hostman Team
Technical writer
Linux
11.02.2025
Reading time: 16 min
Share

GREP (short for "global regular expression print") is one of the most popular utilities in the Linux operating system.

With it, you can search for phrases (sequences of characters) in multiple files simultaneously using regular expressions and filter the output of other commands, keeping only the necessary information.

This guide will cover how to search for specific expressions in a set of text files with various contents using the GREP utility.

All examples shown were run on a cloud server hosted by Hostman running Ubuntu version 22.04.

How Does GREP Work

The GREP command follows this structure:

grep [OPTIONS] [PATTERN] [SOURCES]

Where:

  • OPTIONS: Special parameters (flags) that activate certain mechanisms in the utility related to searching for expressions and displaying results.

  • PATTERN: A regular expression (or plain string) containing the phrase (pattern, template, sequence of characters) you want to find.

  • SOURCES: The path to the files where we will search for the specified expression.

If the GREP command is used to filter the output of another command, its structure looks a bit different:

[COMMAND] | grep [OPTIONS] [PATTERN]

Thus:

  • COMMAND: An arbitrary command with its own set of parameters whose output needs to be filtered.

  • The "pipe" symbol (|) is necessary to create a command pipeline, redirecting streams so that the output of an arbitrary command becomes the input for the GREP command.

Preparation

To understand the nuances of using GREP, it's best to start with small examples of searching for specific phrases. Therefore, we will first create a few text files and then test the GREP command on them.

Let’s first prepare a separate directory where the search will take place:

mkdir texts

Next, create the first file:

nano texts/poem

It will contain one of Langston Hughes's poems:

Hold fast to dreams  
For if dreams die  
Life is a broken-winged bird  
That cannot fly.  
Hold fast to dreams  
For when dreams go  
Life is a barren field  
Frozen with snow.

Now, create the second file:

nano texts/code.py

It will contain a simple Python script:

from datetime import date

dateNow = date.today()
print("Current time:", dateNow)

Finally, create the third file:

nano texts/page.html

This one will have simple HTML markup:

<html>
	<head>
		<title>Some Title</title>
	</head>

	<body>
		<div class="block">
			<p>There's gold here</p>
		</div>

		<div class="block">
			<p>A mixture of wax and clouds</p>
		</div>

		<div class="block block_special">
			<p>Today there's nothing</p>
		</div>
	</body>
</html>

By using files of different formats, we can better understand what the GREP command does by utilizing the full range of the utility's features.

Regular Expressions

Regular expressions are the foundation of the GREP command. Unlike a regular string, regular expressions contain special characters that allow you to specify phrases with a certain degree of variability.

When using the GREP utility, regular expressions are placed within single quotes:

'^date[[:alpha:]]*'

Thus, the full command can look like this:

grep '^date[[:alpha:]]*' texts/*

In this case, the console output will be:

texts/code.py:dateNow = date.today()

However, using double quotes allows you to pass various system data into the expression. For example, you can first create an environment variable with the search expression:

PATTERN="^date[[:alpha:]]*"

And then use it in the GREP command:

grep "$PATTERN" ./texts/*

Additionally, using single backticks allows you to use bash subprocess commands within the GREP command. For example, you can extract a regular expression from a pre-prepared file:

grep `cat somefile` ./texts/*

Note that with the asterisk symbol (wildcard), you can specify all the files in the directory at once. However, the GREP command also allows you to specify just one file: 

grep '^date[[:alpha:]]' texts/code.py 

Because regular expressions are a universal language used in many operating systems and programming languages, their study is a separate vast topic. 

However, it makes sense to briefly cover the main special characters and their functions. It’s important to note that regular expressions in Linux can work in two modes: basic (Basic Regular Expression, BRE) and extended (Extended Regular Expression, ERE). The extended mode is activated with the additional flag -E. The difference between the two modes lies in the number of available special characters and, consequently, the breadth of available functionality.

Basic Syntax

Basic syntax allows you to define only general formal constructs without considering the specific configuration of their characters.

Start of a line — ^

The caret symbol indicates that the sought sequence of characters must be at the beginning of the line:

grep '^Hold' texts/*

The console output will be as follows:

texts/poem:Hold fast to dreams
texts/poem:Hold fast to dreams

End of a line — $

The dollar sign indicates that the sought sequence of characters must be at the end of the line:

grep '</p>$' texts/*

Output:

texts/page.html:                        <p>There's gold here</p>
texts/page.html:                        <p>A mixture of wax and clouds</p>
texts/page.html:                        <p>Today there's nothing</p>

Note that the console output preserves the original representation of the found lines as they appear in the files.

Start of a word — \<

The backslash and less-than symbol indicate that the sought phrase must be at the beginning of a word:

grep '\<br' texts/*

Output:

texts/poem:Life is a broken-winged bird

End of a word — \>

The backslash and greater-than symbol indicate that the sought sequence of characters must be at the end of a word:

grep 'en\>' texts/*

Output:

texts/poem:Life is a broken-winged bird
texts/poem:For when dreams go
texts/poem:Life is a barren field
texts/poem:Frozen with snow.

Start or end of a word — \b

You can specify the start or end of a word using the more universal sequence of characters — backslash and the letter b.

For example, this marks the beginning:

grep '\bdie' texts/*

Output:

texts/poem:For if dreams die

And this marks the end:

grep '<div\b' texts/*

In this case, the console terminal output will be as follows:

texts/page.html:                <div class="block">
texts/page.html:                <div class="block">
texts/page.html:                <div class="block block_special">

Any character — .

Certain characters in the sought phrases can be left unspecified using the dot symbol:

grep '..ere' texts/*

Output:

texts/page.html:                        <p>There's gold here</p>
texts/page.html:                        <p>Today there's nothing</p>

Extended Syntax

Unlike basic syntax, extended syntax allows you to specify the exact number of characters in the sought phrases, thus expanding the range of possible matches.

Combining patterns — |

To avoid running the GREP command multiple times, you can specify several patterns in a single regular expression:

grep -E '^Hold|</p>$' texts/*

The result of running this command will be a combined console output containing the search results for the two separate regular expressions shown earlier.

texts/page.html:                        <p>There's gold here</p>
texts/page.html:                        <p>A mixture of wax and clouds</p>
texts/page.html:                        <p>Today there's nothing</p>
texts/poem:Hold fast to dreams
texts/poem:Hold fast to dreams

Repetition range — {n, d}

In some cases, certain characters in the sought phrase may vary in quantity. Therefore, in the regular expression, you can specify a range of the allowed number of specific characters.

grep -E 'en{1,2}' texts/*

Output:

texts/code.py:print("Current time:", dateNow)
texts/poem:Life is a broken-winged bird
texts/poem:For when dreams go
texts/poem:Life is a barren field
texts/poem:Frozen with snow.

However, frequently used repetition intervals are more conveniently written as special characters, thus simplifying the appearance of the regular expression.

One or more repetitions — +

A repetition interval from one to infinity can be expressed using the plus sign:

grep -E 'en+' texts/*

In this case, the console output will not differ from the previous example.

texts/code.py:print("Current time:", dateNow)
texts/poem:Life is a broken-winged bird
texts/poem:For when dreams go
texts/poem:Life is a barren field
texts/poem:Frozen with snow.

Zero or one repetition — ?

A repetition interval from 0 to 1 can be expressed using the question mark:

grep -E 'ss?' texts/*

As a result, this command will produce the following output in the console terminal:

texts/page.html:                <div class="block">
texts/page.html:                        <p>There's gold here</p>
texts/page.html:                <div class="block">
texts/page.html:                        <p>A mixture of wax and clouds</p>
texts/page.html:                <div class="block block_special">
texts/page.html:                        <p>Today there's nothing</p>
texts/poem:Hold fast to dreams
texts/poem:For if dreams die
texts/poem:Life is a broken-winged bird
texts/poem:Hold fast to dreams
texts/poem:For when dreams go
texts/poem:Life is a barren field
texts/poem:Frozen with snow.

Character set — [abc]

Instead of one specific character, you can specify an entire set enclosed in square brackets:

grep -E '[Hh]o[Ll]' texts/*

Output:

texts/poem:Hold fast to dreams
texts/poem:Hold fast to dreams

Character range — [a-z]

We can replace a large set of allowed characters with a range written using a hyphen:

grep -E 'h[a-z]+' texts/*

Output:

texts/page.html:<html>
texts/page.html:        <head>
texts/page.html:        </head>
texts/page.html:                        <p>There's gold here</p>
texts/page.html:                        <p>Today there's nothing</p>
texts/page.html:</html>
texts/poem:That cannot fly.
texts/poem:For when dreams go

Moreover, character sets and ranges can be combined:

grep -E 'h[abcd-z]+' texts/*

Each range is implicitly transformed into a set of characters:

  • [a-e] into [abcde]
  • [0-6] into [0123456]
  • [a-eA-F] into [abcdeABCDEF]
  • [A-Fa-e] into [ABCDEFabcde]
  • [A-Fa-e0-9] into [ABCDEFabcde0123456789]
  • [a-dA-CE-G] into [abcdABCEFG]
  • [acegi-l5-9] into [acegijkl56789]

Character type — [:alpha:]

Frequently used ranges can be replaced with predefined character types, whose names are specified in square brackets with colons:

[:lower:]

characters from a to z in lowercase

[:upper:]

characters from A to Z in uppercase

[:alpha:]

all alphabetic characters

[:digit:]

all digit characters

[:alnum:]

all alphabetic characters and digits

It is important to understand that the character type is a separate syntactic construct. This means that it must be enclosed in square brackets, which denote a set or range of characters:

grep -E '[[:alpha:]]+ere' texts/*

Output:

texts/page.html:                        <p>There's gold here</p>
texts/page.html:                        <p>Today there's nothing</p>

Filtering Output

To filter the output of another command, you need to write a pipe symbol after it, followed by the standard call to the GREP utility, but without specifying the files to search:

cat texts/code.py | grep 'import'

Like when searching in regular files, the console output will contain the lines with the matches of the specified phrases:

from datetime import date

In this case, the cat command extracts the file content and passes it to the input stream of the GREP utility.

Search Options

In addition to regular expressions, you can specify additional keys for the GREP command, which are special options in flag format that refine the search.

Extended Regular Expressions (-E)

Activates the extended regular expressions mode, allowing the use of more special characters.

Case Insensitivity (-i)

Performs a search for a regular expression without considering the case of characters:

grep -E -i 'b[ar]' texts/*

The console output corresponding to this command will be:

texts/poem:Life is a broken-winged bird
texts/poem:Life is a barren field

You can also specify flags together in a single string:

grep -Ei 'b[ar]' texts/*

Whole Word (-w)

Performs a search so that the specified regular expression is a complete word (not just a substring) in the found line:

grep -w and texts/*

Note that quotes are not required when specifying a regular string without special characters.

The result of this command will be:

texts/page.html: <p>A mixture of wax and clouds</p>

Multiple Expressions (-e)

To avoid running the command multiple times, you can specify several expressions at once:

grep -e 'Hold' -e 'html' texts/*

The result of this command will be identical to this one:

grep -E 'Hold|html' texts/*

In both cases, the console terminal will display the following output:

texts/page.html:<html>
texts/page.html:</html>
texts/poem:Hold fast to dreams
texts/poem:Hold fast to dreams

Recursive Search (-r)

Performs a recursive search in the specified directory to the maximum depth of nesting:

grep -r '[Ff]ilesystem' /root

The console terminal will display output containing file paths at different nesting levels relative to the specified directory:

/root/parser/parser/settings.py:#HTTPCACHE_STORAGE = "scrapy.extensions.httpcache.FilesystemCacheStorage"
/root/resize.log:Resizing the filesystem on /dev/vda1 to 3931904 (4k) blocks.
/root/resize.log:The filesystem on /dev/vda1 is now 3931904 (4k) blocks long.

Search for Special Characters (-F)

Allows the use of special characters as the characters of the search phrase:

grep -F '[' texts/*

Without this flag, you would encounter an error in the console terminal:

grep: Invalid regular expression

An alternative to this flag would be using the escape character in the form of a backslash (\):

grep '\[' texts/*

Including Files (--include)

Allows limiting the search to the specified files only:

grep --include='*.py' 'date' texts/*

The console output will be:

texts/code.py:from datetime import date
texts/code.py:dateNow = date.today()
texts/code.py:print("Current time:", dateNow)

We can also write this command without the wildcard by using an additional recursive search flag:

grep -r --include='*.py' 'date' texts

Excluding Files (--exclude)

Selectively excludes certain files from the list of search sources:

grep --exclude='*.py' 'th' texts/*

The console output will be:

texts/page.html: <p>Today there's nothing</p>
texts/poem:Frozen with snow.

Output Options

Some parameters of the GREP command affect only the output of search results, improving their informativeness and clarity.

Line Numbers (-n)

To increase the informativeness of the GREP results, you can add the line numbers where the search phrases were found:

grep -n '</p>$' texts/*

Each line in the output will be supplemented with the corresponding line number:

texts/page.html:8:                      <p>There's gold here</p>
texts/page.html:12:                     <p>A mixture of wax and clouds</p>
texts/page.html:16:                     <p>Today there's nothing</p>

Lines Before (-B)

Displays a specified number of lines before the lines with found matches:

grep -B3 'mix' texts/*

After the flag, you specify the number of previous lines to be displayed in the console terminal:

texts/page.html-                </div>
texts/page.html-
texts/page.html-                <div class="block">
texts/page.html:                        <p>A mixture of wax and clouds</p>

Lines After (-A)

Displays a specified number of lines after the lines with found matches:

grep -A3 'mix' texts/*

After the flag, you specify the number of subsequent lines to be displayed in the console terminal:

texts/page.html:                        <p>A mixture of wax and clouds</p>
texts/page.html-                </div>
texts/page.html-
texts/page.html-                <div class="block block_special">

Lines Before and After (-C)

Displays a specified number of lines both before and after the lines with found matches:

grep -C3 'mix' texts/*

After the flag, you specify the number of preceding and following lines to be displayed in the console terminal:

texts/page.html-                </div>
texts/page.html-
texts/page.html-                <div class="block">
texts/page.html:                        <p>A mixture of wax and clouds</p>
texts/page.html-                </div>
texts/page.html-
texts/page.html-                <div class="block block_special">

Line Count (-c)

Instead of listing the found lines, the GREP command will output only the number of matches:

grep -c 't' texts/*

The console output will contain the count of matches found in all specified files:

texts/code.py:3
texts/page.html:5
texts/poem:4

If only one file is specified as the source:

grep -c 't' texts/block

The console output will contain only the number:

4

File Names (-l)

This flag allows you to output only the names of the files in which matches were found:

grep -l 't' texts/*

The console output will be as follows:

texts/code.py
texts/page.html
texts/poem

Limit Output (-m)

Limits the number of lines output to the console terminal to the number specified next to the flag:

grep -m2 't' texts/*

The console output will be:

texts/code.py:from datetime import date
texts/code.py:dateNow = date.today()
texts/page.html:<html>
texts/page.html:                <title>Some Title</title>
texts/poem:Hold fast to dreams
texts/poem:That cannot fly.

As you can see, the limiting number affects not the entire output but the lines of each file.

Exact Match of Whole Line (-x)

Searches for an exact match of the entire line with no variability:

grep -x 'Life is a broken-winged bird' texts/*

The console output will be:

texts/poem:Life is a broken-winged bird

Conclusion

The GREP command in Linux is the most flexible and precise tool for searching expressions in large volumes of text data.

When using the command, you need to specify the following elements:

  • A specific set of options (flags) that configure the search and output mechanisms.
  • One or more regular expressions that describe the search phrase.
  • A list of sources (files and directories) where the search will be performed.

Additionally, the utility is used to filter the output of other commands by redirecting input and output streams.

The core of the GREP command is regular expressions. Unlike a simple string, they allow you to define a phrase with a certain degree of variability, making it match multiple similar entries.

There are two modes of operation for regular expressions:

  • Basic Mode: A limited set of special characters that allow you to formalize expressions only in general terms.
  • Extended Mode: A full set of special characters that allows you to formalize expressions with precision down to each character.

The extended mode provides complete flexibility and accuracy when working with regular expressions.

In rare cases where you only need to find matches for trivial patterns, you can limit yourself to the basic mode.

Linux
11.02.2025
Reading time: 16 min
Share

Similar

Linux

Monitoring Linux Server Activity with Falco

Falco is a security tool that allows you to record security events on Linux servers based on rules. It was previously developed by Sysdig and later handed over to Cloud Native Computing Foundation. This guide shows how to install Falco on Linux servers, write rules to detect malicious events executed by processes or users and eventually compares it with Linux Auditd. Prerequisites To follow this guide, you'll need access to a Debian Linux or CentOS Stream 9 server. Alternatively, you could spin up a virtual server using Hostman. The Hostman website has instructions on how to launch a virtual server. Brief Overview of Linux System Calls  In Linux, the user-space is reserved for user-facing services like web browsers, text editors, etc, whilst the kernel space is reserved for the privileged services. Services provided within the kernel space include memory management, process scheduling, file system management, etc. In the context of system calls, when a user executes the cd command, the “chdir system call’’ is invoked via the chdir() wrapper function within the glibc library to change the current working directory and returns the result to the user-space program. Usually, the name of the wrapper function is the same as the invoked system call. The GNU C Library, also known as glibc, contains system functions, acting as a wrapper around the actual function provided by the Linux kernel, allowing applications to access system functionality or make system calls through a standardized C interface. For detailed information on how Linux systems calls work and roles/tasks of glibc wrapper functions, check Linux man page. What is Falco? Falco provides runtime security across hosts, containers, Kubernetes, and other cloud native environments. It relies on both default and custom rules to detect events as malicious on Linux hosts, Kubernetes applications, etc. and associates event data with contextual metadata to deliver meaningful real-time alerts to the SIEM team. Falco relies on different sources to gather events data. It natively supports Linux system call source by default. However, it’s possible to extend Falco capabilities to support other event sources like Kubernetes audit logs, AWS Cloudtrail, KeyCloak Admin/User events via the plugin system. The plugin system consists of shared libraries that allows Falco to include or add new event sources, include new fields that extract information from events, etc. As at the time of writing this guide, some of the following plugins are: K8saudit: Monitors and detects Kubernetes cluster events. Cloudtrail: Tracks events from Cloudtrail logs. Kafka: Records events from Kafka topics. Keycloak: Detects Keycloak user/admin events. Check their website for a complete list of currently supported plugins. In order to consume events at the kernel source, the following drivers are currently supported: eBPF probe modern eBPF probe kernel module Using Modern eBPF Probe eBPF means “extended Berkeley Packet Filter”. It enables us to run isolated programs within the Linux kernel space in order to extend the capabilities of the kernel without loading additional kernel modules. They are programs that execute when specific hook points are triggered or an event takes place. eBPF probe is embedded into the userspace application and works out of the box, regardless of the kernel release. To use the modern eBPF probe, set the engine.kind parameter inside the /etc/falco/falco.yaml file to modern_ebpf to activate this feature. There is no need to install other dependencies such as clang or llvm if you want to use modern eBPF. Installing Falco This section shows how to install Falco on Linux Debian and CentOS servers. Running Falco on Debian Step 1: Import Falco GPG key. curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg Step 2: Setup the apt repository. sudo bash -c 'cat << EOF > /etc/apt/sources.list.d/falcosecurity.listdeb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable mainEOF' Step 3: Install the apt-transport-https package. sudo apt install apt-transport-https Step 4: Update the apt repository. sudo apt update -y Step 5: Install Falco. sudo apt install -y falco Running Falco on CentOS Stream 9 Step 1: Import the Falco GPG key. rpm --import https://falco.org/repo/falcosecurity-packages.asc Step 2: Set up the yum repository. curl -s -o /etc/yum.repos.d/falcosecurity.repo https://falco.org/repo/falcosecurity-rpm.repo Step 3: Update the yum repository. yum update -y Step 4: Install Falco. yum install -y falco Step 5: Execute the command to test whether Falco is successfully installed. falco Managing Falco with systemd In production, it's recommended to manage Falco using Systemd because it provides a centralized way to control and automate service restart instead of manually managing Falco. Systemd is the init process that starts required system services at boot time. Use the following instructions to manually configure Systemd with Falco. Step 1: Execute the following command to search for Falco services. systemctl list-units "falco*" Step 2: Use these commands to enable, start and check the status of falco-modern-bpf.service. The systemctl enable command ensures Falco starts at boot time systemctl enable falco-modern-bpf.service This command starts the service: systemctl start falco-modern-bpf.service And this is how you check if the service is running: systemctl status falco-modern-bpf.service Step 3: Execute the command systemctl list-units | grep falco to search for active related services The screenshot shows that both services are active. The latter is responsible for performing rules updates. If you don't want falcoctl to perform automatic rules update, use the command below to mask it. systemctl mask falcoctl-artifact-follow.service It prevents falcoctl service from being enabled automatically once an aliased falco service is enabled. Check this page for further information on using Systemd to manage Falco. Configuring Falco Settings This section shows how to configure some settings in the Falco configuration file located at /etc/falco/falco.yaml. watch_config_files: This key can be assigned true or false values. The true value ensures that anytime changes are made to the rules or configuration file, it automatically reloads itself to apply the updated configuration settings. rules_files: This key determines which rule files or directories are loaded first based on the values assigned to it. The example below ensures that rules in the /etc/falco/rules.d folder are checked first. rules_files:  - /etc/falco/rules.d  - /etc/falco/falco_rules.yaml - /etc/falco/falco_rules.local.yaml output_channel: Falco supports the following output channels. Syslog standard output http endpoint or webhook file output grpc service You can enable one of these channels to determine where alerts and log messages are sent to. Writing Falco Rules Basically, a rule is made up of an event and specific condition. Example of an event is a filesystem activity such as when a user accesses a file in the etc directory. Another example of an event is when someone or a service decides to connect or transfer a file to a remote host. Conditions are pragmatic expressions that define the exact details Falco should look for. It involves inspecting process arguments, network addresses, etc. Rules are written in YAML, and have a variety of required and optional keys. They are loaded at startup. Following is the structure of a rule in Falco. rule: This key defines the name of the rule, e.g. rule: Unauthorised File Access. desc: The key desc means description. It describes the purpose of the rule, e.g. Detecting unauthorized access to files in the /etc folder by regular users. condition: This key informs Falco to trigger an alert when a specific event takes place, e.g. condition: open_read and fd.name startswith /etc.  output: The message that will be shown in the notification. priority: This key defines the priority level of the rule. Priority levels include WARNING, ERROR, DEBUG, NOTICE, EMERGENCY, INFORMATIONAL, CRITICAL, and ALERT. tags: This key is used to categorize rules, e.g. ["Sensitive_Files", and "Unauthorized_Users"]. For detailed information on Falco rules, check Falco’s website. The following are rules to detect specific filesystem access and outbound network connection. Creating a Rule for Filesystem Activity Use the following steps to create a custom Falco rule. Navigate to the path /etc/falco/rules.d using the cd command. cd /etc/falco/rules.d Create a custom rule file using the following command. touch custom_rules.yaml Open and edit the custom_rules.yaml file using vim or any other text editor. vim custom_rules.yaml Then copy and paste the following into the file custom_rules.yaml. - rule: reading sensitive file desc: Detects when a user reads /etc/ folder condition: open_read and fd.name startswith /etc/ output: “suspicious file read detected file=%fd.name accessed by user=%user.name” priority: WARNING tags: [network, filesystem] Start Falco in the background. falco & To stop the background process falco from running forever, use the following command to search for process ID. pgrep falco Then use the kill command to terminate it by specifying the pid. kill -9 process-pid Now test the rule we just created to check whether Falco would alert us when a user opens or accesses the file /etc/passwd. cat /etc/passwd Creating a Rule for Detecting Outbound Connection Use the following to create a rule to monitor network connection. Navigate to the folder /etc/falco/rules.d using the command: cd /etc/falco/rules.d Use a text editor like vim to create a new file for custom rules. vim custom.yaml Copy and paste the following rule into the file custom.yaml to flag outbound connections to other hosts. - rule: "Suspicious outbound connection" desc: detect outbound connection to other hosts condition: outbound and evt.type = connect and fd.sip != 8.8.8.8 output: "Suspicious outbound connection detected destination=%fd.sip" priority: WARNING tags: [network, exfiltration] Make sure you execute the falco command before testing the preceding rule via the command: ping -c 1 blacklisted_IPaddress We'll receive a warning: Comparison Between Falco and Linux Audit Framework. Auditd is a part of the Linux auditing framework. It is responsible for writing audit records to the disk. Both tools are useful in detecting events registered as malicious via rules. In addition, both tools rely on system calls as their native event source. However, there are differences between these tools: Auditd does not have multiple event sources as compared to Falco. Auditd does not allow users to customize event output but Falco allows. Conclusion  Falco is useful in detecting events defined as malicious via rules. These define whether events are malicious or not. However, it's worth noting that the folder /etc/falco/ should be restricted to privileged users and also be monitored by Falco otherwise anyone can tweak rules in the file to avoid detection.
19 March 2025 · 9 min to read
Mail

How to Send Email in Linux from the Command Line with Sendmail and Mailx

For those managing servers or working on automation tasks, knowing how to send emails from the Linux terminal is essential. It offers complete control over email functions and eliminates the need for complex mail programs. This is useful in scenarios where speed and simplicity matter most. Common tools such as sendmail and mailx are frequently used for sending messages, checking SMTP settings, automating alerts, and integrating with scripts. They are straightforward yet effective, making them perfect for tasks like informing teams about server updates, automating reports, or testing email setups. This guide is designed for users looking to manage their email directly from the terminal. It covers the installation of essential tools and delves into more advanced tasks, such as sending attachments and configuring email tools. Why Choose Command-Line Email Tools? Two commonly used tools, sendmail and mailx, are reliable options for mail transmission in Linux. They come with a certain set of benefits: Efficiency: Traditional email software can be slow and resource-intensive. These tools enable quick and lightweight email sending directly from the terminal. Automation: They integrate smoothly with shell scripts, cron processes, and system monitoring tools. Automating mail alerts and notifications for repeated actions is possible via these Linux mail tools. Troubleshooting SMTP Problems: Debugging SMTP setups becomes more manageable. These commands provide visibility into message delivery, ensuring mail logs and errors are easier to inspect. Flexibility: Whether it’s sending alerts or generating automated reports, command-line tools like sendmail and mailx offer versatility across a range of tasks. Prerequisites  Before utilizing these Linux mail command line tools, ensure you have terminal access. Root privileges may be required in some cases, especially for configuring each mail command on Linux discussed in this guide. Setting Up a SMTP Server SMTP servers are essential for sending emails. These servers fall into two categories: External and Local SMTP servers. External SMTP Servers It refers to a mail server hosted by a third-party provider. These servers are utilized to deliver emails over the internet to recipients who are not part of your local network. They are built to manage global mail delivery while ensuring proper authentication, encryption, and spam prevention. Examples  Gmail  Address: smtp.gmail.com Port: 587 (with TLS) or 465 (with SSL) Outlook  Address: smtp.office365.com Port: 587 These servers need appropriate authentication methods (such as a username, password, or app-specific passwords) and encryption (like TLS or SSL) to ensure secure communication. Note: We’ve already provided a guide for setting up external SMTP servers. The command to send emails through Postfix remains the same as mentioned in this article. Simply configure the SMTP settings using our guide, and replace the email address with Gmail or any other preferred provider for proper email delivery. Local SMTP Servers This server functions solely within a private network or system. It is perfect for: Sending emails between users on the same network or domain (e.g., tom@office.local to jerry@office.local). Local testing and development tasks. Internal communication within an organization. Does not need internet access to operate, as they manage mail delivery internally. Setting Up a Local SMTP Server Here are the procedures to set up a local SMTP server using Postfix: Install Postfix via: sudo apt install postfix Modify the Postfix configuration file: sudo nano /etc/postfix/main.cf Update or confirm these key settings: myhostname = mail.office.local mydomain = office.local myorigin = $mydomain inet_interfaces = loopback-only local_recipient_maps = proxy:unix:passwd.byname mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain Save and exit the file after doing changes, then restart Postfix: sudo systemctl restart postfix To create email addresses like linux@office.local and hostman@office.local, set up user accounts on the server: sudo adduser linuxsudo adduser hostman Overview of sendmail sendmail is a prominent mail transfer agent (MTA) in Linux. It works flawlessly with SMTP servers for mail delivery and allows emails to be sent and routed from local systems or scripts.  Installing sendmail  Before sending emails, you must install the Linux sendmail tool. Execute the commands below based on your distribution: For Debian/Ubuntu sudo apt install sendmail For CentOS/Red Hat sudo yum install sendmail Starting and Enabling Service Once installed, make sure sendmail is running and configured to start at boot: sudo systemctl start sendmailsudo systemctl enable sendmail Testing the Configuration Check the sendmail is set up correctly by executing: echo "Testing sendmail setup" | sendmail -v your-email@example.com Verify email by executing the mail command: mail Note: Install mailutils package in case the mail command is not working. sudo apt install mailutils Or utilize the cat command: cat /var/mail/user Editing the Configuration File To customize settings for sendmail, modify the configuration file located at /etc/mail/sendmail.mc: sudo nano /etc/mail/sendmail.mc Make the required changes to fit your server. For example, if you want to define the domain name for your server, you can add or modify the following line: define(`confDOMAIN_NAME', `your_domain.com')dnl Here, replace your_domain with your actual domain name. Then rebuild the configuration file: sudo m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf If a "permission denied" error occurs, use: sudo sh -c "m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf" Finally, restart the service: sudo systemctl restart sendmail Sending Email Via sendmail With sendmail, you can easily deliver emails, customize subjects, and even add attachments using external tools. Let’s go over the process to send emails: Basic Example To send an email with sendmail, use the below-given instructions: First, create a file to hold the message: nano email.txt Add any content to the file, for example: Subject: Test Email from HostmanThis is a test email sent using sendmail on Linux. Deliver the file's contents: sendmail recipient@example.com < email.txt The contents of email.txt will be sent to the designated recipient. For verification, apply: mail Adding Attachments  sendmail by itself doesn’t support attachments. You’ll need to utilize uuencode or similar tools to include files. First, install sharutils for uuencode: sudo apt install sharutils Here’s how to attach a file: ( echo "Subject: Email with attachment"; uuencode file.txt file.txt ) | sendmail recipient@example.com In the above sendmail example we send an email with file.txt attached. To verify, apply the Linux command mail: mail Overview of mailx  The mailx Linux command is a simple and effective terminal application for managing emails. It is included in the mailutils package found in most Linux distributions. Installing mailx  Install mailutils package on your system to utilize the mailx command on Linux: For Debian/Ubuntu systems sudo apt install mailutils For Red Hat-based systems sudo yum install mailx Sending Email with mailx This is a simple example demonstrating the use of mailx. Include a subject line and message in your email: echo "This is the body of the email" | mailx -s "Test Email from Mailx" recipient@example.com Utilize the Linux mail command for verification: Example with Attachments Use the -A flag with the mailx command to send emails from Linux with attachments: echo "Please find the attached document" | mailx -s "Email with Attachment" -A email.txt recipient@example.com This sends email.txt as an attachment to the recipient. Conclusion Sending email from the Linux command line is an effective method for automating communication tasks, troubleshooting servers, or testing configurations. Using tools such as sendmail and mailx, you can manage everything from simple messages to more complex setups with attachments. This guide has provided detailed instructions to help you begin without difficulty. Utilize these Linux email commands to improve your workflow. If you face any issues, feel free to refer back to this tutorial.
18 March 2025 · 7 min to read
Linux

How to Compress Files in Linux Using tar Command

The tar command basically functions to put together all files and directories into one archive without altering their structure. The approach simplifies organization, creation of the backup, and the transfer of files. Once packaged, you can compress these archives by using multiple ways such as using gzip, bzip2, or xz, which help optimize storage and enhance transfer speeds. Modern Linux distributions come with updated versions of tar, enabling seamless integration with compression tools like gzip for more efficient data handling. This makes tar a valuable asset for users managing large datasets, as it supports both file consolidation and compression in a single command. Thanks to its flexibility, tar is widely used across different Linux environments. It not only facilitates backup creation but also streamlines software distribution and the management of the important data. With an array of choices available, all users can customize archives according to their requirements, whether by excluding particular directories or files, preserving permissions, or securing sensitive data. For anyone dealing with extensive information or complex storage requirements, learning everything about the tar command is crucial. This all makes it an important utility to learn for Linux users. Understand the Syntax of tar  The tar command is fundamentally divided into four distinct parts: tar (keyword) -flags (options), used to execute a specific action name of the archive path to the desired file or directory It would be written as follows: tar -flags (archive_name) (path) Archiving Files and Directories tar used with the flag -cvf has the power to essentially archive the files and also the directories. For a File: tar -cvf collectionX.tar snake.txt For a Directory: tar -cvf DRcollection.tar newDir/ This would essentially archive the file snake.txt to collectionX.tar and the directory newDir to DRcollection.tar respectively.  If desired outcome is to archive multiple files and directories, then use the following commands.For Multiple Files: tar -cvf collectionX.tar snake.txt panther.txt Tiger.txt For Multiple Directories: tar -cvf DRcollection.tar newDir1/ newDir2/ newDir3/ Compressing Files and Directories tar used with the flag -czvf has the power to compress the files as well as the directories: For a File: tar -czvf collectionX.tar.gz snake.txt For a Directory:  tar -czvf DRcollection.tar.gz newDir/ -c archives the directories and files, -z pushes for gzip compression, -v is verbose which essentially shows what’s going on with compression, and -f allows to name the archive that is going to be compressed. Add .gz after tar, if you want to compress the files and directories. For Multiple Files: tar -cvf collectionX.tar.gz snake.txt panther.txt Tiger.txt  For Multiple Directories: tar -cvf DRcollection.tar.gz newDir1/ newDir2/ newDir3/ .bz2 used with tar and both used with -cjf allow to archive and compress files and directories. -j applies bzip2 compression. For a File (with bz2): tar -cjf collectionX.tar.bz2 snake.txt For a Directory (with bz2): tar -cjf DRcollection.tar.bz2 newDir/ .xz used with .tar and both used with -cJf allow you to archive and compress files and directories. In -cJf, -J means compress with xz. For a File (with xz): tar -cJf DRcollection.tar.xz file1.txt For a Directory (with xz): tar -cJf collectionX.tar.xz newDir/ Extracting Compressed .tar Files arch1.tar.gz, arch1.tar.bz2 and arch1.tar.xz are three compressed files. Extract tar.gz: tar -xvzf arch1.tar.gz -x stands for file extraction. Extract tar.bz2: tar -xvjf arch1.tar.bz2 Extract tar.xz: tar -xvJf arch1.tar.xz Extracting Specific Files Using Wildcards If you need to extract only a specific type of file out of an archive, do this: tar -xvf arch1.tar --wildcards '*.sh' It will give you only the files with .sh extension. --wildcards help search that specific type of file and enable pattern matching while *.sh ensures that you only extract the .sh type of files. Extracting to a Specific Directory If you need to extract the complete archive to a specific directory, do this: tar -xvf arch1.tar -C ./destinationDir/pathDir/ -C changes to the specified directory path and -xvf helps extract the archive there.  Managing .tar Archives Check Contents without Extracting If you need to know what's inside an archive but don't want to uncompress files, use commands like this: tar -tzf arch1.tar.gztar -tjf arch1.tar.bz2tar -tJf arch1.tar.xz -t gives details about what’s inside the compressed archives without performing extraction on it. Appending Files to an Existing Archive To append a new file to an archive: tar -rvf arch1.tar new.sh new.sh will be added to arch1.tar. That’s how you append a file into an existing archive.  Removing a Specific File from an Archive What if you need to delete a file from an archive without having to extract it, it can be done by using --delete. tar --delete -f arch1.tar new.sh  This will remove the file new.sh from the archive arch1.tar without extracting it.  Note that --delete does not work on the compressed files, only on archives.  Comparing Archive Contents with Current Directory If you have to examine the contents of your current working directory and compare them with the archive? use: tar --diff -f arch1.tar --diff will help compare the contents of arch1.tar with the content available in the present working directory. Troubleshooting Common .tar Errors "tar: Removing leading '/' from member names" This warning appears when absolute paths are used in an archive: tar -cvf arch1.tar /home/user/file.txt Solution: Use -p to preserve absolute paths. tar -cvpf arch1.tar /home/user/file.txt "tar: Error opening archive: Unrecognized archive format" This error occurs when the archive is corrupt or the wrong decompression command is used. Solution: Verify the file type: file arch1.tar.gz Use the correct decompression command: tar -xvzf arch1.tar.gz  # For .tar.gztar -xvjf arch1.tar.bz2  # For .tar.bz2tar -xvJf arch1.tar.xz   # For .tar.xz If corruption is suspected, check integrity: gzip -t arch1.tar.gzbzip2 -tv arch1.tar.bz2 Conclusion The tar utility serves as an important tool for archiving, compression and extraction. It provides efficiency, making it a crucial component of Linux storage management. With a variety of configurations and settings, tar functions as an evergreen solution catering to diverse use scenarios. Options such as -czvf and -xvzf determine the way files are stored and retrieved, granting users complete control over data compression. Furthermore, tar supports multiple compression tools like gzip, bzip2, and xz, allowing users to optimize both speed and compression ratio based on their specific needs. For Information Technology professionals, developers, and Linux users, learning everything about tar is invaluable. Whether it’s for managing backups, distribution of data effectively, or optimizing storage, tar is by far one of the most influential archiving tools. By selecting the right configurations and commands, users can significantly enhance their workflow, automate tasks, and efficiently handle large datasets.
12 March 2025 · 6 min to read

Do you have questions,
comments, or concerns?

Our professionals are available to assist you at any moment,
whether you need help or are just unsure of where to start.
Email us
Hostman's Support