Sign In
Sign In

Writing Scripts in Linux Bash

Writing Scripts in Linux Bash
Hostman Team
Technical writer
Linux
14.10.2024
Reading time: 12 min

Bash (Bourne-Again SHell) is a command interpreter in UNIX-like operating systems that allows for task automation at the command line level. Bash scripts are files containing a sequence of commands that can be executed by the Bash interpreter.

Bash scripts can be used to automate repetitive tasks. For example, if you need to generate and send a report via email every day, you can write a bash script that performs these actions automatically. This saves a lot of time and reduces the likelihood of errors.

In this article, we will cover the basic concepts and tools for writing Bash scripts in Linux.

Bash Script Syntax

Bash scripts can be written in any text editor and must have executable permissions. Let’s consider some of the most popular editors:

  • Nano is a simple text editor that comes with most Linux distributions. It has an intuitive interface and useful features like syntax highlighting.

  • Vim is one of the most popular text editors for Linux, though it may seem complicated for beginners. Vim offers many features to speed up coding, such as syntax highlighting, autocompletion, and macros.

  • Emacs is another popular text editor for Linux. It also has many features that can simplify the coding process. One of its main features is the ability to run the Bash interpreter inside the editor, allowing you to test scripts without exiting the editor.

At the beginning of each script, there must be a line called a shebang, which tells the operating system which interpreter to use to execute the script. The shebang should start with a hash symbol (#) followed by an exclamation mark (!), and then the path to the interpreter. To use the Bash interpreter, the shebang will look like this:

#!/bin/bash

While writing the script, you can also leave comments that start with a hash symbol and continue until the end of the line. Comments will not be executed by the interpreter and are used to describe the functionality of the script. For example:

# This is a comment

Below, we will write our first script. Suppose we want to create a script in Linux that greets the user and displays the current date and time on the screen. To do this, create a file named greeting.sh in any directory on your computer and add the following code:

#!/bin/bash
echo "Hello, $USER!"
echo "Today is $(date)"

The first line indicates that this is a Bash script. The next line, echo "Hello $USER!", outputs a greeting with the current user's name. $USER is a system variable that contains the name of the current user. The third line, echo "Today is $(date)", displays the current date and time. $(date) is used to call the date command, which returns the current date and time in the system's format.

When creating a Bash script, it’s important to ensure the file is executable. To do this, you need to change the file permissions. We’ll cover this and how to run the script in the next chapter.

Running Scripts

To run a script in Linux, it must have executable permissions. To make a file executable, you can use the chmod command (short for "change mode"). This command allows you to change the access permissions of files and directories in Linux.

The syntax for the chmod command is as follows:

chmod [options] access_rights file

where access_rights is a special code that sets the access permissions for a file or directory, and file is the path to the file or directory whose permissions you want to change.

To make a file executable, you need to add the execute (x) permission to its access rights. For example, to make the greeting.sh file executable, use the following command:

chmod +x greeting.sh

This command will add execute permissions for the current user. Now, we can run the Bash script in Linux by invoking it from the terminal:

./greeting.sh

The result of running the script is shown below.

Image9

Command Line Parameters

Command line parameters allow you to pass arguments to Linux scripts when they are run. Command line parameters can be accessed in the script as $1, $2, $3, etc., where $1 is the first parameter, $2 is the second parameter, and so on.

Let's rewrite the script from the previous chapter to greet the user using a command-line argument:

#!/bin/bash
echo "Hello $1!"

Then run the script, passing the $USER argument:

./greeting.sh $USER

The result is shown below.

Image5

Additionally, you can use special command line parameters:

  • $0 — the name of the script (i.e., the name of the file that was run)

  • $# — the number of passed parameters

  • $* or $@ — a list of all passed parameters (as a single string or array, respectively)

  • $? — the return code of the last executed command

For example, to display the number of passed parameters, you can use the following code:

#!/bin/bash
echo "Hello $1!" 
echo "Number of passed parameters: $#"

The result of running the script is shown below.

Image4

Variables

Variables in Bash are used to store data, such as strings and numbers. They can be explicitly defined by assigning a value or implicitly defined through automatic assignment during certain operations. To create a variable in Bash, you need to assign it a value using an equal sign (=). For example:

company="Hostman"

Note that there should be no spaces between the variable name, the equal sign, and the value.

You can retrieve the value of a variable by specifying its name after the echo command and the $ sign. For example:

echo $company

It's also possible to assign a variable value through user input using the read command. For example, the following script prompts the user for their name and stores it in a variable:

#!/bin/bash
echo "What is your name?"
read name
echo "Hello, $name!"

The result of this script is shown below.

Image8

In Bash, there are several special variables that are automatically defined and filled by the system. For example, the $HOME variable contains the path to the user's home directory, while $PWD contains the path to the current working directory. 

Additionally, there are environment variables that are defined by the system and can be used in scripts. For example, $PATH contains a list of directories where Bash looks for executable files.

Variables can also be used to pass values between different commands and scripts. For example, to pass a variable’s value from one script to another, use the export command:

export variable_name

Conditional Operators

Conditional operators allow you to execute a specific set of actions depending on whether a condition is true or false. In Bash scripts, conditions are written in brackets and passed to the if command.

The syntax of the if operator looks like this:

if [ condition ]
then
  commands to execute if the condition is true
fi

Here, in the square brackets, you specify the condition that needs to be checked. If the condition is true, the commands between then and fi will be executed.

For example, let’s write a Linux script, evenodd.sh, that checks whether the number entered by the user is even or odd:

#!/bin/bash
echo "Enter a number: "
read n
if (( $n % 2 == 0 ))
then
  echo "The number $n is even"
else
  echo "The number $n is odd"
fi

In this example, we use the % operator, which calculates the remainder of division by 2. If the remainder is 0, the number is even; otherwise, it’s odd. The result of running the script is shown below.

Image3

Additionally, there are several comparison operators that can be used in conditional constructions:

  • -eq – equal to;

  • -ne – not equal to;

  • -gt – greater than;

  • -lt – less than;

  • -ge – greater than or equal to;

  • -le – less than or equal to.

For example, to check if the variable $a is greater than the variable $b, you can write the following:

if [ $a -gt $b ]
then
  echo "$a is greater than $b"
fi

It is important to remember that you need to use spaces around the comparison operators in conditional constructions. If there are no spaces, Bash will treat this as one large string instead of a comparison operation.

In addition to if, Bash scripts also use the case structure. This allows you to check a variable's value against several possible options. We will discuss this in the next chapter.

The Case Construction

The case construction in Bash scripts allows you to simplify writing conditional operators for comparing variables with multiple possible values.

The syntax of the case construction is as follows:

case variable in
    pattern1)
        command1
        ;;
    pattern2)
        command2
        ;;
    pattern3)
        command3
        ;;
    *)
        default command
        ;;
esac

where variable is the variable to check, pattern1, pattern2, pattern3 are the possible values to check, and command1, command2, command3 are the commands to execute depending on the value of the variable.

The * symbol at the end of the list of values is used as a default handler if none of the values match the variable.

For example, let’s look at a script that checks the day of the week and performs the corresponding action:

#!/bin/bash

day=$(date +%u)

case $day in
    1)
        echo "Today is Monday"
        ;;
    2)
        echo "Today is Tuesday"
        ;;
    3)
        echo "Today is Wednesday"
        ;;
    4)
        echo "Today is Thursday"
        ;;
    5)
        echo "Today is Friday"
        ;;
    6)
        echo "Today is Saturday"
        ;;
    7)
        echo "Today is Sunday"
        ;;
    *)
        echo "Invalid day of the week"
        ;;
esac

In this example, we use the day variable, which we define using the date +%u command. In this case, %u is used to obtain the numeric value of the day of the week, from 1 (Monday) to 7 (Sunday). Then we compare this variable with the days of the week using the case construction. If its value matches a certain day of the week, we display the corresponding message. If the value does not match any of the listed days, we display an error message.

The result of running the script is shown below. 

Image1

Loops

Loops in Bash are used to perform repetitive actions. There are two types of loops: for and while.

The for loop is used to execute commands for each element in a list.

The syntax of the for loop is as follows:

for variable in list
do
  commands
done

Here, the variable takes the value of an element from the list, and for each of them, the commands between do and done are executed.

Example:

#!/bin/bash

for i in {1..10}; do
    echo "Number: $i"
done

In this example, i takes values from 1 to 10, and for each of them, the echo "Number: $i" command will be executed. The result of running this loop will look like this:

Image10

The while loop is used to execute commands as long as the condition remains true. The syntax of the while loop is as follows:

while [ condition ]
do
  commands
done

Here, in square brackets, you specify the condition that is checked before each iteration of the loop. The commands between do and done will be executed as long as the condition is true.

Example:

#!/bin/bash

count=1
while [ $count -le 10 ]; do
    echo "Count: $count"
    count=$((count+1))
done

In this example, count increases by 1 after each iteration of the loop. When the value of count reaches 10, the loop terminates. The result of running this loop will look like this:

Image2

Functions

Functions in Bash are used to group commands into logically related blocks. Functions can be called from a script using their name. 

The syntax of a function is as follows:

function_name () {
    commands_and_expressions
}

The function name must start with a letter or an underscore and can contain only letters, numbers, and underscores. After the function name comes a list of arguments in parentheses. The commands and expressions to be executed when the function is called must be enclosed in curly braces.

Here’s an example of a function that outputs the current time and date:

#!/bin/bash

print_date () {
    echo "Today's date: $(date)"
}

print_date # Function call

The result of running the script is shown below.

Image7

Functions can also accept arguments, which are passed as parameters inside the parentheses when calling the function. Here’s an example of a function that takes two arguments and outputs their sum:

#!/bin/bash

sum_numbers () {
    result=$(( $1 + $2 ))
    echo "The sum of $1 and $2 is $result"
}

sum_numbers 10 20 # Function call

In this example, $1 and $2 are variables that contain the values of the first and second arguments, respectively. sum_numbers 10 20 will call the sum_numbers function with the arguments 10 and 20, and output the following result:

Image11

Functions can also return values using the return keyword. Let’s rewrite the previous example using this new knowledge:

#!/bin/bash

sum_numbers () {
    result=$(( $1 + $2 ))
    return $result
}

sum_numbers 12 24 # Function call
echo "The sum of the numbers is $?" # Output

Here, the result is stored in the result variable and returned from the function using the return command.

The $? variable contains the return code of the function, which in this case is the result of the sum calculation.

The result of running the script is shown below.

Image12

There is another way to handle the result of a function call without using return. Let’s slightly modify the previous script:

#!/bin/bash

sum_numbers () {
    result=$(( $1 + $2 ))
    echo $result
}
sum=$(sum_numbers 9 11)
echo "The sum of the numbers is $sum" # Output

Here, instead of using $? and return, we store the result of the function call in the sum variable and then output its value. The result is shown below.

Image6

Working with Files and Directories

Bash scripts can be used to perform various operations with files and directories in Linux. For example, to check if a file exists, you can use the following command: 

test -e filename 

If the file exists, the command will return a value of 0; otherwise, it will return a non-zero value.

To work with directories in Bash scripts, you can use commands like cd, mkdir, rmdir, ls, and others.

Script Debugging

Debugging Bash scripts can be a challenging task because problems can be caused by various factors, such as syntax errors, incorrect use of variables or functions, etc. For debugging Bash scripts, you can use tools like set -x, set -v, and set -e.

  • The set -x command allows you to display the commands before they are executed

  • The set -v command displays the values of variables before they are used

  • The set -e command stops the execution of the script in case of an error

Conclusion

Bash scripts are a powerful tool for automating tasks in UNIX-like operating systems. In this article, we covered the basic concepts and tools for writing Bash scripts, such as syntax, variables, conditional operators, loops, functions, and running scripts. We hope this guide helps you become a more productive and experienced Linux user.

You can buy Linux VPS for your projects on Hostman. 

Linux
14.10.2024
Reading time: 12 min

Similar

Linux

How to Use SSH Keys for Authentication

Many cloud applications are built on the popular SSH protocol—it is widely used for managing network infrastructure, transferring files, and executing remote commands. SSH stands for Secure Socket Shell, meaning it provides a shell (command-line interface) around the connection between multiple remote hosts, ensuring that the connection is secure (encrypted and authenticated). SSH connections are available on all popular operating systems, including Linux, Ubuntu, Windows, and Debian. The protocol establishes an encrypted communication channel within an unprotected network by using a pair of public and private keys. Keys: The Foundation of SSH SSH operates on a client-server model. This means the user has an SSH client (a terminal in Linux or a graphical application in Windows), while the server side runs a daemon, which accepts incoming connections from clients. In practice, an SSH channel enables remote terminal management of a server. In other words, after a successful connection, everything entered in the local console is executed directly on the remote server. The SSH protocol uses a pair of keys for encrypting and decrypting information: public key and private key. These keys are mathematically linked. The public key is shared openly, resides on the server, and is used to encrypt data. The private key is confidential, resides on the client, and is used to decrypt data. Of course, keys are not generated manually but with special tools—keygens. These utilities generate new keys using encryption algorithms fundamental to SSH technology. More About How SSH Works Exchange of Public Keys SSH relies on symmetric encryption, meaning two hosts wishing to communicate securely generate a unique session key derived from the public and private data of each host. For example, host A generates a public and private key pair. The public key is sent to host B. Host B does the same, sending its public key to host A. Using the Diffie-Hellman algorithm, host A can create a key by combining its private key with the public key of host B. Likewise, host B can create an identical key by combining its private key with the public key of host A. This results in both hosts independently generating the same symmetric encryption key, which is then used for secure communication. Hence, the term symmetric encryption. Message Verification To verify messages, hosts use a hash function that outputs a fixed-length string based on the following data: The symmetric encryption key The packet number The encrypted message text The result of hashing these elements is called an HMAC (Hash-based Message Authentication Code). The client generates an HMAC and sends it to the server. The server then creates its own HMAC using the same data and compares it to the client's HMAC. If they match, the verification is successful, ensuring that the message is authentic and hasn't been tampered with. Host Authentication Establishing a secure connection is only part of the process. The next step is authenticating the user connecting to the remote host, as the user may not have permission to execute commands. There are several authentication methods: Password Authentication: The user sends an encrypted password to the server. If the password is correct, the server allows the user to execute commands. Certificate-Based Authentication: The user initially provides the server with a password and the public part of a certificate. Once authenticated, the session continues without requiring repeated password entries for subsequent interactions. These methods ensure that only authorized users can access the remote system while maintaining secure communication. Encryption Algorithms A key factor in the robustness of SSH is that decrypting the symmetric key is only possible with the private key, not the public key, even though the symmetric key is derived from both. Achieving this property requires specific encryption algorithms. There are three primary classes of such algorithms: RSA, DSA, and algorithms based on elliptic curves, each with distinct characteristics: RSA: Developed in 1978, RSA is based on integer factorization. Since factoring large semiprime numbers (products of two large primes) is computationally difficult, the security of RSA depends on the size of the chosen factors. The key length ranges from 1024 to 16384 bits. DSA: DSA (Digital Signature Algorithm) is based on discrete logarithms and modular exponentiation. While similar to RSA, it uses a different mathematical approach to link public and private keys. DSA key length is limited to 1024 bits. ECDSA and EdDSA: These algorithms are based on elliptic curves, unlike DSA, which uses modular exponentiation. They assume that no efficient solution exists for the discrete logarithm problem on elliptic curves. Although the keys are shorter, they provide the same level of security. Key Generation Each operating system has its own utilities for quickly generating SSH keys. In Unix-like systems, the command to generate a key pair is: ssh-keygen -t rsa Here, the type of encryption algorithm is specified using the -t flag. Other supported types include: dsa ecdsa ed25519 You can also specify the key length with the -b flag. However, be cautious, as the security of the connection depends on the key length: ssh-keygen -b 2048 -t rsa After entering the command, the terminal will prompt you to specify a file path and name for storing the generated keys. You can accept the default path by pressing Enter, which will create standard file names: id_rsa (private key) and id_rsa.pub (public key). Thus, the public key will be stored in a file with a .pub extension, while the private key will be stored in a file without an extension. Next, the command will prompt you to enter a passphrase. While not mandatory (it is unrelated to the SSH protocol itself), using a passphrase is recommended to prevent unauthorized use of the key by a third-party user on the local Linux system. Note that if a passphrase is used, you must enter it each time you establish the connection. To change the passphrase later, you can use: ssh-keygen -p Or, you can specify all parameters at once with a single command: ssh-keygen -p old_password -N new_password -f path_to_files For Windows, there are two main approaches: Using ssh-keygen from OpenSSH: The OpenSSH client provides the same ssh-keygen command as Linux, following the same steps. Using PuTTY: PuTTY is a graphical application that allows users to generate public and private keys with the press of a button. Installing the Client and Server Components The primary tool for an SSH connection on Linux platforms (both client and server) is OpenSSH. While it is typically pre-installed on most operating systems, there may be situations (such as with Ubuntu) where manual installation is necessary. The general command for installing SSH, followed by entering the superuser password, is: sudo apt-get install ssh However, in some operating systems, SSH may be divided into separate components for the client and server. For the Client To check whether the SSH client is installed on your local machine, simply run the following command in the terminal: ssh If SSH is supported, the terminal will display a description of the command. If nothing appears, you’ll need to install the client manually: sudo apt-get install openssh-client You will be prompted to enter the superuser password during installation. Once completed, SSH connectivity will be available. For the Server Similarly, the server-side part of the OpenSSH toolkit is required on the remote host. To check if the SSH server is available on your remote host, try connecting locally via SSH: ssh localhost If the SSH daemon is running, you will see a message indicating a successful connection. If not, you’ll need to install the SSH server: sudo apt-get install openssh-server As with the client, the terminal will prompt you to enter the superuser password. After installation, you can check whether SSH is active by running: sudo service ssh status Once connected, you can modify SSH settings as needed by editing the configuration file: ./ssh/sshd_config For example, you might want to change the default port to a custom one. Don’t forget that after making changes to the configuration, you must manually restart the SSH service to apply the updates: sudo service ssh restart Copying an SSH Key to the Server On Hostman, you can easily add SSH keys to your servers using the control panel. Using a Special Copy Command After generating a public SSH key, it can be used as an authorized key on a server. This allows quick connections without the need to repeatedly enter a password. The most common way to copy the key is by using the ssh-copy-id command: ssh-copy-id -i ~/.ssh/id_rsa.pub name@server_address This command assumes you used the default paths and filenames during key generation. If not, simply replace ~/.ssh/id_rsa.pub with your custom path and filename. Replace name with the username on the remote server. Replace server_address with the host address. If the usernames on both the client and server are the same, you can shorten the command: ssh-copy-id -i ~/.ssh/id_rsa.pub server_address If you set a passphrase during the SSH key creation, the terminal will prompt you to enter it. Otherwise, the key will be copied immediately. In some cases, the server may be configured to use a non-standard port (the default is 22). If that’s the case, specify the port using the -p flag: ssh-copy-id -i ~/.ssh/id_rsa.pub -p 8129 name@server_address Semi-Manual Copying There are operating systems where the ssh-copy-id command may not be supported, even though SSH connections to the server are possible. In such cases, the copying process can be done manually using a series of commands: ssh name@server_address 'mkdir -pm 700 ~/.ssh; echo ' $(cat ~/.ssh/id_rsa.pub) ' >> ~/.ssh/authorized_keys; chmod 600 ~/.ssh/authorized_keys' This sequence of commands does the following: Creates a special .ssh directory on the server (if it doesn’t already exist) with the correct permissions (700) for reading and writing. Creates or appends to the authorized_keys file, which stores the public keys of all authorized users. The public key from the local file (id_rsa.pub) will be added to it. Sets appropriate permissions (600) on the authorized_keys file to ensure it can only be read and written by the owner. If the authorized_keys file already exists, it will simply be appended with the new key. Once this is done, future connections to the server can be made using the same SSH command, but now the authentication will use the public key added to authorized_keys: ssh name@server_address Manual Copying Some hosting platforms offer server management through alternative interfaces, such as a web-based control panel. In these cases, there is usually an option to manually add a public key to the server. The web interface might even simulate a terminal for interacting with the server. Regardless of the method, the remote host must contain a file named ~/.ssh/authorized_keys, which lists all authorized public keys. Simply copy the client’s public key (found in ~/.ssh/id_rsa.pub by default) into this file. If the key pair was generated using a graphical application (typically PuTTY on Windows), you should copy the public key directly from the application and add it to the existing content in authorized_keys. Connecting to a Server To connect to a remote server on a Linux operating system, enter the following command in the terminal: ssh name@server_address Alternatively, if the local username is identical to the remote username, you can shorten the command to: ssh server_address The system will then prompt you to enter the password. Type it and press Enter. Note that the terminal will not display the password as you type it. Just like with the ssh-copy-id command, you can explicitly specify the port when connecting to a remote server: ssh client@server_address -p 8129 Once connected, you will have control over the remote machine via the terminal; any command you enter will be executed on the server side. Conclusion Today, SSH is one of the most widely used protocols in development and system administration. Therefore, having a basic understanding of its operation is crucial. This article aimed to provide an overview of SSH connections, briefly explain the encryption algorithms (RSA, DSA, ECDSA, and EdDSA), and demonstrate how public and private key pairs can be used to establish secure connections with a personal server, ensuring that exchanged messages remain inaccessible to third parties. We covered the primary commands for UNIX-like operating systems that allow users to generate key pairs and grant clients SSH access by copying the public key to the server, enabling secure connections.
30 January 2025 · 10 min to read
Linux

How to Download Files with cURL

Downloading content from remote servers is a regular task for both administrators and developers. Although there are numerous tools for this job, cURL stands out for its adaptability and simplicity. It’s a command-line utility that supports protocols such as HTTP, HTTPS, FTP, and SFTP, making it crucial for automation, scripting, and efficient file transfers. You can run cURL directly on your computer to fetch files. You can also include it in scripts to streamline data handling, thereby minimizing manual effort and mistakes. This guide demonstrates various ways to download files with cURL. By following these examples, you’ll learn how to deal with redirects, rename files, and monitor download progress. By the end, you should be able to use cURL confidently for tasks on servers or in cloud setups. Basic cURL Command for File Download The curl command works with multiple protocols, but it’s primarily used with HTTP and HTTPS to connect to web servers. It can also interact with FTP or SFTP servers when needed. By default, cURL retrieves a resource from a specified URL and displays it on your terminal (standard output). This is often useful for previewing file contents without saving them, particularly if you’re checking a small text file. Example: To view the content of a text file hosted at https://example.com/file.txt, run: curl https://example.com/file.txt For short text documents, this approach is fine. However, large or binary files can flood the screen with unreadable data, so you’ll usually want to save them instead. Saving Remote Files Often, the main goal is to store the downloaded file on your local machine rather than see it in the terminal. cURL simplifies this with the -O (capital O) option, which preserves the file’s original remote name. curl -O https://example.com/file.txt This retrieves file.txt and saves it in the current directory under the same name. This approach is quick and retains the existing filename, which might be helpful if the file name is significant. Choosing a Different File Name Sometimes, renaming the downloaded file is important to avoid collisions or to create a clear naming scheme. In this case, use the -o (lowercase o) option: curl -o myfile.txt https://example.com/file.txt Here, cURL downloads the remote file file.txt but stores it locally as myfile.txt. This helps keep files organized or prevents accidental overwriting. It’s particularly valuable in scripts that need descriptive file names. Following Redirects When requesting a file, servers might instruct your client to go to a different URL. Understanding and handling redirects is critical for successful downloads. Why Redirects Matter Redirects are commonly used for reorganized websites, relocated files, or mirror links. Without redirect support, cURL stops after receiving an initial “moved” response, and you won’t get the file. Using -L or --location To tell cURL to follow a redirect chain until it reaches the final target, use -L (or --location): curl -L -O https://example.com/redirected-file.jpg This allows cURL to fetch the correct file even if its original URL points elsewhere. If you omit -L, cURL will simply print the redirect message and end, which is problematic for sites with multiple redirects. Downloading Multiple Files cURL can also handle multiple file downloads at once, saving you from running the command repeatedly. Using Curly Braces and Patterns If filenames share a pattern, curly braces {} let you specify each name succinctly: curl -O https://example.com/files/{file1.jpg,file2.jpg,file3.jpg} cURL grabs each file in sequence, making it handy for scripted workflows. Using Ranges For a series of numbered or alphabetically labeled files, specify a range in brackets: curl -O https://example.com/files/file[1-5].jpg cURL automatically iterates through files file1.jpg to file5.jpg. This is great for consistently named sequences of files. Chaining Multiple Downloads If you have different URLs for each file, you can chain them together: curl -O https://example1.com/file1.jpg -O https://example2.com/file2.jpg This approach downloads file1.jpg from the first site and file2.jpg from the second without needing multiple commands. Rate Limiting and Timeouts In certain situations, you may want to control the speed of downloads or prevent cURL from waiting too long for an unresponsive server. Bandwidth Control To keep your network from being overwhelmed or to simulate slow conditions, limit the download rate with --limit-rate: curl --limit-rate 2M -O https://example.com/bigfile.zip 2M stands for 2 megabytes per second. You can also use K for kilobytes or G for gigabytes. Timeouts If a server is too slow, you may want cURL to stop after a set time. The --max-time flag does exactly that: curl --max-time 60 -O https://example.com/file.iso Here, cURL quits after 60 seconds, which is beneficial for scripts that need prompt failures. Silent and Verbose Modes cURL can adjust its output to show minimal information or extensive details. Silent Downloads For batch tasks or cron jobs where you don’t need progress bars, include -s (or --silent): curl -s -O https://example.com/file.jpg This hides progress and errors, which is useful for cleaner logs. However, troubleshooting is harder if there’s a silent failure. Verbose Mode In contrast, -v (or --verbose) prints out detailed request and response information: curl -v https://example.com Verbose output is invaluable when debugging issues like invalid SSL certificates or incorrect redirects. Authentication and Security Some downloads require credentials, or you might need a secure connection. HTTP/FTP Authentication When a server requires a username and password, use -u: curl -u username:password -O https://example.com/protected/file.jpg Directly embedding credentials can be risky, as they might appear in logs or process lists. Consider environment variables or .netrc files for more secure handling. HTTPS and Certificates By default, cURL verifies SSL certificates. If the certificate is invalid, cURL blocks the transfer. You can bypass this check with -k or --insecure, though it introduces security risks. Whenever possible, use a trusted certificate authority so that connections remain authenticated. Using a Proxy In some environments, traffic must route through a proxy server before reaching the target. Downloading Through a Proxy Use the -x or --proxy option to specify the proxy: curl -x http://proxy_host:proxy_port -O https://example.com/file.jpg Replace proxy_host and proxy_port with the relevant details. cURL forwards the request to the proxy, which then retrieves the file on your behalf. Proxy Authentication If your proxy requires credentials, embed them in the URL: curl -x https://proxy.example.com:8080 -U myuser:mypassword -O https://example.com/file.jpg Again, storing sensitive data in plain text can be dangerous, so environment variables or configuration files offer more secure solutions. Monitoring Download Progress Tracking download progress is crucial for large files or slower links. Default Progress Meter By default, cURL shows a progress meter, including total size, transfer speed, and estimated finish time. For example: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100  1256  100  1256    0     0   2243      0 --:--:-- --:--:-- --:--:--  2246 This readout helps you gauge how much remains and if the transfer rate is acceptable. Compact Progress Bar If you want fewer details, add -#: curl -# -O https://example.com/largefile.iso A simpler bar shows the overall progress as a percentage. It’s easier on the eyes but lacks deeper stats like current speed. Capturing Progress in Scripts When using cURL within scripts, you might want to record progress data. cURL typically sends progress info to stderr, so you can redirect it: curl -# -O https://example.com/largefile.iso 2>progress.log Here, progress.log contains the status updates, which you can parse or store for later review. Conclusion cURL shines as a flexible command-line tool for downloading files in multiple protocols and environments. Whether you need to handle complex redirects, rename files on the fly, or throttle bandwidth, cURL has you covered. By mastering its core flags and modes, you’ll be able to integrate cURL seamlessly into your daily workflow for scripting, automation, and more efficient file transfers.
29 January 2025 · 7 min to read
Linux

How to Extract or Unzip .tar.gz Files in Linux

Exploring the Linux landscape often means dealing with several file formats, especially compressed ones like .tar.gz. This format is popular because it combines multiple documents and folders into one compressed archive. Whether you're obtaining software packages, organizing project backups, or overseeing data storage, mastering this format usage is essential.  Throughout this guide, we will examine various strategies for unpacking .gz archives in Linux. From the versatile tar command to the more straightforward gzip and gunzip commands, we'll cover everything. We'll also dive into combining commands like unzip and tar, and using graphical interfaces for those who prefer a more visual approach. Why Choose .tar.gz? Listed below are few key reasons why you might opt to utilize this format: Space Efficiency: The combination of tar and gzip allows for the streamlined compression of large data amounts, enhancing disk space usage. Simplified Data Management: Merging several documents and directories into a single archive enhances data management and organizes storage. Easy Distribution: This widely-adopted format ensures seamless transfers between systems without any compatibility hurdles. Preservation of Metadata: The tar utility maintains file permissions and timestamps, making it perfect for backups and migrating systems. Creating a .tar.gz File Before jumping into extraction, it's helpful to know how to create an archive. This makes it easier to combine and compress many documents into one neat, smaller package. Here is the standard syntax for creation: tar -czf archive-name.tar.gz file1 file2 directory1 Where: c: Creates an entirely new archive. z: Perform compression. f: Assigns a specific name to the archive. For instance, to compress report1, report2, and the directory projects into a file called backup, apply: tar -czf backup.tar.gz report1.txt report2.txt projects For verification, list the directory items via: ls Examining .tar.gz Content To examine the items without extracting them, use a command that lists every compressed item. This is particularly handy for verifying items before unpacking. To list .gz content: tar -ztvf archive-name.tar.gz For instance, to list the items of backup: tar -ztvf backup.tar.gz Extracting .tar.gz in Linux Linux offers a variety of extraction methods for these archives, each bringing its own advantages. Here are comprehensive instructions for utilizing various commands and tools. Method 1: Via tar Utility The tar command is a powerful and flexible utility designed to manage compressed documents, offering functions to create, extract, and display the items of archives. This command is your ultimate tool for handling .gz resources efficiently. Basic Extraction To unpack .gz items directly into the current directory, apply: tar -xvzf archive-name.tar.gz Where: x: Unpacks the archive's items. v: Verbose mode actively displays each file being unpacked. z: Decompresses the data. f: Gives the archive a unique name. For unpacking the backup, apply: tar -xvzf backup.tar.gz Extracting to a Specific Directory For placing the unpacked files in a different location, use the -C option to indicate your chosen directory. This is handy when you need to ensure your retrieved file are neatly arranged in a designated location. To unpack the items into a chosen directory, apply: tar -xvzf archive-name.tar.gz -C /path/to/destination For instance, to unpack the backup into the Documents folder, utilize: tar -xvzf backup.tar.gz -C /home/user/Documents Extracting Specific Content For retrieving certain items from the archive, simply provide their names. This enables you to pinpoint and retrieve just the necessary data.  Here’s the format: tar -xvzf archive-name.tar.gz file1 file2 For example, to retrieve report1 and report2 from backup, apply: tar -xvzf backup.tar.gz report1.txt report2.txt Extracting Contents with a Specific Extension For retrieving items with a particular extension, the --wildcards option proves to be quite useful. This option lets you filter and retrieve data based on their names or extensions. Here's the syntax: tar -xvzf archive-name.tar.gz --wildcards '*.txt' For instance, to retrieve all .txt docs from backup: tar -xvzf backup.tar.gz --wildcards '*.txt' Method 2: Via gzip Utility The gzip is a tool primarily used for compressing data, but it can also decompress them with the -d option. This method is straightforward and effective for handling .gz resources. To unzip a .gz file, apply the subsequent command: gzip -d archive-name.tar.gz For instance, to unpack backup, apply: gzip -d backup.tar.gz After decompressing, retrieve the items via: tar -xf archive-name.tar For instance: tar -xf backup.tar Method 3: Via gunzip Utility The gunzip is a specifically designed tool for decompressing .gz documents, functioning as an alias for gzip -d. This command is simple to use and directly addresses the need to decompress .gz files. To decompress, apply: gunzip archive-name.tar.gz For example: gunzip backup.tar.gz After decompressing, unpack the items through: tar -xf archive-name.tar For example: tar -xf backup.tar Method 4: Via GUI For users who favor a GUI, various Linux desktop environments include file managers equipped with extraction tools. This method is user-friendly and ideal for beginners. Extracting Contents to the Current Directory Find the .gz file within your file manager. Right-click on it and choose "Extract." Extracting Contents to a Specific Directory Spot the .gz file within your file explorer. Right-click on it and select "Extract to…". Choose the destination directory. Handling Large Archives with Parallel Decompression When handling massive archives, pigz (Parallel Implementation of gzip) can significantly enhance decompression speed by using several CPU cores. Here's how to use it: Install pigz on Linux via: sudo apt install pigz To uncompress a .gz file via pigz, apply: pigz -d archive-name.tar.gz After decompression, retrieve the resulting .tar doc with: tar -xf archive-name.tar Utilizing Compression with Encryption For added security, you can encrypt your .gz doc. GPG (GNU Privacy Guard) can be used to encrypt documents, ensuring that sensitive information remains protected during storage and transfer. Encrypting an Archive For encryption, use GPG with the following command: gpg -c archive-name.tar.gz Decrypting an Archive To decrypt an encrypted archive, apply: gpg -d archive-name.tar.gz.gpg > archive-name.tar.gz Tips for Content Extraction in Linux Backup Important Docs: Always create backups before unpacking multiple docs to avoid data loss. Check Permissions: Ensure you possess the required permissions to retrieve documents in the designated directory. Utilize Wildcards Carefully: Be cautious when using wildcards to avoid unintentional extraction. Troubleshooting Frequent Issues with Extraction Here are a few common extraction difficulties and the ways to address them: Corrupted Archives In case an archive is corrupted, try using the --ignore-zeros option to retrieve it: tar -xvzf archive-name.tar.gz --ignore-zeros Insufficient Permissions Confirm that you have the proper permissions to access and modify files. Utilize sudo if required: sudo tar -xvzf archive-name.tar.gz -C /path/to/destination Disk Space Issues Check that you have enough disk space to unzip the documents. Verify disk usage with: df -h Conclusion Unpacking .tar.gz documents in Linux is a simple task, with multiple methods to cater to different user preferences. Whether you're using the tar, gzip, gunzip commands, or a GUI, Linux equips you with efficient tools to handle compressed data seamlessly. This guide empowers you with the know-how to confidently retrieve .gz docs. Whether it's handling software packages, arranging backups, or managing data storage, mastering the creation and extraction of such files keeps your workflow streamlined and efficient.  By mastering the creation and extraction of these files, you streamline your workflow and enhance your overall efficiency, making data management a breeze.
28 January 2025 · 7 min to read

Do you have questions,
comments, or concerns?

Our professionals are available to assist you at any moment,
whether you need help or are just unsure of where to start.
Email us
Hostman's Support