How to Install, Configure, and Use RabbitMQ

Technical writer

Microservices RabbitMQ

08.11.2024

Reading time: 12 min

Message brokers are intermediary applications used in various software architectures, such as microservices. They transfer information in the form of messages from one service to another.

A common concept associated with message brokers is the "publisher-subscriber" pattern, where events are sent from so-called "publishers" (Producers) to consumers (Consumers).

Typically, an intermediary component—a message broker—participates in implementing this pattern.

To simplify, let’s use an analogy. Think of YouTube: there are channels you can subscribe to for notifications. When a new video is published (an event), you’re notified because you’re subscribed.

How Applications Work with Message Brokers

Three main terms are relevant here: Exchange, Queue, and Binding.

Exchange. This is where message events are sent. The Exchange decides which events go to which queue based on connections—bindings—which we’ll discuss shortly.

Queue. The queue is a data structure that operates on a "first in, first out" (FIFO) basis. It stores data on disk or in memory, directly representing the messages. The queue provides copies of these messages to consumers.

Binding. Binding refers to the set of rules that tell the Exchange which queues messages should be routed to. There can be multiple bindings with different parameters between an Exchange and a Queue.

Other important terms are:

Producer is a service that generates messages and sends them to the broker.
Consumer is another service that begins processing it upon receiving a message. In our YouTube analogy, the "publisher" is YouTube itself, and the "consumer" is your phone, receiving push notifications.

Here’s a step-by-step breakdown of how the process works:

The Producer sends a message, which goes to the Exchange.

The Queue stores this message, for instance, on disk.

When a consumer is ready to receive a message (either because it has finished processing the previous message or the event-handling application has started), the server sends a copy of the data from the queue to the consumer.

The Consumer receives the message, processes it, and then sends an acknowledgment (ACK) back to the broker.

After the broker receives the acknowledgement, the message is deleted.

What is RabbitMQ?

RabbitMQ is a popular open-source message broker built on the AMQP (Advanced Message Queuing Protocol), an open protocol for transmitting event messages through a dedicated broker. AMQP provides a wide range of capabilities, and besides RabbitMQ, it’s implemented by other systems like Apache Qpid.

RabbitMQ is written in the Erlang programming language, and its main advantages are its high throughput and maximum routing flexibility. RabbitMQ offers numerous configuration options for defining rules on which messages are sent where for processing.

Connections to RabbitMQ are made over the TCP protocol: the client initiates a connection to the host address and keeps this connection open for as long as interaction with RabbitMQ is needed. RabbitMQ also supports authentication, such as login and password, for secure access.

Installation on Different Operating Systems

RabbitMQ consists of the server and the WebUI (admin interface). The WebUI allows you to monitor what’s happening within the broker, such as checking if the nodes in the cluster are active, seeing how many messages are in process, and more. Let’s go over how to install RabbitMQ on various operating systems.

Before we begin, we need to update the server hostname. Run the command:

sudo hostnamectl set-hostname <new hostname>

For example, we will use rabbitmq as a hostname for this server.

sudo hostnamectl set-hostname rabbitmq

Install nano:

sudo apt install nano

And enter your new hostname in the /etc/hosts file:

sudo nano /etc/hosts

So it looks like this:

127.0.0.1 <new hostname>

In our example:

127.0.0.1 rabbitmq

Ubuntu / Debian

We will be installing RabbitMQ on Ubuntu 22.04. For other Ubuntu/Debian distributions, check the guide on the official website.

Update the package list and install the dependencies:

sudo apt-get update -y
sudo apt-get install curl gnupg -y
sudo apt-get install apt-transport-https

Add repository signing keys:

curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null
curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-erlang.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null
curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-server.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null

Add a repository (Apt Source List) file.

sudo tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
EOF

Update the package list yet again:

sudo apt-get update -y

Install Erlang:

sudo apt-get install -y erlang-base \
                       erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
                       erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
                       erlang-runtime-tools erlang-snmp erlang-ssl \
                       erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl

Finally, install RabbitMQ:

sudo apt-get install rabbitmq-server -y --fix-missing

After installation is complete, you can check that rabbitmq-server is up and running:

systemctl status rabbitmq-server

Create a custom user:

sudo rabbitmqctl add_user <your username> <your password>

And give the necessary permissions:

sudo rabbitmqctl set_user_tags  <your username> administrator
sudo rabbitmqctl set_permissions -p /  <your username> ".*" ".*" ".*"

Now you can enable RabbitMQ Management Console:

sudo rabbitmq-plugins enable rabbitmq_management

And visit <your server IP>:15672. Enter your username and password and access the RabbitMQ web UI.

Windows

You can find the installation guide on the RabbitMQ website. Basically, you can either use Chocolatey or a simple installer.

However, before installing RabbitMQ, we first need to install Erlang/OTP from the official Erlang website.

Run the downloaded file and go through the installation process.

After installing Erlang, we can install RabbitMQ, for example, using the installer from the official website.

Next, let's enable the WebUI:

cd 'C:\Program Files\RabbitMQ Server\rabbitmq_server-3.13.0\sbin'
./rabbitmq-plugins.bat enable rabbitmq_management

Now, you can visit localhost:15672 and access the RabbitMQ web UI using the default username and password guest:guest.

Docker

Installing via Docker is probably the simplest and most convenient way to install RabbitMQ.

We can install RabbitMQ using the following docker run command:

docker run --rm -p 15672:15672 rabbitmq:3.13.7-management

The admin interface will be available on port 15672, where you can log in with the default username and password guest:guest.

However, this method of running RabbitMQ is not suitable for a production environment. Docker assigns a server name to the container, and the message broker stores its state in a folder with that name. With every new container build, RabbitMQ will lose its state information.

We can conveniently define all necessary service parameters with docker-compose, such as changing the default login and password, mounting a folder for state persistence, etc. Here's an example of what the docker-compose.yml file might look like (this can be created on your server where Docker is installed or on your local machine).

version: "3.3"
services:
  rabbit:
    image: rabbitmq:3.13.7-management
    environment:
      - RABBITMQ_DEFAULT_USER=admin #enter your username
      - RABBITMQ_DEFAULT_PASS=password #enter your password
    volumes:
      - ./rabbit:/var/lib/rabbitmq # mounting folder for state persistence
    ports:
      - 15672:15672 # expose port for the admin interface

Run:

docker compose up -d

The RabbitMQ web UI is now available.

Overview of RabbitMQ Management

Let's take a look at the main features of the RabbitMQ management interface. On the main page, you can view general information such as nodes, their status, total message count, and more.

Connections is the list of connections to the cluster.

Channels is the list of channels. Multiple channels can be created within a single connection through which messages are sent. This is done to avoid creating multiple TCP connections.

Exchanges

Let’s look more closely at Exchanges.

There are different types of exchanges, each with its own message filtering mechanism. Based on these rules, events are routed to specific queues.

Direct Exchange

Each message has a key, called the Routing Key. These messages will be routed to queues where the binding to the Exchange specifies the same key.

Topic Exchange

Routing is done based on a pattern key. When creating the pattern, you can use 0 or more words (Latin letters in different cases and numbers) separated by dots (e.g., "key.event"), and the symbols # and *.

Fanout Exchange

No filtering rules are applied. Every message sent to a Fanout Exchange is routed to all queues.

Headers Exchange

Uses message headers and binding headers, comparing key-value pairs in those headers.

Queues

When viewing a specific queue, you can see a graph showing how many messages are in the queue, statistics about delivery times, and message acceptance times.

Messages can have two statuses:

Ready – waiting to be processed.
Unacked – being processed by the consumer.

Implementing Publisher-Subscriber in Python

Let’s go through how to use RabbitMQ to implement the Pub-Sub pattern in Python. You should have Python installed on your system. This tutorial uses version 3.11.5 of Python. We will be using the Pika library.

Create or select a folder where the application code will reside. For example:

mkdir rabbitmq-article

Open this folder in your IDE. It can be Visual Studio Code, PyCharm, or other development environments.
Install the Pika library by running the following command in the terminal:

pip install pika

Now, let's create two files: sender.py and receiver.py.

sender.py code

from pika import BlockingConnection, ConnectionParameters
from pika.exchange_type import ExchangeType

# Create connection, specifying parameters in the ConnectionParameters object
connection = BlockingConnection(ConnectionParameters(host='localhost'))

# Create a channel
channel = connection.channel()

# Declare an exchange
channel.exchange_declare('new_exchange', ExchangeType.direct)

# Declare a queue
queue = channel.queue_declare(queue='new_queue')

# Bind the queue to the exchange
channel.queue_bind(exchange='new_exchange', queue='new_queue', routing_key='key')

# Publish a message
channel.basic_publish(exchange='new_exchange', routing_key='key', body='Hello World!')

print("Message 'Hello World!' sent")
connection.close()

In the above sender.py code:

We establish a connection to RabbitMQ on localhost.
We declare a direct exchange called new_exchange.
We declare a queue called new_queue.
We bind the queue to the exchange using a routing key key.
Finally, we publish the message 'Hello World!' to the exchange with the specified routing key.

Now, let's implement the receiver.py to consume the message.

receiver.py code

from pika import BlockingConnection, ConnectionParameters
from pika.exchange_type import ExchangeType
import sys, os

def main():
    # Create connection
    connection = BlockingConnection(ConnectionParameters(host='localhost'))
    # Create channel
    channel = connection.channel()

    # Declare an exchange
    channel.exchange_declare('new_exchange', ExchangeType.direct)

    # Declare a queue
    queue = channel.queue_declare(queue='new_queue')

    # Bind the queue to the exchange
    channel.queue_bind(exchange='new_exchange', queue='new_queue', routing_key='key')

    # Function to handle incoming messages
    def handle(ch, method, properties, body):
        print(f"Received message: {body.decode()}")

    # Bind the callback function and queue
    channel.basic_consume(queue='new_queue', on_message_callback=handle, auto_ack=True)

    print('Waiting for messages. Press Ctrl+C to exit.')
    channel.start_consuming()

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        try:
            sys.exit(0)
        except SystemExit:
            os._exit(0)

In the above receiver.py code:

We establish a connection and channel to RabbitMQ, just as we did in the sender.
We declare the direct exchange (new_exchange) and queue (new_queue).
The queue is bound to the exchange using the routing key key.
The handle function processes the message by printing it when received.
The basic_consume method subscribes the callback function to the queue.
We use start_consuming to keep the process waiting for messages indefinitely.

Running the Code

Start the receiver first. Open a terminal and run receiver.py:

python receiver.py

You should see the message:

Waiting for messages. Press Ctrl+C to exit.

Now, run the sender. Open a second terminal and run sender.py:

python sender.py

You should see the message:

Message 'Hello World!' sent

In the terminal where the receiver is running, you will see:

Received message: Hello World!

In RabbitMQ Management, you can go to the created exchange (new_exchange) and see that it is bound to the queue (new_queue) using the routing key (key).

Conclusion

In this article, we explored what message brokers are, how applications interact with them, and how to install and use RabbitMQ as a message broker. We successfully implemented a Publisher-Subscriber pattern using Python, RabbitMQ, and the Pika library, where the sender publishes messages to an exchange and the receiver consumes them.

Microservices RabbitMQ

08.11.2024

Reading time: 12 min