RDP Protocol: What It Is, What It's Used For, and How It Works

Remote desktop connection to a host can be achieved in various ways, such as using third-party applications like UltraVNC. However, the most common tool for using the Remote Desktop Protocol (RDP) is Microsoft's built-in utility called Remote Desktop. In this article, we'll explore the history of RDP, how it works, and how data is protected during transmission.

History of RDP

The Remote Desktop Protocol (RDP) was created by Microsoft and initially developed as one of the primary methods for remote access to computers or servers. It was also intended to enable weak local machines to connect to more powerful servers for tasks like complex calculations. Today, RDP is primarily a tool for setting up remote workplaces.

Key milestones in the development of RDP:

• 1998: RDP was first used in Windows NT 4.0 Terminal Server Edition.

• It continues to be supported in all Windows operating systems, up to Windows 11 and Windows Server 2022.

• RDP remains a default remote access tool in Windows, despite the existence of other technologies like VNC.

• There are clients for nearly all major operating systems, including Linux, FreeBSD, macOS, iOS, Android, and more.

• The current version is RDP 10, which includes features like auto-resizing and enhanced graphics compression using the H.264/AVC codec.

General Characteristics of RDP

• Supported color depths: 32-bit and lower (8, 15, 16, and 24-bit).

• Data protection: 128-bit encryption using the RC4 algorithm.

• Other features: Audio redirection, printer and port forwarding, clipboard sharing between the local and remote computers.

Citrix Systems played a key role in the early development of remote desktop technology. In the early 1990s, they developed the WinFrame system based on Windows NT 3.51. Citrix eventually collaborated with Microsoft, and in 1997, Microsoft acquired certain rights to Citrix's technology. As a result, Citrix retained rights to the ICA protocol, while Microsoft developed RDP based on the ITU T.120 standard.

Citrix and Microsoft remain competitors, with Citrix focusing on high-performance systems and Microsoft leading in the mid-range server market with Terminal Services. Both companies continue to expand their product capabilities to reach more users.

Advantages of Terminal Services:

• Easy application setup for clients.

• Centralized session management.

• Functionality tied to active Terminal Services licenses.

Network Setup Using Terminal Services

Microsoft products allow the use of the RDP protocol in two different modes: for connecting to application servers (Terminal Server Mode) or for managing system settings (Remote Administration Mode). Let's look at both options:

Remote Administration Mode

All versions of Microsoft Windows support this mode. The only difference is the number of remote connections allowed simultaneously. In desktop versions, only one connection is supported: a local login or a remote host connection. In server versions, the options are broader: two network connections and one local client.

Terminal Server Mode

This mode is only available on Windows Server editions. The advantage of this mode is that there are no limits on the number of connections—you can have as many as needed. However, you must purchase licenses and configure the system properly to use this mode. The server can be set up as either a dedicated node or a terminal server. Without these steps, access cannot be granted.

How RDP Works

The Remote Desktop Protocol is an application-layer TCP protocol. Initially, the computers establish a connection. After confirming the connection, the system initiates the RDP session at the transport layer. Once this process is successfully completed, the terminal server begins transmitting the desktop image and allows input from the keyboard and mouse. The display can either be a rendered image or graphical primitives.

The system supports the simultaneous use of multiple virtual channels within a single physical connection. This is necessary to enable the following functions:

• Printing to a printer or data exchange via a serial port.

• Clipboard functionality and other operations with the disk subsystem.

• Use of audio playback and recording systems.

The parameters for the virtual channels are set at the start, during the initial connection with the host. Two security options are available when initiating a connection: integrated Standard RDP Security and optional Enhanced RDP Security. Let's look at their functionality in more detail and the features of each approach.

Standard RDP Security

This security approach involves authentication, data encryption, and integrity monitoring through modules integrated into RDP. The encryption uses the RC4 algorithm with a key length between 40-168 bits (depending on the Windows version). When establishing a connection, the system generates a pair of keys to encrypt the information exchanged between the client and the server.

The authentication process works as follows:

1. The system generates a pair of RSA keys.

2. It then creates a public key certificate (Proprietary Certificate).

3. This certificate is signed with the RSA key embedded in Windows.

4. The client receives the Proprietary Certificate to access the terminal server.

5. After verifying the certificate, the server's public key is sent to the client.

Providing the account and password triggers these actions only upon successful verification. All transmitted data undergoes integrity checks using a Message Authentication Code (MAC) algorithm, based on MD5 and SHA1. The system can be switched to use 3DES encryption, which was introduced in Windows 2003 Server to meet FIPS standards.

Enhanced RDP Security

The second security option involves using external modules like TLS 1.0 and CredSSP. The TLS protocol was introduced in Windows Server starting with the 2003 release and is used when the client machine supports RDP. Before connecting, it's recommended to check the utility version (RDP 6.0 or higher is required). Users can choose between generating their own certificate or using an existing one from the Windows system.

CredSSP (Credential Security Support Provider) is a combined solution that integrates TLS, Kerberos, and NTLM technologies. The advantages of this approach include:

• Login authorization is verified before completing the RDP connection, saving server resources when dealing with a high number of requests.

• Encryption and account identification are handled via TLS standards.

• Single sign-on to the host is supported using Kerberos or NTLM.

CredSSP can be selected by enabling the "Use Network Level Authentication" option, available in all operating systems starting with Windows Vista/2008 Server.

Terminal Services Licensing 

A popular method for accessing Terminal Services is through a "thin client." Its main feature is creating and activating a license server, without which the system will not function (although connections through a "thick client" or to a standard desktop are possible, but only in a remote mode).

Licenses are issued in two modes. Without activation, the user is granted a temporary license for the duration of the current session or for a limited period. Once activated, the user is issued a "permanent" license on the server. This process involves issuing a digital certificate confirming the connected workstation's legitimacy.

Types of Client Licenses:

• Temporary Terminal Server CAL: A temporary license limited by a specified period.

• Device Terminal Server CAL: Tied to a specific device.

• User Terminal Server CAL: Linked to the specific user defined in the settings.

• External Terminal Server Connector: Designed for external connections.

Let's look at these options in more detail.

Time-Based License

Upon their first connection to the terminal server, a time-based license is issued to the user. Its validity period is 90 days, regardless of the client's activity. If a second connection is successfully established, the server will attempt to assign a permanent license to the client machine. Licenses must be available in the server's storage for this to happen.

Device-Based License

This option is designed for a client working from a specific physical device. The license is valid for a period ranging from 52 to 89 days. If 7 or fewer days remain before expiration, the server will attempt to replace it with a permanent license upon any connection attempt. If the device is changed, the cycle will restart from the beginning.

User-Based License

The user-based license is the most flexible option, as it allows users to work from any device, creating convenient conditions for deploying remote workspaces. A notable feature of this mode is that Terminal Services does not have a built-in connection counter. As a result, when additional clients connect, the number of available licenses remains unchanged.

This does not comply with Microsoft's licensing rules, but it is a known fact. If you need to issue both device and user licenses simultaneously, the server must be configured for device-based licensing. In a mixed setup, there will also be no counter for tracking different client connections, and limitations can only be enforced by linking them to specific hardware.

External User License

This is a special type of license for granting rights to external users. It does not limit the number of clients, regardless of the volume specified, because, under the EULA agreement, a dedicated server must be allocated for such users. This restricts external clients' access to corporate sessions. This type of license is not very popular due to its high cost.

RDP Connection on macOS

Microsoft has released a Remote Desktop utility for macOS, which you can download from the App Store. The connection process is similar to that on Windows: first, you enter the server's IP address or domain, and then, when prompted, enter the username and password of the user who has remote access rights.

After launching, you may receive a warning about an untrusted certificate. To resolve this, simply click "Show Certificate" and select the "Always trust..." option. From then on, the system will connect without asking again.

Remote Desktop Connection on Ubuntu

When connecting from Windows to a remote host running Ubuntu, the Remote Desktop utility is used, just like on Windows. However, you need to prepare the Ubuntu server by installing support for the technology:

sudo apt install xrdp

After installation, start the service:

sudo systemctl status rdp

If you're using Linux as the local machine, the Remmina utility will be useful. You will need to switch the protocol from VNC to RDP. Despite the different systems, the user experience will feel seamless. Other Linux clients include Gnome Connection, Vinagre, Xfreerdp, and Rdesktop.

To install the Remmina package:

sudo apt-add-repository ppa:remmina-ppa-team/remmina-next
sudo apt-get update
sudo apt-get install remmina remmina-plugin-rdp libfreerdp-plugins-standard

Conclusion

We've completed our overview of the RDP protocol, covering what it is and how it works. One final note is that a license server can be assigned one of two roles:

1. Domain or Workgroup License Server – operates within a domain or workgroup.

2. Entire Enterprise License Server – operates within an organization.

The choice affects how the license server is scanned. In the first case, the system searches through Active Directory, while in the second, it uses NetBIOS broadcast requests.