Working with HTTP Requests in Python

The entire web operates using the HTTP protocol. To open this page, you sent an HTTP request via your browser to the server hosting Hostman tutorials. The server analyzed your request and responded by returning this post so you could learn about the requests library in Python.

The requests library allows you to send HTTP requests. With it, you can write requests in Python to solve a wide range of tasks.

What You Need to Know to Work with requests

Sending HTTP requests and using the responses isn’t rocket science. However, some basic knowledge is still required. At the very least, you need to know the primary methods you'll be sending to servers:

• GET — retrieve a resource.

• POST — create a resource.

• PUT — update a resource.

• DELETE — delete a resource.

These are not all the methods. There are also HEAD, PATCH, and others. But for our practice, we’ll only need GET, the most common method. Users constantly request servers to give them something: pages, posts, images. Processing HTTP requests in Python with other methods works similarly.

In response to requests, you’ll receive status codes. These range from 1xx to 5xx. Every internet user encounters them, though not everyone knows what they are. Error 404 is status code 404. It indicates a client-side error: the user is trying to access a page that doesn’t exist.

The first digit of a status code indicates the type of response:

• 1xx — informational codes.

• 2xx — success.

• 3xx — redirection.

• 4xx — client-side error.

• 5xx — server-side error.

These basic concepts are enough to make a few interesting requests and correctly interpret the responses. Now, let’s see how all this behind-the-scenes web magic is handled with the requests library.

Installing the Library

To work with the library, you need Python installed on your computer. You’ll also need the package manager pip to install requests.

To install Python requests using pip:

python -m pip install requests

The library’s code is hosted on GitHub. You can clone it to your computer:

git clone git://github.com/psf/requests.git

For more details, check the library’s documentation.

Usage Examples

Now, for the most exciting part — practice. 

Let's start with a simple request. We'll give the Python requests library a task to connect to a page containing information about Hostman cloud servers.

Create a file called script.py. Inside it, write the following script:

import requests
result = requests.get('https://hostman.con/cloud-servers')
print(result)

The first line imports the requests library. Then, you define a variable result and use the GET method to get a response from the server, specifying the URI you're interested in. The last line prints the response in the terminal.

To run this short program, open any terminal, navigate to the folder where the script.py file is located, and execute it with the command: 

python script.py

The response should return status code 200, meaning the requested page is available. You can continue working with it.

As an example, let's try using a non-existent URI. You should get a 404 error code.

Change the page address in your script. You can make up your own invalid link or just alter any working one:

import requests
result = requests.get('https://hostman.com/badlink')
print(result)

The response will return a 404 status code.

From Python’s perspective, every response is an object. This object has its own methods and functions that help retrieve additional information or normalize data.

Let's revert to the correct page address and try to get its content. To do this, simply add the text method to the print command. Here’s what the script should look like after editing:

import requests
result = requests.get('https://hostman.com/cloud-servers')
print(result.text)

Running the script will display the entire HTML code of the requested page in the terminal. You can see the markup, linked style sheets, scripts, and content. This is public information, which can also be viewed in a browser using the "View Page Source" option.

Instead of the text method, you can use the url method, which will display the URL you requested. You can also use the status_code method to print the request's status code.

This was just the beginning of working with requests in Python. Now, let’s look at some more interesting examples.

Basic Authentication

Authentication helps an application understand who you are and check if there’s any information about you in the database. Typically, you provide your credentials to the server by passing them through the Authorization header.

One example of an API that requires user authentication is the GitHub Authenticated User API. This service provides information about the profile of an authenticated user. To make a request to the Authenticated User API, you must pass your GitHub username and password as a tuple, using the get() method:

from getpass import getpass
requests.get('https://api.github.com/user', auth=('username', getpass()))

If you passed valid credentials in the auth tuple, you’ll receive a success message in response:

<Response [200]>

If you try to make the same request without valid credentials, you’ll get a 401 Unauthorized status code:

>>> requests.get('https://api.github.com/user')
<Response [401]>

When you pass the username and password in a tuple, the credentials are applied using the basic access authentication scheme. You can make the same request using HTTPBasicAuth:

from requests.auth import HTTPBasicAuth
from getpass import getpass
requests.get(
   'https://api.github.com/user',
   auth=HTTPBasicAuth('username', getpass())
)
<Response [200]>

The requests library also provides other authentication methods, such as HTTPDigestAuth and HTTPProxyAuth. You can also implement your own method. However, keep in mind that poorly designed authentication mechanisms can introduce security vulnerabilities. Unless there's a strong need to create your own solution, using a proven authentication scheme like Basic or OAuth is better.

SSL Verification

Working with requests in Python also allows you to solve another important task — checking if a website has an SSL certificate.

When handling sensitive data, security is crucial. You can protect it during transmission over HTTP by setting up an encrypted connection using an SSL certificate.

To ensure the safety of a site, you need to verify the SSL certificate of the target server. The good news is that the requests library does this by default. However, in some cases, you might want to change this behavior.

If you want to disable SSL certificate verification, set the verify parameter to False:

requests.get('https://api.github.com', verify=False)

This will return a warning:

InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
<Response [200]>

requests even warns you when making a request without SSL verification to help keep your data secure.

Fetching the Weather Forecast

Finally, let’s look at a small project example. We’ll create a utility that fetches the weather forecast and displays the result in the terminal. The data source will be OpenWeather, where you can create a free API key for access.

Create a file called weather.py and write the following:

import requests
result = requests.get('https://api.openweathermap.org/data/2.5/weather?q=Warsaw,pl&appid=someidappkey0ff7889240be2&units=metric')
weather = result.text
print(weather)

First, you import the requests library. Then, you make an API request to the weather service. After the q parameter, you specify the city and country; appid is the key that you need to create on OpenWeather. The units=metric parameter indicates that the metric system should be used, so the temperature will be displayed in Celsius.

By default, the response comes in JSON format, but you can choose XML or HTML by adding the mode parameter to the request with the corresponding value.

Here’s an example of a response in JSON format:

{
   "coord": {"lon": 21.0118, "lat": 52.2298},
   "weather": [
       {
           "id": 800,
           "main": "Clear",
           "description": "clear sky",
           "icon": "01d"
       }
   ],
   "base": "stations",
   "main": {
       "temp": 15.35,
       "feels_like": 14.28,
       "temp_min": 14.78,
       "temp_max": 16.12,
       "pressure": 1021,
       "humidity": 63
   },
   "visibility": 10000,
   "wind": {
       "speed": 3.09,
       "deg": 90
   },
   "clouds": {
       "all": 0
   },
   "dt": 1697721600,
   "sys": {
       "type": 2,
       "id": 2038500,
       "country": "PL",
       "sunrise": 1697689772,
       "sunset": 1697726778
   },
   "timezone": 7200,
   "id": 756135,
   "name": "Warsaw",
   "cod": 200
}

Even without additional data processing, it’s easy to see that at the time of checking, the temperature in Warsaw was slightly above 7 degrees Celsius, with light wind. For example, you can use this information to create a web interface for a more user-friendly display.

Conclusion

You’ve learned the basics of the requests library in Python. Now you can make requests using HTTP methods, perform user authentication, check data, and retrieve information from various sources.

On our app platform you can find Python applications, such as Celery, Django, FastAPI and Flask.