HTTP and Requests

Estimated time needed: 15 minutes

Objectives¶

After completing this lab you will be able to:

• Understand HTTP
• Handle HTTP Requests

Table of Contents

• Overview of HTTP
  • Uniform Resource Locator:URL
  • Request
  • Response
• Requests in Python
  • Get Request with URL Parameters
  • Post Requests

---

Overview of HTTP

When you, the client, use a web page your browser sends an HTTP request to the server where the page is hosted. The server tries to find the desired resource by default "index.html". If your request is successful, the server will send the object to the client in an HTTP response. This includes information like the type of the resource, the length of the resource, and other information.

The figure below represents the process. The circle on the left represents the client, the circle on the right represents the Web server. The table under the Web server represents a list of resources stored in the web server. In this case an HTML file, png image, and txt file .

The HTTP protocol allows you to send and receive information through the web including webpages, images, and other web resources. In this lab, we will provide an overview of the Requests library for interacting with the HTTP protocol.

Uniform Resource Locator: URL

Uniform resource locator (URL) is the most popular way to find resources on the web. We can break the URL into three parts.

• scheme this is this protocol, for this lab it will always be http://
• Internet address or Base URL this will be used to find the location here are some examples: www.ibm.com and www.gitlab.com
• route location on the web server for example: /images/IDSNlogo.png

You may also hear the term Uniform Resource Identifier (URI), URL are actually a subset of URIs. Another popular term is endpoint, this is the URL of an operation provided by a Web server.

Request

The process can be broken into the request and response process. The request using the get method is partially illustrated below. In the start line we have the GET method, this is an HTTP method. Also the location of the resource /index.html and the HTTP version. The Request header passes additional information with an HTTP request:

When an HTTP request is made, an HTTP method is sent, this tells the server what action to perform. A list of several HTTP methods is shown below. We will go over more examples later.

Response

The figure below represents the response; the response start line contains the version number HTTP/1.0, a status code (200) meaning success, followed by a descriptive phrase (OK). The response header contains useful information. Finally, we have the response body containing the requested file, an HTML document. It should be noted that some requests have headers.

Some status code examples are shown in the table below, the prefix indicates the class. These are shown in yellow, with actual status codes shown in white. Check out the following link for more descriptions.

Requests in Python

Requests is a Python Library that allows you to send HTTP/1.1 requests easily. We can import the library as follows:

In [1]:

import requests

import requests

We will also use the following libraries:

In [2]:

import os 
from PIL import Image
from IPython.display import IFrame

import os from PIL import Image from IPython.display import IFrame

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
f:\Lab\research-notes\datascience-notes\data-engineering-notes\PY0101EN-5.3_Requests_HTTP.ipynb Cell 24' in <cell line: 2>()
      <a href='vscode-notebook-cell:/f%3A/Lab/research-notes/datascience-notes/data-engineering-notes/PY0101EN-5.3_Requests_HTTP.ipynb#ch0000023?line=0'>1</a> import os 
----> <a href='vscode-notebook-cell:/f%3A/Lab/research-notes/datascience-notes/data-engineering-notes/PY0101EN-5.3_Requests_HTTP.ipynb#ch0000023?line=1'>2</a> from PIL import Image
      <a href='vscode-notebook-cell:/f%3A/Lab/research-notes/datascience-notes/data-engineering-notes/PY0101EN-5.3_Requests_HTTP.ipynb#ch0000023?line=2'>3</a> from IPython.display import IFrame

ModuleNotFoundError: No module named 'PIL'

You can make a GET request via the method get to www.ibm.com:

In [ ]:

url='https://www.ibm.com/'
r=requests.get(url)

url='https://www.ibm.com/' r=requests.get(url)

We have the response object r, this has information about the request, like the status of the request. We can view the status code using the attribute status_code.

In [ ]:

r.status_code

r.status_code

Out[ ]:

200

You can view the request headers:

In [ ]:

print(r.request.headers)

print(r.request.headers)

{'User-Agent': 'python-requests/2.27.1', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': '_abck=16911B313CF46C73B08AF4E6F4D63671~-1~YAAQx2vcF9viUPeAAQAA0+z8CwdcNnWWDeTIxOObERZglvrnd6v8e/jmc9A3qbs0NYscXJjvib1sZ5God4VbM+lxqQEdstj8Uw9Fub728ymg2viaz1XPtFgdlZITxtLpmacsmY/AkCm4OWCWni+ANK/betCQYJ4rVtP/RZg3oWIJQTLaqEx7XXiDEvlh8HegbxvsOfTZV7P7u4DgQPrPBv5taapamFRZEJmjGj3FOg4irC/Haevu/4KmnDIrDUj8E1EHcuaiBctox8e3+DjwOd+o/V9qnY6PgcOqzDMt1YiNIAITnmHoP4itfYkTuPIRPL6HpK7t1bUNVxFSz1XtVyGTWMB8pp+q9kBanSNO3cw88TM5080=~-1~-1~-1; bm_sz=97F0E2A97ABD896767242782BF6F67CB~YAAQx2vcF9ziUPeAAQAA0+z8Cw/kyvB2hGd7VoPxa+P1tEDfKiIB/JLIXOL3mnyJjFjUKuFBnDPzOi10aj9IbuSR0WvEnEP5i00AmjsoicxZd0U0jWFxZX8LzHWT0oqFTB67NVE2fVt+bmyY8Kpuv9dFeOVUL3gstBiGh9NFIfriYIyv8yDendJ20epw+ATlt2Zk8LKxbk1woM4IHrLR+x5Ksu0DvDnkCggrnyoeGJoybQRZWt2wdeAZptlnMNzQvK5a62Rji2yDmvzN6IYvJa+6IXXZVOIzi7APsDE+CJs=~3293765~3618628'}

You can view the request body, in the following line, as there is no body for a get request we get a None:

In [ ]:

print("request body:", r.request.body)

print("request body:", r.request.body)

request body: None

You can view the HTTP response header using the attribute headers. This returns a python dictionary of HTTP response headers.

In [ ]:

header=r.headers
print(r.headers)

header=r.headers print(r.headers)

{'Cache-Control': 'max-age=301', 'Expires': 'Mon, 23 May 2022 16:20:33 GMT', 'Last-Modified': 'Mon, 23 May 2022 15:23:21 GMT', 'ETag': '"17ecd-5dfaf6ba68345"', 'Accept-Ranges': 'bytes', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'X-Akamai-Transformed': '9 20378 0 pmb=mTOE,2', 'Date': 'Sat, 28 May 2022 18:45:34 GMT', 'Content-Length': '20547', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'x-content-type-options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': 'upgrade-insecure-requests', 'Strict-Transport-Security': 'max-age=31536000'}

We can obtain the date the request was sent using the key Date

In [ ]:

header['date']

header['date']

Out[ ]:

'Sat, 28 May 2022 18:45:34 GMT'

Content-Type indicates the type of data:

In [ ]:

header['Content-Type']

header['Content-Type']

Out[ ]:

'text/html'

You can also check the encoding:

In [ ]:

 r.encoding

r.encoding

Out[ ]:

'utf-8'

As the Content-Type is text/html we can use the attribute text to display the HTML in the body. We can review the first 100 characters:

In [ ]:

r.text[0:100]

r.text[0:100]

Out[ ]:

'{\n  "args": {\n    "ID": "123", \n    "name": "Joseph"\n  }, \n  "headers": {\n    "Accept": "*/*", \n    '

You can load other types of data for non-text requests, like images. Consider the URL of the following image:

In [ ]:

# Use single quotation marks for defining string
url='https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/IDSNlogo.png'

# Use single quotation marks for defining string url='https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/IDSNlogo.png'

We can make a get request:

In [ ]:

r=requests.get(url)

r=requests.get(url)

We can look at the response header:

In [ ]:

print(r.headers)

print(r.headers)

{'Date': 'Sat, 28 May 2022 18:54:27 GMT', 'X-Clv-Request-Id': 'e1e7da64-3b4c-4d51-8137-61cc40f5bf62', 'Server': 'Cleversafe', 'X-Clv-S3-Version': '2.5', 'Accept-Ranges': 'bytes', 'x-amz-request-id': 'e1e7da64-3b4c-4d51-8137-61cc40f5bf62', 'Cache-Control': 'max-age=0,public', 'ETag': '"a831e767d02efd21b904ec485ac0c769"', 'Content-Type': 'image/png', 'Last-Modified': 'Mon, 23 May 2022 12:36:48 GMT', 'Content-Length': '21590'}

We can see the 'Content-Type'

In [ ]:

r.headers['Content-Type']

r.headers['Content-Type']

Out[ ]:

'image/png'

An image is a response object that contains the image as a bytes-like object. As a result, we must save it using a file object. First, we specify the file path and name

In [ ]:

path=os.path.join(os.getcwd(),'image.png')
path

path=os.path.join(os.getcwd(),'image.png') path

Out[ ]:

'/resources/labs/image.png'

We save the file, in order to access the body of the response we use the attribute content then save it using the open function and write method:

In [ ]:

with open(path,'wb') as f:
    f.write(r.content)

with open(path,'wb') as f: f.write(r.content)

We can view the image:

In [ ]:

Image.open(path)  

Image.open(path)

Out[ ]:

Question 1: write wget

In the previous section, we used the wget function to retrieve content from the web server as shown below. Write the python code to perform the same task. The code should be the same as the one used to download the image, but the file name should be 'Example1.txt'.

!wget -O /resources/data/Example1.txt https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/labs/Module%205/data/Example1.txt

In [ ]:

url='https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/labs/Module%205/data/Example1.txt'
path=os.path.join(os.getcwd(),'example1.txt')
r=requests.get(url)
with open(path,'wb') as f:
    f.write(r.content)

url='https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/labs/Module%205/data/Example1.txt' path=os.path.join(os.getcwd(),'example1.txt') r=requests.get(url) with open(path,'wb') as f: f.write(r.content)

Click here for the solution

url='https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/labs/Module%205/data/Example1.txt'
path=os.path.join(os.getcwd(),'example1.txt')
r=requests.get(url)
with open(path,'wb') as f:
    f.write(r.content)

Get Request with URL Parameters

You can use the GET method to modify the results of your query, for example retrieving data from an API. We send a GET request to the server. Like before we have the Base URL, in the Route we append /get, this indicates we would like to preform a GET request. This is demonstrated in the following table:

The Base URL is for http://httpbin.org/ is a simple HTTP Request & Response Service. The URL in Python is given by:

In [ ]:

url_get='http://httpbin.org/get'

url_get='http://httpbin.org/get'

A query string is a part of a uniform resource locator (URL), this sends other information to the web server. The start of the query is a ?, followed by a series of parameter and value pairs, as shown in the table below. The first parameter name is name and the value is Joseph. The second parameter name is ID and the Value is 123. Each pair, parameter, and value is separated by an equals sign, =. The series of pairs is separated by the ampersand &.

To create a Query string, add a dictionary. The keys are the parameter names and the values are the value of the Query string.

In [ ]:

payload={"name":"Joseph","ID":"123"}

payload={"name":"Joseph","ID":"123"}

Then passing the dictionary payload to the params parameter of the get() function:

In [ ]:

r=requests.get(url_get,params=payload)

r=requests.get(url_get,params=payload)

We can print out the URL and see the name and values

In [ ]:

r.url

r.url

Out[ ]:

'http://httpbin.org/get?name=Joseph&ID=123'

There is no request body

In [ ]:

print("request body:", r.request.body)

print("request body:", r.request.body)

request body: None

We can print out the status code

In [ ]:

print(r.status_code)

print(r.status_code)

200

We can view the response as text:

In [ ]:

print(r.text)

print(r.text)

{
  "args": {
    "ID": "123", 
    "name": "Joseph"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.27.1", 
    "X-Amzn-Trace-Id": "Root=1-62927030-6b47a08643eea186697fd787"
  }, 
  "origin": "150.239.85.244", 
  "url": "http://httpbin.org/get?name=Joseph&ID=123"
}

We can look at the 'Content-Type'.

In [3]:

r.headers['Content-Type']

r.headers['Content-Type']

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
f:\Lab\research-notes\datascience-notes\data-engineering-notes\PY0101EN-5.3_Requests_HTTP.ipynb Cell 82' in <cell line: 1>()
----> <a href='vscode-notebook-cell:/f%3A/Lab/research-notes/datascience-notes/data-engineering-notes/PY0101EN-5.3_Requests_HTTP.ipynb#ch0000081?line=0'>1</a> r.headers['Content-Type']

NameError: name 'r' is not defined

As the content 'Content-Type' is in the JSON format we can use the method json(), it returns a Python dict:

In [4]:

r.json()

r.json()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
f:\Lab\research-notes\datascience-notes\data-engineering-notes\PY0101EN-5.3_Requests_HTTP.ipynb Cell 84' in <cell line: 1>()
----> <a href='vscode-notebook-cell:/f%3A/Lab/research-notes/datascience-notes/data-engineering-notes/PY0101EN-5.3_Requests_HTTP.ipynb#ch0000083?line=0'>1</a> r.json()

NameError: name 'r' is not defined

The key args has the name and values:

In [5]:

r.json()['args']

r.json()['args']

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
f:\Lab\research-notes\datascience-notes\data-engineering-notes\PY0101EN-5.3_Requests_HTTP.ipynb Cell 86' in <cell line: 1>()
----> <a href='vscode-notebook-cell:/f%3A/Lab/research-notes/datascience-notes/data-engineering-notes/PY0101EN-5.3_Requests_HTTP.ipynb#ch0000085?line=0'>1</a> r.json()['args']

NameError: name 'r' is not defined

Post Requests

Like a GET request, a POST is used to send data to a server, but the POST request sends the data in a request body. In order to send the Post Request in Python, in the URL we change the route to POST:

In [6]:

url_post='http://httpbin.org/post'

url_post='http://httpbin.org/post'

This endpoint will expect data as a file or as a form. A form is convenient way to configure an HTTP request to send data to a server.

To make a POST request we use the post() function, the variable payload is passed to the parameter data :

In [39]:

r_post=requests.post(url_post,data=payload)

r_post=requests.post(url_post,data=payload)

Comparing the URL from the response object of the GET and POST request we see the POST request has no name or value pairs.

In [40]:

print("POST request URL:",r_post.url )
print("GET request URL:",r.url)

print("POST request URL:",r_post.url ) print("GET request URL:",r.url)

POST request URL: http://httpbin.org/post
GET request URL: http://httpbin.org/get?name=Joseph&ID=123

We can compare the POST and GET request body, we see only the POST request has a body:

In [41]:

print("POST request body:",r_post.request.body)
print("GET request body:",r.request.body)

print("POST request body:",r_post.request.body) print("GET request body:",r.request.body)

POST request body: name=Joseph&ID=123
GET request body: None

We can view the form as well:

In [42]:

r_post.json()['form']

r_post.json()['form']

Out[42]:

{'ID': '123', 'name': 'Joseph'}

There is a lot more you can do. Check out Requests for more.

---

Authors¶

Joseph Santarcangelo
A Data Scientist at IBM, and holds a PhD in Electrical Engineering. His research focused on using Machine Learning, Signal Processing, and Computer Vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.

Other Contributors¶

Mavis Zhou

Change Log¶

Date (YYYY-MM-DD)	Version	Changed By	Change Description
2021-12-20	2.1	Malika	Updated the links
2020-09-02	2.0	Simran	Template updates to the file

© IBM Corporation 2020. All rights reserved.

In [ ]: