Extracting Stock Data Using a Web Scraping
Not all stock data is available via API in this assignment; you will use web-scraping to obtain financial data. You will be quizzed on your results.
Using beautiful soup we will extract historical share data from a web-page.
Table of Contents
- Downloading the Webpage Using Requests Library
- Parsing Webpage HTML Using BeautifulSoup
- Extracting Data and Building DataFrame
Estimated Time Needed: 30 min
In [1]:
Copied!
#!pip install pandas==1.3.3
#!pip install requests==2.26.0
!mamba install bs4==4.10.0 -y
!mamba install html5lib==1.1 -y
!pip install lxml==4.6.4
#!pip install plotly==5.3.1
#!pip install pandas==1.3.3
#!pip install requests==2.26.0
!mamba install bs4==4.10.0 -y
!mamba install html5lib==1.1 -y
!pip install lxml==4.6.4
#!pip install plotly==5.3.1
__ __ __ __
/ \ / \ / \ / \
/ \/ \/ \/ \
███████████████/ /██/ /██/ /██/ /████████████████████████
/ / \ / \ / \ / \ \____
/ / \_/ \_/ \_/ \ o \__,
/ _/ \_____/ `
|/
███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗
████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
██╔████╔██║███████║██╔████╔██║██████╔╝███████║
██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║
╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
mamba (0.15.3) supported by @QuantStack
GitHub: https://github.com/mamba-org/mamba
Twitter: https://twitter.com/QuantStack
█████████████████████████████████████████████████████████████
Looking for: ['bs4==4.10.0']
pkgs/main/linux-64 [<=> ] (00m:00s)
pkgs/main/linux-64 [=> ] (00m:00s) 395 B / ?? (2.46 KB/s)
pkgs/main/linux-64 [=> ] (00m:00s) 395 B / ?? (2.46 KB/s)
pkgs/r/noarch [> ] (--:--) Finalizing...
pkgs/main/linux-64 [=> ] (00m:00s) 395 B / ?? (2.46 KB/s)
pkgs/r/noarch [> ] (--:--) Done
pkgs/r/noarch [====================] (00m:00s) Done
pkgs/main/linux-64 [=> ] (00m:00s) 395 B / ?? (2.46 KB/s)
pkgs/main/linux-64 [=> ] (00m:00s) 395 B / ?? (2.46 KB/s)
pkgs/r/linux-64 [<=> ] (00m:00s)
pkgs/main/linux-64 [=> ] (00m:00s) 395 B / ?? (2.46 KB/s)
pkgs/r/linux-64 [=> ] (00m:00s) 624 KB / ?? (1.99 MB/s)
pkgs/main/linux-64 [<=> ] (00m:00s) 395 B / ?? (2.46 KB/s)
pkgs/r/linux-64 [=> ] (00m:00s) 624 KB / ?? (1.99 MB/s)
pkgs/main/linux-64 [<=> ] (00m:00s) 432 KB / ?? (1.37 MB/s)
pkgs/r/linux-64 [=> ] (00m:00s) 624 KB / ?? (1.99 MB/s)
pkgs/main/linux-64 [<=> ] (00m:00s) 432 KB / ?? (1.37 MB/s)
pkgs/r/linux-64 [=> ] (00m:00s) 624 KB / ?? (1.99 MB/s)
pkgs/main/noarch [<=> ] (00m:00s)
pkgs/main/linux-64 [<=> ] (00m:00s) 432 KB / ?? (1.37 MB/s)
pkgs/r/linux-64 [=> ] (00m:00s) 624 KB / ?? (1.99 MB/s)
pkgs/main/noarch [=> ] (00m:00s) 368 KB / ?? (1.17 MB/s)
pkgs/main/linux-64 [<=> ] (00m:00s) 432 KB / ?? (1.37 MB/s)
pkgs/r/linux-64 [<=> ] (00m:00s) Finalizing...
pkgs/main/noarch [=> ] (00m:00s) 368 KB / ?? (1.17 MB/s)
pkgs/main/linux-64 [<=> ] (00m:00s) 432 KB / ?? (1.37 MB/s)
pkgs/r/linux-64 [<=> ] (00m:00s) Done
pkgs/main/noarch [=> ] (00m:00s) 368 KB / ?? (1.17 MB/s)
pkgs/r/linux-64 [====================] (00m:00s) Done
pkgs/main/linux-64 [<=> ] (00m:00s) 432 KB / ?? (1.37 MB/s)
pkgs/main/noarch [=> ] (00m:00s) 368 KB / ?? (1.17 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 432 KB / ?? (1.37 MB/s)
pkgs/main/noarch [=> ] (00m:00s) 368 KB / ?? (1.17 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 888 KB / ?? (1.89 MB/s)
pkgs/main/noarch [=> ] (00m:00s) 368 KB / ?? (1.17 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 888 KB / ?? (1.89 MB/s)
pkgs/main/noarch [<=> ] (00m:00s) 368 KB / ?? (1.17 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 888 KB / ?? (1.89 MB/s)
pkgs/main/noarch [<=> ] (00m:00s) 764 KB / ?? (1.62 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 888 KB / ?? (1.89 MB/s)
pkgs/main/noarch [ <=> ] (00m:00s) Finalizing...
pkgs/main/linux-64 [ <=> ] (00m:00s) 888 KB / ?? (1.89 MB/s)
pkgs/main/noarch [ <=> ] (00m:00s) Done
pkgs/main/noarch [====================] (00m:00s) Done
pkgs/main/linux-64 [ <=> ] (00m:00s) 888 KB / ?? (1.89 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 888 KB / ?? (1.89 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 1 MB / ?? (2.15 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 1 MB / ?? (2.15 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 2 MB / ?? (2.32 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 2 MB / ?? (2.32 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 2 MB / ?? (2.40 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 2 MB / ?? (2.40 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:00s) 3 MB / ?? (2.45 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:01s) 3 MB / ?? (2.45 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:01s) 3 MB / ?? (2.50 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:01s) 3 MB / ?? (2.50 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:01s) 4 MB / ?? (2.56 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:01s) 4 MB / ?? (2.56 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:01s) 4 MB / ?? (2.59 MB/s)
pkgs/main/linux-64 [ <=> ] (00m:01s) Finalizing...
pkgs/main/linux-64 [ <=> ] (00m:01s) Done
pkgs/main/linux-64 [====================] (00m:01s) Done
Pinned packages:
- python 3.7.*
Transaction
Prefix: /home/jupyterlab/conda/envs/python
Updating specs:
- bs4==4.10.0
- ca-certificates
- certifi
- openssl
Package Version Build Channel Size
────────────────────────────────────────────────────────────────────────────────
Install:
────────────────────────────────────────────────────────────────────────────────
+ beautifulsoup4 4.10.0 pyh06a4308_0 pkgs/main/noarch 85 KB
+ bs4 4.10.0 hd3eb1b0_0 pkgs/main/noarch 10 KB
+ soupsieve 2.3.1 pyhd3eb1b0_0 pkgs/main/noarch 34 KB
Change:
────────────────────────────────────────────────────────────────────────────────
- certifi 2022.5.18.1 py37h89c1867_0 installed
+ certifi 2022.5.18.1 py37h06a4308_0 pkgs/main/linux-64 147 KB
- openssl 1.1.1o h166bdaf_0 installed
+ openssl 1.1.1o h7f8727e_0 pkgs/main/linux-64 3 MB
Summary:
Install: 3 packages
Change: 2 packages
Total download: 3 MB
────────────────────────────────────────────────────────────────────────────────
Downloading [> ] (00m:00s) 2.79 KB/s
Extracting [> ] (--:--)
Downloading [> ] (00m:00s) 2.79 KB/s
Extracting [> ] (--:--)
Downloading [==> ] (00m:00s) 967.84 KB/s
Extracting [> ] (--:--)
Finished certifi (00m:00s) 147 KB 965 KB/s
Downloading [==> ] (00m:00s) 967.84 KB/s
Extracting [> ] (--:--)
Downloading [==> ] (00m:00s) 967.84 KB/s
Extracting [> ] (--:--)
Downloading [==> ] (00m:00s) 967.84 KB/s
Extracting [> ] (--:--)
Downloading [==> ] (00m:00s) 967.84 KB/s
Extracting [========> ] (00m:00s) 1 / 5
Downloading [==> ] (00m:00s) 1.10 MB/s
Extracting [========> ] (00m:00s) 1 / 5
Finished soupsieve (00m:00s) 34 KB 212 KB/s
Downloading [==> ] (00m:00s) 1.10 MB/s
Extracting [========> ] (00m:00s) 1 / 5
Downloading [==> ] (00m:00s) 1.10 MB/s
Extracting [========> ] (00m:00s) 1 / 5
Downloading [==> ] (00m:00s) 1.10 MB/s
Extracting [========> ] (00m:00s) 1 / 5
Downloading [===> ] (00m:00s) 1.59 MB/s
Extracting [========> ] (00m:00s) 1 / 5
Finished beautifulsoup4 (00m:00s) 85 KB 520 KB/s
Downloading [===> ] (00m:00s) 1.59 MB/s
Extracting [========> ] (00m:00s) 1 / 5
Downloading [===> ] (00m:00s) 1.59 MB/s
Extracting [========> ] (00m:00s) 1 / 5
Downloading [===> ] (00m:00s) 1.59 MB/s
Extracting [================> ] (00m:00s) 2 / 5
Downloading [===> ] (00m:00s) 1.59 MB/s
Extracting [================> ] (00m:00s) 2 / 5
Downloading [========================================>] (00m:00s) 16.75 MB/s
Extracting [================> ] (00m:00s) 2 / 5
Downloading [========================================>] (00m:00s) 16.75 MB/s
Extracting [========================> ] (00m:00s) 3 / 5
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [========================> ] (00m:00s) 3 / 5
Finished bs4 (00m:00s) 10 KB 56 KB/s
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [========================> ] (00m:00s) 3 / 5
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [========================> ] (00m:00s) 3 / 5
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [========================> ] (00m:00s) 3 / 5
Finished openssl (00m:00s) 3 MB 15 MB/s
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [========================> ] (00m:00s) 3 / 5
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [========================> ] (00m:00s) 3 / 5
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [================================> ] (00m:00s) 4 / 5
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [================================> ] (00m:00s) 4 / 5
Downloading [=========================================] (00m:00s) 15.41 MB/s
Extracting [=========================================] (00m:00s) 5 / 5
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
__ __ __ __
/ \ / \ / \ / \
/ \/ \/ \/ \
███████████████/ /██/ /██/ /██/ /████████████████████████
/ / \ / \ / \ / \ \____
/ / \_/ \_/ \_/ \ o \__,
/ _/ \_____/ `
|/
███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗
████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
██╔████╔██║███████║██╔████╔██║██████╔╝███████║
██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║
╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
mamba (0.15.3) supported by @QuantStack
GitHub: https://github.com/mamba-org/mamba
Twitter: https://twitter.com/QuantStack
█████████████████████████████████████████████████████████████
Looking for: ['html5lib==1.1']
pkgs/main/linux-64 Using cache
pkgs/main/noarch Using cache
pkgs/r/linux-64 Using cache
pkgs/r/noarch Using cache
Pinned packages:
- python 3.7.*
Transaction
Prefix: /home/jupyterlab/conda/envs/python
Updating specs:
- html5lib==1.1
- ca-certificates
- certifi
- openssl
Package Version Build Channel Size
───────────────────────────────────────────────────────────────────────
Install:
───────────────────────────────────────────────────────────────────────
+ html5lib 1.1 pyhd3eb1b0_0 pkgs/main/noarch 91 KB
+ webencodings 0.5.1 py37_1 pkgs/main/linux-64 19 KB
Summary:
Install: 2 packages
Total download: 110 KB
───────────────────────────────────────────────────────────────────────
Downloading [======> ] (00m:00s) 143.57 KB/s
Extracting [> ] (--:--)
Finished webencodings (00m:00s) 19 KB 143 KB/s
Downloading [======> ] (00m:00s) 143.57 KB/s
Extracting [> ] (--:--)
Downloading [======> ] (00m:00s) 143.57 KB/s
Extracting [> ] (--:--)
Downloading [======> ] (00m:00s) 143.57 KB/s
Extracting [> ] (--:--)
Downloading [======> ] (00m:00s) 143.57 KB/s
Extracting [====================> ] (00m:00s) 1 / 2
Downloading [=============================> ] (00m:00s) 540.66 KB/s
Extracting [====================> ] (00m:00s) 1 / 2
Downloading [=============================> ] (00m:00s) 540.66 KB/s
Extracting [====================> ] (00m:00s) 1 / 2
Downloading [=========================================] (00m:00s) 732.78 KB/s
Extracting [====================> ] (00m:00s) 1 / 2
Finished html5lib (00m:00s) 91 KB 604 KB/s
Downloading [=========================================] (00m:00s) 732.78 KB/s
Extracting [====================> ] (00m:00s) 1 / 2
Downloading [=========================================] (00m:00s) 732.78 KB/s
Extracting [====================> ] (00m:00s) 1 / 2
Downloading [=========================================] (00m:00s) 732.78 KB/s
Extracting [====================> ] (00m:00s) 1 / 2
Downloading [=========================================] (00m:00s) 732.78 KB/s
Extracting [=========================================] (00m:00s) 2 / 2
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Collecting lxml==4.6.4
Downloading lxml-4.6.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 78.1 MB/s eta 0:00:00:00:0100:01
Installing collected packages: lxml
Attempting uninstall: lxml
Found existing installation: lxml 4.8.0
Uninstalling lxml-4.8.0:
Successfully uninstalled lxml-4.8.0
Successfully installed lxml-4.6.4
In [2]:
Copied!
import pandas as pd
import requests
from bs4 import BeautifulSoup
import pandas as pd
import requests
from bs4 import BeautifulSoup
Using Webscraping to Extract Stock Data Example¶
First we must use the request library to downlaod the webpage, and extract the text. We will extract Netflix stock data https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html.
In [3]:
Copied!
url = "https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html"
data = requests.get(url).text
url = "https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html"
data = requests.get(url).text
Next we must parse the text into html using beautiful_soup
In [4]:
Copied!
soup = BeautifulSoup(data, 'html5lib')
soup = BeautifulSoup(data, 'html5lib')
Now we can turn the html table into a pandas dataframe
In [5]:
Copied!
netflix_data = pd.DataFrame(columns=["Date", "Open", "High", "Low", "Close", "Volume"])
# First we isolate the body of the table which contains all the information
# Then we loop through each row and find all the column values for each row
for row in soup.find("tbody").find_all('tr'):
col = row.find_all("td")
date = col[0].text
Open = col[1].text
high = col[2].text
low = col[3].text
close = col[4].text
adj_close = col[5].text
volume = col[6].text
# Finally we append the data of each row to the table
netflix_data = netflix_data.append({"Date":date, "Open":Open, "High":high, "Low":low, "Close":close, "Adj Close":adj_close, "Volume":volume}, ignore_index=True)
netflix_data = pd.DataFrame(columns=["Date", "Open", "High", "Low", "Close", "Volume"])
# First we isolate the body of the table which contains all the information
# Then we loop through each row and find all the column values for each row
for row in soup.find("tbody").find_all('tr'):
col = row.find_all("td")
date = col[0].text
Open = col[1].text
high = col[2].text
low = col[3].text
close = col[4].text
adj_close = col[5].text
volume = col[6].text
# Finally we append the data of each row to the table
netflix_data = netflix_data.append({"Date":date, "Open":Open, "High":high, "Low":low, "Close":close, "Adj Close":adj_close, "Volume":volume}, ignore_index=True)
We can now print out the dataframe
In [6]:
Copied!
netflix_data.head()
netflix_data.head()
Out[6]:
| Date | Open | High | Low | Close | Volume | Adj Close | |
|---|---|---|---|---|---|---|---|
| 0 | Jun 01, 2021 | 504.01 | 536.13 | 482.14 | 528.21 | 78,560,600 | 528.21 |
| 1 | May 01, 2021 | 512.65 | 518.95 | 478.54 | 502.81 | 66,927,600 | 502.81 |
| 2 | Apr 01, 2021 | 529.93 | 563.56 | 499.00 | 513.47 | 111,573,300 | 513.47 |
| 3 | Mar 01, 2021 | 545.57 | 556.99 | 492.85 | 521.66 | 90,183,900 | 521.66 |
| 4 | Feb 01, 2021 | 536.79 | 566.65 | 518.28 | 538.85 | 61,902,300 | 538.85 |
We can also use the pandas read_html function using the url
In [7]:
Copied!
read_html_pandas_data = pd.read_html(url)
read_html_pandas_data = pd.read_html(url)
Or we can convert the BeautifulSoup object to a string
In [8]:
Copied!
read_html_pandas_data = pd.read_html(str(soup))
read_html_pandas_data = pd.read_html(str(soup))
Beacause there is only one table on the page, we just take the first table in the list returned
In [9]:
Copied!
netflix_dataframe = read_html_pandas_data[0]
netflix_dataframe.head()
netflix_dataframe = read_html_pandas_data[0]
netflix_dataframe.head()
Out[9]:
| Date | Open | High | Low | Close* | Adj Close** | Volume | |
|---|---|---|---|---|---|---|---|
| 0 | Jun 01, 2021 | 504.01 | 536.13 | 482.14 | 528.21 | 528.21 | 78560600 |
| 1 | May 01, 2021 | 512.65 | 518.95 | 478.54 | 502.81 | 502.81 | 66927600 |
| 2 | Apr 01, 2021 | 529.93 | 563.56 | 499.00 | 513.47 | 513.47 | 111573300 |
| 3 | Mar 01, 2021 | 545.57 | 556.99 | 492.85 | 521.66 | 521.66 | 90183900 |
| 4 | Feb 01, 2021 | 536.79 | 566.65 | 518.28 | 538.85 | 538.85 | 61902300 |
Using Webscraping to Extract Stock Data Exercise¶
Use the requests library to download the webpage https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html. Save the text of the response as a variable named html_data.
In [10]:
Copied!
url = "https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html"
html_data = requests.get(url).text
url = "https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html"
html_data = requests.get(url).text
Parse the html data using beautiful_soup.
In [11]:
Copied!
soup = BeautifulSoup(html_data, 'html5lib')
soup = BeautifulSoup(html_data, 'html5lib')
Question 1 What is the content of the title attribute:
In [12]:
Copied!
soup.title
soup.title
Out[12]:
<title>Netflix, Inc. (NFLX) Stock Historical Prices & Data - Yahoo Finance</title>
Using beautiful soup extract the table with historical share prices and store it into a dataframe named amazon_data. The dataframe should have columns Date, Open, High, Low, Close, Adj Close, and Volume. Fill in each variable with the correct data from the list col.
In [45]:
Copied!
amazon_data = pd.DataFrame(columns=["Date", "Open", "High", "Low", "Close", "Volume"])
for row in soup.find("tbody").find_all("tr"):
col = row.find_all("td")
date = col[0].text
Open = col[1].text
high = col[2].text
low = col[3].text
close = col[4].text
adj_close = col[5].text
volume = col[6].text
amazon_data = amazon_data.append({"Date":date, "Open":Open, "High":high, "Low":low, "Close":close, "Adj Close":adj_close, "Volume":volume}, ignore_index=True)
amazon_data = pd.DataFrame(columns=["Date", "Open", "High", "Low", "Close", "Volume"])
for row in soup.find("tbody").find_all("tr"):
col = row.find_all("td")
date = col[0].text
Open = col[1].text
high = col[2].text
low = col[3].text
close = col[4].text
adj_close = col[5].text
volume = col[6].text
amazon_data = amazon_data.append({"Date":date, "Open":Open, "High":high, "Low":low, "Close":close, "Adj Close":adj_close, "Volume":volume}, ignore_index=True)
Print out the first five rows of the amazon_data dataframe you created.
In [46]:
Copied!
amazon_data.head()
amazon_data.head()
Out[46]:
| Date | Open | High | Low | Close | Volume | Adj Close | |
|---|---|---|---|---|---|---|---|
| 0 | Jun 01, 2021 | 504.01 | 536.13 | 482.14 | 528.21 | 78,560,600 | 528.21 |
| 1 | May 01, 2021 | 512.65 | 518.95 | 478.54 | 502.81 | 66,927,600 | 502.81 |
| 2 | Apr 01, 2021 | 529.93 | 563.56 | 499.00 | 513.47 | 111,573,300 | 513.47 |
| 3 | Mar 01, 2021 | 545.57 | 556.99 | 492.85 | 521.66 | 90,183,900 | 521.66 |
| 4 | Feb 01, 2021 | 536.79 | 566.65 | 518.28 | 538.85 | 61,902,300 | 538.85 |
Question 2 What is the name of the columns of the dataframe
In [47]:
Copied!
"Date", "Open", "High", "Low", "Close", "Volume"
"Date", "Open", "High", "Low", "Close", "Volume"
Out[47]:
('Date', 'Open', 'High', 'Low', 'Close', 'Volume')
Question 3 What is the Open of the last row of the amazon_data dataframe?
In [54]:
Copied!
# amazon_data[[:-1],'Open']]
# amazon_data.shape
# amazon_data[-1:]
# amazon_data
# amazon_data[[-1:]]['Open'] => 109.35
amazon_data.loc[69,'Open']
# TODO : change the date index to EACH year : from recent year => oldest last
# amazon_data[[:-1],'Open']]
# amazon_data.shape
# amazon_data[-1:]
# amazon_data
# amazon_data[[-1:]]['Open'] => 109.35
amazon_data.loc[69,'Open']
# TODO : change the date index to EACH year : from recent year => oldest last
Out[54]:
'109.35'
Using pandas read_html function
In [49]:
Copied!
read_html_pandas_amazon_data = pd.read_html(url)
read_html_pandas_amazon_data = pd.read_html(url)
In [50]:
Copied!
read_html_pandas_amazon_data = pd.read_html(str(soup))
read_html_pandas_amazon_data = pd.read_html(str(soup))
In [51]:
Copied!
amazon_dataframe = read_html_pandas_amazon_data[0] # read the only table in the html file
amazon_dataframe = read_html_pandas_amazon_data[0] # read the only table in the html file
In [52]:
Copied!
amazon_dataframe.head()
amazon_dataframe.head()
Out[52]:
| Date | Open | High | Low | Close* | Adj Close** | Volume | |
|---|---|---|---|---|---|---|---|
| 0 | Jun 01, 2021 | 504.01 | 536.13 | 482.14 | 528.21 | 528.21 | 78560600 |
| 1 | May 01, 2021 | 512.65 | 518.95 | 478.54 | 502.81 | 502.81 | 66927600 |
| 2 | Apr 01, 2021 | 529.93 | 563.56 | 499.00 | 513.47 | 513.47 | 111573300 |
| 3 | Mar 01, 2021 | 545.57 | 556.99 | 492.85 | 521.66 | 521.66 | 90183900 |
| 4 | Feb 01, 2021 | 536.79 | 566.65 | 518.28 | 538.85 | 538.85 | 61902300 |
In [53]:
Copied!
# amazon_dataframe.describe()
# amazon_dataframe.head()
# amazon_dataframe
# amazon_dataframe.describe()
# amazon_dataframe.head()
# amazon_dataframe
Out[53]:
| Date | Open | High | Low | Close* | Adj Close** | Volume | |
|---|---|---|---|---|---|---|---|
| 0 | Jun 01, 2021 | 504.01 | 536.13 | 482.14 | 528.21 | 528.21 | 78560600 |
| 1 | May 01, 2021 | 512.65 | 518.95 | 478.54 | 502.81 | 502.81 | 66927600 |
| 2 | Apr 01, 2021 | 529.93 | 563.56 | 499.00 | 513.47 | 513.47 | 111573300 |
| 3 | Mar 01, 2021 | 545.57 | 556.99 | 492.85 | 521.66 | 521.66 | 90183900 |
| 4 | Feb 01, 2021 | 536.79 | 566.65 | 518.28 | 538.85 | 538.85 | 61902300 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 66 | Dec 01, 2015 | 124.47 | 133.27 | 113.85 | 114.38 | 114.38 | 319939200 |
| 67 | Nov 01, 2015 | 109.20 | 126.60 | 101.86 | 123.33 | 123.33 | 320321800 |
| 68 | Oct 01, 2015 | 102.91 | 115.83 | 96.26 | 108.38 | 108.38 | 446204400 |
| 69 | Sep 01, 2015 | 109.35 | 111.24 | 93.55 | 103.26 | 103.26 | 497401200 |
| 70 | *Close price adjusted for splits.**Adjusted cl... | *Close price adjusted for splits.**Adjusted cl... | *Close price adjusted for splits.**Adjusted cl... | *Close price adjusted for splits.**Adjusted cl... | *Close price adjusted for splits.**Adjusted cl... | *Close price adjusted for splits.**Adjusted cl... | *Close price adjusted for splits.**Adjusted cl... |
71 rows × 7 columns
About the Authors:
Joseph Santarcangelo has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.
Azim Hirjani
Change Log¶
| Date (YYYY-MM-DD) | Version | Changed By | Change Description |
|---|
| 2021-06-09 | 1.2 | Lakshmi Holla|Added URL in question 3 |
| 2020-11-10 | 1.1 | Malika Singla | Deleted the Optional part | | 2020-08-27 | 1.0 | Malika Singla | Added lab to GitLab |
© IBM Corporation 2020. All rights reserved.