Exploring Web Scraping with Beautiful Soup and Requests: A Learning Journey
Understanding web scraping techniques is essential for data science. The goal was to extract structured data from a real-world website, specifically focusing on the World Economic Forum's Wikipedia page. This hands-on experience allowed me to learn how to programmatically interact with web content and parse HTML to gather information.
The Tools: Requests and Beautiful Soup
I chose to use two powerful Python libraries for this project: requests for making HTTP requests to fetch web pages, and BeautifulSoup for parsing the HTML content.
First, I used requests to get the content of the Wikipedia page:
from bs4 import BeautifulSoup
import requests
url = 'https://en.wikipedia.org/wiki/World_Economic_Forum'
page = requests.get(url)
Once the page content was retrieved, I used BeautifulSoup to parse the HTML. This step transforms the raw HTML into a parseable object, making it easy to navigate and search for specific elements.
soup = BeautifulSoup(page.text, 'html')
print(soup) # This prints the full HTML content of the page
Navigating and Extracting Data from HTML Tables
My primary interest was to extract information presented in tables on the Wikipedia page. I started by identifying the specific table I wanted to work with, which in this case was the "Overview of past annual meetings" table. I found that I could locate the table using its class attribute.
# To find the second table on the page
print(soup.find_all('table')[1])
# To find the specific table by its class
table = soup.find('table', class_='wikitable mw-collapsible')
print(table)
After locating the table, the next step was to extract the table headers. This was crucial for understanding the structure of the data and for creating a meaningful DataFrame later.
table_title = table.find_all('th', scope="col")
print(table_title)
table_title_col = [title.text.strip() for title in table_title]
print(table_title_col)
With the column headers extracted, I initialized a pandas DataFrame to store the scraped data.
import pandas as pd
df = pd.DataFrame(columns=table_title_col)
print(df)
The most involved part was iterating through the table rows and extracting the data for each cell. I learned that I needed to handle potential inconsistencies in row lengths and properly append data to the DataFrame.
row_data = table.find_all('tr')[1:]
print(row_data)
for row in row_data[1:]:
table_data = row.find_all(['td', 'th'])
individual_table_data = [data.text.strip() for data in table_data]
# Ensure the data matches the number of columns before appending
if len(individual_table_data) == len(df.columns):
df.loc[len(df)] = individual_table_data
else:
# Pad missing values if row length is inconsistent
individual_table_data += [None] * (len(df.columns) - len(individual_table_data))
df.loc[len(df)] = individual_table_data # This line was causing duplicates, fixed it.
# Corrected logic for appending rows to avoid duplicates from original snippet:
# Resetting df to re-populate with corrected logic if this were a fresh run
df = pd.DataFrame(columns=table_title_col)
for row in row_data[1:]:
table_data = row.find_all(['td', 'th'])
individual_table_data = [data.text.strip() for data in table_data]
# Handle cases where some rows might have fewer columns than expected
if len(individual_table_data) < len(df.columns):
individual_table_data.extend([None] * (len(df.columns) - len(individual_table_data)))
df.loc[len(df)] = individual_table_data
print(df)
print(type(df[["Dates"]]))
Finally, I saved the extracted data into a CSV file for further analysis.
df.to_csv(r'/Users/etsubfeleke/Documents/practice.csv', index=False)