Web Scraping with Python: requests and BeautifulSoup Step by Step

Lots of useful data doesn't come in a file or through an API: catalog prices, job listings, schedules, public directories. Web scraping is the technique of extracting that information from web pages with a program. It's a valued skill in data analysis, market research and automation, and with what you already know about requests (lesson 12), you're one step away.

Before you start: the rules

A page being public doesn't mean you can do anything you want with its data. Before extracting data from a site, go through this list:

• I looked for an API or an official download first. If one exists, I use it: it's more stable and clearly allowed (lesson 12).
• I read the terms of use and the site's robots.txt file (for example, https://site.com/robots.txt), which lists the parts the site doesn't want programs to crawl.
• I won't overload the server: I pause between requests and download only what I need.
• I don't collect personal data: privacy laws in many places (like Europe's GDPR) cover people's names, emails and phone numbers even when they're posted publicly.
• I don't bypass barriers: logins, paywalls or anti-bot systems mean the site doesn't authorize automated access.

To practice, we'll use books.toscrape.com, a fictional bookstore built specifically for learning to scrape: 1,000 books spread over 50 pages, with explicit permission to practice. (It's a UK-style demo store, which is why prices are in pounds.)

Install and download the page

pip install requests beautifulsoup4 pandas

import requests
from bs4 import BeautifulSoup

URL = "https://books.toscrape.com/"
headers = {"User-Agent": "python-course-scraper/1.0 (educational practice)"}

resp = requests.get(URL, headers=headers, timeout=10)
resp.raise_for_status()                       # error if the response isn't 200

soup = BeautifulSoup(resp.content, "html.parser")
print(soup.title.get_text(strip=True))

All products | Books to Scrape - Sandbox

We pass resp.content (the raw bytes) rather than resp.text so BeautifulSoup can detect the page's encoding itself. This site doesn't declare its encoding in the response headers, so with resp.text the £ sign shows up as "£".

Finding the data: inspecting the HTML

In your browser, right-click a book title and choose Inspect. You'll see that each book has this structure (simplified):

<article class="product_pod">
  <p class="star-rating Three"></p>
  <h3><a href="catalogue/a-light-in-the-attic_1000/index.html"
         title="A Light in the Attic">A Light in the ...</a></h3>
  <div class="product_price">
    <p class="price_color">£51.77</p>
    <p class="instock availability">In stock</p>
  </div>
</article>

With that structure you write CSS selectors, the same ones web designers use to style pages:

In BeautifulSoup, soup.select(selector) returns a list of every matching element, and soup.select_one(selector) returns only the first.

books = soup.select("article.product_pod")
print(len(books))

first = books[0]
print(first.h3.a["title"])                                  # the title attribute
print(first.select_one("p.price_color").get_text(strip=True))

20
A Light in the Attic
£51.77

A function that extracts every book on a page

from urllib.parse import urljoin

STARS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}

def extract_books(soup, page_url):
    books = []
    for art in soup.select("article.product_pod"):
        link = art.h3.a
        price_txt = art.select_one("p.price_color").get_text(strip=True)
        classes = art.select_one("p.star-rating")["class"]     # ["star-rating", "Three"]
        books.append({
            "title": link["title"],
            "price": float(price_txt.replace("£", "")),
            "stars": STARS.get(classes[1], 0),
            "in_stock": "In stock" in art.select_one("p.availability").get_text(),
            "url": urljoin(page_url, link["href"]),
        })
    return books

Two professional touches: we convert the price to a number (so we can calculate with it) and use urljoin to turn relative links (catalogue/…) into full addresses.

Crawling several pages responsibly

import time
import pandas as pd

def scrape(start_url, max_pages=3, delay=1.0):
    all_books, url, page = [], start_url, 0
    with requests.Session() as session:
        session.headers.update(headers)
        while url and page < max_pages:
            resp = session.get(url, timeout=10)
            resp.raise_for_status()
            soup = BeautifulSoup(resp.content, "html.parser")
            all_books.extend(extract_books(soup, url))
            page += 1
            next_link = soup.select_one("li.next a")
            url = urljoin(url, next_link["href"]) if next_link else None
            time.sleep(delay)                          # don't overload the server
    return all_books

books = scrape("https://books.toscrape.com/", max_pages=3)
df = pd.DataFrame(books)
print(len(df), "books")
df.to_csv("books.csv", index=False, encoding="utf-8")

60 books

• requests.Session() reuses the connection: faster for you and kinder to the server.
• max_pages keeps a pagination bug from leaving you downloading forever.
• time.sleep(delay) waits between pages. A one- or two-second pause is basic courtesy.

When the page is built with JavaScript

Some sites send an almost empty page and then load the data with JavaScript. If you run requests.get and can't find the data you see in your browser, that's why. Your options, in order of preference:

Real case: auditing your own store's online catalog

Here's a use of scraping with no ethical gray areas: checking your own online store. Summit Coffee Roasters in Denver, Colorado, wants to catch out-of-stock products and products published without a price every week. With requests you'd download its catalog page; here we use a snippet of that HTML so the example works offline:

from bs4 import BeautifulSoup

# Snippet of your own store's catalog page: Summit Coffee Roasters (Denver, CO)
html = """
<div class="product"><h2>Colombia Huila 12 oz</h2><span class="price">$16.50</span><span class="stock">12</span></div>
<div class="product"><h2>Ethiopia Yirgacheffe 2 lb</h2><span class="price">$54.50</span><span class="stock">0</span></div>
<div class="product"><h2>French press 34 oz</h2><span class="stock">5</span></div>
<div class="product"><h2>Commercial grinder</h2><span class="price">$1,249.00</span><span class="stock">2</span></div>
"""
soup = BeautifulSoup(html, "html.parser")


def price_to_number(text):
    return float(text.replace("$", "").replace(",", "").strip())


products, no_price = [], []
for div in soup.select("div.product"):
    name = div.h2.get_text(strip=True)
    tag = div.select_one("span.price")
    if tag is None:                      # the selector found nothing
        no_price.append(name)
        continue
    products.append({"name": name,
                     "price": price_to_number(tag.get_text()),
                     "stock": int(div.select_one("span.stock").get_text())})

for p in sorted(products, key=lambda p: p["price"], reverse=True):
    status = "OUT OF STOCK" if p["stock"] == 0 else f"{p['stock']} left"
    print(f"{p['name']:26} ${p['price']:8,.2f}  {status}")
print("No price listed:", no_price)

Commercial grinder         $1,249.00  2 left
Ethiopia Yirgacheffe 2 lb  $   54.50  OUT OF STOCK
Colombia Huila 12 oz       $   16.50  12 left
No price listed: ['French press 34 oz']

The key is the line if tag is None. select_one() returns None when it finds nothing, and calling .get_text() on None crashes the program. This way the product without a price gets flagged for fixing on the website, and the rest of the report still runs. Notice price_to_number too: it turns "$1,249.00" into 1249.0 by stripping the dollar sign and comma, just like in lesson 3.

Common scraping errors

Without that check, you'd hit the most typical scraping error:

    price = div.select_one("span.price").get_text()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get_text'

Read it as "the None object has no get_text." The ^^^ marks underline select_one(...): that selector found nothing. Either the element is missing on that listing, or the site changed its HTML and your selector is out of date.

books = scrape("https://books.toscrape.com/", max_pages=5, delay=1.0)
df = pd.DataFrame(books)

assert len(df) == 100, f"Expected 100 books, got {len(df)}"
assert df["price"].notna().all()

picks = (
    df[(df["stars"] >= 4) & (df["price"] < 20)]
    .sort_values("price")
    [["title", "price", "stars", "url"]]
)
picks.to_csv("picks.csv", index=False, encoding="utf-8")
print(len(picks), "recommended books")
print(picks.head())

Summary

• Before scraping: look for an API, read the terms and robots.txt, don't overload the site and leave personal data alone.
• requests downloads the page, BeautifulSoup parses it and CSS selectors find the data.
• Convert text to numbers, complete links with urljoin and validate what you extract.
• Crawl pages with a limit and a delay, and save the results with pandas.
• If the data arrives through JavaScript, find the internal API in the Network tab or use Playwright.

Next lesson: Machine Learning with Python: Your First Model with scikit-learn. In the course's final lesson you'll train a model to predict sales and learn to judge whether its predictions are any good.