Every day, businesses make decisions based on data sitting on public web pages: prices competitors charge, products trending in a category, job listings in a niche, reviews of a new release. Copying that data by hand is soul-crushing and doesn’t scale. Web scraping with python automates the whole job — you write a script that visits pages, extracts exactly the information you want, and saves it in a structured format. In this guide, you’ll go from zero to a working scraper: fetching pages, parsing HTML, handling the messiness of the real web, and storing your results. You’ll also learn the legal and ethical rules that separate legitimate scraping from behavior that gets you blocked or worse.
Python owns this space for good reasons. Its syntax reads almost like English, which keeps scraper code maintainable. Its HTTP and HTML-parsing libraries are mature and battle-tested. And its data ecosystem means the data you scrape flows naturally into analysis — if you want to do something meaningful with scraped data afterward, our beginner’s guide to Python for data analysis is the natural next step. Let’s build.
What Web Scraping Actually Is
Web scraping is automated data extraction from websites. Your browser does something similar every time you load a page: it requests HTML from a server and renders it visually. A scraper skips the rendering — it requests the same HTML and pulls out structured data: the product names, the prices, the dates, the links.
Scraping sits on a spectrum of data-collection methods. Manual copy-paste works for ten items and fails at ten thousand. Official APIs — where websites deliberately expose their data in structured form — are the cleanest option when they exist. Scraping fills the gap: public data with no API, or APIs too limited for your needs.
What can you build with it? Price monitors that alert you when a product drops below a threshold. Datasets for analysis — sports statistics, real estate listings, academic papers. Lead lists for sales teams. Content aggregators. Researchers scrape public data for studies; journalists scrape public records for investigations. The technique is neutral; the use case determines whether it’s valuable or problematic. That distinction matters enough to deserve its own section before we write a single line of code.
The Legal and Ethical Ground Rules
Let’s be direct: scraping exists in a legal gray area that varies by jurisdiction, and this guide isn’t legal advice. But there are widely accepted principles that keep you on safe ground, and every beginner should internalize them before running a scraper against a real site.
Respect robots.txt. Most websites publish a /robots.txt file stating which parts crawlers may access. It’s a voluntary standard, not law, but honoring it is the baseline of ethical scraping. If a site’s robots.txt disallows the pages you want, that’s the site telling you no — find another source or ask for permission.
Read the Terms of Service. Many sites explicitly prohibit automated access in their ToS. Violating ToS can get your accounts banned and your IP blocked, and high-profile cases over scraping public data have led to years of litigation. The landscape is genuinely unsettled, so caution is warranted.
Don’t hammer servers. A scraper firing off hundreds of requests per second is effectively a denial-of-service attack. Rate-limit your requests — a delay of a second or two between requests is polite and almost always fast enough. Identify your scraper with a proper User-Agent string including contact info, so site operators can reach you instead of just blocking you.
Public data only, and think about privacy. Scraping publicly available product prices is fundamentally different from scraping personal data. Avoid collecting personal information unless you have a clear lawful basis and understand regulations like GDPR or CCPA that may apply. When in doubt, don’t collect it.
Prefer the API when one exists. If a site offers an official API, use it. APIs are the site’s sanctioned way of sharing data: they’re stable, documented, rate-limited fairly, and legally unambiguous. Our guide to how APIs work will get you comfortable with them. Scraping is the fallback for when no API exists — not the first resort.
Setting Up Your Toolkit
You need Python 3 installed (any recent version works) and two libraries: requests for fetching pages and beautifulsoup4 for parsing HTML. Install them with pip:
pip install requests beautifulsoup4
That’s genuinely the whole setup. The requests library handles the HTTP layer — sending GET requests, managing headers, cookies, and sessions — with an API so clean it feels like pseudocode. BeautifulSoup (imported as bs4) takes raw HTML and turns it into a navigable tree you can search by tag, class, ID, or CSS selector. Together they cover the large majority of scraping tasks.
For practice, we’ll use quotes.toscrape.com — a real website built specifically for learning web scraping. It exists so beginners can scrape without bothering anyone, which makes it the perfect training ground. Never practice on a production website without permission; use the sandbox until your technique is solid.
A quick note on editors: any code editor works, but one with good Python support — syntax highlighting, a built-in terminal, debugging — makes learning dramatically smoother.
Your First Scraper: Fetching a Page
Every scraper has the same skeleton: fetch, parse, extract, store. Let’s build it piece by piece, starting with fetching.
import requests
url = "https://quotes.toscrape.com/"
headers = {"User-Agent": "LearningScraper/1.0 (learning project; contact: you@example.com)"}
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status() # raises an error if the request failed
print(response.status_code) # 200 means success
print(len(response.text)) # how many characters of HTML we got
Run this and you’ve fetched a real web page programmatically. The headers dict identifies your scraper — a descriptive User-Agent with contact info is basic politeness. The timeout prevents your script from hanging forever on an unresponsive server. And raise_for_status() converts HTTP error codes into Python exceptions instead of letting you silently parse an error page as if it were data — a classic beginner bug.
response.text now holds the page’s raw HTML as a string. It’s ugly — thousands of characters of tags, attributes, and whitespace. Humans read the rendered page; scrapers read this. Our job next is to extract meaning from it, which is where BeautifulSoup earns its keep.
Before moving on, one habit to build now: during development, save fetched pages to disk and parse the saved copy. Re-downloading the same page every time you tweak your parsing code wastes the site’s bandwidth and slows you down. A simple pattern is to write response.text to a file once, then read that file while you iterate on extraction logic.
Parsing HTML With BeautifulSoup
BeautifulSoup converts raw HTML into a searchable tree structure. Here’s the core pattern — find elements, extract their text:
import requests
from bs4 import BeautifulSoup
url = "https://quotes.toscrape.com/"
headers = {"User-Agent": "LearningScraper/1.0 (learning project)"}
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
for quote in soup.select("div.quote"):
text = quote.select_one("span.text").get_text(strip=True)
author = quote.select_one("small.author").get_text(strip=True)
print(f"{text} — {author}")
Let’s unpack it. BeautifulSoup(response.text, "html.parser") builds the parse tree using Python’s built-in HTML parser — no extra dependencies needed. The select method accepts CSS selectors, the same syntax used in stylesheets: div.quote means “div elements with class quote.” select_one returns the first match. get_text(strip=True) extracts an element’s text with surrounding whitespace removed.
Run this script and you’ll see every quote and author on the page printed cleanly. That’s a complete, working scraper in under twenty lines. The pattern generalizes: inspect the target page in your browser’s developer tools (right-click, Inspect), identify the HTML structure around the data you want, and write selectors to match it.
For more complex text extraction — pulling prices out of messy strings, cleaning whitespace, matching patterns — regular expressions are the traditional power tool. Our practical guide to regular expressions covers them thoroughly; you won’t need them on day one, but they become invaluable as your targets get messier.
Handling Real-World Messiness
The sandbox is tidy. The real web is not. Here’s what breaks scrapers in practice and how to handle each problem.
Pagination. Most listing sites spread results across pages — /page/2, ?page=3, or “Load more” buttons. For URL-based pagination, loop over page numbers and fetch each URL in turn, stopping when a page returns no results or a 404. For “Load more” buttons that fetch data via JavaScript, check your browser’s network tab — the button usually calls a JSON API endpoint you can request directly, which is cleaner than scraping HTML at all.
Rate limiting and politeness. Add delays between requests with time.sleep(1). If you need to scrape thousands of pages, consider running during off-peak hours for the target site. Getting your IP temporarily blocked is the web’s way of telling you to slow down — respect it rather than rotating IPs to evade it.
Headers and sessions. Some sites serve different content (or block requests entirely) based on headers. Using requests.Session() persists cookies across requests, which matters for sites requiring login flows or CSRF tokens. Set a realistic User-Agent, accept-language headers if content varies by locale, and handle redirects — requests follows them by default, but verify you land where you expect.
JavaScript-rendered content. requests fetches raw HTML; it doesn’t execute JavaScript. If the data you need is rendered client-side by a framework like React, the initial HTML won’t contain it. Your options, in order of preference: find the underlying JSON API in the network tab (fastest, most reliable), use a headless browser like Playwright or Selenium (heaviest, but handles anything), or check whether the site offers server-side rendering for crawlers. Start with the API hunt — it succeeds more often than beginners expect.
Changing layouts. Websites redesign, and every redesign breaks selectors. Prefer IDs and semantic classes, avoid deep positional chains, and add validation — if your scraper suddenly extracts zero items from a page that should have dozens, that’s a layout change, not an empty page. Log anomalies rather than silently collecting garbage.
CAPTCHAs and bot protection. If a site serves you CAPTCHAs or bot-detection challenges, that’s an unambiguous signal: the operator doesn’t want automated access. Do not attempt to bypass bot protection — it violates most sites’ terms, may violate anti-circumvention laws depending on jurisdiction, and escalates a technical disagreement into a legal one. Find another data source.
Storing What You Collect
Extracting data is half the job; storing it usefully is the other half. Choose your format by scale.
CSV is the simplest option and perfect for tabular data — one row per item, one column per field. Python’s built-in csv module handles it:
import csv
with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["text", "author"])
writer.writeheader()
writer.writerow({"text": "The quote…", "author": "The author"})
CSV opens directly in Excel and Google Sheets, which makes it ideal for sharing results with non-programmers. Its limits: no nested data, and encoding issues bite beginners regularly — always specify encoding="utf-8".
JSON handles nested structures naturally — quotes with lists of tags, products with variant arrays. Python’s json module serializes dicts and lists directly. JSON is the right choice when your scraped items have variable or hierarchical shape, and it’s the native format for feeding data into web applications and APIs.
SQLite enters the picture when your dataset grows or needs querying. It’s a full SQL database in a single file — no server to install — and Python includes support in the standard library via sqlite3. Store each scrape run with timestamps, query across runs to detect changes (price drops, new listings), and index the columns you filter on. If SQL is new to you, our SQL basics guide will get you querying confidently. For serious ongoing collection — daily scrapes feeding a dashboard — SQLite or a proper database beats flat files decisively.
Whatever you choose, store the raw fetched HTML (or at least the extracted fields plus source URLs and timestamps) for a while. When you discover a parsing bug three days later, having the original pages means re-parsing, not re-scraping.
When to Use an API Instead
Scraping is a fallback, not a first choice. Before writing a scraper, spend fifteen minutes checking for an official API — it changes the entire equation.
APIs give you structured data designed for programmatic access, with documented fields, versioning, and clear rate limits. No selector maintenance, no layout breakage, no legal ambiguity. Many sites that aggressively block scrapers welcome API users. Some are free with generous limits; others charge, and the price is often worth it compared to maintaining a scraper.
The decision framework is simple. Use the API when one exists and covers your needs — even a paid one, if the data matters. Scrape when no API exists, when the API omits the fields you need, or when API pricing is genuinely prohibitive for your use case. And sometimes the answer is hybrid: use the API for the bulk of your data and scrape the few fields the API doesn’t expose. If APIs are new territory, our beginner’s guide to how APIs work covers requests, authentication, and JSON handling — skills that transfer directly to API-based data collection.
Key Takeaways
- Web scraping with python means automating data extraction from web pages: fetch HTML with
requests, parse it with BeautifulSoup, extract fields with CSS selectors, and store the results. - Python dominates scraping because of its readability, mature libraries, and smooth handoff to data analysis — a complete working scraper fits in under twenty lines.
- Ethics and legality come first: respect robots.txt and Terms of Service, rate-limit your requests, identify your scraper, collect only public data, and never bypass bot protection.
- Real-world scraping means handling pagination, JavaScript-rendered content, changing layouts, and sessions — write resilient selectors and validate your output.
- Store results as CSV for simple tabular data, JSON for nested structures, or SQLite when you need to query across scrape runs.
- Prefer official APIs whenever they exist — scraping is the fallback for data with no sanctioned access path, not the default.
Frequently Asked Questions
Is web scraping legal?
It depends on what you scrape, where you are, and how you do it. Scraping publicly available data for personal or research use is generally tolerated, but violating Terms of Service can lead to bans and civil action, scraping personal data triggers privacy regulations like GDPR, and bypassing technical protections can violate anti-circumvention laws. Treat scraping as legally gray and consult a lawyer for anything commercial or large-scale.
Do I need to know HTML before learning web scraping?
Basic HTML is essential — you can’t write selectors for elements you can’t identify. Fortunately, you only need the fundamentals: tags, attributes, classes, IDs, and how pages nest elements. A weekend of HTML basics plus your browser’s Inspect tool is enough to start. You’ll learn the rest (forms, tables, semantic markup) as you encounter them in real pages. CSS selector syntax comes along for free since BeautifulSoup’s select uses it.
What’s the difference between BeautifulSoup and Scrapy?
BeautifulSoup is a parsing library — it extracts data from HTML you’ve already fetched, and you handle fetching, pagination, and storage yourself. Scrapy is a full scraping framework: it manages request scheduling, concurrency, pagination, middleware, and export pipelines out of the box. BeautifulSoup (with requests) is ideal for learning and for small-to-medium jobs. Scrapy pays off when you’re crawling thousands of pages across many sites and need production-grade infrastructure. Learn BeautifulSoup first; you’ll know when you’ve outgrown it.
Why is my scraper getting blocked?
The common causes, in order: no or suspicious User-Agent header, too many requests too fast, missing cookies or session state the site expects, and IP-based rate limiting after aggressive crawling. Fix them in order: set a proper User-Agent, add delays between requests, use a session to persist cookies, and slow down further if blocks persist. If the site uses CAPTCHAs or dedicated bot protection, stop — that’s the operator declining automated access.
Can I scrape websites that require login?
Technically yes — a requests.Session() can submit login forms and persist the resulting cookies, letting your scraper access authenticated pages. But logged-in scraping deserves extra caution: you’re accessing data the site gates for a reason, the Terms of Service almost certainly restrict automation on authenticated endpoints, and account bans are the likely outcome if detected. Never scrape another person’s account or credentials. For your own accounts and data portability (exporting your own data), it’s generally reasonable — but check the ToS first.
Conclusion
Web scraping with python is one of the most practical skills a programmer can learn: the web is the world’s largest database, and scraping is a query language for the parts of it with no API. The technical core is genuinely simple — fetch with requests, parse with BeautifulSoup, extract with selectors, store the results — and you proved that to yourself with the quote scraper in this guide. Everything beyond that core is craftsmanship: polite request patterns, resilient selectors, smart storage choices, and the judgment to know when an API is the better tool.
The lasting lessons aren’t technical, though. Scrape politely, scrape only what’s public, respect the operators who publish the data you’re collecting, and stop when a site says no. Start with the practice sandbox, graduate to a real project that matters to you — a price tracker, a research dataset, a personal archive — and store what you collect well.