Day 45 of Udemy's "100 Days of Code: The Complete Python Pro Bootcamp": web scraping with Requests and BeautifulSoup. It parses a local HTML page, scrapes the Hacker News front page and writes Empire's 100 greatest movies to movies.txt. It is for fellow coursists and anyone looking for a small, tested scraping example.

🧭 Overview

Part	What it is for
`src/scrape_movies/`	The code in clean-architecture layers (`domain`, `application`, `infrastructure`, `presentation`) plus `constants.py`; the lecture steps `soup_basics`, `hacker_news` and the movies scraper follow milestone by milestone
`tests/`	pytest tests that run offline against saved pages in `tests/fixtures/`
`docs/`	Project documents (business case, plan, milestones, reviews) and the wiki in English, Danish and Bahasa Malaysia
`Doxyfile`	Generates the API pages from the Doxygen comments in the source

The only runtime dependencies are requests and beautifulsoup4.

📋 Requirements

Python 3.13 or newer (the venv module comes with it).
requests and beautifulsoup4 (installed by the setup below).
Optional: Doxygen to build the API pages.

🛠️ Setup

Create a local virtual environment named .venv, update pip, then install the project with its test tools.

Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Linux and macOS:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Leave out [dev] to install only the runtime dependencies. Personal tokens belong in a git-ignored .env; the project never reads it.

▶️ Run

Every lecture step is a command of the form python -m scrape_movies <lecture> <step>; without arguments the help is printed. Each step below shows the command, what it teaches and a screenshot of its output.

Lecture: Parsing HTML and making soup

The sample page is tests/fixtures/website.html; the code is src/scrape_movies/infrastructure/soup_basics.py.

1. Create a soup and read the title (BeautifulSoup(html, "html.parser"), soup.title.string)

python -m scrape_movies soup title

2. Prettify (soup.prettify() indents one tag per line)

python -m scrape_movies soup prettify

3. Find all tags (find_all("a"), then get_text() and get("href") in a loop)

python -m scrape_movies soup find-all

4. Find by id and by class (find(id=...), find_all("h3", class_=...); class_ because class is a Python keyword)

python -m scrape_movies soup find

5. CSS selectors (select_one("p a") and select(".heading"))

python -m scrape_movies soup select

The screenshots are made by python tools/make_screenshots.py, which renders the command output as an image; no desktop is captured.

Lecture: Scraping Hacker News

The code is src/scrape_movies/infrastructure/hacker_news.py (parsing) and http_client.py (one requests.get with a timeout and a User-Agent). By default the commands read the course's static copy of the front page, --source live reads https://news.ycombinator.com/news and --source fixture reads the saved copy in tests/fixtures/ without any network.

1. Look at the page you will scrape (live data instead of a local file)

2. Inspect the markup (browser developer tools show the tags to look for: a story row tr.athing, its title link .titleline > a, and a span.score in the next row)

python -m scrape_movies hacker-news markup

3. Titles (find_all on the title links, then get_text())

python -m scrape_movies hacker-news titles

4. Links (get("href") of the same anchors)

python -m scrape_movies hacker-news links

5. Upvotes (the text 38 points is split and converted with int() so it can be compared)

python -m scrape_movies hacker-news upvotes

6. The top story (Python's max over the upvotes)

python -m scrape_movies hacker-news top

Lecture: Is web scraping legal and ethical?

A short summary in our own words; it is not legal advice. Reading the site's robots.txt comes first:

curl https://news.ycombinator.com/robots.txt

Public, non-copyrighted data is generally fine to read. Court cases such as Genius v. Google and hiQ v. LinkedIn show that the line is not simple.
Copyrighted content (a video, a blog post) must not be copied, and data behind a login is covered by the site's user agreement, so do not scrape it.
Commercial use of scraped data invites expensive disputes even where the scraping itself was allowed.
Be a good guest: prefer an official API when there is one, read the site's robots.txt to see what it allows, and send at most one request a minute. This project fetches one page per run and identifies itself with a User-Agent.

Project: 100 movies to watch

The assignment: scrape Empire's "100 Greatest Movies" and write movies.txt with the titles numbered from 1) upwards. The code is infrastructure/empire_movies.py (parsing), domain/movie.py (reverse and number) and infrastructure/movie_file.py (writing). --source wayback (default) reads the Internet Archive snapshot of May 2020 that the assignment pins, --source live reads the current Empire page and --source fixture reads the saved headings in tests/fixtures/.

1. Choose the page (the saved headings of the page)

2. Inspect the markup (the 2020 page has <h3 class="title">; the current page has <h2><strong> headings that start with a rank)

python -m scrape_movies movies markup

3. Titles in page order (select("h3.title") and a list comprehension over get_text(); the page counts down from 100)

python -m scrape_movies movies titles

4. Reverse the list (titles[::-1], so the list counts up from 1)

python -m scrape_movies movies reversed

5. Write movies.txt (a with open(...) block, one title per line)

python -m scrape_movies movies write

6. The result

The committed movies.txt was produced with --source fixture on 2026-10-28 from the current Empire list, so its ranking differs from the 2020 list in the lecture video. The Internet Archive answered 429 Too Many Requests while this project was built, so the snapshot is read live only; its h3.title layout is covered by a hand-written sample in the tests.

🧪 Tests

python -m pytest

The tests use saved pages and never touch the network. Lint and types: ruff check ., ruff format --check . and mypy src tests. API pages: doxygen Doxyfile.

📄 License

See LICENSE.

🔗 Links

Repository
Documentation: wiki in English, Dansk and Bahasa Melayu; the pages are in docs/wiki/
Issue tracker

📎 Appendix: Project documents

Document	What it tells you
AGENTS.md	The rules AI agents follow in this repository
Business Case	Why the project exists, its scope and success criteria
Stakeholder Analysis	Who is involved; the stakeholder IDs used for owners and reviewers
Project Plan	The gateways and their schedule
Milestones	One document per gateway: Go/No-Go criteria and tasks
Artifact registry	Where each artifact type lives, the next version, the PO language and domain
Framework README	The framework this project uses
Framework changelog	What changed in each framework release