Tirsvad-Udemy-100-days-of-code/045-scrape-movies
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MIL-002: parsing HTML and making soup #36

Merged
Tirsvad merged 7 commits from mil-002-parse-html into main 2026-10-28 07:16:21 +01:00
Conversation 0 Commits 7 Files Changed 20
+683 -109
20 changed files with 683 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+150 -108
View File
@@ -1,108 +1,150 @@
# 🚀 scrape-movies
Day 45 of Udemy's "100 Days of Code: The Complete Python Pro Bootcamp": web scraping with Requests and BeautifulSoup. It parses a local HTML page, scrapes the Hacker News front page and writes Empire's 100 greatest movies to `movies.txt`. It is for fellow coursists and anyone looking for a small, tested scraping example.
## 📚 Table of Contents
- [Overview](#-overview)
- [Requirements](#-requirements)
- [Setup](#-setup)
- [Run](#-run)
- [Tests](#-tests)
- [License](#-license)
- [Links](#-links)
- [Appendix: Project documents](#-appendix-project-documents)
## 🧭 Overview
| Part | What it is for |
| --- | --- |
| `src/scrape_movies/` | The code in clean-architecture layers (`domain`, `application`, `infrastructure`, `presentation`) plus `constants.py`; the lecture steps `soup_basics`, `hacker_news` and the movies scraper follow milestone by milestone |
| `tests/` | pytest tests that run offline against saved pages in `tests/fixtures/` |
| `docs/` | Project documents (business case, plan, milestones, reviews) and the wiki in English, Danish and Bahasa Malaysia |
| `Doxyfile` | Generates the API pages from the Doxygen comments in the source |
The only runtime dependencies are `requests` and `beautifulsoup4`.
## 📋 Requirements
- Python 3.13 or newer (the `venv` module comes with it).
- `requests` and `beautifulsoup4` (installed by the setup below).
- Optional: [Doxygen](https://www.doxygen.nl/) to build the API pages.
## 🛠️ Setup
Create a local virtual environment named `.venv`, update `pip`, then install the project with its test tools.
Windows (PowerShell):
```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
```
Linux and macOS:
```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
```
Leave out `[dev]` to install only the runtime dependencies. Personal tokens belong in a git-ignored `.env`; the project never reads it.
## ▶️ Run
The lecture steps are added milestone by milestone; each gets a section with its command, its output and a screenshot here.
```bash
python -m scrape_movies # available from the movies milestone
```
## 🧪 Tests
```bash
python -m pytest
```
The tests use saved pages and never touch the network. Lint and types: `ruff check .`, `ruff format --check .` and `mypy src tests`. API pages: `doxygen Doxyfile`.
## 📄 License
See [LICENSE](./LICENSE).
## 🔗 Links
- [Repository][repository]
- [Documentation][documentation]
- [Issue tracker][issues]
## 📎 Appendix: Project documents
| Document | What it tells you |
| --- | --- |
| [AGENTS.md][agents] | The rules AI agents follow in this repository |
| [Business Case][BC-001] | Why the project exists, its scope and success criteria |
| [Stakeholder Analysis][SA-001] | Who is involved; the stakeholder IDs used for owners and reviewers |
| [Project Plan][PP-001] | The gateways and their schedule |
| [Milestones][milestones] | One document per gateway: Go/No-Go criteria and tasks |
| [Artifact registry][registry] | Where each artifact type lives, the next version, the PO language and domain |
| [Framework README][framework-readme] | The framework this project uses |
| [Framework changelog][framework-changelog] | What changed in each framework release |
---
[repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies
[documentation]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/wiki
[issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/issues
[agents]: ./AGENTS.md
[BC-001]: ./docs/business-case.md
[SA-001]: ./docs/stakeholder-analysis.md
[PP-001]: ./docs/project-plan.md
[milestones]: ./docs/milestones/
[registry]: ./docs/artifact-registry.md
[framework-readme]: ./framework/README.md
[framework-changelog]: ./framework/CHANGELOG.md
# 🚀 scrape-movies
Day 45 of Udemy's "100 Days of Code: The Complete Python Pro Bootcamp": web scraping with Requests and BeautifulSoup. It parses a local HTML page, scrapes the Hacker News front page and writes Empire's 100 greatest movies to `movies.txt`. It is for fellow coursists and anyone looking for a small, tested scraping example.
## 📚 Table of Contents
- [Overview](#-overview)
- [Requirements](#-requirements)
- [Setup](#-setup)
- [Run](#-run)
- [Tests](#-tests)
- [License](#-license)
- [Links](#-links)
- [Appendix: Project documents](#-appendix-project-documents)
## 🧭 Overview
| Part | What it is for |
| --- | --- |
| `src/scrape_movies/` | The code in clean-architecture layers (`domain`, `application`, `infrastructure`, `presentation`) plus `constants.py`; the lecture steps `soup_basics`, `hacker_news` and the movies scraper follow milestone by milestone |
| `tests/` | pytest tests that run offline against saved pages in `tests/fixtures/` |
| `docs/` | Project documents (business case, plan, milestones, reviews) and the wiki in English, Danish and Bahasa Malaysia |
| `Doxyfile` | Generates the API pages from the Doxygen comments in the source |
The only runtime dependencies are `requests` and `beautifulsoup4`.
## 📋 Requirements
- Python 3.13 or newer (the `venv` module comes with it).
- `requests` and `beautifulsoup4` (installed by the setup below).
- Optional: [Doxygen](https://www.doxygen.nl/) to build the API pages.
## 🛠️ Setup
Create a local virtual environment named `.venv`, update `pip`, then install the project with its test tools.
Windows (PowerShell):
```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
```
Linux and macOS:
```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
```
Leave out `[dev]` to install only the runtime dependencies. Personal tokens belong in a git-ignored `.env`; the project never reads it.
## ▶️ Run
Every lecture step is a command of the form `python -m scrape_movies <lecture> <step>`; without arguments the help is printed. Each step below shows the command, what it teaches and a screenshot of its output.
### Lecture: Parsing HTML and making soup
The sample page is `tests/fixtures/website.html`; the code is `src/scrape_movies/infrastructure/soup_basics.py`.
**1. Create a soup and read the title** (`BeautifulSoup(html, "html.parser")`, `soup.title.string`)
```bash
python -m scrape_movies soup title
```
![Step 1: soup title](docs/images/soup-1-title.png)
**2. Prettify** (`soup.prettify()` indents one tag per line)
```bash
python -m scrape_movies soup prettify
```
![Step 2: prettify](docs/images/soup-2-prettify.png)
**3. Find all tags** (`find_all("a")`, then `get_text()` and `get("href")` in a loop)
```bash
python -m scrape_movies soup find-all
```
![Step 3: find_all](docs/images/soup-3-find-all.png)
**4. Find by id and by class** (`find(id=...)`, `find_all("h3", class_=...)`; `class_` because `class` is a Python keyword)
```bash
python -m scrape_movies soup find
```
![Step 4: find by attribute](docs/images/soup-4-find.png)
**5. CSS selectors** (`select_one("p a")` and `select(".heading")`)
```bash
python -m scrape_movies soup select
```
![Step 5: select](docs/images/soup-5-select.png)
The screenshots are made by `python tools/make_screenshots.py`, which renders the command output as an image; no desktop is captured.
## 🧪 Tests
```bash
python -m pytest
```
The tests use saved pages and never touch the network. Lint and types: `ruff check .`, `ruff format --check .` and `mypy src tests`. API pages: `doxygen Doxyfile`.
## 📄 License
See [LICENSE](./LICENSE).
## 🔗 Links
- [Repository][repository]
- [Documentation][documentation]
- [Issue tracker][issues]
## 📎 Appendix: Project documents
| Document | What it tells you |
| --- | --- |
| [AGENTS.md][agents] | The rules AI agents follow in this repository |
| [Business Case][BC-001] | Why the project exists, its scope and success criteria |
| [Stakeholder Analysis][SA-001] | Who is involved; the stakeholder IDs used for owners and reviewers |
| [Project Plan][PP-001] | The gateways and their schedule |
| [Milestones][milestones] | One document per gateway: Go/No-Go criteria and tasks |
| [Artifact registry][registry] | Where each artifact type lives, the next version, the PO language and domain |
| [Framework README][framework-readme] | The framework this project uses |
| [Framework changelog][framework-changelog] | What changed in each framework release |
---
[repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies
[documentation]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/wiki
[issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/issues
[agents]: ./AGENTS.md
[BC-001]: ./docs/business-case.md
[SA-001]: ./docs/stakeholder-analysis.md
[PP-001]: ./docs/project-plan.md
[milestones]: ./docs/milestones/
[registry]: ./docs/artifact-registry.md
[framework-readme]: ./framework/README.md
[framework-changelog]: ./framework/CHANGELOG.md
docs/artifact-registry.md
+1 -1
View File
@@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g.
| SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 |
| PP | Project Plan | docs/project-plan.md | 002 |
| MIL | Milestone / Gateway | docs/milestones/*.md | 006 |
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 012 |
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 013 |
## Languages
docs/images/soup-1-title.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.5 KiB

docs/images/soup-2-prettify.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

docs/images/soup-3-find-all.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

docs/images/soup-4-find.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.2 KiB

docs/images/soup-5-select.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.0 KiB

docs/sqa/reviews/rc-012-mil-002-python.md
+62
View File
@@ -0,0 +1,62 @@
# Review of MIL-002 Python code
## Metadata
| Key | Value |
| --- | --- |
| ID | RC-012 |
| CrossReference | [MIL-002], [QC-PY-001] |
## Version History
| Date | Status | Author | Reviewer | Change | Commit |
| --- | --- | --- | --- | --- | --- |
| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | [fda38c9] |
---
## Artifact Under Review
- Instance reviewed: [MIL-002]
- Checklist used: [QC-PY-001]
- Scope: full review. Code reviewed: soup_basics, soup_steps, cli, Link, tools/, tests/unit.
- Language and domain: n/a (technical type)
- Language reviewer: none
- Mode: autonomous, not independent (reviewer is `AI as stand-in for S01`)
## Checklist Results
| # | Criterion | Status | Evidence/Notes |
| --- | --- | --- | --- |
| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE. |
| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | names state purpose; the lecture names (`find_all`, `select_one`) are kept in the helper names. |
| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | `ruff check` and `ruff format --check` are clean with no `noqa`. |
| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | every signature is annotated; `mypy --strict` passes, also in a venv without the third-party packages. |
| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | N-A | no exception is raised or caught. |
| 6 | No mutable default arguments and no shadowed builtins | Pass | no mutable defaults, no shadowed builtins. |
| 7 | Files, locks and connections are managed with context managers | N-A | no file or connection is opened except through `Path.read_text`. |
| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | every public module, class and function has a Doxygen docstring saying what it does. |
| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | N-A | no logging; the command-line output is the product, written with `print` in the presentation layer. |
| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | no Design Class Diagram; all tasks are plain technical tasks (Needs Use Case: No). |
| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | tests are named for behaviour, offline and independent of order. |
| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | `mypy --strict` runs over `src`, `tests` and `tools`. |
| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Fail | dependencies are declared with lower bounds, not pinned (see RC-009 action item); none is unused. |
| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | each constant and format is defined once; the step texts are built in one place. |
| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | `domain` imports nothing from other layers; `infrastructure` imports `domain` and `constants`; `presentation` composes them. |
| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | no class hierarchy; the dataclass `Link` has one reason to change. |
| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | code is in `domain`, `infrastructure` and `presentation`; tests are in `tests/unit`. |
| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | all names are English lowercase with `_`, no space. |
## Overall Verdict
Go — no Mandatory criterion fails and none is Pending. Stand-in review in autonomous mode, not independent; S01 challenges it in the run summary.
## Action Items
| Action | Owner | Due |
| --- | --- | --- |
| Pin dependencies or accept lower bounds (open from RC-009, optional criterion 13) | S01 | 2026-11-14 |
---
[MIL-002]: ../../milestones/mil-002-parse-html.md
[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md
[fda38c9]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/commit/fda38c906e3d470482ae1cc4a8c032991f7efe26
pyproject.toml
+5
View File
@@ -21,6 +21,7 @@ dev = [
"ruff>=0.6",
"mypy>=1.11",
"types-requests>=2.32",
"pillow>=10.4",
]
[tool.setuptools.packages.find]
@@ -46,3 +47,7 @@ ignore = ["D203", "D213"]
strict = true
python_version = "3.13"
mypy_path = "src"
[[tool.mypy.overrides]]
module = ["bs4.*", "requests.*", "PIL.*", "pytest.*"]
ignore_missing_imports = true
src/scrape_movies/__main__.py
+7
View File
@@ -0,0 +1,7 @@
"""@brief Allow `python -m scrape_movies`."""
import sys
from scrape_movies.presentation.cli import main
sys.exit(main())
src/scrape_movies/domain/link.py
+15
View File
@@ -0,0 +1,15 @@
"""@brief The Link value object: the text and the address of an anchor."""
from dataclasses import dataclass
@dataclass(frozen=True)
class Link:
"""@brief A hyperlink found on a page.
@param text The visible text of the anchor.
@param url The value of its `href` attribute.
"""
text: str
url: str
src/scrape_movies/infrastructure/soup_basics.py
+107
View File
@@ -0,0 +1,107 @@
"""@brief BeautifulSoup basics from the "Parsing HTML and Making Soup" lectures."""
from pathlib import Path
from bs4 import BeautifulSoup, Tag
from scrape_movies.constants import HTML_PARSER
from scrape_movies.domain.link import Link
def load_html(path: Path) -> str:
"""@brief Read an HTML file as text.
@param path The file to read.
@return The content of the file.
"""
return path.read_text(encoding="utf-8")
def make_soup(html: str) -> BeautifulSoup:
"""@brief Parse HTML text into a soup object.
@param html The HTML document as text.
@return The parsed document.
"""
return BeautifulSoup(html, HTML_PARSER)
def get_title(soup: BeautifulSoup) -> str | None:
"""@brief Return the text of the `title` tag.
@param soup The parsed document.
@return The title text, or None when the page has no title.
"""
if soup.title is None or soup.title.string is None:
return None
return str(soup.title.string)
def prettify_html(soup: BeautifulSoup) -> str:
"""@brief Return the document indented one tag per line.
@param soup The parsed document.
@return The indented HTML.
"""
return str(soup.prettify())
def find_all_anchors(soup: BeautifulSoup) -> list[Link]:
"""@brief Find every anchor tag with its text and address (`find_all`).
@param soup The parsed document.
@return One Link per anchor that has an `href` attribute.
"""
links: list[Link] = []
for anchor in soup.find_all("a"):
url = anchor.get("href")
if isinstance(url, str):
links.append(Link(text=str(anchor.get_text()), url=url))
return links
def find_text_by_id(soup: BeautifulSoup, element_id: str) -> str | None:
"""@brief Return the text of the first tag with the given id (`find`).
@param soup The parsed document.
@param element_id The value of the `id` attribute.
@return The text, or None when no tag has that id.
"""
tag = soup.find(id=element_id)
return str(tag.get_text()) if isinstance(tag, Tag) else None
def find_all_texts_by_class(
soup: BeautifulSoup, tag_name: str, class_name: str
) -> list[str]:
"""@brief Return the text of every tag with the given class (`find_all`).
The keyword is `class_` because `class` is a reserved word in Python.
@param soup The parsed document.
@param tag_name The tag to look for, for example `h3`.
@param class_name The CSS class the tag must have.
@return The texts in page order.
"""
return [str(tag.get_text()) for tag in soup.find_all(tag_name, class_=class_name)]
def select_one_text(soup: BeautifulSoup, selector: str) -> str | None:
"""@brief Return the text of the first tag matching a CSS selector.
@param soup The parsed document.
@param selector A CSS selector such as `p a` or `#name`.
@return The text, or None when nothing matches.
"""
tag = soup.select_one(selector)
return str(tag.get_text()) if tag is not None else None
def select_texts(soup: BeautifulSoup, selector: str) -> list[str]:
"""@brief Return the text of every tag matching a CSS selector.
@param soup The parsed document.
@param selector A CSS selector such as `.heading`.
@return The texts in page order.
"""
return [str(tag.get_text()) for tag in soup.select(selector)]
src/scrape_movies/presentation/cli.py
+36
View File
@@ -0,0 +1,36 @@
"""@brief Command-line entry point that runs the lecture steps."""
import argparse
from collections.abc import Sequence
from scrape_movies.presentation import soup_steps
def build_parser() -> argparse.ArgumentParser:
"""@brief Build the argument parser.
@return The parser with one sub-command per lecture.
"""
parser = argparse.ArgumentParser(
prog="python -m scrape_movies",
description="Day 45 of 100 Days of Code: web scraping exercises.",
)
commands = parser.add_subparsers(dest="command")
soup = commands.add_parser("soup", help="parse the sample page website.html")
soup.add_argument("step", choices=list(soup_steps.STEPS))
return parser
def main(argv: Sequence[str] | None = None) -> int:
"""@brief Run the command given on the command line.
@param argv The arguments, or None to read `sys.argv`.
@return The process exit code.
"""
parser = build_parser()
args = parser.parse_args(argv)
if args.command == "soup":
print(soup_steps.render_step(args.step))
return 0
parser.print_help()
return 0
src/scrape_movies/presentation/soup_steps.py
+55
View File
@@ -0,0 +1,55 @@
"""@brief Text output of each "Parsing HTML and Making Soup" lecture step."""
from collections.abc import Callable
from bs4 import BeautifulSoup
from scrape_movies.constants import WEBSITE_HTML_PATH
from scrape_movies.infrastructure import soup_basics
def _title(soup: BeautifulSoup) -> str:
return f"soup.title.string -> {soup_basics.get_title(soup)}"
def _prettify(soup: BeautifulSoup) -> str:
return soup_basics.prettify_html(soup).rstrip()
def _find_all(soup: BeautifulSoup) -> str:
return "\n".join(
f"{link.text} -> {link.url}" for link in soup_basics.find_all_anchors(soup)
)
def _find(soup: BeautifulSoup) -> str:
by_id = soup_basics.find_text_by_id(soup, "name")
by_class = soup_basics.find_all_texts_by_class(soup, "h3", "heading")
return f'find(id="name") -> {by_id}\nfind_all("h3", class_="heading") -> {by_class}'
def _select(soup: BeautifulSoup) -> str:
first = soup_basics.select_one_text(soup, "p a")
every = soup_basics.select_texts(soup, ".heading")
return f'select_one("p a") -> {first}\nselect(".heading") -> {every}'
## The lecture steps by command-line name, in lecture order.
STEPS: dict[str, Callable[[BeautifulSoup], str]] = {
"title": _title,
"prettify": _prettify,
"find-all": _find_all,
"find": _find,
"select": _select,
}
def render_step(step: str) -> str:
"""@brief Run one lecture step against the sample page.
@param step A key of STEPS.
@return The text the step prints.
@throws KeyError When the step name is unknown.
"""
soup = soup_basics.make_soup(soup_basics.load_html(WEBSITE_HTML_PATH))
return STEPS[step](soup)
tests/fixtures/.gitkeep
Vendored
View File
tests/fixtures/website.html
Vendored
+26
View File
@@ -0,0 +1,26 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Angela's Personal Site</title>
</head>
<body>
<h1 id="name">Angela Yu</h1>
<p><em>Founder of <strong><a href="https://www.appbrewery.co/">The App Brewery</a></strong>.</em></p>
<p>I am an iOS and Web Developer. I love coffee and motorcycles.</p>
<hr>
<h3 class="heading">Books and Teaching</h3>
<ul>
<li>The Complete iOS App Development Bootcamp</li>
<li>The Complete Web Development Bootcamp</li>
<li>100 Days of Code - The Complete Python Bootcamp</li>
</ul>
<hr>
<h3 class="heading">Other Pages</h3>
<a href="https://angelabauer.github.io/cv/hobbies.html">My Hobbies</a>
<a href="https://angelabauer.github.io/cv/contact-me.html">Contact Me</a>
</body>
</html>
tests/unit/test_cli.py
+24
View File
@@ -0,0 +1,24 @@
import pytest
from scrape_movies.presentation import cli, soup_steps
def test_main_when_soup_step_is_given_prints_output(
capsys: pytest.CaptureFixture[str],
) -> None:
for step in soup_steps.STEPS:
assert cli.main(["soup", step]) == 0
assert capsys.readouterr().out.strip(), step
def test_main_when_no_command_prints_help(
capsys: pytest.CaptureFixture[str],
) -> None:
assert cli.main([]) == 0
assert "usage:" in capsys.readouterr().out
def test_render_step_when_title_returns_page_title() -> None:
assert soup_steps.render_step("title") == (
"soup.title.string -> Angela's Personal Site"
)
tests/unit/test_soup_basics.py
+65
View File
@@ -0,0 +1,65 @@
from bs4 import BeautifulSoup
from scrape_movies.constants import WEBSITE_HTML_PATH
from scrape_movies.domain.link import Link
from scrape_movies.infrastructure import soup_basics
def _soup() -> BeautifulSoup:
return soup_basics.make_soup(soup_basics.load_html(WEBSITE_HTML_PATH))
def test_get_title_when_page_has_title_returns_text() -> None:
assert soup_basics.get_title(_soup()) == "Angela's Personal Site"
def test_get_title_when_page_has_no_title_returns_none() -> None:
assert soup_basics.get_title(soup_basics.make_soup("<p>x</p>")) is None
def test_prettify_html_when_called_puts_tags_on_own_lines() -> None:
pretty = soup_basics.prettify_html(soup_basics.make_soup("<p><b>x</b></p>"))
assert pretty.splitlines()[:3] == ["<p>", " <b>", " x"]
def test_find_all_anchors_when_page_has_links_returns_text_and_url() -> None:
links = soup_basics.find_all_anchors(_soup())
assert links[0] == Link("The App Brewery", "https://www.appbrewery.co/")
assert [link.text for link in links] == [
"The App Brewery",
"My Hobbies",
"Contact Me",
]
def test_find_all_anchors_when_anchor_has_no_href_skips_it() -> None:
soup = soup_basics.make_soup("<a>no link</a><a href='/x'>x</a>")
assert soup_basics.find_all_anchors(soup) == [Link("x", "/x")]
def test_find_text_by_id_when_id_exists_returns_text() -> None:
assert soup_basics.find_text_by_id(_soup(), "name") == "Angela Yu"
def test_find_text_by_id_when_id_is_missing_returns_none() -> None:
assert soup_basics.find_text_by_id(_soup(), "nobody") is None
def test_find_all_texts_by_class_when_class_matches_returns_texts() -> None:
texts = soup_basics.find_all_texts_by_class(_soup(), "h3", "heading")
assert texts == ["Books and Teaching", "Other Pages"]
def test_select_one_text_when_nested_selector_returns_first_match() -> None:
assert soup_basics.select_one_text(_soup(), "p a") == "The App Brewery"
def test_select_one_text_when_nothing_matches_returns_none() -> None:
assert soup_basics.select_one_text(_soup(), "table") is None
def test_select_texts_when_class_selector_returns_all_matches() -> None:
assert soup_basics.select_texts(_soup(), ".heading") == [
"Books and Teaching",
"Other Pages",
]
tools/make_screenshots.py
+55
View File
@@ -0,0 +1,55 @@
"""@brief Run the lecture steps and save one terminal screenshot per step.
Usage: python tools/make_screenshots.py [name-prefix]
The images are written to docs/images/ and used by the README and the wiki.
"""
import subprocess
import sys
from pathlib import Path
from render_terminal_png import render
ROOT = Path(__file__).resolve().parents[1]
IMAGES = ROOT / "docs" / "images"
## File name (without .png) and the arguments given to `python -m scrape_movies`.
STEPS: dict[str, list[str]] = {
"soup-1-title": ["soup", "title"],
"soup-2-prettify": ["soup", "prettify"],
"soup-3-find-all": ["soup", "find-all"],
"soup-4-find": ["soup", "find"],
"soup-5-select": ["soup", "select"],
}
def capture(name: str, args: list[str]) -> Path:
"""@brief Run one step and write its screenshot.
@param name The image name without extension.
@param args The arguments for `python -m scrape_movies`.
@return The path of the PNG.
"""
result = subprocess.run(
[sys.executable, "-m", "scrape_movies", *args],
cwd=ROOT,
capture_output=True,
text=True,
encoding="utf-8",
check=True,
)
target = IMAGES / f"{name}.png"
render("python -m scrape_movies " + " ".join(args), result.stdout, target)
return target
def main() -> None:
"""@brief Capture every step whose name starts with the given prefix."""
prefix = sys.argv[1] if len(sys.argv) > 1 else ""
for name, args in STEPS.items():
if name.startswith(prefix):
print(capture(name, args))
if __name__ == "__main__":
main()
tools/render_terminal_png.py
+75
View File
@@ -0,0 +1,75 @@
"""@brief Render a command and its output as a terminal-style PNG screenshot.
The screenshots of the README and the wiki are made with this script, so no
desktop is captured. Needs Pillow (`pip install -e ".[dev]"`).
"""
from pathlib import Path
from PIL import Image, ImageDraw, ImageFont
BACKGROUND = (30, 30, 30)
TITLE_BAR = (50, 50, 50)
TEXT = (212, 212, 212)
PROMPT = (96, 200, 96)
FONT_SIZE = 16
PADDING = 16
BAR_HEIGHT = 30
MAX_COLUMNS = 110
MAX_LINES = 40
FONT_FILES = (
"consola.ttf",
"DejaVuSansMono.ttf",
"LiberationMono-Regular.ttf",
"Menlo.ttc",
)
def _load_font() -> ImageFont.ImageFont | ImageFont.FreeTypeFont:
for name in FONT_FILES:
try:
return ImageFont.truetype(name, FONT_SIZE)
except OSError:
continue
return ImageFont.load_default()
def _fit(lines: list[str]) -> list[str]:
fitted = [
line if len(line) <= MAX_COLUMNS else line[: MAX_COLUMNS - 3] + "..."
for line in lines
]
if len(fitted) > MAX_LINES:
fitted = fitted[:MAX_LINES] + ["..."]
return fitted
def render(command: str, output: str, target: Path) -> None:
"""@brief Draw the command and its output and save the PNG.
@param command The command line shown after the prompt.
@param output The text the command printed.
@param target The PNG file to write.
"""
font = _load_font()
lines = _fit(output.rstrip().splitlines())
prompt = "$ "
columns = max([len(prompt + command), *(len(line) for line in lines)])
char_width = font.getlength("M")
line_height = FONT_SIZE + 6
width = int(columns * char_width) + 2 * PADDING
height = BAR_HEIGHT + 2 * PADDING + line_height * (len(lines) + 1)
image = Image.new("RGB", (max(width, 480), height), BACKGROUND)
draw = ImageDraw.Draw(image)
draw.rectangle((0, 0, image.width, BAR_HEIGHT), fill=TITLE_BAR)
for index, color in enumerate(((255, 95, 86), (255, 189, 46), (39, 201, 63))):
x = PADDING + index * 22
draw.ellipse((x, 9, x + 12, 21), fill=color)
y = BAR_HEIGHT + PADDING
draw.text((PADDING, y), prompt, font=font, fill=PROMPT)
draw.text((PADDING + font.getlength(prompt), y), command, font=font, fill=TEXT)
for line in lines:
y += line_height
draw.text((PADDING, y), line, font=font, fill=TEXT)
target.parent.mkdir(parents=True, exist_ok=True)
image.save(target)