MIL-002: parsing HTML and making soup #36
@@ -56,12 +56,54 @@ Leave out `[dev]` to install only the runtime dependencies. Personal tokens belo
|
|||||||
|
|
||||||
## ▶️ Run
|
## ▶️ Run
|
||||||
|
|
||||||
The lecture steps are added milestone by milestone; each gets a section with its command, its output and a screenshot here.
|
Every lecture step is a command of the form `python -m scrape_movies <lecture> <step>`; without arguments the help is printed. Each step below shows the command, what it teaches and a screenshot of its output.
|
||||||
|
|
||||||
|
### Lecture: Parsing HTML and making soup
|
||||||
|
|
||||||
|
The sample page is `tests/fixtures/website.html`; the code is `src/scrape_movies/infrastructure/soup_basics.py`.
|
||||||
|
|
||||||
|
**1. Create a soup and read the title** (`BeautifulSoup(html, "html.parser")`, `soup.title.string`)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m scrape_movies # available from the movies milestone
|
python -m scrape_movies soup title
|
||||||
```
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
**2. Prettify** (`soup.prettify()` indents one tag per line)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m scrape_movies soup prettify
|
||||||
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
**3. Find all tags** (`find_all("a")`, then `get_text()` and `get("href")` in a loop)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m scrape_movies soup find-all
|
||||||
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
**4. Find by id and by class** (`find(id=...)`, `find_all("h3", class_=...)`; `class_` because `class` is a Python keyword)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m scrape_movies soup find
|
||||||
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
**5. CSS selectors** (`select_one("p a")` and `select(".heading")`)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m scrape_movies soup select
|
||||||
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
The screenshots are made by `python tools/make_screenshots.py`, which renders the command output as an image; no desktop is captured.
|
||||||
|
|
||||||
## 🧪 Tests
|
## 🧪 Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g.
|
|||||||
| SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 |
|
| SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 |
|
||||||
| PP | Project Plan | docs/project-plan.md | 002 |
|
| PP | Project Plan | docs/project-plan.md | 002 |
|
||||||
| MIL | Milestone / Gateway | docs/milestones/*.md | 006 |
|
| MIL | Milestone / Gateway | docs/milestones/*.md | 006 |
|
||||||
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 012 |
|
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 013 |
|
||||||
|
|
||||||
## Languages
|
## Languages
|
||||||
|
|
||||||
|
|||||||
Binary file not shown.
|
After Width: | Height: | Size: 4.5 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 9.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 9.0 KiB |
@@ -0,0 +1,62 @@
|
|||||||
|
# Review of MIL-002 Python code
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
| Key | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| ID | RC-012 |
|
||||||
|
| CrossReference | [MIL-002], [QC-PY-001] |
|
||||||
|
|
||||||
|
## Version History
|
||||||
|
| Date | Status | Author | Reviewer | Change | Commit |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | [fda38c9] |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Artifact Under Review
|
||||||
|
|
||||||
|
- Instance reviewed: [MIL-002]
|
||||||
|
- Checklist used: [QC-PY-001]
|
||||||
|
- Scope: full review. Code reviewed: soup_basics, soup_steps, cli, Link, tools/, tests/unit.
|
||||||
|
- Language and domain: n/a (technical type)
|
||||||
|
- Language reviewer: none
|
||||||
|
- Mode: autonomous, not independent (reviewer is `AI as stand-in for S01`)
|
||||||
|
|
||||||
|
## Checklist Results
|
||||||
|
|
||||||
|
| # | Criterion | Status | Evidence/Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE. |
|
||||||
|
| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | names state purpose; the lecture names (`find_all`, `select_one`) are kept in the helper names. |
|
||||||
|
| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | `ruff check` and `ruff format --check` are clean with no `noqa`. |
|
||||||
|
| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | every signature is annotated; `mypy --strict` passes, also in a venv without the third-party packages. |
|
||||||
|
| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | N-A | no exception is raised or caught. |
|
||||||
|
| 6 | No mutable default arguments and no shadowed builtins | Pass | no mutable defaults, no shadowed builtins. |
|
||||||
|
| 7 | Files, locks and connections are managed with context managers | N-A | no file or connection is opened except through `Path.read_text`. |
|
||||||
|
| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | every public module, class and function has a Doxygen docstring saying what it does. |
|
||||||
|
| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | N-A | no logging; the command-line output is the product, written with `print` in the presentation layer. |
|
||||||
|
| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | no Design Class Diagram; all tasks are plain technical tasks (Needs Use Case: No). |
|
||||||
|
| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | tests are named for behaviour, offline and independent of order. |
|
||||||
|
| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | `mypy --strict` runs over `src`, `tests` and `tools`. |
|
||||||
|
| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Fail | dependencies are declared with lower bounds, not pinned (see RC-009 action item); none is unused. |
|
||||||
|
| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | each constant and format is defined once; the step texts are built in one place. |
|
||||||
|
| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | `domain` imports nothing from other layers; `infrastructure` imports `domain` and `constants`; `presentation` composes them. |
|
||||||
|
| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | no class hierarchy; the dataclass `Link` has one reason to change. |
|
||||||
|
| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | code is in `domain`, `infrastructure` and `presentation`; tests are in `tests/unit`. |
|
||||||
|
| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | all names are English lowercase with `_`, no space. |
|
||||||
|
|
||||||
|
## Overall Verdict
|
||||||
|
|
||||||
|
Go — no Mandatory criterion fails and none is Pending. Stand-in review in autonomous mode, not independent; S01 challenges it in the run summary.
|
||||||
|
|
||||||
|
## Action Items
|
||||||
|
|
||||||
|
| Action | Owner | Due |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Pin dependencies or accept lower bounds (open from RC-009, optional criterion 13) | S01 | 2026-11-14 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
[MIL-002]: ../../milestones/mil-002-parse-html.md
|
||||||
|
[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md
|
||||||
|
[fda38c9]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/commit/fda38c906e3d470482ae1cc4a8c032991f7efe26
|
||||||
@@ -21,6 +21,7 @@ dev = [
|
|||||||
"ruff>=0.6",
|
"ruff>=0.6",
|
||||||
"mypy>=1.11",
|
"mypy>=1.11",
|
||||||
"types-requests>=2.32",
|
"types-requests>=2.32",
|
||||||
|
"pillow>=10.4",
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.setuptools.packages.find]
|
[tool.setuptools.packages.find]
|
||||||
@@ -46,3 +47,7 @@ ignore = ["D203", "D213"]
|
|||||||
strict = true
|
strict = true
|
||||||
python_version = "3.13"
|
python_version = "3.13"
|
||||||
mypy_path = "src"
|
mypy_path = "src"
|
||||||
|
|
||||||
|
[[tool.mypy.overrides]]
|
||||||
|
module = ["bs4.*", "requests.*", "PIL.*", "pytest.*"]
|
||||||
|
ignore_missing_imports = true
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
"""@brief Allow `python -m scrape_movies`."""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from scrape_movies.presentation.cli import main
|
||||||
|
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
"""@brief The Link value object: the text and the address of an anchor."""
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Link:
|
||||||
|
"""@brief A hyperlink found on a page.
|
||||||
|
|
||||||
|
@param text The visible text of the anchor.
|
||||||
|
@param url The value of its `href` attribute.
|
||||||
|
"""
|
||||||
|
|
||||||
|
text: str
|
||||||
|
url: str
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
"""@brief BeautifulSoup basics from the "Parsing HTML and Making Soup" lectures."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup, Tag
|
||||||
|
|
||||||
|
from scrape_movies.constants import HTML_PARSER
|
||||||
|
from scrape_movies.domain.link import Link
|
||||||
|
|
||||||
|
|
||||||
|
def load_html(path: Path) -> str:
|
||||||
|
"""@brief Read an HTML file as text.
|
||||||
|
|
||||||
|
@param path The file to read.
|
||||||
|
@return The content of the file.
|
||||||
|
"""
|
||||||
|
return path.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def make_soup(html: str) -> BeautifulSoup:
|
||||||
|
"""@brief Parse HTML text into a soup object.
|
||||||
|
|
||||||
|
@param html The HTML document as text.
|
||||||
|
@return The parsed document.
|
||||||
|
"""
|
||||||
|
return BeautifulSoup(html, HTML_PARSER)
|
||||||
|
|
||||||
|
|
||||||
|
def get_title(soup: BeautifulSoup) -> str | None:
|
||||||
|
"""@brief Return the text of the `title` tag.
|
||||||
|
|
||||||
|
@param soup The parsed document.
|
||||||
|
@return The title text, or None when the page has no title.
|
||||||
|
"""
|
||||||
|
if soup.title is None or soup.title.string is None:
|
||||||
|
return None
|
||||||
|
return str(soup.title.string)
|
||||||
|
|
||||||
|
|
||||||
|
def prettify_html(soup: BeautifulSoup) -> str:
|
||||||
|
"""@brief Return the document indented one tag per line.
|
||||||
|
|
||||||
|
@param soup The parsed document.
|
||||||
|
@return The indented HTML.
|
||||||
|
"""
|
||||||
|
return str(soup.prettify())
|
||||||
|
|
||||||
|
|
||||||
|
def find_all_anchors(soup: BeautifulSoup) -> list[Link]:
|
||||||
|
"""@brief Find every anchor tag with its text and address (`find_all`).
|
||||||
|
|
||||||
|
@param soup The parsed document.
|
||||||
|
@return One Link per anchor that has an `href` attribute.
|
||||||
|
"""
|
||||||
|
links: list[Link] = []
|
||||||
|
for anchor in soup.find_all("a"):
|
||||||
|
url = anchor.get("href")
|
||||||
|
if isinstance(url, str):
|
||||||
|
links.append(Link(text=str(anchor.get_text()), url=url))
|
||||||
|
return links
|
||||||
|
|
||||||
|
|
||||||
|
def find_text_by_id(soup: BeautifulSoup, element_id: str) -> str | None:
|
||||||
|
"""@brief Return the text of the first tag with the given id (`find`).
|
||||||
|
|
||||||
|
@param soup The parsed document.
|
||||||
|
@param element_id The value of the `id` attribute.
|
||||||
|
@return The text, or None when no tag has that id.
|
||||||
|
"""
|
||||||
|
tag = soup.find(id=element_id)
|
||||||
|
return str(tag.get_text()) if isinstance(tag, Tag) else None
|
||||||
|
|
||||||
|
|
||||||
|
def find_all_texts_by_class(
|
||||||
|
soup: BeautifulSoup, tag_name: str, class_name: str
|
||||||
|
) -> list[str]:
|
||||||
|
"""@brief Return the text of every tag with the given class (`find_all`).
|
||||||
|
|
||||||
|
The keyword is `class_` because `class` is a reserved word in Python.
|
||||||
|
|
||||||
|
@param soup The parsed document.
|
||||||
|
@param tag_name The tag to look for, for example `h3`.
|
||||||
|
@param class_name The CSS class the tag must have.
|
||||||
|
@return The texts in page order.
|
||||||
|
"""
|
||||||
|
return [str(tag.get_text()) for tag in soup.find_all(tag_name, class_=class_name)]
|
||||||
|
|
||||||
|
|
||||||
|
def select_one_text(soup: BeautifulSoup, selector: str) -> str | None:
|
||||||
|
"""@brief Return the text of the first tag matching a CSS selector.
|
||||||
|
|
||||||
|
@param soup The parsed document.
|
||||||
|
@param selector A CSS selector such as `p a` or `#name`.
|
||||||
|
@return The text, or None when nothing matches.
|
||||||
|
"""
|
||||||
|
tag = soup.select_one(selector)
|
||||||
|
return str(tag.get_text()) if tag is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
def select_texts(soup: BeautifulSoup, selector: str) -> list[str]:
|
||||||
|
"""@brief Return the text of every tag matching a CSS selector.
|
||||||
|
|
||||||
|
@param soup The parsed document.
|
||||||
|
@param selector A CSS selector such as `.heading`.
|
||||||
|
@return The texts in page order.
|
||||||
|
"""
|
||||||
|
return [str(tag.get_text()) for tag in soup.select(selector)]
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
"""@brief Command-line entry point that runs the lecture steps."""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
from scrape_movies.presentation import soup_steps
|
||||||
|
|
||||||
|
|
||||||
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
|
"""@brief Build the argument parser.
|
||||||
|
|
||||||
|
@return The parser with one sub-command per lecture.
|
||||||
|
"""
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
prog="python -m scrape_movies",
|
||||||
|
description="Day 45 of 100 Days of Code: web scraping exercises.",
|
||||||
|
)
|
||||||
|
commands = parser.add_subparsers(dest="command")
|
||||||
|
soup = commands.add_parser("soup", help="parse the sample page website.html")
|
||||||
|
soup.add_argument("step", choices=list(soup_steps.STEPS))
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: Sequence[str] | None = None) -> int:
|
||||||
|
"""@brief Run the command given on the command line.
|
||||||
|
|
||||||
|
@param argv The arguments, or None to read `sys.argv`.
|
||||||
|
@return The process exit code.
|
||||||
|
"""
|
||||||
|
parser = build_parser()
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
if args.command == "soup":
|
||||||
|
print(soup_steps.render_step(args.step))
|
||||||
|
return 0
|
||||||
|
parser.print_help()
|
||||||
|
return 0
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
"""@brief Text output of each "Parsing HTML and Making Soup" lecture step."""
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from scrape_movies.constants import WEBSITE_HTML_PATH
|
||||||
|
from scrape_movies.infrastructure import soup_basics
|
||||||
|
|
||||||
|
|
||||||
|
def _title(soup: BeautifulSoup) -> str:
|
||||||
|
return f"soup.title.string -> {soup_basics.get_title(soup)}"
|
||||||
|
|
||||||
|
|
||||||
|
def _prettify(soup: BeautifulSoup) -> str:
|
||||||
|
return soup_basics.prettify_html(soup).rstrip()
|
||||||
|
|
||||||
|
|
||||||
|
def _find_all(soup: BeautifulSoup) -> str:
|
||||||
|
return "\n".join(
|
||||||
|
f"{link.text} -> {link.url}" for link in soup_basics.find_all_anchors(soup)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _find(soup: BeautifulSoup) -> str:
|
||||||
|
by_id = soup_basics.find_text_by_id(soup, "name")
|
||||||
|
by_class = soup_basics.find_all_texts_by_class(soup, "h3", "heading")
|
||||||
|
return f'find(id="name") -> {by_id}\nfind_all("h3", class_="heading") -> {by_class}'
|
||||||
|
|
||||||
|
|
||||||
|
def _select(soup: BeautifulSoup) -> str:
|
||||||
|
first = soup_basics.select_one_text(soup, "p a")
|
||||||
|
every = soup_basics.select_texts(soup, ".heading")
|
||||||
|
return f'select_one("p a") -> {first}\nselect(".heading") -> {every}'
|
||||||
|
|
||||||
|
|
||||||
|
## The lecture steps by command-line name, in lecture order.
|
||||||
|
STEPS: dict[str, Callable[[BeautifulSoup], str]] = {
|
||||||
|
"title": _title,
|
||||||
|
"prettify": _prettify,
|
||||||
|
"find-all": _find_all,
|
||||||
|
"find": _find,
|
||||||
|
"select": _select,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def render_step(step: str) -> str:
|
||||||
|
"""@brief Run one lecture step against the sample page.
|
||||||
|
|
||||||
|
@param step A key of STEPS.
|
||||||
|
@return The text the step prints.
|
||||||
|
@throws KeyError When the step name is unknown.
|
||||||
|
"""
|
||||||
|
soup = soup_basics.make_soup(soup_basics.load_html(WEBSITE_HTML_PATH))
|
||||||
|
return STEPS[step](soup)
|
||||||
Vendored
Vendored
+26
@@ -0,0 +1,26 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<title>Angela's Personal Site</title>
|
||||||
|
</head>
|
||||||
|
|
||||||
|
<body>
|
||||||
|
<h1 id="name">Angela Yu</h1>
|
||||||
|
<p><em>Founder of <strong><a href="https://www.appbrewery.co/">The App Brewery</a></strong>.</em></p>
|
||||||
|
<p>I am an iOS and Web Developer. I love coffee and motorcycles.</p>
|
||||||
|
<hr>
|
||||||
|
<h3 class="heading">Books and Teaching</h3>
|
||||||
|
<ul>
|
||||||
|
<li>The Complete iOS App Development Bootcamp</li>
|
||||||
|
<li>The Complete Web Development Bootcamp</li>
|
||||||
|
<li>100 Days of Code - The Complete Python Bootcamp</li>
|
||||||
|
</ul>
|
||||||
|
<hr>
|
||||||
|
<h3 class="heading">Other Pages</h3>
|
||||||
|
<a href="https://angelabauer.github.io/cv/hobbies.html">My Hobbies</a>
|
||||||
|
<a href="https://angelabauer.github.io/cv/contact-me.html">Contact Me</a>
|
||||||
|
</body>
|
||||||
|
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
import pytest
|
||||||
|
|
||||||
|
from scrape_movies.presentation import cli, soup_steps
|
||||||
|
|
||||||
|
|
||||||
|
def test_main_when_soup_step_is_given_prints_output(
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
for step in soup_steps.STEPS:
|
||||||
|
assert cli.main(["soup", step]) == 0
|
||||||
|
assert capsys.readouterr().out.strip(), step
|
||||||
|
|
||||||
|
|
||||||
|
def test_main_when_no_command_prints_help(
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
assert cli.main([]) == 0
|
||||||
|
assert "usage:" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_step_when_title_returns_page_title() -> None:
|
||||||
|
assert soup_steps.render_step("title") == (
|
||||||
|
"soup.title.string -> Angela's Personal Site"
|
||||||
|
)
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from scrape_movies.constants import WEBSITE_HTML_PATH
|
||||||
|
from scrape_movies.domain.link import Link
|
||||||
|
from scrape_movies.infrastructure import soup_basics
|
||||||
|
|
||||||
|
|
||||||
|
def _soup() -> BeautifulSoup:
|
||||||
|
return soup_basics.make_soup(soup_basics.load_html(WEBSITE_HTML_PATH))
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_title_when_page_has_title_returns_text() -> None:
|
||||||
|
assert soup_basics.get_title(_soup()) == "Angela's Personal Site"
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_title_when_page_has_no_title_returns_none() -> None:
|
||||||
|
assert soup_basics.get_title(soup_basics.make_soup("<p>x</p>")) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_prettify_html_when_called_puts_tags_on_own_lines() -> None:
|
||||||
|
pretty = soup_basics.prettify_html(soup_basics.make_soup("<p><b>x</b></p>"))
|
||||||
|
assert pretty.splitlines()[:3] == ["<p>", " <b>", " x"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_all_anchors_when_page_has_links_returns_text_and_url() -> None:
|
||||||
|
links = soup_basics.find_all_anchors(_soup())
|
||||||
|
assert links[0] == Link("The App Brewery", "https://www.appbrewery.co/")
|
||||||
|
assert [link.text for link in links] == [
|
||||||
|
"The App Brewery",
|
||||||
|
"My Hobbies",
|
||||||
|
"Contact Me",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_all_anchors_when_anchor_has_no_href_skips_it() -> None:
|
||||||
|
soup = soup_basics.make_soup("<a>no link</a><a href='/x'>x</a>")
|
||||||
|
assert soup_basics.find_all_anchors(soup) == [Link("x", "/x")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_text_by_id_when_id_exists_returns_text() -> None:
|
||||||
|
assert soup_basics.find_text_by_id(_soup(), "name") == "Angela Yu"
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_text_by_id_when_id_is_missing_returns_none() -> None:
|
||||||
|
assert soup_basics.find_text_by_id(_soup(), "nobody") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_all_texts_by_class_when_class_matches_returns_texts() -> None:
|
||||||
|
texts = soup_basics.find_all_texts_by_class(_soup(), "h3", "heading")
|
||||||
|
assert texts == ["Books and Teaching", "Other Pages"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_one_text_when_nested_selector_returns_first_match() -> None:
|
||||||
|
assert soup_basics.select_one_text(_soup(), "p a") == "The App Brewery"
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_one_text_when_nothing_matches_returns_none() -> None:
|
||||||
|
assert soup_basics.select_one_text(_soup(), "table") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_texts_when_class_selector_returns_all_matches() -> None:
|
||||||
|
assert soup_basics.select_texts(_soup(), ".heading") == [
|
||||||
|
"Books and Teaching",
|
||||||
|
"Other Pages",
|
||||||
|
]
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
"""@brief Run the lecture steps and save one terminal screenshot per step.
|
||||||
|
|
||||||
|
Usage: python tools/make_screenshots.py [name-prefix]
|
||||||
|
The images are written to docs/images/ and used by the README and the wiki.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from render_terminal_png import render
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
IMAGES = ROOT / "docs" / "images"
|
||||||
|
|
||||||
|
## File name (without .png) and the arguments given to `python -m scrape_movies`.
|
||||||
|
STEPS: dict[str, list[str]] = {
|
||||||
|
"soup-1-title": ["soup", "title"],
|
||||||
|
"soup-2-prettify": ["soup", "prettify"],
|
||||||
|
"soup-3-find-all": ["soup", "find-all"],
|
||||||
|
"soup-4-find": ["soup", "find"],
|
||||||
|
"soup-5-select": ["soup", "select"],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def capture(name: str, args: list[str]) -> Path:
|
||||||
|
"""@brief Run one step and write its screenshot.
|
||||||
|
|
||||||
|
@param name The image name without extension.
|
||||||
|
@param args The arguments for `python -m scrape_movies`.
|
||||||
|
@return The path of the PNG.
|
||||||
|
"""
|
||||||
|
result = subprocess.run(
|
||||||
|
[sys.executable, "-m", "scrape_movies", *args],
|
||||||
|
cwd=ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
encoding="utf-8",
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
target = IMAGES / f"{name}.png"
|
||||||
|
render("python -m scrape_movies " + " ".join(args), result.stdout, target)
|
||||||
|
return target
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
"""@brief Capture every step whose name starts with the given prefix."""
|
||||||
|
prefix = sys.argv[1] if len(sys.argv) > 1 else ""
|
||||||
|
for name, args in STEPS.items():
|
||||||
|
if name.startswith(prefix):
|
||||||
|
print(capture(name, args))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
"""@brief Render a command and its output as a terminal-style PNG screenshot.
|
||||||
|
|
||||||
|
The screenshots of the README and the wiki are made with this script, so no
|
||||||
|
desktop is captured. Needs Pillow (`pip install -e ".[dev]"`).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from PIL import Image, ImageDraw, ImageFont
|
||||||
|
|
||||||
|
BACKGROUND = (30, 30, 30)
|
||||||
|
TITLE_BAR = (50, 50, 50)
|
||||||
|
TEXT = (212, 212, 212)
|
||||||
|
PROMPT = (96, 200, 96)
|
||||||
|
FONT_SIZE = 16
|
||||||
|
PADDING = 16
|
||||||
|
BAR_HEIGHT = 30
|
||||||
|
MAX_COLUMNS = 110
|
||||||
|
MAX_LINES = 40
|
||||||
|
FONT_FILES = (
|
||||||
|
"consola.ttf",
|
||||||
|
"DejaVuSansMono.ttf",
|
||||||
|
"LiberationMono-Regular.ttf",
|
||||||
|
"Menlo.ttc",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_font() -> ImageFont.ImageFont | ImageFont.FreeTypeFont:
|
||||||
|
for name in FONT_FILES:
|
||||||
|
try:
|
||||||
|
return ImageFont.truetype(name, FONT_SIZE)
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
return ImageFont.load_default()
|
||||||
|
|
||||||
|
|
||||||
|
def _fit(lines: list[str]) -> list[str]:
|
||||||
|
fitted = [
|
||||||
|
line if len(line) <= MAX_COLUMNS else line[: MAX_COLUMNS - 3] + "..."
|
||||||
|
for line in lines
|
||||||
|
]
|
||||||
|
if len(fitted) > MAX_LINES:
|
||||||
|
fitted = fitted[:MAX_LINES] + ["..."]
|
||||||
|
return fitted
|
||||||
|
|
||||||
|
|
||||||
|
def render(command: str, output: str, target: Path) -> None:
|
||||||
|
"""@brief Draw the command and its output and save the PNG.
|
||||||
|
|
||||||
|
@param command The command line shown after the prompt.
|
||||||
|
@param output The text the command printed.
|
||||||
|
@param target The PNG file to write.
|
||||||
|
"""
|
||||||
|
font = _load_font()
|
||||||
|
lines = _fit(output.rstrip().splitlines())
|
||||||
|
prompt = "$ "
|
||||||
|
columns = max([len(prompt + command), *(len(line) for line in lines)])
|
||||||
|
char_width = font.getlength("M")
|
||||||
|
line_height = FONT_SIZE + 6
|
||||||
|
width = int(columns * char_width) + 2 * PADDING
|
||||||
|
height = BAR_HEIGHT + 2 * PADDING + line_height * (len(lines) + 1)
|
||||||
|
image = Image.new("RGB", (max(width, 480), height), BACKGROUND)
|
||||||
|
draw = ImageDraw.Draw(image)
|
||||||
|
draw.rectangle((0, 0, image.width, BAR_HEIGHT), fill=TITLE_BAR)
|
||||||
|
for index, color in enumerate(((255, 95, 86), (255, 189, 46), (39, 201, 63))):
|
||||||
|
x = PADDING + index * 22
|
||||||
|
draw.ellipse((x, 9, x + 12, 21), fill=color)
|
||||||
|
y = BAR_HEIGHT + PADDING
|
||||||
|
draw.text((PADDING, y), prompt, font=font, fill=PROMPT)
|
||||||
|
draw.text((PADDING + font.getlength(prompt), y), command, font=font, fill=TEXT)
|
||||||
|
for line in lines:
|
||||||
|
y += line_height
|
||||||
|
draw.text((PADDING, y), line, font=font, fill=TEXT)
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
image.save(target)
|
||||||
Reference in New Issue
Block a user