diff --git a/README.md b/README.md index 45cbac4..74211cb 100644 --- a/README.md +++ b/README.md @@ -1,108 +1,150 @@ -# ๐Ÿš€ scrape-movies - -Day 45 of Udemy's "100 Days of Code: The Complete Python Pro Bootcamp": web scraping with Requests and BeautifulSoup. It parses a local HTML page, scrapes the Hacker News front page and writes Empire's 100 greatest movies to `movies.txt`. It is for fellow coursists and anyone looking for a small, tested scraping example. - -## ๐Ÿ“š Table of Contents - -- [Overview](#-overview) -- [Requirements](#-requirements) -- [Setup](#-setup) -- [Run](#-run) -- [Tests](#-tests) -- [License](#-license) -- [Links](#-links) -- [Appendix: Project documents](#-appendix-project-documents) - -## ๐Ÿงญ Overview - -| Part | What it is for | -| --- | --- | -| `src/scrape_movies/` | The code in clean-architecture layers (`domain`, `application`, `infrastructure`, `presentation`) plus `constants.py`; the lecture steps `soup_basics`, `hacker_news` and the movies scraper follow milestone by milestone | -| `tests/` | pytest tests that run offline against saved pages in `tests/fixtures/` | -| `docs/` | Project documents (business case, plan, milestones, reviews) and the wiki in English, Danish and Bahasa Malaysia | -| `Doxyfile` | Generates the API pages from the Doxygen comments in the source | - -The only runtime dependencies are `requests` and `beautifulsoup4`. - -## ๐Ÿ“‹ Requirements - -- Python 3.13 or newer (the `venv` module comes with it). -- `requests` and `beautifulsoup4` (installed by the setup below). -- Optional: [Doxygen](https://www.doxygen.nl/) to build the API pages. - -## ๐Ÿ› ๏ธ Setup - -Create a local virtual environment named `.venv`, update `pip`, then install the project with its test tools. - -Windows (PowerShell): - -```powershell -python -m venv .venv -.venv\Scripts\Activate.ps1 -python -m pip install --upgrade pip -python -m pip install -e ".[dev]" -``` - -Linux and macOS: - -```bash -python3 -m venv .venv -source .venv/bin/activate -python -m pip install --upgrade pip -python -m pip install -e ".[dev]" -``` - -Leave out `[dev]` to install only the runtime dependencies. Personal tokens belong in a git-ignored `.env`; the project never reads it. - -## โ–ถ๏ธ Run - -The lecture steps are added milestone by milestone; each gets a section with its command, its output and a screenshot here. - -```bash -python -m scrape_movies # available from the movies milestone -``` - -## ๐Ÿงช Tests - -```bash -python -m pytest -``` - -The tests use saved pages and never touch the network. Lint and types: `ruff check .`, `ruff format --check .` and `mypy src tests`. API pages: `doxygen Doxyfile`. - -## ๐Ÿ“„ License - -See [LICENSE](./LICENSE). - -## ๐Ÿ”— Links - -- [Repository][repository] -- [Documentation][documentation] -- [Issue tracker][issues] - -## ๐Ÿ“Ž Appendix: Project documents - -| Document | What it tells you | -| --- | --- | -| [AGENTS.md][agents] | The rules AI agents follow in this repository | -| [Business Case][BC-001] | Why the project exists, its scope and success criteria | -| [Stakeholder Analysis][SA-001] | Who is involved; the stakeholder IDs used for owners and reviewers | -| [Project Plan][PP-001] | The gateways and their schedule | -| [Milestones][milestones] | One document per gateway: Go/No-Go criteria and tasks | -| [Artifact registry][registry] | Where each artifact type lives, the next version, the PO language and domain | -| [Framework README][framework-readme] | The framework this project uses | -| [Framework changelog][framework-changelog] | What changed in each framework release | - ---- - -[repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies -[documentation]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/wiki -[issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/issues -[agents]: ./AGENTS.md -[BC-001]: ./docs/business-case.md -[SA-001]: ./docs/stakeholder-analysis.md -[PP-001]: ./docs/project-plan.md -[milestones]: ./docs/milestones/ -[registry]: ./docs/artifact-registry.md -[framework-readme]: ./framework/README.md -[framework-changelog]: ./framework/CHANGELOG.md +# ๐Ÿš€ scrape-movies + +Day 45 of Udemy's "100 Days of Code: The Complete Python Pro Bootcamp": web scraping with Requests and BeautifulSoup. It parses a local HTML page, scrapes the Hacker News front page and writes Empire's 100 greatest movies to `movies.txt`. It is for fellow coursists and anyone looking for a small, tested scraping example. + +## ๐Ÿ“š Table of Contents + +- [Overview](#-overview) +- [Requirements](#-requirements) +- [Setup](#-setup) +- [Run](#-run) +- [Tests](#-tests) +- [License](#-license) +- [Links](#-links) +- [Appendix: Project documents](#-appendix-project-documents) + +## ๐Ÿงญ Overview + +| Part | What it is for | +| --- | --- | +| `src/scrape_movies/` | The code in clean-architecture layers (`domain`, `application`, `infrastructure`, `presentation`) plus `constants.py`; the lecture steps `soup_basics`, `hacker_news` and the movies scraper follow milestone by milestone | +| `tests/` | pytest tests that run offline against saved pages in `tests/fixtures/` | +| `docs/` | Project documents (business case, plan, milestones, reviews) and the wiki in English, Danish and Bahasa Malaysia | +| `Doxyfile` | Generates the API pages from the Doxygen comments in the source | + +The only runtime dependencies are `requests` and `beautifulsoup4`. + +## ๐Ÿ“‹ Requirements + +- Python 3.13 or newer (the `venv` module comes with it). +- `requests` and `beautifulsoup4` (installed by the setup below). +- Optional: [Doxygen](https://www.doxygen.nl/) to build the API pages. + +## ๐Ÿ› ๏ธ Setup + +Create a local virtual environment named `.venv`, update `pip`, then install the project with its test tools. + +Windows (PowerShell): + +```powershell +python -m venv .venv +.venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install -e ".[dev]" +``` + +Linux and macOS: + +```bash +python3 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +python -m pip install -e ".[dev]" +``` + +Leave out `[dev]` to install only the runtime dependencies. Personal tokens belong in a git-ignored `.env`; the project never reads it. + +## โ–ถ๏ธ Run + +Every lecture step is a command of the form `python -m scrape_movies `; without arguments the help is printed. Each step below shows the command, what it teaches and a screenshot of its output. + +### Lecture: Parsing HTML and making soup + +The sample page is `tests/fixtures/website.html`; the code is `src/scrape_movies/infrastructure/soup_basics.py`. + +**1. Create a soup and read the title** (`BeautifulSoup(html, "html.parser")`, `soup.title.string`) + +```bash +python -m scrape_movies soup title +``` + +![Step 1: soup title](docs/images/soup-1-title.png) + +**2. Prettify** (`soup.prettify()` indents one tag per line) + +```bash +python -m scrape_movies soup prettify +``` + +![Step 2: prettify](docs/images/soup-2-prettify.png) + +**3. Find all tags** (`find_all("a")`, then `get_text()` and `get("href")` in a loop) + +```bash +python -m scrape_movies soup find-all +``` + +![Step 3: find_all](docs/images/soup-3-find-all.png) + +**4. Find by id and by class** (`find(id=...)`, `find_all("h3", class_=...)`; `class_` because `class` is a Python keyword) + +```bash +python -m scrape_movies soup find +``` + +![Step 4: find by attribute](docs/images/soup-4-find.png) + +**5. CSS selectors** (`select_one("p a")` and `select(".heading")`) + +```bash +python -m scrape_movies soup select +``` + +![Step 5: select](docs/images/soup-5-select.png) + +The screenshots are made by `python tools/make_screenshots.py`, which renders the command output as an image; no desktop is captured. + +## ๐Ÿงช Tests + +```bash +python -m pytest +``` + +The tests use saved pages and never touch the network. Lint and types: `ruff check .`, `ruff format --check .` and `mypy src tests`. API pages: `doxygen Doxyfile`. + +## ๐Ÿ“„ License + +See [LICENSE](./LICENSE). + +## ๐Ÿ”— Links + +- [Repository][repository] +- [Documentation][documentation] +- [Issue tracker][issues] + +## ๐Ÿ“Ž Appendix: Project documents + +| Document | What it tells you | +| --- | --- | +| [AGENTS.md][agents] | The rules AI agents follow in this repository | +| [Business Case][BC-001] | Why the project exists, its scope and success criteria | +| [Stakeholder Analysis][SA-001] | Who is involved; the stakeholder IDs used for owners and reviewers | +| [Project Plan][PP-001] | The gateways and their schedule | +| [Milestones][milestones] | One document per gateway: Go/No-Go criteria and tasks | +| [Artifact registry][registry] | Where each artifact type lives, the next version, the PO language and domain | +| [Framework README][framework-readme] | The framework this project uses | +| [Framework changelog][framework-changelog] | What changed in each framework release | + +--- + +[repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies +[documentation]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/wiki +[issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/issues +[agents]: ./AGENTS.md +[BC-001]: ./docs/business-case.md +[SA-001]: ./docs/stakeholder-analysis.md +[PP-001]: ./docs/project-plan.md +[milestones]: ./docs/milestones/ +[registry]: ./docs/artifact-registry.md +[framework-readme]: ./framework/README.md +[framework-changelog]: ./framework/CHANGELOG.md diff --git a/docs/artifact-registry.md b/docs/artifact-registry.md index 3fe8fa4..7cdb474 100644 --- a/docs/artifact-registry.md +++ b/docs/artifact-registry.md @@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g. | SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 | | PP | Project Plan | docs/project-plan.md | 002 | | MIL | Milestone / Gateway | docs/milestones/*.md | 006 | -| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 012 | +| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 013 | ## Languages diff --git a/docs/images/soup-1-title.png b/docs/images/soup-1-title.png new file mode 100644 index 0000000..c3a207d Binary files /dev/null and b/docs/images/soup-1-title.png differ diff --git a/docs/images/soup-2-prettify.png b/docs/images/soup-2-prettify.png new file mode 100644 index 0000000..eb9a136 Binary files /dev/null and b/docs/images/soup-2-prettify.png differ diff --git a/docs/images/soup-3-find-all.png b/docs/images/soup-3-find-all.png new file mode 100644 index 0000000..9751f45 Binary files /dev/null and b/docs/images/soup-3-find-all.png differ diff --git a/docs/images/soup-4-find.png b/docs/images/soup-4-find.png new file mode 100644 index 0000000..c5d66c6 Binary files /dev/null and b/docs/images/soup-4-find.png differ diff --git a/docs/images/soup-5-select.png b/docs/images/soup-5-select.png new file mode 100644 index 0000000..4fa5d3d Binary files /dev/null and b/docs/images/soup-5-select.png differ diff --git a/docs/sqa/reviews/rc-012-mil-002-python.md b/docs/sqa/reviews/rc-012-mil-002-python.md new file mode 100644 index 0000000..be72a35 --- /dev/null +++ b/docs/sqa/reviews/rc-012-mil-002-python.md @@ -0,0 +1,62 @@ +# Review of MIL-002 Python code + +## Metadata +| Key | Value | +| --- | --- | +| ID | RC-012 | +| CrossReference | [MIL-002], [QC-PY-001] | + +## Version History +| Date | Status | Author | Reviewer | Change | Commit | +| --- | --- | --- | --- | --- | --- | +| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | [fda38c9] | + +--- + +## Artifact Under Review + +- Instance reviewed: [MIL-002] +- Checklist used: [QC-PY-001] +- Scope: full review. Code reviewed: soup_basics, soup_steps, cli, Link, tools/, tests/unit. +- Language and domain: n/a (technical type) +- Language reviewer: none +- Mode: autonomous, not independent (reviewer is `AI as stand-in for S01`) + +## Checklist Results + +| # | Criterion | Status | Evidence/Notes | +| --- | --- | --- | --- | +| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE. | +| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | names state purpose; the lecture names (`find_all`, `select_one`) are kept in the helper names. | +| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | `ruff check` and `ruff format --check` are clean with no `noqa`. | +| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | every signature is annotated; `mypy --strict` passes, also in a venv without the third-party packages. | +| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | N-A | no exception is raised or caught. | +| 6 | No mutable default arguments and no shadowed builtins | Pass | no mutable defaults, no shadowed builtins. | +| 7 | Files, locks and connections are managed with context managers | N-A | no file or connection is opened except through `Path.read_text`. | +| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | every public module, class and function has a Doxygen docstring saying what it does. | +| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | N-A | no logging; the command-line output is the product, written with `print` in the presentation layer. | +| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | no Design Class Diagram; all tasks are plain technical tasks (Needs Use Case: No). | +| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | tests are named for behaviour, offline and independent of order. | +| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | `mypy --strict` runs over `src`, `tests` and `tools`. | +| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Fail | dependencies are declared with lower bounds, not pinned (see RC-009 action item); none is unused. | +| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | each constant and format is defined once; the step texts are built in one place. | +| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | `domain` imports nothing from other layers; `infrastructure` imports `domain` and `constants`; `presentation` composes them. | +| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | no class hierarchy; the dataclass `Link` has one reason to change. | +| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | code is in `domain`, `infrastructure` and `presentation`; tests are in `tests/unit`. | +| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | all names are English lowercase with `_`, no space. | + +## Overall Verdict + +Go โ€” no Mandatory criterion fails and none is Pending. Stand-in review in autonomous mode, not independent; S01 challenges it in the run summary. + +## Action Items + +| Action | Owner | Due | +| --- | --- | --- | +| Pin dependencies or accept lower bounds (open from RC-009, optional criterion 13) | S01 | 2026-11-14 | + +--- + +[MIL-002]: ../../milestones/mil-002-parse-html.md +[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md +[fda38c9]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/commit/fda38c906e3d470482ae1cc4a8c032991f7efe26 diff --git a/pyproject.toml b/pyproject.toml index 352ca05..7e5cd5f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,6 +21,7 @@ dev = [ "ruff>=0.6", "mypy>=1.11", "types-requests>=2.32", + "pillow>=10.4", ] [tool.setuptools.packages.find] @@ -46,3 +47,7 @@ ignore = ["D203", "D213"] strict = true python_version = "3.13" mypy_path = "src" + +[[tool.mypy.overrides]] +module = ["bs4.*", "requests.*", "PIL.*", "pytest.*"] +ignore_missing_imports = true diff --git a/src/scrape_movies/__main__.py b/src/scrape_movies/__main__.py new file mode 100644 index 0000000..3c37d48 --- /dev/null +++ b/src/scrape_movies/__main__.py @@ -0,0 +1,7 @@ +"""@brief Allow `python -m scrape_movies`.""" + +import sys + +from scrape_movies.presentation.cli import main + +sys.exit(main()) diff --git a/src/scrape_movies/domain/link.py b/src/scrape_movies/domain/link.py new file mode 100644 index 0000000..286d670 --- /dev/null +++ b/src/scrape_movies/domain/link.py @@ -0,0 +1,15 @@ +"""@brief The Link value object: the text and the address of an anchor.""" + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Link: + """@brief A hyperlink found on a page. + + @param text The visible text of the anchor. + @param url The value of its `href` attribute. + """ + + text: str + url: str diff --git a/src/scrape_movies/infrastructure/soup_basics.py b/src/scrape_movies/infrastructure/soup_basics.py new file mode 100644 index 0000000..e2e8495 --- /dev/null +++ b/src/scrape_movies/infrastructure/soup_basics.py @@ -0,0 +1,107 @@ +"""@brief BeautifulSoup basics from the "Parsing HTML and Making Soup" lectures.""" + +from pathlib import Path + +from bs4 import BeautifulSoup, Tag + +from scrape_movies.constants import HTML_PARSER +from scrape_movies.domain.link import Link + + +def load_html(path: Path) -> str: + """@brief Read an HTML file as text. + + @param path The file to read. + @return The content of the file. + """ + return path.read_text(encoding="utf-8") + + +def make_soup(html: str) -> BeautifulSoup: + """@brief Parse HTML text into a soup object. + + @param html The HTML document as text. + @return The parsed document. + """ + return BeautifulSoup(html, HTML_PARSER) + + +def get_title(soup: BeautifulSoup) -> str | None: + """@brief Return the text of the `title` tag. + + @param soup The parsed document. + @return The title text, or None when the page has no title. + """ + if soup.title is None or soup.title.string is None: + return None + return str(soup.title.string) + + +def prettify_html(soup: BeautifulSoup) -> str: + """@brief Return the document indented one tag per line. + + @param soup The parsed document. + @return The indented HTML. + """ + return str(soup.prettify()) + + +def find_all_anchors(soup: BeautifulSoup) -> list[Link]: + """@brief Find every anchor tag with its text and address (`find_all`). + + @param soup The parsed document. + @return One Link per anchor that has an `href` attribute. + """ + links: list[Link] = [] + for anchor in soup.find_all("a"): + url = anchor.get("href") + if isinstance(url, str): + links.append(Link(text=str(anchor.get_text()), url=url)) + return links + + +def find_text_by_id(soup: BeautifulSoup, element_id: str) -> str | None: + """@brief Return the text of the first tag with the given id (`find`). + + @param soup The parsed document. + @param element_id The value of the `id` attribute. + @return The text, or None when no tag has that id. + """ + tag = soup.find(id=element_id) + return str(tag.get_text()) if isinstance(tag, Tag) else None + + +def find_all_texts_by_class( + soup: BeautifulSoup, tag_name: str, class_name: str +) -> list[str]: + """@brief Return the text of every tag with the given class (`find_all`). + + The keyword is `class_` because `class` is a reserved word in Python. + + @param soup The parsed document. + @param tag_name The tag to look for, for example `h3`. + @param class_name The CSS class the tag must have. + @return The texts in page order. + """ + return [str(tag.get_text()) for tag in soup.find_all(tag_name, class_=class_name)] + + +def select_one_text(soup: BeautifulSoup, selector: str) -> str | None: + """@brief Return the text of the first tag matching a CSS selector. + + @param soup The parsed document. + @param selector A CSS selector such as `p a` or `#name`. + @return The text, or None when nothing matches. + """ + tag = soup.select_one(selector) + return str(tag.get_text()) if tag is not None else None + + +def select_texts(soup: BeautifulSoup, selector: str) -> list[str]: + """@brief Return the text of every tag matching a CSS selector. + + @param soup The parsed document. + @param selector A CSS selector such as `.heading`. + @return The texts in page order. + """ + return [str(tag.get_text()) for tag in soup.select(selector)] diff --git a/src/scrape_movies/presentation/cli.py b/src/scrape_movies/presentation/cli.py new file mode 100644 index 0000000..5ed8ed1 --- /dev/null +++ b/src/scrape_movies/presentation/cli.py @@ -0,0 +1,36 @@ +"""@brief Command-line entry point that runs the lecture steps.""" + +import argparse +from collections.abc import Sequence + +from scrape_movies.presentation import soup_steps + + +def build_parser() -> argparse.ArgumentParser: + """@brief Build the argument parser. + + @return The parser with one sub-command per lecture. + """ + parser = argparse.ArgumentParser( + prog="python -m scrape_movies", + description="Day 45 of 100 Days of Code: web scraping exercises.", + ) + commands = parser.add_subparsers(dest="command") + soup = commands.add_parser("soup", help="parse the sample page website.html") + soup.add_argument("step", choices=list(soup_steps.STEPS)) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + """@brief Run the command given on the command line. + + @param argv The arguments, or None to read `sys.argv`. + @return The process exit code. + """ + parser = build_parser() + args = parser.parse_args(argv) + if args.command == "soup": + print(soup_steps.render_step(args.step)) + return 0 + parser.print_help() + return 0 diff --git a/src/scrape_movies/presentation/soup_steps.py b/src/scrape_movies/presentation/soup_steps.py new file mode 100644 index 0000000..0368319 --- /dev/null +++ b/src/scrape_movies/presentation/soup_steps.py @@ -0,0 +1,55 @@ +"""@brief Text output of each "Parsing HTML and Making Soup" lecture step.""" + +from collections.abc import Callable + +from bs4 import BeautifulSoup + +from scrape_movies.constants import WEBSITE_HTML_PATH +from scrape_movies.infrastructure import soup_basics + + +def _title(soup: BeautifulSoup) -> str: + return f"soup.title.string -> {soup_basics.get_title(soup)}" + + +def _prettify(soup: BeautifulSoup) -> str: + return soup_basics.prettify_html(soup).rstrip() + + +def _find_all(soup: BeautifulSoup) -> str: + return "\n".join( + f"{link.text} -> {link.url}" for link in soup_basics.find_all_anchors(soup) + ) + + +def _find(soup: BeautifulSoup) -> str: + by_id = soup_basics.find_text_by_id(soup, "name") + by_class = soup_basics.find_all_texts_by_class(soup, "h3", "heading") + return f'find(id="name") -> {by_id}\nfind_all("h3", class_="heading") -> {by_class}' + + +def _select(soup: BeautifulSoup) -> str: + first = soup_basics.select_one_text(soup, "p a") + every = soup_basics.select_texts(soup, ".heading") + return f'select_one("p a") -> {first}\nselect(".heading") -> {every}' + + +## The lecture steps by command-line name, in lecture order. +STEPS: dict[str, Callable[[BeautifulSoup], str]] = { + "title": _title, + "prettify": _prettify, + "find-all": _find_all, + "find": _find, + "select": _select, +} + + +def render_step(step: str) -> str: + """@brief Run one lecture step against the sample page. + + @param step A key of STEPS. + @return The text the step prints. + @throws KeyError When the step name is unknown. + """ + soup = soup_basics.make_soup(soup_basics.load_html(WEBSITE_HTML_PATH)) + return STEPS[step](soup) diff --git a/tests/fixtures/.gitkeep b/tests/fixtures/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/tests/fixtures/website.html b/tests/fixtures/website.html new file mode 100644 index 0000000..ce4dfbb --- /dev/null +++ b/tests/fixtures/website.html @@ -0,0 +1,26 @@ + + + + + + Angela's Personal Site + + + +

Angela Yu

+

Founder of The App Brewery.

+

I am an iOS and Web Developer. I love coffee and motorcycles.

+
+

Books and Teaching

+ +
+

Other Pages

+ My Hobbies + Contact Me + + + \ No newline at end of file diff --git a/tests/unit/test_cli.py b/tests/unit/test_cli.py new file mode 100644 index 0000000..dd83807 --- /dev/null +++ b/tests/unit/test_cli.py @@ -0,0 +1,24 @@ +import pytest + +from scrape_movies.presentation import cli, soup_steps + + +def test_main_when_soup_step_is_given_prints_output( + capsys: pytest.CaptureFixture[str], +) -> None: + for step in soup_steps.STEPS: + assert cli.main(["soup", step]) == 0 + assert capsys.readouterr().out.strip(), step + + +def test_main_when_no_command_prints_help( + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main([]) == 0 + assert "usage:" in capsys.readouterr().out + + +def test_render_step_when_title_returns_page_title() -> None: + assert soup_steps.render_step("title") == ( + "soup.title.string -> Angela's Personal Site" + ) diff --git a/tests/unit/test_soup_basics.py b/tests/unit/test_soup_basics.py new file mode 100644 index 0000000..3a3125e --- /dev/null +++ b/tests/unit/test_soup_basics.py @@ -0,0 +1,65 @@ +from bs4 import BeautifulSoup + +from scrape_movies.constants import WEBSITE_HTML_PATH +from scrape_movies.domain.link import Link +from scrape_movies.infrastructure import soup_basics + + +def _soup() -> BeautifulSoup: + return soup_basics.make_soup(soup_basics.load_html(WEBSITE_HTML_PATH)) + + +def test_get_title_when_page_has_title_returns_text() -> None: + assert soup_basics.get_title(_soup()) == "Angela's Personal Site" + + +def test_get_title_when_page_has_no_title_returns_none() -> None: + assert soup_basics.get_title(soup_basics.make_soup("

x

")) is None + + +def test_prettify_html_when_called_puts_tags_on_own_lines() -> None: + pretty = soup_basics.prettify_html(soup_basics.make_soup("

x

")) + assert pretty.splitlines()[:3] == ["

", " ", " x"] + + +def test_find_all_anchors_when_page_has_links_returns_text_and_url() -> None: + links = soup_basics.find_all_anchors(_soup()) + assert links[0] == Link("The App Brewery", "https://www.appbrewery.co/") + assert [link.text for link in links] == [ + "The App Brewery", + "My Hobbies", + "Contact Me", + ] + + +def test_find_all_anchors_when_anchor_has_no_href_skips_it() -> None: + soup = soup_basics.make_soup("no linkx") + assert soup_basics.find_all_anchors(soup) == [Link("x", "/x")] + + +def test_find_text_by_id_when_id_exists_returns_text() -> None: + assert soup_basics.find_text_by_id(_soup(), "name") == "Angela Yu" + + +def test_find_text_by_id_when_id_is_missing_returns_none() -> None: + assert soup_basics.find_text_by_id(_soup(), "nobody") is None + + +def test_find_all_texts_by_class_when_class_matches_returns_texts() -> None: + texts = soup_basics.find_all_texts_by_class(_soup(), "h3", "heading") + assert texts == ["Books and Teaching", "Other Pages"] + + +def test_select_one_text_when_nested_selector_returns_first_match() -> None: + assert soup_basics.select_one_text(_soup(), "p a") == "The App Brewery" + + +def test_select_one_text_when_nothing_matches_returns_none() -> None: + assert soup_basics.select_one_text(_soup(), "table") is None + + +def test_select_texts_when_class_selector_returns_all_matches() -> None: + assert soup_basics.select_texts(_soup(), ".heading") == [ + "Books and Teaching", + "Other Pages", + ] diff --git a/tools/make_screenshots.py b/tools/make_screenshots.py new file mode 100644 index 0000000..990a025 --- /dev/null +++ b/tools/make_screenshots.py @@ -0,0 +1,55 @@ +"""@brief Run the lecture steps and save one terminal screenshot per step. + +Usage: python tools/make_screenshots.py [name-prefix] +The images are written to docs/images/ and used by the README and the wiki. +""" + +import subprocess +import sys +from pathlib import Path + +from render_terminal_png import render + +ROOT = Path(__file__).resolve().parents[1] +IMAGES = ROOT / "docs" / "images" + +## File name (without .png) and the arguments given to `python -m scrape_movies`. +STEPS: dict[str, list[str]] = { + "soup-1-title": ["soup", "title"], + "soup-2-prettify": ["soup", "prettify"], + "soup-3-find-all": ["soup", "find-all"], + "soup-4-find": ["soup", "find"], + "soup-5-select": ["soup", "select"], +} + + +def capture(name: str, args: list[str]) -> Path: + """@brief Run one step and write its screenshot. + + @param name The image name without extension. + @param args The arguments for `python -m scrape_movies`. + @return The path of the PNG. + """ + result = subprocess.run( + [sys.executable, "-m", "scrape_movies", *args], + cwd=ROOT, + capture_output=True, + text=True, + encoding="utf-8", + check=True, + ) + target = IMAGES / f"{name}.png" + render("python -m scrape_movies " + " ".join(args), result.stdout, target) + return target + + +def main() -> None: + """@brief Capture every step whose name starts with the given prefix.""" + prefix = sys.argv[1] if len(sys.argv) > 1 else "" + for name, args in STEPS.items(): + if name.startswith(prefix): + print(capture(name, args)) + + +if __name__ == "__main__": + main() diff --git a/tools/render_terminal_png.py b/tools/render_terminal_png.py new file mode 100644 index 0000000..b11fe6a --- /dev/null +++ b/tools/render_terminal_png.py @@ -0,0 +1,75 @@ +"""@brief Render a command and its output as a terminal-style PNG screenshot. + +The screenshots of the README and the wiki are made with this script, so no +desktop is captured. Needs Pillow (`pip install -e ".[dev]"`). +""" + +from pathlib import Path + +from PIL import Image, ImageDraw, ImageFont + +BACKGROUND = (30, 30, 30) +TITLE_BAR = (50, 50, 50) +TEXT = (212, 212, 212) +PROMPT = (96, 200, 96) +FONT_SIZE = 16 +PADDING = 16 +BAR_HEIGHT = 30 +MAX_COLUMNS = 110 +MAX_LINES = 40 +FONT_FILES = ( + "consola.ttf", + "DejaVuSansMono.ttf", + "LiberationMono-Regular.ttf", + "Menlo.ttc", +) + + +def _load_font() -> ImageFont.ImageFont | ImageFont.FreeTypeFont: + for name in FONT_FILES: + try: + return ImageFont.truetype(name, FONT_SIZE) + except OSError: + continue + return ImageFont.load_default() + + +def _fit(lines: list[str]) -> list[str]: + fitted = [ + line if len(line) <= MAX_COLUMNS else line[: MAX_COLUMNS - 3] + "..." + for line in lines + ] + if len(fitted) > MAX_LINES: + fitted = fitted[:MAX_LINES] + ["..."] + return fitted + + +def render(command: str, output: str, target: Path) -> None: + """@brief Draw the command and its output and save the PNG. + + @param command The command line shown after the prompt. + @param output The text the command printed. + @param target The PNG file to write. + """ + font = _load_font() + lines = _fit(output.rstrip().splitlines()) + prompt = "$ " + columns = max([len(prompt + command), *(len(line) for line in lines)]) + char_width = font.getlength("M") + line_height = FONT_SIZE + 6 + width = int(columns * char_width) + 2 * PADDING + height = BAR_HEIGHT + 2 * PADDING + line_height * (len(lines) + 1) + image = Image.new("RGB", (max(width, 480), height), BACKGROUND) + draw = ImageDraw.Draw(image) + draw.rectangle((0, 0, image.width, BAR_HEIGHT), fill=TITLE_BAR) + for index, color in enumerate(((255, 95, 86), (255, 189, 46), (39, 201, 63))): + x = PADDING + index * 22 + draw.ellipse((x, 9, x + 12, 21), fill=color) + y = BAR_HEIGHT + PADDING + draw.text((PADDING, y), prompt, font=font, fill=PROMPT) + draw.text((PADDING + font.getlength(prompt), y), command, font=font, fill=TEXT) + for line in lines: + y += line_height + draw.text((PADDING, y), line, font=font, fill=TEXT) + target.parent.mkdir(parents=True, exist_ok=True) + image.save(target)