README.md

@@ -104,6 +104,63 @@ python -m scrape_movies soup select
 
 The screenshots are made by `python tools/make_screenshots.py`, which renders the command output as an image; no desktop is captured.
 
+### Lecture: Scraping Hacker News
+
+The code is `src/scrape_movies/infrastructure/hacker_news.py` (parsing) and `http_client.py` (one `requests.get` with a timeout and a `User-Agent`). By default the commands read the course's static copy of the front page, `--source live` reads <https://news.ycombinator.com/news> and `--source fixture` reads the saved copy in `tests/fixtures/` without any network.
+
+**1. Look at the page you will scrape** (live data instead of a local file)
+
+![Step 1: the page](docs/images/hn-1-page.png)
+
+**2. Inspect the markup** (browser developer tools show the tags to look for: a story row `tr.athing`, its title link `.titleline > a`, and a `span.score` in the next row)
+
+```bash
+python -m scrape_movies hacker-news markup
+```
+
+![Step 2: markup](docs/images/hn-2-markup.png)
+
+**3. Titles** (`find_all` on the title links, then `get_text()`)
+
+```bash
+python -m scrape_movies hacker-news titles
+```
+
+![Step 3: titles](docs/images/hn-3-titles.png)
+
+**4. Links** (`get("href")` of the same anchors)
+
+```bash
+python -m scrape_movies hacker-news links
+```
+
+![Step 4: links](docs/images/hn-4-links.png)
+
+**5. Upvotes** (the text `38 points` is split and converted with `int()` so it can be compared)
+
+```bash
+python -m scrape_movies hacker-news upvotes
+```
+
+![Step 5: upvotes](docs/images/hn-5-upvotes.png)
+
+**6. The top story** (Python's `max` over the upvotes)
+
+```bash
+python -m scrape_movies hacker-news top
+```
+
+![Step 6: top story](docs/images/hn-6-top.png)
+
+### Lecture: Is web scraping legal and ethical?
+
+A short summary in our own words; it is not legal advice.
+
+- **Public, non-copyrighted data** is generally fine to read. Court cases such as Genius v. Google and hiQ v. LinkedIn show that the line is not simple.
+- **Copyrighted content** (a video, a blog post) must not be copied, and **data behind a login** is covered by the site's user agreement, so do not scrape it.
+- **Commercial use** of scraped data invites expensive disputes even where the scraping itself was allowed.
+- **Be a good guest:** prefer an official API when there is one, read the site's `robots.txt` to see what it allows, and send at most one request a minute. This project fetches one page per run and identifies itself with a `User-Agent`.
+
 ## 🧪 Tests
 
 ```bash

docs/artifact-registry.md

@@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g.
 | SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 |
 | PP | Project Plan | docs/project-plan.md | 002 |
 | MIL | Milestone / Gateway | docs/milestones/*.md | 006 |
-| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 013 |
+| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 014 |
 
 ## Languages
 

docs/sqa/reviews/rc-013-mil-003-python.md

@@ -0,0 +1,62 @@
+# Review of MIL-003 Python code
+
+## Metadata
+| Key | Value |
+| --- | --- |
+| ID | RC-013 |
+| CrossReference | [MIL-003], [QC-PY-001] |
+
+## Version History
+| Date | Status | Author | Reviewer | Change | Commit |
+| --- | --- | --- | --- | --- | --- |
+| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | [81af6fa] |
+
+---
+
+## Artifact Under Review
+
+- Instance reviewed: [MIL-003]
+- Checklist used: [QC-PY-001]
+- Scope: full review. Code reviewed: Story, errors, ports, use_cases, http_client, hacker_news, hacker_news_steps, capture_page, tests/unit.
+- Language and domain: n/a (technical type)
+- Language reviewer: none
+- Mode: autonomous, not independent (reviewer is `AI as stand-in for S01`)
+
+## Checklist Results
+
+| # | Criterion | Status | Evidence/Notes |
+| --- | --- | --- | --- |
+| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE. |
+| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | names state purpose; the lecture names (`find_all`, `select_one`) are kept in the helper names. |
+| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | `ruff check` and `ruff format --check` are clean with no `noqa`. |
+| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | every signature is annotated; `mypy --strict` passes, also in a venv without the third-party packages. |
+| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | Pass | `fetch_html` catches only `requests.RequestException` and raises `FetchError` `from` the cause; a test checks `__cause__`. |
+| 6 | No mutable default arguments and no shadowed builtins | Pass | no mutable defaults, no shadowed builtins. |
+| 7 | Files, locks and connections are managed with context managers | Pass | files are read with `Path.read_text`; the HTTP call is a single `requests.get` with no open session. |
+| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | every public module, class and function has a Doxygen docstring saying what it does. |
+| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | N-A | no logging; the command-line output is the product, written with `print` in the presentation layer. |
+| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | no Design Class Diagram; all tasks are plain technical tasks (Needs Use Case: No). |
+| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | tests are named for behaviour, offline and independent of order. |
+| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | `mypy --strict` runs over `src`, `tests` and `tools`. |
+| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Fail | dependencies are declared with lower bounds, not pinned (see RC-009 action item); none is unused. |
+| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | each constant and format is defined once; the step texts are built in one place. |
+| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | `domain` imports nothing from other layers; `infrastructure` imports `domain` and `constants`; `presentation` composes them. |
+| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | no class hierarchy; use cases take callables (ports), so a new source needs no edit of the use case. |
+| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | code is in `domain`, `application`, `infrastructure`, `presentation`; tests are in `tests/unit` (network calls are faked; integration and e2e folders stay empty until needed). |
+| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | all names are English lowercase with `_`, no space. |
+
+## Overall Verdict
+
+Go — no Mandatory criterion fails and none is Pending. Stand-in review in autonomous mode, not independent; S01 challenges it in the run summary.
+
+## Action Items
+
+| Action | Owner | Due |
+| --- | --- | --- |
+| Pin dependencies or accept lower bounds (open from RC-009, optional criterion 13) | S01 | 2026-11-14 |
+
+---
+
+[MIL-003]: ../../milestones/mil-003-hacker-news.md
+[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md
+[81af6fa]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/commit/81af6fa2e9675b4a27740caca50b7b4056688399

pyproject.toml

@@ -38,7 +38,8 @@ extend-exclude = ["framework", ".agents", ".claude"]
 
 [tool.ruff.lint]
 select = ["E", "F", "I", "N", "UP", "B", "D"]
-ignore = ["D203", "D213"]
+# UP040 off: mypy 1.11 (the pinned CI version) does not support PEP 695 aliases.
+ignore = ["D203", "D213", "UP040"]
 
 [tool.ruff.lint.per-file-ignores]
 "tests/**" = ["D"]

src/scrape_movies/application/ports.py

@@ -0,0 +1,11 @@
+"""@brief The ports (interfaces) the use cases need from the outside world."""
+
+from collections.abc import Callable
+from typing import TypeAlias
+
+from scrape_movies.domain.story import Story
+
+## Fetches the HTML text of a page address.
+HtmlFetcher: TypeAlias = Callable[[str], str]
+## Turns Hacker News HTML into stories.
+StoryParser: TypeAlias = Callable[[str], list[Story]]

src/scrape_movies/application/use_cases.py

@@ -0,0 +1,26 @@
+"""@brief Use cases of the scraping exercises."""
+
+from scrape_movies.application.ports import HtmlFetcher, StoryParser
+from scrape_movies.domain.story import Story, top_story
+
+
+def list_stories(fetch: HtmlFetcher, parse: StoryParser, url: str) -> list[Story]:
+    """@brief Fetch a Hacker News page and read its stories.
+
+    @param fetch Fetches the HTML of the page.
+    @param parse Reads the stories from the HTML.
+    @param url The page address.
+    @return The stories in page order.
+    """
+    return parse(fetch(url))
+
+
+def find_top_story(fetch: HtmlFetcher, parse: StoryParser, url: str) -> Story | None:
+    """@brief Find the story with the most upvotes on a Hacker News page.
+
+    @param fetch Fetches the HTML of the page.
+    @param parse Reads the stories from the HTML.
+    @param url The page address.
+    @return The top story, or None when the page has no stories.
+    """
+    return top_story(list_stories(fetch, parse, url))

src/scrape_movies/constants.py

@@ -32,3 +32,6 @@ EMPIRE_LIVE_URL: str = "https://www.empireonline.com/movies/features/best-movies
 
 ## File the movie list is written to.
 MOVIES_OUTPUT_FILE: str = "movies.txt"
+
+## The saved copy of the course's Hacker News page, used offline.
+HACKER_NEWS_FIXTURE_PATH: Path = FIXTURES_DIR / "hacker_news.html"

src/scrape_movies/domain/errors.py

@@ -0,0 +1,5 @@
+"""@brief Errors the project raises."""
+
+
+class FetchError(Exception):
+    """@brief A web page could not be fetched."""

src/scrape_movies/domain/story.py

@@ -0,0 +1,31 @@
+"""@brief The Story entity (a Hacker News article) and the top-story rule."""
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class Story:
+    """@brief An article on the Hacker News front page.
+
+    @param title The headline.
+    @param link The address the headline points to.
+    @param upvotes The number of points; 0 when the story shows none.
+    """
+
+    title: str
+    link: str
+    upvotes: int
+
+
+def top_story(stories: Sequence[Story]) -> Story | None:
+    """@brief Find the story with the most upvotes.
+
+    @param stories The stories in page order.
+    @return The first story with the highest score, or None when there are none.
+    """
+    best: Story | None = None
+    for story in stories:
+        if best is None or story.upvotes > best.upvotes:
+            best = story
+    return best

src/scrape_movies/infrastructure/hacker_news.py

@@ -0,0 +1,81 @@
+"""@brief Reading titles, links and upvotes from a Hacker News page."""
+
+from bs4 import BeautifulSoup, Tag
+
+from scrape_movies.domain.story import Story
+from scrape_movies.infrastructure.soup_basics import make_soup
+
+STORY_ROW = "tr.athing"
+TITLE_LINK = ".titleline > a"
+SCORE = ".score"
+
+
+def _upvotes(row: Tag) -> int:
+    """@brief Read the score shown in the row after a story, 0 when it has none."""
+    following = row.find_next_sibling("tr")
+    score = following.select_one(SCORE) if isinstance(following, Tag) else None
+    if score is None:
+        return 0
+    return int(score.get_text().split()[0])
+
+
+def _story_rows(soup: BeautifulSoup) -> list[Tag]:
+    return [row for row in soup.select(STORY_ROW) if row.select_one(TITLE_LINK)]
+
+
+def parse_stories(html: str) -> list[Story]:
+    """@brief Read every story of a front page.
+
+    @param html The HTML of the page.
+    @return The stories in page order.
+    """
+    stories: list[Story] = []
+    for row in _story_rows(make_soup(html)):
+        anchor = row.select_one(TITLE_LINK)
+        href = anchor.get("href") if anchor is not None else None
+        if anchor is not None and isinstance(href, str):
+            stories.append(
+                Story(title=anchor.get_text(), link=href, upvotes=_upvotes(row))
+            )
+    return stories
+
+
+def get_titles(html: str) -> list[str]:
+    """@brief Return the headlines of a front page.
+
+    @param html The HTML of the page.
+    @return The titles in page order.
+    """
+    return [story.title for story in parse_stories(html)]
+
+
+def get_links(html: str) -> list[str]:
+    """@brief Return the addresses the headlines point to.
+
+    @param html The HTML of the page.
+    @return The links in page order.
+    """
+    return [story.link for story in parse_stories(html)]
+
+
+def get_upvotes(html: str) -> list[int]:
+    """@brief Return the scores as numbers, not text such as "38 points".
+
+    @param html The HTML of the page.
+    @return The upvotes in page order.
+    """
+    return [story.upvotes for story in parse_stories(html)]
+
+
+def first_story_markup(html: str) -> str:
+    """@brief Show the markup of the first story, as seen with "Inspect" in a browser.
+
+    @param html The HTML of the page.
+    @return The indented HTML of its title row and score row, or "" when none.
+    """
+    rows = _story_rows(make_soup(html))
+    if not rows:
+        return ""
+    following = rows[0].find_next_sibling("tr")
+    parts = [rows[0], following] if isinstance(following, Tag) else [rows[0]]
+    return "\n".join(str(part.prettify()) for part in parts)

src/scrape_movies/infrastructure/http_client.py

@@ -0,0 +1,28 @@
+"""@brief Fetching web pages with Requests."""
+
+import requests
+
+from scrape_movies.constants import REQUEST_TIMEOUT_SECONDS, USER_AGENT
+from scrape_movies.domain.errors import FetchError
+
+
+def fetch_html(url: str) -> str:
+    """@brief Download a page with one GET request.
+
+    Sends a User-Agent and gives up after a timeout. Call it once per run:
+    scraping more often than once a minute is not polite.
+
+    @param url The address of the page.
+    @return The HTML text of the response.
+    @throws FetchError When the request fails or the server answers with an error.
+    """
+    try:
+        response = requests.get(
+            url,
+            headers={"User-Agent": USER_AGENT},
+            timeout=REQUEST_TIMEOUT_SECONDS,
+        )
+        response.raise_for_status()
+    except requests.RequestException as error:
+        raise FetchError(f"Could not fetch {url}: {error}") from error
+    return str(response.text)

src/scrape_movies/presentation/cli.py

@@ -1,36 +1,47 @@
-"""@brief Command-line entry point that runs the lecture steps."""
-
-import argparse
-from collections.abc import Sequence
-
-from scrape_movies.presentation import soup_steps
-
-
-def build_parser() -> argparse.ArgumentParser:
-    """@brief Build the argument parser.
-
-    @return The parser with one sub-command per lecture.
-    """
-    parser = argparse.ArgumentParser(
-        prog="python -m scrape_movies",
-        description="Day 45 of 100 Days of Code: web scraping exercises.",
-    )
-    commands = parser.add_subparsers(dest="command")
-    soup = commands.add_parser("soup", help="parse the sample page website.html")
-    soup.add_argument("step", choices=list(soup_steps.STEPS))
-    return parser
-
-
-def main(argv: Sequence[str] | None = None) -> int:
-    """@brief Run the command given on the command line.
-
-    @param argv The arguments, or None to read `sys.argv`.
-    @return The process exit code.
-    """
-    parser = build_parser()
-    args = parser.parse_args(argv)
-    if args.command == "soup":
-        print(soup_steps.render_step(args.step))
-        return 0
-    parser.print_help()
-    return 0
+"""@brief Command-line entry point that runs the lecture steps."""
+
+import argparse
+from collections.abc import Sequence
+
+from scrape_movies.presentation import hacker_news_steps, soup_steps
+
+
+def build_parser() -> argparse.ArgumentParser:
+    """@brief Build the argument parser.
+
+    @return The parser with one sub-command per lecture.
+    """
+    parser = argparse.ArgumentParser(
+        prog="python -m scrape_movies",
+        description="Day 45 of 100 Days of Code: web scraping exercises.",
+    )
+    commands = parser.add_subparsers(dest="command")
+    soup = commands.add_parser("soup", help="parse the sample page website.html")
+    soup.add_argument("step", choices=list(soup_steps.STEPS))
+    news = commands.add_parser("hacker-news", help="scrape the Hacker News front page")
+    news.add_argument("step", choices=hacker_news_steps.STEPS)
+    news.add_argument(
+        "--source",
+        choices=hacker_news_steps.SOURCES,
+        default="mirror",
+        help="the course mirror (default), the live site or the saved copy",
+    )
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    """@brief Run the command given on the command line.
+
+    @param argv The arguments, or None to read `sys.argv`.
+    @return The process exit code.
+    """
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    if args.command == "soup":
+        print(soup_steps.render_step(args.step))
+        return 0
+    if args.command == "hacker-news":
+        print(hacker_news_steps.render_step(args.step, args.source))
+        return 0
+    parser.print_help()
+    return 0

src/scrape_movies/presentation/hacker_news_steps.py

@@ -0,0 +1,62 @@
+"""@brief Text output of each "Scraping Hacker News" lecture step."""
+
+from scrape_movies.application import use_cases
+from scrape_movies.application.ports import HtmlFetcher
+from scrape_movies.constants import (
+    HACKER_NEWS_FIXTURE_PATH,
+    HACKER_NEWS_LIVE_URL,
+    HACKER_NEWS_URL,
+)
+from scrape_movies.infrastructure import hacker_news, http_client, soup_basics
+
+## Where a step reads the page: the course mirror, the live site or the saved copy.
+SOURCES: tuple[str, ...] = ("mirror", "live", "fixture")
+_URLS: dict[str, str] = {
+    "mirror": HACKER_NEWS_URL,
+    "live": HACKER_NEWS_LIVE_URL,
+    "fixture": str(HACKER_NEWS_FIXTURE_PATH),
+}
+
+
+def _fetcher(source: str) -> HtmlFetcher:
+    if source == "fixture":
+        return lambda path: soup_basics.load_html(HACKER_NEWS_FIXTURE_PATH)
+    return http_client.fetch_html
+
+
+def _numbered(lines: list[str]) -> str:
+    return "\n".join(f"{number}. {line}" for number, line in enumerate(lines, 1))
+
+
+def _run(source: str, step: str) -> str:
+    fetch, url = _fetcher(source), _URLS[source]
+    if step == "markup":
+        return hacker_news.first_story_markup(fetch(url)).rstrip()
+    stories = use_cases.list_stories(fetch, hacker_news.parse_stories, url)
+    if step == "titles":
+        return _numbered([story.title for story in stories])
+    if step == "links":
+        return _numbered([story.link for story in stories])
+    if step == "upvotes":
+        return _numbered([str(story.upvotes) for story in stories])
+    top = use_cases.find_top_story(fetch, hacker_news.parse_stories, url)
+    if top is None:
+        return "No stories found."
+    return f"{top.title}\n{top.link}\n{top.upvotes} points"
+
+
+## The lecture steps by command-line name, in lecture order.
+STEPS: tuple[str, ...] = ("markup", "titles", "links", "upvotes", "top")
+
+
+def render_step(step: str, source: str) -> str:
+    """@brief Run one lecture step.
+
+    @param step One of STEPS.
+    @param source One of SOURCES.
+    @return The text the step prints.
+    @throws KeyError When the step or the source is unknown.
+    """
+    if step not in STEPS:
+        raise KeyError(step)
+    return _run(source, step)

tests/unit/test_cli.py

@@ -1,24 +1,41 @@
-import pytest
-
-from scrape_movies.presentation import cli, soup_steps
-
-
-def test_main_when_soup_step_is_given_prints_output(
-    capsys: pytest.CaptureFixture[str],
-) -> None:
-    for step in soup_steps.STEPS:
-        assert cli.main(["soup", step]) == 0
-        assert capsys.readouterr().out.strip(), step
-
-
-def test_main_when_no_command_prints_help(
-    capsys: pytest.CaptureFixture[str],
-) -> None:
-    assert cli.main([]) == 0
-    assert "usage:" in capsys.readouterr().out
-
-
-def test_render_step_when_title_returns_page_title() -> None:
-    assert soup_steps.render_step("title") == (
-        "soup.title.string -> Angela's Personal Site"
-    )
+import pytest
+
+from scrape_movies.presentation import cli, hacker_news_steps, soup_steps
+
+
+def test_main_when_soup_step_is_given_prints_output(
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    for step in soup_steps.STEPS:
+        assert cli.main(["soup", step]) == 0
+        assert capsys.readouterr().out.strip(), step
+
+
+def test_main_when_no_command_prints_help(
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    assert cli.main([]) == 0
+    assert "usage:" in capsys.readouterr().out
+
+
+def test_render_step_when_title_returns_page_title() -> None:
+    assert soup_steps.render_step("title") == (
+        "soup.title.string -> Angela's Personal Site"
+    )
+
+
+def test_main_when_hacker_news_top_from_fixture_prints_top_story(
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    assert cli.main(["hacker-news", "top", "--source", "fixture"]) == 0
+    out = capsys.readouterr().out
+    assert "Mozilla lays off 250 employees" in out
+    assert "1312 points" in out
+
+
+def test_main_when_every_hacker_news_step_runs_from_fixture_prints_output(
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    for step in hacker_news_steps.STEPS:
+        assert cli.main(["hacker-news", step, "--source", "fixture"]) == 0
+        assert capsys.readouterr().out.strip(), step

tests/unit/test_hacker_news.py

@@ -0,0 +1,59 @@
+from scrape_movies.constants import HACKER_NEWS_FIXTURE_PATH
+from scrape_movies.domain.story import Story
+from scrape_movies.infrastructure import hacker_news
+from scrape_movies.infrastructure.soup_basics import load_html
+
+PAGE = """
+<table><tr class="athing" id="1"><td><span class="titleline">
+<a href="https://example.com/one">One</a></span></td></tr>
+<tr><td><span class="score">12 points</span></td></tr>
+<tr class="athing" id="2"><td><span class="titleline">
+<a href="https://example.com/job">A job ad</a></span></td></tr>
+<tr><td>no score here</td></tr></table>
+"""
+
+
+def _fixture() -> str:
+    return load_html(HACKER_NEWS_FIXTURE_PATH)
+
+
+def test_parse_stories_when_page_has_score_reads_integer() -> None:
+    assert hacker_news.parse_stories(PAGE)[0] == Story(
+        "One", "https://example.com/one", 12
+    )
+
+
+def test_parse_stories_when_story_has_no_score_uses_zero() -> None:
+    assert hacker_news.parse_stories(PAGE)[1].upvotes == 0
+
+
+def test_parse_stories_when_page_has_no_stories_returns_empty() -> None:
+    assert hacker_news.parse_stories("<p>nothing</p>") == []
+
+
+def test_parse_stories_when_fixture_returns_thirty_stories() -> None:
+    assert len(hacker_news.parse_stories(_fixture())) == 30
+
+
+def test_get_titles_when_fixture_starts_with_first_headline() -> None:
+    assert hacker_news.get_titles(_fixture())[0] == "Joan Feynman 1927-2020"
+
+
+def test_get_links_when_fixture_returns_one_address_per_story() -> None:
+    links = hacker_news.get_links(_fixture())
+    assert len(links) == 30
+    assert links[0].startswith("https://www.aps.org/")
+
+
+def test_get_upvotes_when_fixture_returns_integers() -> None:
+    upvotes = hacker_news.get_upvotes(_fixture())
+    assert upvotes[:2] == [38, 205]
+    assert all(isinstance(points, int) for points in upvotes)
+
+
+def test_first_story_markup_when_page_has_story_shows_title_row() -> None:
+    assert 'class="titleline"' in hacker_news.first_story_markup(PAGE)
+
+
+def test_first_story_markup_when_page_has_no_story_returns_empty() -> None:
+    assert hacker_news.first_story_markup("<p>x</p>") == ""

tests/unit/test_http_client.py

@@ -0,0 +1,59 @@
+from typing import Any
+
+import pytest
+import requests
+
+from scrape_movies.constants import REQUEST_TIMEOUT_SECONDS, USER_AGENT
+from scrape_movies.domain.errors import FetchError
+from scrape_movies.infrastructure import http_client
+
+
+class _Response:
+    def __init__(self, text: str, error: Exception | None = None) -> None:
+        self.text = text
+        self._error = error
+
+    def raise_for_status(self) -> None:
+        if self._error is not None:
+            raise self._error
+
+
+def test_fetch_html_when_request_succeeds_sends_user_agent_and_timeout(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    calls: list[dict[str, Any]] = []
+
+    def fake_get(url: str, **kwargs: Any) -> _Response:
+        calls.append({"url": url, **kwargs})
+        return _Response("<html></html>")
+
+    monkeypatch.setattr(requests, "get", fake_get)
+    assert http_client.fetch_html("http://example.test") == "<html></html>"
+    assert calls == [
+        {
+            "url": "http://example.test",
+            "headers": {"User-Agent": USER_AGENT},
+            "timeout": REQUEST_TIMEOUT_SECONDS,
+        }
+    ]
+
+
+def test_fetch_html_when_server_errors_raises_fetch_error(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    error = requests.HTTPError("500")
+    monkeypatch.setattr(requests, "get", lambda *a, **k: _Response("", error))
+    with pytest.raises(FetchError) as caught:
+        http_client.fetch_html("http://example.test")
+    assert caught.value.__cause__ is error
+
+
+def test_fetch_html_when_connection_fails_raises_fetch_error(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    def fail(*args: Any, **kwargs: Any) -> _Response:
+        raise requests.ConnectionError("down")
+
+    monkeypatch.setattr(requests, "get", fail)
+    with pytest.raises(FetchError):
+        http_client.fetch_html("http://example.test")

tests/unit/test_story.py

@@ -0,0 +1,15 @@
+from scrape_movies.domain.story import Story, top_story
+
+
+def test_top_story_when_scores_differ_returns_highest() -> None:
+    stories = [Story("a", "/a", 5), Story("b", "/b", 50), Story("c", "/c", 7)]
+    assert top_story(stories) == Story("b", "/b", 50)
+
+
+def test_top_story_when_scores_tie_returns_first() -> None:
+    stories = [Story("a", "/a", 9), Story("b", "/b", 9)]
+    assert top_story(stories) == Story("a", "/a", 9)
+
+
+def test_top_story_when_no_stories_returns_none() -> None:
+    assert top_story([]) is None

tests/unit/test_use_cases.py

@@ -0,0 +1,24 @@
+from scrape_movies.application import use_cases
+from scrape_movies.domain.story import Story
+
+
+def _fetch(url: str) -> str:
+    return f"html of {url}"
+
+
+def _parse(html: str) -> list[Story]:
+    return [Story(html, "/x", 3), Story("best", "/y", 99)]
+
+
+def test_list_stories_when_called_passes_page_through_parser() -> None:
+    stories = use_cases.list_stories(_fetch, _parse, "http://x")
+    assert stories[0].title == "html of http://x"
+
+
+def test_find_top_story_when_stories_exist_returns_highest() -> None:
+    top = use_cases.find_top_story(_fetch, _parse, "http://x")
+    assert top == Story("best", "/y", 99)
+
+
+def test_find_top_story_when_page_is_empty_returns_none() -> None:
+    assert use_cases.find_top_story(_fetch, lambda html: [], "http://x") is None

tools/capture_page.py

@@ -0,0 +1,56 @@
+"""@brief Save a screenshot of a web page with a headless Chromium browser.
+
+A headless browser draws into memory, so no desktop window is opened and no
+mouse or keyboard is used. It needs Edge, Chrome or Chromium on the machine.
+"""
+
+import shutil
+import subprocess
+from pathlib import Path
+
+CANDIDATES = (
+    r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe",
+    r"C:\Program Files\Microsoft\Edge\Application\msedge.exe",
+    r"C:\Program Files\Google\Chrome\Application\chrome.exe",
+    "google-chrome",
+    "chromium",
+    "chromium-browser",
+)
+WINDOW = "1280,1000"
+
+
+def find_browser() -> str:
+    """@brief Locate a Chromium-based browser.
+
+    @return The path or command name of the browser.
+    @throws FileNotFoundError When none is installed.
+    """
+    for candidate in CANDIDATES:
+        if Path(candidate).exists() or shutil.which(candidate):
+            return candidate
+    raise FileNotFoundError("Install Edge, Chrome or Chromium to capture pages.")
+
+
+def capture_page(url: str, target: Path) -> Path:
+    """@brief Render a page and save it as a PNG.
+
+    @param url The page address.
+    @param target The PNG file to write.
+    @return The path of the PNG.
+    """
+    target.parent.mkdir(parents=True, exist_ok=True)
+    subprocess.run(
+        [
+            find_browser(),
+            "--headless=new",
+            "--disable-gpu",
+            "--hide-scrollbars",
+            f"--window-size={WINDOW}",
+            f"--screenshot={target}",
+            url,
+        ],
+        check=True,
+        capture_output=True,
+        timeout=120,
+    )
+    return target

tools/make_screenshots.py

@@ -1,55 +1,72 @@
-"""@brief Run the lecture steps and save one terminal screenshot per step.
-
-Usage: python tools/make_screenshots.py [name-prefix]
-The images are written to docs/images/ and used by the README and the wiki.
-"""
-
-import subprocess
-import sys
-from pathlib import Path
-
-from render_terminal_png import render
-
-ROOT = Path(__file__).resolve().parents[1]
-IMAGES = ROOT / "docs" / "images"
-
-## File name (without .png) and the arguments given to `python -m scrape_movies`.
-STEPS: dict[str, list[str]] = {
-    "soup-1-title": ["soup", "title"],
-    "soup-2-prettify": ["soup", "prettify"],
-    "soup-3-find-all": ["soup", "find-all"],
-    "soup-4-find": ["soup", "find"],
-    "soup-5-select": ["soup", "select"],
-}
-
-
-def capture(name: str, args: list[str]) -> Path:
-    """@brief Run one step and write its screenshot.
-
-    @param name The image name without extension.
-    @param args The arguments for `python -m scrape_movies`.
-    @return The path of the PNG.
-    """
-    result = subprocess.run(
-        [sys.executable, "-m", "scrape_movies", *args],
-        cwd=ROOT,
-        capture_output=True,
-        text=True,
-        encoding="utf-8",
-        check=True,
-    )
-    target = IMAGES / f"{name}.png"
-    render("python -m scrape_movies " + " ".join(args), result.stdout, target)
-    return target
-
-
-def main() -> None:
-    """@brief Capture every step whose name starts with the given prefix."""
-    prefix = sys.argv[1] if len(sys.argv) > 1 else ""
-    for name, args in STEPS.items():
-        if name.startswith(prefix):
-            print(capture(name, args))
-
-
-if __name__ == "__main__":
-    main()
+"""@brief Run the lecture steps and save one terminal screenshot per step.
+
+Usage: python tools/make_screenshots.py [name-prefix]
+The images are written to docs/images/ and used by the README and the wiki.
+"""
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+from capture_page import capture_page
+from render_terminal_png import render
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
+
+ROOT = Path(__file__).resolve().parents[1]
+IMAGES = ROOT / "docs" / "images"
+
+## File name (without .png) and the arguments given to `python -m scrape_movies`.
+STEPS: dict[str, list[str]] = {
+    "soup-1-title": ["soup", "title"],
+    "soup-2-prettify": ["soup", "prettify"],
+    "soup-3-find-all": ["soup", "find-all"],
+    "soup-4-find": ["soup", "find"],
+    "soup-5-select": ["soup", "select"],
+    "hn-2-markup": ["hacker-news", "markup"],
+    "hn-3-titles": ["hacker-news", "titles"],
+    "hn-4-links": ["hacker-news", "links"],
+    "hn-5-upvotes": ["hacker-news", "upvotes"],
+    "hn-6-top": ["hacker-news", "top"],
+}
+## Screenshots of real web pages: image name and the address of the page.
+PAGES: dict[str, str] = {
+    "hn-1-page": "https://appbrewery.github.io/news.ycombinator.com/",
+}
+
+
+def capture(name: str, args: list[str]) -> Path:
+    """@brief Run one step and write its screenshot.
+
+    @param name The image name without extension.
+    @param args The arguments for `python -m scrape_movies`.
+    @return The path of the PNG.
+    """
+    result = subprocess.run(
+        [sys.executable, "-m", "scrape_movies", *args],
+        cwd=ROOT,
+        env={**os.environ, "PYTHONIOENCODING": "utf-8"},
+        capture_output=True,
+        text=True,
+        encoding="utf-8",
+        check=True,
+    )
+    target = IMAGES / f"{name}.png"
+    render("python -m scrape_movies " + " ".join(args), result.stdout, target)
+    return target
+
+
+def main() -> None:
+    """@brief Capture every step whose name starts with the given prefix."""
+    prefix = sys.argv[1] if len(sys.argv) > 1 else ""
+    for name, args in STEPS.items():
+        if name.startswith(prefix):
+            print(capture(name, args))
+    for name, url in PAGES.items():
+        if name.startswith(prefix):
+            print(capture_page(url, IMAGES / f"{name}.png"))
+
+
+if __name__ == "__main__":
+    main()