2026-10-28 07:33:37 +01:00
27 changed files with 775 additions and 85 deletions
@@ -161,6 +161,52 @@ A short summary in our own words; it is not legal advice.
 - **Commercial use** of scraped data invites expensive disputes even where the scraping itself was allowed.
 - **Be a good guest:** prefer an official API when there is one, read the site's `robots.txt` to see what it allows, and send at most one request a minute. This project fetches one page per run and identifies itself with a `User-Agent`.
 ### Project: 100 movies to watch
 The assignment: scrape Empire's "100 Greatest Movies" and write `movies.txt` with the titles numbered from `1)` upwards. The code is `infrastructure/empire_movies.py` (parsing), `domain/movie.py` (reverse and number) and `infrastructure/movie_file.py` (writing). `--source wayback` (default) reads the Internet Archive snapshot of May 2020 that the assignment pins, `--source live` reads the current Empire page and `--source fixture` reads the saved headings in `tests/fixtures/`.
 **1. Choose the page** (the saved headings of the page)
 ![Step 1: the page](docs/images/movies-1-page.png)
 **2. Inspect the markup** (the 2020 page has `<h3 class="title">`; the current page has `<h2><strong>` headings that start with a rank)
 ```bash
 python -m scrape_movies movies markup
 ```
 ![Step 2: markup](docs/images/movies-2-markup.png)
 **3. Titles in page order** (`select("h3.title")` and a list comprehension over `get_text()`; the page counts down from 100)
 ```bash
 python -m scrape_movies movies titles
 ```
 ![Step 3: titles](docs/images/movies-3-titles.png)
 **4. Reverse the list** (`titles[::-1]`, so the list counts up from 1)
 ```bash
 python -m scrape_movies movies reversed
 ```
 ![Step 4: reversed](docs/images/movies-4-reversed.png)
 **5. Write `movies.txt`** (a `with open(...)` block, one title per line)
 ```bash
 python -m scrape_movies movies write
 ```
 ![Step 5: write](docs/images/movies-5-write.png)
 **6. The result**
 ![Step 6: movies.txt](docs/images/movies-6-file.png)
 The committed [`movies.txt`](./movies.txt) was produced with `--source fixture` on 2026-10-28 from the current Empire list, so its ranking differs from the 2020 list in the lecture video. The Internet Archive answered `429 Too Many Requests` while this project was built, so the snapshot is read live only; its `h3.title` layout is covered by a hand-written sample in the tests.
 ## 🧪 Tests
 ```bash
@@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g.
 | SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 |
 | PP | Project Plan | docs/project-plan.md | 002 |
 | MIL | Milestone / Gateway | docs/milestones/*.md | 006 |
-| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 014 |
+| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 015 |
 ## Languages
@@ -0,0 +1,63 @@
 # Review of MIL-004 Python code
 ## Metadata
 | Key | Value |
 | --- | --- |
 | ID | RC-014 |
 | CrossReference | [MIL-004], [QC-PY-001] |
 ## Version History
 | Date | Status | Author | Reviewer | Change | Commit |
 | --- | --- | --- | --- | --- | --- |
 | 2026-10-28 | Proposed | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | [02c339c] |
 ---
 ## Artifact Under Review
 - Instance reviewed: [MIL-004]
 - Checklist used: [QC-PY-001]
 - Scope: full review. Code reviewed: movie, empire_movies, movie_file, use_cases, movies_steps, cli, tools/, tests/unit and tests/integration.
 - Language and domain: n/a (technical type)
 - Language reviewer: none
 - Mode: autonomous, not independent (reviewer is `AI as stand-in for S01`)
 ## Checklist Results
 | # | Criterion | Status | Evidence/Notes |
 | --- | --- | --- | --- |
 | 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE. |
 | 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | names state purpose; the lecture names (`find_all`, `select_one`) are kept in the helper names. |
 | 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | `ruff check` and `ruff format --check` are clean with no `noqa`. |
 | 4 | Every function and method signature is type-annotated, including `-> None` | Pass | every signature is annotated; `mypy --strict` passes, also in a venv without the third-party packages. |
 | 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | Pass | fetch errors are raised as `FetchError` from the cause (MIL-003); no exception is swallowed. |
 | 6 | No mutable default arguments and no shadowed builtins | Pass | no mutable defaults, no shadowed builtins. |
 | 7 | Files, locks and connections are managed with context managers | Pass | `write_movies` writes inside a `with path.open(...)` block. |
 | 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | every public module, class and function has a Doxygen docstring saying what it does. |
 | 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | N-A | no logging; the command-line output is the product, written with `print` in the presentation layer. |
 | 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | no Design Class Diagram; all tasks are plain technical tasks (Needs Use Case: No). |
 | 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | tests are named for behaviour, offline and independent of order. |
 | 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | `mypy --strict` runs over `src`, `tests` and `tools`. |
 | 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Fail | dependencies are declared with lower bounds, not pinned (see RC-009 action item); none is unused. |
 | 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | each constant and format is defined once; the step texts are built in one place. |
 | 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | `domain` imports nothing from other layers; `infrastructure` imports `domain` and `constants`; `presentation` composes them. |
 | 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | use cases take callables (ports); a new page layout needs a new parser, not an edit of the use case. |
 | 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | code is in `domain`, `application`, `infrastructure`, `presentation`; tests in `tests/unit` and `tests/integration` (the CLI-to-file flow); `e2e` stays empty because the CLI is covered by the integration test. |
 | 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | all names are English lowercase with `_`, no space. |
 ## Overall Verdict
 Go — no Mandatory criterion fails and none is Pending. Stand-in review in autonomous mode, not independent; S01 challenges it in the run summary.
 ## Action Items
 | Action | Owner | Due |
 | --- | --- | --- |
 | Pin dependencies or accept lower bounds (open from RC-009, optional criterion 13) | S01 | 2026-11-14 |
 | Refresh the Wayback fixture when archive.org stops answering 429 (the snapshot layout is covered by a hand-written sample) | S01 | 2026-11-14 |
 ---
 [MIL-004]: ../../milestones/mil-004-top-100-movies.md
 [QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md
 [02c339c]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/commit/02c339c2dbe95b0e2eff5e4a3738f6e946014d26
@@ -0,0 +1,100 @@
 1) Jaws (1975)
 2) The Godfather (1972)
 3) The Shawshank Redemption (1994)
 4) Star Wars: The Empire Strikes Back (1980)
 5) Aliens (1986)
 6) Inception (2010)
 7) Goodfellas (1990)
 8) Star Wars: Episode IV – A New Hope (1977)
 9) The Lord Of The Rings: The Fellowship Of The Ring (2001)
 10) The Dark Knight (2008)
 11) Pulp Fiction (1994)
 12) Back To The Future (1985)
 13) Jurassic Park (1993)
 14) Raiders Of The Lost Ark (1981)
 15) Blade Runner (1982)
 16) The Godfather Part II (1974)
 17) Alien (1979)
 18) Terminator 2: Judgment Day (1991)
 19) Interstellar (2014)
 20) Heat (1995)
 21) The Lord Of The Rings: The Return Of The King (2003)
 22) The Matrix (1999)
 23) Die Hard (1988)
 24) There Will Be Blood (2007)
 25) 2001: A Space Odyssey (1968)
 26) Gladiator (2000)
 27) Parasite (2019)
 28) Apocalypse Now (1979)
 29) 12 Angry Men (1957)
 30) Casablanca (1942)
 31) The Thing (1982)
 32) Seven (1995)
 33) The Lord Of The Rings: The Two Towers (2002)
 34) The Shining (1980)
 35) Mad Max: Fury Road (2015)
 36) Point Break (1991)
 37) Fight Club (1999)
 38) The Good, The Bad And The Ugly (1966)
 39) Lawrence Of Arabia (1962)
 40) Mulholland Drive (2001)
 41) Schindler's List (1993)
 42) Saving Private Ryan (1998)
 43) La La Land (2016)
 44) Taxi Driver (1976)
 45) Past Lives (2023)
 46) Whiplash (2014)
 47) It's A Wonderful Life (1946)
 48) Avengers: Infinity War (2018)
 49) Psycho (1960)
 50) No Country For Old Men (2007)
 51) Rear Window (1954)
 52) Citizen Kane (1941)
 53) Vertigo (1958)
 54) Get Out (2017)
 55) Hot Fuzz (2007)
 56) Spirited Away (2001)
 57) Seven Samurai (1954)
 58) Titanic (1997)
 59) The Exorcist (1973)
 60) Fargo (1996)
 61) The Social Network (2010)
 62) Eternal Sunshine Of The Spotless Mind (2004)
 63) The Big Lebowski (1998)
 64) Oppenheimer (2023)
 65) Predator (1987)
 66) Indiana Jones And The Last Crusade (1989)
 67) Do The Right Thing (1989)
 68) Avengers: Endgame (2019)
 69) The Babadook (2014)
 70) Children Of Men (2006)
 71) Arrival (2016)
 72) Magnolia (1999)
 73) The Lion King (1994)
 74) Trainspotting (1996)
 75) The Silence Of The Lambs (1991)
 76) Once Upon A Time In Hollywood (2019)
 77) In The Mood For Love (2000)
 78) Ghostbusters (1984)
 79) Paddington 2 (2017)
 80) Spider-Man: Into The Spider-Verse (2018)
 81) Everything Everywhere All At Once (2022)
 82) Brokeback Mountain (2005)
 83) Inglourious Basterds (2009)
 84) Pan's Labyrinth (2006)
 85) Rocky (1976)
 86) Lady Bird (2017)
 87) Raging Bull (1980)
 88) The Piano (1993)
 89) American Psycho (2000)
 90) Groundhog Day (1993)
 91) The Grand Budapest Hotel (2014)
 92) One Flew Over The Cuckoo's Nest (1975)
 93) Lost In Translation (2003)
 94) Moonlight (2016)
 95) Memento (2000)
 96) E.T. The Extra Terrestrial (1982)
 97) Barbie (2023)
 98) When Harry Met Sally... (1989)
 99) Portrait Of A Lady On Fire (2020)
 100) Avatar (2009)
@@ -1,6 +1,7 @@
 """@brief The ports (interfaces) the use cases need from the outside world."""
-from collections.abc import Callable
+from collections.abc import Callable, Sequence
 from pathlib import Path
 from typing import TypeAlias
 from scrape_movies.domain.story import Story
@@ -9,3 +10,7 @@ from scrape_movies.domain.story import Story
 HtmlFetcher: TypeAlias = Callable[[str], str]
 ## Turns Hacker News HTML into stories.
 StoryParser: TypeAlias = Callable[[str], list[Story]]
 ## Turns the Empire page HTML into the movie lines found on it, in page order.
 MovieParser: TypeAlias = Callable[[str], list[str]]
 ## Writes lines to a file, one per line.
 MovieWriter: TypeAlias = Callable[[Sequence[str], Path], None]
@@ -1,6 +1,15 @@
 """@brief Use cases of the scraping exercises."""
-from scrape_movies.application.ports import HtmlFetcher, StoryParser
+from collections.abc import Sequence
 from pathlib import Path
 from scrape_movies.application.ports import (
    HtmlFetcher,
    MovieParser,
    MovieWriter,
    StoryParser,
 )
 from scrape_movies.domain.movie import format_titles
 from scrape_movies.domain.story import Story, top_story
@@ -24,3 +33,35 @@ def find_top_story(fetch: HtmlFetcher, parse: StoryParser, url: str) -> Story |
    @return The top story, or None when the page has no stories.
    """
    return top_story(list_stories(fetch, parse, url))
 def scrape_movie_titles(fetch: HtmlFetcher, parse: MovieParser, url: str) -> list[str]:
    """@brief Fetch the Empire page and read the titles in page order.
    @param fetch Fetches the HTML of the page.
    @param parse Reads the titles from the HTML.
    @param url The page address.
    @return The titles in page order (100 down to 1).
    """
    return parse(fetch(url))
 def save_movie_list(
    fetch: HtmlFetcher,
    parse: MovieParser,
    write: MovieWriter,
    url: str,
    path: Path,
 ) -> Sequence[str]:
    """@brief Scrape the list and write it, ascending and numbered, to a file.
    @param fetch Fetches the HTML of the page.
    @param parse Reads the titles from the HTML.
    @param write Writes the lines to the file.
    @param url The page address.
    @param path The file to write, for example `movies.txt`.
    @return The lines that were written.
    """
    lines = format_titles(scrape_movie_titles(fetch, parse, url))
    write(lines, path)
    return lines
@@ -35,3 +35,6 @@ MOVIES_OUTPUT_FILE: str = "movies.txt"
 ## The saved copy of the course's Hacker News page, used offline.
 HACKER_NEWS_FIXTURE_PATH: Path = FIXTURES_DIR / "hacker_news.html"
 ## The saved headings of the live Empire page, used offline.
 EMPIRE_FIXTURE_PATH: Path = FIXTURES_DIR / "empire_live_excerpt.html"
@@ -0,0 +1,22 @@
 """@brief The movie list rule: ascending order, every line starting with its rank."""
 import re
 from collections.abc import Sequence
 _RANK_PREFIX = re.compile(r"^\d+\) ")
 def format_titles(titles: Sequence[str]) -> list[str]:
    """@brief Turn a descending page list into an ascending, numbered list.
    Empire lists the movies from 100 down to 1, so the page order is reversed.
    A title that already starts with its rank, such as `3) Jaws`, is kept as
    it is; any other title gets `<rank>) ` in front of it.
    @param titles The titles in page order.
    @return The lines `1) ...`, `2) ...` in ascending order.
    """
    lines: list[str] = []
    for rank, title in enumerate(reversed(titles), 1):
        lines.append(title if _RANK_PREFIX.match(title) else f"{rank}) {title}")
    return lines
@@ -0,0 +1,55 @@
 """@brief Reading the movie titles from the Empire "100 Greatest Movies" page."""
 import re
 from bs4 import Tag
 from scrape_movies.infrastructure.soup_basics import make_soup
 ## The 2020 page (the assignment's snapshot): `<h3 class="title">100) Title</h3>`.
 SNAPSHOT_TITLE = "h3.title"
 ## The current page: `<h2><strong>100) Title (year)</strong></h2>`.
 _RANKED_HEADING = re.compile(r"^\s*\d{1,3}\)")
 def parse_titles(html: str) -> list[str]:
    """@brief Read the movie titles in page order.
    The assignment's snapshot is read through `h3.title`. When the page has no
    such tag, as on the current live page, the `h2` headings that start with a
    rank such as `100)` are used.
    @param html The HTML of the page.
    @return The titles in page order, each as the page shows it.
    """
    soup = make_soup(html)
    titles = [tag.get_text().strip() for tag in soup.select(SNAPSHOT_TITLE)]
    if titles:
        return titles
    headings = [tag for tag in soup.find_all("h2") if isinstance(tag, Tag)]
    return [
        text
        for text in (heading.get_text().strip() for heading in headings)
        if _RANKED_HEADING.match(text)
    ]
 def first_title_markup(html: str) -> str:
    """@brief Show the markup of the first title, as seen with "Inspect".
    @param html The HTML of the page.
    @return The indented HTML of the first title tag, or "" when there is none.
    """
    soup = make_soup(html)
    tag = soup.select_one(SNAPSHOT_TITLE)
    if tag is None:
        tag = next(
            (
                heading
                for heading in soup.find_all("h2")
                if isinstance(heading, Tag)
                and _RANKED_HEADING.match(heading.get_text())
            ),
            None,
        )
    return str(tag.prettify()).rstrip() if tag is not None else ""
@@ -0,0 +1,15 @@
 """@brief Writing the movie list to a text file."""
 from collections.abc import Sequence
 from pathlib import Path
 def write_movies(lines: Sequence[str], path: Path) -> None:
    """@brief Write one line per movie, UTF-8, with a final newline.
    @param lines The lines to write.
    @param path The file to create or overwrite.
    """
    with path.open("w", encoding="utf-8", newline="\n") as file:
        for line in lines:
            file.write(f"{line}\n")
@@ -2,8 +2,9 @@
 import argparse
 from collections.abc import Sequence
 from pathlib import Path
-from scrape_movies.presentation import hacker_news_steps, soup_steps
+from scrape_movies.presentation import hacker_news_steps, movies_steps, soup_steps
 def build_parser() -> argparse.ArgumentParser:
@@ -26,6 +27,17 @@ def build_parser() -> argparse.ArgumentParser:
        default="mirror",
        help="the course mirror (default), the live site or the saved copy",
    )
    movies = commands.add_parser("movies", help="scrape Empire's 100 greatest movies")
    movies.add_argument("step", choices=movies_steps.STEPS)
    movies.add_argument(
        "--source",
        choices=movies_steps.SOURCES,
        default="wayback",
        help="the Wayback snapshot (default), the live site or the saved copy",
    )
    movies.add_argument(
        "--output", type=Path, help="file written by the write step (movies.txt)"
    )
    return parser
@@ -43,5 +55,8 @@ def main(argv: Sequence[str] | None = None) -> int:
    if args.command == "hacker-news":
        print(hacker_news_steps.render_step(args.step, args.source))
        return 0
    if args.command == "movies":
        print(movies_steps.render_step(args.step, args.source, args.output))
        return 0
    parser.print_help()
    return 0
@@ -0,0 +1,63 @@
 """@brief Text output of each "100 Movies to Watch" lecture step."""
 from pathlib import Path
 from scrape_movies.application import use_cases
 from scrape_movies.application.ports import HtmlFetcher
 from scrape_movies.constants import (
    EMPIRE_FIXTURE_PATH,
    EMPIRE_LIVE_URL,
    EMPIRE_WAYBACK_URL,
    MOVIES_OUTPUT_FILE,
 )
 from scrape_movies.infrastructure import empire_movies, http_client, soup_basics
 from scrape_movies.infrastructure.movie_file import write_movies
 ## Where a step reads the page: the Wayback snapshot, the live site or the saved copy.
 SOURCES: tuple[str, ...] = ("wayback", "live", "fixture")
 _URLS: dict[str, str] = {
    "wayback": EMPIRE_WAYBACK_URL,
    "live": EMPIRE_LIVE_URL,
    "fixture": str(EMPIRE_FIXTURE_PATH),
 }
 ## The lecture steps by command-line name, in lecture order.
 STEPS: tuple[str, ...] = ("markup", "titles", "reversed", "write")
 _SHOWN_LINES = 12
 def _fetcher(source: str) -> HtmlFetcher:
    if source == "fixture":
        return lambda path: soup_basics.load_html(EMPIRE_FIXTURE_PATH)
    return http_client.fetch_html
 def _head(lines: list[str]) -> str:
    shown = lines[:_SHOWN_LINES]
    more = len(lines) - len(shown)
    return "\n".join(shown + ([f"... and {more} more"] if more > 0 else []))
 def render_step(step: str, source: str, output: Path | None = None) -> str:
    """@brief Run one lecture step.
    @param step One of STEPS.
    @param source One of SOURCES.
    @param output The file the `write` step creates; `movies.txt` when None.
    @return The text the step prints.
    @throws KeyError When the step or the source is unknown.
    """
    if step not in STEPS:
        raise KeyError(step)
    fetch, url = _fetcher(source), _URLS[source]
    if step == "markup":
        return empire_movies.first_title_markup(fetch(url))
    if step == "titles":
        titles = use_cases.scrape_movie_titles(fetch, empire_movies.parse_titles, url)
        return _head(titles)
    target = output or Path(MOVIES_OUTPUT_FILE)
    lines = use_cases.save_movie_list(
        fetch, empire_movies.parse_titles, write_movies, url, target
    )
    if step == "reversed":
        return _head(list(lines))
    return f"{_head(list(lines))}\nWrote {len(lines)} lines to {target}"
@@ -0,0 +1,105 @@
 <!doctype html>
 <html><head><meta charset="utf-8"><title>Empire best movies (headings excerpt)</title></head>
 <body>
 <!-- Only the 100 numbered headings of https://www.empireonline.com/movies/features/best-movies-2/ as fetched on 2026-10-28, markup unchanged. -->
 <span class="content"><h2><strong>100) Avatar (2009)</strong></h2></span>
 <span class="content"><h2><strong>99) Portrait Of A Lady On Fire (2020)</strong></h2></span>
 <span class="content"><h2><strong>98) When Harry Met Sally... (1989)</strong></h2></span>
 <span class="content"><h2><strong>97) Barbie (2023)</strong></h2></span>
 <span class="content"><h2><strong>96) E.T. The Extra Terrestrial (1982)</strong></h2></span>
 <span class="content"><h2><strong>95) Memento (2000)</strong></h2></span>
 <span class="content"><h2><strong>94) Moonlight (2016)</strong></h2></span>
 <span class="content"><h2><strong>93) Lost In Translation (2003)</strong></h2></span>
 <span class="content"><h2><strong>92) One Flew Over The Cuckoo&#x27;s Nest (1975)</strong></h2></span>
 <span class="content"><h2><strong>91) The Grand Budapest Hotel (2014)</strong></h2></span>
 <span class="content"><h2><strong>90) Groundhog Day (1993)</strong></h2></span>
 <span class="content"><h2><strong>89) American Psycho (2000)</strong></h2></span>
 <span class="content"><h2><strong>88) The Piano (1993)</strong></h2></span>
 <span class="content"><h2><strong>87) Raging Bull (1980)</strong></h2></span>
 <span class="content"><h2><strong>86) Lady Bird (2017)</strong></h2></span>
 <span class="content"><h2><strong>85) Rocky (1976)</strong></h2></span>
 <span class="content"><h2><strong>84) Pan&#x27;s Labyrinth (2006)</strong></h2></span>
 <span class="content"><h2><strong>83) Inglourious Basterds (2009)</strong></h2></span>
 <span class="content"><h2><strong>82) Brokeback Mountain (2005)</strong></h2></span>
 <span class="content"><h2><strong>81) Everything Everywhere All At Once (2022)</strong></h2></span>
 <span class="content"><h2><strong>80) Spider-Man: Into The Spider-Verse (2018)</strong></h2></span>
 <span class="content"><h2><strong>79) Paddington 2 (2017)</strong></h2></span>
 <span class="content"><h2><strong>78) Ghostbusters (1984)</strong></h2></span>
 <span class="content"><h2><strong>77) In The Mood For Love (2000)</strong></h2></span>
 <span class="content"><h2><strong>76) Once Upon A Time In Hollywood (2019)</strong></h2></span>
 <span class="content"><h2><strong>75) The Silence Of The Lambs (1991)</strong></h2></span>
 <span class="content"><h2><strong>74) Trainspotting (1996)</strong></h2></span>
 <span class="content"><h2><strong>73) The Lion King (1994)</strong></h2></span>
 <span class="content"><h2><strong>72) Magnolia (1999)</strong></h2></span>
 <span class="content"><h2><strong>71) Arrival (2016)</strong></h2></span>
 <span class="content"><h2><strong>70) Children Of Men (2006)</strong></h2></span>
 <span class="content"><h2><strong>69) The Babadook</strong> (2014)</h2></span>
 <span class="content"><h2><strong>68) Avengers: Endgame (2019)</strong></h2></span>
 <span class="content"><h2><strong>67) Do The Right Thing (1989)</strong></h2></span>
 <span class="content"><h2><strong>66) Indiana Jones And The Last Crusade (1989)</strong></h2></span>
 <span class="content"><h2><strong>65) Predator (1987)</strong></h2></span>
 <span class="content"><h2><strong>64) Oppenheimer (2023)</strong></h2></span>
 <span class="content"><h2><strong>63) The Big Lebowski (1998)</strong></h2></span>
 <span class="content"><h2><strong>62) Eternal Sunshine Of The Spotless Mind (2004)</strong></h2></span>
 <span class="content"><h2><strong>61) The Social Network (2010)</strong></h2></span>
 <span class="content"><h2><strong>60) Fargo (1996)</strong></h2></span>
 <span class="content"><h2><strong>59) The Exorcist (1973)</strong></h2></span>
 <span class="content"><h2><strong>58) Titanic (1997)</strong></h2></span>
 <span class="content"><h2><strong>57) Seven Samurai (1954)</strong></h2></span>
 <span class="content"><h2><strong>56) Spirited Away (2001)</strong></h2></span>
 <span class="content"><h2><strong>55) Hot Fuzz (2007)</strong></h2></span>
 <span class="content"><h2><strong>54) Get Out (2017)</strong></h2></span>
 <span class="content"><h2><strong>53) Vertigo (1958)</strong></h2></span>
 <span class="content"><h2><strong>52) Citizen Kane (1941)</strong></h2></span>
 <span class="content"><h2><strong>51) Rear Window (1954)</strong></h2></span>
 <span class="content"><h2><strong>50) No Country For Old Men (2007)</strong></h2></span>
 <span class="content"><h2><strong>49) Psycho (1960)</strong></h2></span>
 <span class="content"><h2><strong>48) Avengers: Infinity War (2018)</strong></h2></span>
 <span class="content"><h2><strong>47) It&#x27;s A Wonderful Life (1946)</strong></h2></span>
 <span class="content"><h2><strong>46) Whiplash (2014)</strong></h2></span>
 <span class="content"><h2><strong>45) Past Lives (2023)</strong></h2></span>
 <span class="content"><h2><strong>44) Taxi Driver (1976)</strong></h2></span>
 <span class="content"><h2><strong>43) La La Land (2016)</strong></h2></span>
 <span class="content"><h2><strong>42) Saving Private Ryan (1998)</strong></h2></span>
 <span class="content"><h2><strong>41) Schindler&#x27;s List (1993)</strong></h2></span>
 <span class="content"><h2><strong>40) Mulholland Drive (2001)</strong></h2></span>
 <span class="content"><h2><strong>39) Lawrence Of Arabia (1962)</strong></h2></span>
 <span class="content"><h2><strong>38) The Good, The Bad And The Ugly (1966)</strong></h2></span>
 <span class="content"><h2><strong>37) Fight Club (1999</strong>)</h2></span>
 <span class="content"><h2><strong>36) Point Break (1991)</strong></h2></span>
 <span class="content"><h2><strong>35) Mad Max: Fury Road (2015)</strong></h2></span>
 <span class="content"><h2><strong>34) The Shining (1980)</strong></h2></span>
 <span class="content"><h2><strong>33) The Lord Of The Rings: The Two Towers (2002)</strong></h2></span>
 <span class="content"><h2><strong>32) Seven (1995)</strong></h2></span>
 <span class="content"><h2><strong>31) The Thing (1982)</strong></h2></span>
 <span class="content"><h2><strong>30) Casablanca (1942)</strong></h2></span>
 <span class="content"><h2><strong>29) 12 Angry Men (1957)</strong></h2></span>
 <span class="content"><h2><strong>28) Apocalypse Now (1979)</strong></h2></span>
 <span class="content"><h2><strong>27) Parasite (2019)</strong></h2></span>
 <span class="content"><h2><strong>26) Gladiator (2000)</strong></h2></span>
 <span class="content"><h2><strong>25) 2001: A Space Odyssey (1968)</strong></h2></span>
 <span class="content"><h2><strong>24) There Will Be Blood (2007)</strong></h2></span>
 <span class="content"><h2><strong>23) Die Hard (1988)</strong></h2></span>
 <span class="content"><h2><strong>22) The Matrix (1999)</strong></h2></span>
 <span class="content"><h2><strong>21) The Lord Of The Rings: The Return Of The King (2003)</strong></h2></span>
 <span class="content"><h2><strong>20) Heat (1995)</strong></h2></span>
 <span class="content"><h2><strong>19) Interstellar (2014)</strong></h2></span>
 <span class="content"><h2><strong>18) Terminator 2: Judgment Day (1991)</strong></h2></span>
 <span class="content"><h2><strong>17) Alien (1979)</strong></h2></span>
 <span class="content"><h2><strong>16) The Godfather Part II (1974)</strong></h2></span>
 <span class="content"><h2><strong>15) Blade Runner (1982)</strong></h2></span>
 <span class="content"><h2><strong>14) Raiders Of The Lost Ark (1981)</strong></h2></span>
 <span class="content"><h2><strong>13) Jurassic Park (1993)</strong></h2></span>
 <span class="content"><h2><strong>12) Back To The Future (1985)</strong></h2></span>
 <span class="content"><h2><strong>11) Pulp Fiction (1994)</strong></h2></span>
 <span class="content"><h2><strong>10) The Dark Knight (2008)</strong></h2></span>
 <span class="content"><h2><strong>9) The Lord Of The Rings: The Fellowship Of The Ring (2001)</strong></h2></span>
 <span class="content"><h2><strong>8) Star Wars: Episode IV – A New Hope (1977)</strong></h2></span>
 <span class="content"><h2><strong>7) Goodfellas (1990)</strong></h2></span>
 <span class="content"><h2><strong>6) Inception (2010)</strong></h2></span>
 <span class="content"><h2><strong>5) Aliens (1986)</strong></h2></span>
 <span class="content"><h2><strong>4) Star Wars: The Empire Strikes Back (1980)</strong></h2></span>
 <span class="content"><h2><strong>3) The Shawshank Redemption (1994)</strong></h2></span>
 <span class="content"><h2><strong>2) The Godfather (1972)</strong></h2></span>
 <span class="content"><h2><strong>1) Jaws (1975)</strong></h2></span>
 </body></html>
@@ -0,0 +1,27 @@
 from pathlib import Path
 import pytest
 from scrape_movies.presentation import cli
 def _run(target: Path, capsys: pytest.CaptureFixture[str]) -> str:
    argv = ["movies", "write", "--source", "fixture", "--output", str(target)]
    assert cli.main(argv) == 0
    capsys.readouterr()
    return target.read_text(encoding="utf-8")
 def test_write_when_run_from_fixture_creates_100_ascending_lines(
    tmp_path: Path, capsys: pytest.CaptureFixture[str]
 ) -> None:
    lines = _run(tmp_path / "movies.txt", capsys).splitlines()
    assert len(lines) == 100
    assert [int(line.split(")")[0]) for line in lines] == list(range(1, 101))
 def test_write_when_run_twice_produces_identical_files(
    tmp_path: Path, capsys: pytest.CaptureFixture[str]
 ) -> None:
    first = _run(tmp_path / "a.txt", capsys)
    assert _run(tmp_path / "b.txt", capsys) == first
@@ -39,3 +39,10 @@ def test_main_when_every_hacker_news_step_runs_from_fixture_prints_output(
    for step in hacker_news_steps.STEPS:
        assert cli.main(["hacker-news", step, "--source", "fixture"]) == 0
        assert capsys.readouterr().out.strip(), step
 def test_main_when_movies_markup_from_fixture_prints_heading(
    capsys: pytest.CaptureFixture[str],
 ) -> None:
    assert cli.main(["movies", "markup", "--source", "fixture"]) == 0
    assert "<h2>" in capsys.readouterr().out
@@ -0,0 +1,48 @@
 from scrape_movies.constants import EMPIRE_FIXTURE_PATH
 from scrape_movies.infrastructure import empire_movies
 from scrape_movies.infrastructure.soup_basics import load_html
 SNAPSHOT_LAYOUT = """
 <h3 class="title">4) The Shawshank Redemption</h3>
 <h3 class="title">3) The Dark Knight</h3>
 <h3 class="other">not a title</h3>
 <h3 class="title">2) The Empire Strikes Back</h3>
 """
 def test_parse_titles_when_snapshot_layout_reads_h3_title_in_page_order() -> None:
    assert empire_movies.parse_titles(SNAPSHOT_LAYOUT) == [
        "4) The Shawshank Redemption",
        "3) The Dark Knight",
        "2) The Empire Strikes Back",
    ]
 def test_parse_titles_when_live_layout_reads_ranked_h2_headings() -> None:
    titles = empire_movies.parse_titles(load_html(EMPIRE_FIXTURE_PATH))
    assert len(titles) == 100
    assert titles[0].startswith("100) ")
    assert titles[-1].startswith("1) ")
 def test_parse_titles_when_h2_has_no_rank_ignores_it() -> None:
    assert empire_movies.parse_titles("<h2>Jump to</h2><h2>2) Two</h2>") == ["2) Two"]
 def test_parse_titles_when_page_has_no_titles_returns_empty() -> None:
    assert empire_movies.parse_titles("<p>nothing</p>") == []
 def test_first_title_markup_when_snapshot_layout_shows_h3() -> None:
    assert empire_movies.first_title_markup(SNAPSHOT_LAYOUT).startswith(
        '<h3 class="title">'
    )
 def test_first_title_markup_when_live_layout_shows_h2() -> None:
    markup = empire_movies.first_title_markup(load_html(EMPIRE_FIXTURE_PATH))
    assert markup.startswith("<h2>")
 def test_first_title_markup_when_nothing_found_returns_empty() -> None:
    assert empire_movies.first_title_markup("<p>x</p>") == ""
@@ -0,0 +1,18 @@
 from scrape_movies.domain.movie import format_titles
 def test_format_titles_when_titles_are_ranked_reverses_to_ascending() -> None:
    page = ["3) Three", "2) Two", "1) One"]
    assert format_titles(page) == ["1) One", "2) Two", "3) Three"]
 def test_format_titles_when_titles_have_no_rank_numbers_them() -> None:
    assert format_titles(["Third", "Second", "First"]) == [
        "1) First",
        "2) Second",
        "3) Third",
    ]
 def test_format_titles_when_list_is_empty_returns_empty() -> None:
    assert format_titles([]) == []
@@ -0,0 +1,16 @@
 from pathlib import Path
 from scrape_movies.infrastructure.movie_file import write_movies
 def test_write_movies_when_lines_given_writes_one_per_line(tmp_path: Path) -> None:
    target = tmp_path / "movies.txt"
    write_movies(["1) One", "2) Zoë"], target)
    assert target.read_bytes() == "1) One\n2) Zoë\n".encode()
 def test_write_movies_when_file_exists_replaces_it(tmp_path: Path) -> None:
    target = tmp_path / "movies.txt"
    target.write_text("old\n", encoding="utf-8")
    write_movies(["1) New"], target)
    assert target.read_text(encoding="utf-8") == "1) New\n"
@@ -1,3 +1,6 @@
 from collections.abc import Sequence
 from pathlib import Path
 from scrape_movies.application import use_cases
 from scrape_movies.domain.story import Story
@@ -22,3 +25,21 @@ def test_find_top_story_when_stories_exist_returns_highest() -> None:
 def test_find_top_story_when_page_is_empty_returns_none() -> None:
    assert use_cases.find_top_story(_fetch, lambda html: [], "http://x") is None
 def test_save_movie_list_when_titles_are_descending_writes_ascending() -> None:
    written: list[tuple[list[str], Path]] = []
    def write(lines: Sequence[str], path: Path) -> None:
        written.append((list(lines), path))
    lines = use_cases.save_movie_list(
        _fetch, lambda html: ["2) Two", "1) One"], write, "http://x", Path("m.txt")
    )
    assert list(lines) == ["1) One", "2) Two"]
    assert written == [(["1) One", "2) Two"], Path("m.txt"))]
 def test_scrape_movie_titles_when_called_keeps_page_order() -> None:
    titles = use_cases.scrape_movie_titles(_fetch, lambda html: ["b", "a"], "http://x")
    assert titles == ["b", "a"]
@@ -39,18 +39,21 @@ def capture_page(url: str, target: Path) -> Path:
    @return The path of the PNG.
    """
    target.parent.mkdir(parents=True, exist_ok=True)
-    subprocess.run(
+    browser = find_browser()
-        [
+    for mode in ("new", "old"):
-            find_browser(),
+        result = subprocess.run(
-            "--headless=new",
+            [
-            "--disable-gpu",
+                browser,
-            "--hide-scrollbars",
+                f"--headless={mode}",
-            f"--window-size={WINDOW}",
+                "--disable-gpu",
-            f"--screenshot={target}",
+                "--hide-scrollbars",
-            url,
+                f"--window-size={WINDOW}",
-        ],
+                f"--screenshot={target}",
-        check=True,
+                url,
-        capture_output=True,
+            ],
-        timeout=120,
+            capture_output=True,
-    )
+            timeout=120,
-    return target
+        )
        if result.returncode == 0 and target.exists():
            return target
    raise RuntimeError(f"The browser could not capture {url}")
@@ -29,10 +29,20 @@ STEPS: dict[str, list[str]] = {
    "hn-4-links": ["hacker-news", "links"],
    "hn-5-upvotes": ["hacker-news", "upvotes"],
    "hn-6-top": ["hacker-news", "top"],
    "movies-2-markup": ["movies", "markup", "--source", "fixture"],
    "movies-3-titles": ["movies", "titles", "--source", "fixture"],
    "movies-4-reversed": ["movies", "reversed", "--source", "fixture"],
    "movies-5-write": ["movies", "write", "--source", "fixture"],
 }
 ## Screenshots of a file's first lines: image name, file and number of lines.
 FILES: dict[str, tuple[str, int]] = {"movies-6-file": ("movies.txt", 15)}
 ## Screenshots of real web pages: image name and the address of the page.
 PAGES: dict[str, str] = {
    "hn-1-page": "https://appbrewery.github.io/news.ycombinator.com/",
    # The live page opens a consent wall, so the saved headings are drawn instead.
    "movies-1-page": (
        ROOT / "tests" / "fixtures" / "empire_live_excerpt.html"
    ).as_uri(),
 }
@@ -63,6 +73,13 @@ def main() -> None:
    for name, args in STEPS.items():
        if name.startswith(prefix):
            print(capture(name, args))
    for name, (file_name, count) in FILES.items():
        if name.startswith(prefix):
            text = (ROOT / file_name).read_text(encoding="utf-8")
            target = IMAGES / f"{name}.png"
            shown = os.linesep.join(text.splitlines()[:count])
            render(f"head -n {count} {file_name}", shown, target)
            print(target)
    for name, url in PAGES.items():
        if name.startswith(prefix):
            print(capture_page(url, IMAGES / f"{name}.png"))