diff --git a/README.md b/README.md index 3b87547..65268fe 100644 --- a/README.md +++ b/README.md @@ -161,6 +161,52 @@ A short summary in our own words; it is not legal advice. - **Commercial use** of scraped data invites expensive disputes even where the scraping itself was allowed. - **Be a good guest:** prefer an official API when there is one, read the site's `robots.txt` to see what it allows, and send at most one request a minute. This project fetches one page per run and identifies itself with a `User-Agent`. +### Project: 100 movies to watch + +The assignment: scrape Empire's "100 Greatest Movies" and write `movies.txt` with the titles numbered from `1)` upwards. The code is `infrastructure/empire_movies.py` (parsing), `domain/movie.py` (reverse and number) and `infrastructure/movie_file.py` (writing). `--source wayback` (default) reads the Internet Archive snapshot of May 2020 that the assignment pins, `--source live` reads the current Empire page and `--source fixture` reads the saved headings in `tests/fixtures/`. + +**1. Choose the page** (the saved headings of the page) + +![Step 1: the page](docs/images/movies-1-page.png) + +**2. Inspect the markup** (the 2020 page has `

`; the current page has `

` headings that start with a rank) + +```bash +python -m scrape_movies movies markup +``` + +![Step 2: markup](docs/images/movies-2-markup.png) + +**3. Titles in page order** (`select("h3.title")` and a list comprehension over `get_text()`; the page counts down from 100) + +```bash +python -m scrape_movies movies titles +``` + +![Step 3: titles](docs/images/movies-3-titles.png) + +**4. Reverse the list** (`titles[::-1]`, so the list counts up from 1) + +```bash +python -m scrape_movies movies reversed +``` + +![Step 4: reversed](docs/images/movies-4-reversed.png) + +**5. Write `movies.txt`** (a `with open(...)` block, one title per line) + +```bash +python -m scrape_movies movies write +``` + +![Step 5: write](docs/images/movies-5-write.png) + +**6. The result** + +![Step 6: movies.txt](docs/images/movies-6-file.png) + +The committed [`movies.txt`](./movies.txt) was produced with `--source fixture` on 2026-10-28 from the current Empire list, so its ranking differs from the 2020 list in the lecture video. The Internet Archive answered `429 Too Many Requests` while this project was built, so the snapshot is read live only; its `h3.title` layout is covered by a hand-written sample in the tests. + ## 🧪 Tests ```bash diff --git a/docs/artifact-registry.md b/docs/artifact-registry.md index c7a9d18..54c0f3e 100644 --- a/docs/artifact-registry.md +++ b/docs/artifact-registry.md @@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g. | SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 | | PP | Project Plan | docs/project-plan.md | 002 | | MIL | Milestone / Gateway | docs/milestones/*.md | 006 | -| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 014 | +| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 015 | ## Languages diff --git a/docs/images/movies-1-page.png b/docs/images/movies-1-page.png new file mode 100644 index 0000000..91ff187 Binary files /dev/null and b/docs/images/movies-1-page.png differ diff --git a/docs/images/movies-2-markup.png b/docs/images/movies-2-markup.png new file mode 100644 index 0000000..053ddb1 Binary files /dev/null and b/docs/images/movies-2-markup.png differ diff --git a/docs/images/movies-3-titles.png b/docs/images/movies-3-titles.png new file mode 100644 index 0000000..3244fa3 Binary files /dev/null and b/docs/images/movies-3-titles.png differ diff --git a/docs/images/movies-4-reversed.png b/docs/images/movies-4-reversed.png new file mode 100644 index 0000000..8dac658 Binary files /dev/null and b/docs/images/movies-4-reversed.png differ diff --git a/docs/images/movies-5-write.png b/docs/images/movies-5-write.png new file mode 100644 index 0000000..d6258e1 Binary files /dev/null and b/docs/images/movies-5-write.png differ diff --git a/docs/images/movies-6-file.png b/docs/images/movies-6-file.png new file mode 100644 index 0000000..4cc63ea Binary files /dev/null and b/docs/images/movies-6-file.png differ diff --git a/docs/sqa/reviews/rc-014-mil-004-python.md b/docs/sqa/reviews/rc-014-mil-004-python.md new file mode 100644 index 0000000..2a3c4c3 --- /dev/null +++ b/docs/sqa/reviews/rc-014-mil-004-python.md @@ -0,0 +1,63 @@ +# Review of MIL-004 Python code + +## Metadata +| Key | Value | +| --- | --- | +| ID | RC-014 | +| CrossReference | [MIL-004], [QC-PY-001] | + +## Version History +| Date | Status | Author | Reviewer | Change | Commit | +| --- | --- | --- | --- | --- | --- | +| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | [02c339c] | + +--- + +## Artifact Under Review + +- Instance reviewed: [MIL-004] +- Checklist used: [QC-PY-001] +- Scope: full review. Code reviewed: movie, empire_movies, movie_file, use_cases, movies_steps, cli, tools/, tests/unit and tests/integration. +- Language and domain: n/a (technical type) +- Language reviewer: none +- Mode: autonomous, not independent (reviewer is `AI as stand-in for S01`) + +## Checklist Results + +| # | Criterion | Status | Evidence/Notes | +| --- | --- | --- | --- | +| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE. | +| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | names state purpose; the lecture names (`find_all`, `select_one`) are kept in the helper names. | +| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | `ruff check` and `ruff format --check` are clean with no `noqa`. | +| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | every signature is annotated; `mypy --strict` passes, also in a venv without the third-party packages. | +| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | Pass | fetch errors are raised as `FetchError` from the cause (MIL-003); no exception is swallowed. | +| 6 | No mutable default arguments and no shadowed builtins | Pass | no mutable defaults, no shadowed builtins. | +| 7 | Files, locks and connections are managed with context managers | Pass | `write_movies` writes inside a `with path.open(...)` block. | +| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | every public module, class and function has a Doxygen docstring saying what it does. | +| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | N-A | no logging; the command-line output is the product, written with `print` in the presentation layer. | +| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | no Design Class Diagram; all tasks are plain technical tasks (Needs Use Case: No). | +| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | tests are named for behaviour, offline and independent of order. | +| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | `mypy --strict` runs over `src`, `tests` and `tools`. | +| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Fail | dependencies are declared with lower bounds, not pinned (see RC-009 action item); none is unused. | +| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | each constant and format is defined once; the step texts are built in one place. | +| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | `domain` imports nothing from other layers; `infrastructure` imports `domain` and `constants`; `presentation` composes them. | +| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | use cases take callables (ports); a new page layout needs a new parser, not an edit of the use case. | +| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | code is in `domain`, `application`, `infrastructure`, `presentation`; tests in `tests/unit` and `tests/integration` (the CLI-to-file flow); `e2e` stays empty because the CLI is covered by the integration test. | +| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | all names are English lowercase with `_`, no space. | + +## Overall Verdict + +Go — no Mandatory criterion fails and none is Pending. Stand-in review in autonomous mode, not independent; S01 challenges it in the run summary. + +## Action Items + +| Action | Owner | Due | +| --- | --- | --- | +| Pin dependencies or accept lower bounds (open from RC-009, optional criterion 13) | S01 | 2026-11-14 | +| Refresh the Wayback fixture when archive.org stops answering 429 (the snapshot layout is covered by a hand-written sample) | S01 | 2026-11-14 | + +--- + +[MIL-004]: ../../milestones/mil-004-top-100-movies.md +[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md +[02c339c]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/commit/02c339c2dbe95b0e2eff5e4a3738f6e946014d26 diff --git a/movies.txt b/movies.txt new file mode 100644 index 0000000..003a005 --- /dev/null +++ b/movies.txt @@ -0,0 +1,100 @@ +1) Jaws (1975) +2) The Godfather (1972) +3) The Shawshank Redemption (1994) +4) Star Wars: The Empire Strikes Back (1980) +5) Aliens (1986) +6) Inception (2010) +7) Goodfellas (1990) +8) Star Wars: Episode IV – A New Hope (1977) +9) The Lord Of The Rings: The Fellowship Of The Ring (2001) +10) The Dark Knight (2008) +11) Pulp Fiction (1994) +12) Back To The Future (1985) +13) Jurassic Park (1993) +14) Raiders Of The Lost Ark (1981) +15) Blade Runner (1982) +16) The Godfather Part II (1974) +17) Alien (1979) +18) Terminator 2: Judgment Day (1991) +19) Interstellar (2014) +20) Heat (1995) +21) The Lord Of The Rings: The Return Of The King (2003) +22) The Matrix (1999) +23) Die Hard (1988) +24) There Will Be Blood (2007) +25) 2001: A Space Odyssey (1968) +26) Gladiator (2000) +27) Parasite (2019) +28) Apocalypse Now (1979) +29) 12 Angry Men (1957) +30) Casablanca (1942) +31) The Thing (1982) +32) Seven (1995) +33) The Lord Of The Rings: The Two Towers (2002) +34) The Shining (1980) +35) Mad Max: Fury Road (2015) +36) Point Break (1991) +37) Fight Club (1999) +38) The Good, The Bad And The Ugly (1966) +39) Lawrence Of Arabia (1962) +40) Mulholland Drive (2001) +41) Schindler's List (1993) +42) Saving Private Ryan (1998) +43) La La Land (2016) +44) Taxi Driver (1976) +45) Past Lives (2023) +46) Whiplash (2014) +47) It's A Wonderful Life (1946) +48) Avengers: Infinity War (2018) +49) Psycho (1960) +50) No Country For Old Men (2007) +51) Rear Window (1954) +52) Citizen Kane (1941) +53) Vertigo (1958) +54) Get Out (2017) +55) Hot Fuzz (2007) +56) Spirited Away (2001) +57) Seven Samurai (1954) +58) Titanic (1997) +59) The Exorcist (1973) +60) Fargo (1996) +61) The Social Network (2010) +62) Eternal Sunshine Of The Spotless Mind (2004) +63) The Big Lebowski (1998) +64) Oppenheimer (2023) +65) Predator (1987) +66) Indiana Jones And The Last Crusade (1989) +67) Do The Right Thing (1989) +68) Avengers: Endgame (2019) +69) The Babadook (2014) +70) Children Of Men (2006) +71) Arrival (2016) +72) Magnolia (1999) +73) The Lion King (1994) +74) Trainspotting (1996) +75) The Silence Of The Lambs (1991) +76) Once Upon A Time In Hollywood (2019) +77) In The Mood For Love (2000) +78) Ghostbusters (1984) +79) Paddington 2 (2017) +80) Spider-Man: Into The Spider-Verse (2018) +81) Everything Everywhere All At Once (2022) +82) Brokeback Mountain (2005) +83) Inglourious Basterds (2009) +84) Pan's Labyrinth (2006) +85) Rocky (1976) +86) Lady Bird (2017) +87) Raging Bull (1980) +88) The Piano (1993) +89) American Psycho (2000) +90) Groundhog Day (1993) +91) The Grand Budapest Hotel (2014) +92) One Flew Over The Cuckoo's Nest (1975) +93) Lost In Translation (2003) +94) Moonlight (2016) +95) Memento (2000) +96) E.T. The Extra Terrestrial (1982) +97) Barbie (2023) +98) When Harry Met Sally... (1989) +99) Portrait Of A Lady On Fire (2020) +100) Avatar (2009) diff --git a/src/scrape_movies/application/ports.py b/src/scrape_movies/application/ports.py index 83f921b..b59c9bd 100644 --- a/src/scrape_movies/application/ports.py +++ b/src/scrape_movies/application/ports.py @@ -1,6 +1,7 @@ """@brief The ports (interfaces) the use cases need from the outside world.""" -from collections.abc import Callable +from collections.abc import Callable, Sequence +from pathlib import Path from typing import TypeAlias from scrape_movies.domain.story import Story @@ -9,3 +10,7 @@ from scrape_movies.domain.story import Story HtmlFetcher: TypeAlias = Callable[[str], str] ## Turns Hacker News HTML into stories. StoryParser: TypeAlias = Callable[[str], list[Story]] +## Turns the Empire page HTML into the movie lines found on it, in page order. +MovieParser: TypeAlias = Callable[[str], list[str]] +## Writes lines to a file, one per line. +MovieWriter: TypeAlias = Callable[[Sequence[str], Path], None] diff --git a/src/scrape_movies/application/use_cases.py b/src/scrape_movies/application/use_cases.py index e8b1146..493aa93 100644 --- a/src/scrape_movies/application/use_cases.py +++ b/src/scrape_movies/application/use_cases.py @@ -1,26 +1,67 @@ -"""@brief Use cases of the scraping exercises.""" - -from scrape_movies.application.ports import HtmlFetcher, StoryParser -from scrape_movies.domain.story import Story, top_story - - -def list_stories(fetch: HtmlFetcher, parse: StoryParser, url: str) -> list[Story]: - """@brief Fetch a Hacker News page and read its stories. - - @param fetch Fetches the HTML of the page. - @param parse Reads the stories from the HTML. - @param url The page address. - @return The stories in page order. - """ - return parse(fetch(url)) - - -def find_top_story(fetch: HtmlFetcher, parse: StoryParser, url: str) -> Story | None: - """@brief Find the story with the most upvotes on a Hacker News page. - - @param fetch Fetches the HTML of the page. - @param parse Reads the stories from the HTML. - @param url The page address. - @return The top story, or None when the page has no stories. - """ - return top_story(list_stories(fetch, parse, url)) +"""@brief Use cases of the scraping exercises.""" + +from collections.abc import Sequence +from pathlib import Path + +from scrape_movies.application.ports import ( + HtmlFetcher, + MovieParser, + MovieWriter, + StoryParser, +) +from scrape_movies.domain.movie import format_titles +from scrape_movies.domain.story import Story, top_story + + +def list_stories(fetch: HtmlFetcher, parse: StoryParser, url: str) -> list[Story]: + """@brief Fetch a Hacker News page and read its stories. + + @param fetch Fetches the HTML of the page. + @param parse Reads the stories from the HTML. + @param url The page address. + @return The stories in page order. + """ + return parse(fetch(url)) + + +def find_top_story(fetch: HtmlFetcher, parse: StoryParser, url: str) -> Story | None: + """@brief Find the story with the most upvotes on a Hacker News page. + + @param fetch Fetches the HTML of the page. + @param parse Reads the stories from the HTML. + @param url The page address. + @return The top story, or None when the page has no stories. + """ + return top_story(list_stories(fetch, parse, url)) + + +def scrape_movie_titles(fetch: HtmlFetcher, parse: MovieParser, url: str) -> list[str]: + """@brief Fetch the Empire page and read the titles in page order. + + @param fetch Fetches the HTML of the page. + @param parse Reads the titles from the HTML. + @param url The page address. + @return The titles in page order (100 down to 1). + """ + return parse(fetch(url)) + + +def save_movie_list( + fetch: HtmlFetcher, + parse: MovieParser, + write: MovieWriter, + url: str, + path: Path, +) -> Sequence[str]: + """@brief Scrape the list and write it, ascending and numbered, to a file. + + @param fetch Fetches the HTML of the page. + @param parse Reads the titles from the HTML. + @param write Writes the lines to the file. + @param url The page address. + @param path The file to write, for example `movies.txt`. + @return The lines that were written. + """ + lines = format_titles(scrape_movie_titles(fetch, parse, url)) + write(lines, path) + return lines diff --git a/src/scrape_movies/constants.py b/src/scrape_movies/constants.py index 61c1756..83677fd 100644 --- a/src/scrape_movies/constants.py +++ b/src/scrape_movies/constants.py @@ -35,3 +35,6 @@ MOVIES_OUTPUT_FILE: str = "movies.txt" ## The saved copy of the course's Hacker News page, used offline. HACKER_NEWS_FIXTURE_PATH: Path = FIXTURES_DIR / "hacker_news.html" + +## The saved headings of the live Empire page, used offline. +EMPIRE_FIXTURE_PATH: Path = FIXTURES_DIR / "empire_live_excerpt.html" diff --git a/src/scrape_movies/domain/movie.py b/src/scrape_movies/domain/movie.py new file mode 100644 index 0000000..1372c22 --- /dev/null +++ b/src/scrape_movies/domain/movie.py @@ -0,0 +1,22 @@ +"""@brief The movie list rule: ascending order, every line starting with its rank.""" + +import re +from collections.abc import Sequence + +_RANK_PREFIX = re.compile(r"^\d+\) ") + + +def format_titles(titles: Sequence[str]) -> list[str]: + """@brief Turn a descending page list into an ascending, numbered list. + + Empire lists the movies from 100 down to 1, so the page order is reversed. + A title that already starts with its rank, such as `3) Jaws`, is kept as + it is; any other title gets `) ` in front of it. + + @param titles The titles in page order. + @return The lines `1) ...`, `2) ...` in ascending order. + """ + lines: list[str] = [] + for rank, title in enumerate(reversed(titles), 1): + lines.append(title if _RANK_PREFIX.match(title) else f"{rank}) {title}") + return lines diff --git a/src/scrape_movies/infrastructure/empire_movies.py b/src/scrape_movies/infrastructure/empire_movies.py new file mode 100644 index 0000000..7ce3217 --- /dev/null +++ b/src/scrape_movies/infrastructure/empire_movies.py @@ -0,0 +1,55 @@ +"""@brief Reading the movie titles from the Empire "100 Greatest Movies" page.""" + +import re + +from bs4 import Tag + +from scrape_movies.infrastructure.soup_basics import make_soup + +## The 2020 page (the assignment's snapshot): `

100) Title

`. +SNAPSHOT_TITLE = "h3.title" +## The current page: `

100) Title (year)

`. +_RANKED_HEADING = re.compile(r"^\s*\d{1,3}\)") + + +def parse_titles(html: str) -> list[str]: + """@brief Read the movie titles in page order. + + The assignment's snapshot is read through `h3.title`. When the page has no + such tag, as on the current live page, the `h2` headings that start with a + rank such as `100)` are used. + + @param html The HTML of the page. + @return The titles in page order, each as the page shows it. + """ + soup = make_soup(html) + titles = [tag.get_text().strip() for tag in soup.select(SNAPSHOT_TITLE)] + if titles: + return titles + headings = [tag for tag in soup.find_all("h2") if isinstance(tag, Tag)] + return [ + text + for text in (heading.get_text().strip() for heading in headings) + if _RANKED_HEADING.match(text) + ] + + +def first_title_markup(html: str) -> str: + """@brief Show the markup of the first title, as seen with "Inspect". + + @param html The HTML of the page. + @return The indented HTML of the first title tag, or "" when there is none. + """ + soup = make_soup(html) + tag = soup.select_one(SNAPSHOT_TITLE) + if tag is None: + tag = next( + ( + heading + for heading in soup.find_all("h2") + if isinstance(heading, Tag) + and _RANKED_HEADING.match(heading.get_text()) + ), + None, + ) + return str(tag.prettify()).rstrip() if tag is not None else "" diff --git a/src/scrape_movies/infrastructure/movie_file.py b/src/scrape_movies/infrastructure/movie_file.py new file mode 100644 index 0000000..2ef8b18 --- /dev/null +++ b/src/scrape_movies/infrastructure/movie_file.py @@ -0,0 +1,15 @@ +"""@brief Writing the movie list to a text file.""" + +from collections.abc import Sequence +from pathlib import Path + + +def write_movies(lines: Sequence[str], path: Path) -> None: + """@brief Write one line per movie, UTF-8, with a final newline. + + @param lines The lines to write. + @param path The file to create or overwrite. + """ + with path.open("w", encoding="utf-8", newline="\n") as file: + for line in lines: + file.write(f"{line}\n") diff --git a/src/scrape_movies/presentation/cli.py b/src/scrape_movies/presentation/cli.py index 0b17b87..c35d3fd 100644 --- a/src/scrape_movies/presentation/cli.py +++ b/src/scrape_movies/presentation/cli.py @@ -2,8 +2,9 @@ import argparse from collections.abc import Sequence +from pathlib import Path -from scrape_movies.presentation import hacker_news_steps, soup_steps +from scrape_movies.presentation import hacker_news_steps, movies_steps, soup_steps def build_parser() -> argparse.ArgumentParser: @@ -26,6 +27,17 @@ def build_parser() -> argparse.ArgumentParser: default="mirror", help="the course mirror (default), the live site or the saved copy", ) + movies = commands.add_parser("movies", help="scrape Empire's 100 greatest movies") + movies.add_argument("step", choices=movies_steps.STEPS) + movies.add_argument( + "--source", + choices=movies_steps.SOURCES, + default="wayback", + help="the Wayback snapshot (default), the live site or the saved copy", + ) + movies.add_argument( + "--output", type=Path, help="file written by the write step (movies.txt)" + ) return parser @@ -43,5 +55,8 @@ def main(argv: Sequence[str] | None = None) -> int: if args.command == "hacker-news": print(hacker_news_steps.render_step(args.step, args.source)) return 0 + if args.command == "movies": + print(movies_steps.render_step(args.step, args.source, args.output)) + return 0 parser.print_help() return 0 diff --git a/src/scrape_movies/presentation/movies_steps.py b/src/scrape_movies/presentation/movies_steps.py new file mode 100644 index 0000000..dc9b664 --- /dev/null +++ b/src/scrape_movies/presentation/movies_steps.py @@ -0,0 +1,63 @@ +"""@brief Text output of each "100 Movies to Watch" lecture step.""" + +from pathlib import Path + +from scrape_movies.application import use_cases +from scrape_movies.application.ports import HtmlFetcher +from scrape_movies.constants import ( + EMPIRE_FIXTURE_PATH, + EMPIRE_LIVE_URL, + EMPIRE_WAYBACK_URL, + MOVIES_OUTPUT_FILE, +) +from scrape_movies.infrastructure import empire_movies, http_client, soup_basics +from scrape_movies.infrastructure.movie_file import write_movies + +## Where a step reads the page: the Wayback snapshot, the live site or the saved copy. +SOURCES: tuple[str, ...] = ("wayback", "live", "fixture") +_URLS: dict[str, str] = { + "wayback": EMPIRE_WAYBACK_URL, + "live": EMPIRE_LIVE_URL, + "fixture": str(EMPIRE_FIXTURE_PATH), +} +## The lecture steps by command-line name, in lecture order. +STEPS: tuple[str, ...] = ("markup", "titles", "reversed", "write") +_SHOWN_LINES = 12 + + +def _fetcher(source: str) -> HtmlFetcher: + if source == "fixture": + return lambda path: soup_basics.load_html(EMPIRE_FIXTURE_PATH) + return http_client.fetch_html + + +def _head(lines: list[str]) -> str: + shown = lines[:_SHOWN_LINES] + more = len(lines) - len(shown) + return "\n".join(shown + ([f"... and {more} more"] if more > 0 else [])) + + +def render_step(step: str, source: str, output: Path | None = None) -> str: + """@brief Run one lecture step. + + @param step One of STEPS. + @param source One of SOURCES. + @param output The file the `write` step creates; `movies.txt` when None. + @return The text the step prints. + @throws KeyError When the step or the source is unknown. + """ + if step not in STEPS: + raise KeyError(step) + fetch, url = _fetcher(source), _URLS[source] + if step == "markup": + return empire_movies.first_title_markup(fetch(url)) + if step == "titles": + titles = use_cases.scrape_movie_titles(fetch, empire_movies.parse_titles, url) + return _head(titles) + target = output or Path(MOVIES_OUTPUT_FILE) + lines = use_cases.save_movie_list( + fetch, empire_movies.parse_titles, write_movies, url, target + ) + if step == "reversed": + return _head(list(lines)) + return f"{_head(list(lines))}\nWrote {len(lines)} lines to {target}" diff --git a/tests/fixtures/empire_live_excerpt.html b/tests/fixtures/empire_live_excerpt.html new file mode 100644 index 0000000..2d32137 --- /dev/null +++ b/tests/fixtures/empire_live_excerpt.html @@ -0,0 +1,105 @@ + +Empire best movies (headings excerpt) + + +

100) Avatar (2009)

 +

99) Portrait Of A Lady On Fire (2020)

 +

98) When Harry Met Sally... (1989)

 +

97) Barbie (2023)

 +

96) E.T. The Extra Terrestrial (1982)

 +

95) Memento (2000)

 +

94) Moonlight (2016)

 +

93) Lost In Translation (2003)

 +

92) One Flew Over The Cuckoo's Nest (1975)

 +

91) The Grand Budapest Hotel (2014)

 +

90) Groundhog Day (1993)

 +

89) American Psycho (2000)

 +

88) The Piano (1993)

 +

87) Raging Bull (1980)

 +

86) Lady Bird (2017)

 +

85) Rocky (1976)

 +

84) Pan's Labyrinth (2006)

 +

83) Inglourious Basterds (2009)

 +

82) Brokeback Mountain (2005)

 +

81) Everything Everywhere All At Once (2022)

 +

80) Spider-Man: Into The Spider-Verse (2018)

 +

79) Paddington 2 (2017)

 +

78) Ghostbusters (1984)

 +

77) In The Mood For Love (2000)

 +

76) Once Upon A Time In Hollywood (2019)

 +

75) The Silence Of The Lambs (1991)

 +

74) Trainspotting (1996)

 +

73) The Lion King (1994)

 +

72) Magnolia (1999)

 +

71) Arrival (2016)

 +

70) Children Of Men (2006)

 +

69) The Babadook (2014)

 +

68) Avengers: Endgame (2019)

 +

67) Do The Right Thing (1989)

 +

66) Indiana Jones And The Last Crusade (1989)

 +

65) Predator (1987)

 +

64) Oppenheimer (2023)

 +

63) The Big Lebowski (1998)

 +

62) Eternal Sunshine Of The Spotless Mind (2004)

 +

61) The Social Network (2010)

 +

60) Fargo (1996)

 +

59) The Exorcist (1973)

 +

58) Titanic (1997)

 +

57) Seven Samurai (1954)

 +

56) Spirited Away (2001)

 +

55) Hot Fuzz (2007)

 +

54) Get Out (2017)

 +

53) Vertigo (1958)

 +

52) Citizen Kane (1941)

 +

51) Rear Window (1954)

 +

50) No Country For Old Men (2007)

 +

49) Psycho (1960)

 +

48) Avengers: Infinity War (2018)

 +

47) It's A Wonderful Life (1946)

 +

46) Whiplash (2014)

 +

45) Past Lives (2023)

 +

44) Taxi Driver (1976)

 +

43) La La Land (2016)

 +

42) Saving Private Ryan (1998)

 +

41) Schindler's List (1993)

 +

40) Mulholland Drive (2001)

 +

39) Lawrence Of Arabia (1962)

 +

38) The Good, The Bad And The Ugly (1966)

 +

37) Fight Club (1999)

 +

36) Point Break (1991)

 +

35) Mad Max: Fury Road (2015)

 +

34) The Shining (1980)

 +

33) The Lord Of The Rings: The Two Towers (2002)

 +

32) Seven (1995)

 +

31) The Thing (1982)

 +

30) Casablanca (1942)

 +

29) 12 Angry Men (1957)

 +

28) Apocalypse Now (1979)

 +

27) Parasite (2019)

 +

26) Gladiator (2000)

 +

25) 2001: A Space Odyssey (1968)

 +

24) There Will Be Blood (2007)

 +

23) Die Hard (1988)

 +

22) The Matrix (1999)

 +

21) The Lord Of The Rings: The Return Of The King (2003)

 +

20) Heat (1995)

 +

19) Interstellar (2014)

 +

18) Terminator 2: Judgment Day (1991)

 +

17) Alien (1979)

 +

16) The Godfather Part II (1974)

 +

15) Blade Runner (1982)

 +

14) Raiders Of The Lost Ark (1981)

 +

13) Jurassic Park (1993)

 +

12) Back To The Future (1985)

 +

11) Pulp Fiction (1994)

 +

10) The Dark Knight (2008)

 +

9) The Lord Of The Rings: The Fellowship Of The Ring (2001)

 +

8) Star Wars: Episode IV – A New Hope (1977)

 +

7) Goodfellas (1990)

 +

6) Inception (2010)

 +

5) Aliens (1986)

 +

4) Star Wars: The Empire Strikes Back (1980)

 +

3) The Shawshank Redemption (1994)

 +

2) The Godfather (1972)

 +

1) Jaws (1975)

 + diff --git a/tests/integration/test_movies_flow.py b/tests/integration/test_movies_flow.py new file mode 100644 index 0000000..2087320 --- /dev/null +++ b/tests/integration/test_movies_flow.py @@ -0,0 +1,27 @@ +from pathlib import Path + +import pytest + +from scrape_movies.presentation import cli + + +def _run(target: Path, capsys: pytest.CaptureFixture[str]) -> str: + argv = ["movies", "write", "--source", "fixture", "--output", str(target)] + assert cli.main(argv) == 0 + capsys.readouterr() + return target.read_text(encoding="utf-8") + + +def test_write_when_run_from_fixture_creates_100_ascending_lines( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + lines = _run(tmp_path / "movies.txt", capsys).splitlines() + assert len(lines) == 100 + assert [int(line.split(")")[0]) for line in lines] == list(range(1, 101)) + + +def test_write_when_run_twice_produces_identical_files( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + first = _run(tmp_path / "a.txt", capsys) + assert _run(tmp_path / "b.txt", capsys) == first diff --git a/tests/unit/test_cli.py b/tests/unit/test_cli.py index b288338..13246a8 100644 --- a/tests/unit/test_cli.py +++ b/tests/unit/test_cli.py @@ -39,3 +39,10 @@ def test_main_when_every_hacker_news_step_runs_from_fixture_prints_output( for step in hacker_news_steps.STEPS: assert cli.main(["hacker-news", step, "--source", "fixture"]) == 0 assert capsys.readouterr().out.strip(), step + + +def test_main_when_movies_markup_from_fixture_prints_heading( + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(["movies", "markup", "--source", "fixture"]) == 0 + assert "

" in capsys.readouterr().out diff --git a/tests/unit/test_empire_movies.py b/tests/unit/test_empire_movies.py new file mode 100644 index 0000000..8ead7dc --- /dev/null +++ b/tests/unit/test_empire_movies.py @@ -0,0 +1,48 @@ +from scrape_movies.constants import EMPIRE_FIXTURE_PATH +from scrape_movies.infrastructure import empire_movies +from scrape_movies.infrastructure.soup_basics import load_html + +SNAPSHOT_LAYOUT = """ +

4) The Shawshank Redemption

+

3) The Dark Knight

+

not a title

+

2) The Empire Strikes Back

+""" + + +def test_parse_titles_when_snapshot_layout_reads_h3_title_in_page_order() -> None: + assert empire_movies.parse_titles(SNAPSHOT_LAYOUT) == [ + "4) The Shawshank Redemption", + "3) The Dark Knight", + "2) The Empire Strikes Back", + ] + + +def test_parse_titles_when_live_layout_reads_ranked_h2_headings() -> None: + titles = empire_movies.parse_titles(load_html(EMPIRE_FIXTURE_PATH)) + assert len(titles) == 100 + assert titles[0].startswith("100) ") + assert titles[-1].startswith("1) ") + + +def test_parse_titles_when_h2_has_no_rank_ignores_it() -> None: + assert empire_movies.parse_titles("

Jump to

2) Two

") == ["2) Two"] + + +def test_parse_titles_when_page_has_no_titles_returns_empty() -> None: + assert empire_movies.parse_titles("

nothing

") == [] + + +def test_first_title_markup_when_snapshot_layout_shows_h3() -> None: + assert empire_movies.first_title_markup(SNAPSHOT_LAYOUT).startswith( + '

' + ) + + +def test_first_title_markup_when_live_layout_shows_h2() -> None: + markup = empire_movies.first_title_markup(load_html(EMPIRE_FIXTURE_PATH)) + assert markup.startswith("

") + + +def test_first_title_markup_when_nothing_found_returns_empty() -> None: + assert empire_movies.first_title_markup("

x

") == "" diff --git a/tests/unit/test_movie.py b/tests/unit/test_movie.py new file mode 100644 index 0000000..a7d063b --- /dev/null +++ b/tests/unit/test_movie.py @@ -0,0 +1,18 @@ +from scrape_movies.domain.movie import format_titles + + +def test_format_titles_when_titles_are_ranked_reverses_to_ascending() -> None: + page = ["3) Three", "2) Two", "1) One"] + assert format_titles(page) == ["1) One", "2) Two", "3) Three"] + + +def test_format_titles_when_titles_have_no_rank_numbers_them() -> None: + assert format_titles(["Third", "Second", "First"]) == [ + "1) First", + "2) Second", + "3) Third", + ] + + +def test_format_titles_when_list_is_empty_returns_empty() -> None: + assert format_titles([]) == [] diff --git a/tests/unit/test_movie_file.py b/tests/unit/test_movie_file.py new file mode 100644 index 0000000..1e0660a --- /dev/null +++ b/tests/unit/test_movie_file.py @@ -0,0 +1,16 @@ +from pathlib import Path + +from scrape_movies.infrastructure.movie_file import write_movies + + +def test_write_movies_when_lines_given_writes_one_per_line(tmp_path: Path) -> None: + target = tmp_path / "movies.txt" + write_movies(["1) One", "2) Zoë"], target) + assert target.read_bytes() == "1) One\n2) Zoë\n".encode() + + +def test_write_movies_when_file_exists_replaces_it(tmp_path: Path) -> None: + target = tmp_path / "movies.txt" + target.write_text("old\n", encoding="utf-8") + write_movies(["1) New"], target) + assert target.read_text(encoding="utf-8") == "1) New\n" diff --git a/tests/unit/test_use_cases.py b/tests/unit/test_use_cases.py index b2d17f2..427a1f1 100644 --- a/tests/unit/test_use_cases.py +++ b/tests/unit/test_use_cases.py @@ -1,3 +1,6 @@ +from collections.abc import Sequence +from pathlib import Path + from scrape_movies.application import use_cases from scrape_movies.domain.story import Story @@ -22,3 +25,21 @@ def test_find_top_story_when_stories_exist_returns_highest() -> None: def test_find_top_story_when_page_is_empty_returns_none() -> None: assert use_cases.find_top_story(_fetch, lambda html: [], "http://x") is None + + +def test_save_movie_list_when_titles_are_descending_writes_ascending() -> None: + written: list[tuple[list[str], Path]] = [] + + def write(lines: Sequence[str], path: Path) -> None: + written.append((list(lines), path)) + + lines = use_cases.save_movie_list( + _fetch, lambda html: ["2) Two", "1) One"], write, "http://x", Path("m.txt") + ) + assert list(lines) == ["1) One", "2) Two"] + assert written == [(["1) One", "2) Two"], Path("m.txt"))] + + +def test_scrape_movie_titles_when_called_keeps_page_order() -> None: + titles = use_cases.scrape_movie_titles(_fetch, lambda html: ["b", "a"], "http://x") + assert titles == ["b", "a"] diff --git a/tools/capture_page.py b/tools/capture_page.py index 0dd4c82..fd35bc4 100644 --- a/tools/capture_page.py +++ b/tools/capture_page.py @@ -1,56 +1,59 @@ -"""@brief Save a screenshot of a web page with a headless Chromium browser. - -A headless browser draws into memory, so no desktop window is opened and no -mouse or keyboard is used. It needs Edge, Chrome or Chromium on the machine. -""" - -import shutil -import subprocess -from pathlib import Path - -CANDIDATES = ( - r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe", - r"C:\Program Files\Microsoft\Edge\Application\msedge.exe", - r"C:\Program Files\Google\Chrome\Application\chrome.exe", - "google-chrome", - "chromium", - "chromium-browser", -) -WINDOW = "1280,1000" - - -def find_browser() -> str: - """@brief Locate a Chromium-based browser. - - @return The path or command name of the browser. - @throws FileNotFoundError When none is installed. - """ - for candidate in CANDIDATES: - if Path(candidate).exists() or shutil.which(candidate): - return candidate - raise FileNotFoundError("Install Edge, Chrome or Chromium to capture pages.") - - -def capture_page(url: str, target: Path) -> Path: - """@brief Render a page and save it as a PNG. - - @param url The page address. - @param target The PNG file to write. - @return The path of the PNG. - """ - target.parent.mkdir(parents=True, exist_ok=True) - subprocess.run( - [ - find_browser(), - "--headless=new", - "--disable-gpu", - "--hide-scrollbars", - f"--window-size={WINDOW}", - f"--screenshot={target}", - url, - ], - check=True, - capture_output=True, - timeout=120, - ) - return target +"""@brief Save a screenshot of a web page with a headless Chromium browser. + +A headless browser draws into memory, so no desktop window is opened and no +mouse or keyboard is used. It needs Edge, Chrome or Chromium on the machine. +""" + +import shutil +import subprocess +from pathlib import Path + +CANDIDATES = ( + r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe", + r"C:\Program Files\Microsoft\Edge\Application\msedge.exe", + r"C:\Program Files\Google\Chrome\Application\chrome.exe", + "google-chrome", + "chromium", + "chromium-browser", +) +WINDOW = "1280,1000" + + +def find_browser() -> str: + """@brief Locate a Chromium-based browser. + + @return The path or command name of the browser. + @throws FileNotFoundError When none is installed. + """ + for candidate in CANDIDATES: + if Path(candidate).exists() or shutil.which(candidate): + return candidate + raise FileNotFoundError("Install Edge, Chrome or Chromium to capture pages.") + + +def capture_page(url: str, target: Path) -> Path: + """@brief Render a page and save it as a PNG. + + @param url The page address. + @param target The PNG file to write. + @return The path of the PNG. + """ + target.parent.mkdir(parents=True, exist_ok=True) + browser = find_browser() + for mode in ("new", "old"): + result = subprocess.run( + [ + browser, + f"--headless={mode}", + "--disable-gpu", + "--hide-scrollbars", + f"--window-size={WINDOW}", + f"--screenshot={target}", + url, + ], + capture_output=True, + timeout=120, + ) + if result.returncode == 0 and target.exists(): + return target + raise RuntimeError(f"The browser could not capture {url}") diff --git a/tools/make_screenshots.py b/tools/make_screenshots.py index dd527a1..12b1ca6 100644 --- a/tools/make_screenshots.py +++ b/tools/make_screenshots.py @@ -29,10 +29,20 @@ STEPS: dict[str, list[str]] = { "hn-4-links": ["hacker-news", "links"], "hn-5-upvotes": ["hacker-news", "upvotes"], "hn-6-top": ["hacker-news", "top"], + "movies-2-markup": ["movies", "markup", "--source", "fixture"], + "movies-3-titles": ["movies", "titles", "--source", "fixture"], + "movies-4-reversed": ["movies", "reversed", "--source", "fixture"], + "movies-5-write": ["movies", "write", "--source", "fixture"], } +## Screenshots of a file's first lines: image name, file and number of lines. +FILES: dict[str, tuple[str, int]] = {"movies-6-file": ("movies.txt", 15)} ## Screenshots of real web pages: image name and the address of the page. PAGES: dict[str, str] = { "hn-1-page": "https://appbrewery.github.io/news.ycombinator.com/", + # The live page opens a consent wall, so the saved headings are drawn instead. + "movies-1-page": ( + ROOT / "tests" / "fixtures" / "empire_live_excerpt.html" + ).as_uri(), } @@ -63,6 +73,13 @@ def main() -> None: for name, args in STEPS.items(): if name.startswith(prefix): print(capture(name, args)) + for name, (file_name, count) in FILES.items(): + if name.startswith(prefix): + text = (ROOT / file_name).read_text(encoding="utf-8") + target = IMAGES / f"{name}.png" + shown = os.linesep.join(text.splitlines()[:count]) + render(f"head -n {count} {file_name}", shown, target) + print(target) for name, url in PAGES.items(): if name.startswith(prefix): print(capture_page(url, IMAGES / f"{name}.png"))