diff --git a/README.md b/README.md index 74211cb..3b87547 100644 --- a/README.md +++ b/README.md @@ -104,6 +104,63 @@ python -m scrape_movies soup select The screenshots are made by `python tools/make_screenshots.py`, which renders the command output as an image; no desktop is captured. +### Lecture: Scraping Hacker News + +The code is `src/scrape_movies/infrastructure/hacker_news.py` (parsing) and `http_client.py` (one `requests.get` with a timeout and a `User-Agent`). By default the commands read the course's static copy of the front page, `--source live` reads and `--source fixture` reads the saved copy in `tests/fixtures/` without any network. + +**1. Look at the page you will scrape** (live data instead of a local file) + +![Step 1: the page](docs/images/hn-1-page.png) + +**2. Inspect the markup** (browser developer tools show the tags to look for: a story row `tr.athing`, its title link `.titleline > a`, and a `span.score` in the next row) + +```bash +python -m scrape_movies hacker-news markup +``` + +![Step 2: markup](docs/images/hn-2-markup.png) + +**3. Titles** (`find_all` on the title links, then `get_text()`) + +```bash +python -m scrape_movies hacker-news titles +``` + +![Step 3: titles](docs/images/hn-3-titles.png) + +**4. Links** (`get("href")` of the same anchors) + +```bash +python -m scrape_movies hacker-news links +``` + +![Step 4: links](docs/images/hn-4-links.png) + +**5. Upvotes** (the text `38 points` is split and converted with `int()` so it can be compared) + +```bash +python -m scrape_movies hacker-news upvotes +``` + +![Step 5: upvotes](docs/images/hn-5-upvotes.png) + +**6. The top story** (Python's `max` over the upvotes) + +```bash +python -m scrape_movies hacker-news top +``` + +![Step 6: top story](docs/images/hn-6-top.png) + +### Lecture: Is web scraping legal and ethical? + +A short summary in our own words; it is not legal advice. + +- **Public, non-copyrighted data** is generally fine to read. Court cases such as Genius v. Google and hiQ v. LinkedIn show that the line is not simple. +- **Copyrighted content** (a video, a blog post) must not be copied, and **data behind a login** is covered by the site's user agreement, so do not scrape it. +- **Commercial use** of scraped data invites expensive disputes even where the scraping itself was allowed. +- **Be a good guest:** prefer an official API when there is one, read the site's `robots.txt` to see what it allows, and send at most one request a minute. This project fetches one page per run and identifies itself with a `User-Agent`. + ## ๐Ÿงช Tests ```bash diff --git a/docs/artifact-registry.md b/docs/artifact-registry.md index 7cdb474..c7a9d18 100644 --- a/docs/artifact-registry.md +++ b/docs/artifact-registry.md @@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g. | SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 | | PP | Project Plan | docs/project-plan.md | 002 | | MIL | Milestone / Gateway | docs/milestones/*.md | 006 | -| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 013 | +| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 014 | ## Languages diff --git a/docs/images/hn-1-page.png b/docs/images/hn-1-page.png new file mode 100644 index 0000000..a1f143d Binary files /dev/null and b/docs/images/hn-1-page.png differ diff --git a/docs/images/hn-2-markup.png b/docs/images/hn-2-markup.png new file mode 100644 index 0000000..c039d7d Binary files /dev/null and b/docs/images/hn-2-markup.png differ diff --git a/docs/images/hn-3-titles.png b/docs/images/hn-3-titles.png new file mode 100644 index 0000000..0a6e558 Binary files /dev/null and b/docs/images/hn-3-titles.png differ diff --git a/docs/images/hn-4-links.png b/docs/images/hn-4-links.png new file mode 100644 index 0000000..bc94736 Binary files /dev/null and b/docs/images/hn-4-links.png differ diff --git a/docs/images/hn-5-upvotes.png b/docs/images/hn-5-upvotes.png new file mode 100644 index 0000000..52bd4c8 Binary files /dev/null and b/docs/images/hn-5-upvotes.png differ diff --git a/docs/images/hn-6-top.png b/docs/images/hn-6-top.png new file mode 100644 index 0000000..8ae3fb0 Binary files /dev/null and b/docs/images/hn-6-top.png differ diff --git a/docs/sqa/reviews/rc-013-mil-003-python.md b/docs/sqa/reviews/rc-013-mil-003-python.md new file mode 100644 index 0000000..246cadc --- /dev/null +++ b/docs/sqa/reviews/rc-013-mil-003-python.md @@ -0,0 +1,62 @@ +# Review of MIL-003 Python code + +## Metadata +| Key | Value | +| --- | --- | +| ID | RC-013 | +| CrossReference | [MIL-003], [QC-PY-001] | + +## Version History +| Date | Status | Author | Reviewer | Change | Commit | +| --- | --- | --- | --- | --- | --- | +| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | [81af6fa] | + +--- + +## Artifact Under Review + +- Instance reviewed: [MIL-003] +- Checklist used: [QC-PY-001] +- Scope: full review. Code reviewed: Story, errors, ports, use_cases, http_client, hacker_news, hacker_news_steps, capture_page, tests/unit. +- Language and domain: n/a (technical type) +- Language reviewer: none +- Mode: autonomous, not independent (reviewer is `AI as stand-in for S01`) + +## Checklist Results + +| # | Criterion | Status | Evidence/Notes | +| --- | --- | --- | --- | +| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE. | +| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | names state purpose; the lecture names (`find_all`, `select_one`) are kept in the helper names. | +| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | `ruff check` and `ruff format --check` are clean with no `noqa`. | +| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | every signature is annotated; `mypy --strict` passes, also in a venv without the third-party packages. | +| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | Pass | `fetch_html` catches only `requests.RequestException` and raises `FetchError` `from` the cause; a test checks `__cause__`. | +| 6 | No mutable default arguments and no shadowed builtins | Pass | no mutable defaults, no shadowed builtins. | +| 7 | Files, locks and connections are managed with context managers | Pass | files are read with `Path.read_text`; the HTTP call is a single `requests.get` with no open session. | +| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | every public module, class and function has a Doxygen docstring saying what it does. | +| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | N-A | no logging; the command-line output is the product, written with `print` in the presentation layer. | +| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | no Design Class Diagram; all tasks are plain technical tasks (Needs Use Case: No). | +| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | tests are named for behaviour, offline and independent of order. | +| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | `mypy --strict` runs over `src`, `tests` and `tools`. | +| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Fail | dependencies are declared with lower bounds, not pinned (see RC-009 action item); none is unused. | +| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | each constant and format is defined once; the step texts are built in one place. | +| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | `domain` imports nothing from other layers; `infrastructure` imports `domain` and `constants`; `presentation` composes them. | +| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | no class hierarchy; use cases take callables (ports), so a new source needs no edit of the use case. | +| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | code is in `domain`, `application`, `infrastructure`, `presentation`; tests are in `tests/unit` (network calls are faked; integration and e2e folders stay empty until needed). | +| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | all names are English lowercase with `_`, no space. | + +## Overall Verdict + +Go โ€” no Mandatory criterion fails and none is Pending. Stand-in review in autonomous mode, not independent; S01 challenges it in the run summary. + +## Action Items + +| Action | Owner | Due | +| --- | --- | --- | +| Pin dependencies or accept lower bounds (open from RC-009, optional criterion 13) | S01 | 2026-11-14 | + +--- + +[MIL-003]: ../../milestones/mil-003-hacker-news.md +[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md +[81af6fa]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/commit/81af6fa2e9675b4a27740caca50b7b4056688399 diff --git a/pyproject.toml b/pyproject.toml index 7e5cd5f..f34c9cd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -38,7 +38,8 @@ extend-exclude = ["framework", ".agents", ".claude"] [tool.ruff.lint] select = ["E", "F", "I", "N", "UP", "B", "D"] -ignore = ["D203", "D213"] +# UP040 off: mypy 1.11 (the pinned CI version) does not support PEP 695 aliases. +ignore = ["D203", "D213", "UP040"] [tool.ruff.lint.per-file-ignores] "tests/**" = ["D"] diff --git a/src/scrape_movies/application/ports.py b/src/scrape_movies/application/ports.py new file mode 100644 index 0000000..83f921b --- /dev/null +++ b/src/scrape_movies/application/ports.py @@ -0,0 +1,11 @@ +"""@brief The ports (interfaces) the use cases need from the outside world.""" + +from collections.abc import Callable +from typing import TypeAlias + +from scrape_movies.domain.story import Story + +## Fetches the HTML text of a page address. +HtmlFetcher: TypeAlias = Callable[[str], str] +## Turns Hacker News HTML into stories. +StoryParser: TypeAlias = Callable[[str], list[Story]] diff --git a/src/scrape_movies/application/use_cases.py b/src/scrape_movies/application/use_cases.py new file mode 100644 index 0000000..e8b1146 --- /dev/null +++ b/src/scrape_movies/application/use_cases.py @@ -0,0 +1,26 @@ +"""@brief Use cases of the scraping exercises.""" + +from scrape_movies.application.ports import HtmlFetcher, StoryParser +from scrape_movies.domain.story import Story, top_story + + +def list_stories(fetch: HtmlFetcher, parse: StoryParser, url: str) -> list[Story]: + """@brief Fetch a Hacker News page and read its stories. + + @param fetch Fetches the HTML of the page. + @param parse Reads the stories from the HTML. + @param url The page address. + @return The stories in page order. + """ + return parse(fetch(url)) + + +def find_top_story(fetch: HtmlFetcher, parse: StoryParser, url: str) -> Story | None: + """@brief Find the story with the most upvotes on a Hacker News page. + + @param fetch Fetches the HTML of the page. + @param parse Reads the stories from the HTML. + @param url The page address. + @return The top story, or None when the page has no stories. + """ + return top_story(list_stories(fetch, parse, url)) diff --git a/src/scrape_movies/constants.py b/src/scrape_movies/constants.py index c0d7156..61c1756 100644 --- a/src/scrape_movies/constants.py +++ b/src/scrape_movies/constants.py @@ -32,3 +32,6 @@ EMPIRE_LIVE_URL: str = "https://www.empireonline.com/movies/features/best-movies ## File the movie list is written to. MOVIES_OUTPUT_FILE: str = "movies.txt" + +## The saved copy of the course's Hacker News page, used offline. +HACKER_NEWS_FIXTURE_PATH: Path = FIXTURES_DIR / "hacker_news.html" diff --git a/src/scrape_movies/domain/errors.py b/src/scrape_movies/domain/errors.py new file mode 100644 index 0000000..a790344 --- /dev/null +++ b/src/scrape_movies/domain/errors.py @@ -0,0 +1,5 @@ +"""@brief Errors the project raises.""" + + +class FetchError(Exception): + """@brief A web page could not be fetched.""" diff --git a/src/scrape_movies/domain/story.py b/src/scrape_movies/domain/story.py new file mode 100644 index 0000000..81a642a --- /dev/null +++ b/src/scrape_movies/domain/story.py @@ -0,0 +1,31 @@ +"""@brief The Story entity (a Hacker News article) and the top-story rule.""" + +from collections.abc import Sequence +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Story: + """@brief An article on the Hacker News front page. + + @param title The headline. + @param link The address the headline points to. + @param upvotes The number of points; 0 when the story shows none. + """ + + title: str + link: str + upvotes: int + + +def top_story(stories: Sequence[Story]) -> Story | None: + """@brief Find the story with the most upvotes. + + @param stories The stories in page order. + @return The first story with the highest score, or None when there are none. + """ + best: Story | None = None + for story in stories: + if best is None or story.upvotes > best.upvotes: + best = story + return best diff --git a/src/scrape_movies/infrastructure/hacker_news.py b/src/scrape_movies/infrastructure/hacker_news.py new file mode 100644 index 0000000..cdb43f5 --- /dev/null +++ b/src/scrape_movies/infrastructure/hacker_news.py @@ -0,0 +1,81 @@ +"""@brief Reading titles, links and upvotes from a Hacker News page.""" + +from bs4 import BeautifulSoup, Tag + +from scrape_movies.domain.story import Story +from scrape_movies.infrastructure.soup_basics import make_soup + +STORY_ROW = "tr.athing" +TITLE_LINK = ".titleline > a" +SCORE = ".score" + + +def _upvotes(row: Tag) -> int: + """@brief Read the score shown in the row after a story, 0 when it has none.""" + following = row.find_next_sibling("tr") + score = following.select_one(SCORE) if isinstance(following, Tag) else None + if score is None: + return 0 + return int(score.get_text().split()[0]) + + +def _story_rows(soup: BeautifulSoup) -> list[Tag]: + return [row for row in soup.select(STORY_ROW) if row.select_one(TITLE_LINK)] + + +def parse_stories(html: str) -> list[Story]: + """@brief Read every story of a front page. + + @param html The HTML of the page. + @return The stories in page order. + """ + stories: list[Story] = [] + for row in _story_rows(make_soup(html)): + anchor = row.select_one(TITLE_LINK) + href = anchor.get("href") if anchor is not None else None + if anchor is not None and isinstance(href, str): + stories.append( + Story(title=anchor.get_text(), link=href, upvotes=_upvotes(row)) + ) + return stories + + +def get_titles(html: str) -> list[str]: + """@brief Return the headlines of a front page. + + @param html The HTML of the page. + @return The titles in page order. + """ + return [story.title for story in parse_stories(html)] + + +def get_links(html: str) -> list[str]: + """@brief Return the addresses the headlines point to. + + @param html The HTML of the page. + @return The links in page order. + """ + return [story.link for story in parse_stories(html)] + + +def get_upvotes(html: str) -> list[int]: + """@brief Return the scores as numbers, not text such as "38 points". + + @param html The HTML of the page. + @return The upvotes in page order. + """ + return [story.upvotes for story in parse_stories(html)] + + +def first_story_markup(html: str) -> str: + """@brief Show the markup of the first story, as seen with "Inspect" in a browser. + + @param html The HTML of the page. + @return The indented HTML of its title row and score row, or "" when none. + """ + rows = _story_rows(make_soup(html)) + if not rows: + return "" + following = rows[0].find_next_sibling("tr") + parts = [rows[0], following] if isinstance(following, Tag) else [rows[0]] + return "\n".join(str(part.prettify()) for part in parts) diff --git a/src/scrape_movies/infrastructure/http_client.py b/src/scrape_movies/infrastructure/http_client.py new file mode 100644 index 0000000..b8710a2 --- /dev/null +++ b/src/scrape_movies/infrastructure/http_client.py @@ -0,0 +1,28 @@ +"""@brief Fetching web pages with Requests.""" + +import requests + +from scrape_movies.constants import REQUEST_TIMEOUT_SECONDS, USER_AGENT +from scrape_movies.domain.errors import FetchError + + +def fetch_html(url: str) -> str: + """@brief Download a page with one GET request. + + Sends a User-Agent and gives up after a timeout. Call it once per run: + scraping more often than once a minute is not polite. + + @param url The address of the page. + @return The HTML text of the response. + @throws FetchError When the request fails or the server answers with an error. + """ + try: + response = requests.get( + url, + headers={"User-Agent": USER_AGENT}, + timeout=REQUEST_TIMEOUT_SECONDS, + ) + response.raise_for_status() + except requests.RequestException as error: + raise FetchError(f"Could not fetch {url}: {error}") from error + return str(response.text) diff --git a/src/scrape_movies/presentation/cli.py b/src/scrape_movies/presentation/cli.py index 5ed8ed1..0b17b87 100644 --- a/src/scrape_movies/presentation/cli.py +++ b/src/scrape_movies/presentation/cli.py @@ -1,36 +1,47 @@ -"""@brief Command-line entry point that runs the lecture steps.""" - -import argparse -from collections.abc import Sequence - -from scrape_movies.presentation import soup_steps - - -def build_parser() -> argparse.ArgumentParser: - """@brief Build the argument parser. - - @return The parser with one sub-command per lecture. - """ - parser = argparse.ArgumentParser( - prog="python -m scrape_movies", - description="Day 45 of 100 Days of Code: web scraping exercises.", - ) - commands = parser.add_subparsers(dest="command") - soup = commands.add_parser("soup", help="parse the sample page website.html") - soup.add_argument("step", choices=list(soup_steps.STEPS)) - return parser - - -def main(argv: Sequence[str] | None = None) -> int: - """@brief Run the command given on the command line. - - @param argv The arguments, or None to read `sys.argv`. - @return The process exit code. - """ - parser = build_parser() - args = parser.parse_args(argv) - if args.command == "soup": - print(soup_steps.render_step(args.step)) - return 0 - parser.print_help() - return 0 +"""@brief Command-line entry point that runs the lecture steps.""" + +import argparse +from collections.abc import Sequence + +from scrape_movies.presentation import hacker_news_steps, soup_steps + + +def build_parser() -> argparse.ArgumentParser: + """@brief Build the argument parser. + + @return The parser with one sub-command per lecture. + """ + parser = argparse.ArgumentParser( + prog="python -m scrape_movies", + description="Day 45 of 100 Days of Code: web scraping exercises.", + ) + commands = parser.add_subparsers(dest="command") + soup = commands.add_parser("soup", help="parse the sample page website.html") + soup.add_argument("step", choices=list(soup_steps.STEPS)) + news = commands.add_parser("hacker-news", help="scrape the Hacker News front page") + news.add_argument("step", choices=hacker_news_steps.STEPS) + news.add_argument( + "--source", + choices=hacker_news_steps.SOURCES, + default="mirror", + help="the course mirror (default), the live site or the saved copy", + ) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + """@brief Run the command given on the command line. + + @param argv The arguments, or None to read `sys.argv`. + @return The process exit code. + """ + parser = build_parser() + args = parser.parse_args(argv) + if args.command == "soup": + print(soup_steps.render_step(args.step)) + return 0 + if args.command == "hacker-news": + print(hacker_news_steps.render_step(args.step, args.source)) + return 0 + parser.print_help() + return 0 diff --git a/src/scrape_movies/presentation/hacker_news_steps.py b/src/scrape_movies/presentation/hacker_news_steps.py new file mode 100644 index 0000000..619afb3 --- /dev/null +++ b/src/scrape_movies/presentation/hacker_news_steps.py @@ -0,0 +1,62 @@ +"""@brief Text output of each "Scraping Hacker News" lecture step.""" + +from scrape_movies.application import use_cases +from scrape_movies.application.ports import HtmlFetcher +from scrape_movies.constants import ( + HACKER_NEWS_FIXTURE_PATH, + HACKER_NEWS_LIVE_URL, + HACKER_NEWS_URL, +) +from scrape_movies.infrastructure import hacker_news, http_client, soup_basics + +## Where a step reads the page: the course mirror, the live site or the saved copy. +SOURCES: tuple[str, ...] = ("mirror", "live", "fixture") +_URLS: dict[str, str] = { + "mirror": HACKER_NEWS_URL, + "live": HACKER_NEWS_LIVE_URL, + "fixture": str(HACKER_NEWS_FIXTURE_PATH), +} + + +def _fetcher(source: str) -> HtmlFetcher: + if source == "fixture": + return lambda path: soup_basics.load_html(HACKER_NEWS_FIXTURE_PATH) + return http_client.fetch_html + + +def _numbered(lines: list[str]) -> str: + return "\n".join(f"{number}. {line}" for number, line in enumerate(lines, 1)) + + +def _run(source: str, step: str) -> str: + fetch, url = _fetcher(source), _URLS[source] + if step == "markup": + return hacker_news.first_story_markup(fetch(url)).rstrip() + stories = use_cases.list_stories(fetch, hacker_news.parse_stories, url) + if step == "titles": + return _numbered([story.title for story in stories]) + if step == "links": + return _numbered([story.link for story in stories]) + if step == "upvotes": + return _numbered([str(story.upvotes) for story in stories]) + top = use_cases.find_top_story(fetch, hacker_news.parse_stories, url) + if top is None: + return "No stories found." + return f"{top.title}\n{top.link}\n{top.upvotes} points" + + +## The lecture steps by command-line name, in lecture order. +STEPS: tuple[str, ...] = ("markup", "titles", "links", "upvotes", "top") + + +def render_step(step: str, source: str) -> str: + """@brief Run one lecture step. + + @param step One of STEPS. + @param source One of SOURCES. + @return The text the step prints. + @throws KeyError When the step or the source is unknown. + """ + if step not in STEPS: + raise KeyError(step) + return _run(source, step) diff --git a/tests/fixtures/hacker_news.html b/tests/fixtures/hacker_news.html new file mode 100644 index 0000000..cddd8f4 --- /dev/null +++ b/tests/fixtures/hacker_news.html @@ -0,0 +1,1987 @@ + + + + + + + + Hacker News + + +
+ + + + + + + + + + + +
+ + + + + + +
+ + + + + + + Hacker News + + new + | past | + comments + | ask | + show | + jobs | + submit + + + + login + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ 1. + +
+ +
+ +
+ 		+ + Joan Feynman 1927-2020 + + ( + + aps.org + + ) + + +
+ + 38 points + by + isaacfrond + + 1 hour ago + + + | + hide + | + 2  comments + +
+ 2. + +
+ +
+ +
+ 		+ + Court dismisses Genius lawsuit over lyrics-scraping by Google + + ( + + techcrunch.com + + ) + + +
+ + 205 points + by + neonsunset + + 32 minutes ago + + + | + hide + | + 3  comments + +
+ 3. + +
+ +
+ +
+ 		+ + Facebook's new policy bans blackface and some Jewish stereotypes + + ( + + mashable.com + + ) + + +
+ + 165 points + by + The_suffocated + + 4 hours ago + + + | + hide + | + 30  comments + +
+ 4. + +
+ +
+ +
+ 		+ + Parallel Seam Carving + + ( + + shwestrick.github.io + + ) + + +
+ + 19 points + by + thunderbong + + 3 hours ago + + + | + hide + | + 11  comments + +
+ 5. + +
+ +
+ +
+ 		+ + Amazon's business model meets Sweden's labor unions + + ( + + politico.eu + + ) + + +
+ + 57 points + by + sohkamyung + + 11 hours ago + + + | + hide + | + 141  comments + +
+ 6. + +
+ +
+ +
+ 		+ + 3D Printing Integrated Circuits: What's Possible Now and in the Future? + + ( + + nano-di.com + + ) + + +
+ + 40 points + by + yamrzou + + 4 hours ago + + + | + hide + | + 4  comments + +
+ 7. + +
+ +
+ +
+ 		+ + Ask HN: How Belarus can keep connected despite internet blackout? + + +
+ + 58 points + by + ingve + + 6 hours ago + + + | + hide + | + 20  comments + +
+ 8. + +
+ +
+ +
+ 		+ + Systems Monitoring with Prometheus and Grafana + + ( + + flightaware.engineering + + ) + + +
+ + 138 points + by + tosh + + 1 hour ago + + + | + hide + | + 19  comments + +
+ 9. + +
+ +
+ +
+ 		+ + A Keyboard with Blank Keycaps Made Me and Expert Typist + + ( + + bojanvidanovic.com + + ) + + +
+ + 29 points + by + taubek + + 4 hours ago + + + | + hide + | + 47  comments + +
+ 10. + +
+ +
+ +
+ 		+ + JuliaDB + + ( + + juliadata.github.io + + ) + + +
+ + 189 points + by + akkartik + + 17 hours ago + + + | + hide + | + 52  comments + +
+ 11. + +
+ +
+ +
+ 		+ + Predictions as a Substitute for Reviews + + ( + + acesounderglass.com + + ) + + +
+ + 46 points + by + cainxinth + + 9 hours ago + + + | + hide + | + 6  comments + +
+ 12. + +
+ +
+ +
+ 		+ + Let's Build a 28-Core Raspberry Pi Cluster + + ( + + ikarus.sg + + ) + + +
+ + 108 points + by + danny00 + + 5 hours ago + + + | + hide + | + 25  comments + +
+ 13. + +
+ +
+ +
+ 		+ + Pumas AI: A platform for pharmaceutical modeling and simulation + + ( + + pumas.ai + + ) + + +
+ + 103 points + by + dargscisyhp + + 7 hours ago + + + | + hide + | + 6  comments + +
+ 14. + +
+ +
+ +
+ 		+ + Mitochondria may hold keys to anxiety and mental health + + ( + + quantamagazine.org + + ) + + +
+ + 201 points + by + skilled + + 1 hour ago + + + | + hide + | + 8  comments + +
+ 15. + +
+ +
+ +
+ 		+ + NetSurf, a multi-platform web browser + + ( + + netsurf-browser.org + + ) + + +
+ + 188 points + by + JNRowe + + 10 hours ago + + + | + hide + | + 18  comments + +
+ 16. + +
+ +
+ +
+ 		+ + Launch HN: Xkit (YC S18) - OAuth infrastructure as a service + + +
+ + 110 points + by + belter + + 7 hours ago + + + | + hide + | + 15  comments + +
+ 17. + +
+ +
+ +
+ 		+ + A broken cable smashed a hole 100 feet wide in the Arecibo Observatory + + ( + + businessinsider.com + + ) + + +
+ + 46 points + by + skilled + + 16 hours ago + + + | + hide + | + 84  comments + +
+ 18. + +
+ +
+ +
+ 		+ + The Sail ISA specification language + + ( + + github.com + + ) + + +
+ + 25 points + by + nequo + + 17 hours ago + + + | + hide + | + 103  comments + +
+ 19. + +
+ +
+ +
+ 		+ + Single Page Applications in Rust + + ( + + sheshbabu.com + + ) + + +
+ + 752 points + by + dsego + + 12 hours ago + + + | + hide + | + 43  comments + +
+ 20. + +
+ +
+ +
+ 		+ + Deplacy: CUI-Based Tree Visualizer for Universal Dependencies + + ( + + github.com + + ) + + +
+ + 3 points + by + edu + + 9 hours ago + + + | + hide + | + 51  comments + +
+ 21. + +
+ +
+ +
+ 		+ + Show HN: Radius โ€“ A Meetup.com alternative + + ( + + radius.to + + ) + + +
+ + 242 points + by + radius89 + + 19 hours ago + + + | + hide + | + 94  comments + +
+ 22. + +
+ +
+ +
+ 		+ + Brain circuit scores identify clinically distinct biotypes in depression /anxiety + + ( + + nature.com + + ) + + +
+ + 161 points + by + MBCook + + 17 hours ago + + + | + hide + | + 65  comments + +
+ 23. + +
+ +
+ +
+ 		+ + Astronomers see a black hole awaken in real time + + ( + + eso.org + + ) + + +
+ + 230 points + by + croes + + 1 day ago + + + | + hide + | + 99  comments + +
+ 24. + +
+ +
+ +
+ 		+ + Safe Superintelligence Inc. + + + ( + + ssi.inc + + ) + + +
+ + 1051 points + by + nick_pou + + 18 hours ago + + + | + hide + | + 889  comments + +
+ 25. + +
+ +
+ +
+ 		+ + How airplanes counteract St. Elmo's Fire during thunderstorms + + ( + + phys.org + + ) + + +
+ + 30 points + by + pseudolus + + 18 hours ago + + + | + hide + | + 122  comments + +
+ 26. + +
+ +
+ +
+ 		+ + Datadog releases Incident Management, Profiler, Error Tracking, and more + + ( + + datadoghq.com + + ) + + +
+ + 156 points + by + sharpshadow + + 21 hours ago + + + | + hide + | + 32  comments + +
+ 27. + +
+ +
+ +
+ 		+ + How to stop procrastinating by using the Fogg Behaviour Model + + ( + + deprocrastination.co + + ) + + +
+ + 585 points + by + bookofjoe + + 19 hours ago + + + | + hide + | + 49  comments + +
+ 29. + +
+ +
+ +
+ 		+ + Mozilla lays off 250 employees while it refocuses on commercial products + + ( + + blog.mozilla.org + + ) + + +
+ + 1312 points + by + chenxy88 + + 9 hours ago + + + | + hide + | + 13  comments + +
+ 28. + +
+ +
+ +
+ 		+ + ZX Spectrum Next - Issue 2 + + ( + + kickstarter.com + + ) + + +
+ + 24 points + by + cyberlimerence + + 19 hours ago + + + | + hide + | + 25  comments + +
+ 30. + +
+ +
+ +
+ 		+ + A network of 17th-century female spies (2019) + + ( + + historytoday.com + + ) + + +
+ + 60 points + by + pepys + + 14 hours ago + + + | + hide + | + 13  comments + +
+ +
+
+ + + + + +
+
+
+ + Guidelines + | FAQ | + Lists | + API | + Security + | Legal | + Apply to YC | + Contact + +
+
+
+ Search: + +
+
+
+
+ + + diff --git a/tests/unit/test_cli.py b/tests/unit/test_cli.py index dd83807..b288338 100644 --- a/tests/unit/test_cli.py +++ b/tests/unit/test_cli.py @@ -1,24 +1,41 @@ -import pytest - -from scrape_movies.presentation import cli, soup_steps - - -def test_main_when_soup_step_is_given_prints_output( - capsys: pytest.CaptureFixture[str], -) -> None: - for step in soup_steps.STEPS: - assert cli.main(["soup", step]) == 0 - assert capsys.readouterr().out.strip(), step - - -def test_main_when_no_command_prints_help( - capsys: pytest.CaptureFixture[str], -) -> None: - assert cli.main([]) == 0 - assert "usage:" in capsys.readouterr().out - - -def test_render_step_when_title_returns_page_title() -> None: - assert soup_steps.render_step("title") == ( - "soup.title.string -> Angela's Personal Site" - ) +import pytest + +from scrape_movies.presentation import cli, hacker_news_steps, soup_steps + + +def test_main_when_soup_step_is_given_prints_output( + capsys: pytest.CaptureFixture[str], +) -> None: + for step in soup_steps.STEPS: + assert cli.main(["soup", step]) == 0 + assert capsys.readouterr().out.strip(), step + + +def test_main_when_no_command_prints_help( + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main([]) == 0 + assert "usage:" in capsys.readouterr().out + + +def test_render_step_when_title_returns_page_title() -> None: + assert soup_steps.render_step("title") == ( + "soup.title.string -> Angela's Personal Site" + ) + + +def test_main_when_hacker_news_top_from_fixture_prints_top_story( + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(["hacker-news", "top", "--source", "fixture"]) == 0 + out = capsys.readouterr().out + assert "Mozilla lays off 250 employees" in out + assert "1312 points" in out + + +def test_main_when_every_hacker_news_step_runs_from_fixture_prints_output( + capsys: pytest.CaptureFixture[str], +) -> None: + for step in hacker_news_steps.STEPS: + assert cli.main(["hacker-news", step, "--source", "fixture"]) == 0 + assert capsys.readouterr().out.strip(), step diff --git a/tests/unit/test_hacker_news.py b/tests/unit/test_hacker_news.py new file mode 100644 index 0000000..9022be8 --- /dev/null +++ b/tests/unit/test_hacker_news.py @@ -0,0 +1,59 @@ +from scrape_movies.constants import HACKER_NEWS_FIXTURE_PATH +from scrape_movies.domain.story import Story +from scrape_movies.infrastructure import hacker_news +from scrape_movies.infrastructure.soup_basics import load_html + +PAGE = """ + + + +
+One
12 points
+A job ad
no score here
+""" + + +def _fixture() -> str: + return load_html(HACKER_NEWS_FIXTURE_PATH) + + +def test_parse_stories_when_page_has_score_reads_integer() -> None: + assert hacker_news.parse_stories(PAGE)[0] == Story( + "One", "https://example.com/one", 12 + ) + + +def test_parse_stories_when_story_has_no_score_uses_zero() -> None: + assert hacker_news.parse_stories(PAGE)[1].upvotes == 0 + + +def test_parse_stories_when_page_has_no_stories_returns_empty() -> None: + assert hacker_news.parse_stories("

nothing

") == [] + + +def test_parse_stories_when_fixture_returns_thirty_stories() -> None: + assert len(hacker_news.parse_stories(_fixture())) == 30 + + +def test_get_titles_when_fixture_starts_with_first_headline() -> None: + assert hacker_news.get_titles(_fixture())[0] == "Joan Feynman 1927-2020" + + +def test_get_links_when_fixture_returns_one_address_per_story() -> None: + links = hacker_news.get_links(_fixture()) + assert len(links) == 30 + assert links[0].startswith("https://www.aps.org/") + + +def test_get_upvotes_when_fixture_returns_integers() -> None: + upvotes = hacker_news.get_upvotes(_fixture()) + assert upvotes[:2] == [38, 205] + assert all(isinstance(points, int) for points in upvotes) + + +def test_first_story_markup_when_page_has_story_shows_title_row() -> None: + assert 'class="titleline"' in hacker_news.first_story_markup(PAGE) + + +def test_first_story_markup_when_page_has_no_story_returns_empty() -> None: + assert hacker_news.first_story_markup("

x

") == "" diff --git a/tests/unit/test_http_client.py b/tests/unit/test_http_client.py new file mode 100644 index 0000000..6aa62a2 --- /dev/null +++ b/tests/unit/test_http_client.py @@ -0,0 +1,59 @@ +from typing import Any + +import pytest +import requests + +from scrape_movies.constants import REQUEST_TIMEOUT_SECONDS, USER_AGENT +from scrape_movies.domain.errors import FetchError +from scrape_movies.infrastructure import http_client + + +class _Response: + def __init__(self, text: str, error: Exception | None = None) -> None: + self.text = text + self._error = error + + def raise_for_status(self) -> None: + if self._error is not None: + raise self._error + + +def test_fetch_html_when_request_succeeds_sends_user_agent_and_timeout( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[dict[str, Any]] = [] + + def fake_get(url: str, **kwargs: Any) -> _Response: + calls.append({"url": url, **kwargs}) + return _Response("") + + monkeypatch.setattr(requests, "get", fake_get) + assert http_client.fetch_html("http://example.test") == "" + assert calls == [ + { + "url": "http://example.test", + "headers": {"User-Agent": USER_AGENT}, + "timeout": REQUEST_TIMEOUT_SECONDS, + } + ] + + +def test_fetch_html_when_server_errors_raises_fetch_error( + monkeypatch: pytest.MonkeyPatch, +) -> None: + error = requests.HTTPError("500") + monkeypatch.setattr(requests, "get", lambda *a, **k: _Response("", error)) + with pytest.raises(FetchError) as caught: + http_client.fetch_html("http://example.test") + assert caught.value.__cause__ is error + + +def test_fetch_html_when_connection_fails_raises_fetch_error( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fail(*args: Any, **kwargs: Any) -> _Response: + raise requests.ConnectionError("down") + + monkeypatch.setattr(requests, "get", fail) + with pytest.raises(FetchError): + http_client.fetch_html("http://example.test") diff --git a/tests/unit/test_story.py b/tests/unit/test_story.py new file mode 100644 index 0000000..96bb35f --- /dev/null +++ b/tests/unit/test_story.py @@ -0,0 +1,15 @@ +from scrape_movies.domain.story import Story, top_story + + +def test_top_story_when_scores_differ_returns_highest() -> None: + stories = [Story("a", "/a", 5), Story("b", "/b", 50), Story("c", "/c", 7)] + assert top_story(stories) == Story("b", "/b", 50) + + +def test_top_story_when_scores_tie_returns_first() -> None: + stories = [Story("a", "/a", 9), Story("b", "/b", 9)] + assert top_story(stories) == Story("a", "/a", 9) + + +def test_top_story_when_no_stories_returns_none() -> None: + assert top_story([]) is None diff --git a/tests/unit/test_use_cases.py b/tests/unit/test_use_cases.py new file mode 100644 index 0000000..b2d17f2 --- /dev/null +++ b/tests/unit/test_use_cases.py @@ -0,0 +1,24 @@ +from scrape_movies.application import use_cases +from scrape_movies.domain.story import Story + + +def _fetch(url: str) -> str: + return f"html of {url}" + + +def _parse(html: str) -> list[Story]: + return [Story(html, "/x", 3), Story("best", "/y", 99)] + + +def test_list_stories_when_called_passes_page_through_parser() -> None: + stories = use_cases.list_stories(_fetch, _parse, "http://x") + assert stories[0].title == "html of http://x" + + +def test_find_top_story_when_stories_exist_returns_highest() -> None: + top = use_cases.find_top_story(_fetch, _parse, "http://x") + assert top == Story("best", "/y", 99) + + +def test_find_top_story_when_page_is_empty_returns_none() -> None: + assert use_cases.find_top_story(_fetch, lambda html: [], "http://x") is None diff --git a/tools/capture_page.py b/tools/capture_page.py new file mode 100644 index 0000000..0dd4c82 --- /dev/null +++ b/tools/capture_page.py @@ -0,0 +1,56 @@ +"""@brief Save a screenshot of a web page with a headless Chromium browser. + +A headless browser draws into memory, so no desktop window is opened and no +mouse or keyboard is used. It needs Edge, Chrome or Chromium on the machine. +""" + +import shutil +import subprocess +from pathlib import Path + +CANDIDATES = ( + r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe", + r"C:\Program Files\Microsoft\Edge\Application\msedge.exe", + r"C:\Program Files\Google\Chrome\Application\chrome.exe", + "google-chrome", + "chromium", + "chromium-browser", +) +WINDOW = "1280,1000" + + +def find_browser() -> str: + """@brief Locate a Chromium-based browser. + + @return The path or command name of the browser. + @throws FileNotFoundError When none is installed. + """ + for candidate in CANDIDATES: + if Path(candidate).exists() or shutil.which(candidate): + return candidate + raise FileNotFoundError("Install Edge, Chrome or Chromium to capture pages.") + + +def capture_page(url: str, target: Path) -> Path: + """@brief Render a page and save it as a PNG. + + @param url The page address. + @param target The PNG file to write. + @return The path of the PNG. + """ + target.parent.mkdir(parents=True, exist_ok=True) + subprocess.run( + [ + find_browser(), + "--headless=new", + "--disable-gpu", + "--hide-scrollbars", + f"--window-size={WINDOW}", + f"--screenshot={target}", + url, + ], + check=True, + capture_output=True, + timeout=120, + ) + return target diff --git a/tools/make_screenshots.py b/tools/make_screenshots.py index 990a025..dd527a1 100644 --- a/tools/make_screenshots.py +++ b/tools/make_screenshots.py @@ -1,55 +1,72 @@ -"""@brief Run the lecture steps and save one terminal screenshot per step. - -Usage: python tools/make_screenshots.py [name-prefix] -The images are written to docs/images/ and used by the README and the wiki. -""" - -import subprocess -import sys -from pathlib import Path - -from render_terminal_png import render - -ROOT = Path(__file__).resolve().parents[1] -IMAGES = ROOT / "docs" / "images" - -## File name (without .png) and the arguments given to `python -m scrape_movies`. -STEPS: dict[str, list[str]] = { - "soup-1-title": ["soup", "title"], - "soup-2-prettify": ["soup", "prettify"], - "soup-3-find-all": ["soup", "find-all"], - "soup-4-find": ["soup", "find"], - "soup-5-select": ["soup", "select"], -} - - -def capture(name: str, args: list[str]) -> Path: - """@brief Run one step and write its screenshot. - - @param name The image name without extension. - @param args The arguments for `python -m scrape_movies`. - @return The path of the PNG. - """ - result = subprocess.run( - [sys.executable, "-m", "scrape_movies", *args], - cwd=ROOT, - capture_output=True, - text=True, - encoding="utf-8", - check=True, - ) - target = IMAGES / f"{name}.png" - render("python -m scrape_movies " + " ".join(args), result.stdout, target) - return target - - -def main() -> None: - """@brief Capture every step whose name starts with the given prefix.""" - prefix = sys.argv[1] if len(sys.argv) > 1 else "" - for name, args in STEPS.items(): - if name.startswith(prefix): - print(capture(name, args)) - - -if __name__ == "__main__": - main() +"""@brief Run the lecture steps and save one terminal screenshot per step. + +Usage: python tools/make_screenshots.py [name-prefix] +The images are written to docs/images/ and used by the README and the wiki. +""" + +import os +import subprocess +import sys +from pathlib import Path + +from capture_page import capture_page +from render_terminal_png import render + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +ROOT = Path(__file__).resolve().parents[1] +IMAGES = ROOT / "docs" / "images" + +## File name (without .png) and the arguments given to `python -m scrape_movies`. +STEPS: dict[str, list[str]] = { + "soup-1-title": ["soup", "title"], + "soup-2-prettify": ["soup", "prettify"], + "soup-3-find-all": ["soup", "find-all"], + "soup-4-find": ["soup", "find"], + "soup-5-select": ["soup", "select"], + "hn-2-markup": ["hacker-news", "markup"], + "hn-3-titles": ["hacker-news", "titles"], + "hn-4-links": ["hacker-news", "links"], + "hn-5-upvotes": ["hacker-news", "upvotes"], + "hn-6-top": ["hacker-news", "top"], +} +## Screenshots of real web pages: image name and the address of the page. +PAGES: dict[str, str] = { + "hn-1-page": "https://appbrewery.github.io/news.ycombinator.com/", +} + + +def capture(name: str, args: list[str]) -> Path: + """@brief Run one step and write its screenshot. + + @param name The image name without extension. + @param args The arguments for `python -m scrape_movies`. + @return The path of the PNG. + """ + result = subprocess.run( + [sys.executable, "-m", "scrape_movies", *args], + cwd=ROOT, + env={**os.environ, "PYTHONIOENCODING": "utf-8"}, + capture_output=True, + text=True, + encoding="utf-8", + check=True, + ) + target = IMAGES / f"{name}.png" + render("python -m scrape_movies " + " ".join(args), result.stdout, target) + return target + + +def main() -> None: + """@brief Capture every step whose name starts with the given prefix.""" + prefix = sys.argv[1] if len(sys.argv) > 1 else "" + for name, args in STEPS.items(): + if name.startswith(prefix): + print(capture(name, args)) + for name, url in PAGES.items(): + if name.startswith(prefix): + print(capture_page(url, IMAGES / f"{name}.png")) + + +if __name__ == "__main__": + main()