MIL-002 scrape price (Step 1) #33

Merged
Tirsvad merged 2 commits from mil-002-scrape-price into main 2026-10-28 07:40:45 +01:00
21 changed files with 599 additions and 157 deletions
+9 -1
View File
@@ -19,11 +19,19 @@ The assignment is built in four lecture steps. Each step was delivered on its ow
| Step | What it adds | Status | | Step | What it adds | Status |
| --- | --- | --- | | --- | --- | --- |
| 1 | Scrape the price from the practice page with BeautifulSoup | planned (MIL-002) | | 1 | Scrape the price from the practice page with BeautifulSoup | done |
| 2 | E-mail alert below the target price, secrets in `.env` | planned (MIL-003) | | 2 | E-mail alert below the target price, secrets in `.env` | planned (MIL-003) |
| 3 | Browser-like request headers | planned (MIL-004) | | 3 | Browser-like request headers | planned (MIL-004) |
| 4 | The live Amazon page | planned (MIL-005) | | 4 | The live Amazon page | planned (MIL-005) |
### Step 1: scrape the price
`get_title(html)` and `get_price(html)` in `src/amazon_price_tracker/infrastructure/scraper.py` use BeautifulSoup on the [practice page](https://appbrewery.github.io/instant_pot/). The price element has the class `a-offscreen` and its text is `$99.99`; `split("$")` removes the currency sign and `float()` returns `99.99`.
![The practice page the bot reads](docs/images/step-1-practice-page.png)
![Output of the bot: title and price as a float](docs/images/step-1-run.png)
Repository layout: Repository layout:
| Path | What it is for | | Path | What it is for |
+1 -1
View File
@@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g.
| SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 | | SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 |
| PP | Project Plan | docs/project-plan.md | 002 | | PP | Project Plan | docs/project-plan.md | 002 |
| MIL | Milestone / Gateway | docs/milestones/*.md | 007 | | MIL | Milestone / Gateway | docs/milestones/*.md | 007 |
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 011 | | RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 012 |
## Languages ## Languages
Binary file not shown.

After

Width:  |  Height:  |  Size: 363 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.7 KiB

+62
View File
@@ -0,0 +1,62 @@
# RC-011 Code review of MIL-002
## Metadata
| Key | Value |
| --- | --- |
| ID | RC-011 |
| CrossReference | [MIL-002], [QC-PY-001] |
## Version History
| Date | Status | Author | Reviewer | Change | Commit |
| --- | --- | --- | --- | --- | --- |
| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | AI as stand-in for S01 | Initial version | [5ac60a4] |
---
## Artifact Under Review
- Instance reviewed: [MIL-002]
- Checklist used: [QC-PY-001]
- Scope: full review
- Language and domain: n/a (technical)
- Language reviewer: none
- Mode: autonomous, not independent (AI as stand-in for S01); the author and the stand-in are both the AI working for S01
## Checklist Results
| # | Criterion | Status | Evidence/Notes |
| --- | --- | --- | --- |
| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | Modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE |
| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | Names state purpose (price, target, headers); no abbreviations left unexplained |
| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | ruff format and ruff check pass: 15 pytest tests, ruff 0 findings, mypy strict 0 errors (17 files) |
| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | All signatures annotated; mypy --strict passes: 15 pytest tests, ruff 0 findings, mypy strict 0 errors (17 files) |
| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | Pass | No bare except; errors are specific and chained with `raise ... from` where a cause exists |
| 6 | No mutable default arguments and no shadowed builtins | Pass | No mutable defaults, no shadowed builtins |
| 7 | Files, locks and connections are managed with context managers | Pass | requests.get is used once per call and the response is not kept open; no files are opened |
| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | Public modules and functions have docstrings with Doxygen tags |
| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | Pass | Output uses print only for the assignment's visible result in main; no secret is printed |
| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | No Design Class Diagram exists; the milestone's technical tasks need none (Needs use case: No) |
| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | Tests exist, are named for behaviour and use no network or real mail server: 15 pytest tests, ruff 0 findings, mypy strict 0 errors (17 files) |
| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | mypy --strict passes; no Any |
| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Pass | Dependencies are in pyproject.toml with lower bounds; none unused |
| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | Constants live only in constants.py |
| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | Imports point inward; no cycles |
| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | Ports PageFetcher and ProductParser are narrow Protocols; the use case depends on them, the adapters implement them |
| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | domain, application and infrastructure packages exist; main.py is the composition root; tests mirror them under tests/unit/ |
| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | File names are English snake_case |
## Overall Verdict
Go — all Mandatory criteria pass or are not applicable.
## Action Items
| Action | Owner | Due |
| --- | --- | --- |
| None: no Fail and no Pending criterion | S01 | 2026-10-28 |
---
[MIL-002]: ../../milestones/mil-002-scrape-price.md
[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md
[5ac60a4]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/047-amazon-price-tracker/commit/5ac60a43fc1edac6082b07e5544cc2840968d0a7
+5
View File
@@ -0,0 +1,5 @@
"""Entry point for python -m amazon_price_tracker."""
from amazon_price_tracker.main import main
main()
@@ -0,0 +1 @@
"""Application layer: use cases and the ports they need."""
@@ -0,0 +1,25 @@
"""Ports the use cases need from the outside.
@file ports.py
@brief Protocols for fetching a page and parsing a product from it.
"""
from typing import Protocol
from amazon_price_tracker.domain.product import Product
class PageFetcher(Protocol):
"""Fetches the HTML of a page."""
def fetch(self, url: str) -> str:
"""Return the HTML text of the page at url."""
...
class ProductParser(Protocol):
"""Reads a product from HTML."""
def parse(self, html: str) -> Product:
"""Return the product described by html."""
...
@@ -0,0 +1,20 @@
"""Use case: read a product from a page.
@file read_product.py
@brief Fetches a page and parses the product on it.
"""
from amazon_price_tracker.application.ports import PageFetcher, ProductParser
from amazon_price_tracker.domain.product import Product
def read_product(fetcher: PageFetcher, parser: ProductParser, url: str) -> Product:
"""Fetch the page at url and return its product.
@param fetcher Port that downloads the page.
@param parser Port that reads the product from the HTML.
@param url Address of the product page.
@return The product on the page.
@throws ProductParseError When the page holds no readable product.
"""
return parser.parse(fetcher.fetch(url))
@@ -0,0 +1 @@
"""Domain layer: the product model and its errors."""
+17
View File
@@ -0,0 +1,17 @@
"""Errors of the domain.
@file errors.py
@brief Exceptions raised when a product cannot be read from a page.
"""
class ProductParseError(Exception):
"""A product could not be read from the page."""
class PriceNotFoundError(ProductParseError):
"""The page has no readable price."""
class TitleNotFoundError(ProductParseError):
"""The page has no product title."""
@@ -0,0 +1,18 @@
"""The product model.
@file product.py
@brief Defines the Product value object.
"""
from dataclasses import dataclass
@dataclass(frozen=True, slots=True)
class Product:
"""A product with its title and current price."""
## @brief Product title as shown on the page.
title: str
## @brief Price as a float: the assignment asks for a floating point
# number, so float is used although money is normally Decimal.
price: float
@@ -0,0 +1 @@
"""Infrastructure layer: HTTP and HTML adapters."""
@@ -0,0 +1,24 @@
"""HTTP adapter for the PageFetcher port.
@file http_client.py
@brief Downloads a page with requests.
"""
import requests
from amazon_price_tracker.constants import REQUEST_TIMEOUT_SECONDS
class RequestsPageFetcher:
"""PageFetcher port implemented with requests."""
def fetch(self, url: str) -> str:
"""Return the HTML text of the page at url.
@param url Address of the page.
@return The response body as text.
@throws requests.HTTPError When the server answers with an error status.
"""
response = requests.get(url, timeout=REQUEST_TIMEOUT_SECONDS)
response.raise_for_status()
return response.text
@@ -0,0 +1,65 @@
"""Reads a product from page HTML with BeautifulSoup.
@file scraper.py
@brief Day 47 step 1: get the title and the price as a float.
"""
from bs4 import BeautifulSoup
from amazon_price_tracker.domain.errors import (
PriceNotFoundError,
ProductParseError,
TitleNotFoundError,
)
from amazon_price_tracker.domain.product import Product
def get_title(html: str) -> str:
"""Return the text of the element with id productTitle, whitespace collapsed.
@param html Page HTML.
@return The product title.
@throws TitleNotFoundError When the element is missing.
"""
element = BeautifulSoup(html, "html.parser").select_one("#productTitle")
if element is None:
raise TitleNotFoundError("No element with id 'productTitle' on the page")
return " ".join(element.get_text().split())
def get_price(html: str) -> float:
"""Return the price of the first element with class a-offscreen.
@param html Page HTML.
@return The price as a floating point number.
@throws PriceNotFoundError When the element is missing.
@throws ProductParseError When the price text is not a number.
"""
element = BeautifulSoup(html, "html.parser").select_one(".a-offscreen")
if element is None:
raise PriceNotFoundError("No element with class 'a-offscreen' on the page")
text = element.get_text().strip()
# Text looks like "$99.99": drop the currency sign, as the hint suggests.
amount = text.split("$")[-1].replace(",", "")
try:
return float(amount)
except ValueError as error:
raise ProductParseError(f"Price text is not a number: {text!r}") from error
def parse_product(html: str) -> Product:
"""Return the product described by html.
@param html Page HTML.
@return The product with title and price.
@throws ProductParseError When the title or the price cannot be read.
"""
return Product(title=get_title(html), price=get_price(html))
class SoupProductParser:
"""ProductParser port implemented with BeautifulSoup."""
def parse(self, html: str) -> Product:
"""Return the product described by html."""
return parse_product(html)
+17
View File
@@ -0,0 +1,17 @@
"""Composition root.
@file main.py
@brief Builds the adapters and prints the title and price of the product.
"""
from amazon_price_tracker.application.read_product import read_product
from amazon_price_tracker.constants import PRACTICE_URL
from amazon_price_tracker.infrastructure.http_client import RequestsPageFetcher
from amazon_price_tracker.infrastructure.scraper import SoupProductParser
def main() -> None:
"""Print the title and the price of the practice product."""
product = read_product(RequestsPageFetcher(), SoupProductParser(), PRACTICE_URL)
print(product.title)
print(product.price)
+3
View File
@@ -0,0 +1,3 @@
<!DOCTYPE html>
<html lang="en-us"><head><title>Empty</title></head>
<body><span id="productTitle"> Some product </span></body></html>
+29
View File
@@ -0,0 +1,29 @@
<!DOCTYPE html>
<!-- Trimmed copy of https://appbrewery.github.io/instant_pot/ (markup kept, page reduced). -->
<html lang="en-us">
<head>
<title>Instant Pot - Practice Page</title>
</head>
<body>
<div id="centerCol">
<h1 id="title" class="a-size-large a-spacing-none"> <span id="productTitle"
class="a-size-large product-title-word-break"> Instant Pot Duo Plus 9-in-1
Electric Pressure Cooker, Slow Cooker, Rice Cooker, Steamer, Sauté, Yogurt
Maker, Warmer &amp; Sterilizer, Includes App With Over 800 Recipes, Stainless
Steel, 3 Quart </span> </h1>
</div>
<div data-csa-c-type="widget" data-csa-c-slot-id="apex_dp_offer_display">
<div class="a-section a-spacing-micro">
<span class="a-price aok-align-center" data-a-size="xl" data-a-color="base"><span
class="a-offscreen">$99.99</span><span aria-hidden="true"><span
class="a-price-symbol">$</span><span class="a-price-whole">99<span
class="a-price-decimal">.</span></span><span
class="a-price-fraction">99</span></span></span>
</div>
</div>
<div id="similarItems">
<span class="a-price"><span class="a-offscreen">$16.95</span></span>
<span class="a-price"><span class="a-offscreen">$116</span></span>
</div>
</body>
</html>
@@ -0,0 +1,34 @@
from amazon_price_tracker.application.read_product import read_product
from amazon_price_tracker.domain.product import Product
class FakeFetcher:
def __init__(self, html: str) -> None:
self.html = html
self.urls: list[str] = []
def fetch(self, url: str) -> str:
self.urls.append(url)
return self.html
class FakeParser:
def __init__(self, product: Product) -> None:
self.product = product
self.received: list[str] = []
def parse(self, html: str) -> Product:
self.received.append(html)
return self.product
def test_read_product_fetches_url_and_returns_parsed_product() -> None:
expected = Product(title="Pot", price=9.5)
fetcher = FakeFetcher("<html>page</html>")
parser = FakeParser(expected)
result = read_product(fetcher, parser, "https://example.test/p")
assert result == expected
assert fetcher.urls == ["https://example.test/p"]
assert parser.received == ["<html>page</html>"]
@@ -0,0 +1,46 @@
import pytest
import requests
from amazon_price_tracker.constants import REQUEST_TIMEOUT_SECONDS
from amazon_price_tracker.infrastructure.http_client import RequestsPageFetcher
class FakeResponse:
def __init__(self, text: str, status_code: int) -> None:
self.text = text
self.status_code = status_code
def raise_for_status(self) -> None:
if self.status_code >= 400:
raise requests.HTTPError(f"status {self.status_code}")
def _patch_get(
monkeypatch: pytest.MonkeyPatch, response: FakeResponse
) -> list[tuple[str, float]]:
calls: list[tuple[str, float]] = []
def fake_get(url: str, timeout: float) -> FakeResponse:
calls.append((url, timeout))
return response
monkeypatch.setattr(requests, "get", fake_get)
return calls
def test_fetch_returns_text_using_url_and_timeout(
monkeypatch: pytest.MonkeyPatch,
) -> None:
calls = _patch_get(monkeypatch, FakeResponse("<p>hi</p>", 200))
html = RequestsPageFetcher().fetch("https://example.test/")
assert html == "<p>hi</p>"
assert calls == [("https://example.test/", REQUEST_TIMEOUT_SECONDS)]
def test_fetch_raises_on_http_error(monkeypatch: pytest.MonkeyPatch) -> None:
_patch_get(monkeypatch, FakeResponse("", 404))
with pytest.raises(requests.HTTPError):
RequestsPageFetcher().fetch("https://example.test/")
+66
View File
@@ -0,0 +1,66 @@
from pathlib import Path
import pytest
from amazon_price_tracker.domain.errors import (
PriceNotFoundError,
ProductParseError,
TitleNotFoundError,
)
from amazon_price_tracker.domain.product import Product
from amazon_price_tracker.infrastructure.scraper import (
SoupProductParser,
get_price,
get_title,
parse_product,
)
FIXTURES = Path(__file__).resolve().parents[2] / "fixtures"
def _read(name: str) -> str:
return (FIXTURES / name).read_text(encoding="utf-8")
def test_get_price_returns_first_offscreen_as_float() -> None:
price = get_price(_read("practice_page.html"))
assert price == 99.99
assert isinstance(price, float)
def test_get_title_returns_stripped_text() -> None:
title = get_title(_read("practice_page.html"))
assert title == title.strip()
assert title.startswith("Instant Pot Duo Plus")
def test_get_price_raises_when_price_missing() -> None:
with pytest.raises(PriceNotFoundError):
get_price(_read("no_price_page.html"))
def test_get_title_raises_when_title_missing() -> None:
with pytest.raises(TitleNotFoundError):
get_title("<html></html>")
def test_get_price_raises_when_text_not_a_number() -> None:
html = '<span class="a-offscreen">$abc</span>'
with pytest.raises(ProductParseError):
get_price(html)
def test_parse_product_returns_product() -> None:
product = parse_product(_read("practice_page.html"))
assert isinstance(product, Product)
assert product.price == 99.99
def test_soup_product_parser_parses_html() -> None:
product = SoupProductParser().parse(_read("practice_page.html"))
assert product.price == 99.99