MIL-002 scrape price (Step 1) #33

Merged
Tirsvad merged 2 commits from mil-002-scrape-price into main 2026-10-28 07:40:45 +01:00
21 changed files with 598 additions and 157 deletions
Showing only changes of commit 5ac60a43fc - Show all commits
+164 -156
View File
@@ -1,156 +1,164 @@
# 🚀 Amazon Price Tracker # 🚀 Amazon Price Tracker
A Python bot for Day 47 of Udemy's "100 Days of Code: The Complete Python Pro Bootcamp": it reads a product page, extracts the price and sends an e-mail alert when the price drops below a target. Made for course participants who want to compare solutions and for visitors who browse for ideas. A Python bot for Day 47 of Udemy's "100 Days of Code: The Complete Python Pro Bootcamp": it reads a product page, extracts the price and sends an e-mail alert when the price drops below a target. Made for course participants who want to compare solutions and for visitors who browse for ideas.
## 📚 Table of Contents ## 📚 Table of Contents
- [Overview](#-overview) - [Overview](#-overview)
- [Requirements](#-requirements) - [Requirements](#-requirements)
- [Setup](#-setup) - [Setup](#-setup)
- [Run](#-run) - [Run](#-run)
- [Tests](#-tests) - [Tests](#-tests)
- [License](#-license) - [License](#-license)
- [Links](#-links) - [Links](#-links)
- [Appendix: Project documents](#-appendix-project-documents) - [Appendix: Project documents](#-appendix-project-documents)
## 🧭 Overview ## 🧭 Overview
The assignment is built in four lecture steps. Each step was delivered on its own branch and pull request, and each has a screenshot and a wiki page in English, Danish and Bahasa Malaysia that explains what it teaches. The assignment is built in four lecture steps. Each step was delivered on its own branch and pull request, and each has a screenshot and a wiki page in English, Danish and Bahasa Malaysia that explains what it teaches.
| Step | What it adds | Status | | Step | What it adds | Status |
| --- | --- | --- | | --- | --- | --- |
| 1 | Scrape the price from the practice page with BeautifulSoup | planned (MIL-002) | | 1 | Scrape the price from the practice page with BeautifulSoup | done |
| 2 | E-mail alert below the target price, secrets in `.env` | planned (MIL-003) | | 2 | E-mail alert below the target price, secrets in `.env` | planned (MIL-003) |
| 3 | Browser-like request headers | planned (MIL-004) | | 3 | Browser-like request headers | planned (MIL-004) |
| 4 | The live Amazon page | planned (MIL-005) | | 4 | The live Amazon page | planned (MIL-005) |
Repository layout: ### Step 1: scrape the price
| Path | What it is for | `get_title(html)` and `get_price(html)` in `src/amazon_price_tracker/infrastructure/scraper.py` use BeautifulSoup on the [practice page](https://appbrewery.github.io/instant_pot/). The price element has the class `a-offscreen` and its text is `$99.99`; `split("$")` removes the currency sign and `float()` returns `99.99`.
| --- | --- |
| `src/amazon_price_tracker/` | The package; `constants.py` is the only place for constants | ![The practice page the bot reads](docs/images/step-1-practice-page.png)
| `tests/` | `pytest` tests; they never use the network or a real mail server |
| `docs/` | Project documents (Business Case, plan, milestones, reviews) and the wiki pages | ![Output of the bot: title and price as a float](docs/images/step-1-run.png)
| `Doxyfile` | Doxygen configuration for the source comments |
| `.env.example` | The settings you put in your own `.env` | Repository layout:
## 📋 Requirements | Path | What it is for |
| --- | --- |
- Python 3.13 or newer. | `src/amazon_price_tracker/` | The package; `constants.py` is the only place for constants |
- Runtime packages (installed by `pip`): `requests`, `beautifulsoup4`, `python-dotenv`. Nothing else is needed to run the bot. | `tests/` | `pytest` tests; they never use the network or a real mail server |
- Optional, for development: `pytest`, `ruff`, `mypy` (the `dev` extra) and [Doxygen](https://www.doxygen.nl/) for the API documentation. | `docs/` | Project documents (Business Case, plan, milestones, reviews) and the wiki pages |
| `Doxyfile` | Doxygen configuration for the source comments |
## 🛠️ Setup | `.env.example` | The settings you put in your own `.env` |
Create a local virtual environment named `.venv`, activate it and upgrade `pip`: ## 📋 Requirements
```bash - Python 3.13 or newer.
python -m venv .venv - Runtime packages (installed by `pip`): `requests`, `beautifulsoup4`, `python-dotenv`. Nothing else is needed to run the bot.
``` - Optional, for development: `pytest`, `ruff`, `mypy` (the `dev` extra) and [Doxygen](https://www.doxygen.nl/) for the API documentation.
Activate it (Linux and macOS): ## 🛠️ Setup
```bash Create a local virtual environment named `.venv`, activate it and upgrade `pip`:
source .venv/bin/activate
``` ```bash
python -m venv .venv
Activate it (Windows PowerShell): ```
```powershell Activate it (Linux and macOS):
.venv\Scripts\Activate.ps1
``` ```bash
source .venv/bin/activate
Activate it (Windows Git Bash): ```
```bash Activate it (Windows PowerShell):
source .venv/Scripts/activate
``` ```powershell
.venv\Scripts\Activate.ps1
Upgrade `pip` and install the project with the development tools: ```
```bash Activate it (Windows Git Bash):
python -m pip install --upgrade pip
``` ```bash
source .venv/Scripts/activate
```bash ```
python -m pip install -e ".[dev]"
``` Upgrade `pip` and install the project with the development tools:
For use without development tools, install with `python -m pip install -e .`. ```bash
python -m pip install --upgrade pip
Create your own settings file from the example and fill in the values. `.env` is ignored by git; never commit it: ```
```bash ```bash
cp .env.example .env python -m pip install -e ".[dev]"
``` ```
Gmail needs 2-Step Verification and an App Password; use the App Password as `EMAIL_PASSWORD`. SMTP addresses: Gmail `smtp.gmail.com`, Hotmail `smtp.live.com`, Outlook `outlook.office365.com`, Yahoo `smtp.mail.yahoo.com`. The bot connects on port 587. For use without development tools, install with `python -m pip install -e .`.
## ▶️ Run Create your own settings file from the example and fill in the values. `.env` is ignored by git; never commit it:
The runnable steps are added milestone by milestone (see the table above). After step 1 the bot starts with: ```bash
cp .env.example .env
```bash ```
python -m amazon_price_tracker
``` Gmail needs 2-Step Verification and an App Password; use the App Password as `EMAIL_PASSWORD`. SMTP addresses: Gmail `smtp.gmail.com`, Hotmail `smtp.live.com`, Outlook `outlook.office365.com`, Yahoo `smtp.mail.yahoo.com`. The bot connects on port 587.
## 🧪 Tests ## ▶️ Run
```bash The runnable steps are added milestone by milestone (see the table above). After step 1 the bot starts with:
python -m pytest
``` ```bash
python -m amazon_price_tracker
Lint and type checks: ```
```bash ## 🧪 Tests
ruff check .
``` ```bash
python -m pytest
```bash ```
mypy
``` Lint and type checks:
Build the API documentation (written to `docs/doxygen/`, which is git-ignored): ```bash
ruff check .
```bash ```
doxygen Doxyfile
``` ```bash
mypy
## 📄 License ```
GNU Affero General Public License v3.0, see [LICENSE](LICENSE). Build the API documentation (written to `docs/doxygen/`, which is git-ignored):
## 🔗 Links ```bash
doxygen Doxyfile
- [Repository][repository] ```
- [Documentation][documentation]
- [Issue tracker][issues] ## 📄 License
## 📎 Appendix: Project documents GNU Affero General Public License v3.0, see [LICENSE](LICENSE).
The documents a newcomer needs, in reading order. ## 🔗 Links
| Document | What it tells you | - [Repository][repository]
| --- | --- | - [Documentation][documentation]
| [AGENTS.md][agents] | The rules AI agents follow in this repository | - [Issue tracker][issues]
| [Business Case][BC-001] | Why the project exists, its scope and success criteria |
| [Stakeholder Analysis][SA-001] | Who is involved; the stakeholder IDs used for owners and reviewers | ## 📎 Appendix: Project documents
| [Project Plan][PP-001] | The gateways and their schedule |
| [Milestones][milestones] | One document per gateway: Go/No-Go criteria and tasks | The documents a newcomer needs, in reading order.
| [Artifact registry][registry] | Where each artifact type lives, the next version, the PO language and domain |
| [Framework README][framework-readme] | The framework this project uses | | Document | What it tells you |
| [Framework changelog][framework-changelog] | What changed in each framework release | | --- | --- |
| [AGENTS.md][agents] | The rules AI agents follow in this repository |
--- | [Business Case][BC-001] | Why the project exists, its scope and success criteria |
| [Stakeholder Analysis][SA-001] | Who is involved; the stakeholder IDs used for owners and reviewers |
[repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/047-amazon-price-tracker | [Project Plan][PP-001] | The gateways and their schedule |
[documentation]: ./docs/ | [Milestones][milestones] | One document per gateway: Go/No-Go criteria and tasks |
[issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/047-amazon-price-tracker/issues | [Artifact registry][registry] | Where each artifact type lives, the next version, the PO language and domain |
[agents]: ./AGENTS.md | [Framework README][framework-readme] | The framework this project uses |
[BC-001]: ./docs/business-case.md | [Framework changelog][framework-changelog] | What changed in each framework release |
[SA-001]: ./docs/stakeholder-analysis.md
[PP-001]: ./docs/project-plan.md ---
[milestones]: ./docs/milestones/
[registry]: ./docs/artifact-registry.md [repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/047-amazon-price-tracker
[framework-readme]: ./framework/README.md [documentation]: ./docs/
[framework-changelog]: ./framework/CHANGELOG.md [issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/047-amazon-price-tracker/issues
[agents]: ./AGENTS.md
[BC-001]: ./docs/business-case.md
[SA-001]: ./docs/stakeholder-analysis.md
[PP-001]: ./docs/project-plan.md
[milestones]: ./docs/milestones/
[registry]: ./docs/artifact-registry.md
[framework-readme]: ./framework/README.md
[framework-changelog]: ./framework/CHANGELOG.md
+1 -1
View File
@@ -15,7 +15,7 @@ document of a type. `Primary File` may contain a glob (e.g.
| SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 | | SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 |
| PP | Project Plan | docs/project-plan.md | 002 | | PP | Project Plan | docs/project-plan.md | 002 |
| MIL | Milestone / Gateway | docs/milestones/*.md | 007 | | MIL | Milestone / Gateway | docs/milestones/*.md | 007 |
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 011 | | RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 012 |
## Languages ## Languages
Binary file not shown.

After

Width:  |  Height:  |  Size: 363 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.7 KiB

+61
View File
@@ -0,0 +1,61 @@
# RC-011 Code review of MIL-002
## Metadata
| Key | Value |
| --- | --- |
| ID | RC-011 |
| CrossReference | [MIL-002], [QC-PY-001] |
## Version History
| Date | Status | Author | Reviewer | Change | Commit |
| --- | --- | --- | --- | --- | --- |
| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | AI as stand-in for S01 | Initial version | pending |
---
## Artifact Under Review
- Instance reviewed: [MIL-002]
- Checklist used: [QC-PY-001]
- Scope: full review
- Language and domain: n/a (technical)
- Language reviewer: none
- Mode: autonomous, not independent (AI as stand-in for S01); the author and the stand-in are both the AI working for S01
## Checklist Results
| # | Criterion | Status | Evidence/Notes |
| --- | --- | --- | --- |
| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | Modules and functions are snake_case, classes PascalCase, constants UPPER_SNAKE |
| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | Names state purpose (price, target, headers); no abbreviations left unexplained |
| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | ruff format and ruff check pass: 15 pytest tests, ruff 0 findings, mypy strict 0 errors (17 files) |
| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | All signatures annotated; mypy --strict passes: 15 pytest tests, ruff 0 findings, mypy strict 0 errors (17 files) |
| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | Pass | No bare except; errors are specific and chained with `raise ... from` where a cause exists |
| 6 | No mutable default arguments and no shadowed builtins | Pass | No mutable defaults, no shadowed builtins |
| 7 | Files, locks and connections are managed with context managers | Pass | requests.get is used once per call and the response is not kept open; no files are opened |
| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | Public modules and functions have docstrings with Doxygen tags |
| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | Pass | Output uses print only for the assignment's visible result in main; no secret is printed |
| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | No Design Class Diagram exists; the milestone's technical tasks need none (Needs use case: No) |
| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | Tests exist, are named for behaviour and use no network or real mail server: 15 pytest tests, ruff 0 findings, mypy strict 0 errors (17 files) |
| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | mypy --strict passes; no Any |
| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Pass | Dependencies are in pyproject.toml with lower bounds; none unused |
| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | Constants live only in constants.py |
| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | Imports point inward; no cycles |
| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | Ports PageFetcher and ProductParser are narrow Protocols; the use case depends on them, the adapters implement them |
| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | domain, application and infrastructure packages exist; main.py is the composition root; tests mirror them under tests/unit/ |
| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | File names are English snake_case |
## Overall Verdict
Go — all Mandatory criteria pass or are not applicable.
## Action Items
| Action | Owner | Due |
| --- | --- | --- |
| None: no Fail and no Pending criterion | S01 | 2026-10-28 |
---
[MIL-002]: ../../milestones/mil-002-scrape-price.md
[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md
+5
View File
@@ -0,0 +1,5 @@
"""Entry point for python -m amazon_price_tracker."""
from amazon_price_tracker.main import main
main()
@@ -0,0 +1 @@
"""Application layer: use cases and the ports they need."""
@@ -0,0 +1,25 @@
"""Ports the use cases need from the outside.
@file ports.py
@brief Protocols for fetching a page and parsing a product from it.
"""
from typing import Protocol
from amazon_price_tracker.domain.product import Product
class PageFetcher(Protocol):
"""Fetches the HTML of a page."""
def fetch(self, url: str) -> str:
"""Return the HTML text of the page at url."""
...
class ProductParser(Protocol):
"""Reads a product from HTML."""
def parse(self, html: str) -> Product:
"""Return the product described by html."""
...
@@ -0,0 +1,20 @@
"""Use case: read a product from a page.
@file read_product.py
@brief Fetches a page and parses the product on it.
"""
from amazon_price_tracker.application.ports import PageFetcher, ProductParser
from amazon_price_tracker.domain.product import Product
def read_product(fetcher: PageFetcher, parser: ProductParser, url: str) -> Product:
"""Fetch the page at url and return its product.
@param fetcher Port that downloads the page.
@param parser Port that reads the product from the HTML.
@param url Address of the product page.
@return The product on the page.
@throws ProductParseError When the page holds no readable product.
"""
return parser.parse(fetcher.fetch(url))
@@ -0,0 +1 @@
"""Domain layer: the product model and its errors."""
+17
View File
@@ -0,0 +1,17 @@
"""Errors of the domain.
@file errors.py
@brief Exceptions raised when a product cannot be read from a page.
"""
class ProductParseError(Exception):
"""A product could not be read from the page."""
class PriceNotFoundError(ProductParseError):
"""The page has no readable price."""
class TitleNotFoundError(ProductParseError):
"""The page has no product title."""
@@ -0,0 +1,18 @@
"""The product model.
@file product.py
@brief Defines the Product value object.
"""
from dataclasses import dataclass
@dataclass(frozen=True, slots=True)
class Product:
"""A product with its title and current price."""
## @brief Product title as shown on the page.
title: str
## @brief Price as a float: the assignment asks for a floating point
# number, so float is used although money is normally Decimal.
price: float
@@ -0,0 +1 @@
"""Infrastructure layer: HTTP and HTML adapters."""
@@ -0,0 +1,24 @@
"""HTTP adapter for the PageFetcher port.
@file http_client.py
@brief Downloads a page with requests.
"""
import requests
from amazon_price_tracker.constants import REQUEST_TIMEOUT_SECONDS
class RequestsPageFetcher:
"""PageFetcher port implemented with requests."""
def fetch(self, url: str) -> str:
"""Return the HTML text of the page at url.
@param url Address of the page.
@return The response body as text.
@throws requests.HTTPError When the server answers with an error status.
"""
response = requests.get(url, timeout=REQUEST_TIMEOUT_SECONDS)
response.raise_for_status()
return response.text
@@ -0,0 +1,65 @@
"""Reads a product from page HTML with BeautifulSoup.
@file scraper.py
@brief Day 47 step 1: get the title and the price as a float.
"""
from bs4 import BeautifulSoup
from amazon_price_tracker.domain.errors import (
PriceNotFoundError,
ProductParseError,
TitleNotFoundError,
)
from amazon_price_tracker.domain.product import Product
def get_title(html: str) -> str:
"""Return the text of the element with id productTitle, whitespace collapsed.
@param html Page HTML.
@return The product title.
@throws TitleNotFoundError When the element is missing.
"""
element = BeautifulSoup(html, "html.parser").select_one("#productTitle")
if element is None:
raise TitleNotFoundError("No element with id 'productTitle' on the page")
return " ".join(element.get_text().split())
def get_price(html: str) -> float:
"""Return the price of the first element with class a-offscreen.
@param html Page HTML.
@return The price as a floating point number.
@throws PriceNotFoundError When the element is missing.
@throws ProductParseError When the price text is not a number.
"""
element = BeautifulSoup(html, "html.parser").select_one(".a-offscreen")
if element is None:
raise PriceNotFoundError("No element with class 'a-offscreen' on the page")
text = element.get_text().strip()
# Text looks like "$99.99": drop the currency sign, as the hint suggests.
amount = text.split("$")[-1].replace(",", "")
try:
return float(amount)
except ValueError as error:
raise ProductParseError(f"Price text is not a number: {text!r}") from error
def parse_product(html: str) -> Product:
"""Return the product described by html.
@param html Page HTML.
@return The product with title and price.
@throws ProductParseError When the title or the price cannot be read.
"""
return Product(title=get_title(html), price=get_price(html))
class SoupProductParser:
"""ProductParser port implemented with BeautifulSoup."""
def parse(self, html: str) -> Product:
"""Return the product described by html."""
return parse_product(html)
+17
View File
@@ -0,0 +1,17 @@
"""Composition root.
@file main.py
@brief Builds the adapters and prints the title and price of the product.
"""
from amazon_price_tracker.application.read_product import read_product
from amazon_price_tracker.constants import PRACTICE_URL
from amazon_price_tracker.infrastructure.http_client import RequestsPageFetcher
from amazon_price_tracker.infrastructure.scraper import SoupProductParser
def main() -> None:
"""Print the title and the price of the practice product."""
product = read_product(RequestsPageFetcher(), SoupProductParser(), PRACTICE_URL)
print(product.title)
print(product.price)
+3
View File
@@ -0,0 +1,3 @@
<!DOCTYPE html>
<html lang="en-us"><head><title>Empty</title></head>
<body><span id="productTitle"> Some product </span></body></html>
+29
View File
@@ -0,0 +1,29 @@
<!DOCTYPE html>
<!-- Trimmed copy of https://appbrewery.github.io/instant_pot/ (markup kept, page reduced). -->
<html lang="en-us">
<head>
<title>Instant Pot - Practice Page</title>
</head>
<body>
<div id="centerCol">
<h1 id="title" class="a-size-large a-spacing-none"> <span id="productTitle"
class="a-size-large product-title-word-break"> Instant Pot Duo Plus 9-in-1
Electric Pressure Cooker, Slow Cooker, Rice Cooker, Steamer, Sauté, Yogurt
Maker, Warmer &amp; Sterilizer, Includes App With Over 800 Recipes, Stainless
Steel, 3 Quart </span> </h1>
</div>
<div data-csa-c-type="widget" data-csa-c-slot-id="apex_dp_offer_display">
<div class="a-section a-spacing-micro">
<span class="a-price aok-align-center" data-a-size="xl" data-a-color="base"><span
class="a-offscreen">$99.99</span><span aria-hidden="true"><span
class="a-price-symbol">$</span><span class="a-price-whole">99<span
class="a-price-decimal">.</span></span><span
class="a-price-fraction">99</span></span></span>
</div>
</div>
<div id="similarItems">
<span class="a-price"><span class="a-offscreen">$16.95</span></span>
<span class="a-price"><span class="a-offscreen">$116</span></span>
</div>
</body>
</html>
@@ -0,0 +1,34 @@
from amazon_price_tracker.application.read_product import read_product
from amazon_price_tracker.domain.product import Product
class FakeFetcher:
def __init__(self, html: str) -> None:
self.html = html
self.urls: list[str] = []
def fetch(self, url: str) -> str:
self.urls.append(url)
return self.html
class FakeParser:
def __init__(self, product: Product) -> None:
self.product = product
self.received: list[str] = []
def parse(self, html: str) -> Product:
self.received.append(html)
return self.product
def test_read_product_fetches_url_and_returns_parsed_product() -> None:
expected = Product(title="Pot", price=9.5)
fetcher = FakeFetcher("<html>page</html>")
parser = FakeParser(expected)
result = read_product(fetcher, parser, "https://example.test/p")
assert result == expected
assert fetcher.urls == ["https://example.test/p"]
assert parser.received == ["<html>page</html>"]
@@ -0,0 +1,46 @@
import pytest
import requests
from amazon_price_tracker.constants import REQUEST_TIMEOUT_SECONDS
from amazon_price_tracker.infrastructure.http_client import RequestsPageFetcher
class FakeResponse:
def __init__(self, text: str, status_code: int) -> None:
self.text = text
self.status_code = status_code
def raise_for_status(self) -> None:
if self.status_code >= 400:
raise requests.HTTPError(f"status {self.status_code}")
def _patch_get(
monkeypatch: pytest.MonkeyPatch, response: FakeResponse
) -> list[tuple[str, float]]:
calls: list[tuple[str, float]] = []
def fake_get(url: str, timeout: float) -> FakeResponse:
calls.append((url, timeout))
return response
monkeypatch.setattr(requests, "get", fake_get)
return calls
def test_fetch_returns_text_using_url_and_timeout(
monkeypatch: pytest.MonkeyPatch,
) -> None:
calls = _patch_get(monkeypatch, FakeResponse("<p>hi</p>", 200))
html = RequestsPageFetcher().fetch("https://example.test/")
assert html == "<p>hi</p>"
assert calls == [("https://example.test/", REQUEST_TIMEOUT_SECONDS)]
def test_fetch_raises_on_http_error(monkeypatch: pytest.MonkeyPatch) -> None:
_patch_get(monkeypatch, FakeResponse("", 404))
with pytest.raises(requests.HTTPError):
RequestsPageFetcher().fetch("https://example.test/")
+66
View File
@@ -0,0 +1,66 @@
from pathlib import Path
import pytest
from amazon_price_tracker.domain.errors import (
PriceNotFoundError,
ProductParseError,
TitleNotFoundError,
)
from amazon_price_tracker.domain.product import Product
from amazon_price_tracker.infrastructure.scraper import (
SoupProductParser,
get_price,
get_title,
parse_product,
)
FIXTURES = Path(__file__).resolve().parents[2] / "fixtures"
def _read(name: str) -> str:
return (FIXTURES / name).read_text(encoding="utf-8")
def test_get_price_returns_first_offscreen_as_float() -> None:
price = get_price(_read("practice_page.html"))
assert price == 99.99
assert isinstance(price, float)
def test_get_title_returns_stripped_text() -> None:
title = get_title(_read("practice_page.html"))
assert title == title.strip()
assert title.startswith("Instant Pot Duo Plus")
def test_get_price_raises_when_price_missing() -> None:
with pytest.raises(PriceNotFoundError):
get_price(_read("no_price_page.html"))
def test_get_title_raises_when_title_missing() -> None:
with pytest.raises(TitleNotFoundError):
get_title("<html></html>")
def test_get_price_raises_when_text_not_a_number() -> None:
html = '<span class="a-offscreen">$abc</span>'
with pytest.raises(ProductParseError):
get_price(html)
def test_parse_product_returns_product() -> None:
product = parse_product(_read("practice_page.html"))
assert isinstance(product, Product)
assert product.price == 99.99
def test_soup_product_parser_parses_html() -> None:
product = SoupProductParser().parse(_read("practice_page.html"))
assert product.price == 99.99