MIL-002: Add interaction script and tests #24

Merged
Tirsvad merged 2 commits from mil-002-interaction-script into main 2026-10-28 09:13:35 +01:00
10 changed files with 438 additions and 2 deletions
Showing only changes of commit 0f7b1b1e5c - Show all commits
+57
View File
@@ -0,0 +1,57 @@
# ADR-0001 Flat package layout
## Metadata
| Key | Value |
| --- | --- |
| ID | ADR-0001 |
| CrossReference | [BC-001], [MIL-002] |
## Version History
| Date | Status | Author | Reviewer | Change | Commit |
| --- | --- | --- | --- | --- | --- |
| 2026-10-28 | Accepted | Claude (AI) | AI as stand-in for S01 | Initial version; accepted after a Go review (RC-009) | pending |
Allowed Status values: `Proposed`, `Accepted`, `Rejected`, `Deprecated`, `Superseded by ADR-NNNN`.
---
## Context
The Python checklist `QC-PY-001` (criterion 17) expects the layers `domain`, `application`, `infrastructure` and `presentation` under `src/`. The `clean-architecture` skill says a small project may merge `Infrastructure` and `Presentation`, never `Domain`, and that a layer with nothing in it yet is created only when its first file is.
This project is one lecture exercise: about 30 lines that start a Chrome driver, open a page, read one element and print it. It has no business rule, no entity, no storage and no second use case. The stakeholders ask for readable code with the assignment's names (S02, CN-004), and the lecture itself names one file, `interaction.py`.
Options:
1. Full four-layer layout (`domain`, `application`, `infrastructure`, `presentation`) with a port for the driver.
2. A flat package `scraping_wiki` with `constants.py` and `interaction.py`; the Selenium adapter and the console entry point stay together in `interaction.py`.
Forces: readability for S02 and S03, the lecture's file names, the amount of code, and the framework's wish for layers that point inward.
## Decision
Use a flat package `src/scraping_wiki/` with `constants.py` and `interaction.py`, and create no `domain`, `application`, `infrastructure` or `presentation` package until the project has code that belongs in one. `interaction.py` holds the Selenium adapter and the console entry point (the merged infrastructure and presentation layers); there is no domain code.
## Consequences
**Positive:**
- The code matches the lecture and the assignment's names, so S02 can compare it line by line.
- No empty packages and no ports that add indirection without a second implementation.
- Tests use a fake driver without a port, because the functions take a `WebDriver`.
**Negative:**
- Criterion 17 of `QC-PY-001` is met only through this ADR, not by the folder structure.
- Later lectures (clicking links, typing into fields) may need the layers; the structure then has to be refactored and this ADR superseded.
- Selenium types appear in the same module as the console entry point.
## Affected Artifacts
- [BC-001] — keeps the single runtime dependency and the lecture-sized scope
- [MIL-002] — the deliverable `interaction.py` and `constants.py` follow this layout
---
[BC-001]: ../business-case.md
[MIL-002]: ../milestones/mil-002-interaction-script.md
+2 -1
View File
@@ -15,7 +15,8 @@ document of a type. `Primary File` may contain a glob (e.g.
| SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 |
| PP | Project Plan | docs/project-plan.md | 002 |
| MIL | Milestone / Gateway | docs/milestones/*.md | 006 |
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 009 |
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 011 |
| ADR | Architecture Decision Record | docs/adr/adr-NNNN-*.md | 0002 |
## Languages
@@ -0,0 +1,54 @@
# RC-009 Review of ADR-0001
## Metadata
| Key | Value |
| --- | --- |
| ID | RC-009 |
| CrossReference | [ADR-0001], [QC-ADR-001] |
## Version History
| Date | Status | Author | Reviewer | Change | Commit |
| --- | --- | --- | --- | --- | --- |
| 2026-10-28 | Accepted | Claude (AI) | AI as stand-in for S01 | Initial version: Go | pending |
---
## Artifact Under Review
- Instance reviewed: [ADR-0001]
- Checklist used: [QC-ADR-001]
- Scope: full review
- Language and domain: n/a (technical type)
- Language reviewer: none
- Mode: autonomous, not independent (the reviewer is `AI as stand-in for S01`)
## Checklist Results
| # | Criterion | Status | Evidence/Notes |
| --- | --- | --- | --- |
| 1 | The ID (`ADR-NNNN`, 4-digit) matches the filename number and is unique and sequential | Pass | ID ADR-0001 matches the file name adr-0001 and is the first ADR. |
| 2 | Context states the problem, the forces (cost, risk, constraints) and the options evaluated | Pass | Context states the problem, the forces and the two options. |
| 3 | At least two options were considered, or the absence of alternatives is justified | Pass | Two options: four-layer layout, flat package. |
| 4 | Decision is stated in one or two unhedged sentences and matches one of the evaluated options | Pass | Decision is two sentences, unhedged, and is option 2. |
| 5 | Consequences list both Positive and Negative outcomes | Pass | Consequences list three positive and three negative outcomes. |
| 6 | Quality-attribute impacts (performance, security, portability, etc.) are named where the decision affects them | Pass | Maintainability and flexibility impacts are named (later refactoring). |
| 7 | Affected Artifacts are listed as links, and each is also in CrossReference | Pass | Affected Artifacts BC-001 and MIL-002 are links and are in CrossReference. |
| 8 | Status changes are recorded as new Version History rows with a Change summary and Commit link; retained rows are never rewritten | Pass | First version; Version History has one row. |
| 9 | A superseded or deprecated ADR names its successor, and the successor links back | Pass | Not superseded or deprecated; not applicable to the status, so the row passes by absence. |
| 10 | Written in business or technical language of the domain, with no unexplained abbreviations | Pass | Plain technical language; abbreviations are the framework's own IDs. |
| 11 | Status is one of `Proposed`, `Accepted`, `Rejected`, `Deprecated` or `Superseded by ADR-NNNN`, and every Version History row uses only these | Pass | Status is Accepted, an allowed value. |
## Overall Verdict
Go — Every Mandatory criterion of the checklist passes; no Fail. Stand-in review as `AI as stand-in for S01`, not independent: S01 can challenge it in the end-of-run summary. ADR-0001 is the recorded reason for the merged layers that criterion 17 of QC-PY-001 allows.
## Action Items
| Action | Owner | Due |
| --- | --- | --- |
| None | S01 | n/a |
---
[ADR-0001]: ../../adr/adr-0001-flat-package-layout.md
[QC-ADR-001]: ../../../framework/qc/artifacts/qc-adr.md
+61
View File
@@ -0,0 +1,61 @@
# RC-010 Code review of MIL-002 (src and tests)
## Metadata
| Key | Value |
| --- | --- |
| ID | RC-010 |
| CrossReference | [MIL-002], [QC-PY-001] |
## Version History
| Date | Status | Author | Reviewer | Change | Commit |
| --- | --- | --- | --- | --- | --- |
| 2026-10-28 | Accepted | Claude (AI) | AI as stand-in for S01 | Initial version: Go | pending |
---
## Artifact Under Review
- Instance reviewed: [MIL-002]
- Checklist used: [QC-PY-001]
- Scope: full review
- Language and domain: n/a (technical type)
- Language reviewer: none
- Mode: autonomous, not independent (the reviewer is `AI as stand-in for S01`)
## Checklist Results
| # | Criterion | Status | Evidence/Notes |
| --- | --- | --- | --- |
| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | snake_case functions, variables and modules; UPPER_SNAKE constants. |
| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | create_driver, open_main_page, read_article_count state their purpose; no abbreviations or one-letter names. |
| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | ruff format and ruff check (E, F, I, B, UP) are clean; no noqa suppressions. |
| 4 | Every function and method signature is type-annotated, including `-> None` | Pass | Every function, test and fake is annotated, including -> None; mypy --strict is clean. |
| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | Pass | No except clause; main uses try/finally so the driver is always quit; errors propagate (tested). |
| 6 | No mutable default arguments and no shadowed builtins | Pass | The only default is the immutable `headless: bool = False`; no builtins are shadowed. |
| 7 | Files, locks and connections are managed with context managers | Pass | The browser is closed with try/finally and driver.quit(); kept close to the lecture instead of a with block (judgement). |
| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | Modules and all public functions have Doxygen @brief/@param/@return; doxygen Doxyfile builds with 0 warnings. |
| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | Pass | print is the program's console output, which the lecture asks for, not logging; no secrets are handled (judgement). |
| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | Pass | No classes and no Design Class Diagram; the test fakes are not production classes (judgement; passes by absence). |
| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | 9 unit tests named for behaviour with a fake driver; no browser, network or order dependence. |
| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | Pass | mypy --strict is clean; the six type: ignore comments mark fakes standing in for WebDriver; no Any. |
| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Pass | selenium>=4.20 is the only runtime dependency, dev tools are pytest, ruff, mypy; lower bounds rather than exact pins. |
| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | URL, selector, Chrome options and env names are defined once in constants.py; tests import them. |
| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | interaction imports constants and selenium; constants imports nothing; no cycle. |
| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | Pass | Each module and function has one reason to change; no type tests. |
| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | Flat package by decision: ADR-0001 records why no layer packages exist; tests/unit now, tests/e2e in MIL-003. |
| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | All names are English, snake_case, no spaces. |
## Overall Verdict
Go — Every Mandatory criterion of the checklist passes; no Fail. Stand-in review as `AI as stand-in for S01`, not independent: S01 can challenge it in the end-of-run summary. Verified: pytest 9 passed, ruff, mypy strict and doxygen clean.
## Action Items
| Action | Owner | Due |
| --- | --- | --- |
| None | S01 | n/a |
---
[MIL-002]: ../../milestones/mil-002-interaction-script.md
[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md
+13 -1
View File
@@ -12,7 +12,7 @@ license = { text = "AGPL-3.0-or-later" }
dependencies = ["selenium>=4.20"]
[project.optional-dependencies]
dev = ["pytest>=8"]
dev = ["pytest>=8", "ruff>=0.5", "mypy>=1.10"]
[project.scripts]
scraping-wiki = "scraping_wiki.interaction:main"
@@ -23,3 +23,15 @@ where = ["src"]
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
# The e2e tests come in a later milestone and need Docker.
addopts = "--ignore=tests/e2e"
[tool.ruff]
line-length = 100
[tool.ruff.lint]
select = ["E", "F", "I", "B", "UP"]
[tool.mypy]
strict = true
mypy_path = "src"
+30
View File
@@ -0,0 +1,30 @@
"""@file
@brief Constants of the Wikipedia scraper: the page, the selector and the Chrome options.
"""
## @brief The Wikipedia main page that the scraper opens.
WIKIPEDIA_MAIN_PAGE_URL: str = "https://en.wikipedia.org/wiki/Main_Page"
## @brief CSS selector of the anchor inside the div whose id is `articlecount`.
ARTICLE_COUNT_SELECTOR: str = "#articlecount a"
## @brief Chrome option that runs the browser without a window.
HEADLESS_ARGUMENT: str = "--headless=new"
## @brief Chrome option needed when Chromium runs as a user inside a container.
NO_SANDBOX_ARGUMENT: str = "--no-sandbox"
## @brief Chrome option that keeps Chromium from using the small shared memory of a container.
DISABLE_SHM_ARGUMENT: str = "--disable-dev-shm-usage"
## @brief Chrome option that gives a headless run a window size, so screenshots are readable.
WINDOW_SIZE_ARGUMENT: str = "--window-size=1280,1000"
## @brief Environment variable that names the Chrome or Chromium binary to use (optional).
CHROME_BINARY_ENV: str = "CHROME_BINARY"
## @brief Environment variable that names the chromedriver binary to use (optional).
CHROMEDRIVER_PATH_ENV: str = "CHROMEDRIVER_PATH"
## @brief Name of the command line flag that starts the scraper without a browser window.
HEADLESS_FLAG: str = "--headless"
+91
View File
@@ -0,0 +1,91 @@
"""@file
@brief The lecture script: open the Wikipedia main page and print the article count.
Run it with `python -m scraping_wiki.interaction`, or add `--headless` to run
without a browser window (as the Docker run does).
"""
import argparse
import os
from collections.abc import Sequence
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.remote.webdriver import WebDriver
from scraping_wiki.constants import (
ARTICLE_COUNT_SELECTOR,
CHROME_BINARY_ENV,
CHROMEDRIVER_PATH_ENV,
DISABLE_SHM_ARGUMENT,
HEADLESS_ARGUMENT,
HEADLESS_FLAG,
NO_SANDBOX_ARGUMENT,
WIKIPEDIA_MAIN_PAGE_URL,
WINDOW_SIZE_ARGUMENT,
)
def create_driver(headless: bool = False) -> WebDriver:
"""@brief Initialise the Chrome driver.
@param headless True to run Chrome without a window and with the options a container needs.
@return The Chrome driver that controls the browser.
"""
options = webdriver.ChromeOptions()
if headless:
options.add_argument(HEADLESS_ARGUMENT)
options.add_argument(NO_SANDBOX_ARGUMENT)
options.add_argument(DISABLE_SHM_ARGUMENT)
options.add_argument(WINDOW_SIZE_ARGUMENT)
binary = os.environ.get(CHROME_BINARY_ENV)
if binary:
options.binary_location = binary
driver_path = os.environ.get(CHROMEDRIVER_PATH_ENV)
service = Service(executable_path=driver_path) if driver_path else Service()
return webdriver.Chrome(options=options, service=service)
def open_main_page(driver: WebDriver) -> None:
"""@brief Navigate to the Wikipedia main page with `get()`.
@param driver The driver that controls the browser.
"""
driver.get(WIKIPEDIA_MAIN_PAGE_URL)
def read_article_count(driver: WebDriver) -> str:
"""@brief Find the article count link with a CSS selector and return its text.
@param driver A driver that has the Wikipedia main page open.
@return The text of the anchor inside the `articlecount` div, for example `7,000,000`.
"""
article_count = driver.find_element(By.CSS_SELECTOR, ARTICLE_COUNT_SELECTOR)
return article_count.text
def main(argv: Sequence[str] | None = None) -> int:
"""@brief Open the main page, print the article count and close the browser.
@param argv Command line arguments; `--headless` runs without a browser window.
@return The exit code, 0 when the count was printed.
"""
parser = argparse.ArgumentParser(description="Print the Wikipedia article count.")
parser.add_argument(
HEADLESS_FLAG,
action="store_true",
help="run Chrome without a window",
)
arguments = parser.parse_args(argv)
driver = create_driver(headless=arguments.headless)
try:
open_main_page(driver)
print(read_article_count(driver))
finally:
driver.quit()
return 0
if __name__ == "__main__":
raise SystemExit(main())
View File
+11
View File
@@ -0,0 +1,11 @@
"""Tests of the constants: they are the one place that names the page and the selector."""
from scraping_wiki import constants
def test_url_is_the_wikipedia_main_page() -> None:
assert constants.WIKIPEDIA_MAIN_PAGE_URL == "https://en.wikipedia.org/wiki/Main_Page"
def test_selector_targets_the_anchor_in_the_article_count_div() -> None:
assert constants.ARTICLE_COUNT_SELECTOR == "#articlecount a"
+119
View File
@@ -0,0 +1,119 @@
"""Tests of the lecture script. A fake driver stands in for Chrome, so no browser starts."""
import pytest
from selenium import webdriver
from selenium.webdriver.common.by import By
from scraping_wiki import interaction
from scraping_wiki.constants import (
ARTICLE_COUNT_SELECTOR,
CHROME_BINARY_ENV,
CHROMEDRIVER_PATH_ENV,
HEADLESS_ARGUMENT,
WIKIPEDIA_MAIN_PAGE_URL,
)
class FakeElement:
def __init__(self, text: str) -> None:
self.text = text
class FakeDriver:
def __init__(self, count: str = "7,123,456") -> None:
self.count = count
self.visited: list[str] = []
self.searched: list[tuple[str, str]] = []
self.is_quit = False
def get(self, url: str) -> None:
self.visited.append(url)
def find_element(self, by: str, value: str) -> FakeElement:
self.searched.append((by, value))
return FakeElement(self.count)
def quit(self) -> None:
self.is_quit = True
class RecordingChrome:
def __init__(self, options: object, service: object) -> None:
self.options = options
self.service = service
def test_open_main_page_gets_the_wikipedia_url() -> None:
driver = FakeDriver()
interaction.open_main_page(driver) # type: ignore[arg-type]
assert driver.visited == [WIKIPEDIA_MAIN_PAGE_URL]
def test_read_article_count_uses_the_css_selector_and_returns_the_text() -> None:
driver = FakeDriver("7,000,001")
assert interaction.read_article_count(driver) == "7,000,001" # type: ignore[arg-type]
assert driver.searched == [(By.CSS_SELECTOR, ARTICLE_COUNT_SELECTOR)]
def test_main_prints_the_count_and_quits_the_driver(
monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
) -> None:
driver = FakeDriver("6,999,999")
monkeypatch.setattr(interaction, "create_driver", lambda headless=False: driver)
assert interaction.main([]) == 0
assert capsys.readouterr().out == "6,999,999\n"
assert driver.visited == [WIKIPEDIA_MAIN_PAGE_URL]
assert driver.is_quit
def test_main_quits_the_driver_when_reading_fails(monkeypatch: pytest.MonkeyPatch) -> None:
driver = FakeDriver()
def fail(_driver: object) -> str:
raise RuntimeError("element not found")
monkeypatch.setattr(interaction, "create_driver", lambda headless=False: driver)
monkeypatch.setattr(interaction, "read_article_count", fail)
with pytest.raises(RuntimeError):
interaction.main([])
assert driver.is_quit
def test_main_passes_the_headless_flag_to_create_driver(monkeypatch: pytest.MonkeyPatch) -> None:
seen: list[bool] = []
def fake_create(headless: bool = False) -> FakeDriver:
seen.append(headless)
return FakeDriver()
monkeypatch.setattr(interaction, "create_driver", fake_create)
interaction.main(["--headless"])
interaction.main([])
assert seen == [True, False]
def test_create_driver_adds_the_headless_options_only_when_asked(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.delenv(CHROME_BINARY_ENV, raising=False)
monkeypatch.delenv(CHROMEDRIVER_PATH_ENV, raising=False)
monkeypatch.setattr(
webdriver, "Chrome", lambda options, service: RecordingChrome(options, service)
)
plain = interaction.create_driver()
headless = interaction.create_driver(headless=True)
assert HEADLESS_ARGUMENT not in plain.options.arguments # type: ignore[attr-defined]
assert HEADLESS_ARGUMENT in headless.options.arguments # type: ignore[attr-defined]
def test_create_driver_uses_the_binaries_named_in_the_environment(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv(CHROME_BINARY_ENV, "/usr/bin/chromium")
monkeypatch.setenv(CHROMEDRIVER_PATH_ENV, "/usr/bin/chromedriver")
monkeypatch.setattr(
webdriver, "Chrome", lambda options, service: RecordingChrome(options, service)
)
driver = interaction.create_driver()
assert driver.options.binary_location == "/usr/bin/chromium" # type: ignore[attr-defined]
assert driver.service.path == "/usr/bin/chromedriver" # type: ignore[attr-defined]