diff --git a/README.md b/README.md index af7f658..fa75137 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,182 @@ -# 048-scrapping-wiki +# ๐Ÿš€ Scraping Wiki +A small Python project from Udemy's *100 Days of Code: The Complete Python Pro Bootcamp*: it drives Chrome with Selenium, opens the Wikipedia main page and prints the article count. It is for course participants who want to compare solutions and for visitors looking for ideas. + +## ๐Ÿ“š Table of Contents + +- [Overview](#-overview) +- [Requirements](#-requirements) +- [Setup](#-setup) +- [Run](#-run) +- [Tests](#-tests) +- [License](#-license) +- [Links](#-links) +- [Appendix: Project documents](#-appendix-project-documents) + +## ๐Ÿงญ Overview + +The lecture builds `interaction.py` in seven steps: create the file, import Selenium, initialise the Chrome driver, navigate to Wikipedia with `get()`, identify the HTML element with a CSS selector, find it with `find_element` and print its text, and a hint at the next lectures (clicking links and typing into search fields). Each step has a screenshot below and an explanation in the [wiki][wiki] (English, Danish and Bahasa Malaysia). + +| Part | What it is | +| --- | --- | +| `src/scraping_wiki/interaction.py` | The script: `create_driver`, `open_main_page`, `read_article_count`, `main` | +| `src/scraping_wiki/constants.py` | The page URL, the CSS selectors and the Chrome options | +| `tests/unit/` | pytest tests with a fake driver; they start no browser | +| `tests/e2e/` | End to end test and screenshot capture; run only in Docker | +| `docs/lecture-steps/` | The code of each lecture step, as built up in the lecture | +| `docs/screenshots/` | One screenshot per lecture step | +| `Dockerfile` | Headless Chromium for the end to end run | + +### The lecture steps + +**Step 1: create `interaction.py`.** +![Step 1: create the file](docs/screenshots/step-1-create-the-file.png) + +**Step 2: import Selenium.** +![Step 2: import Selenium](docs/screenshots/step-2-import-selenium.png) + +**Step 3: initialise the Chrome driver.** +![Step 3: initialise the Chrome driver](docs/screenshots/step-3-initialise-chrome-driver.png) + +**Step 4: navigate to the Wikipedia main page with `get()`.** +![Step 4: navigate with get](docs/screenshots/step-4-navigate-with-get.png) + +**Step 5: identify the HTML element.** The article count is a link inside the `div` with the id `articlecount`, so the lecture uses the CSS selector `#articlecount a`. Wikipedia has since added a second link in that `div`: the first match (red) is now the number of active editors, and the article count (green) is `#articlecount li:nth-of-type(2) a`, the selector the program uses (see [ADR-0002][ADR-0002]). +![Step 5: identify the element](docs/screenshots/step-5-identify-the-element.png) + +**Step 6: find the element with `find_element` and print its text.** +![Step 6: find the element and print](docs/screenshots/step-6-find-element-and-print.png) + +**Step 7: next lectures: click links and type into search fields.** +![Step 7: future interactions](docs/screenshots/step-7-future-interactions.png) + +## ๐Ÿ“‹ Requirements + +- Python 3.13 or newer +- Google Chrome (to run the script on your own machine; Selenium downloads the matching driver) +- `selenium`, the only runtime dependency (installed with the project) +- `pytest`, `ruff` and `mypy` for the tests and checks (installed with the `dev` extra) +- Optional: [Docker][docker] for the end to end run and the screenshots +- Optional: [Doxygen][doxygen] to build the API documentation + +## ๐Ÿ› ๏ธ Setup + +Create a local virtual environment in the project folder: + +```bash +python -m venv .venv +``` + +Activate it: + +```bash +# Windows (PowerShell) +.venv\Scripts\Activate.ps1 + +# Windows (Git Bash) +source .venv/Scripts/activate + +# Linux / macOS +source .venv/bin/activate +``` + +Upgrade pip and install the project with the test tools: + +```bash +python -m pip install --upgrade pip +python -m pip install -e ".[dev]" +``` + +Leave the environment with `deactivate`. The `.env` file in a clone is for personal tooling only; the project never reads or imports it. + +## โ–ถ๏ธ Run + +With the environment active, this opens Chrome, loads the Wikipedia main page, prints the article count and closes the browser: + +```bash +python -m scraping_wiki.interaction +``` + +The install also adds a console script, and `--headless` runs without a browser window: + +```bash +scraping-wiki --headless +``` + +The output is one line, for example `7,256,081`. + +## ๐Ÿงช Tests + +The unit tests start no browser: + +```bash +python -m pytest +``` + +Style and types: + +```bash +python -m ruff check src tests +python -m mypy src tests +``` + +The end to end test and the screenshots need a browser, so they run in a Docker container and never on your desktop. The results (log, exit code and screenshots) land in `results/`: + +```bash +docker build -t scraping-wiki-e2e . +docker run --rm -v "$(pwd)/results:/results" scraping-wiki-e2e +``` + +Build the API documentation from the Doxygen comments (written to `docs/doxygen/`): + +```bash +doxygen Doxyfile +``` + +## ๐Ÿ“„ License + +[GNU Affero General Public License v3.0][license]. + +## ๐Ÿ”— Links + +- [Repository][repository] +- [Documentation][documentation] +- [Issue tracker][issues] + +## ๐Ÿ“Ž Appendix: Project documents + +The documents a newcomer needs, in reading order. + +| Document | What it tells you | +| --- | --- | +| [AGENTS.md][agents] | The rules AI agents follow in this repository | +| [Business Case][BC-001] | Why the project exists, its scope and success criteria | +| [Stakeholder Analysis][SA-001] | Who is involved; the stakeholder IDs used for owners and reviewers | +| [Project Plan][PP-001] | The gateways and their schedule | +| [Milestones][milestones] | One document per gateway: Go/No-Go criteria and tasks | +| [Architecture decisions][adr] | Why the layout is flat and why the selector changed | +| [Review records][reviews] | The review of each document and of the code | +| [Artifact registry][registry] | Where each artifact type lives, the next version, the PO language and domain | +| [Framework README][framework-readme] | The framework this project uses | +| [Framework changelog][framework-changelog] | What changed in each framework release | + +--- + +[repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/048-scrapping-wiki +[documentation]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/048-scrapping-wiki/wiki +[wiki]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/048-scrapping-wiki/wiki +[issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/048-scrapping-wiki/issues +[license]: ./LICENSE +[docker]: https://docs.docker.com/get-docker/ +[doxygen]: https://www.doxygen.nl/ +[ADR-0002]: ./docs/adr/adr-0002-article-count-selector.md +[agents]: ./AGENTS.md +[BC-001]: ./docs/business-case.md +[SA-001]: ./docs/stakeholder-analysis.md +[PP-001]: ./docs/project-plan.md +[milestones]: ./docs/milestones/ +[adr]: ./docs/adr/ +[reviews]: ./docs/sqa/reviews/ +[registry]: ./docs/artifact-registry.md +[framework-readme]: ./framework/README.md +[framework-changelog]: ./framework/CHANGELOG.md