Modernise Day 48 python.org scraper #1
@@ -0,0 +1,8 @@
|
||||
.env
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
.mypy_cache/
|
||||
*.egg-info/
|
||||
@@ -0,0 +1,3 @@
|
||||
[submodule "framework"]
|
||||
path = framework
|
||||
url = ssh://git@git.tirsystem.com:10022/TirSystem/SQA-QC-Framework.git
|
||||
@@ -1,2 +1,120 @@
|
||||
# 048-scrapping-python-org
|
||||
# 🚀 048 – Scraping python.org with Selenium
|
||||
|
||||
Day 48 of Udemy's *100 Days of Code: The Complete Python Pro Bootcamp*: scrape
|
||||
the five upcoming events from [python.org](https://www.python.org/) with
|
||||
Selenium and print them as a dictionary. This is a modernised version of the
|
||||
original solution.
|
||||
|
||||
## 📚 Table of Contents
|
||||
|
||||
- [Overview](#-overview)
|
||||
- [Requirements](#-requirements)
|
||||
- [Setup](#-setup)
|
||||
- [Run](#-run)
|
||||
- [Tests](#-tests)
|
||||
- [License](#-license)
|
||||
- [Links](#-links)
|
||||
- [Appendix: Project documents](#-appendix-project-documents)
|
||||
|
||||
## 🧭 Overview
|
||||
|
||||
The program opens python.org in headless Chrome, reads the *Upcoming Events*
|
||||
widget and prints:
|
||||
|
||||
```python
|
||||
{0: {'time': '2026-10-29', 'name': 'PyConFR 2026'}, 1: {...}, ..., 4: {...}}
|
||||
```
|
||||
|
||||
| Path | Purpose |
|
||||
| --- | --- |
|
||||
| `src/scraping_python_org/domain` | The `Event` value object |
|
||||
| `src/scraping_python_org/application` | The use case and the `EventSource` port |
|
||||
| `src/scraping_python_org/infrastructure` | Selenium and Chrome adapters |
|
||||
| `src/scraping_python_org/presentation` | Command line entry point and screenshot capture |
|
||||
| `tests/` | Unit tests and a headless browser test against a saved page |
|
||||
| `docs/wiki/` | Wiki in English, Danish and Bahasa Malaysia |
|
||||
|
||||
### The lecture steps
|
||||
|
||||
**1. Open the page.** python.org loads in the browser.
|
||||
|
||||

|
||||
|
||||
**2. Locate the widget.** Inspect the page: the events sit in the
|
||||
`.event-widget` element, each in an `li` with a `time` and an `a` tag.
|
||||
|
||||

|
||||
|
||||
**3. Select the five list items** with the CSS selector `.event-widget li`
|
||||
(outlined in red).
|
||||
|
||||

|
||||
|
||||
**4. Build the dictionary** with a `for` loop and print it.
|
||||
|
||||

|
||||
|
||||
What each step teaches is in the [wiki](docs/wiki/Home.md).
|
||||
|
||||
## 📋 Requirements
|
||||
|
||||
- Python 3.11 or newer (developed on 3.13).
|
||||
- Google Chrome. Selenium Manager downloads a matching driver by itself.
|
||||
|
||||
## 🛠️ Setup
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
source .venv/Scripts/activate # Windows Git Bash; on Linux/macOS: .venv/bin/activate
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
## ▶️ Run
|
||||
|
||||
```bash
|
||||
python -m scraping_python_org
|
||||
```
|
||||
|
||||
Regenerate the screenshots:
|
||||
|
||||
```bash
|
||||
python -m scraping_python_org.presentation.screenshots
|
||||
```
|
||||
|
||||
## 🧪 Tests
|
||||
|
||||
```bash
|
||||
pytest
|
||||
```
|
||||
|
||||
The `browser` test starts headless Chrome on a saved HTML page; skip it with
|
||||
`pytest -m "not browser"`.
|
||||
|
||||
## 📄 License
|
||||
|
||||
See [LICENSE](./LICENSE).
|
||||
|
||||
## 🔗 Links
|
||||
|
||||
- [Repository][repository]
|
||||
- [Documentation][documentation]
|
||||
- [Issue tracker][issues]
|
||||
|
||||
## 📎 Appendix: Project documents
|
||||
|
||||
| Document | What it tells you |
|
||||
| --- | --- |
|
||||
| [AGENTS.md][agents] | The rules AI agents follow in this repository |
|
||||
| [Artifact registry][registry] | The PO language and where artifacts live |
|
||||
| [Framework README][framework-readme] | The framework this project uses |
|
||||
| [Framework changelog][framework-changelog] | What changed in each framework release |
|
||||
|
||||
---
|
||||
|
||||
[repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/048-scrapping-python-org
|
||||
[documentation]: ./docs/wiki/Home.md
|
||||
[issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/048-scrapping-python-org/issues
|
||||
[agents]: ./AGENTS.md
|
||||
[registry]: ./docs/artifact-registry.md
|
||||
[framework-readme]: ./framework/README.md
|
||||
[framework-changelog]: ./framework/CHANGELOG.md
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
# Session 001
|
||||
|
||||
## Prompt 1
|
||||
|
||||
> Day 48 python org. Autonomous mode. Create a Python project (Udemy 100 Days of Code assignment): scrape the five upcoming events from python.org with Selenium into a dictionary `{index: {"time", "name"}}`. We do not make artifacts. Code in src, wiki in docs/wiki in English, Danish and Bahasa Malaysia, a screenshot per lecture step in README and wiki, wiki explains what we gained. Use my solution from github.com/Tirsvad/udemy.python.100-days-of-code (day-48-scrapping-python-org) and modernise it.
|
||||
|
||||
**Outcome:** Plan proposed; PO language set to da. Scraper built under `src/` with tests, README and three-language wiki written.
|
||||
|
||||
## Prompt 2
|
||||
|
||||
> do not make artifacts.
|
||||
|
||||
**Outcome:** Treated as waiver of plan documents. Work continued; nothing pushed.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Artifact Registry
|
||||
|
||||
This project's artifact state. Types, short names and `CrossReference
|
||||
Candidates` come from the framework catalog
|
||||
(`framework/registry/artifact-catalog.md`); this file only records where
|
||||
each document lives in *this* project and the next version to use.
|
||||
|
||||
Delete rows for types you don't use. Add a row the first time you create a
|
||||
document of a type. `Primary File` may contain a glob (e.g.
|
||||
`docs/use-cases/uc-*/uc.md`); `framework/scripts/find-crossreferences.sh` reads it.
|
||||
|
||||
| Short Name | Artifact Type | Primary File | Next Available Version |
|
||||
| --- | --- | --- | --- |
|
||||
| BC | Business Case | docs/business-case.md | 001 |
|
||||
| SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 001 |
|
||||
|
||||
## Languages
|
||||
|
||||
Set the PO language when the project starts; `project-planning` asks for it
|
||||
if it is missing. An artifact of a type marked "Written in the PO language"
|
||||
exists once, in that language, under its normal name (`business-case.md`); the
|
||||
`artifact` skill ("One file per artifact") has the rule.
|
||||
|
||||
| Setting | Value |
|
||||
| --- | --- |
|
||||
| PO language | da |
|
||||
| PO domain | it |
|
||||
| High-level register | IT Executive English |
|
||||
| Technical register | IT Professional English |
|
||||
| Wiki languages | en and the PO language, at least (`framework/process/wiki.md`) |
|
||||
|
||||
Every artifact of a type marked "Yes" below states its language and domain in
|
||||
its `Language` and `Domain` Metadata rows; `new-artifact.sh` fills them from
|
||||
`PO language` and `PO domain`. `Language` is a BCP 47 code (`da`, `en`).
|
||||
`Domain` is a value from this list; add a row to introduce a domain, so a
|
||||
reviewer can see at once which professional vocabulary a document uses.
|
||||
|
||||
| Domain | Meaning |
|
||||
| --- | --- |
|
||||
| it | Software and IT |
|
||||
| medical | Healthcare and medical devices |
|
||||
| construction | Construction and civil engineering |
|
||||
|
||||
| Artifact types | Register | Written in the PO language |
|
||||
| --- | --- | --- |
|
||||
| BC, KPI, PP, MIL | IT Executive English | Yes |
|
||||
| SA, BMC, BPMN, UCD, US, UC, SSD, DM, RA, GOV, DICT | IT Professional English | Yes |
|
||||
| OC, SD, DCD, ERD, TP, AD, ADR, TM, RC, QC, source code | IT Professional English | No |
|
||||
|
||||
## Notes
|
||||
|
||||
- "Next Available Version" is the zero-padded (3-digit) version to use the
|
||||
*next* time a new document of that type is created. Increment it only when
|
||||
a brand-new document is created, not when an existing document's
|
||||
`## Version History` gets a row.
|
||||
- `ADR` uses 4 digits (`0001`); `RC` is sequential across all artifact types.
|
||||
|
After Width: | Height: | Size: 138 KiB |
|
After Width: | Height: | Size: 24 KiB |
|
After Width: | Height: | Size: 25 KiB |
|
After Width: | Height: | Size: 14 KiB |
@@ -0,0 +1,21 @@
|
||||
[English](Home) · **Dansk** · [Bahasa Malaysia](Home-ms)
|
||||
|
||||
# Dag 48 – Scraping af python.org med Selenium
|
||||
|
||||
I denne lektion bruger vi Selenium til at læse de fem kommende begivenheder på
|
||||
[python.org](https://www.python.org/) og samle dem i en dictionary:
|
||||
|
||||
```python
|
||||
{0: {"time": "2026-10-29", "name": "PyConFR 2026"}, 1: {...}, ...}
|
||||
```
|
||||
|
||||
## Sider
|
||||
|
||||
- [Lektionens trin](Lecture-Steps-da): de fire trin, hver med et skærmbillede og hvad du får ud af det.
|
||||
- [Modernisering](Modernisation-da): hvad der er ændret i forhold til den oprindelige løsning, og hvorfor.
|
||||
|
||||
## Hvad du får ud af denne lektion
|
||||
|
||||
- Du læser en websides HTML-struktur med browserens inspektionsværktøj.
|
||||
- Du vælger locators (`By.CSS_SELECTOR`, `By.TAG_NAME`), der er korte og stabile.
|
||||
- Du omsætter scrapede elementer til ren, struktureret Python-data.
|
||||
@@ -0,0 +1,22 @@
|
||||
[English](Home) · [Dansk](Home-da) · **Bahasa Malaysia**
|
||||
|
||||
# Hari 48 – Mengikis python.org dengan Selenium
|
||||
|
||||
Dalam pelajaran ini kita menggunakan Selenium untuk membaca lima acara akan
|
||||
datang di [python.org](https://www.python.org/) dan menyusunnya dalam kamus
|
||||
(dictionary):
|
||||
|
||||
```python
|
||||
{0: {"time": "2026-10-29", "name": "PyConFR 2026"}, 1: {...}, ...}
|
||||
```
|
||||
|
||||
## Halaman
|
||||
|
||||
- [Langkah pelajaran](Lecture-Steps-ms): empat langkah, setiap satu dengan tangkapan skrin dan apa yang anda peroleh.
|
||||
- [Pemodenan](Modernisation-ms): apa yang berubah daripada penyelesaian asal dan sebabnya.
|
||||
|
||||
## Apa yang anda peroleh daripada pelajaran ini
|
||||
|
||||
- Anda membaca struktur HTML sesebuah laman web dengan alat pemeriksa pelayar.
|
||||
- Anda memilih locator (`By.CSS_SELECTOR`, `By.TAG_NAME`) yang ringkas dan stabil.
|
||||
- Anda menukar elemen yang dikikis kepada data Python yang bersih dan tersusun.
|
||||
@@ -0,0 +1,21 @@
|
||||
**English** · [Dansk](Home-da) · [Bahasa Malaysia](Home-ms)
|
||||
|
||||
# Day 48 – Scraping python.org with Selenium
|
||||
|
||||
In this lecture we use Selenium to read the five upcoming events from
|
||||
[python.org](https://www.python.org/) and organise them in a dictionary:
|
||||
|
||||
```python
|
||||
{0: {"time": "2026-10-29", "name": "PyConFR 2026"}, 1: {...}, ...}
|
||||
```
|
||||
|
||||
## Pages
|
||||
|
||||
- [Lecture steps](Lecture-Steps): the four steps, each with a screenshot and what you gain from it.
|
||||
- [Modernisation](Modernisation): what changed from the original solution and why.
|
||||
|
||||
## What you gain in this lecture
|
||||
|
||||
- You read a web page's HTML structure with the browser's inspector.
|
||||
- You choose locators (`By.CSS_SELECTOR`, `By.TAG_NAME`) that are short and stable.
|
||||
- You turn scraped elements into clean, structured Python data.
|
||||
@@ -0,0 +1,43 @@
|
||||
[English](Lecture-Steps) · **Dansk** · [Bahasa Malaysia](Lecture-Steps-ms)
|
||||
|
||||
# Lektionens trin
|
||||
|
||||
## Trin 1 – Åbn siden
|
||||
|
||||

|
||||
|
||||
`driver.get("https://www.python.org/")` indlæser siden i Chrome.
|
||||
|
||||
**Hvad du får ud af det:** driveren er din fjernbetjening til en rigtig browser.
|
||||
Den skal altid lukkes igen, så den moderne kode bruger `try/finally`.
|
||||
|
||||
## Trin 2 – Find widgetten med inspektionsværktøjet
|
||||
|
||||

|
||||
|
||||
Højreklik på widgetten og vælg *Inspect*. Begivenhederne ligger i et element med
|
||||
klassen `event-widget`; hver begivenhed er en `li` med et `time`- og et `a`-tag.
|
||||
|
||||
**Hvad du får ud af det:** du lærer at læse HTML-træet og finde den beholder,
|
||||
der rummer præcis de data, du vil have.
|
||||
|
||||
## Trin 3 – Vælg de fem listepunkter med en CSS-selector
|
||||
|
||||

|
||||
|
||||
`driver.find_elements(By.CSS_SELECTOR, ".event-widget li")` returnerer de fem
|
||||
punkter. En klassebaseret selector er kortere og mere stabil end den lange
|
||||
XPath, man kopierer fra inspektøren, og den rammer ikke urelaterede links.
|
||||
|
||||
**Hvad du får ud af det:** du kender Seleniums locator-strategier og ved,
|
||||
hvornår CSS-selectors slår XPath.
|
||||
|
||||
## Trin 4 – Byg dictionary'en
|
||||
|
||||

|
||||
|
||||
For hvert punkt læser vi `time` og `a` og gemmer dem under punktets indeks, med
|
||||
`enumerate` i stedet for en manuel tæller.
|
||||
|
||||
**Hvad du får ud af det:** du omsætter sideelementer til struktureret data: en
|
||||
dictionary af dictionaries, klar til brug i resten af et program.
|
||||
@@ -0,0 +1,43 @@
|
||||
[English](Lecture-Steps) · [Dansk](Lecture-Steps-da) · **Bahasa Malaysia**
|
||||
|
||||
# Langkah pelajaran
|
||||
|
||||
## Langkah 1 – Buka laman
|
||||
|
||||

|
||||
|
||||
`driver.get("https://www.python.org/")` memuatkan laman dalam Chrome.
|
||||
|
||||
**Apa yang anda peroleh:** driver ialah alat kawalan jauh untuk pelayar sebenar.
|
||||
Ia mesti ditutup semula, jadi kod moden menggunakan `try/finally`.
|
||||
|
||||
## Langkah 2 – Cari widget dengan alat pemeriksa
|
||||
|
||||

|
||||
|
||||
Klik kanan pada widget dan pilih *Inspect*. Acara berada dalam elemen berkelas
|
||||
`event-widget`; setiap acara ialah `li` yang mengandungi teg `time` dan `a`.
|
||||
|
||||
**Apa yang anda peroleh:** anda belajar membaca pepohon HTML dan mencari
|
||||
bekas yang menyimpan data yang tepat seperti yang dikehendaki.
|
||||
|
||||
## Langkah 3 – Pilih lima item senarai dengan pemilih CSS
|
||||
|
||||

|
||||
|
||||
`driver.find_elements(By.CSS_SELECTOR, ".event-widget li")` mengembalikan lima
|
||||
item. Pemilih berasaskan kelas lebih pendek dan lebih stabil daripada XPath
|
||||
panjang yang disalin dari pemeriksa, dan ia tidak memadankan pautan lain.
|
||||
|
||||
**Apa yang anda peroleh:** anda mengenali strategi locator Selenium dan bila
|
||||
pemilih CSS lebih baik daripada XPath.
|
||||
|
||||
## Langkah 4 – Bina kamus
|
||||
|
||||

|
||||
|
||||
Untuk setiap item kita baca `time` dan `a` lalu simpan di bawah indeks item itu,
|
||||
dengan `enumerate` dan bukannya pembilang manual.
|
||||
|
||||
**Apa yang anda peroleh:** anda menukar elemen halaman kepada data tersusun:
|
||||
kamus dalam kamus, sedia digunakan dalam bahagian program yang lain.
|
||||
@@ -0,0 +1,44 @@
|
||||
**English** · [Dansk](Lecture-Steps-da) · [Bahasa Malaysia](Lecture-Steps-ms)
|
||||
|
||||
# Lecture steps
|
||||
|
||||
## Step 1 – Open the page
|
||||
|
||||

|
||||
|
||||
`driver.get("https://www.python.org/")` loads the page in Chrome.
|
||||
|
||||
**What you gain:** the driver is your remote control for a real browser. It
|
||||
must always be closed again, so the modern code wraps it in `try/finally`.
|
||||
|
||||
## Step 2 – Locate the widget with the inspector
|
||||
|
||||

|
||||
|
||||
Right-click the widget and choose *Inspect*. The events live inside an element
|
||||
with the class `event-widget`; each event is an `li` holding a `time` and an
|
||||
`a` tag.
|
||||
|
||||
**What you gain:** you learn to read the HTML tree and find the container that
|
||||
holds exactly the data you want.
|
||||
|
||||
## Step 3 – Select the five list items with a CSS selector
|
||||
|
||||

|
||||
|
||||
`driver.find_elements(By.CSS_SELECTOR, ".event-widget li")` returns the five
|
||||
items. A class-based selector is shorter and more stable than the long XPath
|
||||
copied from the inspector, and it does not match unrelated links.
|
||||
|
||||
**What you gain:** you know the Selenium locator strategies and when CSS
|
||||
selectors beat XPath.
|
||||
|
||||
## Step 4 – Build the dictionary
|
||||
|
||||

|
||||
|
||||
For every item we read `time` and `a` and store them under the item's index,
|
||||
with `enumerate` instead of a manual counter.
|
||||
|
||||
**What you gain:** you turn page elements into structured data: a dictionary
|
||||
of dictionaries, ready to use in the rest of a program.
|
||||
@@ -0,0 +1,16 @@
|
||||
[English](Modernisation) · **Dansk** · [Bahasa Malaysia](Modernisation-ms)
|
||||
|
||||
# Modernisering
|
||||
|
||||
| Oprindelig løsning | Moderne løsning | Hvorfor |
|
||||
| --- | --- | --- |
|
||||
| Én `main.py` | Lag i `src/scraping_python_org` (domain, application, infrastructure, presentation) | Forretningsreglerne afhænger ikke af Selenium; use casen testes med en fake |
|
||||
| Lang absolut XPath | CSS-selector `.event-widget li` | Kort, læsbar, tåler ændringer i layoutet |
|
||||
| Manuel tæller `c` | `enumerate` | Mindre kode, ingen fejl med off-by-one |
|
||||
| `detach=True` efterfulgt af `driver.quit()` | Headless Chrome i en context manager, der altid lukker | Ingen efterladte browservinduer, kører uden skrivebord |
|
||||
| `selenium` uden version i `requirements.txt` | `pyproject.toml` med minimumsversioner | Reproducerbare installationer |
|
||||
| Ingen tests | Enhedstests og en headless browsertest på en gemt side | Ændringer er trygge |
|
||||
| Ingen typeannotationer | `mypy --strict`, `ruff` | Fejl findes før kørsel |
|
||||
|
||||
**Hvad du får ud af det:** en scraper, du kan ændre med ro i sindet; samme idé
|
||||
som i lektionen, men bygget som et rigtigt projekt.
|
||||
@@ -0,0 +1,16 @@
|
||||
[English](Modernisation) · [Dansk](Modernisation-da) · **Bahasa Malaysia**
|
||||
|
||||
# Pemodenan
|
||||
|
||||
| Penyelesaian asal | Penyelesaian moden | Sebab |
|
||||
| --- | --- | --- |
|
||||
| Satu `main.py` | Lapisan dalam `src/scraping_python_org` (domain, application, infrastructure, presentation) | Peraturan perniagaan tidak bergantung pada Selenium; use case diuji dengan palsu (fake) |
|
||||
| XPath mutlak yang panjang | Pemilih CSS `.event-widget li` | Pendek, mudah dibaca, tahan perubahan reka letak |
|
||||
| Pembilang manual `c` | `enumerate` | Kod lebih sedikit, tiada ralat off-by-one |
|
||||
| `detach=True` diikuti `driver.quit()` | Chrome headless dalam context manager yang sentiasa menutup | Tiada tetingkap pelayar tertinggal, berjalan tanpa desktop |
|
||||
| `selenium` tanpa versi dalam `requirements.txt` | `pyproject.toml` dengan versi minimum | Pemasangan yang boleh diulang |
|
||||
| Tiada ujian | Ujian unit dan ujian pelayar headless pada halaman tersimpan | Perubahan selamat |
|
||||
| Tiada petunjuk jenis | `mypy --strict`, `ruff` | Ralat ditemui sebelum dijalankan |
|
||||
|
||||
**Apa yang anda peroleh:** pengikis yang boleh diubah dengan yakin; idea yang
|
||||
sama seperti dalam pelajaran, tetapi dibina seperti projek sebenar.
|
||||
@@ -0,0 +1,16 @@
|
||||
**English** · [Dansk](Modernisation-da) · [Bahasa Malaysia](Modernisation-ms)
|
||||
|
||||
# Modernisation
|
||||
|
||||
| Original solution | Modern solution | Why |
|
||||
| --- | --- | --- |
|
||||
| One `main.py` | Layers in `src/scraping_python_org` (domain, application, infrastructure, presentation) | Business rules do not depend on Selenium; the use case is tested with a fake |
|
||||
| Long absolute XPath | CSS selector `.event-widget li` | Short, readable, survives layout changes |
|
||||
| Manual counter `c` | `enumerate` | Less code, no off-by-one risk |
|
||||
| `detach=True` followed by `driver.quit()` | Headless Chrome in a context manager that always quits | No stray browser windows, runs without a desktop |
|
||||
| Unpinned `selenium` in `requirements.txt` | `pyproject.toml` with minimum versions | Reproducible installs |
|
||||
| No tests | Unit tests plus a headless browser test on a saved page | Changes are safe |
|
||||
| No type hints | `mypy --strict`, `ruff` | Errors found before running |
|
||||
|
||||
**What you gain:** a scraper you can change with confidence, the same idea
|
||||
as the lecture but built the way a real project is.
|
||||
@@ -0,0 +1,17 @@
|
||||
**English** · [Dansk](Home-da) · [Bahasa Malaysia](Home-ms)
|
||||
|
||||
- [Home](Home)
|
||||
- [Lecture steps](Lecture-Steps)
|
||||
- [Modernisation](Modernisation)
|
||||
|
||||
**Dansk**
|
||||
|
||||
- [Hjem](Home-da)
|
||||
- [Lektionens trin](Lecture-Steps-da)
|
||||
- [Modernisering](Modernisation-da)
|
||||
|
||||
**Bahasa Malaysia**
|
||||
|
||||
- [Utama](Home-ms)
|
||||
- [Langkah pelajaran](Lecture-Steps-ms)
|
||||
- [Pemodenan](Modernisation-ms)
|
||||
|
After Width: | Height: | Size: 138 KiB |
|
After Width: | Height: | Size: 24 KiB |
|
After Width: | Height: | Size: 25 KiB |
|
After Width: | Height: | Size: 14 KiB |
@@ -0,0 +1,37 @@
|
||||
[build-system]
|
||||
requires = ["hatchling>=1.27"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "scraping-python-org"
|
||||
version = "0.1.0"
|
||||
description = "Scrape the upcoming events from python.org with Selenium."
|
||||
requires-python = ">=3.12"
|
||||
dependencies = ["selenium>=4.25"]
|
||||
|
||||
[dependency-groups]
|
||||
dev = ["pytest>=8.3", "ruff>=0.8", "mypy>=1.13"]
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/scraping_python_org"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
markers = ["browser: needs a headless Chrome installed"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
src = ["src", "tests"]
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "UP", "B", "D", "ANN", "SIM", "T20"]
|
||||
ignore = ["D203", "D213"]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"tests/**" = ["D"]
|
||||
"src/scraping_python_org/presentation/cli.py" = ["T201"]
|
||||
|
||||
[tool.mypy]
|
||||
strict = true
|
||||
mypy_path = "src"
|
||||
@@ -0,0 +1 @@
|
||||
"""Scrape the upcoming events from python.org."""
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Entry point for ``python -m scraping_python_org``."""
|
||||
|
||||
from scraping_python_org.presentation.cli import main
|
||||
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1 @@
|
||||
"""Use cases and the ports they need."""
|
||||
@@ -0,0 +1,13 @@
|
||||
"""Port for anything that can supply upcoming events."""
|
||||
|
||||
from typing import Protocol
|
||||
|
||||
from scraping_python_org.domain.event import Event
|
||||
|
||||
|
||||
class EventSource(Protocol):
|
||||
"""Supplies upcoming events in page order."""
|
||||
|
||||
def fetch_events(self) -> list[Event]:
|
||||
"""Return the upcoming events."""
|
||||
...
|
||||
@@ -0,0 +1,13 @@
|
||||
"""Use case: list the upcoming events as a numbered dictionary."""
|
||||
|
||||
from scraping_python_org.application.event_source import EventSource
|
||||
|
||||
DEFAULT_LIMIT = 5
|
||||
|
||||
|
||||
def get_upcoming_events(
|
||||
source: EventSource, limit: int = DEFAULT_LIMIT
|
||||
) -> dict[int, dict[str, str]]:
|
||||
"""Return up to ``limit`` events as ``{index: {"time": ..., "name": ...}}``."""
|
||||
events = source.fetch_events()[:limit]
|
||||
return {index: event.to_dict() for index, event in enumerate(events)}
|
||||
@@ -0,0 +1 @@
|
||||
"""Business rules; no input or output."""
|
||||
@@ -0,0 +1,15 @@
|
||||
"""The Event value object."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Event:
|
||||
"""An upcoming event with its date text and name."""
|
||||
|
||||
time: str
|
||||
name: str
|
||||
|
||||
def to_dict(self) -> dict[str, str]:
|
||||
"""Return the event as a plain dictionary."""
|
||||
return {"time": self.time, "name": self.name}
|
||||
@@ -0,0 +1 @@
|
||||
"""Drivers and adapters for outside technology."""
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Chrome WebDriver lifecycle."""
|
||||
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
|
||||
from selenium import webdriver
|
||||
|
||||
WINDOW_SIZE = "1280,1000"
|
||||
|
||||
|
||||
@contextmanager
|
||||
def chrome_driver(headless: bool = True) -> Iterator[webdriver.Chrome]:
|
||||
"""Yield a Chrome driver (found by Selenium Manager) and always quit it."""
|
||||
options = webdriver.ChromeOptions()
|
||||
if headless:
|
||||
options.add_argument("--headless=new")
|
||||
options.add_argument(f"--window-size={WINDOW_SIZE}")
|
||||
driver = webdriver.Chrome(options=options)
|
||||
try:
|
||||
yield driver
|
||||
finally:
|
||||
driver.quit()
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Selenium adapter for the EventSource port."""
|
||||
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.remote.webdriver import WebDriver
|
||||
|
||||
from scraping_python_org.domain.event import Event
|
||||
|
||||
EVENT_WIDGET_SELECTOR = ".event-widget"
|
||||
EVENT_ITEM_SELECTOR = f"{EVENT_WIDGET_SELECTOR} li"
|
||||
|
||||
|
||||
class SeleniumEventSource:
|
||||
"""Reads the events from the event widget of a page."""
|
||||
|
||||
def __init__(self, driver: WebDriver, url: str) -> None:
|
||||
"""Remember the driver and the page to read."""
|
||||
self._driver = driver
|
||||
self._url = url
|
||||
|
||||
def fetch_events(self) -> list[Event]:
|
||||
"""Open the page and return one Event per widget list item."""
|
||||
self._driver.get(self._url)
|
||||
items = self._driver.find_elements(By.CSS_SELECTOR, EVENT_ITEM_SELECTOR)
|
||||
return [
|
||||
Event(
|
||||
time=item.find_element(By.TAG_NAME, "time").text,
|
||||
name=item.find_element(By.TAG_NAME, "a").text,
|
||||
)
|
||||
for item in items
|
||||
]
|
||||
@@ -0,0 +1 @@
|
||||
"""Entry points and the composition root."""
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Command line entry point and composition root."""
|
||||
|
||||
from scraping_python_org.application.get_upcoming_events import get_upcoming_events
|
||||
from scraping_python_org.infrastructure.chrome import chrome_driver
|
||||
from scraping_python_org.infrastructure.selenium_event_source import SeleniumEventSource
|
||||
|
||||
PYTHON_ORG_URL = "https://www.python.org/"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Scrape the upcoming events from python.org and print them."""
|
||||
with chrome_driver() as driver:
|
||||
source = SeleniumEventSource(driver, PYTHON_ORG_URL)
|
||||
events = get_upcoming_events(source)
|
||||
print(events)
|
||||
return 0
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Capture one screenshot per lecture step into docs/screenshots/.
|
||||
|
||||
Run with ``python -m scraping_python_org.presentation.screenshots``.
|
||||
"""
|
||||
|
||||
import html
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.remote.webdriver import WebDriver
|
||||
|
||||
from scraping_python_org.application.get_upcoming_events import get_upcoming_events
|
||||
from scraping_python_org.infrastructure.chrome import chrome_driver
|
||||
from scraping_python_org.infrastructure.selenium_event_source import (
|
||||
EVENT_ITEM_SELECTOR,
|
||||
EVENT_WIDGET_SELECTOR,
|
||||
SeleniumEventSource,
|
||||
)
|
||||
from scraping_python_org.presentation.cli import PYTHON_ORG_URL
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
OUTPUT_DIR = Path(__file__).resolve().parents[3] / "docs" / "screenshots"
|
||||
HIGHLIGHT_SCRIPT = (
|
||||
"document.querySelectorAll(arguments[0])"
|
||||
".forEach(li => { li.style.outline = '3px solid red'; });"
|
||||
)
|
||||
OUTPUT_PAGE = (
|
||||
"<html><body style='margin:0;background:#fff'>"
|
||||
"<pre style='display:inline-block;font:18px Consolas,monospace;"
|
||||
"white-space:pre-wrap;margin:0;padding:24px'>{text}</pre>"
|
||||
"</body></html>"
|
||||
)
|
||||
|
||||
|
||||
def _render_text(driver: WebDriver, text: str, path: Path) -> None:
|
||||
"""Screenshot ``text`` rendered in a <pre> block."""
|
||||
page = OUTPUT_PAGE.format(text=html.escape(text))
|
||||
driver.get("data:text/html;charset=utf-8," + quote(page))
|
||||
driver.find_element(By.TAG_NAME, "pre").screenshot(str(path))
|
||||
|
||||
|
||||
def capture(output_dir: Path = OUTPUT_DIR) -> list[Path]:
|
||||
"""Write the four lecture screenshots and return their paths."""
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
home = output_dir / "01-python-org-home.png"
|
||||
widget = output_dir / "02-event-widget.png"
|
||||
highlighted = output_dir / "03-event-list-highlighted.png"
|
||||
output = output_dir / "04-output-dictionary.png"
|
||||
with chrome_driver() as driver:
|
||||
events = get_upcoming_events(SeleniumEventSource(driver, PYTHON_ORG_URL))
|
||||
driver.save_screenshot(str(home))
|
||||
driver.find_element(By.CSS_SELECTOR, EVENT_WIDGET_SELECTOR).screenshot(str(widget))
|
||||
driver.execute_script(HIGHLIGHT_SCRIPT, EVENT_ITEM_SELECTOR)
|
||||
driver.find_element(By.CSS_SELECTOR, EVENT_WIDGET_SELECTOR).screenshot(str(highlighted))
|
||||
_render_text(driver, str(events), output)
|
||||
paths = [home, widget, highlighted, output]
|
||||
for path in paths:
|
||||
logger.info("wrote %s", path)
|
||||
return paths
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
capture()
|
||||
@@ -0,0 +1,35 @@
|
||||
from scraping_python_org.application.get_upcoming_events import get_upcoming_events
|
||||
from scraping_python_org.domain.event import Event
|
||||
|
||||
|
||||
class FakeEventSource:
|
||||
def __init__(self, events: list[Event]) -> None:
|
||||
self._events = events
|
||||
|
||||
def fetch_events(self) -> list[Event]:
|
||||
return self._events
|
||||
|
||||
|
||||
def _events(count: int) -> list[Event]:
|
||||
return [Event(f"11-{i:02d}", f"Event {i}") for i in range(count)]
|
||||
|
||||
|
||||
def test_get_upcoming_events_numbers_events_from_zero() -> None:
|
||||
result = get_upcoming_events(FakeEventSource(_events(2)))
|
||||
|
||||
assert result == {
|
||||
0: {"time": "11-00", "name": "Event 0"},
|
||||
1: {"time": "11-01", "name": "Event 1"},
|
||||
}
|
||||
|
||||
|
||||
def test_get_upcoming_events_limits_to_five_by_default() -> None:
|
||||
assert list(get_upcoming_events(FakeEventSource(_events(8)))) == [0, 1, 2, 3, 4]
|
||||
|
||||
|
||||
def test_get_upcoming_events_honours_custom_limit() -> None:
|
||||
assert len(get_upcoming_events(FakeEventSource(_events(8)), limit=3)) == 3
|
||||
|
||||
|
||||
def test_get_upcoming_events_returns_empty_dict_without_events() -> None:
|
||||
assert get_upcoming_events(FakeEventSource([])) == {}
|
||||
@@ -0,0 +1,15 @@
|
||||
import dataclasses
|
||||
|
||||
import pytest
|
||||
|
||||
from scraping_python_org.domain.event import Event
|
||||
|
||||
|
||||
def test_to_dict_returns_time_and_name() -> None:
|
||||
assert Event("2026-11-02", "PyCon").to_dict() == {"time": "2026-11-02", "name": "PyCon"}
|
||||
|
||||
|
||||
def test_event_is_immutable() -> None:
|
||||
event = Event("2026-11-02", "PyCon")
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
event.name = "Other" # type: ignore[misc]
|
||||
@@ -0,0 +1,22 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head><meta charset="utf-8"><title>Welcome to Python.org (fixture)</title></head>
|
||||
<body>
|
||||
<div id="content">
|
||||
<div class="medium-widget blog-widget"><ul class="menu"><li><time>2026-01-01</time> <a href="/blog/">Not an event</a></li></ul></div>
|
||||
<div class="medium-widget event-widget last">
|
||||
<div class="shrubbery">
|
||||
<h2 class="widget-title"><span class="prompt">>>></span> Upcoming Events</h2>
|
||||
<p class="give-me-more"><a href="/events/" title="More Events">More</a></p>
|
||||
<ul class="menu">
|
||||
<li><time datetime="2026-11-02T00:00:00+00:00"><span class="say-no-more">2026-</span>11-02</time> <a href="/events/python-events/1001/">PyCon Alpha</a></li>
|
||||
<li><time datetime="2026-11-05T00:00:00+00:00"><span class="say-no-more">2026-</span>11-05</time> <a href="/events/python-events/1002/">PyData Beta</a></li>
|
||||
<li><time datetime="2026-11-09T00:00:00+00:00"><span class="say-no-more">2026-</span>11-09</time> <a href="/events/python-events/1003/">Django Gamma</a></li>
|
||||
<li><time datetime="2026-11-14T00:00:00+00:00"><span class="say-no-more">2026-</span>11-14</time> <a href="/events/python-events/1004/">EuroPython Delta</a></li>
|
||||
<li><time datetime="2026-11-20T00:00:00+00:00"><span class="say-no-more">2026-</span>11-20</time> <a href="/events/python-events/1005/">PyLadies Epsilon</a></li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,30 @@
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from selenium import webdriver
|
||||
|
||||
from scraping_python_org.infrastructure.chrome import chrome_driver
|
||||
from scraping_python_org.infrastructure.selenium_event_source import SeleniumEventSource
|
||||
|
||||
FIXTURE = Path(__file__).resolve().parents[1] / "fixtures" / "python_org_events.html"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def driver() -> Iterator[webdriver.Chrome]:
|
||||
with chrome_driver() as chrome:
|
||||
yield chrome
|
||||
|
||||
|
||||
@pytest.mark.browser
|
||||
def test_fetch_events_reads_widget_items_only(driver: webdriver.Chrome) -> None:
|
||||
events = SeleniumEventSource(driver, FIXTURE.as_uri()).fetch_events()
|
||||
|
||||
assert [event.name for event in events] == [
|
||||
"PyCon Alpha",
|
||||
"PyData Beta",
|
||||
"Django Gamma",
|
||||
"EuroPython Delta",
|
||||
"PyLadies Epsilon",
|
||||
]
|
||||
assert events[0].time == "2026-11-02"
|
||||