MIL-002: parsing HTML and making soup #36
@@ -56,12 +56,54 @@ Leave out `[dev]` to install only the runtime dependencies. Personal tokens belo
|
|||||||
|
|
||||||
## ▶️ Run
|
## ▶️ Run
|
||||||
|
|
||||||
The lecture steps are added milestone by milestone; each gets a section with its command, its output and a screenshot here.
|
Every lecture step is a command of the form `python -m scrape_movies <lecture> <step>`; without arguments the help is printed. Each step below shows the command, what it teaches and a screenshot of its output.
|
||||||
|
|
||||||
|
### Lecture: Parsing HTML and making soup
|
||||||
|
|
||||||
|
The sample page is `tests/fixtures/website.html`; the code is `src/scrape_movies/infrastructure/soup_basics.py`.
|
||||||
|
|
||||||
|
**1. Create a soup and read the title** (`BeautifulSoup(html, "html.parser")`, `soup.title.string`)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m scrape_movies # available from the movies milestone
|
python -m scrape_movies soup title
|
||||||
```
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
**2. Prettify** (`soup.prettify()` indents one tag per line)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m scrape_movies soup prettify
|
||||||
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
**3. Find all tags** (`find_all("a")`, then `get_text()` and `get("href")` in a loop)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m scrape_movies soup find-all
|
||||||
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
**4. Find by id and by class** (`find(id=...)`, `find_all("h3", class_=...)`; `class_` because `class` is a Python keyword)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m scrape_movies soup find
|
||||||
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
**5. CSS selectors** (`select_one("p a")` and `select(".heading")`)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m scrape_movies soup select
|
||||||
|
```
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
The screenshots are made by `python tools/make_screenshots.py`, which renders the command output as an image; no desktop is captured.
|
||||||
|
|
||||||
## 🧪 Tests
|
## 🧪 Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
Reference in New Issue
Block a user