18 lines
1.0 KiB
Markdown
18 lines
1.0 KiB
Markdown
**English** · [Dansk](Gains-da) · [Bahasa Melayu](Gains-ms)
|
|
|
|
# What we gained
|
|
|
|
| Lecture | What we can do now |
|
|
| --- | --- |
|
|
| [Parsing HTML](Parsing-HTML) | Make a soup, read its structure and select elements by tag, id, class or CSS selector |
|
|
| [Hacker News](Hacker-News) | Fetch a live page, read titles, links and scores, convert text to numbers, find the maximum |
|
|
| [Scraping ethics](Scraping-Ethics) | Check `robots.txt`, respect copyright and logins, keep requests rare |
|
|
| [Top 100 movies](Top-100-Movies) | Scrape a ranked list, reverse it and write it to a file |
|
|
| [Setup](Setup) | Work in a virtual environment with `pyproject.toml` and offline tests |
|
|
|
|
## Beyond the lectures
|
|
|
|
- **Parsing is separate from fetching.** The parsing functions take text, so the tests use saved pages and never need the network.
|
|
- **Layers keep rules apart.** The ranking rule has no knowledge of BeautifulSoup or Requests.
|
|
- **Screenshots are made by a script** (`tools/make_screenshots.py`), so they can be regenerated after a change.
|