1.0 KiB
1.0 KiB
English · Dansk · Bahasa Melayu
What we gained
| Lecture | What we can do now |
|---|---|
| Parsing HTML | Make a soup, read its structure and select elements by tag, id, class or CSS selector |
| Hacker News | Fetch a live page, read titles, links and scores, convert text to numbers, find the maximum |
| Scraping ethics | Check robots.txt, respect copyright and logins, keep requests rare |
| Top 100 movies | Scrape a ranked list, reverse it and write it to a file |
| Setup | Work in a virtual environment with pyproject.toml and offline tests |
Beyond the lectures
- Parsing is separate from fetching. The parsing functions take text, so the tests use saved pages and never need the network.
- Layers keep rules apart. The ranking rule has no knowledge of BeautifulSoup or Requests.
- Screenshots are made by a script (
tools/make_screenshots.py), so they can be regenerated after a change.