2026-10-28 07:24:45 +01:00
1 changed files with 57 additions and 0 deletions
@@ -104,6 +104,63 @@ python -m scrape_movies soup select

 The screenshots are made by `python tools/make_screenshots.py`, which renders the command output as an image; no desktop is captured.

+### Lecture: Scraping Hacker News
+
+The code is `src/scrape_movies/infrastructure/hacker_news.py` (parsing) and `http_client.py` (one `requests.get` with a timeout and a `User-Agent`). By default the commands read the course's static copy of the front page, `--source live` reads <https://news.ycombinator.com/news> and `--source fixture` reads the saved copy in `tests/fixtures/` without any network.
+
+**1. Look at the page you will scrape** (live data instead of a local file)
+
+![Step 1: the page](docs/images/hn-1-page.png)
+
+**2. Inspect the markup** (browser developer tools show the tags to look for: a story row `tr.athing`, its title link `.titleline > a`, and a `span.score` in the next row)
+
+```bash
+python -m scrape_movies hacker-news markup
+```
+
+![Step 2: markup](docs/images/hn-2-markup.png)
+
+**3. Titles** (`find_all` on the title links, then `get_text()`)
+
+```bash
+python -m scrape_movies hacker-news titles
+```
+
+![Step 3: titles](docs/images/hn-3-titles.png)
+
+**4. Links** (`get("href")` of the same anchors)
+
+```bash
+python -m scrape_movies hacker-news links
+```
+
+![Step 4: links](docs/images/hn-4-links.png)
+
+**5. Upvotes** (the text `38 points` is split and converted with `int()` so it can be compared)
+
+```bash
+python -m scrape_movies hacker-news upvotes
+```
+
+![Step 5: upvotes](docs/images/hn-5-upvotes.png)
+
+**6. The top story** (Python's `max` over the upvotes)
+
+```bash
+python -m scrape_movies hacker-news top
+```
+
+![Step 6: top story](docs/images/hn-6-top.png)
+
+### Lecture: Is web scraping legal and ethical?
+
+A short summary in our own words; it is not legal advice.
+
+- **Public, non-copyrighted data** is generally fine to read. Court cases such as Genius v. Google and hiQ v. LinkedIn show that the line is not simple.
+- **Copyrighted content** (a video, a blog post) must not be copied, and **data behind a login** is covered by the site's user agreement, so do not scrape it.
+- **Commercial use** of scraped data invites expensive disputes even where the scraping itself was allowed.
+- **Be a good guest:** prefer an official API when there is one, read the site's `robots.txt` to see what it allows, and send at most one request a minute. This project fetches one page per run and identifies itself with a `User-Agent`.
+
 ## 🧪 Tests

 ```bash