3.3 KiB
3.3 KiB
Scraping Hacker News
Metadata
| Key | Value |
|---|---|
| ID | MIL-003 |
| CrossReference | BC-001 |
| Language | en |
| Domain | it |
Version History
| Date | Status | Author | Reviewer | Change | Commit |
|---|---|---|---|---|---|
| 2026-10-28 | Accepted | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | dcef5e6 |
Purpose
Decide whether scraping a live site is reproduced safely: Hacker News titles, links and upvotes are extracted, the top story found, and the ethics of scraping documented.
Deliverable
Module hacker_news with fetch and parse functions, the saved front page as a fixture, tests, ethics note and screenshots of each step.
Go / No-Go Criteria
| # | Criterion (objectively checkable) | Go | No-Go |
|---|---|---|---|
| 1 | Parsing returns titles, links and integer upvotes from the fixture | Met | Not met |
| 2 | The article with the most upvotes is found with max |
Met | Not met |
| 3 | Fetching sends one request with a User-Agent and a timeout; no loops of requests |
Met | Not met |
| 4 | Tests run offline | Met | Not met |
| 5 | Ethics and legality notes from the lecture are in README and wiki source text | Met | Not met |
| 6 | One screenshot per lecture step exists under docs/images/ |
Met | Not met |
Dependencies
| Depends on | Reason |
|---|---|
| MIL-002 (soup helpers) | Builds on it |
Traceability
| Business Case objective / KPI / user story | Reference |
|---|---|
| O1, O3, O5; criteria 1, 4 | BC-001 |
Ownership
| Role | Stakeholder ID (SA) |
|---|---|
| Owner | S01 |
| Approving reviewer | S01 |
Target Date
2026-11-06 — no hard deadline in the Business Case; dates follow the order of the milestones.
Tasks
| # | Task | Summary | Needs its own Use Case/User Story? | Reference |
|---|---|---|---|---|
| 1 | Add fetch function for Hacker News | fetch_html(url) using requests with a timeout, User-Agent and status check; URLs live in constants.py. The course mirror and the live site are both named. |
No | |
| 2 | Add Hacker News fixture | Save the course mirror page appbrewery.github.io/news.ycombinator.com as tests/fixtures/hacker_news.html so tests never touch the network. |
No | |
| 3 | Extract titles, links and upvotes | Functions returning lists of titles, links and integer scores from the title links and score spans, as in the lecture. | No | |
| 4 | Find the top story by upvotes | top_story() using max over the scraped rows and a test for ties and an empty page. |
No | |
| 5 | Write tests for Hacker News parsing | pytest tests against the fixture, with requests patched for fetch tests. |
No | |
| 6 | Document ethics and legality of scraping | Summarise the lecture: robots.txt, copyrighted and authenticated data, commercial risk, request frequency. Written in the README in our own words. | No | |
| 7 | Capture screenshots of the Hacker News steps | Save a screenshot per lecture step (page, inspected markup, extracted output, top story) in docs/images/. The live pages are opened in the built-in browser pane, not the host desktop. |
No |