76 lines
3.3 KiB
Markdown
76 lines
3.3 KiB
Markdown
# Scraping Hacker News
|
|
|
|
## Metadata
|
|
| Key | Value |
|
|
| --- | --- |
|
|
| ID | MIL-003 |
|
|
| CrossReference | [BC-001] |
|
|
| Language | en |
|
|
| Domain | it |
|
|
|
|
## Version History
|
|
| Date | Status | Author | Reviewer | Change | Commit |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| 2026-10-28 | Accepted | Jens Tirsvad Nielsen | S01 (AI as stand-in, autonomous mode) | Initial version | [dcef5e6] |
|
|
|
|
---
|
|
|
|
## Purpose
|
|
|
|
Decide whether scraping a live site is reproduced safely: Hacker News titles, links and upvotes are extracted, the top story found, and the ethics of scraping documented.
|
|
|
|
## Deliverable
|
|
|
|
Module `hacker_news` with fetch and parse functions, the saved front page as a fixture, tests, ethics note and screenshots of each step.
|
|
|
|
## Go / No-Go Criteria
|
|
|
|
| # | Criterion (objectively checkable) | Go | No-Go |
|
|
| --- | --- | --- | --- |
|
|
| 1 | Parsing returns titles, links and integer upvotes from the fixture | Met | Not met |
|
|
| 2 | The article with the most upvotes is found with `max` | Met | Not met |
|
|
| 3 | Fetching sends one request with a `User-Agent` and a timeout; no loops of requests | Met | Not met |
|
|
| 4 | Tests run offline | Met | Not met |
|
|
| 5 | Ethics and legality notes from the lecture are in README and wiki source text | Met | Not met |
|
|
| 6 | One screenshot per lecture step exists under `docs/images/` | Met | Not met |
|
|
|
|
## Dependencies
|
|
|
|
| Depends on | Reason |
|
|
| --- | --- |
|
|
| MIL-002 (soup helpers) | Builds on it |
|
|
|
|
## Traceability
|
|
|
|
| Business Case objective / KPI / user story | Reference |
|
|
| --- | --- |
|
|
| O1, O3, O5; criteria 1, 4 | [BC-001] |
|
|
|
|
## Ownership
|
|
|
|
| Role | Stakeholder ID (SA) |
|
|
| --- | --- |
|
|
| Owner | S01 |
|
|
| Approving reviewer | S01 |
|
|
|
|
## Target Date
|
|
|
|
2026-11-06 — no hard deadline in the Business Case; dates follow the order of the milestones.
|
|
|
|
## Tasks
|
|
|
|
| # | Task | Summary | Needs its own Use Case/User Story? | Reference |
|
|
| --- | --- | --- | --- | --- |
|
|
| 1 | Add fetch function for Hacker News | `fetch_html(url)` using `requests` with a timeout, `User-Agent` and status check; URLs live in `constants.py`. The course mirror and the live site are both named. | No | |
|
|
| 2 | Add Hacker News fixture | Save the course mirror page `appbrewery.github.io/news.ycombinator.com` as `tests/fixtures/hacker_news.html` so tests never touch the network. | No | |
|
|
| 3 | Extract titles, links and upvotes | Functions returning lists of titles, links and integer scores from the title links and score spans, as in the lecture. | No | |
|
|
| 4 | Find the top story by upvotes | `top_story()` using `max` over the scraped rows and a test for ties and an empty page. | No | |
|
|
| 5 | Write tests for Hacker News parsing | pytest tests against the fixture, with `requests` patched for fetch tests. | No | |
|
|
| 6 | Document ethics and legality of scraping | Summarise the lecture: robots.txt, copyrighted and authenticated data, commercial risk, request frequency. Written in the README in our own words. | No | |
|
|
| 7 | Capture screenshots of the Hacker News steps | Save a screenshot per lecture step (page, inspected markup, extracted output, top story) in `docs/images/`. The live pages are opened in the built-in browser pane, not the host desktop. | No | |
|
|
|
|
---
|
|
|
|
[BC-001]: ../business-case.md
|
|
[dcef5e6]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/045-scrape-movies/commit/dcef5e6e6e7ec16162817d3ec0ce1b303e0a3a63
|