Tirsvad-Udemy-100-days-of-code/046-scraping-the-billboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MIL-002: Project scaffold #44

Merged
Tirsvad merged 2 commits from mil-002-project-scaffold into main 2026-10-28 07:23:19 +01:00
Conversation 0 Commits 2 Files Changed 14
+359 -3
14 changed files with 359 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -183,3 +183,4 @@ docs/doxygen/
.venv/
browser.json
html/
/docs/doxygen/
Doxyfile
+10
View File
@@ -0,0 +1,10 @@
PROJECT_NAME = "Musical Time Machine"
INPUT = src
RECURSIVE = YES
FILE_PATTERNS = *.py
OPTIMIZE_OUTPUT_JAVA = YES
EXTRACT_ALL = YES
OUTPUT_DIRECTORY = docs/doxygen
GENERATE_LATEX = NO
QUIET = YES
WARN_AS_ERROR = NO
README.md
+141 -1
View File
@@ -1,2 +1,142 @@
# 046-scraping-the-billboard
# 🚀 Musical Time Machine
Pick a date, scrape the Billboard Hot 100 of that day and get a private YouTube Music playlist of the songs. A solution to day 46 of Udemy's *100 Days of Code: The Complete Python Pro Bootcamp*, for fellow course participants and anyone browsing for ideas.
## 📚 Table of Contents
- [Overview](#-overview)
- [Requirements](#-requirements)
- [Setup](#-setup)
- [Run](#-run)
- [Tests](#-tests)
- [License](#-license)
- [Links](#-links)
- [Appendix: Project documents](#-appendix-project-documents)
## 🧭 Overview
The program follows the four steps of the lecture:
1. **Scrape the chart.** Ask for a date (`YYYY-MM-DD`) and read the 100 song titles from the Bakeboard clone of the Billboard Hot 100 (`https://appbrewery.github.io/bakeboard-hot-100/`) with `requests` and `BeautifulSoup`.
2. **Authenticate.** Open a YouTube Music session with `ytmusicapi` from a `browser.json` file.
3. **Create the playlist.** Create a private playlist named `{date} Billboard 100`, unless it already exists.
4. **Add the songs.** Search every song and add the top result; a song that fails is skipped.
Layout of the repository:
| Folder | What is in it |
| --- | --- |
| `src/musical_time_machine/` | The program, in the layers `domain`, `application`, `infrastructure` and `presentation`; all constants are in `constants.py` |
| `tests/` | The automated tests (pytest); they use a stored chart page and a fake music service, never the network |
| `docs/` | Business case, plan, milestones, reviews and the wiki sources |
The wiki is written in English, Danish and Bahasa Malaysia and explains what each lecture step teaches.
## 📋 Requirements
- Python 3.13 or later.
- Three runtime libraries, each needed by the assignment: `requests` (fetch the page), `beautifulsoup4` (read the HTML) and `ytmusicapi` (talk to YouTube Music).
- A Google account and Firefox, only for steps 2 to 4 (to create `browser.json`).
- Optional: [Doxygen](https://www.doxygen.nl/) to build the API documentation.
## 🛠️ Setup
Create and use a local virtual environment named `.venv`, then upgrade `pip`:
```bash
python -m venv .venv
```
Activate it (Windows PowerShell):
```powershell
.venv\Scripts\Activate.ps1
```
Activate it (Linux and macOS):
```bash
source .venv/bin/activate
```
```bash
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
```
Leave out `[dev]` when you only want to run the program.
`browser.json` (steps 2 to 4 only) holds your YouTube Music login cookies. It is ignored by git: never commit it.
1. Open Firefox, go to <https://music.youtube.com> and log in.
2. Open the developer tools (`Ctrl+Shift+I`, on macOS `Cmd+Option+I`), choose the **Network** tab and type `browse` in the filter.
3. Pick a successful `browse?...` request to `music.youtube.com`, right-click it and choose *Copy Value*, then *Copy Request Headers*.
4. In the project folder run `ytmusicapi browser` and paste the headers, then finish with `Ctrl+D` (`Ctrl+Z` and Enter on Windows). On macOS run `pbpaste | ytmusicapi browser`.
The cookies expire after a while; repeat the steps to get a fresh file.
## ▶️ Run
The program is added step by step (one pull request per lecture step):
| Step | Status |
| --- | --- |
| 1. Scrape the Billboard Hot 100 | planned in milestone 3 |
| 2. YouTube Music authentication | planned in milestone 4 |
| 3. Create the playlist | planned in milestone 5 |
| 4. Search and add songs | planned in milestone 6 |
The run command and a screenshot for each step are added with the step.
## 🧪 Tests
```bash
python -m pytest
```
To build the API documentation (written as Doxygen comments in the source):
```bash
doxygen Doxyfile
```
## 📄 License
GNU Affero General Public License v3.0, see [LICENSE](./LICENSE).
## 🔗 Links
- [Repository][repository]
- [Documentation][documentation]
- [Issue tracker][issues]
## 📎 Appendix: Project documents
| Document | What it tells you |
| --- | --- |
| [AGENTS.md][agents] | The rules AI agents follow in this repository |
| [Business Case][BC-001] | Why the project exists, its scope and success criteria |
| [Stakeholder Analysis][SA-001] | Who is involved; the stakeholder IDs used for owners and reviewers |
| [Project Plan][PP-001] | The gateways and their schedule |
| [Milestones][milestones] | One document per gateway: Go/No-Go criteria and tasks |
| [Artifact registry][registry] | Where each artifact type lives, the next version, the PO language and domain |
| [Traceability matrix][TM-001] | How the artifacts link and when each was last reviewed |
| [Governance][GOV-001] | The review workflow, who decides and how disputes escalate |
| [Framework README][framework-readme] | The framework this project uses |
| [Framework changelog][framework-changelog] | What changed in each framework release |
---
[repository]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/046-scraping-the-billboard
[documentation]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/046-scraping-the-billboard/wiki
[issues]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/046-scraping-the-billboard/issues
[agents]: ./AGENTS.md
[BC-001]: ./docs/business-case.md
[SA-001]: ./docs/stakeholder-analysis.md
[PP-001]: ./docs/project-plan.md
[milestones]: ./docs/milestones/
[registry]: ./docs/artifact-registry.md
[TM-001]: ./docs/sqa/traceability-matrix.md
[GOV-001]: ./docs/sqa/governance.md
[framework-readme]: ./framework/README.md
[framework-changelog]: ./framework/CHANGELOG.md
docs/artifact-registry.md
+1 -1
View File
@@ -17,7 +17,7 @@ document of a type. `Primary File` may contain a glob (e.g.
| MIL | Milestone / Gateway | docs/milestones/mil-*.md | 008 |
| GOV | Governance / ARB Workflow | docs/sqa/governance.md | 002 |
| TM | Traceability Matrix | docs/sqa/traceability-matrix.md | 002 |
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 012 |
| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 013 |
## Languages
docs/sqa/reviews/rc-012-mil-002-code.md
+62
View File
@@ -0,0 +1,62 @@
# Review of the code of MIL-002
## Metadata
| Key | Value |
| --- | --- |
| ID | RC-012 |
| CrossReference | [MIL-002], [QC-PY-001] |
## Version History
| Date | Status | Author | Reviewer | Change | Commit |
| --- | --- | --- | --- | --- | --- |
| 2026-10-28 | Proposed | Jens Tirsvad Nielsen | AI as stand-in for S01 | Initial version | [891c98a] |
---
## Artifact Under Review
- Instance reviewed: [MIL-002] (the code of its pull request, `src/` and `tests/`)
- Checklist used: [QC-PY-001] (`QC-PY-001`)
- Scope: full review
- Language and domain: n/a
- Language reviewer: none
- Mode: autonomous, not independent (reviewer is `AI as stand-in for S01`)
## Checklist Results
| # | Criterion | Status | Evidence/Notes |
| --- | --- | --- | --- |
| 1 | Packages, modules, functions, variables, classes and constants follow PEP 8 casing (`snake_case`, `PascalCase`, `UPPER_SNAKE`) | Pass | Constants are UPPER_SNAKE, package names snake_case. |
| 2 | Names state purpose in the domain's language; no unexplained abbreviations, no single-letter names outside tiny scopes | Pass | Names state purpose (REQUEST_TIMEOUT_SECONDS, SELECTOR_ROW). |
| 3 | Code is produced by the project's formatter and passes its linter with no unexplained suppressions | Pass | `ruff format` and `ruff check` (configured in pyproject.toml) report no finding. |
| 4 | Every function and method signature is type-annotated, including `-> None` | N-A | The scaffold defines no function; constants carry `Final` annotations. |
| 5 | No bare `except:`, no swallowed exceptions; specific exceptions are raised and the cause is kept (`raise ... from`) | N-A | No logic exists in the scaffold; nothing to check yet. |
| 6 | No mutable default arguments and no shadowed builtins | N-A | No logic exists in the scaffold; nothing to check yet. |
| 7 | Files, locks and connections are managed with context managers | N-A | No logic exists in the scaffold; nothing to check yet. |
| 8 | Public modules, classes and functions have docstrings that say what, not how | Pass | Modules and constant groups have Doxygen comments; `doxygen Doxyfile` has 0 warnings. |
| 9 | Logging uses `logging`, not `print`; no secrets or personal data in log output | N-A | No logic exists in the scaffold; nothing to check yet. |
| 10 | Classes and operations trace to the Design Class Diagram they implement; deviations are recorded | N-A | No logic exists in the scaffold; nothing to check yet. |
| 11 | Tests exist for new behaviour, are named for the behaviour, and do not depend on order or the network | Pass | tests/unit/test_smoke.py checks the constants and runs without the network. |
| 12 | Type checker runs in strict mode without errors; `Any` is justified in a comment | N-A | No logic exists in the scaffold; nothing to check yet. |
| 13 | Dependencies are declared and pinned in the project's dependency file, none unused | Pass | Three runtime dependencies and the dev extra are declared with lower bounds; none unused yet (they are used from MIL-003). |
| 14 | Each piece of knowledge (a business rule, constant, format, validation or query) is defined in one place; code is merged only where it expresses the same knowledge, not where it merely looks alike | Pass | Every constant is defined once in constants.py. |
| 15 | Imports point inward: business-rule modules import no framework, database, UI or delivery-mechanism code, and there are no import cycles | Pass | Layer packages are empty; no imports yet. |
| 16 | SOLID holds: a class or module has one reason to change; new behaviour is added by extension, not by editing a type test in several places; subclasses honour the contract of their base; protocols are narrow; high-level code depends on protocols or abstract types and receives its implementations | N-A | No logic exists in the scaffold; nothing to check yet. |
| 17 | Source is laid out in the layers and folders of the clean architecture folder structure (`domain`, `application`, `infrastructure`, `presentation` packages under `src/`, and `unit`, `integration` and `e2e` under `tests/`), in `snake_case`, and imports between the layers point inward | Pass | src/musical_time_machine/{domain,application,infrastructure,presentation} and tests/unit exist. |
| 18 | File and folder names are English, contain no space and no character outside letters, digits, `-`, `_` and `.`, and follow the casing of the language for files | Pass | All names are English, lowercase with underscores. |
## Overall Verdict
Go — every Mandatory criterion passes or does not apply; no blocking finding.
## Action Items
| Action | Owner | Due |
| --- | --- | --- |
| None | S01 | - |
---
[MIL-002]: ../../milestones/mil-002-project-scaffold.md
[QC-PY-001]: ../../../framework/qc/code/qc-programming-python.md
[891c98a]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/046-scraping-the-billboard/commit/891c98ac5077def329e36d8c1ebf2b35c9216349
docs/sqa/traceability-matrix.md
+2 -1
View File
@@ -26,7 +26,7 @@ Tracks backward and forward links between artifact instances so that the traceab
| [PP-001] | PP | en | it | [BC-001], [SA-001] | [MIL-001] | [RC-003] |
| [GOV-001] | GOV | en | it | [BC-001], [SA-001] | [TM-001] | [RC-004] |
| [MIL-001] | MIL | en | it | [BC-001], [PP-001] | [MIL-002] | [RC-005] |
| [MIL-002] | MIL | en | it | [MIL-001] | [MIL-003] | [RC-006] |
| [MIL-002] | MIL | en | it | [MIL-001] | [MIL-003] | [RC-006], [RC-012] |
| [MIL-003] | MIL | en | it | [MIL-002] | [MIL-004] | [RC-007] |
| [MIL-004] | MIL | en | it | [MIL-003] | [MIL-005] | [RC-008] |
| [MIL-005] | MIL | en | it | [MIL-004] | [MIL-006] | [RC-009] |
@@ -81,3 +81,4 @@ Tracks backward and forward links between artifact instances so that the traceab
[RC-010]: ./reviews/rc-010-mil-006.md
[RC-011]: ./reviews/rc-011-mil-007.md
[01e016e]: https://git.tirsystem.com/Tirsvad-Udemy-100-days-of-code/046-scraping-the-billboard/commit/01e016e15dd93cface4f4ae9e9e99919e0aace2b
[RC-012]: ./reviews/rc-012-mil-002-code.md
pyproject.toml
+34
View File
@@ -0,0 +1,34 @@
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "musical-time-machine"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
"requests>=2.34",
"beautifulsoup4>=4.15",
"ytmusicapi>=1.12",
]
[project.optional-dependencies]
dev = [
"pytest>=9.2",
"pytest-cov>=7.1",
"ruff>=0.17",
]
[tool.setuptools.packages.find]
where = ["src"]
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
[tool.ruff]
line-length = 100
target-version = "py313"
[tool.ruff.lint]
select = ["E", "F", "I", "N", "UP", "B"]
src/musical_time_machine/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""@file
@brief Musical Time Machine: build a YouTube Music playlist from a Billboard Hot 100 chart.
"""
src/musical_time_machine/application/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""@file
@brief Application layer: use cases and the ports they need.
"""
src/musical_time_machine/constants.py
+67
View File
@@ -0,0 +1,67 @@
"""@file
@brief All constants of the Musical Time Machine, defined once.
"""
from typing import Final
## @name Chart source
## @{
## @brief Base URL of the Bakeboard Hot 100 clone; the chart date and a slash are appended.
BASE_URL: Final[str] = "https://appbrewery.github.io/bakeboard-hot-100/"
## @brief strptime/strftime format of the chart date the user enters.
DATE_FORMAT: Final[str] = "%Y-%m-%d"
## @brief Number of entries on one Hot 100 chart.
CHART_SIZE: Final[int] = 100
## @}
## @name HTTP request
## @{
## @brief Headers sent with every chart request.
REQUEST_HEADERS: Final[dict[str, str]] = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
),
}
## @brief Seconds to wait for the chart page before giving up.
REQUEST_TIMEOUT_SECONDS: Final[int] = 10
## @}
## @name CSS selectors of the Bakeboard chart markup
## @{
## @brief One chart row.
SELECTOR_ROW: Final[str] = "div.chart-entry"
## @brief Song title inside a row.
SELECTOR_TITLE: Final[str] = "h3.chart-entry__title"
## @brief Artist inside a row.
SELECTOR_ARTIST: Final[str] = "span.chart-entry__artist"
## @}
## @name YouTube Music
## @{
## @brief Playlist title; format with the chart date.
PLAYLIST_NAME_TEMPLATE: Final[str] = "{date} Billboard 100"
## @brief Playlist description; format with the chart date.
PLAYLIST_DESCRIPTION_TEMPLATE: Final[str] = (
"Billboard Hot 100 of {date}, made by Musical Time Machine."
)
## @brief File holding the ytmusicapi browser authentication headers (git-ignored).
BROWSER_FILE_NAME: Final[str] = "browser.json"
## @brief ytmusicapi search filter, so only songs are matched.
SEARCH_FILTER: Final[str] = "songs"
## @brief Privacy status of the created playlist.
PLAYLIST_PRIVACY: Final[str] = "PRIVATE"
## @}
src/musical_time_machine/domain/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""@file
@brief Domain layer: business rules, no input or output.
"""
src/musical_time_machine/infrastructure/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""@file
@brief Infrastructure layer: scraping, YouTube Music and configuration adapters.
"""
src/musical_time_machine/presentation/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""@file
@brief Presentation layer: command line entry point and composition root.
"""
tests/unit/test_smoke.py
+26
View File
@@ -0,0 +1,26 @@
"""@file
@brief Smoke test: the package imports and its constants are sane.
"""
import musical_time_machine
from musical_time_machine import constants
def test_package_imports() -> None:
"""@brief The package and its layers can be imported."""
from musical_time_machine import application, domain, infrastructure, presentation
assert all(
m is not None
for m in (musical_time_machine, domain, application, infrastructure, presentation)
)
def test_chart_size_is_one_hundred() -> None:
"""@brief The Hot 100 has 100 entries."""
assert constants.CHART_SIZE == 100
def test_playlist_name_template_formats_date() -> None:
"""@brief The playlist name template accepts a chart date."""
assert constants.PLAYLIST_NAME_TEMPLATE.format(date="2026-04-18") == "2026-04-18 Billboard 100"