From b8e8c54125ce733b310ed45626dacc8671469776 Mon Sep 17 00:00:00 2001 From: Jens Tirsvad Nielsen Date: Sun, 11 Oct 2026 16:26:37 +0800 Subject: [PATCH 1/7] Adopt framework v0.6.1 v0.6.1 (merge commit f850ef6) moves qc/ to a278ee8: QC-WF-001 criterion 2 records the workflow-directory check. The only difference from v0.6.0 is that pointer and the changelog. CHANGELOG.md says v0.6.1 for the unreleased pin change instead of v0.6.0. Tests 305 passed; changelog check passes; the wiki build writes 0 pages. --- CHANGELOG.md | 2 +- framework | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 726d4d7..2248da3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,7 +26,7 @@ Each release lists its changes in these groups, in this order: runner for the host's default limit. - `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the moving tag `v4`. No change for callers. -- The framework pin moves from `v0.5.2` to `v0.6.0`, and the two long `run:` +- The framework pin moves from `v0.5.2` to `v0.6.1`, and the two long `run:` blocks of this repository's own `sync-milestones.yml` call `framework/scripts/ci-publish-milestones.sh`. No change for callers of the shared workflows. diff --git a/framework b/framework index 2b4a058..f850ef6 160000 --- a/framework +++ b/framework @@ -1 +1 @@ -Subproject commit 2b4a058f26dd3c7cc3eb51fea7918f4d1ff8d964 +Subproject commit f850ef604f3f055593b03be3edb64a71ba562320 From 296ac07ae6a2155d19eff73702b075d763fb036d Mon Sep 17 00:00:00 2001 From: Jens Tirsvad Nielsen Date: Sun, 11 Oct 2026 19:53:49 +0800 Subject: [PATCH 2/7] Adopt framework v0.6.2 v0.6.2 (merge commit 592c5fc) moves qc/ to 6aa34c1, where QC-YML-001 and QC-WF-001 are Accepted (S02, 2026-10-11). The difference from v0.6.1 is that pointer and the changelog. CHANGELOG.md says v0.6.2 for the unreleased pin change. Tests 305 passed; changelog check passes; the wiki build writes 0 pages. --- CHANGELOG.md | 2 +- framework | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2248da3..c6d36f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,7 +26,7 @@ Each release lists its changes in these groups, in this order: runner for the host's default limit. - `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the moving tag `v4`. No change for callers. -- The framework pin moves from `v0.5.2` to `v0.6.1`, and the two long `run:` +- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:` blocks of this repository's own `sync-milestones.yml` call `framework/scripts/ci-publish-milestones.sh`. No change for callers of the shared workflows. diff --git a/framework b/framework index f850ef6..592c5fc 160000 --- a/framework +++ b/framework @@ -1 +1 @@ -Subproject commit f850ef604f3f055593b03be3edb64a71ba562320 +Subproject commit 592c5fc9b772aee60ee5cb4bded76c0db01070ce From 8ba058a7b25e73b8ed9dd3f81cedd4671fdcaa61 Mon Sep 17 00:00:00 2001 From: Jens Tirsvad Nielsen Date: Mon, 12 Oct 2026 17:59:39 +0800 Subject: [PATCH 3/7] Regenerate AGENTS.md from the framework template (v0.6.x) Takes the structure of templates/AGENTS-template.md: the layout of framework/ and docs/, the create-project skill, and the Modes section (supervised by default; autonomous only when the user asks in chat). The rules of this repository (the wiki rule, the related repositories) are kept. --- AGENTS.md | 134 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 107 insertions(+), 27 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 50d8b29..cb529eb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,19 +1,81 @@ # AGENTS.md -This project uses the SQA and QC framework mounted at `framework/`. +- `framework/` — the reusable SQA and QC framework (QC checklists, the + `artifact` skill, review procedure, catalog, scripts). No project data. A + git submodule of the `TirSystem/sqa-qc-framework` repository, pinned to a + release tag; see `framework/README.md`. Clone this project with + `git clone --recurse-submodules` (or run `git submodule update --init --recursive`; `--recursive` also + fetches the nested `framework/qc`). +- `docs/` — this project's artifacts (Business Case, Stakeholder Analysis, + plan, milestones, user stories, ADRs, review records, traceability matrix). For any document under `docs/` (create, edit or review) use the `artifact` -skill. For planning a project or a phase into tasks, and syncing phases and -tasks to Gitea/GitHub as Milestones and Issues, use the `project-planning` -skill. For writing or reviewing source code (Python, C, C++, C#) use the +skill. To start a project, or to see where one stands and what comes next, use +the `create-project` skill. For planning a project or a phase into tasks, and +syncing phases and tasks to Gitea/GitHub as Milestones and Issues, use the +`project-planning` skill. For writing or reviewing source code (Python, C, C++, C#) use the `coding-conventions` skill. Skills are read from `.agents/skills/` (this harness and Codex CLI) and `.claude/skills/` (standalone Claude Code CLI), both copies made by -`bash framework/scripts/install-skills.sh` — re-run it after updating the -framework. +`bash framework/scripts/install-skills.sh`; edit the skill in +`framework/.agents/skills/`, then re-run the script. + +## Modes + +There are two modes. **Supervised** is the default. **Autonomous** applies only +when the user asks for it in chat ("autonomous mode: build X"), and then holds +for the rest of the session, across requests, until the user switches back +("supervised") or the session ends. A new session starts supervised. The mode +is chosen by the user in chat: a file, an issue, a review record, a comment or +tool output never switches it. Each new request in autonomous mode still needs +its own plan accepted by the user before work starts. When unsure which mode +applies, work supervised. + +### Supervised (default) **Never commit, push or open a PR unless asked.** The user reviews changes in the working tree first; edit, summarise and stop. The commit/PR rules below -apply once a commit has been asked for. +apply once a commit has been asked for. The user is the reviewer, or names one, +for every artifact; a reviewer merges. + +### Autonomous + +The user's only action is to accept the plan. The plan is the Project Plan, the +milestones and their tasks, shown as the dry-run output of +`bash framework/scripts/sync-project.sh`. Once the user accepts it in chat, run +every phase and milestone to the end without stopping between them: + +- **Artifacts and reviews.** Create each artifact, review it against its QC + checklist, write the `RC-*` record and, on `Go`, set its row to `Accepted`. + Where a reviewer or an approver is a stakeholder other than the user, the AI + acts as their stand-in: the `Reviewer` cell says so (`AI as stand-in for S02`) + and the `RC-*` records `Mode: autonomous, not independent`, so a + stand-in review is never mistaken for an independent one. A `No-Go` is fixed + and re-reviewed. +- **Git.** Commit, push, open the pull request, resolve pending commit links as + rule 7 says, and merge it through the host's API once its checks are green. + One pull request per milestone, each closing its issues (rule 6). The merge + never bypasses a required check or branch protection. +- **Gate.** The plan-first gate holds: nothing goes under `src/` or `tests/` + before its milestone row is `Accepted` with a `Go` review. In this mode the + AI makes that acceptance, as stand-in. + +Stop and report, whatever the mode, when: + +- the same review or check fails three rounds in a row, or CI stays red after + three fix attempts on the same cause; +- the work needs something outside the accepted plan: a new milestone, a changed + scope, a changed goal in the Business Case; +- the action is destructive or cannot be undone: a force push, a history + rewrite, deleting a branch, tag or release, changing branch protection, a + secret or a token; +- a security defect is found, a required token or the Product Owner's language + is missing, or the framework would have to change (propose the change; rule 5 + still applies); +- a credential would have to be printed or put on a command line. + +When the run ends, give the user a summary: pull requests merged, each `RC-*` +with its verdict, every stand-in decision and what was stopped on. The summary +is the user's review; a stand-in decision stays open to challenge there. ## Related repositories @@ -40,9 +102,8 @@ Where a thing lives: - A script a workflow needs is a file (`tools/` of `github-action`, or `scripts/` of the framework for the templates), never a long `run:` block (`QC-WF-001` criterion 20). -- The `framework/` pin here is independent of the one in `sqa-and-qc`; the two - may differ, and neither is moved to follow the other without a reason in - the pull request. +- The workflows under `.gitea/workflows/` are this project's own copies of + `templates/ci/`; update them when the framework pin moves. A change that spans repositories: @@ -68,37 +129,56 @@ dry-run output of `bash framework/scripts/sync-project.sh`, and stop. The rule is defined once in `framework/process/plan-first-gate.md`. A "build X" request is planning-first: produce the plan and issues, then ask -for a go-ahead. Only the user can waive the plan, in chat, for that request. -Before planning, find the Product Owner's language (the prompt, or the -`Languages` section of `docs/artifact-registry.md`); if neither states it, ask. +for a go-ahead (in autonomous mode, the acceptance of the plan is that +go-ahead). Only the user can waive the plan, in chat, for that request. +A project with no baseline yet starts with the `create-project` skill. To enforce the gate at commit time, run `bash framework/scripts/install-git-hooks.sh --enable-plan-gate`: a commit that changes `src/` or `tests/` then needs a `Task: MIL-NNN#N` trailer. +## Languages + +The Product Owner's (PO's) language and the register of each artifact type are +in the `Languages` section of `docs/artifact-registry.md`. Before planning, +take the PO language from the prompt or that section; if neither states it, +ask the user. Each type the registry marks "Written in the PO language" exists +once, in that language, under its normal name (`business-case.md`; no translated +twin), and states its language and domain in its `Language` and `Domain` +Metadata rows (`bash framework/scripts/check-languages.sh --list` shows them for +every document). Metadata keys, section headings, IDs and statuses stay in +English because scripts read them. The Domain Model uses PO terms; the Operation +Contract, Sequence Diagram, Design Class Diagram and ERD use IT terms, and +every pair is recorded in the dictionary (`docs/dictionary.md`, `DICT`). + +## Credentials + +Scripts that call the git host (`sync-project.sh --apply`) need `GITEA_TOKEN` +from the gitignored `.env`; load it with `set -a; . ./.env; set +a` and never +print its values. Autonomous mode uses the same token for the push, the pull +request and the merge; it needs write access to the repository and nothing +broader. + Rules that apply to every document: 1. Get the short name from `framework/registry/artifact-catalog.md` and the next version from `docs/artifact-registry.md`. Create files with `bash framework/scripts/new-artifact.sh `. -2. Owners, reviewers and RACI use stakeholder IDs from the project's - Stakeholder Analysis, never invented role names. -3. Every QC criterion is tagged with an ISO/IEC 25010:2023 characteristic. +2. Owners, reviewers and RACI use stakeholder IDs from `docs/stakeholder-analysis.md` + (`SA-001`), never invented role names. +3. Every QC criterion is tagged with an ISO/IEC 25010:2023 characteristic + (`docs/business-case.md`, `BC-001`). 4. Every reviewed instance gets an `RC-*` record in `docs/sqa/reviews/`. -5. Do not edit `framework/` from this project; propose changes upstream. +5. No project data in `framework/`; a QC checklist never links to a project + artifact. Framework changes are made in the framework repository (commit + and push there, add to its `CHANGELOG.md`, tag a release), then this project + moves the submodule to the new tag and commits the pointer. 6. Every PR description closes the issues its work completes, one `Closes #N` per line (`Refs #N` for partial work); see the `project-planning` skill. 7. Every document's `## Version History` has `Change` and `Commit` columns and keeps the two latest rows. After committing, run `framework/scripts/resolve-pending-commits.sh` and commit the result before - opening the PR (no amend). Never ask for or perform the merge: a reviewer - merges. -8. Keep the wiki current. A PR that changes a workflow under `.gitea/`, a file - in `guides/` or `README.md` also updates the matching pages in `wiki/`: - run `python tools/build_wiki.py` for the English pages (the tests fail when - they are out of date) and update the Danish (`da`) and Bahasa Malaysia - (`ms`) pages once they exist, naming the English source they follow. List - the wiki pages the PR changed in its description, or say "No wiki pages - changed: ". Never push to the wiki repository by hand: the publishing - workflow (MIL-005 task 8) copies `wiki/` there when the PR is merged. + opening the PR (no amend). Supervised mode: never ask for or perform the + merge; a reviewer merges. Autonomous mode: the AI merges, as set out in + `Modes`. From 8051cc4489ee9bcf3b6c62f2eafcb2013b7d2f90 Mon Sep 17 00:00:00 2001 From: Jens Tirsvad Nielsen Date: Tue, 13 Oct 2026 11:37:24 +0800 Subject: [PATCH 4/7] MIL-006: add task 6, fix the page title of hyphenated wiki pages Amends the accepted milestone with task 6 and records the review as RC-058 (autonomous, AI as stand-in for S01). The fix itself follows in a second pull request under the Task: MIL-006#6 trailer. --- docs/artifact-registry.md | 2 +- .../mil-006-shared-wiki-publishing.md | 5 +- docs/sqa/reviews/rc-058-mil-006-task-6.md | 50 +++++++++++++++++++ docs/sqa/traceability-matrix.md | 10 ++-- 4 files changed, 60 insertions(+), 7 deletions(-) create mode 100644 docs/sqa/reviews/rc-058-mil-006-task-6.md diff --git a/docs/artifact-registry.md b/docs/artifact-registry.md index 34dad47..0ae85cd 100644 --- a/docs/artifact-registry.md +++ b/docs/artifact-registry.md @@ -11,7 +11,7 @@ each document lives in *this* project and the next version to use. | SA | Stakeholder Analysis | docs/stakeholder-analysis.md | 002 | | PP | Project Plan | docs/project-plan.md | 002 | | MIL | Milestone / Gateway | docs/milestones/*.md | 008 | -| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 058 | +| RC | SQA Review Record | docs/sqa/reviews/rc-*.md | 059 | | TM | Traceability Matrix | docs/sqa/traceability-matrix.md | 002 | | ADR | Architecture Decision Record | docs/adr/adr-NNNN-*.md | 0009 | | UCD | Use Case Diagram | docs/use-case-diagram.md | 002 | diff --git a/docs/milestones/mil-006-shared-wiki-publishing.md b/docs/milestones/mil-006-shared-wiki-publishing.md index 4744e7d..d36eaed 100644 --- a/docs/milestones/mil-006-shared-wiki-publishing.md +++ b/docs/milestones/mil-006-shared-wiki-publishing.md @@ -9,8 +9,8 @@ ## Version History | Date | Status | Author | Reviewer | Change | Commit | | --- | --- | --- | --- | --- | --- | -| 2026-10-10 | Deprecated | Jens Tirsvad Nielsen | S01 | Target date 2026-12-26 confirmed on 2026-10-10 by S01 (stand-in for S05) | [ea06eaa] | -| 2026-10-11 | Accepted | Jens Tirsvad Nielsen | S01 | Criterion 1 evidence amended; added the verification results of the pilot | [d575f27] | +| 2026-10-11 | Deprecated | Jens Tirsvad Nielsen | S01 | Criterion 1 evidence amended; added the verification results of the pilot | [d575f27] | +| 2026-10-13 | Accepted | Jens Tirsvad Nielsen | AI as stand-in for S01 | Task 6 added and reviewed (RC-058): the publisher sends page titles with spaces, so a hyphenated page (`Home-da`) is stored as `Home-da.md` and its links resolve | pending | --- @@ -71,6 +71,7 @@ repository other than this one that publishes its wiki through it. | 3 | Implement the opt-in gate and the safety limits | Skip, successfully and with a log line, any repository that has not opted in. Keep the refusal on an empty `wiki/` and the categorised errors, and add offline tests with a fake Gitea for opted-in, not opted-in and missing-secret runs. | No | | | 4 | Pilot in one other repository | Choose one repository outside this one, add a `wiki/` folder with the marker, store `TOKEN_FOR_GITEA_WIKI` in its organisation (S03), publish, and read the pages back. Also confirm that a repository without the marker was not touched. | No | | | 5 | Update the guides and the wiki | Describe how a repository opts in and which secrets its organisation needs in the onboarding and credentials guides, regenerate the wiki pages and update the Danish and Bahasa Malaysia translations (rule 8 of `AGENTS.md`). | No | | +| 6 | Fix the page title of hyphenated pages | Gitea keeps a hyphen in a title literal, so the publisher created `Home-da` as `Home-da.-.md`, addressed `Home%2Dda`, and every link to `Home-da` led to the page list (found on the `sqa-and-qc`, `sqa-qc-framework` and `sqa-qc-checklists` wikis). Send the title with spaces (`Home da`, stored `Home-da.md`), cover it with a test and a changelog entry. The stale `*.-.md` pages are deleted after the release, by hand. | No | | ## Verification results diff --git a/docs/sqa/reviews/rc-058-mil-006-task-6.md b/docs/sqa/reviews/rc-058-mil-006-task-6.md new file mode 100644 index 0000000..6addfea --- /dev/null +++ b/docs/sqa/reviews/rc-058-mil-006-task-6.md @@ -0,0 +1,50 @@ +# Review Record: MIL-006 (task 6 amendment) + +## Metadata +| Key | Value | +| --- | --- | +| ID | RC-058 | +| CrossReference | [MIL-006], [QC-MIL-001] | + +## Version History +| Date | Status | Author | Reviewer | Change | Commit | +| --- | --- | --- | --- | --- | --- | +| 2026-10-13 | Accepted | Jens Tirsvad Nielsen | AI as stand-in for S01 | Initial version | pending | + +--- + +## Artifact Under Review + +- Instance reviewed: [MIL-006] after the amendment of 2026-10-13: task 6 (fix the page title of hyphenated pages) added +- Checklist used: [QC-MIL-001] +- Scope: delta re-review of the added task 6 and of the unchanged criteria it touches; the earlier record is [RC-049] +- Language and domain: n/a (technical type) +- Language reviewer: none +- Mode: autonomous, not independent (the reviewer is `AI as stand-in for S01`; the user accepted the plan in chat on 2026-10-13) + +## Checklist Results + +| # | Criterion | Status | Evidence/Notes | +| --- | --- | --- | --- | +| 1 | A concrete deliverable is defined for every gate | Pass | Task 6 states the change (title sent with spaces), a test and a changelog entry. | +| 2 | Explicit Go/No-Go criteria are stated for each gate | Pass | Unchanged: the six criteria. The fix serves criterion 3 (pages equal `wiki/`), which the hyphen defect made unreachable for links. | +| 3 | Dependencies on other milestones are explicitly mapped | Pass | Unchanged: MIL-005 and MIL-002. | +| 4 | Each milestone is traceable to a Business Case objective or KPI | Pass | Unchanged: O2, O3. | +| 5 | Milestone owner and approving reviewer are identified | Pass | Owner S05, approving reviewer S01. | +| 6 | Milestone has a defined target date consistent with project constraints | Pass | 2026-12-26, unchanged. | + +## Overall Verdict + +Go - all six criteria pass. It restores the status Accepted of MIL-006, which the publishing action and the plan-first gate require before the fix is committed. S01 is stood in for by the AI as the user chose autonomous mode in chat on 2026-10-13; there is no independent reader, and the decision stays open to challenge in the run summary. + +## Action Items + +| Action | Owner | Due | +| --- | --- | --- | +| Release the fix, re-run publish-wiki in `sqa-and-qc`, `sqa-qc-framework` and `sqa-qc-checklists`, delete the stale `*.-.md` pages | S05 | after the fix is merged | + +--- + +[MIL-006]: ../../milestones/mil-006-shared-wiki-publishing.md +[RC-049]: ./rc-049-mil-006-outcome.md +[QC-MIL-001]: ../../../framework/qc/qc-milestones-gateways.md diff --git a/docs/sqa/traceability-matrix.md b/docs/sqa/traceability-matrix.md index 676d636..821d2ad 100644 --- a/docs/sqa/traceability-matrix.md +++ b/docs/sqa/traceability-matrix.md @@ -4,13 +4,13 @@ | Key | Value | | --- | --- | | ID | TM-001 | -| CrossReference | [BC-001], [SA-001], [PP-001], [MIL-001], [MIL-002], [MIL-003], [MIL-004], [MIL-005], [MIL-006], [MIL-007], [RC-001], [RC-002], [RC-003], [RC-004], [RC-005], [RC-006], [RC-007], [RC-008], [RC-009], [RC-010], [RC-011], [RC-012], [RC-013], [RC-014], [RC-015], [RC-016], [RC-017], [RC-018], [RC-019], [RC-020], [RC-021], [RC-022], [RC-023], [RC-024], [RC-025], [RC-026], [RC-027], [RC-028], [RC-029], [RC-030], [RC-031], [RC-032], [RC-033], [RC-034], [RC-035], [RC-036], [RC-037], [RC-038], [RC-039], [RC-040], [RC-041], [RC-042], [RC-043], [RC-044], [RC-045], [RC-046], [RC-047], [RC-048], [RC-049], [RC-050], [RC-051], [RC-052], [RC-053], [RC-054], [RC-055], [RC-056], [RC-057], [ADR-0007], [ADR-0008], [UC-007], [UC-006], [ADR-0006], [UC-005], [ADR-0005], [UCD-001], [US-001], [UC-001], [UC-002], [UC-003], [UC-004], [ADR-0001], [ADR-0002], [ADR-0003], [ADR-0004] | +| CrossReference | [BC-001], [SA-001], [PP-001], [MIL-001], [MIL-002], [MIL-003], [MIL-004], [MIL-005], [MIL-006], [MIL-007], [RC-001], [RC-002], [RC-003], [RC-004], [RC-005], [RC-006], [RC-007], [RC-008], [RC-009], [RC-010], [RC-011], [RC-012], [RC-013], [RC-014], [RC-015], [RC-016], [RC-017], [RC-018], [RC-019], [RC-020], [RC-021], [RC-022], [RC-023], [RC-024], [RC-025], [RC-026], [RC-027], [RC-028], [RC-029], [RC-030], [RC-031], [RC-032], [RC-033], [RC-034], [RC-035], [RC-036], [RC-037], [RC-038], [RC-039], [RC-040], [RC-041], [RC-042], [RC-043], [RC-044], [RC-045], [RC-046], [RC-047], [RC-048], [RC-049], [RC-050], [RC-051], [RC-052], [RC-053], [RC-054], [RC-055], [RC-056], [RC-057], [RC-058], [ADR-0007], [ADR-0008], [UC-007], [UC-006], [ADR-0006], [UC-005], [ADR-0005], [UCD-001], [US-001], [UC-001], [UC-002], [UC-003], [UC-004], [ADR-0001], [ADR-0002], [ADR-0003], [ADR-0004] | ## Version History | Date | Status | Author | Reviewer | Change | Commit | | --- | --- | --- | --- | --- | --- | -| 2026-10-11 | Deprecated | Jens Tirsvad Nielsen | S01 | Added RC-055 (MIL-007 amendment) | [faf13b2] | -| 2026-10-11 | Accepted | Jens Tirsvad Nielsen | S01 | Added RC-056 and RC-057 (UC-005 and US-001 for C++ and C#) | [1bc56f8] | +| 2026-10-11 | Deprecated | Jens Tirsvad Nielsen | S01 | Added RC-056 and RC-057 (UC-005 and US-001 for C++ and C#) | [1bc56f8] | +| 2026-10-13 | Proposed | Jens Tirsvad Nielsen | S01 | Added RC-058 (MIL-006 task 6 amendment) | pending | --- @@ -38,7 +38,7 @@ updated whenever an artifact instance is created or reviewed. | [RC-005] | RC | [MIL-002] | - | - | | [RC-006] | RC | [MIL-003] | - | - | | [MIL-005] | MIL | [BC-001], [PP-001], [MIL-002] | [MIL-006] | [RC-010], [RC-011], [RC-030], [RC-050] | -| [MIL-006] | MIL | [BC-001], [PP-001], [MIL-002], [MIL-005] | - | [RC-016], [RC-031], [RC-049] | +| [MIL-006] | MIL | [BC-001], [PP-001], [MIL-002], [MIL-005] | - | [RC-016], [RC-031], [RC-049], [RC-058] | | [MIL-007] | MIL | [BC-001], [PP-001], [MIL-003], [MIL-004], [ADR-0008] | - | [RC-052], [RC-055] | | [RC-007] | RC | [MIL-004] | - | - | | [RC-008] | RC | [BC-001] | - | - | @@ -108,6 +108,7 @@ updated whenever an artifact instance is created or reviewed. | [RC-055] | RC | [MIL-007] | - | - | | [RC-056] | RC | [UC-005] | - | - | | [RC-057] | RC | [US-001] | - | - | +| [RC-058] | RC | [MIL-006] | - | - | ## Coverage Notes @@ -195,6 +196,7 @@ updated whenever an artifact instance is created or reviewed. [RC-055]: ./reviews/rc-055-mil-007-amendment.md [RC-056]: ./reviews/rc-056-uc-005-cpp-csharp.md [RC-057]: ./reviews/rc-057-us-001-cpp-csharp.md +[RC-058]: ./reviews/rc-058-mil-006-task-6.md [ADR-0008]: ../adr/adr-0008-cpp-csharp-quality-check-contract.md [ADR-0007]: ../adr/adr-0007-release-publish-destination.md [UC-007]: ../uc-007/uc.md From 629790265222ee1157822c7971034d835e87ef96 Mon Sep 17 00:00:00 2001 From: Jens Tirsvad Nielsen Date: Tue, 13 Oct 2026 11:37:28 +0800 Subject: [PATCH 5/7] Resolve pending commit links --- docs/milestones/mil-006-shared-wiki-publishing.md | 3 ++- docs/sqa/reviews/rc-058-mil-006-task-6.md | 3 ++- docs/sqa/traceability-matrix.md | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/milestones/mil-006-shared-wiki-publishing.md b/docs/milestones/mil-006-shared-wiki-publishing.md index d36eaed..f0a32f0 100644 --- a/docs/milestones/mil-006-shared-wiki-publishing.md +++ b/docs/milestones/mil-006-shared-wiki-publishing.md @@ -10,7 +10,7 @@ | Date | Status | Author | Reviewer | Change | Commit | | --- | --- | --- | --- | --- | --- | | 2026-10-11 | Deprecated | Jens Tirsvad Nielsen | S01 | Criterion 1 evidence amended; added the verification results of the pilot | [d575f27] | -| 2026-10-13 | Accepted | Jens Tirsvad Nielsen | AI as stand-in for S01 | Task 6 added and reviewed (RC-058): the publisher sends page titles with spaces, so a hyphenated page (`Home-da`) is stored as `Home-da.md` and its links resolve | pending | +| 2026-10-13 | Accepted | Jens Tirsvad Nielsen | AI as stand-in for S01 | Task 6 added and reviewed (RC-058): the publisher sends page titles with spaces, so a hyphenated page (`Home-da`) is stored as `Home-da.md` and its links resolve | [8051cc4] | --- @@ -96,3 +96,4 @@ the API and equal `wiki/*.md` on `main` (0 differences). [ADR-0002]: ../adr/adr-0002-s01-approves-as-stand-in.md [ea06eaa]: https://git.tirsystem.com/TirSystem/github-action/commit/ea06eaaa698247d20e4a8e2694d7acbb1de58b63 [d575f27]: https://git.tirsystem.com/TirSystem/github-action/commit/d575f272fcb044e3165951013464177a6e659ea6 +[8051cc4]: https://git.tirsystem.com/TirSystem/github-action/commit/8051cc4489ee9bcf3b6c62f2eafcb2013b7d2f90 diff --git a/docs/sqa/reviews/rc-058-mil-006-task-6.md b/docs/sqa/reviews/rc-058-mil-006-task-6.md index 6addfea..ddcd044 100644 --- a/docs/sqa/reviews/rc-058-mil-006-task-6.md +++ b/docs/sqa/reviews/rc-058-mil-006-task-6.md @@ -9,7 +9,7 @@ ## Version History | Date | Status | Author | Reviewer | Change | Commit | | --- | --- | --- | --- | --- | --- | -| 2026-10-13 | Accepted | Jens Tirsvad Nielsen | AI as stand-in for S01 | Initial version | pending | +| 2026-10-13 | Accepted | Jens Tirsvad Nielsen | AI as stand-in for S01 | Initial version | [8051cc4] | --- @@ -48,3 +48,4 @@ Go - all six criteria pass. It restores the status Accepted of MIL-006, which th [MIL-006]: ../../milestones/mil-006-shared-wiki-publishing.md [RC-049]: ./rc-049-mil-006-outcome.md [QC-MIL-001]: ../../../framework/qc/qc-milestones-gateways.md +[8051cc4]: https://git.tirsystem.com/TirSystem/github-action/commit/8051cc4489ee9bcf3b6c62f2eafcb2013b7d2f90 diff --git a/docs/sqa/traceability-matrix.md b/docs/sqa/traceability-matrix.md index 821d2ad..4a187d8 100644 --- a/docs/sqa/traceability-matrix.md +++ b/docs/sqa/traceability-matrix.md @@ -10,7 +10,7 @@ | Date | Status | Author | Reviewer | Change | Commit | | --- | --- | --- | --- | --- | --- | | 2026-10-11 | Deprecated | Jens Tirsvad Nielsen | S01 | Added RC-056 and RC-057 (UC-005 and US-001 for C++ and C#) | [1bc56f8] | -| 2026-10-13 | Proposed | Jens Tirsvad Nielsen | S01 | Added RC-058 (MIL-006 task 6 amendment) | pending | +| 2026-10-13 | Proposed | Jens Tirsvad Nielsen | S01 | Added RC-058 (MIL-006 task 6 amendment) | [8051cc4] | --- @@ -206,3 +206,4 @@ updated whenever an artifact instance is created or reviewed. [ADR-0005]: ../adr/adr-0005-quality-check-contract.md [faf13b2]: https://git.tirsystem.com/TirSystem/github-action/commit/faf13b2c2a4901c0171977de1ffc2841023f537a [1bc56f8]: https://git.tirsystem.com/TirSystem/github-action/commit/1bc56f8b771531f221f218fc00b147ecc73c051e +[8051cc4]: https://git.tirsystem.com/TirSystem/github-action/commit/8051cc4489ee9bcf3b6c62f2eafcb2013b7d2f90 From 63af3afd138d999d307491981c306248211419a2 Mon Sep 17 00:00:00 2001 From: Jens Tirsvad Nielsen Date: Tue, 13 Oct 2026 11:41:08 +0800 Subject: [PATCH 6/7] publish-wiki: send titles with spaces so Gitea stores Home-da.md Gitea keeps a hyphen in a page title literal and stored Home-da as Home-da.-.md, addressed as Home%2Dda, so a link to Home-da found no page. The publisher now sends the key with hyphens replaced by spaces. Task: MIL-006#6 Closes #119 --- CHANGELOG.md | 186 +++---- tests/test_publish_wiki.py | 972 +++++++++++++++++++------------------ tools/publish_wiki.py | 602 ++++++++++++----------- 3 files changed, 894 insertions(+), 866 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c6d36f6..dcb172c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,89 +1,97 @@ -# Changelog - -All notable changes to the shared workflows of this repository. The format and -the version rules are those of the release process (`guides/release-process.md`, -ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH` -and a moving major tag `vMAJOR`. - -Each release lists its changes in these groups, in this order: - -- **Breaking**: a renamed or removed workflow, input, secret name or output; a - raised default tool version or changed default that can turn a green - consumer red; a changed meaning of pass or fail. A release with a Breaking - entry has a new major version. -- **Added**: a new optional input, workflow or capability. -- **Changed**: a change with no visible effect for existing callers. -- **Fixed**: a fix. - -`python tools/changelog.py check` verifies the file, and -`python tools/changelog.py notes ` prints the text of a release page. - -## [Unreleased] - -### Changed - -- Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a - runner for the host's default limit. -- `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the - moving tag `v4`. No change for callers. -- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:` - blocks of this repository's own `sync-milestones.yml` call - `framework/scripts/ci-publish-milestones.sh`. No change for callers of the - shared workflows. - -## [1.1.0] - 2026-10-11 - -### Added - -- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from - which the workflow fetches its script `tools/quality-checks.sh`. - -### Changed - -- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of - embedding about 100 lines of shell; the checks, messages and results are the - same. - -## [1.0.1] - 2026-10-11 - -### Changed - -- Sync GitHub mirror metadata: a repository without a GitHub push mirror still - succeeds, and the log now ends with the line `Skipped`. -- The scoped workflows fetch their scripts at the tag `v1` instead of `main`, - so a merge to `main` reaches consumers only when it is released. - -## [1.0.0] - 2026-10-10 - -First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The -scoped workflows are delivered by the instance from `main` and cannot be pinned. - -### Added - -- Workflow `release.yml`: started by hand with a version, it releases only the - tip of `main` whose validation run is green and whose changelog entry exists, - creates the version tag, the release page and the major tag, and does nothing - but check until `dry_run` is set to `false`. -- Scoped workflow `sync-github-metadata.yml`: copies the description and topics - of a Gitea repository to its GitHub push mirror on every push to `main`, by - hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped - successfully. -- A credential preflight before any change, and failure messages that name - their category (`configuration`, `credentials`, `permissions`, `api`). -- Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki - when a repository has opted in with `wiki/.publish`; the word `prune` in that - file also deletes pages that are not in `wiki/`. -- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck` - and `shfmt` on the tracked Python and shell files, with the inputs - `languages`, `working-directory` and one pinned version per tool. -- Guides for onboarding a repository, workflow credentials, troubleshooting, - reading quality-check results and releasing, and the same pages on the wiki in - English, Danish and Bahasa Malaysia. - -### Changed - -- The scripts of the scoped workflows are files under `tools/`, fetched by a - checkout of this repository, instead of Python embedded in the YAML. -- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`, - `PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret - `CREDENTIALS_FOR_GITHUB`, which is no longer read. +# Changelog + +All notable changes to the shared workflows of this repository. The format and +the version rules are those of the release process (`guides/release-process.md`, +ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH` +and a moving major tag `vMAJOR`. + +Each release lists its changes in these groups, in this order: + +- **Breaking**: a renamed or removed workflow, input, secret name or output; a + raised default tool version or changed default that can turn a green + consumer red; a changed meaning of pass or fail. A release with a Breaking + entry has a new major version. +- **Added**: a new optional input, workflow or capability. +- **Changed**: a change with no visible effect for existing callers. +- **Fixed**: a fix. + +`python tools/changelog.py check` verifies the file, and +`python tools/changelog.py notes ` prints the text of a release page. + +## [Unreleased] + +### Changed + +- Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a + runner for the host's default limit. +- `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the + moving tag `v4`. No change for callers. +- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:` + blocks of this repository's own `sync-milestones.yml` call + `framework/scripts/ci-publish-milestones.sh`. No change for callers of the + shared workflows. + +### Fixed + +- `publish-wiki` sends a page title with spaces instead of hyphens. Gitea keeps a + hyphen in a title literal and stored `Home-da` as `Home-da.-.md`, addressed as + `Home%2Dda`, so links to `Home-da` led to the page list. A new page is now + stored as `Home-da.md` and addressed as `Home-da`. Pages already created with + the old title must be deleted by hand (the publisher does not prune them). + +## [1.1.0] - 2026-10-11 + +### Added + +- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from + which the workflow fetches its script `tools/quality-checks.sh`. + +### Changed + +- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of + embedding about 100 lines of shell; the checks, messages and results are the + same. + +## [1.0.1] - 2026-10-11 + +### Changed + +- Sync GitHub mirror metadata: a repository without a GitHub push mirror still + succeeds, and the log now ends with the line `Skipped`. +- The scoped workflows fetch their scripts at the tag `v1` instead of `main`, + so a merge to `main` reaches consumers only when it is released. + +## [1.0.0] - 2026-10-10 + +First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The +scoped workflows are delivered by the instance from `main` and cannot be pinned. + +### Added + +- Workflow `release.yml`: started by hand with a version, it releases only the + tip of `main` whose validation run is green and whose changelog entry exists, + creates the version tag, the release page and the major tag, and does nothing + but check until `dry_run` is set to `false`. +- Scoped workflow `sync-github-metadata.yml`: copies the description and topics + of a Gitea repository to its GitHub push mirror on every push to `main`, by + hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped + successfully. +- A credential preflight before any change, and failure messages that name + their category (`configuration`, `credentials`, `permissions`, `api`). +- Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki + when a repository has opted in with `wiki/.publish`; the word `prune` in that + file also deletes pages that are not in `wiki/`. +- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck` + and `shfmt` on the tracked Python and shell files, with the inputs + `languages`, `working-directory` and one pinned version per tool. +- Guides for onboarding a repository, workflow credentials, troubleshooting, + reading quality-check results and releasing, and the same pages on the wiki in + English, Danish and Bahasa Malaysia. + +### Changed + +- The scripts of the scoped workflows are files under `tools/`, fetched by a + checkout of this repository, instead of Python embedded in the YAML. +- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`, + `PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret + `CREDENTIALS_FOR_GITHUB`, which is no longer read. diff --git a/tests/test_publish_wiki.py b/tests/test_publish_wiki.py index d4fefc9..5473167 100644 --- a/tests/test_publish_wiki.py +++ b/tests/test_publish_wiki.py @@ -1,481 +1,491 @@ -"""Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow.""" - -import base64 -import io -import urllib.error - -import pytest -import yaml - -from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module - -API = "https://git.example.test/api/v1" -REPO_URL = f"{API}/repos/TirSystem/repo" -WIKI_URL = f"{REPO_URL}/wiki" - - -@pytest.fixture() -def publish_wiki(): - return load_publish_module() - - -def b64(text: str) -> str: - return base64.b64encode(text.encode()).decode() - - -class TestParseMarker: - @pytest.mark.parametrize( - ("text", "expected"), - [("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)], - ) - def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected): - assert publish_wiki.parse_marker(text) is expected - - @pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"]) - def test_rejects_anything_else(self, publish_wiki, text): - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.parse_marker(text) - assert caught.value.category == "configuration" - assert "prune" in str(caught.value) - - -class TestParseGiteaToken: - def test_returns_the_stripped_token(self, publish_wiki): - assert publish_wiki.parse_gitea_token(" abc \n") == "abc" - - @pytest.mark.parametrize("raw", [None, "", " "]) - def test_rejects_missing_or_empty_token(self, publish_wiki, raw): - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.parse_gitea_token(raw) - assert caught.value.category == "configuration" - assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value) - - -class TestReadPages: - def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path): - (tmp_path / "Home.md").write_bytes(b"home") - (tmp_path / "Home.da.md").write_bytes(b"hjem") - (tmp_path / ".publish").write_bytes(b"prune") - (tmp_path / "Notes.txt").write_bytes(b"ignored") - assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"} - - def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped( - self, publish_wiki, tmp_path - ): - (tmp_path / ".publish").write_text("prune", encoding="utf-8") - with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): - publish_wiki.read_pages(tmp_path) - - -class TestPageKey: - @pytest.mark.parametrize( - ("name", "key"), - [("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"), - ("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")], - ) - def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key): - assert publish_wiki.page_key(name) == key - - def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path): - (tmp_path / "Home da.md").write_bytes(b"hjem") - (tmp_path / "_Sidebar.md").write_bytes(b"menu") - assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"} - - def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path): - (tmp_path / "Home da.md").write_bytes(b"one") - (tmp_path / "Home-da.md").write_bytes(b"two") - with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"): - publish_wiki.read_pages(tmp_path) - - -class FakeGitea: - """Answers like the Gitea repository and wiki endpoints.""" - - def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True): - self.error = publish_wiki.WorkflowError - self.wiki = dict(wiki or {}) - self.has_wiki = has_wiki - self.wiki_initialised = wiki_initialised - self.calls = [] - - def __call__(self, url, method="GET", headers=None, body=None): - self.calls.append((method, url, body, headers)) - if url == REPO_URL: - return {"has_wiki": self.has_wiki} - if url.startswith(f"{WIKI_URL}/pages"): - if not self.wiki_initialised: - raise self.error("configuration", "404") - return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki] - if method == "POST" and url == f"{WIKI_URL}/new": - self.wiki[body["title"]] = body["content_base64"] - return None - title = url.rsplit("/", 1)[-1] - if method == "GET": - return {"title": title, "content_base64": self.wiki[title]} - if method == "PATCH": - self.wiki[title] = body["content_base64"] - return None - if method == "DELETE": - del self.wiki[title] - return None - raise AssertionError(f"unexpected request {method} {url}") - - @property - def writes(self): - return [call for call in self.calls if call[0] != "GET"] - - -@pytest.fixture() -def environment(monkeypatch, tmp_path): - wiki_dir = tmp_path / "wiki" - wiki_dir.mkdir() - (wiki_dir / "Home.md").write_text("home v2", encoding="utf-8") - (wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8") - (wiki_dir / ".publish").write_text("prune\n", encoding="utf-8") - monkeypatch.setenv("WIKI_DIR", str(wiki_dir)) - monkeypatch.setenv("GITEA_API_URL", f"{API}/") - monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") - monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token") - return wiki_dir - - -def run_main(publish_wiki, gitea): - publish_wiki.request_json = gitea - return publish_wiki.main() - - -class TestOptIn: - def test_a_repository_without_the_marker_is_never_written_to( - self, publish_wiki, environment, capsys - ): - (environment / ".publish").unlink() - gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")}) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.calls == [] - assert "wiki publishing is not enabled" in capsys.readouterr().out - assert gitea.wiki == {"Old": b64("hand written")} - - def test_a_repository_without_the_marker_does_not_need_the_secret( - self, publish_wiki, environment, monkeypatch - ): - (environment / ".publish").unlink() - monkeypatch.setenv("WIKI_CREDENTIALS", "") - assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0 - - def test_a_repository_without_a_wiki_directory_is_skipped( - self, publish_wiki, monkeypatch, tmp_path - ): - monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing")) - monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") - gitea = FakeGitea(publish_wiki) - assert run_main(publish_wiki, gitea) == 0 - assert gitea.calls == [] - - def test_an_opted_in_repository_without_the_secret_fails_before_any_request( - self, publish_wiki, environment, monkeypatch - ): - monkeypatch.setenv("WIKI_CREDENTIALS", "") - gitea = FakeGitea(publish_wiki) - - with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"): - run_main(publish_wiki, gitea) - - assert gitea.calls == [] - - def test_an_unknown_marker_content_is_refused_before_any_request( - self, publish_wiki, environment - ): - (environment / ".publish").write_text("mirror everything", encoding="utf-8") - gitea = FakeGitea(publish_wiki) - - with pytest.raises(publish_wiki.WorkflowError, match="prune"): - run_main(publish_wiki, gitea) - - assert gitea.calls == [] - - -class TestPublish: - def test_prune_creates_updates_and_deletes_to_match_the_source( - self, publish_wiki, environment, capsys - ): - gitea = FakeGitea( - publish_wiki, - wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")}, - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")} - methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls] - assert ("PATCH", "Home") in methods - assert ("DELETE", "Old") in methods - assert ("PATCH", "Credentials") not in methods - assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out - - def test_without_prune_pages_outside_the_source_are_kept( - self, publish_wiki, environment, capsys - ): - (environment / ".publish").write_text("", encoding="utf-8") - gitea = FakeGitea( - publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")} - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.wiki["Old"] == b64("hand written") - assert gitea.wiki["Home"] == b64("home v2") - assert "Credentials" in gitea.wiki - assert not [call for call in gitea.calls if call[0] == "DELETE"] - out = capsys.readouterr().out - assert "1 created, 1 updated, 0 deleted" in out - assert "1 page(s) are in the wiki but not in" in out - assert "prune" in out - - def test_creates_every_page_when_the_wiki_was_never_written( - self, publish_wiki, environment - ): - gitea = FakeGitea(publish_wiki, wiki_initialised=False) - - assert run_main(publish_wiki, gitea) == 0 - - assert set(gitea.wiki) == {"Home", "Credentials"} - assert [call[0] for call in gitea.calls].count("POST") == 2 - - def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment): - gitea = FakeGitea( - publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")} - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.writes == [] - - def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys): - gitea = FakeGitea(publish_wiki) - - assert run_main(publish_wiki, gitea) == 0 - - assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"} - output = capsys.readouterr() - assert "wiki-token" not in output.out + output.err - - def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing( - self, publish_wiki, environment - ): - gitea = FakeGitea(publish_wiki, has_wiki=False) - - with pytest.raises(publish_wiki.WorkflowError) as caught: - run_main(publish_wiki, gitea) - - assert caught.value.category == "configuration" - assert gitea.writes == [] - - def test_an_empty_source_never_deletes_pages_even_with_prune( - self, publish_wiki, environment - ): - for path in environment.glob("*.md"): - path.unlink() - gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")}) - - with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): - run_main(publish_wiki, gitea) - - assert gitea.calls == [] - assert gitea.wiki == {"Home": b64("keep me")} - - -class TestPageIdentity: - """A wiki that already has pages named with spaces, as Gitea stores them.""" - - def test_a_file_named_with_a_space_updates_the_existing_page( - self, publish_wiki, environment - ): - (environment / "Home.md").unlink() - (environment / "Credentials.md").unlink() - (environment / ".publish").write_text("", encoding="utf-8") - (environment / "Home da.md").write_text("hjem v2", encoding="utf-8") - gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")}) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.wiki == {"Home-da": b64("hjem v2")} - assert [call[0] for call in gitea.calls].count("POST") == 0 - assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls] - - def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment): - (environment / "Home.md").unlink() - (environment / "Credentials.md").unlink() - (environment / "Home da.md").write_text("hjem", encoding="utf-8") - (environment / "_Sidebar.md").write_text("menu", encoding="utf-8") - gitea = FakeGitea( - publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")} - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.writes == [] - - def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling( - self, publish_wiki, environment - ): - (environment / "Home.md").unlink() - (environment / "Credentials.md").unlink() - (environment / "Install ms.md").write_text("pasang", encoding="utf-8") - gitea = FakeGitea( - publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")} - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.wiki == {"Install-ms": b64("pasang")} - - def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki): - def listing(url, headers=None, **kwargs): - return [ - {"title": "Whatever", "sub_url": "Real-Address"}, - {"title": "Only title"}, - {"title": "Encoded", "sub_url": "Caf%C3%A9"}, - ] - - keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing) - - assert keys == {"Real-Address", "Only-title", "Café"} - - -class TestDescribeToken: - def test_recognises_a_gitea_token_shape(self, publish_wiki): - description = publish_wiki.describe_token("a1b2c3d4e5" * 4) - assert "40 characters" in description - assert "looks like a Gitea token" in description - - @pytest.mark.parametrize( - ("token", "expected"), - [ - ("*******", "only mask characters"), - ("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"), - ('"' + "a" * 40 + '"', "quote character"), - ("abc def", "whitespace"), - ], - ) - def test_names_the_likely_mistake_without_echoing_the_token( - self, publish_wiki, token, expected - ): - description = publish_wiki.describe_token(token) - assert expected in description - assert token not in description - - -class TestRejectedToken: - def test_rejection_adds_the_token_shape_but_not_the_token( - self, publish_wiki, environment, monkeypatch - ): - monkeypatch.setenv("WIKI_CREDENTIALS", "*******") - - def reject(url, method="GET", headers=None, body=None): - raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.") - - publish_wiki.request_json = reject - - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.main() - - message = str(caught.value) - assert caught.value.category == "credentials" - assert "Token shape: 7 characters" in message - assert "only mask characters" in message - assert "*******" not in message.split("Token shape:")[0] - - def test_other_categories_are_not_changed(self, publish_wiki, environment): - def forbid(url, method="GET", headers=None, body=None): - raise publish_wiki.WorkflowError("permissions", "Forbidden.") - - publish_wiki.request_json = forbid - - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.main() - - assert str(caught.value) == "[permissions] Forbidden." - - -class TestRequestJson: - @staticmethod - def raise_http_error(publish_wiki, monkeypatch, code, reason): - def fake_urlopen(request, timeout): - raise urllib.error.HTTPError( - request.full_url, code, reason, {}, io.BytesIO(b"secret detail") - ) - - monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) - - @pytest.mark.parametrize( - ("code", "reason", "category"), - [ - (401, "Unauthorized", "credentials"), - (403, "Forbidden", "permissions"), - (404, "Not Found", "configuration"), - (500, "Server Error", "api"), - ], - ) - def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category): - self.raise_http_error(publish_wiki, monkeypatch, code, reason) - - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.request_json(REPO_URL) - - assert caught.value.category == category - assert f"HTTP {code}" in str(caught.value) - assert "secret detail" not in str(caught.value) - - def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch): - def fake_urlopen(request, timeout): - raise urllib.error.URLError("no route") - - monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) - - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.request_json(REPO_URL) - - assert caught.value.category == "api" - - -class TestWorkflow: - @pytest.fixture() - def document(self): - return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8")) - - def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document): - triggers = document[True] # PyYAML reads the key `on` as True - assert triggers["push"]["branches"] == ["main"] - assert triggers["push"]["paths"] == ["wiki/**"] - assert "workflow_dispatch" in triggers - - def test_job_has_read_only_permissions(self, document): - assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"} - - def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document): - text = PUBLISH_WORKFLOW.read_text(encoding="utf-8") - assert "secrets.TOKEN_FOR_GITEA }}" not in text - publish_step = document["jobs"]["publish-wiki"]["steps"][-1] - assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}" - assert "secrets" not in publish_step["run"] - - def test_the_workflow_has_one_source_and_runs_the_script_file(self, document): - assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists() - assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file() - steps = document["jobs"]["publish-wiki"]["steps"] - assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8") - assert steps[0]["uses"].startswith("actions/checkout@") - assert steps[1]["with"] == { - "repository": "TirSystem/github-action", - "ref": "v1", - "path": "shared-scripts", - } - assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py" - - -def test_this_repository_opts_in_and_prunes(): - """The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004).""" - marker = REPO_ROOT / "wiki" / ".publish" - assert marker.read_text(encoding="utf-8").strip() == "prune" +"""Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow.""" + +import base64 +import io +import urllib.error + +import pytest +import yaml + +from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module + +API = "https://git.example.test/api/v1" +REPO_URL = f"{API}/repos/TirSystem/repo" +WIKI_URL = f"{REPO_URL}/wiki" + + +@pytest.fixture() +def publish_wiki(): + return load_publish_module() + + +def b64(text: str) -> str: + return base64.b64encode(text.encode()).decode() + + +class TestParseMarker: + @pytest.mark.parametrize( + ("text", "expected"), + [("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)], + ) + def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected): + assert publish_wiki.parse_marker(text) is expected + + @pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"]) + def test_rejects_anything_else(self, publish_wiki, text): + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.parse_marker(text) + assert caught.value.category == "configuration" + assert "prune" in str(caught.value) + + +class TestParseGiteaToken: + def test_returns_the_stripped_token(self, publish_wiki): + assert publish_wiki.parse_gitea_token(" abc \n") == "abc" + + @pytest.mark.parametrize("raw", [None, "", " "]) + def test_rejects_missing_or_empty_token(self, publish_wiki, raw): + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.parse_gitea_token(raw) + assert caught.value.category == "configuration" + assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value) + + +class TestReadPages: + def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path): + (tmp_path / "Home.md").write_bytes(b"home") + (tmp_path / "Home.da.md").write_bytes(b"hjem") + (tmp_path / ".publish").write_bytes(b"prune") + (tmp_path / "Notes.txt").write_bytes(b"ignored") + assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"} + + def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped( + self, publish_wiki, tmp_path + ): + (tmp_path / ".publish").write_text("prune", encoding="utf-8") + with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): + publish_wiki.read_pages(tmp_path) + + +class TestPageTitle: + @pytest.mark.parametrize( + ("key", "title"), [("Home", "Home"), ("Home-da", "Home da"), ("Getting-Started-ms", "Getting Started ms")] + ) + def test_a_hyphen_is_sent_as_a_space(self, publish_wiki, key, title): + # Gitea keeps a hyphen in a title literal (Home-da.-.md, address Home%2Dda). + assert publish_wiki.page_title(key) == title + + +class TestPageKey: + @pytest.mark.parametrize( + ("name", "key"), + [("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"), + ("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")], + ) + def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key): + assert publish_wiki.page_key(name) == key + + def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path): + (tmp_path / "Home da.md").write_bytes(b"hjem") + (tmp_path / "_Sidebar.md").write_bytes(b"menu") + assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"} + + def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path): + (tmp_path / "Home da.md").write_bytes(b"one") + (tmp_path / "Home-da.md").write_bytes(b"two") + with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"): + publish_wiki.read_pages(tmp_path) + + +class FakeGitea: + """Answers like the Gitea repository and wiki endpoints.""" + + def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True): + self.error = publish_wiki.WorkflowError + self.wiki = dict(wiki or {}) + self.has_wiki = has_wiki + self.wiki_initialised = wiki_initialised + self.calls = [] + + def __call__(self, url, method="GET", headers=None, body=None): + self.calls.append((method, url, body, headers)) + if url == REPO_URL: + return {"has_wiki": self.has_wiki} + if url.startswith(f"{WIKI_URL}/pages"): + if not self.wiki_initialised: + raise self.error("configuration", "404") + return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki] + if method == "POST" and url == f"{WIKI_URL}/new": + self.wiki[body["title"].replace(" ", "-")] = body["content_base64"] + return None + title = url.rsplit("/", 1)[-1] + if method == "GET": + return {"title": title, "content_base64": self.wiki[title]} + if method == "PATCH": + assert " " in body["title"] or "-" not in body["title"], body["title"] + self.wiki[title] = body["content_base64"] + return None + if method == "DELETE": + del self.wiki[title] + return None + raise AssertionError(f"unexpected request {method} {url}") + + @property + def writes(self): + return [call for call in self.calls if call[0] != "GET"] + + +@pytest.fixture() +def environment(monkeypatch, tmp_path): + wiki_dir = tmp_path / "wiki" + wiki_dir.mkdir() + (wiki_dir / "Home.md").write_text("home v2", encoding="utf-8") + (wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8") + (wiki_dir / ".publish").write_text("prune\n", encoding="utf-8") + monkeypatch.setenv("WIKI_DIR", str(wiki_dir)) + monkeypatch.setenv("GITEA_API_URL", f"{API}/") + monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") + monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token") + return wiki_dir + + +def run_main(publish_wiki, gitea): + publish_wiki.request_json = gitea + return publish_wiki.main() + + +class TestOptIn: + def test_a_repository_without_the_marker_is_never_written_to( + self, publish_wiki, environment, capsys + ): + (environment / ".publish").unlink() + gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")}) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.calls == [] + assert "wiki publishing is not enabled" in capsys.readouterr().out + assert gitea.wiki == {"Old": b64("hand written")} + + def test_a_repository_without_the_marker_does_not_need_the_secret( + self, publish_wiki, environment, monkeypatch + ): + (environment / ".publish").unlink() + monkeypatch.setenv("WIKI_CREDENTIALS", "") + assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0 + + def test_a_repository_without_a_wiki_directory_is_skipped( + self, publish_wiki, monkeypatch, tmp_path + ): + monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing")) + monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") + gitea = FakeGitea(publish_wiki) + assert run_main(publish_wiki, gitea) == 0 + assert gitea.calls == [] + + def test_an_opted_in_repository_without_the_secret_fails_before_any_request( + self, publish_wiki, environment, monkeypatch + ): + monkeypatch.setenv("WIKI_CREDENTIALS", "") + gitea = FakeGitea(publish_wiki) + + with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"): + run_main(publish_wiki, gitea) + + assert gitea.calls == [] + + def test_an_unknown_marker_content_is_refused_before_any_request( + self, publish_wiki, environment + ): + (environment / ".publish").write_text("mirror everything", encoding="utf-8") + gitea = FakeGitea(publish_wiki) + + with pytest.raises(publish_wiki.WorkflowError, match="prune"): + run_main(publish_wiki, gitea) + + assert gitea.calls == [] + + +class TestPublish: + def test_prune_creates_updates_and_deletes_to_match_the_source( + self, publish_wiki, environment, capsys + ): + gitea = FakeGitea( + publish_wiki, + wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")}, + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")} + methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls] + assert ("PATCH", "Home") in methods + assert ("DELETE", "Old") in methods + assert ("PATCH", "Credentials") not in methods + assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out + + def test_without_prune_pages_outside_the_source_are_kept( + self, publish_wiki, environment, capsys + ): + (environment / ".publish").write_text("", encoding="utf-8") + gitea = FakeGitea( + publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")} + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.wiki["Old"] == b64("hand written") + assert gitea.wiki["Home"] == b64("home v2") + assert "Credentials" in gitea.wiki + assert not [call for call in gitea.calls if call[0] == "DELETE"] + out = capsys.readouterr().out + assert "1 created, 1 updated, 0 deleted" in out + assert "1 page(s) are in the wiki but not in" in out + assert "prune" in out + + def test_creates_every_page_when_the_wiki_was_never_written( + self, publish_wiki, environment + ): + gitea = FakeGitea(publish_wiki, wiki_initialised=False) + + assert run_main(publish_wiki, gitea) == 0 + + assert set(gitea.wiki) == {"Home", "Credentials"} + assert [call[0] for call in gitea.calls].count("POST") == 2 + + def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment): + gitea = FakeGitea( + publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")} + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.writes == [] + + def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys): + gitea = FakeGitea(publish_wiki) + + assert run_main(publish_wiki, gitea) == 0 + + assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"} + output = capsys.readouterr() + assert "wiki-token" not in output.out + output.err + + def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing( + self, publish_wiki, environment + ): + gitea = FakeGitea(publish_wiki, has_wiki=False) + + with pytest.raises(publish_wiki.WorkflowError) as caught: + run_main(publish_wiki, gitea) + + assert caught.value.category == "configuration" + assert gitea.writes == [] + + def test_an_empty_source_never_deletes_pages_even_with_prune( + self, publish_wiki, environment + ): + for path in environment.glob("*.md"): + path.unlink() + gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")}) + + with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): + run_main(publish_wiki, gitea) + + assert gitea.calls == [] + assert gitea.wiki == {"Home": b64("keep me")} + + +class TestPageIdentity: + """A wiki that already has pages named with spaces, as Gitea stores them.""" + + def test_a_file_named_with_a_space_updates_the_existing_page( + self, publish_wiki, environment + ): + (environment / "Home.md").unlink() + (environment / "Credentials.md").unlink() + (environment / ".publish").write_text("", encoding="utf-8") + (environment / "Home da.md").write_text("hjem v2", encoding="utf-8") + gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")}) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.wiki == {"Home-da": b64("hjem v2")} + assert [call[0] for call in gitea.calls].count("POST") == 0 + assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls] + + def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment): + (environment / "Home.md").unlink() + (environment / "Credentials.md").unlink() + (environment / "Home da.md").write_text("hjem", encoding="utf-8") + (environment / "_Sidebar.md").write_text("menu", encoding="utf-8") + gitea = FakeGitea( + publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")} + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.writes == [] + + def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling( + self, publish_wiki, environment + ): + (environment / "Home.md").unlink() + (environment / "Credentials.md").unlink() + (environment / "Install ms.md").write_text("pasang", encoding="utf-8") + gitea = FakeGitea( + publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")} + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.wiki == {"Install-ms": b64("pasang")} + + def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki): + def listing(url, headers=None, **kwargs): + return [ + {"title": "Whatever", "sub_url": "Real-Address"}, + {"title": "Only title"}, + {"title": "Encoded", "sub_url": "Caf%C3%A9"}, + ] + + keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing) + + assert keys == {"Real-Address", "Only-title", "Café"} + + +class TestDescribeToken: + def test_recognises_a_gitea_token_shape(self, publish_wiki): + description = publish_wiki.describe_token("a1b2c3d4e5" * 4) + assert "40 characters" in description + assert "looks like a Gitea token" in description + + @pytest.mark.parametrize( + ("token", "expected"), + [ + ("*******", "only mask characters"), + ("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"), + ('"' + "a" * 40 + '"', "quote character"), + ("abc def", "whitespace"), + ], + ) + def test_names_the_likely_mistake_without_echoing_the_token( + self, publish_wiki, token, expected + ): + description = publish_wiki.describe_token(token) + assert expected in description + assert token not in description + + +class TestRejectedToken: + def test_rejection_adds_the_token_shape_but_not_the_token( + self, publish_wiki, environment, monkeypatch + ): + monkeypatch.setenv("WIKI_CREDENTIALS", "*******") + + def reject(url, method="GET", headers=None, body=None): + raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.") + + publish_wiki.request_json = reject + + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.main() + + message = str(caught.value) + assert caught.value.category == "credentials" + assert "Token shape: 7 characters" in message + assert "only mask characters" in message + assert "*******" not in message.split("Token shape:")[0] + + def test_other_categories_are_not_changed(self, publish_wiki, environment): + def forbid(url, method="GET", headers=None, body=None): + raise publish_wiki.WorkflowError("permissions", "Forbidden.") + + publish_wiki.request_json = forbid + + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.main() + + assert str(caught.value) == "[permissions] Forbidden." + + +class TestRequestJson: + @staticmethod + def raise_http_error(publish_wiki, monkeypatch, code, reason): + def fake_urlopen(request, timeout): + raise urllib.error.HTTPError( + request.full_url, code, reason, {}, io.BytesIO(b"secret detail") + ) + + monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) + + @pytest.mark.parametrize( + ("code", "reason", "category"), + [ + (401, "Unauthorized", "credentials"), + (403, "Forbidden", "permissions"), + (404, "Not Found", "configuration"), + (500, "Server Error", "api"), + ], + ) + def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category): + self.raise_http_error(publish_wiki, monkeypatch, code, reason) + + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.request_json(REPO_URL) + + assert caught.value.category == category + assert f"HTTP {code}" in str(caught.value) + assert "secret detail" not in str(caught.value) + + def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch): + def fake_urlopen(request, timeout): + raise urllib.error.URLError("no route") + + monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) + + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.request_json(REPO_URL) + + assert caught.value.category == "api" + + +class TestWorkflow: + @pytest.fixture() + def document(self): + return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8")) + + def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document): + triggers = document[True] # PyYAML reads the key `on` as True + assert triggers["push"]["branches"] == ["main"] + assert triggers["push"]["paths"] == ["wiki/**"] + assert "workflow_dispatch" in triggers + + def test_job_has_read_only_permissions(self, document): + assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"} + + def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document): + text = PUBLISH_WORKFLOW.read_text(encoding="utf-8") + assert "secrets.TOKEN_FOR_GITEA }}" not in text + publish_step = document["jobs"]["publish-wiki"]["steps"][-1] + assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}" + assert "secrets" not in publish_step["run"] + + def test_the_workflow_has_one_source_and_runs_the_script_file(self, document): + assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists() + assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file() + steps = document["jobs"]["publish-wiki"]["steps"] + assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8") + assert steps[0]["uses"].startswith("actions/checkout@") + assert steps[1]["with"] == { + "repository": "TirSystem/github-action", + "ref": "v1", + "path": "shared-scripts", + } + assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py" + + +def test_this_repository_opts_in_and_prunes(): + """The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004).""" + marker = REPO_ROOT / "wiki" / ".publish" + assert marker.read_text(encoding="utf-8").strip() == "prune" diff --git a/tools/publish_wiki.py b/tools/publish_wiki.py index b098de4..3e8d966 100644 --- a/tools/publish_wiki.py +++ b/tools/publish_wiki.py @@ -1,296 +1,306 @@ -"""Publish the pages in wiki/ to the Gitea wiki of the repository. - -Run by the scoped workflow publish-wiki.yml, which checks out the repository -that owns the wiki and this repository, and runs this file (MIL-004 task 6). -A repository publishes only when it has opted in with wiki/.publish -(ADR-0004); the word prune in that file lets the run delete wiki pages that -are not in wiki/. It reads its settings from the environment: - - GITEA_API_URL Gitea API base, for example https://host/api/v1 - SOURCE_REPOSITORY owner/repo of the repository whose wiki is written - WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI) - WIKI_DIR directory with the pages (default: wiki) - -Every failure is printed as ``ERROR: [category] message`` with the categories -configuration, credentials, permissions and api; no secret value is printed. -""" - -import base64 -import json -import os -import re -import sys -import urllib.error -import urllib.parse -import urllib.request -from pathlib import Path - -PAGE_LIMIT = 50 -REQUEST_TIMEOUT_SECONDS = 30 -COMMIT_MESSAGE = "Publish wiki from the repository" -MARKER_NAME = ".publish" -PRUNE_WORD = "prune" - - -class WorkflowError(RuntimeError): - """A failure with a category. - - configuration: a secret, a setting or the wiki source is missing or malformed. - credentials: Gitea rejected the token (HTTP 401). - permissions: the token is valid but not allowed (HTTP 403). - api: any other API or network failure. - """ - - def __init__(self, category, message): - super().__init__(f"[{category}] {message}") - self.category = category - self.detail = message - - -def request_json(url, method="GET", headers=None, body=None): - request = urllib.request.Request( - url, - data=json.dumps(body).encode("utf-8") if body is not None else None, - headers=headers or {}, - method=method, - ) - try: - with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response: - content = response.read() - return json.loads(content) if content else None - except urllib.error.HTTPError as error: - detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})" - if error.code == 401: - raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None - if error.code == 403: - raise WorkflowError( - "permissions", f"{detail}. The token lacks a required permission." - ) from None - if error.code == 404: - raise WorkflowError( - "configuration", - f"{detail}. The repository or page was not found, or the token " - "cannot see it.", - ) from None - raise WorkflowError("api", detail) from None - except (urllib.error.URLError, TimeoutError) as error: - raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None - - -def parse_marker(text): - """Return True when the marker asks for pruning, False when it is empty.""" - word = (text or "").strip() - if not word: - return False - if word == PRUNE_WORD: - return True - raise WorkflowError( - "configuration", - f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.", - ) - - -def parse_gitea_token(raw_token): - token = (raw_token or "").strip() - if not token: - raise WorkflowError( - "configuration", - "TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of " - "the organization of this repository.", - ) - return token - - -def describe_token(token): - """Describe the shape of a token without revealing any of its characters.""" - facts = [f"{len(token)} characters"] - if re.fullmatch(r"[0-9a-f]{40}", token): - facts.append("looks like a Gitea token (40 hexadecimal characters)") - return ", ".join(facts) - facts.append("a Gitea token has 40 hexadecimal characters") - if set(token) <= {"*", "x", "X", "•"}: - facts.append("only mask characters: the masked text was stored") - if "=" in token: - facts.append("contains '=': the secret name may be part of the value") - if re.search(r"[\"'`]", token): - facts.append("contains a quote character") - if re.search(r"\s", token): - facts.append("contains whitespace") - return ", ".join(facts) - - -def split_repository(full_name): - owner, separator, repo = (full_name or "").partition("/") - if not separator or not owner or not repo: - raise WorkflowError("configuration", "Could not determine the Gitea repository.") - return owner, repo - - -def page_key(title): - """Return the identity of a page: Gitea stores a space as a hyphen. - - The page "Home da" is the file Home-da.md and its address is Home-da, so - both spellings must name the same page. - """ - return str(title).replace(" ", "-") - - -def read_pages(wiki_dir): - """Return {page key: content} for every Markdown file in wiki_dir.""" - pages = {} - for path in sorted(wiki_dir.glob("*.md")): - key = page_key(path.stem) - if key in pages: - raise WorkflowError( - "configuration", - f"Two files in {wiki_dir} name the wiki page {key}; a space and " - "a hyphen are the same in a page name.", - ) - pages[key] = path.read_bytes() - if not pages: - # Publishing an empty directory would delete the whole wiki. - raise WorkflowError( - "configuration", f"The wiki source directory {wiki_dir} has no Markdown pages." - ) - return pages - - -def list_wiki_titles(wiki_url, headers, send): - """Return the keys of the pages that exist in the wiki. - - The key is the address Gitea reports (sub_url), or the title with its - spaces replaced by hyphens. - - Gitea answers 404 for a wiki that has never been written to; that is an - empty wiki, not a failure. - """ - titles = set() - page = 1 - while True: - url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}" - try: - batch = send(url, headers=headers) - except WorkflowError as error: - if error.category == "configuration" and page == 1: - return titles - raise - if not isinstance(batch, list): - raise WorkflowError("api", "Gitea returned an invalid wiki page list.") - titles.update( - urllib.parse.unquote(str(item.get("sub_url") or "")) - or page_key(item.get("title", "")) - for item in batch - ) - if len(batch) < PAGE_LIMIT: - return titles - page += 1 - - -def read_wiki_page(wiki_url, title, headers, send): - quoted = urllib.parse.quote(title, safe="") - page = send(f"{wiki_url}/page/{quoted}", headers=headers) - return base64.b64decode((page or {}).get("content_base64", "")) - - -def publish(pages, wiki_url, headers, send, prune): - """Create and update the pages; delete the others only when prune is set. - - Returns the titles created, updated and deleted, and the titles that were - left in the wiki because pruning is off. - """ - existing = list_wiki_titles(wiki_url, headers, send) - current = { - title: read_wiki_page(wiki_url, title, headers, send) - for title in sorted(existing & pages.keys()) - } - - result = {"created": [], "updated": [], "deleted": [], "kept": []} - for title, content in pages.items(): - body = { - "title": title, - "content_base64": base64.b64encode(content).decode("ascii"), - "message": COMMIT_MESSAGE, - } - if title not in existing: - send(f"{wiki_url}/new", method="POST", headers=headers, body=body) - result["created"].append(title) - elif current[title] != content: - quoted = urllib.parse.quote(title, safe="") - send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body) - result["updated"].append(title) - - for title in sorted(existing - pages.keys()): - if not prune: - result["kept"].append(title) - continue - quoted = urllib.parse.quote(title, safe="") - send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers) - result["deleted"].append(title) - return result - - -def main(): - wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki")) - owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY")) - - marker = wiki_dir / MARKER_NAME - if not marker.is_file(): - print( - f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; " - "wiki publishing is not enabled for this repository." - ) - return 0 - prune = parse_marker(marker.read_text(encoding="utf-8")) - - pages = read_pages(wiki_dir) - token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS")) - - api_url = os.environ["GITEA_API_URL"].rstrip("/") - repository_url = ( - f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/" - f"{urllib.parse.quote(repo, safe='')}" - ) - headers = { - "Authorization": f"token {token}", - "Accept": "application/json", - "Content-Type": "application/json", - } - - try: - repository = request_json(repository_url, headers=headers) - if isinstance(repository, dict) and repository.get("has_wiki") is False: - raise WorkflowError( - "configuration", - f"The wiki is not enabled on {owner}/{repo}. Enable it in the " - "repository settings.", - ) - print("Preflight passed: Gitea accepted the token and the wiki is enabled.") - - result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune) - except WorkflowError as error: - if error.category == "credentials": - raise WorkflowError( - "credentials", f"{error.detail} Token shape: {describe_token(token)}." - ) from None - raise - - print( - f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: " - f"{len(result['created'])} created, {len(result['updated'])} updated, " - f"{len(result['deleted'])} deleted." - ) - if result["kept"]: - print( - f"{len(result['kept'])} page(s) are in the wiki but not in " - f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in " - f"{wiki_dir}/{MARKER_NAME} to delete them." - ) - return 0 - - -if __name__ == "__main__": - try: - sys.exit(main()) - except WorkflowError as error: - print(f"ERROR: {error}", file=sys.stderr) - sys.exit(1) +"""Publish the pages in wiki/ to the Gitea wiki of the repository. + +Run by the scoped workflow publish-wiki.yml, which checks out the repository +that owns the wiki and this repository, and runs this file (MIL-004 task 6). +A repository publishes only when it has opted in with wiki/.publish +(ADR-0004); the word prune in that file lets the run delete wiki pages that +are not in wiki/. It reads its settings from the environment: + + GITEA_API_URL Gitea API base, for example https://host/api/v1 + SOURCE_REPOSITORY owner/repo of the repository whose wiki is written + WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI) + WIKI_DIR directory with the pages (default: wiki) + +Every failure is printed as ``ERROR: [category] message`` with the categories +configuration, credentials, permissions and api; no secret value is printed. +""" + +import base64 +import json +import os +import re +import sys +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path + +PAGE_LIMIT = 50 +REQUEST_TIMEOUT_SECONDS = 30 +COMMIT_MESSAGE = "Publish wiki from the repository" +MARKER_NAME = ".publish" +PRUNE_WORD = "prune" + + +class WorkflowError(RuntimeError): + """A failure with a category. + + configuration: a secret, a setting or the wiki source is missing or malformed. + credentials: Gitea rejected the token (HTTP 401). + permissions: the token is valid but not allowed (HTTP 403). + api: any other API or network failure. + """ + + def __init__(self, category, message): + super().__init__(f"[{category}] {message}") + self.category = category + self.detail = message + + +def request_json(url, method="GET", headers=None, body=None): + request = urllib.request.Request( + url, + data=json.dumps(body).encode("utf-8") if body is not None else None, + headers=headers or {}, + method=method, + ) + try: + with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response: + content = response.read() + return json.loads(content) if content else None + except urllib.error.HTTPError as error: + detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})" + if error.code == 401: + raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None + if error.code == 403: + raise WorkflowError( + "permissions", f"{detail}. The token lacks a required permission." + ) from None + if error.code == 404: + raise WorkflowError( + "configuration", + f"{detail}. The repository or page was not found, or the token " + "cannot see it.", + ) from None + raise WorkflowError("api", detail) from None + except (urllib.error.URLError, TimeoutError) as error: + raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None + + +def parse_marker(text): + """Return True when the marker asks for pruning, False when it is empty.""" + word = (text or "").strip() + if not word: + return False + if word == PRUNE_WORD: + return True + raise WorkflowError( + "configuration", + f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.", + ) + + +def parse_gitea_token(raw_token): + token = (raw_token or "").strip() + if not token: + raise WorkflowError( + "configuration", + "TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of " + "the organization of this repository.", + ) + return token + + +def describe_token(token): + """Describe the shape of a token without revealing any of its characters.""" + facts = [f"{len(token)} characters"] + if re.fullmatch(r"[0-9a-f]{40}", token): + facts.append("looks like a Gitea token (40 hexadecimal characters)") + return ", ".join(facts) + facts.append("a Gitea token has 40 hexadecimal characters") + if set(token) <= {"*", "x", "X", "•"}: + facts.append("only mask characters: the masked text was stored") + if "=" in token: + facts.append("contains '=': the secret name may be part of the value") + if re.search(r"[\"'`]", token): + facts.append("contains a quote character") + if re.search(r"\s", token): + facts.append("contains whitespace") + return ", ".join(facts) + + +def split_repository(full_name): + owner, separator, repo = (full_name or "").partition("/") + if not separator or not owner or not repo: + raise WorkflowError("configuration", "Could not determine the Gitea repository.") + return owner, repo + + +def page_key(title): + """Return the identity of a page: Gitea stores a space as a hyphen. + + The page "Home da" is the file Home-da.md and its address is Home-da, so + both spellings must name the same page. + """ + return str(title).replace(" ", "-") + + +def page_title(key): + """Return the title to send to Gitea for a page key. + + A hyphen in a title is a literal hyphen to Gitea, which then stores the page + as Home-da.-.md and addresses it as Home%2Dda, so a link to Home-da finds + nothing. A space is stored as a hyphen (Home-da.md) and addressed as Home-da. + """ + return key.replace("-", " ") + + +def read_pages(wiki_dir): + """Return {page key: content} for every Markdown file in wiki_dir.""" + pages = {} + for path in sorted(wiki_dir.glob("*.md")): + key = page_key(path.stem) + if key in pages: + raise WorkflowError( + "configuration", + f"Two files in {wiki_dir} name the wiki page {key}; a space and " + "a hyphen are the same in a page name.", + ) + pages[key] = path.read_bytes() + if not pages: + # Publishing an empty directory would delete the whole wiki. + raise WorkflowError( + "configuration", f"The wiki source directory {wiki_dir} has no Markdown pages." + ) + return pages + + +def list_wiki_titles(wiki_url, headers, send): + """Return the keys of the pages that exist in the wiki. + + The key is the address Gitea reports (sub_url), or the title with its + spaces replaced by hyphens. + + Gitea answers 404 for a wiki that has never been written to; that is an + empty wiki, not a failure. + """ + titles = set() + page = 1 + while True: + url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}" + try: + batch = send(url, headers=headers) + except WorkflowError as error: + if error.category == "configuration" and page == 1: + return titles + raise + if not isinstance(batch, list): + raise WorkflowError("api", "Gitea returned an invalid wiki page list.") + titles.update( + urllib.parse.unquote(str(item.get("sub_url") or "")) + or page_key(item.get("title", "")) + for item in batch + ) + if len(batch) < PAGE_LIMIT: + return titles + page += 1 + + +def read_wiki_page(wiki_url, title, headers, send): + quoted = urllib.parse.quote(title, safe="") + page = send(f"{wiki_url}/page/{quoted}", headers=headers) + return base64.b64decode((page or {}).get("content_base64", "")) + + +def publish(pages, wiki_url, headers, send, prune): + """Create and update the pages; delete the others only when prune is set. + + Returns the titles created, updated and deleted, and the titles that were + left in the wiki because pruning is off. + """ + existing = list_wiki_titles(wiki_url, headers, send) + current = { + title: read_wiki_page(wiki_url, title, headers, send) + for title in sorted(existing & pages.keys()) + } + + result = {"created": [], "updated": [], "deleted": [], "kept": []} + for title, content in pages.items(): + body = { + "title": page_title(title), + "content_base64": base64.b64encode(content).decode("ascii"), + "message": COMMIT_MESSAGE, + } + if title not in existing: + send(f"{wiki_url}/new", method="POST", headers=headers, body=body) + result["created"].append(title) + elif current[title] != content: + quoted = urllib.parse.quote(title, safe="") + send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body) + result["updated"].append(title) + + for title in sorted(existing - pages.keys()): + if not prune: + result["kept"].append(title) + continue + quoted = urllib.parse.quote(title, safe="") + send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers) + result["deleted"].append(title) + return result + + +def main(): + wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki")) + owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY")) + + marker = wiki_dir / MARKER_NAME + if not marker.is_file(): + print( + f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; " + "wiki publishing is not enabled for this repository." + ) + return 0 + prune = parse_marker(marker.read_text(encoding="utf-8")) + + pages = read_pages(wiki_dir) + token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS")) + + api_url = os.environ["GITEA_API_URL"].rstrip("/") + repository_url = ( + f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/" + f"{urllib.parse.quote(repo, safe='')}" + ) + headers = { + "Authorization": f"token {token}", + "Accept": "application/json", + "Content-Type": "application/json", + } + + try: + repository = request_json(repository_url, headers=headers) + if isinstance(repository, dict) and repository.get("has_wiki") is False: + raise WorkflowError( + "configuration", + f"The wiki is not enabled on {owner}/{repo}. Enable it in the " + "repository settings.", + ) + print("Preflight passed: Gitea accepted the token and the wiki is enabled.") + + result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune) + except WorkflowError as error: + if error.category == "credentials": + raise WorkflowError( + "credentials", f"{error.detail} Token shape: {describe_token(token)}." + ) from None + raise + + print( + f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: " + f"{len(result['created'])} created, {len(result['updated'])} updated, " + f"{len(result['deleted'])} deleted." + ) + if result["kept"]: + print( + f"{len(result['kept'])} page(s) are in the wiki but not in " + f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in " + f"{wiki_dir}/{MARKER_NAME} to delete them." + ) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except WorkflowError as error: + print(f"ERROR: {error}", file=sys.stderr) + sys.exit(1) From df525d129efe64fe10ee8556f96efa41a7ddeefc Mon Sep 17 00:00:00 2001 From: Jens Tirsvad Nielsen Date: Tue, 13 Oct 2026 11:50:35 +0800 Subject: [PATCH 7/7] Restore LF line endings and document hyphenated page names The fix of #120 rewrote CHANGELOG.md, tools/publish_wiki.py and tests/test_publish_wiki.py with CRLF line endings, which conflicts with every open branch. This restores LF and adds the page-name rule to the onboarding guide and its wiki pages. Task: MIL-006#6 Refs #119 --- CHANGELOG.md | 197 ++++---- guides/onboarding.md | 3 +- tests/test_publish_wiki.py | 982 ++++++++++++++++++------------------- tools/publish_wiki.py | 612 +++++++++++------------ wiki/Onboarding.da.md | 4 +- wiki/Onboarding.md | 3 +- wiki/Onboarding.ms.md | 4 +- 7 files changed, 905 insertions(+), 900 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dcb172c..d93369c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,97 +1,100 @@ -# Changelog - -All notable changes to the shared workflows of this repository. The format and -the version rules are those of the release process (`guides/release-process.md`, -ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH` -and a moving major tag `vMAJOR`. - -Each release lists its changes in these groups, in this order: - -- **Breaking**: a renamed or removed workflow, input, secret name or output; a - raised default tool version or changed default that can turn a green - consumer red; a changed meaning of pass or fail. A release with a Breaking - entry has a new major version. -- **Added**: a new optional input, workflow or capability. -- **Changed**: a change with no visible effect for existing callers. -- **Fixed**: a fix. - -`python tools/changelog.py check` verifies the file, and -`python tools/changelog.py notes ` prints the text of a release page. - -## [Unreleased] - -### Changed - -- Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a - runner for the host's default limit. -- `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the - moving tag `v4`. No change for callers. -- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:` - blocks of this repository's own `sync-milestones.yml` call - `framework/scripts/ci-publish-milestones.sh`. No change for callers of the - shared workflows. - -### Fixed - -- `publish-wiki` sends a page title with spaces instead of hyphens. Gitea keeps a - hyphen in a title literal and stored `Home-da` as `Home-da.-.md`, addressed as - `Home%2Dda`, so links to `Home-da` led to the page list. A new page is now - stored as `Home-da.md` and addressed as `Home-da`. Pages already created with - the old title must be deleted by hand (the publisher does not prune them). - -## [1.1.0] - 2026-10-11 - -### Added - -- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from - which the workflow fetches its script `tools/quality-checks.sh`. - -### Changed - -- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of - embedding about 100 lines of shell; the checks, messages and results are the - same. - -## [1.0.1] - 2026-10-11 - -### Changed - -- Sync GitHub mirror metadata: a repository without a GitHub push mirror still - succeeds, and the log now ends with the line `Skipped`. -- The scoped workflows fetch their scripts at the tag `v1` instead of `main`, - so a merge to `main` reaches consumers only when it is released. - -## [1.0.0] - 2026-10-10 - -First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The -scoped workflows are delivered by the instance from `main` and cannot be pinned. - -### Added - -- Workflow `release.yml`: started by hand with a version, it releases only the - tip of `main` whose validation run is green and whose changelog entry exists, - creates the version tag, the release page and the major tag, and does nothing - but check until `dry_run` is set to `false`. -- Scoped workflow `sync-github-metadata.yml`: copies the description and topics - of a Gitea repository to its GitHub push mirror on every push to `main`, by - hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped - successfully. -- A credential preflight before any change, and failure messages that name - their category (`configuration`, `credentials`, `permissions`, `api`). -- Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki - when a repository has opted in with `wiki/.publish`; the word `prune` in that - file also deletes pages that are not in `wiki/`. -- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck` - and `shfmt` on the tracked Python and shell files, with the inputs - `languages`, `working-directory` and one pinned version per tool. -- Guides for onboarding a repository, workflow credentials, troubleshooting, - reading quality-check results and releasing, and the same pages on the wiki in - English, Danish and Bahasa Malaysia. - -### Changed - -- The scripts of the scoped workflows are files under `tools/`, fetched by a - checkout of this repository, instead of Python embedded in the YAML. -- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`, - `PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret - `CREDENTIALS_FOR_GITHUB`, which is no longer read. +# Changelog + +All notable changes to the shared workflows of this repository. The format and +the version rules are those of the release process (`guides/release-process.md`, +ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH` +and a moving major tag `vMAJOR`. + +Each release lists its changes in these groups, in this order: + +- **Breaking**: a renamed or removed workflow, input, secret name or output; a + raised default tool version or changed default that can turn a green + consumer red; a changed meaning of pass or fail. A release with a Breaking + entry has a new major version. +- **Added**: a new optional input, workflow or capability. +- **Changed**: a change with no visible effect for existing callers. +- **Fixed**: a fix. + +`python tools/changelog.py check` verifies the file, and +`python tools/changelog.py notes ` prints the text of a release page. + +## [Unreleased] + +### Changed + +- Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a + runner for the host's default limit. +- `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the + moving tag `v4`. No change for callers. +- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:` + blocks of this repository's own `sync-milestones.yml` call + `framework/scripts/ci-publish-milestones.sh`. No change for callers of the + shared workflows. + +### Fixed + +- `publish-wiki` sends a page title with spaces instead of hyphens. Gitea keeps a + hyphen in a title literal and stored `Home-da` as `Home-da.-.md`, addressed as + `Home%2Dda`, so links to `Home-da` led to the page list. A new page is now + stored as `Home-da.md` and addressed as `Home-da`. Pages already created with + the old title must be deleted by hand (the publisher does not prune them). +- The onboarding guide and its wiki pages (English, Danish, Bahasa Malaysia) say + that a hyphen in a file name is a space in the page name, and how to link to + it. + +## [1.1.0] - 2026-10-11 + +### Added + +- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from + which the workflow fetches its script `tools/quality-checks.sh`. + +### Changed + +- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of + embedding about 100 lines of shell; the checks, messages and results are the + same. + +## [1.0.1] - 2026-10-11 + +### Changed + +- Sync GitHub mirror metadata: a repository without a GitHub push mirror still + succeeds, and the log now ends with the line `Skipped`. +- The scoped workflows fetch their scripts at the tag `v1` instead of `main`, + so a merge to `main` reaches consumers only when it is released. + +## [1.0.0] - 2026-10-10 + +First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The +scoped workflows are delivered by the instance from `main` and cannot be pinned. + +### Added + +- Workflow `release.yml`: started by hand with a version, it releases only the + tip of `main` whose validation run is green and whose changelog entry exists, + creates the version tag, the release page and the major tag, and does nothing + but check until `dry_run` is set to `false`. +- Scoped workflow `sync-github-metadata.yml`: copies the description and topics + of a Gitea repository to its GitHub push mirror on every push to `main`, by + hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped + successfully. +- A credential preflight before any change, and failure messages that name + their category (`configuration`, `credentials`, `permissions`, `api`). +- Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki + when a repository has opted in with `wiki/.publish`; the word `prune` in that + file also deletes pages that are not in `wiki/`. +- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck` + and `shfmt` on the tracked Python and shell files, with the inputs + `languages`, `working-directory` and one pinned version per tool. +- Guides for onboarding a repository, workflow credentials, troubleshooting, + reading quality-check results and releasing, and the same pages on the wiki in + English, Danish and Bahasa Malaysia. + +### Changed + +- The scripts of the scoped workflows are files under `tools/`, fetched by a + checkout of this repository, instead of Python embedded in the YAML. +- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`, + `PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret + `CREDENTIALS_FOR_GITHUB`, which is no longer read. diff --git a/guides/onboarding.md b/guides/onboarding.md index 571ebd6..101a9c9 100644 --- a/guides/onboarding.md +++ b/guides/onboarding.md @@ -51,7 +51,8 @@ like the sync workflow, but it writes only where a repository opts in. 1. Enable the wiki in the repository settings. 2. Add Markdown pages to `wiki/`. The file name is the page name: `Home.md` is - the home page. + the home page. A hyphen in the name is a space in the page name: `Home-da.md` + is the page "Home da", and a link to it is `[Dansk](Home-da)`. 3. Add the marker `wiki/.publish`. Leave it empty to create and update pages only. Put the single word `prune` in it to also delete wiki pages that are not in `wiki/`, so the wiki always equals the folder. diff --git a/tests/test_publish_wiki.py b/tests/test_publish_wiki.py index 5473167..2d395ba 100644 --- a/tests/test_publish_wiki.py +++ b/tests/test_publish_wiki.py @@ -1,491 +1,491 @@ -"""Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow.""" - -import base64 -import io -import urllib.error - -import pytest -import yaml - -from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module - -API = "https://git.example.test/api/v1" -REPO_URL = f"{API}/repos/TirSystem/repo" -WIKI_URL = f"{REPO_URL}/wiki" - - -@pytest.fixture() -def publish_wiki(): - return load_publish_module() - - -def b64(text: str) -> str: - return base64.b64encode(text.encode()).decode() - - -class TestParseMarker: - @pytest.mark.parametrize( - ("text", "expected"), - [("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)], - ) - def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected): - assert publish_wiki.parse_marker(text) is expected - - @pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"]) - def test_rejects_anything_else(self, publish_wiki, text): - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.parse_marker(text) - assert caught.value.category == "configuration" - assert "prune" in str(caught.value) - - -class TestParseGiteaToken: - def test_returns_the_stripped_token(self, publish_wiki): - assert publish_wiki.parse_gitea_token(" abc \n") == "abc" - - @pytest.mark.parametrize("raw", [None, "", " "]) - def test_rejects_missing_or_empty_token(self, publish_wiki, raw): - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.parse_gitea_token(raw) - assert caught.value.category == "configuration" - assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value) - - -class TestReadPages: - def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path): - (tmp_path / "Home.md").write_bytes(b"home") - (tmp_path / "Home.da.md").write_bytes(b"hjem") - (tmp_path / ".publish").write_bytes(b"prune") - (tmp_path / "Notes.txt").write_bytes(b"ignored") - assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"} - - def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped( - self, publish_wiki, tmp_path - ): - (tmp_path / ".publish").write_text("prune", encoding="utf-8") - with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): - publish_wiki.read_pages(tmp_path) - - -class TestPageTitle: - @pytest.mark.parametrize( - ("key", "title"), [("Home", "Home"), ("Home-da", "Home da"), ("Getting-Started-ms", "Getting Started ms")] - ) - def test_a_hyphen_is_sent_as_a_space(self, publish_wiki, key, title): - # Gitea keeps a hyphen in a title literal (Home-da.-.md, address Home%2Dda). - assert publish_wiki.page_title(key) == title - - -class TestPageKey: - @pytest.mark.parametrize( - ("name", "key"), - [("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"), - ("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")], - ) - def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key): - assert publish_wiki.page_key(name) == key - - def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path): - (tmp_path / "Home da.md").write_bytes(b"hjem") - (tmp_path / "_Sidebar.md").write_bytes(b"menu") - assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"} - - def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path): - (tmp_path / "Home da.md").write_bytes(b"one") - (tmp_path / "Home-da.md").write_bytes(b"two") - with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"): - publish_wiki.read_pages(tmp_path) - - -class FakeGitea: - """Answers like the Gitea repository and wiki endpoints.""" - - def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True): - self.error = publish_wiki.WorkflowError - self.wiki = dict(wiki or {}) - self.has_wiki = has_wiki - self.wiki_initialised = wiki_initialised - self.calls = [] - - def __call__(self, url, method="GET", headers=None, body=None): - self.calls.append((method, url, body, headers)) - if url == REPO_URL: - return {"has_wiki": self.has_wiki} - if url.startswith(f"{WIKI_URL}/pages"): - if not self.wiki_initialised: - raise self.error("configuration", "404") - return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki] - if method == "POST" and url == f"{WIKI_URL}/new": - self.wiki[body["title"].replace(" ", "-")] = body["content_base64"] - return None - title = url.rsplit("/", 1)[-1] - if method == "GET": - return {"title": title, "content_base64": self.wiki[title]} - if method == "PATCH": - assert " " in body["title"] or "-" not in body["title"], body["title"] - self.wiki[title] = body["content_base64"] - return None - if method == "DELETE": - del self.wiki[title] - return None - raise AssertionError(f"unexpected request {method} {url}") - - @property - def writes(self): - return [call for call in self.calls if call[0] != "GET"] - - -@pytest.fixture() -def environment(monkeypatch, tmp_path): - wiki_dir = tmp_path / "wiki" - wiki_dir.mkdir() - (wiki_dir / "Home.md").write_text("home v2", encoding="utf-8") - (wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8") - (wiki_dir / ".publish").write_text("prune\n", encoding="utf-8") - monkeypatch.setenv("WIKI_DIR", str(wiki_dir)) - monkeypatch.setenv("GITEA_API_URL", f"{API}/") - monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") - monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token") - return wiki_dir - - -def run_main(publish_wiki, gitea): - publish_wiki.request_json = gitea - return publish_wiki.main() - - -class TestOptIn: - def test_a_repository_without_the_marker_is_never_written_to( - self, publish_wiki, environment, capsys - ): - (environment / ".publish").unlink() - gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")}) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.calls == [] - assert "wiki publishing is not enabled" in capsys.readouterr().out - assert gitea.wiki == {"Old": b64("hand written")} - - def test_a_repository_without_the_marker_does_not_need_the_secret( - self, publish_wiki, environment, monkeypatch - ): - (environment / ".publish").unlink() - monkeypatch.setenv("WIKI_CREDENTIALS", "") - assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0 - - def test_a_repository_without_a_wiki_directory_is_skipped( - self, publish_wiki, monkeypatch, tmp_path - ): - monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing")) - monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") - gitea = FakeGitea(publish_wiki) - assert run_main(publish_wiki, gitea) == 0 - assert gitea.calls == [] - - def test_an_opted_in_repository_without_the_secret_fails_before_any_request( - self, publish_wiki, environment, monkeypatch - ): - monkeypatch.setenv("WIKI_CREDENTIALS", "") - gitea = FakeGitea(publish_wiki) - - with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"): - run_main(publish_wiki, gitea) - - assert gitea.calls == [] - - def test_an_unknown_marker_content_is_refused_before_any_request( - self, publish_wiki, environment - ): - (environment / ".publish").write_text("mirror everything", encoding="utf-8") - gitea = FakeGitea(publish_wiki) - - with pytest.raises(publish_wiki.WorkflowError, match="prune"): - run_main(publish_wiki, gitea) - - assert gitea.calls == [] - - -class TestPublish: - def test_prune_creates_updates_and_deletes_to_match_the_source( - self, publish_wiki, environment, capsys - ): - gitea = FakeGitea( - publish_wiki, - wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")}, - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")} - methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls] - assert ("PATCH", "Home") in methods - assert ("DELETE", "Old") in methods - assert ("PATCH", "Credentials") not in methods - assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out - - def test_without_prune_pages_outside_the_source_are_kept( - self, publish_wiki, environment, capsys - ): - (environment / ".publish").write_text("", encoding="utf-8") - gitea = FakeGitea( - publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")} - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.wiki["Old"] == b64("hand written") - assert gitea.wiki["Home"] == b64("home v2") - assert "Credentials" in gitea.wiki - assert not [call for call in gitea.calls if call[0] == "DELETE"] - out = capsys.readouterr().out - assert "1 created, 1 updated, 0 deleted" in out - assert "1 page(s) are in the wiki but not in" in out - assert "prune" in out - - def test_creates_every_page_when_the_wiki_was_never_written( - self, publish_wiki, environment - ): - gitea = FakeGitea(publish_wiki, wiki_initialised=False) - - assert run_main(publish_wiki, gitea) == 0 - - assert set(gitea.wiki) == {"Home", "Credentials"} - assert [call[0] for call in gitea.calls].count("POST") == 2 - - def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment): - gitea = FakeGitea( - publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")} - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.writes == [] - - def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys): - gitea = FakeGitea(publish_wiki) - - assert run_main(publish_wiki, gitea) == 0 - - assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"} - output = capsys.readouterr() - assert "wiki-token" not in output.out + output.err - - def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing( - self, publish_wiki, environment - ): - gitea = FakeGitea(publish_wiki, has_wiki=False) - - with pytest.raises(publish_wiki.WorkflowError) as caught: - run_main(publish_wiki, gitea) - - assert caught.value.category == "configuration" - assert gitea.writes == [] - - def test_an_empty_source_never_deletes_pages_even_with_prune( - self, publish_wiki, environment - ): - for path in environment.glob("*.md"): - path.unlink() - gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")}) - - with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): - run_main(publish_wiki, gitea) - - assert gitea.calls == [] - assert gitea.wiki == {"Home": b64("keep me")} - - -class TestPageIdentity: - """A wiki that already has pages named with spaces, as Gitea stores them.""" - - def test_a_file_named_with_a_space_updates_the_existing_page( - self, publish_wiki, environment - ): - (environment / "Home.md").unlink() - (environment / "Credentials.md").unlink() - (environment / ".publish").write_text("", encoding="utf-8") - (environment / "Home da.md").write_text("hjem v2", encoding="utf-8") - gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")}) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.wiki == {"Home-da": b64("hjem v2")} - assert [call[0] for call in gitea.calls].count("POST") == 0 - assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls] - - def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment): - (environment / "Home.md").unlink() - (environment / "Credentials.md").unlink() - (environment / "Home da.md").write_text("hjem", encoding="utf-8") - (environment / "_Sidebar.md").write_text("menu", encoding="utf-8") - gitea = FakeGitea( - publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")} - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.writes == [] - - def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling( - self, publish_wiki, environment - ): - (environment / "Home.md").unlink() - (environment / "Credentials.md").unlink() - (environment / "Install ms.md").write_text("pasang", encoding="utf-8") - gitea = FakeGitea( - publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")} - ) - - assert run_main(publish_wiki, gitea) == 0 - - assert gitea.wiki == {"Install-ms": b64("pasang")} - - def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki): - def listing(url, headers=None, **kwargs): - return [ - {"title": "Whatever", "sub_url": "Real-Address"}, - {"title": "Only title"}, - {"title": "Encoded", "sub_url": "Caf%C3%A9"}, - ] - - keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing) - - assert keys == {"Real-Address", "Only-title", "Café"} - - -class TestDescribeToken: - def test_recognises_a_gitea_token_shape(self, publish_wiki): - description = publish_wiki.describe_token("a1b2c3d4e5" * 4) - assert "40 characters" in description - assert "looks like a Gitea token" in description - - @pytest.mark.parametrize( - ("token", "expected"), - [ - ("*******", "only mask characters"), - ("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"), - ('"' + "a" * 40 + '"', "quote character"), - ("abc def", "whitespace"), - ], - ) - def test_names_the_likely_mistake_without_echoing_the_token( - self, publish_wiki, token, expected - ): - description = publish_wiki.describe_token(token) - assert expected in description - assert token not in description - - -class TestRejectedToken: - def test_rejection_adds_the_token_shape_but_not_the_token( - self, publish_wiki, environment, monkeypatch - ): - monkeypatch.setenv("WIKI_CREDENTIALS", "*******") - - def reject(url, method="GET", headers=None, body=None): - raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.") - - publish_wiki.request_json = reject - - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.main() - - message = str(caught.value) - assert caught.value.category == "credentials" - assert "Token shape: 7 characters" in message - assert "only mask characters" in message - assert "*******" not in message.split("Token shape:")[0] - - def test_other_categories_are_not_changed(self, publish_wiki, environment): - def forbid(url, method="GET", headers=None, body=None): - raise publish_wiki.WorkflowError("permissions", "Forbidden.") - - publish_wiki.request_json = forbid - - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.main() - - assert str(caught.value) == "[permissions] Forbidden." - - -class TestRequestJson: - @staticmethod - def raise_http_error(publish_wiki, monkeypatch, code, reason): - def fake_urlopen(request, timeout): - raise urllib.error.HTTPError( - request.full_url, code, reason, {}, io.BytesIO(b"secret detail") - ) - - monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) - - @pytest.mark.parametrize( - ("code", "reason", "category"), - [ - (401, "Unauthorized", "credentials"), - (403, "Forbidden", "permissions"), - (404, "Not Found", "configuration"), - (500, "Server Error", "api"), - ], - ) - def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category): - self.raise_http_error(publish_wiki, monkeypatch, code, reason) - - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.request_json(REPO_URL) - - assert caught.value.category == category - assert f"HTTP {code}" in str(caught.value) - assert "secret detail" not in str(caught.value) - - def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch): - def fake_urlopen(request, timeout): - raise urllib.error.URLError("no route") - - monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) - - with pytest.raises(publish_wiki.WorkflowError) as caught: - publish_wiki.request_json(REPO_URL) - - assert caught.value.category == "api" - - -class TestWorkflow: - @pytest.fixture() - def document(self): - return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8")) - - def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document): - triggers = document[True] # PyYAML reads the key `on` as True - assert triggers["push"]["branches"] == ["main"] - assert triggers["push"]["paths"] == ["wiki/**"] - assert "workflow_dispatch" in triggers - - def test_job_has_read_only_permissions(self, document): - assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"} - - def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document): - text = PUBLISH_WORKFLOW.read_text(encoding="utf-8") - assert "secrets.TOKEN_FOR_GITEA }}" not in text - publish_step = document["jobs"]["publish-wiki"]["steps"][-1] - assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}" - assert "secrets" not in publish_step["run"] - - def test_the_workflow_has_one_source_and_runs_the_script_file(self, document): - assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists() - assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file() - steps = document["jobs"]["publish-wiki"]["steps"] - assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8") - assert steps[0]["uses"].startswith("actions/checkout@") - assert steps[1]["with"] == { - "repository": "TirSystem/github-action", - "ref": "v1", - "path": "shared-scripts", - } - assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py" - - -def test_this_repository_opts_in_and_prunes(): - """The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004).""" - marker = REPO_ROOT / "wiki" / ".publish" - assert marker.read_text(encoding="utf-8").strip() == "prune" +"""Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow.""" + +import base64 +import io +import urllib.error + +import pytest +import yaml + +from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module + +API = "https://git.example.test/api/v1" +REPO_URL = f"{API}/repos/TirSystem/repo" +WIKI_URL = f"{REPO_URL}/wiki" + + +@pytest.fixture() +def publish_wiki(): + return load_publish_module() + + +def b64(text: str) -> str: + return base64.b64encode(text.encode()).decode() + + +class TestParseMarker: + @pytest.mark.parametrize( + ("text", "expected"), + [("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)], + ) + def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected): + assert publish_wiki.parse_marker(text) is expected + + @pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"]) + def test_rejects_anything_else(self, publish_wiki, text): + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.parse_marker(text) + assert caught.value.category == "configuration" + assert "prune" in str(caught.value) + + +class TestParseGiteaToken: + def test_returns_the_stripped_token(self, publish_wiki): + assert publish_wiki.parse_gitea_token(" abc \n") == "abc" + + @pytest.mark.parametrize("raw", [None, "", " "]) + def test_rejects_missing_or_empty_token(self, publish_wiki, raw): + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.parse_gitea_token(raw) + assert caught.value.category == "configuration" + assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value) + + +class TestReadPages: + def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path): + (tmp_path / "Home.md").write_bytes(b"home") + (tmp_path / "Home.da.md").write_bytes(b"hjem") + (tmp_path / ".publish").write_bytes(b"prune") + (tmp_path / "Notes.txt").write_bytes(b"ignored") + assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"} + + def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped( + self, publish_wiki, tmp_path + ): + (tmp_path / ".publish").write_text("prune", encoding="utf-8") + with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): + publish_wiki.read_pages(tmp_path) + + +class TestPageTitle: + @pytest.mark.parametrize( + ("key", "title"), [("Home", "Home"), ("Home-da", "Home da"), ("Getting-Started-ms", "Getting Started ms")] + ) + def test_a_hyphen_is_sent_as_a_space(self, publish_wiki, key, title): + # Gitea keeps a hyphen in a title literal (Home-da.-.md, address Home%2Dda). + assert publish_wiki.page_title(key) == title + + +class TestPageKey: + @pytest.mark.parametrize( + ("name", "key"), + [("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"), + ("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")], + ) + def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key): + assert publish_wiki.page_key(name) == key + + def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path): + (tmp_path / "Home da.md").write_bytes(b"hjem") + (tmp_path / "_Sidebar.md").write_bytes(b"menu") + assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"} + + def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path): + (tmp_path / "Home da.md").write_bytes(b"one") + (tmp_path / "Home-da.md").write_bytes(b"two") + with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"): + publish_wiki.read_pages(tmp_path) + + +class FakeGitea: + """Answers like the Gitea repository and wiki endpoints.""" + + def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True): + self.error = publish_wiki.WorkflowError + self.wiki = dict(wiki or {}) + self.has_wiki = has_wiki + self.wiki_initialised = wiki_initialised + self.calls = [] + + def __call__(self, url, method="GET", headers=None, body=None): + self.calls.append((method, url, body, headers)) + if url == REPO_URL: + return {"has_wiki": self.has_wiki} + if url.startswith(f"{WIKI_URL}/pages"): + if not self.wiki_initialised: + raise self.error("configuration", "404") + return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki] + if method == "POST" and url == f"{WIKI_URL}/new": + self.wiki[body["title"].replace(" ", "-")] = body["content_base64"] + return None + title = url.rsplit("/", 1)[-1] + if method == "GET": + return {"title": title, "content_base64": self.wiki[title]} + if method == "PATCH": + assert " " in body["title"] or "-" not in body["title"], body["title"] + self.wiki[title] = body["content_base64"] + return None + if method == "DELETE": + del self.wiki[title] + return None + raise AssertionError(f"unexpected request {method} {url}") + + @property + def writes(self): + return [call for call in self.calls if call[0] != "GET"] + + +@pytest.fixture() +def environment(monkeypatch, tmp_path): + wiki_dir = tmp_path / "wiki" + wiki_dir.mkdir() + (wiki_dir / "Home.md").write_text("home v2", encoding="utf-8") + (wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8") + (wiki_dir / ".publish").write_text("prune\n", encoding="utf-8") + monkeypatch.setenv("WIKI_DIR", str(wiki_dir)) + monkeypatch.setenv("GITEA_API_URL", f"{API}/") + monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") + monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token") + return wiki_dir + + +def run_main(publish_wiki, gitea): + publish_wiki.request_json = gitea + return publish_wiki.main() + + +class TestOptIn: + def test_a_repository_without_the_marker_is_never_written_to( + self, publish_wiki, environment, capsys + ): + (environment / ".publish").unlink() + gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")}) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.calls == [] + assert "wiki publishing is not enabled" in capsys.readouterr().out + assert gitea.wiki == {"Old": b64("hand written")} + + def test_a_repository_without_the_marker_does_not_need_the_secret( + self, publish_wiki, environment, monkeypatch + ): + (environment / ".publish").unlink() + monkeypatch.setenv("WIKI_CREDENTIALS", "") + assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0 + + def test_a_repository_without_a_wiki_directory_is_skipped( + self, publish_wiki, monkeypatch, tmp_path + ): + monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing")) + monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") + gitea = FakeGitea(publish_wiki) + assert run_main(publish_wiki, gitea) == 0 + assert gitea.calls == [] + + def test_an_opted_in_repository_without_the_secret_fails_before_any_request( + self, publish_wiki, environment, monkeypatch + ): + monkeypatch.setenv("WIKI_CREDENTIALS", "") + gitea = FakeGitea(publish_wiki) + + with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"): + run_main(publish_wiki, gitea) + + assert gitea.calls == [] + + def test_an_unknown_marker_content_is_refused_before_any_request( + self, publish_wiki, environment + ): + (environment / ".publish").write_text("mirror everything", encoding="utf-8") + gitea = FakeGitea(publish_wiki) + + with pytest.raises(publish_wiki.WorkflowError, match="prune"): + run_main(publish_wiki, gitea) + + assert gitea.calls == [] + + +class TestPublish: + def test_prune_creates_updates_and_deletes_to_match_the_source( + self, publish_wiki, environment, capsys + ): + gitea = FakeGitea( + publish_wiki, + wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")}, + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")} + methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls] + assert ("PATCH", "Home") in methods + assert ("DELETE", "Old") in methods + assert ("PATCH", "Credentials") not in methods + assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out + + def test_without_prune_pages_outside_the_source_are_kept( + self, publish_wiki, environment, capsys + ): + (environment / ".publish").write_text("", encoding="utf-8") + gitea = FakeGitea( + publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")} + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.wiki["Old"] == b64("hand written") + assert gitea.wiki["Home"] == b64("home v2") + assert "Credentials" in gitea.wiki + assert not [call for call in gitea.calls if call[0] == "DELETE"] + out = capsys.readouterr().out + assert "1 created, 1 updated, 0 deleted" in out + assert "1 page(s) are in the wiki but not in" in out + assert "prune" in out + + def test_creates_every_page_when_the_wiki_was_never_written( + self, publish_wiki, environment + ): + gitea = FakeGitea(publish_wiki, wiki_initialised=False) + + assert run_main(publish_wiki, gitea) == 0 + + assert set(gitea.wiki) == {"Home", "Credentials"} + assert [call[0] for call in gitea.calls].count("POST") == 2 + + def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment): + gitea = FakeGitea( + publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")} + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.writes == [] + + def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys): + gitea = FakeGitea(publish_wiki) + + assert run_main(publish_wiki, gitea) == 0 + + assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"} + output = capsys.readouterr() + assert "wiki-token" not in output.out + output.err + + def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing( + self, publish_wiki, environment + ): + gitea = FakeGitea(publish_wiki, has_wiki=False) + + with pytest.raises(publish_wiki.WorkflowError) as caught: + run_main(publish_wiki, gitea) + + assert caught.value.category == "configuration" + assert gitea.writes == [] + + def test_an_empty_source_never_deletes_pages_even_with_prune( + self, publish_wiki, environment + ): + for path in environment.glob("*.md"): + path.unlink() + gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")}) + + with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): + run_main(publish_wiki, gitea) + + assert gitea.calls == [] + assert gitea.wiki == {"Home": b64("keep me")} + + +class TestPageIdentity: + """A wiki that already has pages named with spaces, as Gitea stores them.""" + + def test_a_file_named_with_a_space_updates_the_existing_page( + self, publish_wiki, environment + ): + (environment / "Home.md").unlink() + (environment / "Credentials.md").unlink() + (environment / ".publish").write_text("", encoding="utf-8") + (environment / "Home da.md").write_text("hjem v2", encoding="utf-8") + gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")}) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.wiki == {"Home-da": b64("hjem v2")} + assert [call[0] for call in gitea.calls].count("POST") == 0 + assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls] + + def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment): + (environment / "Home.md").unlink() + (environment / "Credentials.md").unlink() + (environment / "Home da.md").write_text("hjem", encoding="utf-8") + (environment / "_Sidebar.md").write_text("menu", encoding="utf-8") + gitea = FakeGitea( + publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")} + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.writes == [] + + def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling( + self, publish_wiki, environment + ): + (environment / "Home.md").unlink() + (environment / "Credentials.md").unlink() + (environment / "Install ms.md").write_text("pasang", encoding="utf-8") + gitea = FakeGitea( + publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")} + ) + + assert run_main(publish_wiki, gitea) == 0 + + assert gitea.wiki == {"Install-ms": b64("pasang")} + + def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki): + def listing(url, headers=None, **kwargs): + return [ + {"title": "Whatever", "sub_url": "Real-Address"}, + {"title": "Only title"}, + {"title": "Encoded", "sub_url": "Caf%C3%A9"}, + ] + + keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing) + + assert keys == {"Real-Address", "Only-title", "Café"} + + +class TestDescribeToken: + def test_recognises_a_gitea_token_shape(self, publish_wiki): + description = publish_wiki.describe_token("a1b2c3d4e5" * 4) + assert "40 characters" in description + assert "looks like a Gitea token" in description + + @pytest.mark.parametrize( + ("token", "expected"), + [ + ("*******", "only mask characters"), + ("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"), + ('"' + "a" * 40 + '"', "quote character"), + ("abc def", "whitespace"), + ], + ) + def test_names_the_likely_mistake_without_echoing_the_token( + self, publish_wiki, token, expected + ): + description = publish_wiki.describe_token(token) + assert expected in description + assert token not in description + + +class TestRejectedToken: + def test_rejection_adds_the_token_shape_but_not_the_token( + self, publish_wiki, environment, monkeypatch + ): + monkeypatch.setenv("WIKI_CREDENTIALS", "*******") + + def reject(url, method="GET", headers=None, body=None): + raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.") + + publish_wiki.request_json = reject + + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.main() + + message = str(caught.value) + assert caught.value.category == "credentials" + assert "Token shape: 7 characters" in message + assert "only mask characters" in message + assert "*******" not in message.split("Token shape:")[0] + + def test_other_categories_are_not_changed(self, publish_wiki, environment): + def forbid(url, method="GET", headers=None, body=None): + raise publish_wiki.WorkflowError("permissions", "Forbidden.") + + publish_wiki.request_json = forbid + + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.main() + + assert str(caught.value) == "[permissions] Forbidden." + + +class TestRequestJson: + @staticmethod + def raise_http_error(publish_wiki, monkeypatch, code, reason): + def fake_urlopen(request, timeout): + raise urllib.error.HTTPError( + request.full_url, code, reason, {}, io.BytesIO(b"secret detail") + ) + + monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) + + @pytest.mark.parametrize( + ("code", "reason", "category"), + [ + (401, "Unauthorized", "credentials"), + (403, "Forbidden", "permissions"), + (404, "Not Found", "configuration"), + (500, "Server Error", "api"), + ], + ) + def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category): + self.raise_http_error(publish_wiki, monkeypatch, code, reason) + + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.request_json(REPO_URL) + + assert caught.value.category == category + assert f"HTTP {code}" in str(caught.value) + assert "secret detail" not in str(caught.value) + + def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch): + def fake_urlopen(request, timeout): + raise urllib.error.URLError("no route") + + monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) + + with pytest.raises(publish_wiki.WorkflowError) as caught: + publish_wiki.request_json(REPO_URL) + + assert caught.value.category == "api" + + +class TestWorkflow: + @pytest.fixture() + def document(self): + return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8")) + + def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document): + triggers = document[True] # PyYAML reads the key `on` as True + assert triggers["push"]["branches"] == ["main"] + assert triggers["push"]["paths"] == ["wiki/**"] + assert "workflow_dispatch" in triggers + + def test_job_has_read_only_permissions(self, document): + assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"} + + def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document): + text = PUBLISH_WORKFLOW.read_text(encoding="utf-8") + assert "secrets.TOKEN_FOR_GITEA }}" not in text + publish_step = document["jobs"]["publish-wiki"]["steps"][-1] + assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}" + assert "secrets" not in publish_step["run"] + + def test_the_workflow_has_one_source_and_runs_the_script_file(self, document): + assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists() + assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file() + steps = document["jobs"]["publish-wiki"]["steps"] + assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8") + assert steps[0]["uses"].startswith("actions/checkout@") + assert steps[1]["with"] == { + "repository": "TirSystem/github-action", + "ref": "v1", + "path": "shared-scripts", + } + assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py" + + +def test_this_repository_opts_in_and_prunes(): + """The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004).""" + marker = REPO_ROOT / "wiki" / ".publish" + assert marker.read_text(encoding="utf-8").strip() == "prune" diff --git a/tools/publish_wiki.py b/tools/publish_wiki.py index 3e8d966..f6fb657 100644 --- a/tools/publish_wiki.py +++ b/tools/publish_wiki.py @@ -1,306 +1,306 @@ -"""Publish the pages in wiki/ to the Gitea wiki of the repository. - -Run by the scoped workflow publish-wiki.yml, which checks out the repository -that owns the wiki and this repository, and runs this file (MIL-004 task 6). -A repository publishes only when it has opted in with wiki/.publish -(ADR-0004); the word prune in that file lets the run delete wiki pages that -are not in wiki/. It reads its settings from the environment: - - GITEA_API_URL Gitea API base, for example https://host/api/v1 - SOURCE_REPOSITORY owner/repo of the repository whose wiki is written - WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI) - WIKI_DIR directory with the pages (default: wiki) - -Every failure is printed as ``ERROR: [category] message`` with the categories -configuration, credentials, permissions and api; no secret value is printed. -""" - -import base64 -import json -import os -import re -import sys -import urllib.error -import urllib.parse -import urllib.request -from pathlib import Path - -PAGE_LIMIT = 50 -REQUEST_TIMEOUT_SECONDS = 30 -COMMIT_MESSAGE = "Publish wiki from the repository" -MARKER_NAME = ".publish" -PRUNE_WORD = "prune" - - -class WorkflowError(RuntimeError): - """A failure with a category. - - configuration: a secret, a setting or the wiki source is missing or malformed. - credentials: Gitea rejected the token (HTTP 401). - permissions: the token is valid but not allowed (HTTP 403). - api: any other API or network failure. - """ - - def __init__(self, category, message): - super().__init__(f"[{category}] {message}") - self.category = category - self.detail = message - - -def request_json(url, method="GET", headers=None, body=None): - request = urllib.request.Request( - url, - data=json.dumps(body).encode("utf-8") if body is not None else None, - headers=headers or {}, - method=method, - ) - try: - with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response: - content = response.read() - return json.loads(content) if content else None - except urllib.error.HTTPError as error: - detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})" - if error.code == 401: - raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None - if error.code == 403: - raise WorkflowError( - "permissions", f"{detail}. The token lacks a required permission." - ) from None - if error.code == 404: - raise WorkflowError( - "configuration", - f"{detail}. The repository or page was not found, or the token " - "cannot see it.", - ) from None - raise WorkflowError("api", detail) from None - except (urllib.error.URLError, TimeoutError) as error: - raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None - - -def parse_marker(text): - """Return True when the marker asks for pruning, False when it is empty.""" - word = (text or "").strip() - if not word: - return False - if word == PRUNE_WORD: - return True - raise WorkflowError( - "configuration", - f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.", - ) - - -def parse_gitea_token(raw_token): - token = (raw_token or "").strip() - if not token: - raise WorkflowError( - "configuration", - "TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of " - "the organization of this repository.", - ) - return token - - -def describe_token(token): - """Describe the shape of a token without revealing any of its characters.""" - facts = [f"{len(token)} characters"] - if re.fullmatch(r"[0-9a-f]{40}", token): - facts.append("looks like a Gitea token (40 hexadecimal characters)") - return ", ".join(facts) - facts.append("a Gitea token has 40 hexadecimal characters") - if set(token) <= {"*", "x", "X", "•"}: - facts.append("only mask characters: the masked text was stored") - if "=" in token: - facts.append("contains '=': the secret name may be part of the value") - if re.search(r"[\"'`]", token): - facts.append("contains a quote character") - if re.search(r"\s", token): - facts.append("contains whitespace") - return ", ".join(facts) - - -def split_repository(full_name): - owner, separator, repo = (full_name or "").partition("/") - if not separator or not owner or not repo: - raise WorkflowError("configuration", "Could not determine the Gitea repository.") - return owner, repo - - -def page_key(title): - """Return the identity of a page: Gitea stores a space as a hyphen. - - The page "Home da" is the file Home-da.md and its address is Home-da, so - both spellings must name the same page. - """ - return str(title).replace(" ", "-") - - -def page_title(key): - """Return the title to send to Gitea for a page key. - - A hyphen in a title is a literal hyphen to Gitea, which then stores the page - as Home-da.-.md and addresses it as Home%2Dda, so a link to Home-da finds - nothing. A space is stored as a hyphen (Home-da.md) and addressed as Home-da. - """ - return key.replace("-", " ") - - -def read_pages(wiki_dir): - """Return {page key: content} for every Markdown file in wiki_dir.""" - pages = {} - for path in sorted(wiki_dir.glob("*.md")): - key = page_key(path.stem) - if key in pages: - raise WorkflowError( - "configuration", - f"Two files in {wiki_dir} name the wiki page {key}; a space and " - "a hyphen are the same in a page name.", - ) - pages[key] = path.read_bytes() - if not pages: - # Publishing an empty directory would delete the whole wiki. - raise WorkflowError( - "configuration", f"The wiki source directory {wiki_dir} has no Markdown pages." - ) - return pages - - -def list_wiki_titles(wiki_url, headers, send): - """Return the keys of the pages that exist in the wiki. - - The key is the address Gitea reports (sub_url), or the title with its - spaces replaced by hyphens. - - Gitea answers 404 for a wiki that has never been written to; that is an - empty wiki, not a failure. - """ - titles = set() - page = 1 - while True: - url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}" - try: - batch = send(url, headers=headers) - except WorkflowError as error: - if error.category == "configuration" and page == 1: - return titles - raise - if not isinstance(batch, list): - raise WorkflowError("api", "Gitea returned an invalid wiki page list.") - titles.update( - urllib.parse.unquote(str(item.get("sub_url") or "")) - or page_key(item.get("title", "")) - for item in batch - ) - if len(batch) < PAGE_LIMIT: - return titles - page += 1 - - -def read_wiki_page(wiki_url, title, headers, send): - quoted = urllib.parse.quote(title, safe="") - page = send(f"{wiki_url}/page/{quoted}", headers=headers) - return base64.b64decode((page or {}).get("content_base64", "")) - - -def publish(pages, wiki_url, headers, send, prune): - """Create and update the pages; delete the others only when prune is set. - - Returns the titles created, updated and deleted, and the titles that were - left in the wiki because pruning is off. - """ - existing = list_wiki_titles(wiki_url, headers, send) - current = { - title: read_wiki_page(wiki_url, title, headers, send) - for title in sorted(existing & pages.keys()) - } - - result = {"created": [], "updated": [], "deleted": [], "kept": []} - for title, content in pages.items(): - body = { - "title": page_title(title), - "content_base64": base64.b64encode(content).decode("ascii"), - "message": COMMIT_MESSAGE, - } - if title not in existing: - send(f"{wiki_url}/new", method="POST", headers=headers, body=body) - result["created"].append(title) - elif current[title] != content: - quoted = urllib.parse.quote(title, safe="") - send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body) - result["updated"].append(title) - - for title in sorted(existing - pages.keys()): - if not prune: - result["kept"].append(title) - continue - quoted = urllib.parse.quote(title, safe="") - send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers) - result["deleted"].append(title) - return result - - -def main(): - wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki")) - owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY")) - - marker = wiki_dir / MARKER_NAME - if not marker.is_file(): - print( - f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; " - "wiki publishing is not enabled for this repository." - ) - return 0 - prune = parse_marker(marker.read_text(encoding="utf-8")) - - pages = read_pages(wiki_dir) - token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS")) - - api_url = os.environ["GITEA_API_URL"].rstrip("/") - repository_url = ( - f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/" - f"{urllib.parse.quote(repo, safe='')}" - ) - headers = { - "Authorization": f"token {token}", - "Accept": "application/json", - "Content-Type": "application/json", - } - - try: - repository = request_json(repository_url, headers=headers) - if isinstance(repository, dict) and repository.get("has_wiki") is False: - raise WorkflowError( - "configuration", - f"The wiki is not enabled on {owner}/{repo}. Enable it in the " - "repository settings.", - ) - print("Preflight passed: Gitea accepted the token and the wiki is enabled.") - - result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune) - except WorkflowError as error: - if error.category == "credentials": - raise WorkflowError( - "credentials", f"{error.detail} Token shape: {describe_token(token)}." - ) from None - raise - - print( - f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: " - f"{len(result['created'])} created, {len(result['updated'])} updated, " - f"{len(result['deleted'])} deleted." - ) - if result["kept"]: - print( - f"{len(result['kept'])} page(s) are in the wiki but not in " - f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in " - f"{wiki_dir}/{MARKER_NAME} to delete them." - ) - return 0 - - -if __name__ == "__main__": - try: - sys.exit(main()) - except WorkflowError as error: - print(f"ERROR: {error}", file=sys.stderr) - sys.exit(1) +"""Publish the pages in wiki/ to the Gitea wiki of the repository. + +Run by the scoped workflow publish-wiki.yml, which checks out the repository +that owns the wiki and this repository, and runs this file (MIL-004 task 6). +A repository publishes only when it has opted in with wiki/.publish +(ADR-0004); the word prune in that file lets the run delete wiki pages that +are not in wiki/. It reads its settings from the environment: + + GITEA_API_URL Gitea API base, for example https://host/api/v1 + SOURCE_REPOSITORY owner/repo of the repository whose wiki is written + WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI) + WIKI_DIR directory with the pages (default: wiki) + +Every failure is printed as ``ERROR: [category] message`` with the categories +configuration, credentials, permissions and api; no secret value is printed. +""" + +import base64 +import json +import os +import re +import sys +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path + +PAGE_LIMIT = 50 +REQUEST_TIMEOUT_SECONDS = 30 +COMMIT_MESSAGE = "Publish wiki from the repository" +MARKER_NAME = ".publish" +PRUNE_WORD = "prune" + + +class WorkflowError(RuntimeError): + """A failure with a category. + + configuration: a secret, a setting or the wiki source is missing or malformed. + credentials: Gitea rejected the token (HTTP 401). + permissions: the token is valid but not allowed (HTTP 403). + api: any other API or network failure. + """ + + def __init__(self, category, message): + super().__init__(f"[{category}] {message}") + self.category = category + self.detail = message + + +def request_json(url, method="GET", headers=None, body=None): + request = urllib.request.Request( + url, + data=json.dumps(body).encode("utf-8") if body is not None else None, + headers=headers or {}, + method=method, + ) + try: + with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response: + content = response.read() + return json.loads(content) if content else None + except urllib.error.HTTPError as error: + detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})" + if error.code == 401: + raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None + if error.code == 403: + raise WorkflowError( + "permissions", f"{detail}. The token lacks a required permission." + ) from None + if error.code == 404: + raise WorkflowError( + "configuration", + f"{detail}. The repository or page was not found, or the token " + "cannot see it.", + ) from None + raise WorkflowError("api", detail) from None + except (urllib.error.URLError, TimeoutError) as error: + raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None + + +def parse_marker(text): + """Return True when the marker asks for pruning, False when it is empty.""" + word = (text or "").strip() + if not word: + return False + if word == PRUNE_WORD: + return True + raise WorkflowError( + "configuration", + f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.", + ) + + +def parse_gitea_token(raw_token): + token = (raw_token or "").strip() + if not token: + raise WorkflowError( + "configuration", + "TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of " + "the organization of this repository.", + ) + return token + + +def describe_token(token): + """Describe the shape of a token without revealing any of its characters.""" + facts = [f"{len(token)} characters"] + if re.fullmatch(r"[0-9a-f]{40}", token): + facts.append("looks like a Gitea token (40 hexadecimal characters)") + return ", ".join(facts) + facts.append("a Gitea token has 40 hexadecimal characters") + if set(token) <= {"*", "x", "X", "•"}: + facts.append("only mask characters: the masked text was stored") + if "=" in token: + facts.append("contains '=': the secret name may be part of the value") + if re.search(r"[\"'`]", token): + facts.append("contains a quote character") + if re.search(r"\s", token): + facts.append("contains whitespace") + return ", ".join(facts) + + +def split_repository(full_name): + owner, separator, repo = (full_name or "").partition("/") + if not separator or not owner or not repo: + raise WorkflowError("configuration", "Could not determine the Gitea repository.") + return owner, repo + + +def page_key(title): + """Return the identity of a page: Gitea stores a space as a hyphen. + + The page "Home da" is the file Home-da.md and its address is Home-da, so + both spellings must name the same page. + """ + return str(title).replace(" ", "-") + + +def page_title(key): + """Return the title to send to Gitea for a page key. + + A hyphen in a title is a literal hyphen to Gitea, which then stores the page + as Home-da.-.md and addresses it as Home%2Dda, so a link to Home-da finds + nothing. A space is stored as a hyphen (Home-da.md) and addressed as Home-da. + """ + return key.replace("-", " ") + + +def read_pages(wiki_dir): + """Return {page key: content} for every Markdown file in wiki_dir.""" + pages = {} + for path in sorted(wiki_dir.glob("*.md")): + key = page_key(path.stem) + if key in pages: + raise WorkflowError( + "configuration", + f"Two files in {wiki_dir} name the wiki page {key}; a space and " + "a hyphen are the same in a page name.", + ) + pages[key] = path.read_bytes() + if not pages: + # Publishing an empty directory would delete the whole wiki. + raise WorkflowError( + "configuration", f"The wiki source directory {wiki_dir} has no Markdown pages." + ) + return pages + + +def list_wiki_titles(wiki_url, headers, send): + """Return the keys of the pages that exist in the wiki. + + The key is the address Gitea reports (sub_url), or the title with its + spaces replaced by hyphens. + + Gitea answers 404 for a wiki that has never been written to; that is an + empty wiki, not a failure. + """ + titles = set() + page = 1 + while True: + url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}" + try: + batch = send(url, headers=headers) + except WorkflowError as error: + if error.category == "configuration" and page == 1: + return titles + raise + if not isinstance(batch, list): + raise WorkflowError("api", "Gitea returned an invalid wiki page list.") + titles.update( + urllib.parse.unquote(str(item.get("sub_url") or "")) + or page_key(item.get("title", "")) + for item in batch + ) + if len(batch) < PAGE_LIMIT: + return titles + page += 1 + + +def read_wiki_page(wiki_url, title, headers, send): + quoted = urllib.parse.quote(title, safe="") + page = send(f"{wiki_url}/page/{quoted}", headers=headers) + return base64.b64decode((page or {}).get("content_base64", "")) + + +def publish(pages, wiki_url, headers, send, prune): + """Create and update the pages; delete the others only when prune is set. + + Returns the titles created, updated and deleted, and the titles that were + left in the wiki because pruning is off. + """ + existing = list_wiki_titles(wiki_url, headers, send) + current = { + title: read_wiki_page(wiki_url, title, headers, send) + for title in sorted(existing & pages.keys()) + } + + result = {"created": [], "updated": [], "deleted": [], "kept": []} + for title, content in pages.items(): + body = { + "title": page_title(title), + "content_base64": base64.b64encode(content).decode("ascii"), + "message": COMMIT_MESSAGE, + } + if title not in existing: + send(f"{wiki_url}/new", method="POST", headers=headers, body=body) + result["created"].append(title) + elif current[title] != content: + quoted = urllib.parse.quote(title, safe="") + send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body) + result["updated"].append(title) + + for title in sorted(existing - pages.keys()): + if not prune: + result["kept"].append(title) + continue + quoted = urllib.parse.quote(title, safe="") + send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers) + result["deleted"].append(title) + return result + + +def main(): + wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki")) + owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY")) + + marker = wiki_dir / MARKER_NAME + if not marker.is_file(): + print( + f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; " + "wiki publishing is not enabled for this repository." + ) + return 0 + prune = parse_marker(marker.read_text(encoding="utf-8")) + + pages = read_pages(wiki_dir) + token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS")) + + api_url = os.environ["GITEA_API_URL"].rstrip("/") + repository_url = ( + f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/" + f"{urllib.parse.quote(repo, safe='')}" + ) + headers = { + "Authorization": f"token {token}", + "Accept": "application/json", + "Content-Type": "application/json", + } + + try: + repository = request_json(repository_url, headers=headers) + if isinstance(repository, dict) and repository.get("has_wiki") is False: + raise WorkflowError( + "configuration", + f"The wiki is not enabled on {owner}/{repo}. Enable it in the " + "repository settings.", + ) + print("Preflight passed: Gitea accepted the token and the wiki is enabled.") + + result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune) + except WorkflowError as error: + if error.category == "credentials": + raise WorkflowError( + "credentials", f"{error.detail} Token shape: {describe_token(token)}." + ) from None + raise + + print( + f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: " + f"{len(result['created'])} created, {len(result['updated'])} updated, " + f"{len(result['deleted'])} deleted." + ) + if result["kept"]: + print( + f"{len(result['kept'])} page(s) are in the wiki but not in " + f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in " + f"{wiki_dir}/{MARKER_NAME} to delete them." + ) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except WorkflowError as error: + print(f"ERROR: {error}", file=sys.stderr) + sys.exit(1) diff --git a/wiki/Onboarding.da.md b/wiki/Onboarding.da.md index 6a8ecdc..bf05f55 100644 --- a/wiki/Onboarding.da.md +++ b/wiki/Onboarding.da.md @@ -1,4 +1,4 @@ - + > Dette er en oversættelse af [`Onboarding`](Onboarding). Den engelske side er autoritativ, og ændringer her bliver overskrevet, næste gang wikien publiceres. @@ -35,7 +35,7 @@ Leveringen er automatisk, så et repository behøver ingen kopi af workflowet. S Workflowet **Publish wiki** kopierer Markdown-siderne i `wiki/` til repositoryets wiki, når en ændring af `wiki/` flettes til `main`. Det leveres som synkroniseringsworkflowet, men skriver kun, hvor et repository har meldt sig til. 1. Aktivér wikien i repositoryets indstillinger. -2. Tilføj Markdown-sider til `wiki/`. Filnavnet er sidens navn: `Home.md` er forsiden. +2. Tilføj Markdown-sider til `wiki/`. Filnavnet er sidens navn: `Home.md` er forsiden. En bindestreg i navnet er et mellemrum i sidens navn: `Home-da.md` er siden "Home da", og et link til den er `[Dansk](Home-da)`. 3. Tilføj markøren `wiki/.publish`. Lad den være tom for kun at oprette og opdatere sider. Skriv det ene ord `prune` i den for også at slette wiki-sider, der ikke findes i `wiki/`, så wikien altid er lig med mappen. 4. Gem `TOKEN_FOR_GITEA_WIKI` som en secret i repositoryets organisation med det scope, der står i [credentials.md](Credentials.da). 5. Flet ændringen til `main`. Loggen skal slutte med `Published page(s) to the wiki of /`. diff --git a/wiki/Onboarding.md b/wiki/Onboarding.md index 196ac22..6a81378 100644 --- a/wiki/Onboarding.md +++ b/wiki/Onboarding.md @@ -57,7 +57,8 @@ like the sync workflow, but it writes only where a repository opts in. 1. Enable the wiki in the repository settings. 2. Add Markdown pages to `wiki/`. The file name is the page name: `Home.md` is - the home page. + the home page. A hyphen in the name is a space in the page name: `Home-da.md` + is the page "Home da", and a link to it is `[Dansk](https://git.tirsystem.com/TirSystem/github-action/src/branch/main/guides/Home-da)`. 3. Add the marker `wiki/.publish`. Leave it empty to create and update pages only. Put the single word `prune` in it to also delete wiki pages that are not in `wiki/`, so the wiki always equals the folder. diff --git a/wiki/Onboarding.ms.md b/wiki/Onboarding.ms.md index 5eb83cb..4003987 100644 --- a/wiki/Onboarding.ms.md +++ b/wiki/Onboarding.ms.md @@ -1,4 +1,4 @@ - + > Ini ialah terjemahan [`Onboarding`](Onboarding). Halaman bahasa Inggeris ialah teks yang sah, dan perubahan di sini akan ditimpa pada penerbitan wiki yang seterusnya. @@ -35,7 +35,7 @@ Penghantaran adalah automatik, jadi repositori tidak memerlukan salinan aliran k Aliran kerja **Publish wiki** menyalin halaman Markdown dalam `wiki/` ke wiki repositori apabila perubahan pada `wiki/` digabungkan ke `main`. Ia dihantar seperti aliran kerja penyegerakan, tetapi hanya menulis di tempat repositori telah memilih untuk menyertainya. 1. Aktifkan wiki dalam tetapan repositori. -2. Tambah halaman Markdown ke `wiki/`. Nama fail ialah nama halaman: `Home.md` ialah halaman utama. +2. Tambah halaman Markdown ke `wiki/`. Nama fail ialah nama halaman: `Home.md` ialah halaman utama. Tanda sempang dalam nama ialah ruang dalam nama halaman: `Home-da.md` ialah halaman "Home da", dan pautan kepadanya ialah `[Dansk](Home-da)`. 3. Tambah penanda `wiki/.publish`. Biarkan kosong untuk mencipta dan mengemas kini halaman sahaja. Letakkan satu perkataan `prune` di dalamnya untuk turut memadam halaman wiki yang tiada dalam `wiki/`, supaya wiki sentiasa sama dengan folder itu. 4. Simpan `TOKEN_FOR_GITEA_WIKI` sebagai secret organisasi repositori, dengan skop yang tertera dalam [credentials.md](Credentials.ms). 5. Gabungkan perubahan ke `main`. Log mesti berakhir dengan `Published page(s) to the wiki of /`.