TirSystem/github-action
0
1
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 6 Wiki Activity

publish-wiki: send titles with spaces so Gitea stores Home-da.md
TirSystem/github-action: Publish wiki / publish-wiki (push) Skipped
TirSystem/github-action: Sync GitHub mirror metadata / sync-metadata (push) Skipped
Validate workflows / validate (pull_request) Successful in 18s
Details

Browse Source 
Gitea keeps a hyphen in a page title literal and stored Home-da as
Home-da.-.md, addressed as Home%2Dda, so a link to Home-da found no page.
The publisher now sends the key with hyphens replaced by spaces.

Task: MIL-006#6
Closes #119
This commit is contained in:
Tirsvad
2026-10-13 11:41:08 +08:00
parent 3dae3d4dfa
commit 63af3afd13
3 changed files with 894 additions and 866 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+97 -89
View File
@@ -1,89 +1,97 @@
# Changelog # Changelog
All notable changes to the shared workflows of this repository. The format and All notable changes to the shared workflows of this repository. The format and
the version rules are those of the release process (`guides/release-process.md`, the version rules are those of the release process (`guides/release-process.md`,
ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH` ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH`
and a moving major tag `vMAJOR`. and a moving major tag `vMAJOR`.
Each release lists its changes in these groups, in this order: Each release lists its changes in these groups, in this order:
- **Breaking**: a renamed or removed workflow, input, secret name or output; a - **Breaking**: a renamed or removed workflow, input, secret name or output; a
raised default tool version or changed default that can turn a green raised default tool version or changed default that can turn a green
consumer red; a changed meaning of pass or fail. A release with a Breaking consumer red; a changed meaning of pass or fail. A release with a Breaking
entry has a new major version. entry has a new major version.
- **Added**: a new optional input, workflow or capability. - **Added**: a new optional input, workflow or capability.
- **Changed**: a change with no visible effect for existing callers. - **Changed**: a change with no visible effect for existing callers.
- **Fixed**: a fix. - **Fixed**: a fix.
`python tools/changelog.py check` verifies the file, and `python tools/changelog.py check` verifies the file, and
`python tools/changelog.py notes <version>` prints the text of a release page. `python tools/changelog.py notes <version>` prints the text of a release page.
## [Unreleased] ## [Unreleased]
### Changed ### Changed
- Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a - Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a
runner for the host's default limit. runner for the host's default limit.
- `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the - `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the
moving tag `v4`. No change for callers. moving tag `v4`. No change for callers.
- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:` - The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:`
blocks of this repository's own `sync-milestones.yml` call blocks of this repository's own `sync-milestones.yml` call
`framework/scripts/ci-publish-milestones.sh`. No change for callers of the `framework/scripts/ci-publish-milestones.sh`. No change for callers of the
shared workflows. shared workflows.
## [1.1.0] - 2026-10-11 ### Fixed
### Added - `publish-wiki` sends a page title with spaces instead of hyphens. Gitea keeps a
hyphen in a title literal and stored `Home-da` as `Home-da.-.md`, addressed as
- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from `Home%2Dda`, so links to `Home-da` led to the page list. A new page is now
which the workflow fetches its script `tools/quality-checks.sh`. stored as `Home-da.md` and addressed as `Home-da`. Pages already created with
the old title must be deleted by hand (the publisher does not prune them).
### Changed
## [1.1.0] - 2026-10-11
- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of
embedding about 100 lines of shell; the checks, messages and results are the ### Added
same.
- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from
## [1.0.1] - 2026-10-11 which the workflow fetches its script `tools/quality-checks.sh`.
### Changed ### Changed
- Sync GitHub mirror metadata: a repository without a GitHub push mirror still - `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of
succeeds, and the log now ends with the line `Skipped`. embedding about 100 lines of shell; the checks, messages and results are the
- The scoped workflows fetch their scripts at the tag `v1` instead of `main`, same.
so a merge to `main` reaches consumers only when it is released.
## [1.0.1] - 2026-10-11
## [1.0.0] - 2026-10-10
### Changed
First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The
scoped workflows are delivered by the instance from `main` and cannot be pinned. - Sync GitHub mirror metadata: a repository without a GitHub push mirror still
succeeds, and the log now ends with the line `Skipped`.
### Added - The scoped workflows fetch their scripts at the tag `v1` instead of `main`,
so a merge to `main` reaches consumers only when it is released.
- Workflow `release.yml`: started by hand with a version, it releases only the
tip of `main` whose validation run is green and whose changelog entry exists, ## [1.0.0] - 2026-10-10
creates the version tag, the release page and the major tag, and does nothing
but check until `dry_run` is set to `false`. First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The
- Scoped workflow `sync-github-metadata.yml`: copies the description and topics scoped workflows are delivered by the instance from `main` and cannot be pinned.
of a Gitea repository to its GitHub push mirror on every push to `main`, by
hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped ### Added
successfully.
- A credential preflight before any change, and failure messages that name - Workflow `release.yml`: started by hand with a version, it releases only the
their category (`configuration`, `credentials`, `permissions`, `api`). tip of `main` whose validation run is green and whose changelog entry exists,
- Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki creates the version tag, the release page and the major tag, and does nothing
when a repository has opted in with `wiki/.publish`; the word `prune` in that but check until `dry_run` is set to `false`.
file also deletes pages that are not in `wiki/`. - Scoped workflow `sync-github-metadata.yml`: copies the description and topics
- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck` of a Gitea repository to its GitHub push mirror on every push to `main`, by
and `shfmt` on the tracked Python and shell files, with the inputs hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped
`languages`, `working-directory` and one pinned version per tool. successfully.
- Guides for onboarding a repository, workflow credentials, troubleshooting, - A credential preflight before any change, and failure messages that name
reading quality-check results and releasing, and the same pages on the wiki in their category (`configuration`, `credentials`, `permissions`, `api`).
English, Danish and Bahasa Malaysia. - Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki
when a repository has opted in with `wiki/.publish`; the word `prune` in that
### Changed file also deletes pages that are not in `wiki/`.
- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck`
- The scripts of the scoped workflows are files under `tools/`, fetched by a and `shfmt` on the tracked Python and shell files, with the inputs
checkout of this repository, instead of Python embedded in the YAML. `languages`, `working-directory` and one pinned version per tool.
- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`, - Guides for onboarding a repository, workflow credentials, troubleshooting,
`PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret reading quality-check results and releasing, and the same pages on the wiki in
`CREDENTIALS_FOR_GITHUB`, which is no longer read. English, Danish and Bahasa Malaysia.
### Changed
- The scripts of the scoped workflows are files under `tools/`, fetched by a
checkout of this repository, instead of Python embedded in the YAML.
- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`,
`PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret
`CREDENTIALS_FOR_GITHUB`, which is no longer read.
tests/test_publish_wiki.py
+491 -481
View File
@@ -1,481 +1,491 @@
"""Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow.""" """Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow."""
import base64 import base64
import io import io
import urllib.error import urllib.error
import pytest import pytest
import yaml import yaml
from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module
API = "https://git.example.test/api/v1" API = "https://git.example.test/api/v1"
REPO_URL = f"{API}/repos/TirSystem/repo" REPO_URL = f"{API}/repos/TirSystem/repo"
WIKI_URL = f"{REPO_URL}/wiki" WIKI_URL = f"{REPO_URL}/wiki"
@pytest.fixture() @pytest.fixture()
def publish_wiki(): def publish_wiki():
return load_publish_module() return load_publish_module()
def b64(text: str) -> str: def b64(text: str) -> str:
return base64.b64encode(text.encode()).decode() return base64.b64encode(text.encode()).decode()
class TestParseMarker: class TestParseMarker:
@pytest.mark.parametrize( @pytest.mark.parametrize(
("text", "expected"), ("text", "expected"),
[("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)], [("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)],
) )
def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected): def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected):
assert publish_wiki.parse_marker(text) is expected assert publish_wiki.parse_marker(text) is expected
@pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"]) @pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"])
def test_rejects_anything_else(self, publish_wiki, text): def test_rejects_anything_else(self, publish_wiki, text):
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
publish_wiki.parse_marker(text) publish_wiki.parse_marker(text)
assert caught.value.category == "configuration" assert caught.value.category == "configuration"
assert "prune" in str(caught.value) assert "prune" in str(caught.value)
class TestParseGiteaToken: class TestParseGiteaToken:
def test_returns_the_stripped_token(self, publish_wiki): def test_returns_the_stripped_token(self, publish_wiki):
assert publish_wiki.parse_gitea_token(" abc \n") == "abc" assert publish_wiki.parse_gitea_token(" abc \n") == "abc"
@pytest.mark.parametrize("raw", [None, "", " "]) @pytest.mark.parametrize("raw", [None, "", " "])
def test_rejects_missing_or_empty_token(self, publish_wiki, raw): def test_rejects_missing_or_empty_token(self, publish_wiki, raw):
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
publish_wiki.parse_gitea_token(raw) publish_wiki.parse_gitea_token(raw)
assert caught.value.category == "configuration" assert caught.value.category == "configuration"
assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value) assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value)
class TestReadPages: class TestReadPages:
def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path): def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path):
(tmp_path / "Home.md").write_bytes(b"home") (tmp_path / "Home.md").write_bytes(b"home")
(tmp_path / "Home.da.md").write_bytes(b"hjem") (tmp_path / "Home.da.md").write_bytes(b"hjem")
(tmp_path / ".publish").write_bytes(b"prune") (tmp_path / ".publish").write_bytes(b"prune")
(tmp_path / "Notes.txt").write_bytes(b"ignored") (tmp_path / "Notes.txt").write_bytes(b"ignored")
assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"} assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"}
def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped( def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped(
self, publish_wiki, tmp_path self, publish_wiki, tmp_path
): ):
(tmp_path / ".publish").write_text("prune", encoding="utf-8") (tmp_path / ".publish").write_text("prune", encoding="utf-8")
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
publish_wiki.read_pages(tmp_path) publish_wiki.read_pages(tmp_path)
class TestPageKey: class TestPageTitle:
@pytest.mark.parametrize( @pytest.mark.parametrize(
("name", "key"), ("key", "title"), [("Home", "Home"), ("Home-da", "Home da"), ("Getting-Started-ms", "Getting Started ms")]
[("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"), )
("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")], def test_a_hyphen_is_sent_as_a_space(self, publish_wiki, key, title):
) # Gitea keeps a hyphen in a title literal (Home-da.-.md, address Home%2Dda).
def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key): assert publish_wiki.page_title(key) == title
assert publish_wiki.page_key(name) == key
def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path): class TestPageKey:
(tmp_path / "Home da.md").write_bytes(b"hjem") @pytest.mark.parametrize(
(tmp_path / "_Sidebar.md").write_bytes(b"menu") ("name", "key"),
assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"} [("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"),
("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")],
def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path): )
(tmp_path / "Home da.md").write_bytes(b"one") def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key):
(tmp_path / "Home-da.md").write_bytes(b"two") assert publish_wiki.page_key(name) == key
with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"):
publish_wiki.read_pages(tmp_path) def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path):
(tmp_path / "Home da.md").write_bytes(b"hjem")
(tmp_path / "_Sidebar.md").write_bytes(b"menu")
class FakeGitea: assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"}
"""Answers like the Gitea repository and wiki endpoints."""
def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path):
def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True): (tmp_path / "Home da.md").write_bytes(b"one")
self.error = publish_wiki.WorkflowError (tmp_path / "Home-da.md").write_bytes(b"two")
self.wiki = dict(wiki or {}) with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"):
self.has_wiki = has_wiki publish_wiki.read_pages(tmp_path)
self.wiki_initialised = wiki_initialised
self.calls = []
class FakeGitea:
def __call__(self, url, method="GET", headers=None, body=None): """Answers like the Gitea repository and wiki endpoints."""
self.calls.append((method, url, body, headers))
if url == REPO_URL: def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True):
return {"has_wiki": self.has_wiki} self.error = publish_wiki.WorkflowError
if url.startswith(f"{WIKI_URL}/pages"): self.wiki = dict(wiki or {})
if not self.wiki_initialised: self.has_wiki = has_wiki
raise self.error("configuration", "404") self.wiki_initialised = wiki_initialised
return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki] self.calls = []
if method == "POST" and url == f"{WIKI_URL}/new":
self.wiki[body["title"]] = body["content_base64"] def __call__(self, url, method="GET", headers=None, body=None):
return None self.calls.append((method, url, body, headers))
title = url.rsplit("/", 1)[-1] if url == REPO_URL:
if method == "GET": return {"has_wiki": self.has_wiki}
return {"title": title, "content_base64": self.wiki[title]} if url.startswith(f"{WIKI_URL}/pages"):
if method == "PATCH": if not self.wiki_initialised:
self.wiki[title] = body["content_base64"] raise self.error("configuration", "404")
return None return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki]
if method == "DELETE": if method == "POST" and url == f"{WIKI_URL}/new":
del self.wiki[title] self.wiki[body["title"].replace(" ", "-")] = body["content_base64"]
return None return None
raise AssertionError(f"unexpected request {method} {url}") title = url.rsplit("/", 1)[-1]
if method == "GET":
@property return {"title": title, "content_base64": self.wiki[title]}
def writes(self): if method == "PATCH":
return [call for call in self.calls if call[0] != "GET"] assert " " in body["title"] or "-" not in body["title"], body["title"]
self.wiki[title] = body["content_base64"]
return None
@pytest.fixture() if method == "DELETE":
def environment(monkeypatch, tmp_path): del self.wiki[title]
wiki_dir = tmp_path / "wiki" return None
wiki_dir.mkdir() raise AssertionError(f"unexpected request {method} {url}")
(wiki_dir / "Home.md").write_text("home v2", encoding="utf-8")
(wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8") @property
(wiki_dir / ".publish").write_text("prune\n", encoding="utf-8") def writes(self):
monkeypatch.setenv("WIKI_DIR", str(wiki_dir)) return [call for call in self.calls if call[0] != "GET"]
monkeypatch.setenv("GITEA_API_URL", f"{API}/")
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token") @pytest.fixture()
return wiki_dir def environment(monkeypatch, tmp_path):
wiki_dir = tmp_path / "wiki"
wiki_dir.mkdir()
def run_main(publish_wiki, gitea): (wiki_dir / "Home.md").write_text("home v2", encoding="utf-8")
publish_wiki.request_json = gitea (wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8")
return publish_wiki.main() (wiki_dir / ".publish").write_text("prune\n", encoding="utf-8")
monkeypatch.setenv("WIKI_DIR", str(wiki_dir))
monkeypatch.setenv("GITEA_API_URL", f"{API}/")
class TestOptIn: monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
def test_a_repository_without_the_marker_is_never_written_to( monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token")
self, publish_wiki, environment, capsys return wiki_dir
):
(environment / ".publish").unlink()
gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")}) def run_main(publish_wiki, gitea):
publish_wiki.request_json = gitea
assert run_main(publish_wiki, gitea) == 0 return publish_wiki.main()
assert gitea.calls == []
assert "wiki publishing is not enabled" in capsys.readouterr().out class TestOptIn:
assert gitea.wiki == {"Old": b64("hand written")} def test_a_repository_without_the_marker_is_never_written_to(
self, publish_wiki, environment, capsys
def test_a_repository_without_the_marker_does_not_need_the_secret( ):
self, publish_wiki, environment, monkeypatch (environment / ".publish").unlink()
): gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")})
(environment / ".publish").unlink()
monkeypatch.setenv("WIKI_CREDENTIALS", "") assert run_main(publish_wiki, gitea) == 0
assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0
assert gitea.calls == []
def test_a_repository_without_a_wiki_directory_is_skipped( assert "wiki publishing is not enabled" in capsys.readouterr().out
self, publish_wiki, monkeypatch, tmp_path assert gitea.wiki == {"Old": b64("hand written")}
):
monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing")) def test_a_repository_without_the_marker_does_not_need_the_secret(
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") self, publish_wiki, environment, monkeypatch
gitea = FakeGitea(publish_wiki) ):
assert run_main(publish_wiki, gitea) == 0 (environment / ".publish").unlink()
assert gitea.calls == [] monkeypatch.setenv("WIKI_CREDENTIALS", "")
assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0
def test_an_opted_in_repository_without_the_secret_fails_before_any_request(
self, publish_wiki, environment, monkeypatch def test_a_repository_without_a_wiki_directory_is_skipped(
): self, publish_wiki, monkeypatch, tmp_path
monkeypatch.setenv("WIKI_CREDENTIALS", "") ):
gitea = FakeGitea(publish_wiki) monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing"))
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"): gitea = FakeGitea(publish_wiki)
run_main(publish_wiki, gitea) assert run_main(publish_wiki, gitea) == 0
assert gitea.calls == []
assert gitea.calls == []
def test_an_opted_in_repository_without_the_secret_fails_before_any_request(
def test_an_unknown_marker_content_is_refused_before_any_request( self, publish_wiki, environment, monkeypatch
self, publish_wiki, environment ):
): monkeypatch.setenv("WIKI_CREDENTIALS", "")
(environment / ".publish").write_text("mirror everything", encoding="utf-8") gitea = FakeGitea(publish_wiki)
gitea = FakeGitea(publish_wiki)
with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"):
with pytest.raises(publish_wiki.WorkflowError, match="prune"): run_main(publish_wiki, gitea)
run_main(publish_wiki, gitea)
assert gitea.calls == []
assert gitea.calls == []
def test_an_unknown_marker_content_is_refused_before_any_request(
self, publish_wiki, environment
class TestPublish: ):
def test_prune_creates_updates_and_deletes_to_match_the_source( (environment / ".publish").write_text("mirror everything", encoding="utf-8")
self, publish_wiki, environment, capsys gitea = FakeGitea(publish_wiki)
):
gitea = FakeGitea( with pytest.raises(publish_wiki.WorkflowError, match="prune"):
publish_wiki, run_main(publish_wiki, gitea)
wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")},
) assert gitea.calls == []
assert run_main(publish_wiki, gitea) == 0
class TestPublish:
assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")} def test_prune_creates_updates_and_deletes_to_match_the_source(
methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls] self, publish_wiki, environment, capsys
assert ("PATCH", "Home") in methods ):
assert ("DELETE", "Old") in methods gitea = FakeGitea(
assert ("PATCH", "Credentials") not in methods publish_wiki,
assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")},
)
def test_without_prune_pages_outside_the_source_are_kept(
self, publish_wiki, environment, capsys assert run_main(publish_wiki, gitea) == 0
):
(environment / ".publish").write_text("", encoding="utf-8") assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")}
gitea = FakeGitea( methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls]
publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")} assert ("PATCH", "Home") in methods
) assert ("DELETE", "Old") in methods
assert ("PATCH", "Credentials") not in methods
assert run_main(publish_wiki, gitea) == 0 assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out
assert gitea.wiki["Old"] == b64("hand written") def test_without_prune_pages_outside_the_source_are_kept(
assert gitea.wiki["Home"] == b64("home v2") self, publish_wiki, environment, capsys
assert "Credentials" in gitea.wiki ):
assert not [call for call in gitea.calls if call[0] == "DELETE"] (environment / ".publish").write_text("", encoding="utf-8")
out = capsys.readouterr().out gitea = FakeGitea(
assert "1 created, 1 updated, 0 deleted" in out publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")}
assert "1 page(s) are in the wiki but not in" in out )
assert "prune" in out
assert run_main(publish_wiki, gitea) == 0
def test_creates_every_page_when_the_wiki_was_never_written(
self, publish_wiki, environment assert gitea.wiki["Old"] == b64("hand written")
): assert gitea.wiki["Home"] == b64("home v2")
gitea = FakeGitea(publish_wiki, wiki_initialised=False) assert "Credentials" in gitea.wiki
assert not [call for call in gitea.calls if call[0] == "DELETE"]
assert run_main(publish_wiki, gitea) == 0 out = capsys.readouterr().out
assert "1 created, 1 updated, 0 deleted" in out
assert set(gitea.wiki) == {"Home", "Credentials"} assert "1 page(s) are in the wiki but not in" in out
assert [call[0] for call in gitea.calls].count("POST") == 2 assert "prune" in out
def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment): def test_creates_every_page_when_the_wiki_was_never_written(
gitea = FakeGitea( self, publish_wiki, environment
publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")} ):
) gitea = FakeGitea(publish_wiki, wiki_initialised=False)
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.writes == [] assert set(gitea.wiki) == {"Home", "Credentials"}
assert [call[0] for call in gitea.calls].count("POST") == 2
def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys):
gitea = FakeGitea(publish_wiki) def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment):
gitea = FakeGitea(
assert run_main(publish_wiki, gitea) == 0 publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")}
)
assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"}
output = capsys.readouterr() assert run_main(publish_wiki, gitea) == 0
assert "wiki-token" not in output.out + output.err
assert gitea.writes == []
def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing(
self, publish_wiki, environment def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys):
): gitea = FakeGitea(publish_wiki)
gitea = FakeGitea(publish_wiki, has_wiki=False)
assert run_main(publish_wiki, gitea) == 0
with pytest.raises(publish_wiki.WorkflowError) as caught:
run_main(publish_wiki, gitea) assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"}
output = capsys.readouterr()
assert caught.value.category == "configuration" assert "wiki-token" not in output.out + output.err
assert gitea.writes == []
def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing(
def test_an_empty_source_never_deletes_pages_even_with_prune( self, publish_wiki, environment
self, publish_wiki, environment ):
): gitea = FakeGitea(publish_wiki, has_wiki=False)
for path in environment.glob("*.md"):
path.unlink() with pytest.raises(publish_wiki.WorkflowError) as caught:
gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")}) run_main(publish_wiki, gitea)
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): assert caught.value.category == "configuration"
run_main(publish_wiki, gitea) assert gitea.writes == []
assert gitea.calls == [] def test_an_empty_source_never_deletes_pages_even_with_prune(
assert gitea.wiki == {"Home": b64("keep me")} self, publish_wiki, environment
):
for path in environment.glob("*.md"):
class TestPageIdentity: path.unlink()
"""A wiki that already has pages named with spaces, as Gitea stores them.""" gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")})
def test_a_file_named_with_a_space_updates_the_existing_page( with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
self, publish_wiki, environment run_main(publish_wiki, gitea)
):
(environment / "Home.md").unlink() assert gitea.calls == []
(environment / "Credentials.md").unlink() assert gitea.wiki == {"Home": b64("keep me")}
(environment / ".publish").write_text("", encoding="utf-8")
(environment / "Home da.md").write_text("hjem v2", encoding="utf-8")
gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")}) class TestPageIdentity:
"""A wiki that already has pages named with spaces, as Gitea stores them."""
assert run_main(publish_wiki, gitea) == 0
def test_a_file_named_with_a_space_updates_the_existing_page(
assert gitea.wiki == {"Home-da": b64("hjem v2")} self, publish_wiki, environment
assert [call[0] for call in gitea.calls].count("POST") == 0 ):
assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls] (environment / "Home.md").unlink()
(environment / "Credentials.md").unlink()
def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment): (environment / ".publish").write_text("", encoding="utf-8")
(environment / "Home.md").unlink() (environment / "Home da.md").write_text("hjem v2", encoding="utf-8")
(environment / "Credentials.md").unlink() gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")})
(environment / "Home da.md").write_text("hjem", encoding="utf-8")
(environment / "_Sidebar.md").write_text("menu", encoding="utf-8") assert run_main(publish_wiki, gitea) == 0
gitea = FakeGitea(
publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")} assert gitea.wiki == {"Home-da": b64("hjem v2")}
) assert [call[0] for call in gitea.calls].count("POST") == 0
assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls]
assert run_main(publish_wiki, gitea) == 0
def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment):
assert gitea.writes == [] (environment / "Home.md").unlink()
(environment / "Credentials.md").unlink()
def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling( (environment / "Home da.md").write_text("hjem", encoding="utf-8")
self, publish_wiki, environment (environment / "_Sidebar.md").write_text("menu", encoding="utf-8")
): gitea = FakeGitea(
(environment / "Home.md").unlink() publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")}
(environment / "Credentials.md").unlink() )
(environment / "Install ms.md").write_text("pasang", encoding="utf-8")
gitea = FakeGitea( assert run_main(publish_wiki, gitea) == 0
publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")}
) assert gitea.writes == []
assert run_main(publish_wiki, gitea) == 0 def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling(
self, publish_wiki, environment
assert gitea.wiki == {"Install-ms": b64("pasang")} ):
(environment / "Home.md").unlink()
def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki): (environment / "Credentials.md").unlink()
def listing(url, headers=None, **kwargs): (environment / "Install ms.md").write_text("pasang", encoding="utf-8")
return [ gitea = FakeGitea(
{"title": "Whatever", "sub_url": "Real-Address"}, publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")}
{"title": "Only title"}, )
{"title": "Encoded", "sub_url": "Caf%C3%A9"},
] assert run_main(publish_wiki, gitea) == 0
keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing) assert gitea.wiki == {"Install-ms": b64("pasang")}
assert keys == {"Real-Address", "Only-title", "Café"} def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki):
def listing(url, headers=None, **kwargs):
return [
class TestDescribeToken: {"title": "Whatever", "sub_url": "Real-Address"},
def test_recognises_a_gitea_token_shape(self, publish_wiki): {"title": "Only title"},
description = publish_wiki.describe_token("a1b2c3d4e5" * 4) {"title": "Encoded", "sub_url": "Caf%C3%A9"},
assert "40 characters" in description ]
assert "looks like a Gitea token" in description
keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing)
@pytest.mark.parametrize(
("token", "expected"), assert keys == {"Real-Address", "Only-title", "Café"}
[
("*******", "only mask characters"),
("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"), class TestDescribeToken:
('"' + "a" * 40 + '"', "quote character"), def test_recognises_a_gitea_token_shape(self, publish_wiki):
("abc def", "whitespace"), description = publish_wiki.describe_token("a1b2c3d4e5" * 4)
], assert "40 characters" in description
) assert "looks like a Gitea token" in description
def test_names_the_likely_mistake_without_echoing_the_token(
self, publish_wiki, token, expected @pytest.mark.parametrize(
): ("token", "expected"),
description = publish_wiki.describe_token(token) [
assert expected in description ("*******", "only mask characters"),
assert token not in description ("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"),
('"' + "a" * 40 + '"', "quote character"),
("abc def", "whitespace"),
class TestRejectedToken: ],
def test_rejection_adds_the_token_shape_but_not_the_token( )
self, publish_wiki, environment, monkeypatch def test_names_the_likely_mistake_without_echoing_the_token(
): self, publish_wiki, token, expected
monkeypatch.setenv("WIKI_CREDENTIALS", "*******") ):
description = publish_wiki.describe_token(token)
def reject(url, method="GET", headers=None, body=None): assert expected in description
raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.") assert token not in description
publish_wiki.request_json = reject
class TestRejectedToken:
with pytest.raises(publish_wiki.WorkflowError) as caught: def test_rejection_adds_the_token_shape_but_not_the_token(
publish_wiki.main() self, publish_wiki, environment, monkeypatch
):
message = str(caught.value) monkeypatch.setenv("WIKI_CREDENTIALS", "*******")
assert caught.value.category == "credentials"
assert "Token shape: 7 characters" in message def reject(url, method="GET", headers=None, body=None):
assert "only mask characters" in message raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.")
assert "*******" not in message.split("Token shape:")[0]
publish_wiki.request_json = reject
def test_other_categories_are_not_changed(self, publish_wiki, environment):
def forbid(url, method="GET", headers=None, body=None): with pytest.raises(publish_wiki.WorkflowError) as caught:
raise publish_wiki.WorkflowError("permissions", "Forbidden.") publish_wiki.main()
publish_wiki.request_json = forbid message = str(caught.value)
assert caught.value.category == "credentials"
with pytest.raises(publish_wiki.WorkflowError) as caught: assert "Token shape: 7 characters" in message
publish_wiki.main() assert "only mask characters" in message
assert "*******" not in message.split("Token shape:")[0]
assert str(caught.value) == "[permissions] Forbidden."
def test_other_categories_are_not_changed(self, publish_wiki, environment):
def forbid(url, method="GET", headers=None, body=None):
class TestRequestJson: raise publish_wiki.WorkflowError("permissions", "Forbidden.")
@staticmethod
def raise_http_error(publish_wiki, monkeypatch, code, reason): publish_wiki.request_json = forbid
def fake_urlopen(request, timeout):
raise urllib.error.HTTPError( with pytest.raises(publish_wiki.WorkflowError) as caught:
request.full_url, code, reason, {}, io.BytesIO(b"secret detail") publish_wiki.main()
)
assert str(caught.value) == "[permissions] Forbidden."
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
@pytest.mark.parametrize( class TestRequestJson:
("code", "reason", "category"), @staticmethod
[ def raise_http_error(publish_wiki, monkeypatch, code, reason):
(401, "Unauthorized", "credentials"), def fake_urlopen(request, timeout):
(403, "Forbidden", "permissions"), raise urllib.error.HTTPError(
(404, "Not Found", "configuration"), request.full_url, code, reason, {}, io.BytesIO(b"secret detail")
(500, "Server Error", "api"), )
],
) monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category):
self.raise_http_error(publish_wiki, monkeypatch, code, reason) @pytest.mark.parametrize(
("code", "reason", "category"),
with pytest.raises(publish_wiki.WorkflowError) as caught: [
publish_wiki.request_json(REPO_URL) (401, "Unauthorized", "credentials"),
(403, "Forbidden", "permissions"),
assert caught.value.category == category (404, "Not Found", "configuration"),
assert f"HTTP {code}" in str(caught.value) (500, "Server Error", "api"),
assert "secret detail" not in str(caught.value) ],
)
def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch): def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category):
def fake_urlopen(request, timeout): self.raise_http_error(publish_wiki, monkeypatch, code, reason)
raise urllib.error.URLError("no route")
with pytest.raises(publish_wiki.WorkflowError) as caught:
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) publish_wiki.request_json(REPO_URL)
with pytest.raises(publish_wiki.WorkflowError) as caught: assert caught.value.category == category
publish_wiki.request_json(REPO_URL) assert f"HTTP {code}" in str(caught.value)
assert "secret detail" not in str(caught.value)
assert caught.value.category == "api"
def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch):
def fake_urlopen(request, timeout):
class TestWorkflow: raise urllib.error.URLError("no route")
@pytest.fixture()
def document(self): monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8"))
with pytest.raises(publish_wiki.WorkflowError) as caught:
def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document): publish_wiki.request_json(REPO_URL)
triggers = document[True] # PyYAML reads the key `on` as True
assert triggers["push"]["branches"] == ["main"] assert caught.value.category == "api"
assert triggers["push"]["paths"] == ["wiki/**"]
assert "workflow_dispatch" in triggers
class TestWorkflow:
def test_job_has_read_only_permissions(self, document): @pytest.fixture()
assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"} def document(self):
return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8"))
def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document):
text = PUBLISH_WORKFLOW.read_text(encoding="utf-8") def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document):
assert "secrets.TOKEN_FOR_GITEA }}" not in text triggers = document[True] # PyYAML reads the key `on` as True
publish_step = document["jobs"]["publish-wiki"]["steps"][-1] assert triggers["push"]["branches"] == ["main"]
assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}" assert triggers["push"]["paths"] == ["wiki/**"]
assert "secrets" not in publish_step["run"] assert "workflow_dispatch" in triggers
def test_the_workflow_has_one_source_and_runs_the_script_file(self, document): def test_job_has_read_only_permissions(self, document):
assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists() assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"}
assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file()
steps = document["jobs"]["publish-wiki"]["steps"] def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document):
assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8") text = PUBLISH_WORKFLOW.read_text(encoding="utf-8")
assert steps[0]["uses"].startswith("actions/checkout@") assert "secrets.TOKEN_FOR_GITEA }}" not in text
assert steps[1]["with"] == { publish_step = document["jobs"]["publish-wiki"]["steps"][-1]
"repository": "TirSystem/github-action", assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}"
"ref": "v1", assert "secrets" not in publish_step["run"]
"path": "shared-scripts",
} def test_the_workflow_has_one_source_and_runs_the_script_file(self, document):
assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py" assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists()
assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file()
steps = document["jobs"]["publish-wiki"]["steps"]
def test_this_repository_opts_in_and_prunes(): assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8")
"""The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004).""" assert steps[0]["uses"].startswith("actions/checkout@")
marker = REPO_ROOT / "wiki" / ".publish" assert steps[1]["with"] == {
assert marker.read_text(encoding="utf-8").strip() == "prune" "repository": "TirSystem/github-action",
"ref": "v1",
"path": "shared-scripts",
}
assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py"
def test_this_repository_opts_in_and_prunes():
"""The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004)."""
marker = REPO_ROOT / "wiki" / ".publish"
assert marker.read_text(encoding="utf-8").strip() == "prune"
tools/publish_wiki.py
+306 -296
View File
@@ -1,296 +1,306 @@
"""Publish the pages in wiki/ to the Gitea wiki of the repository. """Publish the pages in wiki/ to the Gitea wiki of the repository.
Run by the scoped workflow publish-wiki.yml, which checks out the repository Run by the scoped workflow publish-wiki.yml, which checks out the repository
that owns the wiki and this repository, and runs this file (MIL-004 task 6). that owns the wiki and this repository, and runs this file (MIL-004 task 6).
A repository publishes only when it has opted in with wiki/.publish A repository publishes only when it has opted in with wiki/.publish
(ADR-0004); the word prune in that file lets the run delete wiki pages that (ADR-0004); the word prune in that file lets the run delete wiki pages that
are not in wiki/. It reads its settings from the environment: are not in wiki/. It reads its settings from the environment:
GITEA_API_URL Gitea API base, for example https://host/api/v1 GITEA_API_URL Gitea API base, for example https://host/api/v1
SOURCE_REPOSITORY owner/repo of the repository whose wiki is written SOURCE_REPOSITORY owner/repo of the repository whose wiki is written
WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI) WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI)
WIKI_DIR directory with the pages (default: wiki) WIKI_DIR directory with the pages (default: wiki)
Every failure is printed as ``ERROR: [category] message`` with the categories Every failure is printed as ``ERROR: [category] message`` with the categories
configuration, credentials, permissions and api; no secret value is printed. configuration, credentials, permissions and api; no secret value is printed.
""" """
import base64 import base64
import json import json
import os import os
import re import re
import sys import sys
import urllib.error import urllib.error
import urllib.parse import urllib.parse
import urllib.request import urllib.request
from pathlib import Path from pathlib import Path
PAGE_LIMIT = 50 PAGE_LIMIT = 50
REQUEST_TIMEOUT_SECONDS = 30 REQUEST_TIMEOUT_SECONDS = 30
COMMIT_MESSAGE = "Publish wiki from the repository" COMMIT_MESSAGE = "Publish wiki from the repository"
MARKER_NAME = ".publish" MARKER_NAME = ".publish"
PRUNE_WORD = "prune" PRUNE_WORD = "prune"
class WorkflowError(RuntimeError): class WorkflowError(RuntimeError):
"""A failure with a category. """A failure with a category.
configuration: a secret, a setting or the wiki source is missing or malformed. configuration: a secret, a setting or the wiki source is missing or malformed.
credentials: Gitea rejected the token (HTTP 401). credentials: Gitea rejected the token (HTTP 401).
permissions: the token is valid but not allowed (HTTP 403). permissions: the token is valid but not allowed (HTTP 403).
api: any other API or network failure. api: any other API or network failure.
""" """
def __init__(self, category, message): def __init__(self, category, message):
super().__init__(f"[{category}] {message}") super().__init__(f"[{category}] {message}")
self.category = category self.category = category
self.detail = message self.detail = message
def request_json(url, method="GET", headers=None, body=None): def request_json(url, method="GET", headers=None, body=None):
request = urllib.request.Request( request = urllib.request.Request(
url, url,
data=json.dumps(body).encode("utf-8") if body is not None else None, data=json.dumps(body).encode("utf-8") if body is not None else None,
headers=headers or {}, headers=headers or {},
method=method, method=method,
) )
try: try:
with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response: with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response:
content = response.read() content = response.read()
return json.loads(content) if content else None return json.loads(content) if content else None
except urllib.error.HTTPError as error: except urllib.error.HTTPError as error:
detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})" detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})"
if error.code == 401: if error.code == 401:
raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None
if error.code == 403: if error.code == 403:
raise WorkflowError( raise WorkflowError(
"permissions", f"{detail}. The token lacks a required permission." "permissions", f"{detail}. The token lacks a required permission."
) from None ) from None
if error.code == 404: if error.code == 404:
raise WorkflowError( raise WorkflowError(
"configuration", "configuration",
f"{detail}. The repository or page was not found, or the token " f"{detail}. The repository or page was not found, or the token "
"cannot see it.", "cannot see it.",
) from None ) from None
raise WorkflowError("api", detail) from None raise WorkflowError("api", detail) from None
except (urllib.error.URLError, TimeoutError) as error: except (urllib.error.URLError, TimeoutError) as error:
raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None
def parse_marker(text): def parse_marker(text):
"""Return True when the marker asks for pruning, False when it is empty.""" """Return True when the marker asks for pruning, False when it is empty."""
word = (text or "").strip() word = (text or "").strip()
if not word: if not word:
return False return False
if word == PRUNE_WORD: if word == PRUNE_WORD:
return True return True
raise WorkflowError( raise WorkflowError(
"configuration", "configuration",
f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.", f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.",
) )
def parse_gitea_token(raw_token): def parse_gitea_token(raw_token):
token = (raw_token or "").strip() token = (raw_token or "").strip()
if not token: if not token:
raise WorkflowError( raise WorkflowError(
"configuration", "configuration",
"TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of " "TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of "
"the organization of this repository.", "the organization of this repository.",
) )
return token return token
def describe_token(token): def describe_token(token):
"""Describe the shape of a token without revealing any of its characters.""" """Describe the shape of a token without revealing any of its characters."""
facts = [f"{len(token)} characters"] facts = [f"{len(token)} characters"]
if re.fullmatch(r"[0-9a-f]{40}", token): if re.fullmatch(r"[0-9a-f]{40}", token):
facts.append("looks like a Gitea token (40 hexadecimal characters)") facts.append("looks like a Gitea token (40 hexadecimal characters)")
return ", ".join(facts) return ", ".join(facts)
facts.append("a Gitea token has 40 hexadecimal characters") facts.append("a Gitea token has 40 hexadecimal characters")
if set(token) <= {"*", "x", "X", "•"}: if set(token) <= {"*", "x", "X", "•"}:
facts.append("only mask characters: the masked text was stored") facts.append("only mask characters: the masked text was stored")
if "=" in token: if "=" in token:
facts.append("contains '=': the secret name may be part of the value") facts.append("contains '=': the secret name may be part of the value")
if re.search(r"[\"'`]", token): if re.search(r"[\"'`]", token):
facts.append("contains a quote character") facts.append("contains a quote character")
if re.search(r"\s", token): if re.search(r"\s", token):
facts.append("contains whitespace") facts.append("contains whitespace")
return ", ".join(facts) return ", ".join(facts)
def split_repository(full_name): def split_repository(full_name):
owner, separator, repo = (full_name or "").partition("/") owner, separator, repo = (full_name or "").partition("/")
if not separator or not owner or not repo: if not separator or not owner or not repo:
raise WorkflowError("configuration", "Could not determine the Gitea repository.") raise WorkflowError("configuration", "Could not determine the Gitea repository.")
return owner, repo return owner, repo
def page_key(title): def page_key(title):
"""Return the identity of a page: Gitea stores a space as a hyphen. """Return the identity of a page: Gitea stores a space as a hyphen.
The page "Home da" is the file Home-da.md and its address is Home-da, so The page "Home da" is the file Home-da.md and its address is Home-da, so
both spellings must name the same page. both spellings must name the same page.
""" """
return str(title).replace(" ", "-") return str(title).replace(" ", "-")
def read_pages(wiki_dir): def page_title(key):
"""Return {page key: content} for every Markdown file in wiki_dir.""" """Return the title to send to Gitea for a page key.
pages = {}
for path in sorted(wiki_dir.glob("*.md")): A hyphen in a title is a literal hyphen to Gitea, which then stores the page
key = page_key(path.stem) as Home-da.-.md and addresses it as Home%2Dda, so a link to Home-da finds
if key in pages: nothing. A space is stored as a hyphen (Home-da.md) and addressed as Home-da.
raise WorkflowError( """
"configuration", return key.replace("-", " ")
f"Two files in {wiki_dir} name the wiki page {key}; a space and "
"a hyphen are the same in a page name.",
) def read_pages(wiki_dir):
pages[key] = path.read_bytes() """Return {page key: content} for every Markdown file in wiki_dir."""
if not pages: pages = {}
# Publishing an empty directory would delete the whole wiki. for path in sorted(wiki_dir.glob("*.md")):
raise WorkflowError( key = page_key(path.stem)
"configuration", f"The wiki source directory {wiki_dir} has no Markdown pages." if key in pages:
) raise WorkflowError(
return pages "configuration",
f"Two files in {wiki_dir} name the wiki page {key}; a space and "
"a hyphen are the same in a page name.",
def list_wiki_titles(wiki_url, headers, send): )
"""Return the keys of the pages that exist in the wiki. pages[key] = path.read_bytes()
if not pages:
The key is the address Gitea reports (sub_url), or the title with its # Publishing an empty directory would delete the whole wiki.
spaces replaced by hyphens. raise WorkflowError(
"configuration", f"The wiki source directory {wiki_dir} has no Markdown pages."
Gitea answers 404 for a wiki that has never been written to; that is an )
empty wiki, not a failure. return pages
"""
titles = set()
page = 1 def list_wiki_titles(wiki_url, headers, send):
while True: """Return the keys of the pages that exist in the wiki.
url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}"
try: The key is the address Gitea reports (sub_url), or the title with its
batch = send(url, headers=headers) spaces replaced by hyphens.
except WorkflowError as error:
if error.category == "configuration" and page == 1: Gitea answers 404 for a wiki that has never been written to; that is an
return titles empty wiki, not a failure.
raise """
if not isinstance(batch, list): titles = set()
raise WorkflowError("api", "Gitea returned an invalid wiki page list.") page = 1
titles.update( while True:
urllib.parse.unquote(str(item.get("sub_url") or "")) url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}"
or page_key(item.get("title", "")) try:
for item in batch batch = send(url, headers=headers)
) except WorkflowError as error:
if len(batch) < PAGE_LIMIT: if error.category == "configuration" and page == 1:
return titles return titles
page += 1 raise
if not isinstance(batch, list):
raise WorkflowError("api", "Gitea returned an invalid wiki page list.")
def read_wiki_page(wiki_url, title, headers, send): titles.update(
quoted = urllib.parse.quote(title, safe="") urllib.parse.unquote(str(item.get("sub_url") or ""))
page = send(f"{wiki_url}/page/{quoted}", headers=headers) or page_key(item.get("title", ""))
return base64.b64decode((page or {}).get("content_base64", "")) for item in batch
)
if len(batch) < PAGE_LIMIT:
def publish(pages, wiki_url, headers, send, prune): return titles
"""Create and update the pages; delete the others only when prune is set. page += 1
Returns the titles created, updated and deleted, and the titles that were
left in the wiki because pruning is off. def read_wiki_page(wiki_url, title, headers, send):
""" quoted = urllib.parse.quote(title, safe="")
existing = list_wiki_titles(wiki_url, headers, send) page = send(f"{wiki_url}/page/{quoted}", headers=headers)
current = { return base64.b64decode((page or {}).get("content_base64", ""))
title: read_wiki_page(wiki_url, title, headers, send)
for title in sorted(existing & pages.keys())
} def publish(pages, wiki_url, headers, send, prune):
"""Create and update the pages; delete the others only when prune is set.
result = {"created": [], "updated": [], "deleted": [], "kept": []}
for title, content in pages.items(): Returns the titles created, updated and deleted, and the titles that were
body = { left in the wiki because pruning is off.
"title": title, """
"content_base64": base64.b64encode(content).decode("ascii"), existing = list_wiki_titles(wiki_url, headers, send)
"message": COMMIT_MESSAGE, current = {
} title: read_wiki_page(wiki_url, title, headers, send)
if title not in existing: for title in sorted(existing & pages.keys())
send(f"{wiki_url}/new", method="POST", headers=headers, body=body) }
result["created"].append(title)
elif current[title] != content: result = {"created": [], "updated": [], "deleted": [], "kept": []}
quoted = urllib.parse.quote(title, safe="") for title, content in pages.items():
send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body) body = {
result["updated"].append(title) "title": page_title(title),
"content_base64": base64.b64encode(content).decode("ascii"),
for title in sorted(existing - pages.keys()): "message": COMMIT_MESSAGE,
if not prune: }
result["kept"].append(title) if title not in existing:
continue send(f"{wiki_url}/new", method="POST", headers=headers, body=body)
quoted = urllib.parse.quote(title, safe="") result["created"].append(title)
send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers) elif current[title] != content:
result["deleted"].append(title) quoted = urllib.parse.quote(title, safe="")
return result send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body)
result["updated"].append(title)
def main(): for title in sorted(existing - pages.keys()):
wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki")) if not prune:
owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY")) result["kept"].append(title)
continue
marker = wiki_dir / MARKER_NAME quoted = urllib.parse.quote(title, safe="")
if not marker.is_file(): send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers)
print( result["deleted"].append(title)
f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; " return result
"wiki publishing is not enabled for this repository."
)
return 0 def main():
prune = parse_marker(marker.read_text(encoding="utf-8")) wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki"))
owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY"))
pages = read_pages(wiki_dir)
token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS")) marker = wiki_dir / MARKER_NAME
if not marker.is_file():
api_url = os.environ["GITEA_API_URL"].rstrip("/") print(
repository_url = ( f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; "
f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/" "wiki publishing is not enabled for this repository."
f"{urllib.parse.quote(repo, safe='')}" )
) return 0
headers = { prune = parse_marker(marker.read_text(encoding="utf-8"))
"Authorization": f"token {token}",
"Accept": "application/json", pages = read_pages(wiki_dir)
"Content-Type": "application/json", token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS"))
}
api_url = os.environ["GITEA_API_URL"].rstrip("/")
try: repository_url = (
repository = request_json(repository_url, headers=headers) f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/"
if isinstance(repository, dict) and repository.get("has_wiki") is False: f"{urllib.parse.quote(repo, safe='')}"
raise WorkflowError( )
"configuration", headers = {
f"The wiki is not enabled on {owner}/{repo}. Enable it in the " "Authorization": f"token {token}",
"repository settings.", "Accept": "application/json",
) "Content-Type": "application/json",
print("Preflight passed: Gitea accepted the token and the wiki is enabled.") }
result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune) try:
except WorkflowError as error: repository = request_json(repository_url, headers=headers)
if error.category == "credentials": if isinstance(repository, dict) and repository.get("has_wiki") is False:
raise WorkflowError( raise WorkflowError(
"credentials", f"{error.detail} Token shape: {describe_token(token)}." "configuration",
) from None f"The wiki is not enabled on {owner}/{repo}. Enable it in the "
raise "repository settings.",
)
print( print("Preflight passed: Gitea accepted the token and the wiki is enabled.")
f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: "
f"{len(result['created'])} created, {len(result['updated'])} updated, " result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune)
f"{len(result['deleted'])} deleted." except WorkflowError as error:
) if error.category == "credentials":
if result["kept"]: raise WorkflowError(
print( "credentials", f"{error.detail} Token shape: {describe_token(token)}."
f"{len(result['kept'])} page(s) are in the wiki but not in " ) from None
f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in " raise
f"{wiki_dir}/{MARKER_NAME} to delete them."
) print(
return 0 f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: "
f"{len(result['created'])} created, {len(result['updated'])} updated, "
f"{len(result['deleted'])} deleted."
if __name__ == "__main__": )
try: if result["kept"]:
sys.exit(main()) print(
except WorkflowError as error: f"{len(result['kept'])} page(s) are in the wiki but not in "
print(f"ERROR: {error}", file=sys.stderr) f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in "
sys.exit(1) f"{wiki_dir}/{MARKER_NAME} to delete them."
)
return 0
if __name__ == "__main__":
try:
sys.exit(main())
except WorkflowError as error:
print(f"ERROR: {error}", file=sys.stderr)
sys.exit(1)