TirSystem/github-action
0
1
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Restore LF line endings and document hyphenated page names #121

Merged
Tirsvad merged 1 commits from fix-lf-endings into main 2026-10-13 05:51:12 +02:00
Conversation 0 Commits 1 Files Changed 7
+905 -900
7 changed files with 905 additions and 900 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit df525d129e - Show all commits
CHANGELOG.md
+100 -97
View File
@@ -1,97 +1,100 @@
# Changelog # Changelog
All notable changes to the shared workflows of this repository. The format and All notable changes to the shared workflows of this repository. The format and
the version rules are those of the release process (`guides/release-process.md`, the version rules are those of the release process (`guides/release-process.md`,
ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH` ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH`
and a moving major tag `vMAJOR`. and a moving major tag `vMAJOR`.
Each release lists its changes in these groups, in this order: Each release lists its changes in these groups, in this order:
- **Breaking**: a renamed or removed workflow, input, secret name or output; a - **Breaking**: a renamed or removed workflow, input, secret name or output; a
raised default tool version or changed default that can turn a green raised default tool version or changed default that can turn a green
consumer red; a changed meaning of pass or fail. A release with a Breaking consumer red; a changed meaning of pass or fail. A release with a Breaking
entry has a new major version. entry has a new major version.
- **Added**: a new optional input, workflow or capability. - **Added**: a new optional input, workflow or capability.
- **Changed**: a change with no visible effect for existing callers. - **Changed**: a change with no visible effect for existing callers.
- **Fixed**: a fix. - **Fixed**: a fix.
`python tools/changelog.py check` verifies the file, and `python tools/changelog.py check` verifies the file, and
`python tools/changelog.py notes <version>` prints the text of a release page. `python tools/changelog.py notes <version>` prints the text of a release page.
## [Unreleased] ## [Unreleased]
### Changed ### Changed
- Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a - Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a
runner for the host's default limit. runner for the host's default limit.
- `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the - `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the
moving tag `v4`. No change for callers. moving tag `v4`. No change for callers.
- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:` - The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:`
blocks of this repository's own `sync-milestones.yml` call blocks of this repository's own `sync-milestones.yml` call
`framework/scripts/ci-publish-milestones.sh`. No change for callers of the `framework/scripts/ci-publish-milestones.sh`. No change for callers of the
shared workflows. shared workflows.
### Fixed ### Fixed
- `publish-wiki` sends a page title with spaces instead of hyphens. Gitea keeps a - `publish-wiki` sends a page title with spaces instead of hyphens. Gitea keeps a
hyphen in a title literal and stored `Home-da` as `Home-da.-.md`, addressed as hyphen in a title literal and stored `Home-da` as `Home-da.-.md`, addressed as
`Home%2Dda`, so links to `Home-da` led to the page list. A new page is now `Home%2Dda`, so links to `Home-da` led to the page list. A new page is now
stored as `Home-da.md` and addressed as `Home-da`. Pages already created with stored as `Home-da.md` and addressed as `Home-da`. Pages already created with
the old title must be deleted by hand (the publisher does not prune them). the old title must be deleted by hand (the publisher does not prune them).
- The onboarding guide and its wiki pages (English, Danish, Bahasa Malaysia) say
## [1.1.0] - 2026-10-11 that a hyphen in a file name is a space in the page name, and how to link to
it.
### Added
## [1.1.0] - 2026-10-11
- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from
which the workflow fetches its script `tools/quality-checks.sh`. ### Added
### Changed - Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from
which the workflow fetches its script `tools/quality-checks.sh`.
- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of
embedding about 100 lines of shell; the checks, messages and results are the ### Changed
same.
- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of
## [1.0.1] - 2026-10-11 embedding about 100 lines of shell; the checks, messages and results are the
same.
### Changed
## [1.0.1] - 2026-10-11
- Sync GitHub mirror metadata: a repository without a GitHub push mirror still
succeeds, and the log now ends with the line `Skipped`. ### Changed
- The scoped workflows fetch their scripts at the tag `v1` instead of `main`,
so a merge to `main` reaches consumers only when it is released. - Sync GitHub mirror metadata: a repository without a GitHub push mirror still
succeeds, and the log now ends with the line `Skipped`.
## [1.0.0] - 2026-10-10 - The scoped workflows fetch their scripts at the tag `v1` instead of `main`,
so a merge to `main` reaches consumers only when it is released.
First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The
scoped workflows are delivered by the instance from `main` and cannot be pinned. ## [1.0.0] - 2026-10-10
### Added First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The
scoped workflows are delivered by the instance from `main` and cannot be pinned.
- Workflow `release.yml`: started by hand with a version, it releases only the
tip of `main` whose validation run is green and whose changelog entry exists, ### Added
creates the version tag, the release page and the major tag, and does nothing
but check until `dry_run` is set to `false`. - Workflow `release.yml`: started by hand with a version, it releases only the
- Scoped workflow `sync-github-metadata.yml`: copies the description and topics tip of `main` whose validation run is green and whose changelog entry exists,
of a Gitea repository to its GitHub push mirror on every push to `main`, by creates the version tag, the release page and the major tag, and does nothing
hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped but check until `dry_run` is set to `false`.
successfully. - Scoped workflow `sync-github-metadata.yml`: copies the description and topics
- A credential preflight before any change, and failure messages that name of a Gitea repository to its GitHub push mirror on every push to `main`, by
their category (`configuration`, `credentials`, `permissions`, `api`). hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped
- Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki successfully.
when a repository has opted in with `wiki/.publish`; the word `prune` in that - A credential preflight before any change, and failure messages that name
file also deletes pages that are not in `wiki/`. their category (`configuration`, `credentials`, `permissions`, `api`).
- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck` - Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki
and `shfmt` on the tracked Python and shell files, with the inputs when a repository has opted in with `wiki/.publish`; the word `prune` in that
`languages`, `working-directory` and one pinned version per tool. file also deletes pages that are not in `wiki/`.
- Guides for onboarding a repository, workflow credentials, troubleshooting, - Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck`
reading quality-check results and releasing, and the same pages on the wiki in and `shfmt` on the tracked Python and shell files, with the inputs
English, Danish and Bahasa Malaysia. `languages`, `working-directory` and one pinned version per tool.
- Guides for onboarding a repository, workflow credentials, troubleshooting,
### Changed reading quality-check results and releasing, and the same pages on the wiki in
English, Danish and Bahasa Malaysia.
- The scripts of the scoped workflows are files under `tools/`, fetched by a
checkout of this repository, instead of Python embedded in the YAML. ### Changed
- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`,
`PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret - The scripts of the scoped workflows are files under `tools/`, fetched by a
`CREDENTIALS_FOR_GITHUB`, which is no longer read. checkout of this repository, instead of Python embedded in the YAML.
- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`,
`PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret
`CREDENTIALS_FOR_GITHUB`, which is no longer read.
guides/onboarding.md
+2 -1
View File
@@ -51,7 +51,8 @@ like the sync workflow, but it writes only where a repository opts in.
1. Enable the wiki in the repository settings. 1. Enable the wiki in the repository settings.
2. Add Markdown pages to `wiki/`. The file name is the page name: `Home.md` is 2. Add Markdown pages to `wiki/`. The file name is the page name: `Home.md` is
the home page. the home page. A hyphen in the name is a space in the page name: `Home-da.md`
is the page "Home da", and a link to it is `[Dansk](Home-da)`.
3. Add the marker `wiki/.publish`. Leave it empty to create and update pages 3. Add the marker `wiki/.publish`. Leave it empty to create and update pages
only. Put the single word `prune` in it to also delete wiki pages that are only. Put the single word `prune` in it to also delete wiki pages that are
not in `wiki/`, so the wiki always equals the folder. not in `wiki/`, so the wiki always equals the folder.
tests/test_publish_wiki.py
+491 -491
View File
@@ -1,491 +1,491 @@
"""Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow.""" """Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow."""
import base64 import base64
import io import io
import urllib.error import urllib.error
import pytest import pytest
import yaml import yaml
from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module
API = "https://git.example.test/api/v1" API = "https://git.example.test/api/v1"
REPO_URL = f"{API}/repos/TirSystem/repo" REPO_URL = f"{API}/repos/TirSystem/repo"
WIKI_URL = f"{REPO_URL}/wiki" WIKI_URL = f"{REPO_URL}/wiki"
@pytest.fixture() @pytest.fixture()
def publish_wiki(): def publish_wiki():
return load_publish_module() return load_publish_module()
def b64(text: str) -> str: def b64(text: str) -> str:
return base64.b64encode(text.encode()).decode() return base64.b64encode(text.encode()).decode()
class TestParseMarker: class TestParseMarker:
@pytest.mark.parametrize( @pytest.mark.parametrize(
("text", "expected"), ("text", "expected"),
[("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)], [("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)],
) )
def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected): def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected):
assert publish_wiki.parse_marker(text) is expected assert publish_wiki.parse_marker(text) is expected
@pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"]) @pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"])
def test_rejects_anything_else(self, publish_wiki, text): def test_rejects_anything_else(self, publish_wiki, text):
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
publish_wiki.parse_marker(text) publish_wiki.parse_marker(text)
assert caught.value.category == "configuration" assert caught.value.category == "configuration"
assert "prune" in str(caught.value) assert "prune" in str(caught.value)
class TestParseGiteaToken: class TestParseGiteaToken:
def test_returns_the_stripped_token(self, publish_wiki): def test_returns_the_stripped_token(self, publish_wiki):
assert publish_wiki.parse_gitea_token(" abc \n") == "abc" assert publish_wiki.parse_gitea_token(" abc \n") == "abc"
@pytest.mark.parametrize("raw", [None, "", " "]) @pytest.mark.parametrize("raw", [None, "", " "])
def test_rejects_missing_or_empty_token(self, publish_wiki, raw): def test_rejects_missing_or_empty_token(self, publish_wiki, raw):
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
publish_wiki.parse_gitea_token(raw) publish_wiki.parse_gitea_token(raw)
assert caught.value.category == "configuration" assert caught.value.category == "configuration"
assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value) assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value)
class TestReadPages: class TestReadPages:
def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path): def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path):
(tmp_path / "Home.md").write_bytes(b"home") (tmp_path / "Home.md").write_bytes(b"home")
(tmp_path / "Home.da.md").write_bytes(b"hjem") (tmp_path / "Home.da.md").write_bytes(b"hjem")
(tmp_path / ".publish").write_bytes(b"prune") (tmp_path / ".publish").write_bytes(b"prune")
(tmp_path / "Notes.txt").write_bytes(b"ignored") (tmp_path / "Notes.txt").write_bytes(b"ignored")
assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"} assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"}
def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped( def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped(
self, publish_wiki, tmp_path self, publish_wiki, tmp_path
): ):
(tmp_path / ".publish").write_text("prune", encoding="utf-8") (tmp_path / ".publish").write_text("prune", encoding="utf-8")
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
publish_wiki.read_pages(tmp_path) publish_wiki.read_pages(tmp_path)
class TestPageTitle: class TestPageTitle:
@pytest.mark.parametrize( @pytest.mark.parametrize(
("key", "title"), [("Home", "Home"), ("Home-da", "Home da"), ("Getting-Started-ms", "Getting Started ms")] ("key", "title"), [("Home", "Home"), ("Home-da", "Home da"), ("Getting-Started-ms", "Getting Started ms")]
) )
def test_a_hyphen_is_sent_as_a_space(self, publish_wiki, key, title): def test_a_hyphen_is_sent_as_a_space(self, publish_wiki, key, title):
# Gitea keeps a hyphen in a title literal (Home-da.-.md, address Home%2Dda). # Gitea keeps a hyphen in a title literal (Home-da.-.md, address Home%2Dda).
assert publish_wiki.page_title(key) == title assert publish_wiki.page_title(key) == title
class TestPageKey: class TestPageKey:
@pytest.mark.parametrize( @pytest.mark.parametrize(
("name", "key"), ("name", "key"),
[("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"), [("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"),
("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")], ("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")],
) )
def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key): def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key):
assert publish_wiki.page_key(name) == key assert publish_wiki.page_key(name) == key
def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path): def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path):
(tmp_path / "Home da.md").write_bytes(b"hjem") (tmp_path / "Home da.md").write_bytes(b"hjem")
(tmp_path / "_Sidebar.md").write_bytes(b"menu") (tmp_path / "_Sidebar.md").write_bytes(b"menu")
assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"} assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"}
def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path): def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path):
(tmp_path / "Home da.md").write_bytes(b"one") (tmp_path / "Home da.md").write_bytes(b"one")
(tmp_path / "Home-da.md").write_bytes(b"two") (tmp_path / "Home-da.md").write_bytes(b"two")
with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"): with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"):
publish_wiki.read_pages(tmp_path) publish_wiki.read_pages(tmp_path)
class FakeGitea: class FakeGitea:
"""Answers like the Gitea repository and wiki endpoints.""" """Answers like the Gitea repository and wiki endpoints."""
def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True): def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True):
self.error = publish_wiki.WorkflowError self.error = publish_wiki.WorkflowError
self.wiki = dict(wiki or {}) self.wiki = dict(wiki or {})
self.has_wiki = has_wiki self.has_wiki = has_wiki
self.wiki_initialised = wiki_initialised self.wiki_initialised = wiki_initialised
self.calls = [] self.calls = []
def __call__(self, url, method="GET", headers=None, body=None): def __call__(self, url, method="GET", headers=None, body=None):
self.calls.append((method, url, body, headers)) self.calls.append((method, url, body, headers))
if url == REPO_URL: if url == REPO_URL:
return {"has_wiki": self.has_wiki} return {"has_wiki": self.has_wiki}
if url.startswith(f"{WIKI_URL}/pages"): if url.startswith(f"{WIKI_URL}/pages"):
if not self.wiki_initialised: if not self.wiki_initialised:
raise self.error("configuration", "404") raise self.error("configuration", "404")
return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki] return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki]
if method == "POST" and url == f"{WIKI_URL}/new": if method == "POST" and url == f"{WIKI_URL}/new":
self.wiki[body["title"].replace(" ", "-")] = body["content_base64"] self.wiki[body["title"].replace(" ", "-")] = body["content_base64"]
return None return None
title = url.rsplit("/", 1)[-1] title = url.rsplit("/", 1)[-1]
if method == "GET": if method == "GET":
return {"title": title, "content_base64": self.wiki[title]} return {"title": title, "content_base64": self.wiki[title]}
if method == "PATCH": if method == "PATCH":
assert " " in body["title"] or "-" not in body["title"], body["title"] assert " " in body["title"] or "-" not in body["title"], body["title"]
self.wiki[title] = body["content_base64"] self.wiki[title] = body["content_base64"]
return None return None
if method == "DELETE": if method == "DELETE":
del self.wiki[title] del self.wiki[title]
return None return None
raise AssertionError(f"unexpected request {method} {url}") raise AssertionError(f"unexpected request {method} {url}")
@property @property
def writes(self): def writes(self):
return [call for call in self.calls if call[0] != "GET"] return [call for call in self.calls if call[0] != "GET"]
@pytest.fixture() @pytest.fixture()
def environment(monkeypatch, tmp_path): def environment(monkeypatch, tmp_path):
wiki_dir = tmp_path / "wiki" wiki_dir = tmp_path / "wiki"
wiki_dir.mkdir() wiki_dir.mkdir()
(wiki_dir / "Home.md").write_text("home v2", encoding="utf-8") (wiki_dir / "Home.md").write_text("home v2", encoding="utf-8")
(wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8") (wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8")
(wiki_dir / ".publish").write_text("prune\n", encoding="utf-8") (wiki_dir / ".publish").write_text("prune\n", encoding="utf-8")
monkeypatch.setenv("WIKI_DIR", str(wiki_dir)) monkeypatch.setenv("WIKI_DIR", str(wiki_dir))
monkeypatch.setenv("GITEA_API_URL", f"{API}/") monkeypatch.setenv("GITEA_API_URL", f"{API}/")
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token") monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token")
return wiki_dir return wiki_dir
def run_main(publish_wiki, gitea): def run_main(publish_wiki, gitea):
publish_wiki.request_json = gitea publish_wiki.request_json = gitea
return publish_wiki.main() return publish_wiki.main()
class TestOptIn: class TestOptIn:
def test_a_repository_without_the_marker_is_never_written_to( def test_a_repository_without_the_marker_is_never_written_to(
self, publish_wiki, environment, capsys self, publish_wiki, environment, capsys
): ):
(environment / ".publish").unlink() (environment / ".publish").unlink()
gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")}) gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")})
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.calls == [] assert gitea.calls == []
assert "wiki publishing is not enabled" in capsys.readouterr().out assert "wiki publishing is not enabled" in capsys.readouterr().out
assert gitea.wiki == {"Old": b64("hand written")} assert gitea.wiki == {"Old": b64("hand written")}
def test_a_repository_without_the_marker_does_not_need_the_secret( def test_a_repository_without_the_marker_does_not_need_the_secret(
self, publish_wiki, environment, monkeypatch self, publish_wiki, environment, monkeypatch
): ):
(environment / ".publish").unlink() (environment / ".publish").unlink()
monkeypatch.setenv("WIKI_CREDENTIALS", "") monkeypatch.setenv("WIKI_CREDENTIALS", "")
assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0 assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0
def test_a_repository_without_a_wiki_directory_is_skipped( def test_a_repository_without_a_wiki_directory_is_skipped(
self, publish_wiki, monkeypatch, tmp_path self, publish_wiki, monkeypatch, tmp_path
): ):
monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing")) monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing"))
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo") monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
gitea = FakeGitea(publish_wiki) gitea = FakeGitea(publish_wiki)
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.calls == [] assert gitea.calls == []
def test_an_opted_in_repository_without_the_secret_fails_before_any_request( def test_an_opted_in_repository_without_the_secret_fails_before_any_request(
self, publish_wiki, environment, monkeypatch self, publish_wiki, environment, monkeypatch
): ):
monkeypatch.setenv("WIKI_CREDENTIALS", "") monkeypatch.setenv("WIKI_CREDENTIALS", "")
gitea = FakeGitea(publish_wiki) gitea = FakeGitea(publish_wiki)
with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"): with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"):
run_main(publish_wiki, gitea) run_main(publish_wiki, gitea)
assert gitea.calls == [] assert gitea.calls == []
def test_an_unknown_marker_content_is_refused_before_any_request( def test_an_unknown_marker_content_is_refused_before_any_request(
self, publish_wiki, environment self, publish_wiki, environment
): ):
(environment / ".publish").write_text("mirror everything", encoding="utf-8") (environment / ".publish").write_text("mirror everything", encoding="utf-8")
gitea = FakeGitea(publish_wiki) gitea = FakeGitea(publish_wiki)
with pytest.raises(publish_wiki.WorkflowError, match="prune"): with pytest.raises(publish_wiki.WorkflowError, match="prune"):
run_main(publish_wiki, gitea) run_main(publish_wiki, gitea)
assert gitea.calls == [] assert gitea.calls == []
class TestPublish: class TestPublish:
def test_prune_creates_updates_and_deletes_to_match_the_source( def test_prune_creates_updates_and_deletes_to_match_the_source(
self, publish_wiki, environment, capsys self, publish_wiki, environment, capsys
): ):
gitea = FakeGitea( gitea = FakeGitea(
publish_wiki, publish_wiki,
wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")}, wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")},
) )
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")} assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")}
methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls] methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls]
assert ("PATCH", "Home") in methods assert ("PATCH", "Home") in methods
assert ("DELETE", "Old") in methods assert ("DELETE", "Old") in methods
assert ("PATCH", "Credentials") not in methods assert ("PATCH", "Credentials") not in methods
assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out
def test_without_prune_pages_outside_the_source_are_kept( def test_without_prune_pages_outside_the_source_are_kept(
self, publish_wiki, environment, capsys self, publish_wiki, environment, capsys
): ):
(environment / ".publish").write_text("", encoding="utf-8") (environment / ".publish").write_text("", encoding="utf-8")
gitea = FakeGitea( gitea = FakeGitea(
publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")} publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")}
) )
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.wiki["Old"] == b64("hand written") assert gitea.wiki["Old"] == b64("hand written")
assert gitea.wiki["Home"] == b64("home v2") assert gitea.wiki["Home"] == b64("home v2")
assert "Credentials" in gitea.wiki assert "Credentials" in gitea.wiki
assert not [call for call in gitea.calls if call[0] == "DELETE"] assert not [call for call in gitea.calls if call[0] == "DELETE"]
out = capsys.readouterr().out out = capsys.readouterr().out
assert "1 created, 1 updated, 0 deleted" in out assert "1 created, 1 updated, 0 deleted" in out
assert "1 page(s) are in the wiki but not in" in out assert "1 page(s) are in the wiki but not in" in out
assert "prune" in out assert "prune" in out
def test_creates_every_page_when_the_wiki_was_never_written( def test_creates_every_page_when_the_wiki_was_never_written(
self, publish_wiki, environment self, publish_wiki, environment
): ):
gitea = FakeGitea(publish_wiki, wiki_initialised=False) gitea = FakeGitea(publish_wiki, wiki_initialised=False)
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert set(gitea.wiki) == {"Home", "Credentials"} assert set(gitea.wiki) == {"Home", "Credentials"}
assert [call[0] for call in gitea.calls].count("POST") == 2 assert [call[0] for call in gitea.calls].count("POST") == 2
def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment): def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment):
gitea = FakeGitea( gitea = FakeGitea(
publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")} publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")}
) )
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.writes == [] assert gitea.writes == []
def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys): def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys):
gitea = FakeGitea(publish_wiki) gitea = FakeGitea(publish_wiki)
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"} assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"}
output = capsys.readouterr() output = capsys.readouterr()
assert "wiki-token" not in output.out + output.err assert "wiki-token" not in output.out + output.err
def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing( def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing(
self, publish_wiki, environment self, publish_wiki, environment
): ):
gitea = FakeGitea(publish_wiki, has_wiki=False) gitea = FakeGitea(publish_wiki, has_wiki=False)
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
run_main(publish_wiki, gitea) run_main(publish_wiki, gitea)
assert caught.value.category == "configuration" assert caught.value.category == "configuration"
assert gitea.writes == [] assert gitea.writes == []
def test_an_empty_source_never_deletes_pages_even_with_prune( def test_an_empty_source_never_deletes_pages_even_with_prune(
self, publish_wiki, environment self, publish_wiki, environment
): ):
for path in environment.glob("*.md"): for path in environment.glob("*.md"):
path.unlink() path.unlink()
gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")}) gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")})
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"): with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
run_main(publish_wiki, gitea) run_main(publish_wiki, gitea)
assert gitea.calls == [] assert gitea.calls == []
assert gitea.wiki == {"Home": b64("keep me")} assert gitea.wiki == {"Home": b64("keep me")}
class TestPageIdentity: class TestPageIdentity:
"""A wiki that already has pages named with spaces, as Gitea stores them.""" """A wiki that already has pages named with spaces, as Gitea stores them."""
def test_a_file_named_with_a_space_updates_the_existing_page( def test_a_file_named_with_a_space_updates_the_existing_page(
self, publish_wiki, environment self, publish_wiki, environment
): ):
(environment / "Home.md").unlink() (environment / "Home.md").unlink()
(environment / "Credentials.md").unlink() (environment / "Credentials.md").unlink()
(environment / ".publish").write_text("", encoding="utf-8") (environment / ".publish").write_text("", encoding="utf-8")
(environment / "Home da.md").write_text("hjem v2", encoding="utf-8") (environment / "Home da.md").write_text("hjem v2", encoding="utf-8")
gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")}) gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")})
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.wiki == {"Home-da": b64("hjem v2")} assert gitea.wiki == {"Home-da": b64("hjem v2")}
assert [call[0] for call in gitea.calls].count("POST") == 0 assert [call[0] for call in gitea.calls].count("POST") == 0
assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls] assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls]
def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment): def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment):
(environment / "Home.md").unlink() (environment / "Home.md").unlink()
(environment / "Credentials.md").unlink() (environment / "Credentials.md").unlink()
(environment / "Home da.md").write_text("hjem", encoding="utf-8") (environment / "Home da.md").write_text("hjem", encoding="utf-8")
(environment / "_Sidebar.md").write_text("menu", encoding="utf-8") (environment / "_Sidebar.md").write_text("menu", encoding="utf-8")
gitea = FakeGitea( gitea = FakeGitea(
publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")} publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")}
) )
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.writes == [] assert gitea.writes == []
def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling( def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling(
self, publish_wiki, environment self, publish_wiki, environment
): ):
(environment / "Home.md").unlink() (environment / "Home.md").unlink()
(environment / "Credentials.md").unlink() (environment / "Credentials.md").unlink()
(environment / "Install ms.md").write_text("pasang", encoding="utf-8") (environment / "Install ms.md").write_text("pasang", encoding="utf-8")
gitea = FakeGitea( gitea = FakeGitea(
publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")} publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")}
) )
assert run_main(publish_wiki, gitea) == 0 assert run_main(publish_wiki, gitea) == 0
assert gitea.wiki == {"Install-ms": b64("pasang")} assert gitea.wiki == {"Install-ms": b64("pasang")}
def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki): def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki):
def listing(url, headers=None, **kwargs): def listing(url, headers=None, **kwargs):
return [ return [
{"title": "Whatever", "sub_url": "Real-Address"}, {"title": "Whatever", "sub_url": "Real-Address"},
{"title": "Only title"}, {"title": "Only title"},
{"title": "Encoded", "sub_url": "Caf%C3%A9"}, {"title": "Encoded", "sub_url": "Caf%C3%A9"},
] ]
keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing) keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing)
assert keys == {"Real-Address", "Only-title", "Café"} assert keys == {"Real-Address", "Only-title", "Café"}
class TestDescribeToken: class TestDescribeToken:
def test_recognises_a_gitea_token_shape(self, publish_wiki): def test_recognises_a_gitea_token_shape(self, publish_wiki):
description = publish_wiki.describe_token("a1b2c3d4e5" * 4) description = publish_wiki.describe_token("a1b2c3d4e5" * 4)
assert "40 characters" in description assert "40 characters" in description
assert "looks like a Gitea token" in description assert "looks like a Gitea token" in description
@pytest.mark.parametrize( @pytest.mark.parametrize(
("token", "expected"), ("token", "expected"),
[ [
("*******", "only mask characters"), ("*******", "only mask characters"),
("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"), ("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"),
('"' + "a" * 40 + '"', "quote character"), ('"' + "a" * 40 + '"', "quote character"),
("abc def", "whitespace"), ("abc def", "whitespace"),
], ],
) )
def test_names_the_likely_mistake_without_echoing_the_token( def test_names_the_likely_mistake_without_echoing_the_token(
self, publish_wiki, token, expected self, publish_wiki, token, expected
): ):
description = publish_wiki.describe_token(token) description = publish_wiki.describe_token(token)
assert expected in description assert expected in description
assert token not in description assert token not in description
class TestRejectedToken: class TestRejectedToken:
def test_rejection_adds_the_token_shape_but_not_the_token( def test_rejection_adds_the_token_shape_but_not_the_token(
self, publish_wiki, environment, monkeypatch self, publish_wiki, environment, monkeypatch
): ):
monkeypatch.setenv("WIKI_CREDENTIALS", "*******") monkeypatch.setenv("WIKI_CREDENTIALS", "*******")
def reject(url, method="GET", headers=None, body=None): def reject(url, method="GET", headers=None, body=None):
raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.") raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.")
publish_wiki.request_json = reject publish_wiki.request_json = reject
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
publish_wiki.main() publish_wiki.main()
message = str(caught.value) message = str(caught.value)
assert caught.value.category == "credentials" assert caught.value.category == "credentials"
assert "Token shape: 7 characters" in message assert "Token shape: 7 characters" in message
assert "only mask characters" in message assert "only mask characters" in message
assert "*******" not in message.split("Token shape:")[0] assert "*******" not in message.split("Token shape:")[0]
def test_other_categories_are_not_changed(self, publish_wiki, environment): def test_other_categories_are_not_changed(self, publish_wiki, environment):
def forbid(url, method="GET", headers=None, body=None): def forbid(url, method="GET", headers=None, body=None):
raise publish_wiki.WorkflowError("permissions", "Forbidden.") raise publish_wiki.WorkflowError("permissions", "Forbidden.")
publish_wiki.request_json = forbid publish_wiki.request_json = forbid
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
publish_wiki.main() publish_wiki.main()
assert str(caught.value) == "[permissions] Forbidden." assert str(caught.value) == "[permissions] Forbidden."
class TestRequestJson: class TestRequestJson:
@staticmethod @staticmethod
def raise_http_error(publish_wiki, monkeypatch, code, reason): def raise_http_error(publish_wiki, monkeypatch, code, reason):
def fake_urlopen(request, timeout): def fake_urlopen(request, timeout):
raise urllib.error.HTTPError( raise urllib.error.HTTPError(
request.full_url, code, reason, {}, io.BytesIO(b"secret detail") request.full_url, code, reason, {}, io.BytesIO(b"secret detail")
) )
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
@pytest.mark.parametrize( @pytest.mark.parametrize(
("code", "reason", "category"), ("code", "reason", "category"),
[ [
(401, "Unauthorized", "credentials"), (401, "Unauthorized", "credentials"),
(403, "Forbidden", "permissions"), (403, "Forbidden", "permissions"),
(404, "Not Found", "configuration"), (404, "Not Found", "configuration"),
(500, "Server Error", "api"), (500, "Server Error", "api"),
], ],
) )
def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category): def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category):
self.raise_http_error(publish_wiki, monkeypatch, code, reason) self.raise_http_error(publish_wiki, monkeypatch, code, reason)
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
publish_wiki.request_json(REPO_URL) publish_wiki.request_json(REPO_URL)
assert caught.value.category == category assert caught.value.category == category
assert f"HTTP {code}" in str(caught.value) assert f"HTTP {code}" in str(caught.value)
assert "secret detail" not in str(caught.value) assert "secret detail" not in str(caught.value)
def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch): def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch):
def fake_urlopen(request, timeout): def fake_urlopen(request, timeout):
raise urllib.error.URLError("no route") raise urllib.error.URLError("no route")
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen) monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
with pytest.raises(publish_wiki.WorkflowError) as caught: with pytest.raises(publish_wiki.WorkflowError) as caught:
publish_wiki.request_json(REPO_URL) publish_wiki.request_json(REPO_URL)
assert caught.value.category == "api" assert caught.value.category == "api"
class TestWorkflow: class TestWorkflow:
@pytest.fixture() @pytest.fixture()
def document(self): def document(self):
return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8")) return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8"))
def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document): def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document):
triggers = document[True] # PyYAML reads the key `on` as True triggers = document[True] # PyYAML reads the key `on` as True
assert triggers["push"]["branches"] == ["main"] assert triggers["push"]["branches"] == ["main"]
assert triggers["push"]["paths"] == ["wiki/**"] assert triggers["push"]["paths"] == ["wiki/**"]
assert "workflow_dispatch" in triggers assert "workflow_dispatch" in triggers
def test_job_has_read_only_permissions(self, document): def test_job_has_read_only_permissions(self, document):
assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"} assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"}
def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document): def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document):
text = PUBLISH_WORKFLOW.read_text(encoding="utf-8") text = PUBLISH_WORKFLOW.read_text(encoding="utf-8")
assert "secrets.TOKEN_FOR_GITEA }}" not in text assert "secrets.TOKEN_FOR_GITEA }}" not in text
publish_step = document["jobs"]["publish-wiki"]["steps"][-1] publish_step = document["jobs"]["publish-wiki"]["steps"][-1]
assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}" assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}"
assert "secrets" not in publish_step["run"] assert "secrets" not in publish_step["run"]
def test_the_workflow_has_one_source_and_runs_the_script_file(self, document): def test_the_workflow_has_one_source_and_runs_the_script_file(self, document):
assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists() assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists()
assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file() assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file()
steps = document["jobs"]["publish-wiki"]["steps"] steps = document["jobs"]["publish-wiki"]["steps"]
assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8") assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8")
assert steps[0]["uses"].startswith("actions/checkout@") assert steps[0]["uses"].startswith("actions/checkout@")
assert steps[1]["with"] == { assert steps[1]["with"] == {
"repository": "TirSystem/github-action", "repository": "TirSystem/github-action",
"ref": "v1", "ref": "v1",
"path": "shared-scripts", "path": "shared-scripts",
} }
assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py" assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py"
def test_this_repository_opts_in_and_prunes(): def test_this_repository_opts_in_and_prunes():
"""The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004).""" """The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004)."""
marker = REPO_ROOT / "wiki" / ".publish" marker = REPO_ROOT / "wiki" / ".publish"
assert marker.read_text(encoding="utf-8").strip() == "prune" assert marker.read_text(encoding="utf-8").strip() == "prune"
tools/publish_wiki.py
+306 -306
View File
@@ -1,306 +1,306 @@
"""Publish the pages in wiki/ to the Gitea wiki of the repository. """Publish the pages in wiki/ to the Gitea wiki of the repository.
Run by the scoped workflow publish-wiki.yml, which checks out the repository Run by the scoped workflow publish-wiki.yml, which checks out the repository
that owns the wiki and this repository, and runs this file (MIL-004 task 6). that owns the wiki and this repository, and runs this file (MIL-004 task 6).
A repository publishes only when it has opted in with wiki/.publish A repository publishes only when it has opted in with wiki/.publish
(ADR-0004); the word prune in that file lets the run delete wiki pages that (ADR-0004); the word prune in that file lets the run delete wiki pages that
are not in wiki/. It reads its settings from the environment: are not in wiki/. It reads its settings from the environment:
GITEA_API_URL Gitea API base, for example https://host/api/v1 GITEA_API_URL Gitea API base, for example https://host/api/v1
SOURCE_REPOSITORY owner/repo of the repository whose wiki is written SOURCE_REPOSITORY owner/repo of the repository whose wiki is written
WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI) WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI)
WIKI_DIR directory with the pages (default: wiki) WIKI_DIR directory with the pages (default: wiki)
Every failure is printed as ``ERROR: [category] message`` with the categories Every failure is printed as ``ERROR: [category] message`` with the categories
configuration, credentials, permissions and api; no secret value is printed. configuration, credentials, permissions and api; no secret value is printed.
""" """
import base64 import base64
import json import json
import os import os
import re import re
import sys import sys
import urllib.error import urllib.error
import urllib.parse import urllib.parse
import urllib.request import urllib.request
from pathlib import Path from pathlib import Path
PAGE_LIMIT = 50 PAGE_LIMIT = 50
REQUEST_TIMEOUT_SECONDS = 30 REQUEST_TIMEOUT_SECONDS = 30
COMMIT_MESSAGE = "Publish wiki from the repository" COMMIT_MESSAGE = "Publish wiki from the repository"
MARKER_NAME = ".publish" MARKER_NAME = ".publish"
PRUNE_WORD = "prune" PRUNE_WORD = "prune"
class WorkflowError(RuntimeError): class WorkflowError(RuntimeError):
"""A failure with a category. """A failure with a category.
configuration: a secret, a setting or the wiki source is missing or malformed. configuration: a secret, a setting or the wiki source is missing or malformed.
credentials: Gitea rejected the token (HTTP 401). credentials: Gitea rejected the token (HTTP 401).
permissions: the token is valid but not allowed (HTTP 403). permissions: the token is valid but not allowed (HTTP 403).
api: any other API or network failure. api: any other API or network failure.
""" """
def __init__(self, category, message): def __init__(self, category, message):
super().__init__(f"[{category}] {message}") super().__init__(f"[{category}] {message}")
self.category = category self.category = category
self.detail = message self.detail = message
def request_json(url, method="GET", headers=None, body=None): def request_json(url, method="GET", headers=None, body=None):
request = urllib.request.Request( request = urllib.request.Request(
url, url,
data=json.dumps(body).encode("utf-8") if body is not None else None, data=json.dumps(body).encode("utf-8") if body is not None else None,
headers=headers or {}, headers=headers or {},
method=method, method=method,
) )
try: try:
with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response: with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response:
content = response.read() content = response.read()
return json.loads(content) if content else None return json.loads(content) if content else None
except urllib.error.HTTPError as error: except urllib.error.HTTPError as error:
detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})" detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})"
if error.code == 401: if error.code == 401:
raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None
if error.code == 403: if error.code == 403:
raise WorkflowError( raise WorkflowError(
"permissions", f"{detail}. The token lacks a required permission." "permissions", f"{detail}. The token lacks a required permission."
) from None ) from None
if error.code == 404: if error.code == 404:
raise WorkflowError( raise WorkflowError(
"configuration", "configuration",
f"{detail}. The repository or page was not found, or the token " f"{detail}. The repository or page was not found, or the token "
"cannot see it.", "cannot see it.",
) from None ) from None
raise WorkflowError("api", detail) from None raise WorkflowError("api", detail) from None
except (urllib.error.URLError, TimeoutError) as error: except (urllib.error.URLError, TimeoutError) as error:
raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None
def parse_marker(text): def parse_marker(text):
"""Return True when the marker asks for pruning, False when it is empty.""" """Return True when the marker asks for pruning, False when it is empty."""
word = (text or "").strip() word = (text or "").strip()
if not word: if not word:
return False return False
if word == PRUNE_WORD: if word == PRUNE_WORD:
return True return True
raise WorkflowError( raise WorkflowError(
"configuration", "configuration",
f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.", f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.",
) )
def parse_gitea_token(raw_token): def parse_gitea_token(raw_token):
token = (raw_token or "").strip() token = (raw_token or "").strip()
if not token: if not token:
raise WorkflowError( raise WorkflowError(
"configuration", "configuration",
"TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of " "TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of "
"the organization of this repository.", "the organization of this repository.",
) )
return token return token
def describe_token(token): def describe_token(token):
"""Describe the shape of a token without revealing any of its characters.""" """Describe the shape of a token without revealing any of its characters."""
facts = [f"{len(token)} characters"] facts = [f"{len(token)} characters"]
if re.fullmatch(r"[0-9a-f]{40}", token): if re.fullmatch(r"[0-9a-f]{40}", token):
facts.append("looks like a Gitea token (40 hexadecimal characters)") facts.append("looks like a Gitea token (40 hexadecimal characters)")
return ", ".join(facts) return ", ".join(facts)
facts.append("a Gitea token has 40 hexadecimal characters") facts.append("a Gitea token has 40 hexadecimal characters")
if set(token) <= {"*", "x", "X", "•"}: if set(token) <= {"*", "x", "X", "•"}:
facts.append("only mask characters: the masked text was stored") facts.append("only mask characters: the masked text was stored")
if "=" in token: if "=" in token:
facts.append("contains '=': the secret name may be part of the value") facts.append("contains '=': the secret name may be part of the value")
if re.search(r"[\"'`]", token): if re.search(r"[\"'`]", token):
facts.append("contains a quote character") facts.append("contains a quote character")
if re.search(r"\s", token): if re.search(r"\s", token):
facts.append("contains whitespace") facts.append("contains whitespace")
return ", ".join(facts) return ", ".join(facts)
def split_repository(full_name): def split_repository(full_name):
owner, separator, repo = (full_name or "").partition("/") owner, separator, repo = (full_name or "").partition("/")
if not separator or not owner or not repo: if not separator or not owner or not repo:
raise WorkflowError("configuration", "Could not determine the Gitea repository.") raise WorkflowError("configuration", "Could not determine the Gitea repository.")
return owner, repo return owner, repo
def page_key(title): def page_key(title):
"""Return the identity of a page: Gitea stores a space as a hyphen. """Return the identity of a page: Gitea stores a space as a hyphen.
The page "Home da" is the file Home-da.md and its address is Home-da, so The page "Home da" is the file Home-da.md and its address is Home-da, so
both spellings must name the same page. both spellings must name the same page.
""" """
return str(title).replace(" ", "-") return str(title).replace(" ", "-")
def page_title(key): def page_title(key):
"""Return the title to send to Gitea for a page key. """Return the title to send to Gitea for a page key.
A hyphen in a title is a literal hyphen to Gitea, which then stores the page A hyphen in a title is a literal hyphen to Gitea, which then stores the page
as Home-da.-.md and addresses it as Home%2Dda, so a link to Home-da finds as Home-da.-.md and addresses it as Home%2Dda, so a link to Home-da finds
nothing. A space is stored as a hyphen (Home-da.md) and addressed as Home-da. nothing. A space is stored as a hyphen (Home-da.md) and addressed as Home-da.
""" """
return key.replace("-", " ") return key.replace("-", " ")
def read_pages(wiki_dir): def read_pages(wiki_dir):
"""Return {page key: content} for every Markdown file in wiki_dir.""" """Return {page key: content} for every Markdown file in wiki_dir."""
pages = {} pages = {}
for path in sorted(wiki_dir.glob("*.md")): for path in sorted(wiki_dir.glob("*.md")):
key = page_key(path.stem) key = page_key(path.stem)
if key in pages: if key in pages:
raise WorkflowError( raise WorkflowError(
"configuration", "configuration",
f"Two files in {wiki_dir} name the wiki page {key}; a space and " f"Two files in {wiki_dir} name the wiki page {key}; a space and "
"a hyphen are the same in a page name.", "a hyphen are the same in a page name.",
) )
pages[key] = path.read_bytes() pages[key] = path.read_bytes()
if not pages: if not pages:
# Publishing an empty directory would delete the whole wiki. # Publishing an empty directory would delete the whole wiki.
raise WorkflowError( raise WorkflowError(
"configuration", f"The wiki source directory {wiki_dir} has no Markdown pages." "configuration", f"The wiki source directory {wiki_dir} has no Markdown pages."
) )
return pages return pages
def list_wiki_titles(wiki_url, headers, send): def list_wiki_titles(wiki_url, headers, send):
"""Return the keys of the pages that exist in the wiki. """Return the keys of the pages that exist in the wiki.
The key is the address Gitea reports (sub_url), or the title with its The key is the address Gitea reports (sub_url), or the title with its
spaces replaced by hyphens. spaces replaced by hyphens.
Gitea answers 404 for a wiki that has never been written to; that is an Gitea answers 404 for a wiki that has never been written to; that is an
empty wiki, not a failure. empty wiki, not a failure.
""" """
titles = set() titles = set()
page = 1 page = 1
while True: while True:
url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}" url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}"
try: try:
batch = send(url, headers=headers) batch = send(url, headers=headers)
except WorkflowError as error: except WorkflowError as error:
if error.category == "configuration" and page == 1: if error.category == "configuration" and page == 1:
return titles return titles
raise raise
if not isinstance(batch, list): if not isinstance(batch, list):
raise WorkflowError("api", "Gitea returned an invalid wiki page list.") raise WorkflowError("api", "Gitea returned an invalid wiki page list.")
titles.update( titles.update(
urllib.parse.unquote(str(item.get("sub_url") or "")) urllib.parse.unquote(str(item.get("sub_url") or ""))
or page_key(item.get("title", "")) or page_key(item.get("title", ""))
for item in batch for item in batch
) )
if len(batch) < PAGE_LIMIT: if len(batch) < PAGE_LIMIT:
return titles return titles
page += 1 page += 1
def read_wiki_page(wiki_url, title, headers, send): def read_wiki_page(wiki_url, title, headers, send):
quoted = urllib.parse.quote(title, safe="") quoted = urllib.parse.quote(title, safe="")
page = send(f"{wiki_url}/page/{quoted}", headers=headers) page = send(f"{wiki_url}/page/{quoted}", headers=headers)
return base64.b64decode((page or {}).get("content_base64", "")) return base64.b64decode((page or {}).get("content_base64", ""))
def publish(pages, wiki_url, headers, send, prune): def publish(pages, wiki_url, headers, send, prune):
"""Create and update the pages; delete the others only when prune is set. """Create and update the pages; delete the others only when prune is set.
Returns the titles created, updated and deleted, and the titles that were Returns the titles created, updated and deleted, and the titles that were
left in the wiki because pruning is off. left in the wiki because pruning is off.
""" """
existing = list_wiki_titles(wiki_url, headers, send) existing = list_wiki_titles(wiki_url, headers, send)
current = { current = {
title: read_wiki_page(wiki_url, title, headers, send) title: read_wiki_page(wiki_url, title, headers, send)
for title in sorted(existing & pages.keys()) for title in sorted(existing & pages.keys())
} }
result = {"created": [], "updated": [], "deleted": [], "kept": []} result = {"created": [], "updated": [], "deleted": [], "kept": []}
for title, content in pages.items(): for title, content in pages.items():
body = { body = {
"title": page_title(title), "title": page_title(title),
"content_base64": base64.b64encode(content).decode("ascii"), "content_base64": base64.b64encode(content).decode("ascii"),
"message": COMMIT_MESSAGE, "message": COMMIT_MESSAGE,
} }
if title not in existing: if title not in existing:
send(f"{wiki_url}/new", method="POST", headers=headers, body=body) send(f"{wiki_url}/new", method="POST", headers=headers, body=body)
result["created"].append(title) result["created"].append(title)
elif current[title] != content: elif current[title] != content:
quoted = urllib.parse.quote(title, safe="") quoted = urllib.parse.quote(title, safe="")
send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body) send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body)
result["updated"].append(title) result["updated"].append(title)
for title in sorted(existing - pages.keys()): for title in sorted(existing - pages.keys()):
if not prune: if not prune:
result["kept"].append(title) result["kept"].append(title)
continue continue
quoted = urllib.parse.quote(title, safe="") quoted = urllib.parse.quote(title, safe="")
send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers) send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers)
result["deleted"].append(title) result["deleted"].append(title)
return result return result
def main(): def main():
wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki")) wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki"))
owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY")) owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY"))
marker = wiki_dir / MARKER_NAME marker = wiki_dir / MARKER_NAME
if not marker.is_file(): if not marker.is_file():
print( print(
f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; " f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; "
"wiki publishing is not enabled for this repository." "wiki publishing is not enabled for this repository."
) )
return 0 return 0
prune = parse_marker(marker.read_text(encoding="utf-8")) prune = parse_marker(marker.read_text(encoding="utf-8"))
pages = read_pages(wiki_dir) pages = read_pages(wiki_dir)
token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS")) token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS"))
api_url = os.environ["GITEA_API_URL"].rstrip("/") api_url = os.environ["GITEA_API_URL"].rstrip("/")
repository_url = ( repository_url = (
f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/" f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/"
f"{urllib.parse.quote(repo, safe='')}" f"{urllib.parse.quote(repo, safe='')}"
) )
headers = { headers = {
"Authorization": f"token {token}", "Authorization": f"token {token}",
"Accept": "application/json", "Accept": "application/json",
"Content-Type": "application/json", "Content-Type": "application/json",
} }
try: try:
repository = request_json(repository_url, headers=headers) repository = request_json(repository_url, headers=headers)
if isinstance(repository, dict) and repository.get("has_wiki") is False: if isinstance(repository, dict) and repository.get("has_wiki") is False:
raise WorkflowError( raise WorkflowError(
"configuration", "configuration",
f"The wiki is not enabled on {owner}/{repo}. Enable it in the " f"The wiki is not enabled on {owner}/{repo}. Enable it in the "
"repository settings.", "repository settings.",
) )
print("Preflight passed: Gitea accepted the token and the wiki is enabled.") print("Preflight passed: Gitea accepted the token and the wiki is enabled.")
result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune) result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune)
except WorkflowError as error: except WorkflowError as error:
if error.category == "credentials": if error.category == "credentials":
raise WorkflowError( raise WorkflowError(
"credentials", f"{error.detail} Token shape: {describe_token(token)}." "credentials", f"{error.detail} Token shape: {describe_token(token)}."
) from None ) from None
raise raise
print( print(
f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: " f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: "
f"{len(result['created'])} created, {len(result['updated'])} updated, " f"{len(result['created'])} created, {len(result['updated'])} updated, "
f"{len(result['deleted'])} deleted." f"{len(result['deleted'])} deleted."
) )
if result["kept"]: if result["kept"]:
print( print(
f"{len(result['kept'])} page(s) are in the wiki but not in " f"{len(result['kept'])} page(s) are in the wiki but not in "
f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in " f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in "
f"{wiki_dir}/{MARKER_NAME} to delete them." f"{wiki_dir}/{MARKER_NAME} to delete them."
) )
return 0 return 0
if __name__ == "__main__": if __name__ == "__main__":
try: try:
sys.exit(main()) sys.exit(main())
except WorkflowError as error: except WorkflowError as error:
print(f"ERROR: {error}", file=sys.stderr) print(f"ERROR: {error}", file=sys.stderr)
sys.exit(1) sys.exit(1)
wiki/Onboarding.da.md
+2 -2
View File
@@ -1,4 +1,4 @@
<!-- translation of Onboarding.md, source sha256 1d431898754e --> <!-- translation of Onboarding.md, source sha256 0b11defd563a -->
> Dette er en oversættelse af [`Onboarding`](Onboarding). Den engelske side er autoritativ, og ændringer her bliver overskrevet, næste gang wikien publiceres. > Dette er en oversættelse af [`Onboarding`](Onboarding). Den engelske side er autoritativ, og ændringer her bliver overskrevet, næste gang wikien publiceres.
@@ -35,7 +35,7 @@ Leveringen er automatisk, så et repository behøver ingen kopi af workflowet. S
Workflowet **Publish wiki** kopierer Markdown-siderne i `wiki/` til repositoryets wiki, når en ændring af `wiki/` flettes til `main`. Det leveres som synkroniseringsworkflowet, men skriver kun, hvor et repository har meldt sig til. Workflowet **Publish wiki** kopierer Markdown-siderne i `wiki/` til repositoryets wiki, når en ændring af `wiki/` flettes til `main`. Det leveres som synkroniseringsworkflowet, men skriver kun, hvor et repository har meldt sig til.
1. Aktivér wikien i repositoryets indstillinger. 1. Aktivér wikien i repositoryets indstillinger.
2. Tilføj Markdown-sider til `wiki/`. Filnavnet er sidens navn: `Home.md` er forsiden. 2. Tilføj Markdown-sider til `wiki/`. Filnavnet er sidens navn: `Home.md` er forsiden. En bindestreg i navnet er et mellemrum i sidens navn: `Home-da.md` er siden "Home da", og et link til den er `[Dansk](Home-da)`.
3. Tilføj markøren `wiki/.publish`. Lad den være tom for kun at oprette og opdatere sider. Skriv det ene ord `prune` i den for også at slette wiki-sider, der ikke findes i `wiki/`, så wikien altid er lig med mappen. 3. Tilføj markøren `wiki/.publish`. Lad den være tom for kun at oprette og opdatere sider. Skriv det ene ord `prune` i den for også at slette wiki-sider, der ikke findes i `wiki/`, så wikien altid er lig med mappen.
4. Gem `TOKEN_FOR_GITEA_WIKI` som en secret i repositoryets organisation med det scope, der står i [credentials.md](Credentials.da). 4. Gem `TOKEN_FOR_GITEA_WIKI` som en secret i repositoryets organisation med det scope, der står i [credentials.md](Credentials.da).
5. Flet ændringen til `main`. Loggen skal slutte med `Published <n> page(s) to the wiki of <owner>/<repo>`. 5. Flet ændringen til `main`. Loggen skal slutte med `Published <n> page(s) to the wiki of <owner>/<repo>`.
wiki/Onboarding.md
+2 -1
View File
@@ -57,7 +57,8 @@ like the sync workflow, but it writes only where a repository opts in.
1. Enable the wiki in the repository settings. 1. Enable the wiki in the repository settings.
2. Add Markdown pages to `wiki/`. The file name is the page name: `Home.md` is 2. Add Markdown pages to `wiki/`. The file name is the page name: `Home.md` is
the home page. the home page. A hyphen in the name is a space in the page name: `Home-da.md`
is the page "Home da", and a link to it is `[Dansk](https://git.tirsystem.com/TirSystem/github-action/src/branch/main/guides/Home-da)`.
3. Add the marker `wiki/.publish`. Leave it empty to create and update pages 3. Add the marker `wiki/.publish`. Leave it empty to create and update pages
only. Put the single word `prune` in it to also delete wiki pages that are only. Put the single word `prune` in it to also delete wiki pages that are
not in `wiki/`, so the wiki always equals the folder. not in `wiki/`, so the wiki always equals the folder.
wiki/Onboarding.ms.md
+2 -2
View File
@@ -1,4 +1,4 @@
<!-- translation of Onboarding.md, source sha256 1d431898754e --> <!-- translation of Onboarding.md, source sha256 0b11defd563a -->
> Ini ialah terjemahan [`Onboarding`](Onboarding). Halaman bahasa Inggeris ialah teks yang sah, dan perubahan di sini akan ditimpa pada penerbitan wiki yang seterusnya. > Ini ialah terjemahan [`Onboarding`](Onboarding). Halaman bahasa Inggeris ialah teks yang sah, dan perubahan di sini akan ditimpa pada penerbitan wiki yang seterusnya.
@@ -35,7 +35,7 @@ Penghantaran adalah automatik, jadi repositori tidak memerlukan salinan aliran k
Aliran kerja **Publish wiki** menyalin halaman Markdown dalam `wiki/` ke wiki repositori apabila perubahan pada `wiki/` digabungkan ke `main`. Ia dihantar seperti aliran kerja penyegerakan, tetapi hanya menulis di tempat repositori telah memilih untuk menyertainya. Aliran kerja **Publish wiki** menyalin halaman Markdown dalam `wiki/` ke wiki repositori apabila perubahan pada `wiki/` digabungkan ke `main`. Ia dihantar seperti aliran kerja penyegerakan, tetapi hanya menulis di tempat repositori telah memilih untuk menyertainya.
1. Aktifkan wiki dalam tetapan repositori. 1. Aktifkan wiki dalam tetapan repositori.
2. Tambah halaman Markdown ke `wiki/`. Nama fail ialah nama halaman: `Home.md` ialah halaman utama. 2. Tambah halaman Markdown ke `wiki/`. Nama fail ialah nama halaman: `Home.md` ialah halaman utama. Tanda sempang dalam nama ialah ruang dalam nama halaman: `Home-da.md` ialah halaman "Home da", dan pautan kepadanya ialah `[Dansk](Home-da)`.
3. Tambah penanda `wiki/.publish`. Biarkan kosong untuk mencipta dan mengemas kini halaman sahaja. Letakkan satu perkataan `prune` di dalamnya untuk turut memadam halaman wiki yang tiada dalam `wiki/`, supaya wiki sentiasa sama dengan folder itu. 3. Tambah penanda `wiki/.publish`. Biarkan kosong untuk mencipta dan mengemas kini halaman sahaja. Letakkan satu perkataan `prune` di dalamnya untuk turut memadam halaman wiki yang tiada dalam `wiki/`, supaya wiki sentiasa sama dengan folder itu.
4. Simpan `TOKEN_FOR_GITEA_WIKI` sebagai secret organisasi repositori, dengan skop yang tertera dalam [credentials.md](Credentials.ms). 4. Simpan `TOKEN_FOR_GITEA_WIKI` sebagai secret organisasi repositori, dengan skop yang tertera dalam [credentials.md](Credentials.ms).
5. Gabungkan perubahan ke `main`. Log mesti berakhir dengan `Published <n> page(s) to the wiki of <owner>/<repo>`. 5. Gabungkan perubahan ke `main`. Log mesti berakhir dengan `Published <n> page(s) to the wiki of <owner>/<repo>`.