Merge pull request 'Restore LF line endings and document hyphenated page names' (#121) from fix-lf-endings into main
This commit was merged in pull request #121.
This commit is contained in:
+100
-97
@@ -1,97 +1,100 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to the shared workflows of this repository. The format and
|
||||
the version rules are those of the release process (`guides/release-process.md`,
|
||||
ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH`
|
||||
and a moving major tag `vMAJOR`.
|
||||
|
||||
Each release lists its changes in these groups, in this order:
|
||||
|
||||
- **Breaking**: a renamed or removed workflow, input, secret name or output; a
|
||||
raised default tool version or changed default that can turn a green
|
||||
consumer red; a changed meaning of pass or fail. A release with a Breaking
|
||||
entry has a new major version.
|
||||
- **Added**: a new optional input, workflow or capability.
|
||||
- **Changed**: a change with no visible effect for existing callers.
|
||||
- **Fixed**: a fix.
|
||||
|
||||
`python tools/changelog.py check` verifies the file, and
|
||||
`python tools/changelog.py notes <version>` prints the text of a release page.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
|
||||
- Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a
|
||||
runner for the host's default limit.
|
||||
- `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the
|
||||
moving tag `v4`. No change for callers.
|
||||
- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:`
|
||||
blocks of this repository's own `sync-milestones.yml` call
|
||||
`framework/scripts/ci-publish-milestones.sh`. No change for callers of the
|
||||
shared workflows.
|
||||
|
||||
### Fixed
|
||||
|
||||
- `publish-wiki` sends a page title with spaces instead of hyphens. Gitea keeps a
|
||||
hyphen in a title literal and stored `Home-da` as `Home-da.-.md`, addressed as
|
||||
`Home%2Dda`, so links to `Home-da` led to the page list. A new page is now
|
||||
stored as `Home-da.md` and addressed as `Home-da`. Pages already created with
|
||||
the old title must be deleted by hand (the publisher does not prune them).
|
||||
|
||||
## [1.1.0] - 2026-10-11
|
||||
|
||||
### Added
|
||||
|
||||
- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from
|
||||
which the workflow fetches its script `tools/quality-checks.sh`.
|
||||
|
||||
### Changed
|
||||
|
||||
- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of
|
||||
embedding about 100 lines of shell; the checks, messages and results are the
|
||||
same.
|
||||
|
||||
## [1.0.1] - 2026-10-11
|
||||
|
||||
### Changed
|
||||
|
||||
- Sync GitHub mirror metadata: a repository without a GitHub push mirror still
|
||||
succeeds, and the log now ends with the line `Skipped`.
|
||||
- The scoped workflows fetch their scripts at the tag `v1` instead of `main`,
|
||||
so a merge to `main` reaches consumers only when it is released.
|
||||
|
||||
## [1.0.0] - 2026-10-10
|
||||
|
||||
First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The
|
||||
scoped workflows are delivered by the instance from `main` and cannot be pinned.
|
||||
|
||||
### Added
|
||||
|
||||
- Workflow `release.yml`: started by hand with a version, it releases only the
|
||||
tip of `main` whose validation run is green and whose changelog entry exists,
|
||||
creates the version tag, the release page and the major tag, and does nothing
|
||||
but check until `dry_run` is set to `false`.
|
||||
- Scoped workflow `sync-github-metadata.yml`: copies the description and topics
|
||||
of a Gitea repository to its GitHub push mirror on every push to `main`, by
|
||||
hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped
|
||||
successfully.
|
||||
- A credential preflight before any change, and failure messages that name
|
||||
their category (`configuration`, `credentials`, `permissions`, `api`).
|
||||
- Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki
|
||||
when a repository has opted in with `wiki/.publish`; the word `prune` in that
|
||||
file also deletes pages that are not in `wiki/`.
|
||||
- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck`
|
||||
and `shfmt` on the tracked Python and shell files, with the inputs
|
||||
`languages`, `working-directory` and one pinned version per tool.
|
||||
- Guides for onboarding a repository, workflow credentials, troubleshooting,
|
||||
reading quality-check results and releasing, and the same pages on the wiki in
|
||||
English, Danish and Bahasa Malaysia.
|
||||
|
||||
### Changed
|
||||
|
||||
- The scripts of the scoped workflows are files under `tools/`, fetched by a
|
||||
checkout of this repository, instead of Python embedded in the YAML.
|
||||
- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`,
|
||||
`PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret
|
||||
`CREDENTIALS_FOR_GITHUB`, which is no longer read.
|
||||
# Changelog
|
||||
|
||||
All notable changes to the shared workflows of this repository. The format and
|
||||
the version rules are those of the release process (`guides/release-process.md`,
|
||||
ADR-0006): one version for the whole repository, tags `vMAJOR.MINOR.PATCH`
|
||||
and a moving major tag `vMAJOR`.
|
||||
|
||||
Each release lists its changes in these groups, in this order:
|
||||
|
||||
- **Breaking**: a renamed or removed workflow, input, secret name or output; a
|
||||
raised default tool version or changed default that can turn a green
|
||||
consumer red; a changed meaning of pass or fail. A release with a Breaking
|
||||
entry has a new major version.
|
||||
- **Added**: a new optional input, workflow or capability.
|
||||
- **Changed**: a change with no visible effect for existing callers.
|
||||
- **Fixed**: a fix.
|
||||
|
||||
`python tools/changelog.py check` verifies the file, and
|
||||
`python tools/changelog.py notes <version>` prints the text of a release page.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
|
||||
- Every job has a `timeout-minutes` (5 to 15), so a hung step no longer holds a
|
||||
runner for the host's default limit.
|
||||
- `actions/checkout` is referenced by the exact tag `v4.2.2` instead of the
|
||||
moving tag `v4`. No change for callers.
|
||||
- The framework pin moves from `v0.5.2` to `v0.6.2`, and the two long `run:`
|
||||
blocks of this repository's own `sync-milestones.yml` call
|
||||
`framework/scripts/ci-publish-milestones.sh`. No change for callers of the
|
||||
shared workflows.
|
||||
|
||||
### Fixed
|
||||
|
||||
- `publish-wiki` sends a page title with spaces instead of hyphens. Gitea keeps a
|
||||
hyphen in a title literal and stored `Home-da` as `Home-da.-.md`, addressed as
|
||||
`Home%2Dda`, so links to `Home-da` led to the page list. A new page is now
|
||||
stored as `Home-da.md` and addressed as `Home-da`. Pages already created with
|
||||
the old title must be deleted by hand (the publisher does not prune them).
|
||||
- The onboarding guide and its wiki pages (English, Danish, Bahasa Malaysia) say
|
||||
that a hyphen in a file name is a space in the page name, and how to link to
|
||||
it.
|
||||
|
||||
## [1.1.0] - 2026-10-11
|
||||
|
||||
### Added
|
||||
|
||||
- Input `tools-ref` of `quality-checks.yml` (default `v1`): the tag or branch from
|
||||
which the workflow fetches its script `tools/quality-checks.sh`.
|
||||
|
||||
### Changed
|
||||
|
||||
- `quality-checks.yml` runs the script file `tools/quality-checks.sh` instead of
|
||||
embedding about 100 lines of shell; the checks, messages and results are the
|
||||
same.
|
||||
|
||||
## [1.0.1] - 2026-10-11
|
||||
|
||||
### Changed
|
||||
|
||||
- Sync GitHub mirror metadata: a repository without a GitHub push mirror still
|
||||
succeeds, and the log now ends with the line `Skipped`.
|
||||
- The scoped workflows fetch their scripts at the tag `v1` instead of `main`,
|
||||
so a merge to `main` reaches consumers only when it is released.
|
||||
|
||||
## [1.0.0] - 2026-10-10
|
||||
|
||||
First release. The reusable workflow can be pinned at `v1` or `v1.0.0`. The
|
||||
scoped workflows are delivered by the instance from `main` and cannot be pinned.
|
||||
|
||||
### Added
|
||||
|
||||
- Workflow `release.yml`: started by hand with a version, it releases only the
|
||||
tip of `main` whose validation run is green and whose changelog entry exists,
|
||||
creates the version tag, the release page and the major tag, and does nothing
|
||||
but check until `dry_run` is set to `false`.
|
||||
- Scoped workflow `sync-github-metadata.yml`: copies the description and topics
|
||||
of a Gitea repository to its GitHub push mirror on every push to `main`, by
|
||||
hand and daily at 03:17 UTC. A repository without a GitHub mirror is skipped
|
||||
successfully.
|
||||
- A credential preflight before any change, and failure messages that name
|
||||
their category (`configuration`, `credentials`, `permissions`, `api`).
|
||||
- Scoped workflow `publish-wiki.yml`: publishes `wiki/` to the repository wiki
|
||||
when a repository has opted in with `wiki/.publish`; the word `prune` in that
|
||||
file also deletes pages that are not in `wiki/`.
|
||||
- Reusable workflow `quality-checks.yml`: `ruff`, `mypy --strict`, `shellcheck`
|
||||
and `shfmt` on the tracked Python and shell files, with the inputs
|
||||
`languages`, `working-directory` and one pinned version per tool.
|
||||
- Guides for onboarding a repository, workflow credentials, troubleshooting,
|
||||
reading quality-check results and releasing, and the same pages on the wiki in
|
||||
English, Danish and Bahasa Malaysia.
|
||||
|
||||
### Changed
|
||||
|
||||
- The scripts of the scoped workflows are files under `tools/`, fetched by a
|
||||
checkout of this repository, instead of Python embedded in the YAML.
|
||||
- Secrets hold the bare token as clear text: `TOKEN_FOR_GITEA`,
|
||||
`PAT_FOR_GITHUB` and `TOKEN_FOR_GITEA_WIKI`. They replace the JSON secret
|
||||
`CREDENTIALS_FOR_GITHUB`, which is no longer read.
|
||||
|
||||
@@ -51,7 +51,8 @@ like the sync workflow, but it writes only where a repository opts in.
|
||||
|
||||
1. Enable the wiki in the repository settings.
|
||||
2. Add Markdown pages to `wiki/`. The file name is the page name: `Home.md` is
|
||||
the home page.
|
||||
the home page. A hyphen in the name is a space in the page name: `Home-da.md`
|
||||
is the page "Home da", and a link to it is `[Dansk](Home-da)`.
|
||||
3. Add the marker `wiki/.publish`. Leave it empty to create and update pages
|
||||
only. Put the single word `prune` in it to also delete wiki pages that are
|
||||
not in `wiki/`, so the wiki always equals the folder.
|
||||
|
||||
+491
-491
@@ -1,491 +1,491 @@
|
||||
"""Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow."""
|
||||
|
||||
import base64
|
||||
import io
|
||||
import urllib.error
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module
|
||||
|
||||
API = "https://git.example.test/api/v1"
|
||||
REPO_URL = f"{API}/repos/TirSystem/repo"
|
||||
WIKI_URL = f"{REPO_URL}/wiki"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def publish_wiki():
|
||||
return load_publish_module()
|
||||
|
||||
|
||||
def b64(text: str) -> str:
|
||||
return base64.b64encode(text.encode()).decode()
|
||||
|
||||
|
||||
class TestParseMarker:
|
||||
@pytest.mark.parametrize(
|
||||
("text", "expected"),
|
||||
[("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)],
|
||||
)
|
||||
def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected):
|
||||
assert publish_wiki.parse_marker(text) is expected
|
||||
|
||||
@pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"])
|
||||
def test_rejects_anything_else(self, publish_wiki, text):
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.parse_marker(text)
|
||||
assert caught.value.category == "configuration"
|
||||
assert "prune" in str(caught.value)
|
||||
|
||||
|
||||
class TestParseGiteaToken:
|
||||
def test_returns_the_stripped_token(self, publish_wiki):
|
||||
assert publish_wiki.parse_gitea_token(" abc \n") == "abc"
|
||||
|
||||
@pytest.mark.parametrize("raw", [None, "", " "])
|
||||
def test_rejects_missing_or_empty_token(self, publish_wiki, raw):
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.parse_gitea_token(raw)
|
||||
assert caught.value.category == "configuration"
|
||||
assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value)
|
||||
|
||||
|
||||
class TestReadPages:
|
||||
def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path):
|
||||
(tmp_path / "Home.md").write_bytes(b"home")
|
||||
(tmp_path / "Home.da.md").write_bytes(b"hjem")
|
||||
(tmp_path / ".publish").write_bytes(b"prune")
|
||||
(tmp_path / "Notes.txt").write_bytes(b"ignored")
|
||||
assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"}
|
||||
|
||||
def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped(
|
||||
self, publish_wiki, tmp_path
|
||||
):
|
||||
(tmp_path / ".publish").write_text("prune", encoding="utf-8")
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
|
||||
publish_wiki.read_pages(tmp_path)
|
||||
|
||||
|
||||
class TestPageTitle:
|
||||
@pytest.mark.parametrize(
|
||||
("key", "title"), [("Home", "Home"), ("Home-da", "Home da"), ("Getting-Started-ms", "Getting Started ms")]
|
||||
)
|
||||
def test_a_hyphen_is_sent_as_a_space(self, publish_wiki, key, title):
|
||||
# Gitea keeps a hyphen in a title literal (Home-da.-.md, address Home%2Dda).
|
||||
assert publish_wiki.page_title(key) == title
|
||||
|
||||
|
||||
class TestPageKey:
|
||||
@pytest.mark.parametrize(
|
||||
("name", "key"),
|
||||
[("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"),
|
||||
("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")],
|
||||
)
|
||||
def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key):
|
||||
assert publish_wiki.page_key(name) == key
|
||||
|
||||
def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path):
|
||||
(tmp_path / "Home da.md").write_bytes(b"hjem")
|
||||
(tmp_path / "_Sidebar.md").write_bytes(b"menu")
|
||||
assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"}
|
||||
|
||||
def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path):
|
||||
(tmp_path / "Home da.md").write_bytes(b"one")
|
||||
(tmp_path / "Home-da.md").write_bytes(b"two")
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"):
|
||||
publish_wiki.read_pages(tmp_path)
|
||||
|
||||
|
||||
class FakeGitea:
|
||||
"""Answers like the Gitea repository and wiki endpoints."""
|
||||
|
||||
def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True):
|
||||
self.error = publish_wiki.WorkflowError
|
||||
self.wiki = dict(wiki or {})
|
||||
self.has_wiki = has_wiki
|
||||
self.wiki_initialised = wiki_initialised
|
||||
self.calls = []
|
||||
|
||||
def __call__(self, url, method="GET", headers=None, body=None):
|
||||
self.calls.append((method, url, body, headers))
|
||||
if url == REPO_URL:
|
||||
return {"has_wiki": self.has_wiki}
|
||||
if url.startswith(f"{WIKI_URL}/pages"):
|
||||
if not self.wiki_initialised:
|
||||
raise self.error("configuration", "404")
|
||||
return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki]
|
||||
if method == "POST" and url == f"{WIKI_URL}/new":
|
||||
self.wiki[body["title"].replace(" ", "-")] = body["content_base64"]
|
||||
return None
|
||||
title = url.rsplit("/", 1)[-1]
|
||||
if method == "GET":
|
||||
return {"title": title, "content_base64": self.wiki[title]}
|
||||
if method == "PATCH":
|
||||
assert " " in body["title"] or "-" not in body["title"], body["title"]
|
||||
self.wiki[title] = body["content_base64"]
|
||||
return None
|
||||
if method == "DELETE":
|
||||
del self.wiki[title]
|
||||
return None
|
||||
raise AssertionError(f"unexpected request {method} {url}")
|
||||
|
||||
@property
|
||||
def writes(self):
|
||||
return [call for call in self.calls if call[0] != "GET"]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def environment(monkeypatch, tmp_path):
|
||||
wiki_dir = tmp_path / "wiki"
|
||||
wiki_dir.mkdir()
|
||||
(wiki_dir / "Home.md").write_text("home v2", encoding="utf-8")
|
||||
(wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8")
|
||||
(wiki_dir / ".publish").write_text("prune\n", encoding="utf-8")
|
||||
monkeypatch.setenv("WIKI_DIR", str(wiki_dir))
|
||||
monkeypatch.setenv("GITEA_API_URL", f"{API}/")
|
||||
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
|
||||
monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token")
|
||||
return wiki_dir
|
||||
|
||||
|
||||
def run_main(publish_wiki, gitea):
|
||||
publish_wiki.request_json = gitea
|
||||
return publish_wiki.main()
|
||||
|
||||
|
||||
class TestOptIn:
|
||||
def test_a_repository_without_the_marker_is_never_written_to(
|
||||
self, publish_wiki, environment, capsys
|
||||
):
|
||||
(environment / ".publish").unlink()
|
||||
gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")})
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.calls == []
|
||||
assert "wiki publishing is not enabled" in capsys.readouterr().out
|
||||
assert gitea.wiki == {"Old": b64("hand written")}
|
||||
|
||||
def test_a_repository_without_the_marker_does_not_need_the_secret(
|
||||
self, publish_wiki, environment, monkeypatch
|
||||
):
|
||||
(environment / ".publish").unlink()
|
||||
monkeypatch.setenv("WIKI_CREDENTIALS", "")
|
||||
assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0
|
||||
|
||||
def test_a_repository_without_a_wiki_directory_is_skipped(
|
||||
self, publish_wiki, monkeypatch, tmp_path
|
||||
):
|
||||
monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing"))
|
||||
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
|
||||
gitea = FakeGitea(publish_wiki)
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
assert gitea.calls == []
|
||||
|
||||
def test_an_opted_in_repository_without_the_secret_fails_before_any_request(
|
||||
self, publish_wiki, environment, monkeypatch
|
||||
):
|
||||
monkeypatch.setenv("WIKI_CREDENTIALS", "")
|
||||
gitea = FakeGitea(publish_wiki)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"):
|
||||
run_main(publish_wiki, gitea)
|
||||
|
||||
assert gitea.calls == []
|
||||
|
||||
def test_an_unknown_marker_content_is_refused_before_any_request(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
(environment / ".publish").write_text("mirror everything", encoding="utf-8")
|
||||
gitea = FakeGitea(publish_wiki)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="prune"):
|
||||
run_main(publish_wiki, gitea)
|
||||
|
||||
assert gitea.calls == []
|
||||
|
||||
|
||||
class TestPublish:
|
||||
def test_prune_creates_updates_and_deletes_to_match_the_source(
|
||||
self, publish_wiki, environment, capsys
|
||||
):
|
||||
gitea = FakeGitea(
|
||||
publish_wiki,
|
||||
wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")},
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")}
|
||||
methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls]
|
||||
assert ("PATCH", "Home") in methods
|
||||
assert ("DELETE", "Old") in methods
|
||||
assert ("PATCH", "Credentials") not in methods
|
||||
assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out
|
||||
|
||||
def test_without_prune_pages_outside_the_source_are_kept(
|
||||
self, publish_wiki, environment, capsys
|
||||
):
|
||||
(environment / ".publish").write_text("", encoding="utf-8")
|
||||
gitea = FakeGitea(
|
||||
publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")}
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.wiki["Old"] == b64("hand written")
|
||||
assert gitea.wiki["Home"] == b64("home v2")
|
||||
assert "Credentials" in gitea.wiki
|
||||
assert not [call for call in gitea.calls if call[0] == "DELETE"]
|
||||
out = capsys.readouterr().out
|
||||
assert "1 created, 1 updated, 0 deleted" in out
|
||||
assert "1 page(s) are in the wiki but not in" in out
|
||||
assert "prune" in out
|
||||
|
||||
def test_creates_every_page_when_the_wiki_was_never_written(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
gitea = FakeGitea(publish_wiki, wiki_initialised=False)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert set(gitea.wiki) == {"Home", "Credentials"}
|
||||
assert [call[0] for call in gitea.calls].count("POST") == 2
|
||||
|
||||
def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment):
|
||||
gitea = FakeGitea(
|
||||
publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")}
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.writes == []
|
||||
|
||||
def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys):
|
||||
gitea = FakeGitea(publish_wiki)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"}
|
||||
output = capsys.readouterr()
|
||||
assert "wiki-token" not in output.out + output.err
|
||||
|
||||
def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
gitea = FakeGitea(publish_wiki, has_wiki=False)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
run_main(publish_wiki, gitea)
|
||||
|
||||
assert caught.value.category == "configuration"
|
||||
assert gitea.writes == []
|
||||
|
||||
def test_an_empty_source_never_deletes_pages_even_with_prune(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
for path in environment.glob("*.md"):
|
||||
path.unlink()
|
||||
gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")})
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
|
||||
run_main(publish_wiki, gitea)
|
||||
|
||||
assert gitea.calls == []
|
||||
assert gitea.wiki == {"Home": b64("keep me")}
|
||||
|
||||
|
||||
class TestPageIdentity:
|
||||
"""A wiki that already has pages named with spaces, as Gitea stores them."""
|
||||
|
||||
def test_a_file_named_with_a_space_updates_the_existing_page(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
(environment / "Home.md").unlink()
|
||||
(environment / "Credentials.md").unlink()
|
||||
(environment / ".publish").write_text("", encoding="utf-8")
|
||||
(environment / "Home da.md").write_text("hjem v2", encoding="utf-8")
|
||||
gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")})
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.wiki == {"Home-da": b64("hjem v2")}
|
||||
assert [call[0] for call in gitea.calls].count("POST") == 0
|
||||
assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls]
|
||||
|
||||
def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment):
|
||||
(environment / "Home.md").unlink()
|
||||
(environment / "Credentials.md").unlink()
|
||||
(environment / "Home da.md").write_text("hjem", encoding="utf-8")
|
||||
(environment / "_Sidebar.md").write_text("menu", encoding="utf-8")
|
||||
gitea = FakeGitea(
|
||||
publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")}
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.writes == []
|
||||
|
||||
def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
(environment / "Home.md").unlink()
|
||||
(environment / "Credentials.md").unlink()
|
||||
(environment / "Install ms.md").write_text("pasang", encoding="utf-8")
|
||||
gitea = FakeGitea(
|
||||
publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")}
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.wiki == {"Install-ms": b64("pasang")}
|
||||
|
||||
def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki):
|
||||
def listing(url, headers=None, **kwargs):
|
||||
return [
|
||||
{"title": "Whatever", "sub_url": "Real-Address"},
|
||||
{"title": "Only title"},
|
||||
{"title": "Encoded", "sub_url": "Caf%C3%A9"},
|
||||
]
|
||||
|
||||
keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing)
|
||||
|
||||
assert keys == {"Real-Address", "Only-title", "Café"}
|
||||
|
||||
|
||||
class TestDescribeToken:
|
||||
def test_recognises_a_gitea_token_shape(self, publish_wiki):
|
||||
description = publish_wiki.describe_token("a1b2c3d4e5" * 4)
|
||||
assert "40 characters" in description
|
||||
assert "looks like a Gitea token" in description
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("token", "expected"),
|
||||
[
|
||||
("*******", "only mask characters"),
|
||||
("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"),
|
||||
('"' + "a" * 40 + '"', "quote character"),
|
||||
("abc def", "whitespace"),
|
||||
],
|
||||
)
|
||||
def test_names_the_likely_mistake_without_echoing_the_token(
|
||||
self, publish_wiki, token, expected
|
||||
):
|
||||
description = publish_wiki.describe_token(token)
|
||||
assert expected in description
|
||||
assert token not in description
|
||||
|
||||
|
||||
class TestRejectedToken:
|
||||
def test_rejection_adds_the_token_shape_but_not_the_token(
|
||||
self, publish_wiki, environment, monkeypatch
|
||||
):
|
||||
monkeypatch.setenv("WIKI_CREDENTIALS", "*******")
|
||||
|
||||
def reject(url, method="GET", headers=None, body=None):
|
||||
raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.")
|
||||
|
||||
publish_wiki.request_json = reject
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.main()
|
||||
|
||||
message = str(caught.value)
|
||||
assert caught.value.category == "credentials"
|
||||
assert "Token shape: 7 characters" in message
|
||||
assert "only mask characters" in message
|
||||
assert "*******" not in message.split("Token shape:")[0]
|
||||
|
||||
def test_other_categories_are_not_changed(self, publish_wiki, environment):
|
||||
def forbid(url, method="GET", headers=None, body=None):
|
||||
raise publish_wiki.WorkflowError("permissions", "Forbidden.")
|
||||
|
||||
publish_wiki.request_json = forbid
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.main()
|
||||
|
||||
assert str(caught.value) == "[permissions] Forbidden."
|
||||
|
||||
|
||||
class TestRequestJson:
|
||||
@staticmethod
|
||||
def raise_http_error(publish_wiki, monkeypatch, code, reason):
|
||||
def fake_urlopen(request, timeout):
|
||||
raise urllib.error.HTTPError(
|
||||
request.full_url, code, reason, {}, io.BytesIO(b"secret detail")
|
||||
)
|
||||
|
||||
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("code", "reason", "category"),
|
||||
[
|
||||
(401, "Unauthorized", "credentials"),
|
||||
(403, "Forbidden", "permissions"),
|
||||
(404, "Not Found", "configuration"),
|
||||
(500, "Server Error", "api"),
|
||||
],
|
||||
)
|
||||
def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category):
|
||||
self.raise_http_error(publish_wiki, monkeypatch, code, reason)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.request_json(REPO_URL)
|
||||
|
||||
assert caught.value.category == category
|
||||
assert f"HTTP {code}" in str(caught.value)
|
||||
assert "secret detail" not in str(caught.value)
|
||||
|
||||
def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch):
|
||||
def fake_urlopen(request, timeout):
|
||||
raise urllib.error.URLError("no route")
|
||||
|
||||
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.request_json(REPO_URL)
|
||||
|
||||
assert caught.value.category == "api"
|
||||
|
||||
|
||||
class TestWorkflow:
|
||||
@pytest.fixture()
|
||||
def document(self):
|
||||
return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8"))
|
||||
|
||||
def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document):
|
||||
triggers = document[True] # PyYAML reads the key `on` as True
|
||||
assert triggers["push"]["branches"] == ["main"]
|
||||
assert triggers["push"]["paths"] == ["wiki/**"]
|
||||
assert "workflow_dispatch" in triggers
|
||||
|
||||
def test_job_has_read_only_permissions(self, document):
|
||||
assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"}
|
||||
|
||||
def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document):
|
||||
text = PUBLISH_WORKFLOW.read_text(encoding="utf-8")
|
||||
assert "secrets.TOKEN_FOR_GITEA }}" not in text
|
||||
publish_step = document["jobs"]["publish-wiki"]["steps"][-1]
|
||||
assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}"
|
||||
assert "secrets" not in publish_step["run"]
|
||||
|
||||
def test_the_workflow_has_one_source_and_runs_the_script_file(self, document):
|
||||
assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists()
|
||||
assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file()
|
||||
steps = document["jobs"]["publish-wiki"]["steps"]
|
||||
assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8")
|
||||
assert steps[0]["uses"].startswith("actions/checkout@")
|
||||
assert steps[1]["with"] == {
|
||||
"repository": "TirSystem/github-action",
|
||||
"ref": "v1",
|
||||
"path": "shared-scripts",
|
||||
}
|
||||
assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py"
|
||||
|
||||
|
||||
def test_this_repository_opts_in_and_prunes():
|
||||
"""The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004)."""
|
||||
marker = REPO_ROOT / "wiki" / ".publish"
|
||||
assert marker.read_text(encoding="utf-8").strip() == "prune"
|
||||
"""Offline tests for tools/publish_wiki.py and the scoped publish-wiki workflow."""
|
||||
|
||||
import base64
|
||||
import io
|
||||
import urllib.error
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from workflow_source import PUBLISH_WORKFLOW, REPO_ROOT, load_publish_module
|
||||
|
||||
API = "https://git.example.test/api/v1"
|
||||
REPO_URL = f"{API}/repos/TirSystem/repo"
|
||||
WIKI_URL = f"{REPO_URL}/wiki"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def publish_wiki():
|
||||
return load_publish_module()
|
||||
|
||||
|
||||
def b64(text: str) -> str:
|
||||
return base64.b64encode(text.encode()).decode()
|
||||
|
||||
|
||||
class TestParseMarker:
|
||||
@pytest.mark.parametrize(
|
||||
("text", "expected"),
|
||||
[("", False), (" \n", False), (None, False), ("prune", True), (" prune\n", True)],
|
||||
)
|
||||
def test_reads_the_empty_marker_and_the_prune_word(self, publish_wiki, text, expected):
|
||||
assert publish_wiki.parse_marker(text) is expected
|
||||
|
||||
@pytest.mark.parametrize("text", ["Prune", "prune: true", "delete", "prune\nmore"])
|
||||
def test_rejects_anything_else(self, publish_wiki, text):
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.parse_marker(text)
|
||||
assert caught.value.category == "configuration"
|
||||
assert "prune" in str(caught.value)
|
||||
|
||||
|
||||
class TestParseGiteaToken:
|
||||
def test_returns_the_stripped_token(self, publish_wiki):
|
||||
assert publish_wiki.parse_gitea_token(" abc \n") == "abc"
|
||||
|
||||
@pytest.mark.parametrize("raw", [None, "", " "])
|
||||
def test_rejects_missing_or_empty_token(self, publish_wiki, raw):
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.parse_gitea_token(raw)
|
||||
assert caught.value.category == "configuration"
|
||||
assert "TOKEN_FOR_GITEA_WIKI is missing or empty" in str(caught.value)
|
||||
|
||||
|
||||
class TestReadPages:
|
||||
def test_reads_markdown_files_by_stem_and_ignores_the_marker(self, publish_wiki, tmp_path):
|
||||
(tmp_path / "Home.md").write_bytes(b"home")
|
||||
(tmp_path / "Home.da.md").write_bytes(b"hjem")
|
||||
(tmp_path / ".publish").write_bytes(b"prune")
|
||||
(tmp_path / "Notes.txt").write_bytes(b"ignored")
|
||||
assert publish_wiki.read_pages(tmp_path) == {"Home": b"home", "Home.da": b"hjem"}
|
||||
|
||||
def test_rejects_a_directory_without_pages_so_the_wiki_is_never_wiped(
|
||||
self, publish_wiki, tmp_path
|
||||
):
|
||||
(tmp_path / ".publish").write_text("prune", encoding="utf-8")
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
|
||||
publish_wiki.read_pages(tmp_path)
|
||||
|
||||
|
||||
class TestPageTitle:
|
||||
@pytest.mark.parametrize(
|
||||
("key", "title"), [("Home", "Home"), ("Home-da", "Home da"), ("Getting-Started-ms", "Getting Started ms")]
|
||||
)
|
||||
def test_a_hyphen_is_sent_as_a_space(self, publish_wiki, key, title):
|
||||
# Gitea keeps a hyphen in a title literal (Home-da.-.md, address Home%2Dda).
|
||||
assert publish_wiki.page_title(key) == title
|
||||
|
||||
|
||||
class TestPageKey:
|
||||
@pytest.mark.parametrize(
|
||||
("name", "key"),
|
||||
[("Home", "Home"), ("Home da", "Home-da"), ("Home-da", "Home-da"), ("Home.da", "Home.da"),
|
||||
("AI Prompt Template ms", "AI-Prompt-Template-ms"), ("_Sidebar", "_Sidebar")],
|
||||
)
|
||||
def test_a_space_and_a_hyphen_are_the_same_page(self, publish_wiki, name, key):
|
||||
assert publish_wiki.page_key(name) == key
|
||||
|
||||
def test_file_names_with_spaces_and_hyphens_are_keyed_the_same(self, publish_wiki, tmp_path):
|
||||
(tmp_path / "Home da.md").write_bytes(b"hjem")
|
||||
(tmp_path / "_Sidebar.md").write_bytes(b"menu")
|
||||
assert publish_wiki.read_pages(tmp_path) == {"Home-da": b"hjem", "_Sidebar": b"menu"}
|
||||
|
||||
def test_two_files_for_one_page_are_refused(self, publish_wiki, tmp_path):
|
||||
(tmp_path / "Home da.md").write_bytes(b"one")
|
||||
(tmp_path / "Home-da.md").write_bytes(b"two")
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="name the wiki page Home-da"):
|
||||
publish_wiki.read_pages(tmp_path)
|
||||
|
||||
|
||||
class FakeGitea:
|
||||
"""Answers like the Gitea repository and wiki endpoints."""
|
||||
|
||||
def __init__(self, publish_wiki, wiki=None, has_wiki=True, wiki_initialised=True):
|
||||
self.error = publish_wiki.WorkflowError
|
||||
self.wiki = dict(wiki or {})
|
||||
self.has_wiki = has_wiki
|
||||
self.wiki_initialised = wiki_initialised
|
||||
self.calls = []
|
||||
|
||||
def __call__(self, url, method="GET", headers=None, body=None):
|
||||
self.calls.append((method, url, body, headers))
|
||||
if url == REPO_URL:
|
||||
return {"has_wiki": self.has_wiki}
|
||||
if url.startswith(f"{WIKI_URL}/pages"):
|
||||
if not self.wiki_initialised:
|
||||
raise self.error("configuration", "404")
|
||||
return [{"title": key.replace("-", " "), "sub_url": key} for key in self.wiki]
|
||||
if method == "POST" and url == f"{WIKI_URL}/new":
|
||||
self.wiki[body["title"].replace(" ", "-")] = body["content_base64"]
|
||||
return None
|
||||
title = url.rsplit("/", 1)[-1]
|
||||
if method == "GET":
|
||||
return {"title": title, "content_base64": self.wiki[title]}
|
||||
if method == "PATCH":
|
||||
assert " " in body["title"] or "-" not in body["title"], body["title"]
|
||||
self.wiki[title] = body["content_base64"]
|
||||
return None
|
||||
if method == "DELETE":
|
||||
del self.wiki[title]
|
||||
return None
|
||||
raise AssertionError(f"unexpected request {method} {url}")
|
||||
|
||||
@property
|
||||
def writes(self):
|
||||
return [call for call in self.calls if call[0] != "GET"]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def environment(monkeypatch, tmp_path):
|
||||
wiki_dir = tmp_path / "wiki"
|
||||
wiki_dir.mkdir()
|
||||
(wiki_dir / "Home.md").write_text("home v2", encoding="utf-8")
|
||||
(wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8")
|
||||
(wiki_dir / ".publish").write_text("prune\n", encoding="utf-8")
|
||||
monkeypatch.setenv("WIKI_DIR", str(wiki_dir))
|
||||
monkeypatch.setenv("GITEA_API_URL", f"{API}/")
|
||||
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
|
||||
monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token")
|
||||
return wiki_dir
|
||||
|
||||
|
||||
def run_main(publish_wiki, gitea):
|
||||
publish_wiki.request_json = gitea
|
||||
return publish_wiki.main()
|
||||
|
||||
|
||||
class TestOptIn:
|
||||
def test_a_repository_without_the_marker_is_never_written_to(
|
||||
self, publish_wiki, environment, capsys
|
||||
):
|
||||
(environment / ".publish").unlink()
|
||||
gitea = FakeGitea(publish_wiki, wiki={"Old": b64("hand written")})
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.calls == []
|
||||
assert "wiki publishing is not enabled" in capsys.readouterr().out
|
||||
assert gitea.wiki == {"Old": b64("hand written")}
|
||||
|
||||
def test_a_repository_without_the_marker_does_not_need_the_secret(
|
||||
self, publish_wiki, environment, monkeypatch
|
||||
):
|
||||
(environment / ".publish").unlink()
|
||||
monkeypatch.setenv("WIKI_CREDENTIALS", "")
|
||||
assert run_main(publish_wiki, FakeGitea(publish_wiki)) == 0
|
||||
|
||||
def test_a_repository_without_a_wiki_directory_is_skipped(
|
||||
self, publish_wiki, monkeypatch, tmp_path
|
||||
):
|
||||
monkeypatch.setenv("WIKI_DIR", str(tmp_path / "missing"))
|
||||
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
|
||||
gitea = FakeGitea(publish_wiki)
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
assert gitea.calls == []
|
||||
|
||||
def test_an_opted_in_repository_without_the_secret_fails_before_any_request(
|
||||
self, publish_wiki, environment, monkeypatch
|
||||
):
|
||||
monkeypatch.setenv("WIKI_CREDENTIALS", "")
|
||||
gitea = FakeGitea(publish_wiki)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"):
|
||||
run_main(publish_wiki, gitea)
|
||||
|
||||
assert gitea.calls == []
|
||||
|
||||
def test_an_unknown_marker_content_is_refused_before_any_request(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
(environment / ".publish").write_text("mirror everything", encoding="utf-8")
|
||||
gitea = FakeGitea(publish_wiki)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="prune"):
|
||||
run_main(publish_wiki, gitea)
|
||||
|
||||
assert gitea.calls == []
|
||||
|
||||
|
||||
class TestPublish:
|
||||
def test_prune_creates_updates_and_deletes_to_match_the_source(
|
||||
self, publish_wiki, environment, capsys
|
||||
):
|
||||
gitea = FakeGitea(
|
||||
publish_wiki,
|
||||
wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")},
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")}
|
||||
methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls]
|
||||
assert ("PATCH", "Home") in methods
|
||||
assert ("DELETE", "Old") in methods
|
||||
assert ("PATCH", "Credentials") not in methods
|
||||
assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out
|
||||
|
||||
def test_without_prune_pages_outside_the_source_are_kept(
|
||||
self, publish_wiki, environment, capsys
|
||||
):
|
||||
(environment / ".publish").write_text("", encoding="utf-8")
|
||||
gitea = FakeGitea(
|
||||
publish_wiki, wiki={"Home": b64("home v1"), "Old": b64("hand written")}
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.wiki["Old"] == b64("hand written")
|
||||
assert gitea.wiki["Home"] == b64("home v2")
|
||||
assert "Credentials" in gitea.wiki
|
||||
assert not [call for call in gitea.calls if call[0] == "DELETE"]
|
||||
out = capsys.readouterr().out
|
||||
assert "1 created, 1 updated, 0 deleted" in out
|
||||
assert "1 page(s) are in the wiki but not in" in out
|
||||
assert "prune" in out
|
||||
|
||||
def test_creates_every_page_when_the_wiki_was_never_written(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
gitea = FakeGitea(publish_wiki, wiki_initialised=False)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert set(gitea.wiki) == {"Home", "Credentials"}
|
||||
assert [call[0] for call in gitea.calls].count("POST") == 2
|
||||
|
||||
def test_does_nothing_when_the_wiki_is_already_current(self, publish_wiki, environment):
|
||||
gitea = FakeGitea(
|
||||
publish_wiki, wiki={"Home": b64("home v2"), "Credentials": b64("credentials")}
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.writes == []
|
||||
|
||||
def test_sends_the_token_and_never_prints_it(self, publish_wiki, environment, capsys):
|
||||
gitea = FakeGitea(publish_wiki)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert {call[3]["Authorization"] for call in gitea.calls} == {"token wiki-token"}
|
||||
output = capsys.readouterr()
|
||||
assert "wiki-token" not in output.out + output.err
|
||||
|
||||
def test_fails_when_the_wiki_is_not_enabled_and_writes_nothing(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
gitea = FakeGitea(publish_wiki, has_wiki=False)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
run_main(publish_wiki, gitea)
|
||||
|
||||
assert caught.value.category == "configuration"
|
||||
assert gitea.writes == []
|
||||
|
||||
def test_an_empty_source_never_deletes_pages_even_with_prune(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
for path in environment.glob("*.md"):
|
||||
path.unlink()
|
||||
gitea = FakeGitea(publish_wiki, wiki={"Home": b64("keep me")})
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
|
||||
run_main(publish_wiki, gitea)
|
||||
|
||||
assert gitea.calls == []
|
||||
assert gitea.wiki == {"Home": b64("keep me")}
|
||||
|
||||
|
||||
class TestPageIdentity:
|
||||
"""A wiki that already has pages named with spaces, as Gitea stores them."""
|
||||
|
||||
def test_a_file_named_with_a_space_updates_the_existing_page(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
(environment / "Home.md").unlink()
|
||||
(environment / "Credentials.md").unlink()
|
||||
(environment / ".publish").write_text("", encoding="utf-8")
|
||||
(environment / "Home da.md").write_text("hjem v2", encoding="utf-8")
|
||||
gitea = FakeGitea(publish_wiki, wiki={"Home-da": b64("hjem v1")})
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.wiki == {"Home-da": b64("hjem v2")}
|
||||
assert [call[0] for call in gitea.calls].count("POST") == 0
|
||||
assert ("PATCH", f"{WIKI_URL}/page/Home-da") in [(c[0], c[1]) for c in gitea.calls]
|
||||
|
||||
def test_identical_pages_with_spaces_are_a_no_op(self, publish_wiki, environment):
|
||||
(environment / "Home.md").unlink()
|
||||
(environment / "Credentials.md").unlink()
|
||||
(environment / "Home da.md").write_text("hjem", encoding="utf-8")
|
||||
(environment / "_Sidebar.md").write_text("menu", encoding="utf-8")
|
||||
gitea = FakeGitea(
|
||||
publish_wiki, wiki={"Home-da": b64("hjem"), "_Sidebar": b64("menu")}
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.writes == []
|
||||
|
||||
def test_prune_does_not_delete_a_page_that_is_in_the_folder_under_another_spelling(
|
||||
self, publish_wiki, environment
|
||||
):
|
||||
(environment / "Home.md").unlink()
|
||||
(environment / "Credentials.md").unlink()
|
||||
(environment / "Install ms.md").write_text("pasang", encoding="utf-8")
|
||||
gitea = FakeGitea(
|
||||
publish_wiki, wiki={"Install-ms": b64("pasang"), "Old-page": b64("x")}
|
||||
)
|
||||
|
||||
assert run_main(publish_wiki, gitea) == 0
|
||||
|
||||
assert gitea.wiki == {"Install-ms": b64("pasang")}
|
||||
|
||||
def test_the_address_wins_over_the_title_and_is_decoded(self, publish_wiki):
|
||||
def listing(url, headers=None, **kwargs):
|
||||
return [
|
||||
{"title": "Whatever", "sub_url": "Real-Address"},
|
||||
{"title": "Only title"},
|
||||
{"title": "Encoded", "sub_url": "Caf%C3%A9"},
|
||||
]
|
||||
|
||||
keys = publish_wiki.list_wiki_titles(WIKI_URL, {}, listing)
|
||||
|
||||
assert keys == {"Real-Address", "Only-title", "Café"}
|
||||
|
||||
|
||||
class TestDescribeToken:
|
||||
def test_recognises_a_gitea_token_shape(self, publish_wiki):
|
||||
description = publish_wiki.describe_token("a1b2c3d4e5" * 4)
|
||||
assert "40 characters" in description
|
||||
assert "looks like a Gitea token" in description
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("token", "expected"),
|
||||
[
|
||||
("*******", "only mask characters"),
|
||||
("TOKEN_FOR_GITEA_WIKI=" + "a" * 40, "contains '='"),
|
||||
('"' + "a" * 40 + '"', "quote character"),
|
||||
("abc def", "whitespace"),
|
||||
],
|
||||
)
|
||||
def test_names_the_likely_mistake_without_echoing_the_token(
|
||||
self, publish_wiki, token, expected
|
||||
):
|
||||
description = publish_wiki.describe_token(token)
|
||||
assert expected in description
|
||||
assert token not in description
|
||||
|
||||
|
||||
class TestRejectedToken:
|
||||
def test_rejection_adds_the_token_shape_but_not_the_token(
|
||||
self, publish_wiki, environment, monkeypatch
|
||||
):
|
||||
monkeypatch.setenv("WIKI_CREDENTIALS", "*******")
|
||||
|
||||
def reject(url, method="GET", headers=None, body=None):
|
||||
raise publish_wiki.WorkflowError("credentials", "Gitea rejected the token.")
|
||||
|
||||
publish_wiki.request_json = reject
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.main()
|
||||
|
||||
message = str(caught.value)
|
||||
assert caught.value.category == "credentials"
|
||||
assert "Token shape: 7 characters" in message
|
||||
assert "only mask characters" in message
|
||||
assert "*******" not in message.split("Token shape:")[0]
|
||||
|
||||
def test_other_categories_are_not_changed(self, publish_wiki, environment):
|
||||
def forbid(url, method="GET", headers=None, body=None):
|
||||
raise publish_wiki.WorkflowError("permissions", "Forbidden.")
|
||||
|
||||
publish_wiki.request_json = forbid
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.main()
|
||||
|
||||
assert str(caught.value) == "[permissions] Forbidden."
|
||||
|
||||
|
||||
class TestRequestJson:
|
||||
@staticmethod
|
||||
def raise_http_error(publish_wiki, monkeypatch, code, reason):
|
||||
def fake_urlopen(request, timeout):
|
||||
raise urllib.error.HTTPError(
|
||||
request.full_url, code, reason, {}, io.BytesIO(b"secret detail")
|
||||
)
|
||||
|
||||
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("code", "reason", "category"),
|
||||
[
|
||||
(401, "Unauthorized", "credentials"),
|
||||
(403, "Forbidden", "permissions"),
|
||||
(404, "Not Found", "configuration"),
|
||||
(500, "Server Error", "api"),
|
||||
],
|
||||
)
|
||||
def test_categorizes_http_errors(self, publish_wiki, monkeypatch, code, reason, category):
|
||||
self.raise_http_error(publish_wiki, monkeypatch, code, reason)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.request_json(REPO_URL)
|
||||
|
||||
assert caught.value.category == category
|
||||
assert f"HTTP {code}" in str(caught.value)
|
||||
assert "secret detail" not in str(caught.value)
|
||||
|
||||
def test_reports_unreachable_service_as_api_failure(self, publish_wiki, monkeypatch):
|
||||
def fake_urlopen(request, timeout):
|
||||
raise urllib.error.URLError("no route")
|
||||
|
||||
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
|
||||
|
||||
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||
publish_wiki.request_json(REPO_URL)
|
||||
|
||||
assert caught.value.category == "api"
|
||||
|
||||
|
||||
class TestWorkflow:
|
||||
@pytest.fixture()
|
||||
def document(self):
|
||||
return yaml.safe_load(PUBLISH_WORKFLOW.read_text(encoding="utf-8"))
|
||||
|
||||
def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document):
|
||||
triggers = document[True] # PyYAML reads the key `on` as True
|
||||
assert triggers["push"]["branches"] == ["main"]
|
||||
assert triggers["push"]["paths"] == ["wiki/**"]
|
||||
assert "workflow_dispatch" in triggers
|
||||
|
||||
def test_job_has_read_only_permissions(self, document):
|
||||
assert document["jobs"]["publish-wiki"]["permissions"] == {"contents": "read"}
|
||||
|
||||
def test_secret_is_passed_only_through_the_environment_and_has_no_fallback(self, document):
|
||||
text = PUBLISH_WORKFLOW.read_text(encoding="utf-8")
|
||||
assert "secrets.TOKEN_FOR_GITEA }}" not in text
|
||||
publish_step = document["jobs"]["publish-wiki"]["steps"][-1]
|
||||
assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}"
|
||||
assert "secrets" not in publish_step["run"]
|
||||
|
||||
def test_the_workflow_has_one_source_and_runs_the_script_file(self, document):
|
||||
assert not (REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml").exists()
|
||||
assert (REPO_ROOT / "tools" / "publish_wiki.py").is_file()
|
||||
steps = document["jobs"]["publish-wiki"]["steps"]
|
||||
assert "python3 - <<" not in PUBLISH_WORKFLOW.read_text(encoding="utf-8")
|
||||
assert steps[0]["uses"].startswith("actions/checkout@")
|
||||
assert steps[1]["with"] == {
|
||||
"repository": "TirSystem/github-action",
|
||||
"ref": "v1",
|
||||
"path": "shared-scripts",
|
||||
}
|
||||
assert steps[-1]["run"] == "python3 shared-scripts/tools/publish_wiki.py"
|
||||
|
||||
|
||||
def test_this_repository_opts_in_and_prunes():
|
||||
"""The wiki of this repository must equal wiki/ (ADR-0001, ADR-0004)."""
|
||||
marker = REPO_ROOT / "wiki" / ".publish"
|
||||
assert marker.read_text(encoding="utf-8").strip() == "prune"
|
||||
|
||||
+306
-306
@@ -1,306 +1,306 @@
|
||||
"""Publish the pages in wiki/ to the Gitea wiki of the repository.
|
||||
|
||||
Run by the scoped workflow publish-wiki.yml, which checks out the repository
|
||||
that owns the wiki and this repository, and runs this file (MIL-004 task 6).
|
||||
A repository publishes only when it has opted in with wiki/.publish
|
||||
(ADR-0004); the word prune in that file lets the run delete wiki pages that
|
||||
are not in wiki/. It reads its settings from the environment:
|
||||
|
||||
GITEA_API_URL Gitea API base, for example https://host/api/v1
|
||||
SOURCE_REPOSITORY owner/repo of the repository whose wiki is written
|
||||
WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI)
|
||||
WIKI_DIR directory with the pages (default: wiki)
|
||||
|
||||
Every failure is printed as ``ERROR: [category] message`` with the categories
|
||||
configuration, credentials, permissions and api; no secret value is printed.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
PAGE_LIMIT = 50
|
||||
REQUEST_TIMEOUT_SECONDS = 30
|
||||
COMMIT_MESSAGE = "Publish wiki from the repository"
|
||||
MARKER_NAME = ".publish"
|
||||
PRUNE_WORD = "prune"
|
||||
|
||||
|
||||
class WorkflowError(RuntimeError):
|
||||
"""A failure with a category.
|
||||
|
||||
configuration: a secret, a setting or the wiki source is missing or malformed.
|
||||
credentials: Gitea rejected the token (HTTP 401).
|
||||
permissions: the token is valid but not allowed (HTTP 403).
|
||||
api: any other API or network failure.
|
||||
"""
|
||||
|
||||
def __init__(self, category, message):
|
||||
super().__init__(f"[{category}] {message}")
|
||||
self.category = category
|
||||
self.detail = message
|
||||
|
||||
|
||||
def request_json(url, method="GET", headers=None, body=None):
|
||||
request = urllib.request.Request(
|
||||
url,
|
||||
data=json.dumps(body).encode("utf-8") if body is not None else None,
|
||||
headers=headers or {},
|
||||
method=method,
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response:
|
||||
content = response.read()
|
||||
return json.loads(content) if content else None
|
||||
except urllib.error.HTTPError as error:
|
||||
detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})"
|
||||
if error.code == 401:
|
||||
raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None
|
||||
if error.code == 403:
|
||||
raise WorkflowError(
|
||||
"permissions", f"{detail}. The token lacks a required permission."
|
||||
) from None
|
||||
if error.code == 404:
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
f"{detail}. The repository or page was not found, or the token "
|
||||
"cannot see it.",
|
||||
) from None
|
||||
raise WorkflowError("api", detail) from None
|
||||
except (urllib.error.URLError, TimeoutError) as error:
|
||||
raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None
|
||||
|
||||
|
||||
def parse_marker(text):
|
||||
"""Return True when the marker asks for pruning, False when it is empty."""
|
||||
word = (text or "").strip()
|
||||
if not word:
|
||||
return False
|
||||
if word == PRUNE_WORD:
|
||||
return True
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.",
|
||||
)
|
||||
|
||||
|
||||
def parse_gitea_token(raw_token):
|
||||
token = (raw_token or "").strip()
|
||||
if not token:
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
"TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of "
|
||||
"the organization of this repository.",
|
||||
)
|
||||
return token
|
||||
|
||||
|
||||
def describe_token(token):
|
||||
"""Describe the shape of a token without revealing any of its characters."""
|
||||
facts = [f"{len(token)} characters"]
|
||||
if re.fullmatch(r"[0-9a-f]{40}", token):
|
||||
facts.append("looks like a Gitea token (40 hexadecimal characters)")
|
||||
return ", ".join(facts)
|
||||
facts.append("a Gitea token has 40 hexadecimal characters")
|
||||
if set(token) <= {"*", "x", "X", "•"}:
|
||||
facts.append("only mask characters: the masked text was stored")
|
||||
if "=" in token:
|
||||
facts.append("contains '=': the secret name may be part of the value")
|
||||
if re.search(r"[\"'`]", token):
|
||||
facts.append("contains a quote character")
|
||||
if re.search(r"\s", token):
|
||||
facts.append("contains whitespace")
|
||||
return ", ".join(facts)
|
||||
|
||||
|
||||
def split_repository(full_name):
|
||||
owner, separator, repo = (full_name or "").partition("/")
|
||||
if not separator or not owner or not repo:
|
||||
raise WorkflowError("configuration", "Could not determine the Gitea repository.")
|
||||
return owner, repo
|
||||
|
||||
|
||||
def page_key(title):
|
||||
"""Return the identity of a page: Gitea stores a space as a hyphen.
|
||||
|
||||
The page "Home da" is the file Home-da.md and its address is Home-da, so
|
||||
both spellings must name the same page.
|
||||
"""
|
||||
return str(title).replace(" ", "-")
|
||||
|
||||
|
||||
def page_title(key):
|
||||
"""Return the title to send to Gitea for a page key.
|
||||
|
||||
A hyphen in a title is a literal hyphen to Gitea, which then stores the page
|
||||
as Home-da.-.md and addresses it as Home%2Dda, so a link to Home-da finds
|
||||
nothing. A space is stored as a hyphen (Home-da.md) and addressed as Home-da.
|
||||
"""
|
||||
return key.replace("-", " ")
|
||||
|
||||
|
||||
def read_pages(wiki_dir):
|
||||
"""Return {page key: content} for every Markdown file in wiki_dir."""
|
||||
pages = {}
|
||||
for path in sorted(wiki_dir.glob("*.md")):
|
||||
key = page_key(path.stem)
|
||||
if key in pages:
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
f"Two files in {wiki_dir} name the wiki page {key}; a space and "
|
||||
"a hyphen are the same in a page name.",
|
||||
)
|
||||
pages[key] = path.read_bytes()
|
||||
if not pages:
|
||||
# Publishing an empty directory would delete the whole wiki.
|
||||
raise WorkflowError(
|
||||
"configuration", f"The wiki source directory {wiki_dir} has no Markdown pages."
|
||||
)
|
||||
return pages
|
||||
|
||||
|
||||
def list_wiki_titles(wiki_url, headers, send):
|
||||
"""Return the keys of the pages that exist in the wiki.
|
||||
|
||||
The key is the address Gitea reports (sub_url), or the title with its
|
||||
spaces replaced by hyphens.
|
||||
|
||||
Gitea answers 404 for a wiki that has never been written to; that is an
|
||||
empty wiki, not a failure.
|
||||
"""
|
||||
titles = set()
|
||||
page = 1
|
||||
while True:
|
||||
url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}"
|
||||
try:
|
||||
batch = send(url, headers=headers)
|
||||
except WorkflowError as error:
|
||||
if error.category == "configuration" and page == 1:
|
||||
return titles
|
||||
raise
|
||||
if not isinstance(batch, list):
|
||||
raise WorkflowError("api", "Gitea returned an invalid wiki page list.")
|
||||
titles.update(
|
||||
urllib.parse.unquote(str(item.get("sub_url") or ""))
|
||||
or page_key(item.get("title", ""))
|
||||
for item in batch
|
||||
)
|
||||
if len(batch) < PAGE_LIMIT:
|
||||
return titles
|
||||
page += 1
|
||||
|
||||
|
||||
def read_wiki_page(wiki_url, title, headers, send):
|
||||
quoted = urllib.parse.quote(title, safe="")
|
||||
page = send(f"{wiki_url}/page/{quoted}", headers=headers)
|
||||
return base64.b64decode((page or {}).get("content_base64", ""))
|
||||
|
||||
|
||||
def publish(pages, wiki_url, headers, send, prune):
|
||||
"""Create and update the pages; delete the others only when prune is set.
|
||||
|
||||
Returns the titles created, updated and deleted, and the titles that were
|
||||
left in the wiki because pruning is off.
|
||||
"""
|
||||
existing = list_wiki_titles(wiki_url, headers, send)
|
||||
current = {
|
||||
title: read_wiki_page(wiki_url, title, headers, send)
|
||||
for title in sorted(existing & pages.keys())
|
||||
}
|
||||
|
||||
result = {"created": [], "updated": [], "deleted": [], "kept": []}
|
||||
for title, content in pages.items():
|
||||
body = {
|
||||
"title": page_title(title),
|
||||
"content_base64": base64.b64encode(content).decode("ascii"),
|
||||
"message": COMMIT_MESSAGE,
|
||||
}
|
||||
if title not in existing:
|
||||
send(f"{wiki_url}/new", method="POST", headers=headers, body=body)
|
||||
result["created"].append(title)
|
||||
elif current[title] != content:
|
||||
quoted = urllib.parse.quote(title, safe="")
|
||||
send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body)
|
||||
result["updated"].append(title)
|
||||
|
||||
for title in sorted(existing - pages.keys()):
|
||||
if not prune:
|
||||
result["kept"].append(title)
|
||||
continue
|
||||
quoted = urllib.parse.quote(title, safe="")
|
||||
send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers)
|
||||
result["deleted"].append(title)
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki"))
|
||||
owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY"))
|
||||
|
||||
marker = wiki_dir / MARKER_NAME
|
||||
if not marker.is_file():
|
||||
print(
|
||||
f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; "
|
||||
"wiki publishing is not enabled for this repository."
|
||||
)
|
||||
return 0
|
||||
prune = parse_marker(marker.read_text(encoding="utf-8"))
|
||||
|
||||
pages = read_pages(wiki_dir)
|
||||
token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS"))
|
||||
|
||||
api_url = os.environ["GITEA_API_URL"].rstrip("/")
|
||||
repository_url = (
|
||||
f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/"
|
||||
f"{urllib.parse.quote(repo, safe='')}"
|
||||
)
|
||||
headers = {
|
||||
"Authorization": f"token {token}",
|
||||
"Accept": "application/json",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
try:
|
||||
repository = request_json(repository_url, headers=headers)
|
||||
if isinstance(repository, dict) and repository.get("has_wiki") is False:
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
f"The wiki is not enabled on {owner}/{repo}. Enable it in the "
|
||||
"repository settings.",
|
||||
)
|
||||
print("Preflight passed: Gitea accepted the token and the wiki is enabled.")
|
||||
|
||||
result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune)
|
||||
except WorkflowError as error:
|
||||
if error.category == "credentials":
|
||||
raise WorkflowError(
|
||||
"credentials", f"{error.detail} Token shape: {describe_token(token)}."
|
||||
) from None
|
||||
raise
|
||||
|
||||
print(
|
||||
f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: "
|
||||
f"{len(result['created'])} created, {len(result['updated'])} updated, "
|
||||
f"{len(result['deleted'])} deleted."
|
||||
)
|
||||
if result["kept"]:
|
||||
print(
|
||||
f"{len(result['kept'])} page(s) are in the wiki but not in "
|
||||
f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in "
|
||||
f"{wiki_dir}/{MARKER_NAME} to delete them."
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except WorkflowError as error:
|
||||
print(f"ERROR: {error}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
"""Publish the pages in wiki/ to the Gitea wiki of the repository.
|
||||
|
||||
Run by the scoped workflow publish-wiki.yml, which checks out the repository
|
||||
that owns the wiki and this repository, and runs this file (MIL-004 task 6).
|
||||
A repository publishes only when it has opted in with wiki/.publish
|
||||
(ADR-0004); the word prune in that file lets the run delete wiki pages that
|
||||
are not in wiki/. It reads its settings from the environment:
|
||||
|
||||
GITEA_API_URL Gitea API base, for example https://host/api/v1
|
||||
SOURCE_REPOSITORY owner/repo of the repository whose wiki is written
|
||||
WIKI_CREDENTIALS the Gitea token, as clear text (TOKEN_FOR_GITEA_WIKI)
|
||||
WIKI_DIR directory with the pages (default: wiki)
|
||||
|
||||
Every failure is printed as ``ERROR: [category] message`` with the categories
|
||||
configuration, credentials, permissions and api; no secret value is printed.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
PAGE_LIMIT = 50
|
||||
REQUEST_TIMEOUT_SECONDS = 30
|
||||
COMMIT_MESSAGE = "Publish wiki from the repository"
|
||||
MARKER_NAME = ".publish"
|
||||
PRUNE_WORD = "prune"
|
||||
|
||||
|
||||
class WorkflowError(RuntimeError):
|
||||
"""A failure with a category.
|
||||
|
||||
configuration: a secret, a setting or the wiki source is missing or malformed.
|
||||
credentials: Gitea rejected the token (HTTP 401).
|
||||
permissions: the token is valid but not allowed (HTTP 403).
|
||||
api: any other API or network failure.
|
||||
"""
|
||||
|
||||
def __init__(self, category, message):
|
||||
super().__init__(f"[{category}] {message}")
|
||||
self.category = category
|
||||
self.detail = message
|
||||
|
||||
|
||||
def request_json(url, method="GET", headers=None, body=None):
|
||||
request = urllib.request.Request(
|
||||
url,
|
||||
data=json.dumps(body).encode("utf-8") if body is not None else None,
|
||||
headers=headers or {},
|
||||
method=method,
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response:
|
||||
content = response.read()
|
||||
return json.loads(content) if content else None
|
||||
except urllib.error.HTTPError as error:
|
||||
detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})"
|
||||
if error.code == 401:
|
||||
raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None
|
||||
if error.code == 403:
|
||||
raise WorkflowError(
|
||||
"permissions", f"{detail}. The token lacks a required permission."
|
||||
) from None
|
||||
if error.code == 404:
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
f"{detail}. The repository or page was not found, or the token "
|
||||
"cannot see it.",
|
||||
) from None
|
||||
raise WorkflowError("api", detail) from None
|
||||
except (urllib.error.URLError, TimeoutError) as error:
|
||||
raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None
|
||||
|
||||
|
||||
def parse_marker(text):
|
||||
"""Return True when the marker asks for pruning, False when it is empty."""
|
||||
word = (text or "").strip()
|
||||
if not word:
|
||||
return False
|
||||
if word == PRUNE_WORD:
|
||||
return True
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
f"wiki/{MARKER_NAME} may only be empty or contain the word '{PRUNE_WORD}'.",
|
||||
)
|
||||
|
||||
|
||||
def parse_gitea_token(raw_token):
|
||||
token = (raw_token or "").strip()
|
||||
if not token:
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
"TOKEN_FOR_GITEA_WIKI is missing or empty. Store it as a secret of "
|
||||
"the organization of this repository.",
|
||||
)
|
||||
return token
|
||||
|
||||
|
||||
def describe_token(token):
|
||||
"""Describe the shape of a token without revealing any of its characters."""
|
||||
facts = [f"{len(token)} characters"]
|
||||
if re.fullmatch(r"[0-9a-f]{40}", token):
|
||||
facts.append("looks like a Gitea token (40 hexadecimal characters)")
|
||||
return ", ".join(facts)
|
||||
facts.append("a Gitea token has 40 hexadecimal characters")
|
||||
if set(token) <= {"*", "x", "X", "•"}:
|
||||
facts.append("only mask characters: the masked text was stored")
|
||||
if "=" in token:
|
||||
facts.append("contains '=': the secret name may be part of the value")
|
||||
if re.search(r"[\"'`]", token):
|
||||
facts.append("contains a quote character")
|
||||
if re.search(r"\s", token):
|
||||
facts.append("contains whitespace")
|
||||
return ", ".join(facts)
|
||||
|
||||
|
||||
def split_repository(full_name):
|
||||
owner, separator, repo = (full_name or "").partition("/")
|
||||
if not separator or not owner or not repo:
|
||||
raise WorkflowError("configuration", "Could not determine the Gitea repository.")
|
||||
return owner, repo
|
||||
|
||||
|
||||
def page_key(title):
|
||||
"""Return the identity of a page: Gitea stores a space as a hyphen.
|
||||
|
||||
The page "Home da" is the file Home-da.md and its address is Home-da, so
|
||||
both spellings must name the same page.
|
||||
"""
|
||||
return str(title).replace(" ", "-")
|
||||
|
||||
|
||||
def page_title(key):
|
||||
"""Return the title to send to Gitea for a page key.
|
||||
|
||||
A hyphen in a title is a literal hyphen to Gitea, which then stores the page
|
||||
as Home-da.-.md and addresses it as Home%2Dda, so a link to Home-da finds
|
||||
nothing. A space is stored as a hyphen (Home-da.md) and addressed as Home-da.
|
||||
"""
|
||||
return key.replace("-", " ")
|
||||
|
||||
|
||||
def read_pages(wiki_dir):
|
||||
"""Return {page key: content} for every Markdown file in wiki_dir."""
|
||||
pages = {}
|
||||
for path in sorted(wiki_dir.glob("*.md")):
|
||||
key = page_key(path.stem)
|
||||
if key in pages:
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
f"Two files in {wiki_dir} name the wiki page {key}; a space and "
|
||||
"a hyphen are the same in a page name.",
|
||||
)
|
||||
pages[key] = path.read_bytes()
|
||||
if not pages:
|
||||
# Publishing an empty directory would delete the whole wiki.
|
||||
raise WorkflowError(
|
||||
"configuration", f"The wiki source directory {wiki_dir} has no Markdown pages."
|
||||
)
|
||||
return pages
|
||||
|
||||
|
||||
def list_wiki_titles(wiki_url, headers, send):
|
||||
"""Return the keys of the pages that exist in the wiki.
|
||||
|
||||
The key is the address Gitea reports (sub_url), or the title with its
|
||||
spaces replaced by hyphens.
|
||||
|
||||
Gitea answers 404 for a wiki that has never been written to; that is an
|
||||
empty wiki, not a failure.
|
||||
"""
|
||||
titles = set()
|
||||
page = 1
|
||||
while True:
|
||||
url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}"
|
||||
try:
|
||||
batch = send(url, headers=headers)
|
||||
except WorkflowError as error:
|
||||
if error.category == "configuration" and page == 1:
|
||||
return titles
|
||||
raise
|
||||
if not isinstance(batch, list):
|
||||
raise WorkflowError("api", "Gitea returned an invalid wiki page list.")
|
||||
titles.update(
|
||||
urllib.parse.unquote(str(item.get("sub_url") or ""))
|
||||
or page_key(item.get("title", ""))
|
||||
for item in batch
|
||||
)
|
||||
if len(batch) < PAGE_LIMIT:
|
||||
return titles
|
||||
page += 1
|
||||
|
||||
|
||||
def read_wiki_page(wiki_url, title, headers, send):
|
||||
quoted = urllib.parse.quote(title, safe="")
|
||||
page = send(f"{wiki_url}/page/{quoted}", headers=headers)
|
||||
return base64.b64decode((page or {}).get("content_base64", ""))
|
||||
|
||||
|
||||
def publish(pages, wiki_url, headers, send, prune):
|
||||
"""Create and update the pages; delete the others only when prune is set.
|
||||
|
||||
Returns the titles created, updated and deleted, and the titles that were
|
||||
left in the wiki because pruning is off.
|
||||
"""
|
||||
existing = list_wiki_titles(wiki_url, headers, send)
|
||||
current = {
|
||||
title: read_wiki_page(wiki_url, title, headers, send)
|
||||
for title in sorted(existing & pages.keys())
|
||||
}
|
||||
|
||||
result = {"created": [], "updated": [], "deleted": [], "kept": []}
|
||||
for title, content in pages.items():
|
||||
body = {
|
||||
"title": page_title(title),
|
||||
"content_base64": base64.b64encode(content).decode("ascii"),
|
||||
"message": COMMIT_MESSAGE,
|
||||
}
|
||||
if title not in existing:
|
||||
send(f"{wiki_url}/new", method="POST", headers=headers, body=body)
|
||||
result["created"].append(title)
|
||||
elif current[title] != content:
|
||||
quoted = urllib.parse.quote(title, safe="")
|
||||
send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body)
|
||||
result["updated"].append(title)
|
||||
|
||||
for title in sorted(existing - pages.keys()):
|
||||
if not prune:
|
||||
result["kept"].append(title)
|
||||
continue
|
||||
quoted = urllib.parse.quote(title, safe="")
|
||||
send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers)
|
||||
result["deleted"].append(title)
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki"))
|
||||
owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY"))
|
||||
|
||||
marker = wiki_dir / MARKER_NAME
|
||||
if not marker.is_file():
|
||||
print(
|
||||
f"No {wiki_dir}/{MARKER_NAME} marker in {owner}/{repo}; "
|
||||
"wiki publishing is not enabled for this repository."
|
||||
)
|
||||
return 0
|
||||
prune = parse_marker(marker.read_text(encoding="utf-8"))
|
||||
|
||||
pages = read_pages(wiki_dir)
|
||||
token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS"))
|
||||
|
||||
api_url = os.environ["GITEA_API_URL"].rstrip("/")
|
||||
repository_url = (
|
||||
f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/"
|
||||
f"{urllib.parse.quote(repo, safe='')}"
|
||||
)
|
||||
headers = {
|
||||
"Authorization": f"token {token}",
|
||||
"Accept": "application/json",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
try:
|
||||
repository = request_json(repository_url, headers=headers)
|
||||
if isinstance(repository, dict) and repository.get("has_wiki") is False:
|
||||
raise WorkflowError(
|
||||
"configuration",
|
||||
f"The wiki is not enabled on {owner}/{repo}. Enable it in the "
|
||||
"repository settings.",
|
||||
)
|
||||
print("Preflight passed: Gitea accepted the token and the wiki is enabled.")
|
||||
|
||||
result = publish(pages, f"{repository_url}/wiki", headers, request_json, prune)
|
||||
except WorkflowError as error:
|
||||
if error.category == "credentials":
|
||||
raise WorkflowError(
|
||||
"credentials", f"{error.detail} Token shape: {describe_token(token)}."
|
||||
) from None
|
||||
raise
|
||||
|
||||
print(
|
||||
f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: "
|
||||
f"{len(result['created'])} created, {len(result['updated'])} updated, "
|
||||
f"{len(result['deleted'])} deleted."
|
||||
)
|
||||
if result["kept"]:
|
||||
print(
|
||||
f"{len(result['kept'])} page(s) are in the wiki but not in "
|
||||
f"{wiki_dir}/ and were kept; put the word '{PRUNE_WORD}' in "
|
||||
f"{wiki_dir}/{MARKER_NAME} to delete them."
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except WorkflowError as error:
|
||||
print(f"ERROR: {error}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<!-- translation of Onboarding.md, source sha256 1d431898754e -->
|
||||
<!-- translation of Onboarding.md, source sha256 0b11defd563a -->
|
||||
|
||||
> Dette er en oversættelse af [`Onboarding`](Onboarding). Den engelske side er autoritativ, og ændringer her bliver overskrevet, næste gang wikien publiceres.
|
||||
|
||||
@@ -35,7 +35,7 @@ Leveringen er automatisk, så et repository behøver ingen kopi af workflowet. S
|
||||
Workflowet **Publish wiki** kopierer Markdown-siderne i `wiki/` til repositoryets wiki, når en ændring af `wiki/` flettes til `main`. Det leveres som synkroniseringsworkflowet, men skriver kun, hvor et repository har meldt sig til.
|
||||
|
||||
1. Aktivér wikien i repositoryets indstillinger.
|
||||
2. Tilføj Markdown-sider til `wiki/`. Filnavnet er sidens navn: `Home.md` er forsiden.
|
||||
2. Tilføj Markdown-sider til `wiki/`. Filnavnet er sidens navn: `Home.md` er forsiden. En bindestreg i navnet er et mellemrum i sidens navn: `Home-da.md` er siden "Home da", og et link til den er `[Dansk](Home-da)`.
|
||||
3. Tilføj markøren `wiki/.publish`. Lad den være tom for kun at oprette og opdatere sider. Skriv det ene ord `prune` i den for også at slette wiki-sider, der ikke findes i `wiki/`, så wikien altid er lig med mappen.
|
||||
4. Gem `TOKEN_FOR_GITEA_WIKI` som en secret i repositoryets organisation med det scope, der står i [credentials.md](Credentials.da).
|
||||
5. Flet ændringen til `main`. Loggen skal slutte med `Published <n> page(s) to the wiki of <owner>/<repo>`.
|
||||
|
||||
+2
-1
@@ -57,7 +57,8 @@ like the sync workflow, but it writes only where a repository opts in.
|
||||
|
||||
1. Enable the wiki in the repository settings.
|
||||
2. Add Markdown pages to `wiki/`. The file name is the page name: `Home.md` is
|
||||
the home page.
|
||||
the home page. A hyphen in the name is a space in the page name: `Home-da.md`
|
||||
is the page "Home da", and a link to it is `[Dansk](https://git.tirsystem.com/TirSystem/github-action/src/branch/main/guides/Home-da)`.
|
||||
3. Add the marker `wiki/.publish`. Leave it empty to create and update pages
|
||||
only. Put the single word `prune` in it to also delete wiki pages that are
|
||||
not in `wiki/`, so the wiki always equals the folder.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<!-- translation of Onboarding.md, source sha256 1d431898754e -->
|
||||
<!-- translation of Onboarding.md, source sha256 0b11defd563a -->
|
||||
|
||||
> Ini ialah terjemahan [`Onboarding`](Onboarding). Halaman bahasa Inggeris ialah teks yang sah, dan perubahan di sini akan ditimpa pada penerbitan wiki yang seterusnya.
|
||||
|
||||
@@ -35,7 +35,7 @@ Penghantaran adalah automatik, jadi repositori tidak memerlukan salinan aliran k
|
||||
Aliran kerja **Publish wiki** menyalin halaman Markdown dalam `wiki/` ke wiki repositori apabila perubahan pada `wiki/` digabungkan ke `main`. Ia dihantar seperti aliran kerja penyegerakan, tetapi hanya menulis di tempat repositori telah memilih untuk menyertainya.
|
||||
|
||||
1. Aktifkan wiki dalam tetapan repositori.
|
||||
2. Tambah halaman Markdown ke `wiki/`. Nama fail ialah nama halaman: `Home.md` ialah halaman utama.
|
||||
2. Tambah halaman Markdown ke `wiki/`. Nama fail ialah nama halaman: `Home.md` ialah halaman utama. Tanda sempang dalam nama ialah ruang dalam nama halaman: `Home-da.md` ialah halaman "Home da", dan pautan kepadanya ialah `[Dansk](Home-da)`.
|
||||
3. Tambah penanda `wiki/.publish`. Biarkan kosong untuk mencipta dan mengemas kini halaman sahaja. Letakkan satu perkataan `prune` di dalamnya untuk turut memadam halaman wiki yang tiada dalam `wiki/`, supaya wiki sentiasa sama dengan folder itu.
|
||||
4. Simpan `TOKEN_FOR_GITEA_WIKI` sebagai secret organisasi repositori, dengan skop yang tertera dalam [credentials.md](Credentials.ms).
|
||||
5. Gabungkan perubahan ke `main`. Log mesti berakhir dengan `Published <n> page(s) to the wiki of <owner>/<repo>`.
|
||||
|
||||
Reference in New Issue
Block a user