Add tools/publish_wiki.py and the Publish wiki workflow, which make the Gitea wiki equal wiki/ through the wiki REST API when a change to wiki/ is merged to main. It needs its own secret TOKEN_FOR_GITEA_WIKI, refuses an empty source and checks that the English pages are current first. Document the secret in the credentials and troubleshooting guides and regenerate the affected wiki pages. Task: MIL-005#8
This commit is contained in:
@@ -0,0 +1,31 @@
|
|||||||
|
name: Publish wiki
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [ main ]
|
||||||
|
paths:
|
||||||
|
- "wiki/**"
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: publish-wiki
|
||||||
|
cancel-in-progress: false
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
publish-wiki:
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Check out the repository
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Check that the English pages are current
|
||||||
|
run: python3 tools/build_wiki.py --check
|
||||||
|
|
||||||
|
- name: Publish the pages to the wiki
|
||||||
|
env:
|
||||||
|
GITEA_API_URL: ${{ gitea.api_url }}
|
||||||
|
SOURCE_REPOSITORY: ${{ gitea.repository }}
|
||||||
|
WIKI_CREDENTIALS: ${{ secrets.TOKEN_FOR_GITEA_WIKI }}
|
||||||
|
run: python3 tools/publish_wiki.py
|
||||||
@@ -25,6 +25,7 @@ It currently contains the Gitea workflow [`sync-github-metadata.yml`](.gitea/sco
|
|||||||
- Repository secrets:
|
- Repository secrets:
|
||||||
- `TOKEN_FOR_GITEA`: Gitea API token (plain string, or JSON with `GITEA_TOKEN`)
|
- `TOKEN_FOR_GITEA`: Gitea API token (plain string, or JSON with `GITEA_TOKEN`)
|
||||||
- `CREDENTIALS_FOR_GITHUB`: JSON object containing `GITHUB_PAT`, a GitHub personal access token allowed to edit the mirror repository
|
- `CREDENTIALS_FOR_GITHUB`: JSON object containing `GITHUB_PAT`, a GitHub personal access token allowed to edit the mirror repository
|
||||||
|
- `TOKEN_FOR_GITEA_WIKI` (only in this repository, for the **Publish wiki** workflow): Gitea token that can write the repository wiki
|
||||||
|
|
||||||
## 🛠️ Setup
|
## 🛠️ Setup
|
||||||
|
|
||||||
|
|||||||
@@ -56,6 +56,28 @@ credentials.` Failures start with a category such as `[credentials]` or
|
|||||||
`[permissions]`; see [troubleshooting.md](troubleshooting.md). No secret value
|
`[permissions]`; see [troubleshooting.md](troubleshooting.md). No secret value
|
||||||
is ever printed.
|
is ever printed.
|
||||||
|
|
||||||
|
## Wiki publishing secret
|
||||||
|
|
||||||
|
The workflow **Publish wiki** (this repository only) copies `wiki/` to the Gitea
|
||||||
|
wiki when a change to `wiki/` is merged to `main`. It uses its own secret so
|
||||||
|
that the metadata sync tokens never get write access.
|
||||||
|
|
||||||
|
| Secret | Content | Used for |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `TOKEN_FOR_GITEA_WIKI` | A bare token, a JSON string, or JSON with `GITEA_TOKEN` | Writing the wiki pages of this repository |
|
||||||
|
|
||||||
|
- A Gitea personal access token with scope `write:repository`. Gitea has no
|
||||||
|
wiki-only scope, so create the token for a dedicated user that has write
|
||||||
|
access to this repository only. S05 approves the scope and S03 stores the
|
||||||
|
secret.
|
||||||
|
- Calls made: `GET /repos/{owner}/{repo}`, `GET /repos/{owner}/{repo}/wiki/pages`,
|
||||||
|
`GET`, `POST` and `PATCH /repos/{owner}/{repo}/wiki/...` and, for a page that
|
||||||
|
no longer exists in `wiki/`, `DELETE /repos/{owner}/{repo}/wiki/page/{name}`.
|
||||||
|
- The wiki must be enabled in the repository settings. The workflow refuses to
|
||||||
|
run when it is not.
|
||||||
|
- The wiki always equals `wiki/`: pages edited or added in the Gitea web UI are
|
||||||
|
overwritten or removed on the next publish.
|
||||||
|
|
||||||
## Rotating a token
|
## Rotating a token
|
||||||
|
|
||||||
1. Create the new token with the permissions above.
|
1. Create the new token with the permissions above.
|
||||||
|
|||||||
@@ -30,6 +30,9 @@ never starts, or stops before the step) is a **runner** problem.
|
|||||||
| `TOKEN_FOR_GITEA must contain a non-empty GITEA_TOKEN.` | configuration | The JSON form needs a `GITEA_TOKEN` key |
|
| `TOKEN_FOR_GITEA must contain a non-empty GITEA_TOKEN.` | configuration | The JSON form needs a `GITEA_TOKEN` key |
|
||||||
| `Could not determine the Gitea source repository.` | configuration | The run has no repository name; report as a defect |
|
| `Could not determine the Gitea source repository.` | configuration | The run has no repository name; report as a defect |
|
||||||
| `Expected at most one GitHub push mirror ... found N` | configuration | Keep one GitHub push mirror |
|
| `Expected at most one GitHub push mirror ... found N` | configuration | Keep one GitHub push mirror |
|
||||||
|
| `TOKEN_FOR_GITEA_WIKI is missing or empty.` | configuration | Wiki publishing only: add the secret (see [credentials.md](credentials.md)) |
|
||||||
|
| `The wiki is not enabled on <repo>` | configuration | Wiki publishing only: enable the wiki in the repository settings |
|
||||||
|
| `The wiki source directory ... has no Markdown pages` | configuration | Wiki publishing only: `wiki/` is empty or missing; nothing was changed |
|
||||||
| `Could not determine the GitHub owner and repository ...` | configuration | The mirror address must look like `https://github.com/<owner>/<repo>.git` |
|
| `Could not determine the GitHub owner and repository ...` | configuration | The mirror address must look like `https://github.com/<owner>/<repo>.git` |
|
||||||
| `Gitea API request failed with HTTP 401` | credentials | Create a new Gitea token and update `TOKEN_FOR_GITEA` |
|
| `Gitea API request failed with HTTP 401` | credentials | Create a new Gitea token and update `TOKEN_FOR_GITEA` |
|
||||||
| `GitHub API request failed with HTTP 401` | credentials | Create a new GitHub token and update `CREDENTIALS_FOR_GITHUB` |
|
| `GitHub API request failed with HTTP 401` | credentials | Create a new GitHub token and update `CREDENTIALS_FOR_GITHUB` |
|
||||||
|
|||||||
@@ -0,0 +1,262 @@
|
|||||||
|
"""Offline tests for tools/publish_wiki.py and its workflow."""
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import importlib.util
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import urllib.error
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "publish-wiki.yml"
|
||||||
|
_SPEC = importlib.util.spec_from_file_location(
|
||||||
|
"publish_wiki", REPO_ROOT / "tools" / "publish_wiki.py"
|
||||||
|
)
|
||||||
|
assert _SPEC is not None and _SPEC.loader is not None
|
||||||
|
publish_wiki = importlib.util.module_from_spec(_SPEC)
|
||||||
|
_SPEC.loader.exec_module(publish_wiki)
|
||||||
|
|
||||||
|
API = "https://git.example.test/api/v1"
|
||||||
|
REPO_URL = f"{API}/repos/TirSystem/repo"
|
||||||
|
WIKI_URL = f"{REPO_URL}/wiki"
|
||||||
|
|
||||||
|
|
||||||
|
def b64(text: str) -> str:
|
||||||
|
return base64.b64encode(text.encode()).decode()
|
||||||
|
|
||||||
|
|
||||||
|
class TestParseGiteaToken:
|
||||||
|
def test_accepts_bare_token(self):
|
||||||
|
assert publish_wiki.parse_gitea_token(" abc ") == "abc"
|
||||||
|
|
||||||
|
def test_accepts_json_object_and_string(self):
|
||||||
|
assert publish_wiki.parse_gitea_token('{"GITEA_TOKEN": "abc"}') == "abc"
|
||||||
|
assert publish_wiki.parse_gitea_token('"abc"') == "abc"
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("raw", [None, "", " ", "{}", '{"GITEA_TOKEN": ""}', "[1]"])
|
||||||
|
def test_rejects_missing_or_empty_token(self, raw):
|
||||||
|
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||||
|
publish_wiki.parse_gitea_token(raw)
|
||||||
|
assert caught.value.category == "configuration"
|
||||||
|
assert "TOKEN_FOR_GITEA_WIKI" in str(caught.value)
|
||||||
|
|
||||||
|
|
||||||
|
class TestReadPages:
|
||||||
|
def test_reads_markdown_files_by_stem(self, tmp_path):
|
||||||
|
(tmp_path / "Home.md").write_bytes(b"home")
|
||||||
|
(tmp_path / "Notes.txt").write_bytes(b"ignored")
|
||||||
|
assert publish_wiki.read_pages(tmp_path) == {"Home": b"home"}
|
||||||
|
|
||||||
|
def test_rejects_missing_directory(self, tmp_path):
|
||||||
|
with pytest.raises(publish_wiki.WorkflowError, match="missing"):
|
||||||
|
publish_wiki.read_pages(tmp_path / "nope")
|
||||||
|
|
||||||
|
def test_rejects_empty_directory_so_the_wiki_is_never_wiped(self, tmp_path):
|
||||||
|
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
|
||||||
|
publish_wiki.read_pages(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
class FakeGitea:
|
||||||
|
"""Answers like the Gitea repository and wiki endpoints."""
|
||||||
|
|
||||||
|
def __init__(self, wiki=None, has_wiki=True, wiki_initialised=True):
|
||||||
|
self.wiki = dict(wiki or {})
|
||||||
|
self.has_wiki = has_wiki
|
||||||
|
self.wiki_initialised = wiki_initialised
|
||||||
|
self.calls = []
|
||||||
|
|
||||||
|
def __call__(self, url, method="GET", headers=None, body=None):
|
||||||
|
self.calls.append((method, url, body))
|
||||||
|
if url == REPO_URL:
|
||||||
|
return {"has_wiki": self.has_wiki}
|
||||||
|
if url.startswith(f"{WIKI_URL}/pages"):
|
||||||
|
if not self.wiki_initialised:
|
||||||
|
raise publish_wiki.WorkflowError("configuration", "404")
|
||||||
|
return [{"title": title} for title in self.wiki]
|
||||||
|
if method == "POST" and url == f"{WIKI_URL}/new":
|
||||||
|
self.wiki[body["title"]] = body["content_base64"]
|
||||||
|
return None
|
||||||
|
title = url.rsplit("/", 1)[-1]
|
||||||
|
if method == "GET":
|
||||||
|
return {"title": title, "content_base64": self.wiki[title]}
|
||||||
|
if method == "PATCH":
|
||||||
|
self.wiki[title] = body["content_base64"]
|
||||||
|
return None
|
||||||
|
if method == "DELETE":
|
||||||
|
del self.wiki[title]
|
||||||
|
return None
|
||||||
|
raise AssertionError(f"unexpected request {method} {url}")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def environment(monkeypatch, tmp_path):
|
||||||
|
wiki_dir = tmp_path / "wiki"
|
||||||
|
wiki_dir.mkdir()
|
||||||
|
(wiki_dir / "Home.md").write_text("home v2", encoding="utf-8")
|
||||||
|
(wiki_dir / "Credentials.md").write_text("credentials", encoding="utf-8")
|
||||||
|
monkeypatch.setenv("WIKI_DIR", str(wiki_dir))
|
||||||
|
monkeypatch.setenv("GITEA_API_URL", f"{API}/")
|
||||||
|
monkeypatch.setenv("SOURCE_REPOSITORY", "TirSystem/repo")
|
||||||
|
monkeypatch.setenv("WIKI_CREDENTIALS", "wiki-token")
|
||||||
|
return wiki_dir
|
||||||
|
|
||||||
|
|
||||||
|
def run_main(monkeypatch, gitea):
|
||||||
|
monkeypatch.setattr(publish_wiki, "request_json", gitea)
|
||||||
|
return publish_wiki.main()
|
||||||
|
|
||||||
|
|
||||||
|
class TestMain:
|
||||||
|
def test_creates_updates_and_deletes_to_match_the_source(
|
||||||
|
self, monkeypatch, environment, capsys
|
||||||
|
):
|
||||||
|
gitea = FakeGitea(
|
||||||
|
wiki={"Home": b64("home v1"), "Credentials": b64("credentials"), "Old": b64("x")}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert run_main(monkeypatch, gitea) == 0
|
||||||
|
|
||||||
|
assert gitea.wiki == {"Home": b64("home v2"), "Credentials": b64("credentials")}
|
||||||
|
methods = [(call[0], call[1].rsplit("/", 1)[-1]) for call in gitea.calls]
|
||||||
|
assert ("PATCH", "Home") in methods
|
||||||
|
assert ("DELETE", "Old") in methods
|
||||||
|
assert ("PATCH", "Credentials") not in methods
|
||||||
|
assert "0 created, 1 updated, 1 deleted" in capsys.readouterr().out
|
||||||
|
|
||||||
|
def test_creates_every_page_when_the_wiki_was_never_written(
|
||||||
|
self, monkeypatch, environment
|
||||||
|
):
|
||||||
|
gitea = FakeGitea(wiki_initialised=False)
|
||||||
|
|
||||||
|
assert run_main(monkeypatch, gitea) == 0
|
||||||
|
|
||||||
|
assert set(gitea.wiki) == {"Home", "Credentials"}
|
||||||
|
assert [call[0] for call in gitea.calls].count("POST") == 2
|
||||||
|
|
||||||
|
def test_does_nothing_when_the_wiki_is_already_current(self, monkeypatch, environment):
|
||||||
|
gitea = FakeGitea(wiki={"Home": b64("home v2"), "Credentials": b64("credentials")})
|
||||||
|
|
||||||
|
assert run_main(monkeypatch, gitea) == 0
|
||||||
|
|
||||||
|
assert all(call[0] == "GET" for call in gitea.calls)
|
||||||
|
|
||||||
|
def test_sends_the_token_and_never_prints_it(self, monkeypatch, environment, capsys):
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
class Recording(FakeGitea):
|
||||||
|
def __call__(self, url, method="GET", headers=None, body=None):
|
||||||
|
seen.append(headers["Authorization"])
|
||||||
|
return super().__call__(url, method, headers, body)
|
||||||
|
|
||||||
|
assert run_main(monkeypatch, Recording()) == 0
|
||||||
|
|
||||||
|
assert set(seen) == {"token wiki-token"}
|
||||||
|
output = capsys.readouterr()
|
||||||
|
assert "wiki-token" not in output.out + output.err
|
||||||
|
|
||||||
|
def test_fails_when_the_wiki_is_not_enabled(self, monkeypatch, environment):
|
||||||
|
gitea = FakeGitea(has_wiki=False)
|
||||||
|
|
||||||
|
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||||
|
run_main(monkeypatch, gitea)
|
||||||
|
|
||||||
|
assert caught.value.category == "configuration"
|
||||||
|
assert all(call[0] == "GET" for call in gitea.calls)
|
||||||
|
|
||||||
|
def test_fails_before_any_request_when_the_secret_is_missing(
|
||||||
|
self, monkeypatch, environment
|
||||||
|
):
|
||||||
|
monkeypatch.setenv("WIKI_CREDENTIALS", "")
|
||||||
|
gitea = FakeGitea()
|
||||||
|
|
||||||
|
with pytest.raises(publish_wiki.WorkflowError, match="TOKEN_FOR_GITEA_WIKI"):
|
||||||
|
run_main(monkeypatch, gitea)
|
||||||
|
|
||||||
|
assert gitea.calls == []
|
||||||
|
|
||||||
|
def test_fails_before_any_request_when_the_source_is_empty(
|
||||||
|
self, monkeypatch, environment
|
||||||
|
):
|
||||||
|
for path in environment.glob("*.md"):
|
||||||
|
path.unlink()
|
||||||
|
gitea = FakeGitea(wiki={"Home": b64("keep me")})
|
||||||
|
|
||||||
|
with pytest.raises(publish_wiki.WorkflowError, match="no Markdown pages"):
|
||||||
|
run_main(monkeypatch, gitea)
|
||||||
|
|
||||||
|
assert gitea.calls == []
|
||||||
|
assert gitea.wiki == {"Home": b64("keep me")}
|
||||||
|
|
||||||
|
|
||||||
|
class TestRequestJson:
|
||||||
|
@staticmethod
|
||||||
|
def raise_http_error(monkeypatch, code, reason):
|
||||||
|
def fake_urlopen(request, timeout):
|
||||||
|
raise urllib.error.HTTPError(
|
||||||
|
request.full_url, code, reason, {}, io.BytesIO(b"secret detail")
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("code", "reason", "category"),
|
||||||
|
[
|
||||||
|
(401, "Unauthorized", "credentials"),
|
||||||
|
(403, "Forbidden", "permissions"),
|
||||||
|
(404, "Not Found", "configuration"),
|
||||||
|
(500, "Server Error", "api"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_categorizes_http_errors(self, monkeypatch, code, reason, category):
|
||||||
|
self.raise_http_error(monkeypatch, code, reason)
|
||||||
|
|
||||||
|
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||||
|
publish_wiki.request_json(REPO_URL)
|
||||||
|
|
||||||
|
assert caught.value.category == category
|
||||||
|
assert f"HTTP {code}" in str(caught.value)
|
||||||
|
assert "secret detail" not in str(caught.value)
|
||||||
|
|
||||||
|
def test_reports_unreachable_service_as_api_failure(self, monkeypatch):
|
||||||
|
def fake_urlopen(request, timeout):
|
||||||
|
raise urllib.error.URLError("no route")
|
||||||
|
|
||||||
|
monkeypatch.setattr(publish_wiki.urllib.request, "urlopen", fake_urlopen)
|
||||||
|
|
||||||
|
with pytest.raises(publish_wiki.WorkflowError) as caught:
|
||||||
|
publish_wiki.request_json(REPO_URL)
|
||||||
|
|
||||||
|
assert caught.value.category == "api"
|
||||||
|
|
||||||
|
|
||||||
|
class TestWorkflow:
|
||||||
|
@pytest.fixture()
|
||||||
|
def document(self):
|
||||||
|
return yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
def test_runs_only_on_main_when_wiki_changes_or_by_hand(self, document):
|
||||||
|
triggers = document[True] # PyYAML reads the key `on` as True
|
||||||
|
assert triggers["push"]["branches"] == ["main"]
|
||||||
|
assert triggers["push"]["paths"] == ["wiki/**"]
|
||||||
|
assert "workflow_dispatch" in triggers
|
||||||
|
|
||||||
|
def test_job_has_read_only_permissions(self, document):
|
||||||
|
job = document["jobs"]["publish-wiki"]
|
||||||
|
assert job["permissions"] == {"contents": "read"}
|
||||||
|
|
||||||
|
def test_checks_the_pages_before_publishing_them(self, document):
|
||||||
|
runs = [step.get("run", "") for step in document["jobs"]["publish-wiki"]["steps"]]
|
||||||
|
assert runs.index("python3 tools/build_wiki.py --check") < runs.index(
|
||||||
|
"python3 tools/publish_wiki.py"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_secret_is_passed_only_through_the_environment(self, document):
|
||||||
|
text = WORKFLOW.read_text(encoding="utf-8")
|
||||||
|
assert text.count("secrets.TOKEN_FOR_GITEA_WIKI") == 1
|
||||||
|
steps = document["jobs"]["publish-wiki"]["steps"]
|
||||||
|
publish_step = steps[-1]
|
||||||
|
assert publish_step["env"]["WIKI_CREDENTIALS"] == "${{ secrets.TOKEN_FOR_GITEA_WIKI }}"
|
||||||
|
assert "secrets" not in publish_step["run"]
|
||||||
@@ -0,0 +1,240 @@
|
|||||||
|
"""Publish the pages in ``wiki/`` to the Gitea wiki of this repository.
|
||||||
|
|
||||||
|
The pages are written through the Gitea wiki REST API. A page that is missing
|
||||||
|
in ``wiki/`` is removed from the wiki, so the wiki always equals the reviewed
|
||||||
|
source; edits made in the web UI are overwritten (MIL-005).
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
GITEA_API_URL Gitea API base, for example https://host/api/v1
|
||||||
|
SOURCE_REPOSITORY owner/repo of the repository whose wiki is written
|
||||||
|
WIKI_CREDENTIALS bare token, JSON string, or JSON with GITEA_TOKEN
|
||||||
|
WIKI_DIR directory with the pages (default: wiki)
|
||||||
|
|
||||||
|
Exit codes: 0 published or nothing to do, 1 failed (message names a category).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
from collections.abc import Callable
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
PAGE_LIMIT = 50
|
||||||
|
REQUEST_TIMEOUT_SECONDS = 30
|
||||||
|
COMMIT_MESSAGE = "Publish wiki from the repository"
|
||||||
|
|
||||||
|
JsonRequest = Callable[..., Any]
|
||||||
|
|
||||||
|
|
||||||
|
class WorkflowError(RuntimeError):
|
||||||
|
"""A failure with a category.
|
||||||
|
|
||||||
|
configuration: a secret, a setting or the wiki source is missing or malformed.
|
||||||
|
credentials: Gitea rejected the token (HTTP 401).
|
||||||
|
permissions: the token is valid but not allowed (HTTP 403).
|
||||||
|
api: any other API or network failure.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, category: str, message: str) -> None:
|
||||||
|
super().__init__(f"[{category}] {message}")
|
||||||
|
self.category = category
|
||||||
|
|
||||||
|
|
||||||
|
def request_json(
|
||||||
|
url: str,
|
||||||
|
method: str = "GET",
|
||||||
|
headers: dict[str, str] | None = None,
|
||||||
|
body: dict[str, Any] | None = None,
|
||||||
|
) -> Any:
|
||||||
|
"""Send one JSON request to Gitea and return the decoded response."""
|
||||||
|
request = urllib.request.Request(
|
||||||
|
url,
|
||||||
|
data=json.dumps(body).encode("utf-8") if body is not None else None,
|
||||||
|
headers=headers or {},
|
||||||
|
method=method,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response:
|
||||||
|
content = response.read()
|
||||||
|
return json.loads(content) if content else None
|
||||||
|
except urllib.error.HTTPError as error:
|
||||||
|
detail = f"Gitea API request failed with HTTP {error.code} ({error.reason})"
|
||||||
|
if error.code == 401:
|
||||||
|
raise WorkflowError("credentials", f"{detail}. The token was rejected.") from None
|
||||||
|
if error.code == 403:
|
||||||
|
raise WorkflowError(
|
||||||
|
"permissions", f"{detail}. The token lacks a required permission."
|
||||||
|
) from None
|
||||||
|
if error.code == 404:
|
||||||
|
raise WorkflowError(
|
||||||
|
"configuration",
|
||||||
|
f"{detail}. The repository or page was not found, or the token "
|
||||||
|
"cannot see it.",
|
||||||
|
) from None
|
||||||
|
raise WorkflowError("api", detail) from None
|
||||||
|
except (urllib.error.URLError, TimeoutError) as error:
|
||||||
|
raise WorkflowError("api", f"Gitea API is unreachable ({error}).") from None
|
||||||
|
|
||||||
|
|
||||||
|
def parse_gitea_token(raw_credentials: str | None) -> str:
|
||||||
|
"""Return the token from a bare value, a JSON string or JSON with GITEA_TOKEN."""
|
||||||
|
raw_credentials = (raw_credentials or "").strip()
|
||||||
|
if not raw_credentials:
|
||||||
|
raise WorkflowError("configuration", "TOKEN_FOR_GITEA_WIKI is missing or empty.")
|
||||||
|
|
||||||
|
token: Any
|
||||||
|
try:
|
||||||
|
credentials = json.loads(raw_credentials)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
token = raw_credentials
|
||||||
|
else:
|
||||||
|
if isinstance(credentials, dict):
|
||||||
|
token = credentials.get("GITEA_TOKEN")
|
||||||
|
elif isinstance(credentials, str):
|
||||||
|
token = credentials
|
||||||
|
else:
|
||||||
|
token = None
|
||||||
|
|
||||||
|
if not isinstance(token, str) or not token.strip():
|
||||||
|
raise WorkflowError(
|
||||||
|
"configuration", "TOKEN_FOR_GITEA_WIKI must contain a non-empty GITEA_TOKEN."
|
||||||
|
)
|
||||||
|
return token.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def split_repository(full_name: str | None) -> tuple[str, str]:
|
||||||
|
"""Split ``owner/repo`` into its two parts."""
|
||||||
|
owner, separator, repo = (full_name or "").partition("/")
|
||||||
|
if not separator or not owner or not repo:
|
||||||
|
raise WorkflowError("configuration", "Could not determine the Gitea repository.")
|
||||||
|
return owner, repo
|
||||||
|
|
||||||
|
|
||||||
|
def read_pages(wiki_dir: Path) -> dict[str, bytes]:
|
||||||
|
"""Return ``{page title: content}`` for every Markdown file in ``wiki_dir``."""
|
||||||
|
if not wiki_dir.is_dir():
|
||||||
|
raise WorkflowError("configuration", f"The wiki source directory {wiki_dir} is missing.")
|
||||||
|
pages = {path.stem: path.read_bytes() for path in sorted(wiki_dir.glob("*.md"))}
|
||||||
|
if not pages:
|
||||||
|
# Publishing an empty directory would delete the whole wiki.
|
||||||
|
raise WorkflowError(
|
||||||
|
"configuration", f"The wiki source directory {wiki_dir} has no Markdown pages."
|
||||||
|
)
|
||||||
|
return pages
|
||||||
|
|
||||||
|
|
||||||
|
def list_wiki_titles(wiki_url: str, headers: dict[str, str], send: JsonRequest) -> set[str]:
|
||||||
|
"""Return the titles of the pages that exist in the wiki.
|
||||||
|
|
||||||
|
Gitea answers 404 for a wiki that has never been written to; that is an
|
||||||
|
empty wiki, not a failure.
|
||||||
|
"""
|
||||||
|
titles: set[str] = set()
|
||||||
|
page = 1
|
||||||
|
while True:
|
||||||
|
url = f"{wiki_url}/pages?page={page}&limit={PAGE_LIMIT}"
|
||||||
|
try:
|
||||||
|
batch = send(url, headers=headers)
|
||||||
|
except WorkflowError as error:
|
||||||
|
if error.category == "configuration" and page == 1:
|
||||||
|
return titles
|
||||||
|
raise
|
||||||
|
if not isinstance(batch, list):
|
||||||
|
raise WorkflowError("api", "Gitea returned an invalid wiki page list.")
|
||||||
|
titles.update(str(item.get("title", "")) for item in batch)
|
||||||
|
if len(batch) < PAGE_LIMIT:
|
||||||
|
return titles
|
||||||
|
page += 1
|
||||||
|
|
||||||
|
|
||||||
|
def read_wiki_page(
|
||||||
|
wiki_url: str, title: str, headers: dict[str, str], send: JsonRequest
|
||||||
|
) -> bytes:
|
||||||
|
"""Return the current content of one wiki page."""
|
||||||
|
quoted = urllib.parse.quote(title, safe="")
|
||||||
|
page = send(f"{wiki_url}/page/{quoted}", headers=headers)
|
||||||
|
encoded = (page or {}).get("content_base64", "")
|
||||||
|
return base64.b64decode(encoded)
|
||||||
|
|
||||||
|
|
||||||
|
def publish(
|
||||||
|
pages: dict[str, bytes],
|
||||||
|
wiki_url: str,
|
||||||
|
headers: dict[str, str],
|
||||||
|
send: JsonRequest,
|
||||||
|
) -> dict[str, list[str]]:
|
||||||
|
"""Make the wiki equal ``pages``; return the titles created, updated, deleted."""
|
||||||
|
existing = list_wiki_titles(wiki_url, headers, send)
|
||||||
|
current = {
|
||||||
|
title: read_wiki_page(wiki_url, title, headers, send)
|
||||||
|
for title in sorted(existing & pages.keys())
|
||||||
|
}
|
||||||
|
|
||||||
|
result: dict[str, list[str]] = {"created": [], "updated": [], "deleted": []}
|
||||||
|
for title, content in pages.items():
|
||||||
|
body = {
|
||||||
|
"title": title,
|
||||||
|
"content_base64": base64.b64encode(content).decode("ascii"),
|
||||||
|
"message": COMMIT_MESSAGE,
|
||||||
|
}
|
||||||
|
if title not in existing:
|
||||||
|
send(f"{wiki_url}/new", method="POST", headers=headers, body=body)
|
||||||
|
result["created"].append(title)
|
||||||
|
elif current[title] != content:
|
||||||
|
quoted = urllib.parse.quote(title, safe="")
|
||||||
|
send(f"{wiki_url}/page/{quoted}", method="PATCH", headers=headers, body=body)
|
||||||
|
result["updated"].append(title)
|
||||||
|
|
||||||
|
for title in sorted(existing - pages.keys()):
|
||||||
|
quoted = urllib.parse.quote(title, safe="")
|
||||||
|
send(f"{wiki_url}/page/{quoted}", method="DELETE", headers=headers)
|
||||||
|
result["deleted"].append(title)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
"""Run the publish; return the process exit code."""
|
||||||
|
wiki_dir = Path(os.environ.get("WIKI_DIR", "wiki"))
|
||||||
|
pages = read_pages(wiki_dir)
|
||||||
|
token = parse_gitea_token(os.environ.get("WIKI_CREDENTIALS"))
|
||||||
|
owner, repo = split_repository(os.environ.get("SOURCE_REPOSITORY"))
|
||||||
|
|
||||||
|
api_url = os.environ["GITEA_API_URL"].rstrip("/")
|
||||||
|
repository_url = (
|
||||||
|
f"{api_url}/repos/{urllib.parse.quote(owner, safe='')}/"
|
||||||
|
f"{urllib.parse.quote(repo, safe='')}"
|
||||||
|
)
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"token {token}",
|
||||||
|
"Accept": "application/json",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
}
|
||||||
|
|
||||||
|
repository = request_json(repository_url, headers=headers)
|
||||||
|
if isinstance(repository, dict) and repository.get("has_wiki") is False:
|
||||||
|
raise WorkflowError(
|
||||||
|
"configuration",
|
||||||
|
f"The wiki is not enabled on {owner}/{repo}. Enable it in the repository settings.",
|
||||||
|
)
|
||||||
|
print("Preflight passed: Gitea accepted the token and the wiki is enabled.")
|
||||||
|
|
||||||
|
result = publish(pages, f"{repository_url}/wiki", headers, request_json)
|
||||||
|
print(
|
||||||
|
f"Published {len(pages)} page(s) to the wiki of {owner}/{repo}: "
|
||||||
|
f"{len(result['created'])} created, {len(result['updated'])} updated, "
|
||||||
|
f"{len(result['deleted'])} deleted."
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
sys.exit(main())
|
||||||
|
except WorkflowError as error:
|
||||||
|
print(f"ERROR: {error}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
@@ -60,6 +60,28 @@ credentials.` Failures start with a category such as `[credentials]` or
|
|||||||
`[permissions]`; see [troubleshooting.md](Troubleshooting). No secret value
|
`[permissions]`; see [troubleshooting.md](Troubleshooting). No secret value
|
||||||
is ever printed.
|
is ever printed.
|
||||||
|
|
||||||
|
## Wiki publishing secret
|
||||||
|
|
||||||
|
The workflow **Publish wiki** (this repository only) copies `wiki/` to the Gitea
|
||||||
|
wiki when a change to `wiki/` is merged to `main`. It uses its own secret so
|
||||||
|
that the metadata sync tokens never get write access.
|
||||||
|
|
||||||
|
| Secret | Content | Used for |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `TOKEN_FOR_GITEA_WIKI` | A bare token, a JSON string, or JSON with `GITEA_TOKEN` | Writing the wiki pages of this repository |
|
||||||
|
|
||||||
|
- A Gitea personal access token with scope `write:repository`. Gitea has no
|
||||||
|
wiki-only scope, so create the token for a dedicated user that has write
|
||||||
|
access to this repository only. S05 approves the scope and S03 stores the
|
||||||
|
secret.
|
||||||
|
- Calls made: `GET /repos/{owner}/{repo}`, `GET /repos/{owner}/{repo}/wiki/pages`,
|
||||||
|
`GET`, `POST` and `PATCH /repos/{owner}/{repo}/wiki/...` and, for a page that
|
||||||
|
no longer exists in `wiki/`, `DELETE /repos/{owner}/{repo}/wiki/page/{name}`.
|
||||||
|
- The wiki must be enabled in the repository settings. The workflow refuses to
|
||||||
|
run when it is not.
|
||||||
|
- The wiki always equals `wiki/`: pages edited or added in the Gitea web UI are
|
||||||
|
overwritten or removed on the next publish.
|
||||||
|
|
||||||
## Rotating a token
|
## Rotating a token
|
||||||
|
|
||||||
1. Create the new token with the permissions above.
|
1. Create the new token with the permissions above.
|
||||||
|
|||||||
@@ -21,6 +21,7 @@ It currently contains the Gitea workflow [`sync-github-metadata.yml`](https://gi
|
|||||||
- Repository secrets:
|
- Repository secrets:
|
||||||
- `TOKEN_FOR_GITEA`: Gitea API token (plain string, or JSON with `GITEA_TOKEN`)
|
- `TOKEN_FOR_GITEA`: Gitea API token (plain string, or JSON with `GITEA_TOKEN`)
|
||||||
- `CREDENTIALS_FOR_GITHUB`: JSON object containing `GITHUB_PAT`, a GitHub personal access token allowed to edit the mirror repository
|
- `CREDENTIALS_FOR_GITHUB`: JSON object containing `GITHUB_PAT`, a GitHub personal access token allowed to edit the mirror repository
|
||||||
|
- `TOKEN_FOR_GITEA_WIKI` (only in this repository, for the **Publish wiki** workflow): Gitea token that can write the repository wiki
|
||||||
|
|
||||||
|
|
||||||
## Setup
|
## Setup
|
||||||
|
|||||||
@@ -34,6 +34,9 @@ never starts, or stops before the step) is a **runner** problem.
|
|||||||
| `TOKEN_FOR_GITEA must contain a non-empty GITEA_TOKEN.` | configuration | The JSON form needs a `GITEA_TOKEN` key |
|
| `TOKEN_FOR_GITEA must contain a non-empty GITEA_TOKEN.` | configuration | The JSON form needs a `GITEA_TOKEN` key |
|
||||||
| `Could not determine the Gitea source repository.` | configuration | The run has no repository name; report as a defect |
|
| `Could not determine the Gitea source repository.` | configuration | The run has no repository name; report as a defect |
|
||||||
| `Expected at most one GitHub push mirror ... found N` | configuration | Keep one GitHub push mirror |
|
| `Expected at most one GitHub push mirror ... found N` | configuration | Keep one GitHub push mirror |
|
||||||
|
| `TOKEN_FOR_GITEA_WIKI is missing or empty.` | configuration | Wiki publishing only: add the secret (see [credentials.md](Credentials)) |
|
||||||
|
| `The wiki is not enabled on <repo>` | configuration | Wiki publishing only: enable the wiki in the repository settings |
|
||||||
|
| `The wiki source directory ... has no Markdown pages` | configuration | Wiki publishing only: `wiki/` is empty or missing; nothing was changed |
|
||||||
| `Could not determine the GitHub owner and repository ...` | configuration | The mirror address must look like `https://github.com/<owner>/<repo>.git` |
|
| `Could not determine the GitHub owner and repository ...` | configuration | The mirror address must look like `https://github.com/<owner>/<repo>.git` |
|
||||||
| `Gitea API request failed with HTTP 401` | credentials | Create a new Gitea token and update `TOKEN_FOR_GITEA` |
|
| `Gitea API request failed with HTTP 401` | credentials | Create a new Gitea token and update `TOKEN_FOR_GITEA` |
|
||||||
| `GitHub API request failed with HTTP 401` | credentials | Create a new GitHub token and update `CREDENTIALS_FOR_GITHUB` |
|
| `GitHub API request failed with HTTP 401` | credentials | Create a new GitHub token and update `CREDENTIALS_FOR_GITHUB` |
|
||||||
|
|||||||
Reference in New Issue
Block a user