docs/adr/adr-0012-english-comment-detection.md

@@ -52,6 +52,7 @@ A comment is judged **not English** when `langid` 1.1.6, restricted to `en da no
 
 - drop every token that is not a word: a token with a code character (`/ \ { } [ ] " ` = < > | $ @ # % & + ^ ~ *`), an underscore, a digit or `::` (a path, a URL, a JSON key, an identifier); a token that starts with `-` (an option), contains `.` (a file name), is written in capitals (an acronym or a constant) or is camelCase; a leading label such as `Usage:` or `Note:`; and the words of the allowed-words file. Split what is left on `-`;
 - keep a token only if it is letters, with at most one inner apostrophe (letters include `æ ø å ä ö ü` and the other letters of Latin scripts);
+- a block of several consecutive comment lines is judged **line by line first**, each line only with at least four words and a probability of at least 0.7, so an English line cannot hide a Danish one next to it (the first version judged the joined block and missed exactly that case in the trial of [MIL-013] task 8); when no line is flagged, the joined block is judged with the values above, so a sentence cut over short lines is not missed. A single comment line is judged once, with the values above;
 - a comment with fewer than three words left is **not judged** (it passes), and so is a comment that holds only identifiers, URLs or numbers.
 
 On the corpus this gives 0 false positives in 300 English comments, Danish 44/45 (97.8 %), Malay 41/45 (91.1 %), and 48/48 for the other languages, meeting criterion 1 of [MIL-013]. The threshold, the minimum of three words and the candidate languages are inputs of the workflow with these values as defaults; the rates are a test that fails when the code changes them (task 7).

docs/wiki/EnglishComments.da.md

@@ -1,4 +1,4 @@
-<!-- translation of EnglishComments.md, source sha256 866ce29fd1ea -->
+<!-- translation of EnglishComments.md, source sha256 98f3d6e44d62 -->
 
 > Dette er en oversættelse af [`EnglishComments`](EnglishComments). Den engelske side er autoritativ, og ændringer her bliver overskrevet, næste gang wikien publiceres.
 
@@ -52,7 +52,10 @@ kommentar.
 Kommentarerne og docstrings i filer på de sprog, kodetælleren kender
 ([kodemålinger](CodeMetrics)): Python, shell, PowerShell, C, C++, C#, Java,
 JavaScript, TypeScript, Go, Rust og SQL. Strenge og kode læses ikke. Kommentarlinjer
-efter hinanden bedømmes som én blok.
+efter hinanden danner en blok. Hver linje i en blok bedømmes for sig (mindst fire ord og en
+sandsynlighed på mindst 0,7), så en engelsk linje ikke kan skjule en dansk ved siden af; når ingen linje
+markeres, bedømmes blokken som helhed (mindst `min-words` ord og `threshold`), så en sætning delt over korte
+linjer ikke overses.
 
 Før en blok bedømmes, fjernes alt, der ikke er et ord: stier, URL'er, tilvalg
 (`--output`), filnavne, identifikatorer (`snake_case`, `camelCase`), tal,

docs/wiki/EnglishComments.md

@@ -51,7 +51,10 @@ the check still fails the run, but no comment is written.
 The comments and docstrings of the files of the languages in the code counter
 ([code metrics](CodeMetrics)): Python, shell, PowerShell, C, C++, C#, Java,
 JavaScript, TypeScript, Go, Rust and SQL. Strings and code are not read.
-Consecutive comment lines are judged as one block.
+Consecutive comment lines form a block. Each line of a block is judged on its own
+(at least four words and a probability of at least 0.7), so an English line cannot
+hide a Danish one next to it; when no line is flagged, the block is judged as a whole
+(at least `min-words` words and `threshold`), so a sentence cut over short lines is not missed.
 
 Before a block is judged, everything that is not a word is removed: paths, URLs,
 options (`--output`), file names, identifiers (`snake_case`, `camelCase`),

docs/wiki/EnglishComments.ms.md

@@ -1,4 +1,4 @@
-<!-- translation of EnglishComments.md, source sha256 866ce29fd1ea -->
+<!-- translation of EnglishComments.md, source sha256 98f3d6e44d62 -->
 
 > Ini ialah terjemahan [`EnglishComments`](EnglishComments). Halaman bahasa Inggeris ialah teks yang sah, dan perubahan di sini akan ditimpa pada penerbitan wiki yang seterusnya.
 
@@ -52,7 +52,10 @@ semakan masih menggagalkan larian, tetapi tiada komen ditulis.
 Komen dan docstring dalam fail bahasa yang dikenali oleh pengira kod
 ([metrik kod](CodeMetrics)): Python, shell, PowerShell, C, C++, C#, Java,
 JavaScript, TypeScript, Go, Rust dan SQL. String dan kod tidak dibaca. Baris komen
-yang berturutan dinilai sebagai satu blok.
+yang berturutan membentuk satu blok. Setiap baris dalam blok dinilai secara berasingan (sekurang-kurangnya
+empat perkataan dan kebarangkalian sekurang-kurangnya 0.7), jadi baris bahasa Inggeris tidak dapat menyembunyikan
+baris bahasa Denmark di sebelahnya; apabila tiada baris ditandakan, blok dinilai secara keseluruhan
+(sekurang-kurangnya `min-words` perkataan dan `threshold`), supaya ayat yang dipecahkan kepada baris pendek tidak terlepas.
 
 Sebelum sesuatu blok dinilai, segala yang bukan perkataan dibuang: laluan, URL,
 pilihan (`--output`), nama fail, pengecam (`snake_case`, `camelCase`), nombor,

guides/english-comments.md

@@ -45,7 +45,10 @@ the check still fails the run, but no comment is written.
 The comments and docstrings of the files of the languages in the code counter
 ([code metrics](code-metrics.md)): Python, shell, PowerShell, C, C++, C#, Java,
 JavaScript, TypeScript, Go, Rust and SQL. Strings and code are not read.
-Consecutive comment lines are judged as one block.
+Consecutive comment lines form a block. Each line of a block is judged on its own
+(at least four words and a probability of at least 0.7), so an English line cannot
+hide a Danish one next to it; when no line is flagged, the block is judged as a whole
+(at least `min-words` words and `threshold`), so a sentence cut over short lines is not missed.
 
 Before a block is judged, everything that is not a word is removed: paths, URLs,
 options (`--output`), file names, identifiers (`snake_case`, `camelCase`),

tests/test_english_comments.py

@@ -114,9 +114,9 @@ class TestExtract:
     def test_a_file_of_an_unknown_language_has_no_comments(self):
         assert ec.extract("notes.md", "# Heading\n") == []
 
-    def test_consecutive_lines_are_one_block(self):
+    def test_consecutive_lines_are_one_group(self):
         lines = [(1, "a"), (2, "b"), (4, "c"), (5, "d"), (5, "e")]
-        assert ec.blocks(lines) == [(1, 2, "a b"), (4, 5, "c d e")]
+        assert ec.groups(lines) == [[(1, "a"), (2, "b")], [(4, "c"), (5, "d"), (5, "e")]]
 
 
 class TestCheckText:
@@ -134,6 +134,20 @@ class TestCheckText:
         assert [f.line for f in ec.check_text("a.py", text, {3})] == [3]
         assert ec.check_text("a.py", text, {2}) == []
 
+    def test_a_danish_line_next_to_an_english_line_is_still_found(self):
+        english = "# This function adds the totals of the basket together and returns the sum\n"
+        findings = ec.check_text("a.py", english + self.DANISH)
+        assert [(f.line, f.end_line, f.language) for f in findings] == [(2, 2, "da")]
+
+    def test_a_sentence_cut_over_short_lines_is_judged_as_a_whole(self):
+        text = "# Denne funktion\n# beregner totalen\n# for kurven\n# og gemmer resultatet\n"
+        [finding] = ec.check_text("a.py", text)
+        assert (finding.line, finding.end_line, finding.language) == (1, 4, "da")
+
+    def test_only_a_flagged_line_is_reported_when_the_block_is_mixed(self):
+        text = self.DANISH + "# and this line is plain English text about the totals\n"
+        assert [f.line for f in ec.check_text("a.py", text, {2})] == []
+
     def test_the_marker_skips_the_block(self):
         text = self.DANISH.rstrip() + "  # english-comments: ignore\n"
         assert ec.check_text("a.py", text) == []

tools/english_comments.py

@@ -1,11 +1,12 @@
 """Find comments in source code that are not written in English (MIL-013, ADR-0012).
 
 Run by .gitea/workflows/english-comments.yml. It reads the comments and docstrings
-of the source files of the languages of ADR-0010, joins the lines of one comment
-block, and judges the block with ``langid`` (restricted to a few candidate
-languages) after removing everything that is not a word: paths, options,
-identifiers, numbers and the words of the allowed-words file. A block with fewer
-than ``--min-words`` words left is not judged.
+of the source files of the languages of ADR-0010 and judges each comment line
+with ``langid`` (restricted to a few candidate languages) after removing
+everything that is not a word: paths, options, identifiers, numbers and the words
+of the allowed-words file. When no line of a block of consecutive comment lines
+is flagged, the block is judged as a whole, so a sentence cut over short lines is
+not missed. A text with fewer than ``--min-words`` words left is not judged.
 
 Usage:
     python tools/english_comments.py check [--directory DIR] [--scope changed|all]
@@ -38,6 +39,9 @@ from pathlib import Path
 
 DEFAULT_THRESHOLD = 0.6
 DEFAULT_MIN_WORDS = 3
+# a single line of a longer block is judged more strictly than a whole block (ADR-0012)
+LINE_THRESHOLD = 0.7
+LINE_MIN_WORDS = 4
 MAX_FILE_BYTES = 1_000_000
 CANDIDATES = ["en", "da", "no", "sv", "de", "fr", "ms", "id", "nl"]
 LANGUAGE_NAMES = {
@@ -149,16 +153,15 @@ def extract(path: str, text: str) -> list[tuple[int, str]]:
     return sorted((n, t) for n, t in found if t and not _PRAGMA.match(t))
 
 
-def blocks(lines: list[tuple[int, str]]) -> list[tuple[int, int, str]]:
-    """Join comment lines on consecutive line numbers: (first line, last line, text)."""
-    joined: list[tuple[int, int, str]] = []
+def groups(lines: list[tuple[int, str]]) -> list[list[tuple[int, str]]]:
+    """Group comment lines on consecutive line numbers: one group is one comment block."""
+    found: list[list[tuple[int, str]]] = []
     for number, text in lines:
-        if joined and joined[-1][1] in (number, number - 1):
-            first, _, previous = joined[-1]
-            joined[-1] = (first, number, f"{previous} {text}")
+        if found and found[-1][-1][0] in (number, number - 1):
+            found[-1].append((number, text))
         else:
-            joined.append((number, number, text))
-    return joined
+            found.append([(number, text)])
+    return found
 
 
 def words(text: str, allowed: frozenset[str] = frozenset()) -> list[str]:
@@ -169,7 +172,7 @@ def words(text: str, allowed: frozenset[str] = frozenset()) -> list[str]:
         tokens = tokens[1:]  # a label such as "Usage:" or "Note:" is not prose
     for token in tokens:
         if len(token) > 1 and token[0] == "." and token[1].isalpha():
-            continue  # a hidden file such as .clang-format
+            continue  # a dotfile, for example one of a linter
         token = token.strip("().,;:!?")
         # a token with code characters (quotes, braces, `=`, `/`, `_`, digits) is not a word
         if not token or _NOT_WORD_CHARS.search(token) or "::" in token:
@@ -225,14 +228,30 @@ def check_text(
 ) -> list[Finding]:
     """Findings of one file; with ``lines`` only the blocks that touch those line numbers."""
     findings: list[Finding] = []
-    for first, last, body in blocks(extract(path, text)):
+    for group in groups(extract(path, text)):
+        body = " ".join(line for _, line in group)
         if IGNORE_MARKER.search(body):
             continue
-        if lines is not None and not any(first <= n <= last for n in lines):
-            continue
-        verdict = judge(body, allowed, threshold, min_words)
-        if verdict:
-            findings.append(Finding(path, first, last, verdict[0], round(verdict[1], 2), body))
+        found: list[Finding] = []
+        if len(group) > 1:
+            # each line on its own, so an English line cannot hide a Danish one next to it
+            found = [
+                Finding(path, number, number, verdict[0], round(verdict[1], 2), line)
+                for number, line in group
+                if (
+                    verdict := judge(
+                        line, allowed, max(threshold, LINE_THRESHOLD), max(min_words, LINE_MIN_WORDS)
+                    )
+                )
+            ]
+        if not found and (verdict := judge(body, allowed, threshold, min_words)):
+            # the block as a whole, so a sentence cut over short lines is not missed
+            found = [Finding(path, group[0][0], group[-1][0], verdict[0], round(verdict[1], 2), body)]
+        findings += [
+            f
+            for f in found
+            if lines is None or any(f.line <= n <= f.end_line for n in lines)
+        ]
     return findings