diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a5c8a47..dcaeaaf9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,15 @@ All notable changes to ForgeLM are documented here. ### Fixed +- CI/CD examples capture ForgeLM and `jq` statuses separately: a JSON parser's exit 4 cannot be mistaken for an + approval hold, and a multi-stage approval hold retains ForgeLM's exit 4. The guide's JSON-reference section links + use the compiled heading IDs and preserve the page route and heading on reload + (F-PR1073-A1-01, F-PR1073-A2-01). +- CI/CD user-manual examples in English and Turkish read the training envelope's boolean `success` field, + keep plain-text stderr logs separate from the JSON result, and distinguish the multi-stage `final_status` + summary. The canonical JSON reference documents the pipeline-state exception and its actual fields in both languages. + Bash gate examples use `pipefail` and `jq -e` so failed or awaiting-approval runs do not pass + (#433, #216; CR-A04-008, DOC-D3B02-016). Non-JSON library output before the envelope remains tracked in #412. - **`forgelm purge --row-id` warns about the weights a run actually leaves behind.** This restores the documented contract that `data.erasure_warning_memorisation` fires when a trained model exists for the erased row. The check used to look only for a literal `<--output-dir>/final_model/`, so a default invocation (`--output-dir` omitted, diff --git a/docs/standards/testing.md b/docs/standards/testing.md index 069920de..f077cb9c 100644 --- a/docs/standards/testing.md +++ b/docs/standards/testing.md @@ -59,6 +59,13 @@ tests/ - `conftest.py` holds shared fixtures only. Domain-specific helpers live in the test files that need them. - A new feature PR adds the matching `test_*.py` in the **same** PR. No "tests in next PR." +## Documentation regressions + +The CI/CD manual regression cases execute the shipped snippets with POSIX Bash and `jq`. +They are skipped with an explicit prerequisite reason on hosts without these tools; GitHub's Linux runners provide both. +Guide navigation regressions compile the EN/TR Markdown with the development-only `markdown` dependency and run +the shipped router in the Node VM harness, checking the compiled heading target and route after reload. + ## Browser wizard regressions Browser wizard regressions live in `tests/site/*.test.cjs`. Run diff --git a/docs/usermanuals/en/operations/cicd.md b/docs/usermanuals/en/operations/cicd.md index aed09f0a..231aeef5 100644 --- a/docs/usermanuals/en/operations/cicd.md +++ b/docs/usermanuals/en/operations/cicd.md @@ -143,14 +143,67 @@ pipeline { ## JSON output mode -For programmatic parsing of run outputs, use `--output-format json`: +For a single-stage training run (a config without `pipeline:`), the result envelope has a boolean `success` field: ```shell -$ forgelm --config configs/run.yaml --output-format json | jq '.verdict' -"success" +$ forgelm --config configs/run.yaml --output-format json | jq '.success' +true ``` -Every event ForgeLM logs to stderr is also emitted as a JSON object on stdout, ready to pipe into your log aggregator or dashboard. +The JSON envelope goes to stdout; human-readable logs remain plain text on stderr. +JSON mode suppresses logs below WARNING and does not mirror log events to stdout. +See [JSON output schemas](#/reference/json-output) for every field. + +Printing `.success` is not a CI gate: `jq` normally exits 0 even for `false`. +Use `PIPESTATUS` to capture ForgeLM and `jq` results separately before another command overwrites them. +Only ForgeLM exit 4 means awaiting approval; `jq` can also exit 4 on empty input. +Preserve non-zero ForgeLM exits. If ForgeLM exits 0 but JSON validation fails, this wrapper exits 1. +See [Exit Codes](#/reference/exit-codes) for the ForgeLM contract. + +```bash +set -o pipefail +command_status=(0 0) +forgelm --config configs/run.yaml --output-format json \ + | jq -e '.success == true' || command_status=("${PIPESTATUS[@]}") +forgelm_status=${command_status[0]} +json_status=${command_status[1]} +if [ "$forgelm_status" -eq 4 ]; then + printf 'Awaiting human approval\n' >&2 + exit 4 +elif [ "$forgelm_status" -ne 0 ]; then + exit "$forgelm_status" +elif [ "$json_status" -ne 0 ]; then + printf 'JSON validation failed (jq exit %s)\n' "$json_status" >&2 + exit 1 +fi +exit 0 +``` + +A config with `pipeline:` emits a pipeline state with `final_status`, not a top-level `success`. +For a multi-stage run, require `completed`; `stopped_at_stage` and `gated_pending_approval` must not pass: + +```bash +set -o pipefail +command_status=(0 0) +forgelm --config configs/pipeline.yaml --output-format json \ + | jq -e '.final_status == "completed"' || command_status=("${PIPESTATUS[@]}") +forgelm_status=${command_status[0]} +json_status=${command_status[1]} +if [ "$forgelm_status" -eq 4 ]; then + printf 'Awaiting human approval\n' >&2 + exit 4 +elif [ "$forgelm_status" -ne 0 ]; then + exit "$forgelm_status" +elif [ "$json_status" -ne 0 ]; then + printf 'JSON validation failed (jq exit %s)\n' "$json_status" >&2 + exit 1 +fi +exit 0 +``` + +Known limitation: training and pipeline runs can print non-JSON library output before the envelope +([#412](https://github.com/HodeTech/ForgeLM/issues/412)). The examples require clean JSON stdout; +a JSON parse error must fail the CI step. ## Caching models diff --git a/docs/usermanuals/en/reference/json-output.md b/docs/usermanuals/en/reference/json-output.md index c0ebbabe..e59b54b9 100644 --- a/docs/usermanuals/en/reference/json-output.md +++ b/docs/usermanuals/en/reference/json-output.md @@ -11,12 +11,40 @@ This page is the canonical reference. CI/CD pipelines that parse `forgelm` outpu ## Common conventions -- **stdout vs stderr.** The JSON envelope goes to **stdout**. Human-friendly logs (info / warning / error) go to **stderr**. Pipe `forgelm ... --output-format json | jq .` and read your operator-facing messages from `2>` separately. -- **Top-level wrapper.** Every envelope starts with `"success": true | false`. Consumers can branch on this single key before parsing the rest. +- **stdout vs stderr.** The JSON envelope goes to **stdout**; human-friendly logs go to **stderr**. + Training and pipeline runs can also print non-JSON library output before the envelope + ([#412](https://github.com/HodeTech/ForgeLM/issues/412)); a parse error must fail the CI step. + See [CI/CD Pipelines](#/operations/cicd) for Bash gate examples. +- **Top-level wrapper.** Most envelopes start with `"success": true | false`. + Summaries for configs containing `pipeline:` are the exception: they use `final_status` instead. + See [Multi-stage pipeline state](#/reference/json-output#multi-stage-pipeline-state). - **Error envelope.** When `success: false`, the envelope carries `"error": ""` (string). Optional richer fields (`exit_code`, `error_type`, `details`) MAY be present per the [error-handling standard on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/standards/error-handling.md). Consumers that need certainty on these fields should also check the process exit code via `$?`. - **Exit codes.** See [Exit Codes](#/reference/exit-codes). `success: false` always pairs with a non-zero exit. `success: true` pairs with exit `0` in the normal case, with **one exception**: a training run halted at the Article 14 human-approval gate emits `success: true` **and** `awaiting_approval: true` while exiting `4` (the model is staged, not yet promoted). Consumers that must distinguish "done" from "awaiting human approval" should read the `awaiting_approval` discriminator (or check `$?`), not `success` alone. - **Argument-parsing errors are the exception.** The JSON-envelope guarantee begins **after** arguments parse successfully. A usage error detected at parse time (unknown flag, bad `--choice`, out-of-range validator value such as `--workers 0`) is reported by `argparse` itself: a human-readable `usage: ... error: ...` line on **stderr** and exit code `2`, with **no** JSON envelope on stdout. A consumer that reads stdout-JSON on every non-zero exit must treat empty stdout (paired with non-zero `$?`) as a parse/usage error and fall back to stderr for the message. The envelope contract applies only to errors raised once the dispatcher is running. +## Multi-stage pipeline state + +`forgelm --config --output-format json` with a `pipeline:` block emits a pipeline-state summary. +The same state is saved in `pipeline_state.json`. This summary has no top-level `success` or `error` wrapper; +an error caught before a state summary is produced may instead use the common `success: false` error envelope. + +| Key | Type | Notes | +|---|---|---| +| `pipeline_run_id` | str | Identifier of the pipeline run. | +| `pipeline_config_hash` | str | Hash of the pipeline configuration bytes. | +| `forgelm_version` | str | ForgeLM version that produced the state. | +| `started_at` | str | UTC start timestamp. | +| `finished_at` | str or null | UTC finish timestamp, or null while the run is in progress. | +| `final_status` | str | Pipeline status; see the values below. | +| `stopped_at` | str or null | Stage where the chain stopped, or null. | +| `stages` | list[object] | Per-stage snapshots: status, exit code, error, metrics, model paths and timestamps. | + +`final_status` is one of `in_progress`, `completed`, `stopped_at_stage`, or `gated_pending_approval`. +Check this field together with the process exit code, rather than testing `.success` on a state summary. +See [Multi-stage pipelines](#/training/pipelines) for stage behavior and +the [PipelineStageState source](https://github.com/HodeTech/ForgeLM/blob/main/forgelm/cli/_pipeline.py) +for the nested stage fields. + ## `forgelm doctor` Environment check. See [Doctor command](#/getting-started/first-run). diff --git a/docs/usermanuals/tr/operations/cicd.md b/docs/usermanuals/tr/operations/cicd.md index 2f8afe51..6b7caba8 100644 --- a/docs/usermanuals/tr/operations/cicd.md +++ b/docs/usermanuals/tr/operations/cicd.md @@ -144,14 +144,68 @@ pipeline { ## JSON çıktı modu -Koşu çıktısının programatik parse'ı için `--output-format json` kullanın: +Tek aşamalı bir eğitim koşusunda (`pipeline:` içermeyen config), sonuç zarfındaki `success` alanı boolean türündedir: ```shell -$ forgelm --config configs/run.yaml --output-format json | jq '.verdict' -"success" +$ forgelm --config configs/run.yaml --output-format json | jq '.success' +true ``` -ForgeLM'in stderr'a yazdığı her olay stdout'ta da JSON nesnesi olarak çıkar — log aggregator'ınıza veya dashboard'unuza pipe'lamaya hazır. +JSON zarfı stdout'a gider; okunabilir loglar stderr'de düz metin olarak kalır. +JSON modu WARNING altındaki logları bastırır ve log olaylarını stdout'a kopyalamaz. +Tüm alanlar için [JSON çıktı şemaları](#/reference/json-output) sayfasına bakın. + +`.success` değerini yazdırmak bir CI kontrolü değildir: `jq`, `false` için de normalde 0 ile çıkar. +Başka bir komut üzerine yazmadan önce ForgeLM ve `jq` sonuçlarını `PIPESTATUS` ile ayrı yakalayın. +Yalnızca ForgeLM'in 4 ile çıkması onay beklendiğini gösterir; `jq` boş girdide de 4 ile çıkabilir. +ForgeLM'in sıfır dışındaki kodlarını koruyun. ForgeLM 0 ile çıkıp JSON doğrulaması başarısızsa bu betik 1 ile çıkar. +ForgeLM sözleşmesi için [Çıkış Kodları](#/reference/exit-codes) sayfasına bakın. + +```bash +set -o pipefail +command_status=(0 0) +forgelm --config configs/run.yaml --output-format json \ + | jq -e '.success == true' || command_status=("${PIPESTATUS[@]}") +forgelm_status=${command_status[0]} +json_status=${command_status[1]} +if [ "$forgelm_status" -eq 4 ]; then + printf 'Awaiting human approval\n' >&2 + exit 4 +elif [ "$forgelm_status" -ne 0 ]; then + exit "$forgelm_status" +elif [ "$json_status" -ne 0 ]; then + printf 'JSON validation failed (jq exit %s)\n' "$json_status" >&2 + exit 1 +fi +exit 0 +``` + +`pipeline:` içeren config, üst düzey `success` yerine `final_status` taşıyan bir pipeline durumu üretir. +Çok aşamalı bir koşuda `completed` şartını arayın; +`stopped_at_stage` ve `gated_pending_approval` başarılı sayılmamalıdır: + +```bash +set -o pipefail +command_status=(0 0) +forgelm --config configs/pipeline.yaml --output-format json \ + | jq -e '.final_status == "completed"' || command_status=("${PIPESTATUS[@]}") +forgelm_status=${command_status[0]} +json_status=${command_status[1]} +if [ "$forgelm_status" -eq 4 ]; then + printf 'Awaiting human approval\n' >&2 + exit 4 +elif [ "$forgelm_status" -ne 0 ]; then + exit "$forgelm_status" +elif [ "$json_status" -ne 0 ]; then + printf 'JSON validation failed (jq exit %s)\n' "$json_status" >&2 + exit 1 +fi +exit 0 +``` + +Bilinen sınırlama: eğitim ve pipeline koşuları, zarftan önce JSON olmayan kütüphane çıktısı yazabilir +([#412](https://github.com/HodeTech/ForgeLM/issues/412)). Örnekler temiz JSON stdout gerektirir; +JSON ayrıştırma hatası CI adımını başarısız kılmalıdır. ## Model cache diff --git a/docs/usermanuals/tr/reference/json-output.md b/docs/usermanuals/tr/reference/json-output.md index e1f3a911..b1ee72e4 100644 --- a/docs/usermanuals/tr/reference/json-output.md +++ b/docs/usermanuals/tr/reference/json-output.md @@ -11,12 +11,41 @@ Bu sayfa kanonik referanstır. `forgelm` çıktısını parse eden CI/CD pipelin ## Ortak konvansiyonlar -- **stdout vs stderr.** JSON envelope **stdout**'a gider. İnsana okunabilir loglar (info / warning / error) **stderr**'e gider. `forgelm ... --output-format json | jq .` ile pipe edip operatör mesajlarını `2>` ile ayrı okuyun. -- **Üst seviye sarmalayıcı.** Her envelope `"success": true | false` ile başlar. Tüketiciler önce bu tek anahtarda branch açabilir. +- **stdout vs stderr.** JSON zarfı **stdout**'a, okunabilir loglar **stderr**'e gider. + Eğitim ve pipeline koşuları, zarftan önce JSON olmayan kütüphane çıktısı da yazabilir + ([#412](https://github.com/HodeTech/ForgeLM/issues/412)); ayrıştırma hatası CI adımını başarısız kılmalıdır. + Bash kontrol örnekleri için [CI/CD Pipeline'ları](#/operations/cicd) sayfasına bakın. +- **Üst seviye sarmalayıcı.** Zarfların çoğu `"success": true | false` ile başlar. + `pipeline:` içeren config'lerin özetleri istisnadır: bunun yerine `final_status` kullanırlar. + Bkz. [Çok aşamalı pipeline durumu](#/reference/json-output#cok-asamal-pipeline-durumu). - **Hata envelope'u.** `success: false` olduğunda envelope `"error": ""` (string) taşır. Opsiyonel zenginleştirilmiş alanlar (`exit_code`, `error_type`, `details`) [GitHub'daki error-handling standardı](https://github.com/HodeTech/ForgeLM/blob/main/docs/standards/error-handling.md) gereği var olabilir. Bu alanlardan kesinlik isteyen tüketiciler `$?` ile süreç exit kodunu da kontrol etmelidir. - **Exit kodları.** Bkz. [Exit Kodları](#/reference/exit-codes). `success: false` her zaman non-zero bir exit ile eşleşir. `success: true` normal durumda exit `0` ile eşleşir, **tek istisna** dışında: Article 14 insan-onay gate'inde duraklayan bir eğitim çalışması `success: true` **ve** `awaiting_approval: true` emit ederken exit `4` ile çıkar (model stage'lenmiştir, henüz promote edilmemiştir). "Bitti" ile "insan onayı bekliyor"u ayırması gereken tüketiciler tek başına `success` yerine `awaiting_approval` discriminator'ını (veya `$?`'i) okumalıdır. - **Argüman-ayrıştırma hataları istisnadır.** JSON-envelope garantisi argümanlar başarıyla ayrıştırıldıktan **sonra** başlar. Ayrıştırma anında saptanan bir kullanım hatası (bilinmeyen flag, hatalı `--choice`, `--workers 0` gibi aralık-dışı validator değeri) `argparse`'in kendisi tarafından raporlanır: **stderr**'e insana okunabilir bir `usage: ... error: ...` satırı ve exit kodu `2`, stdout'ta **hiçbir** JSON envelope olmadan. Her non-zero exit'te stdout-JSON okuyan bir tüketici, boş stdout'u (non-zero `$?` ile eşleşen) bir ayrıştırma/kullanım hatası olarak ele almalı ve mesaj için stderr'e dönmelidir. Envelope kontratı yalnızca dispatcher çalışmaya başladıktan sonra ortaya çıkan hatalar için geçerlidir. +## Çok aşamalı pipeline durumu + +`pipeline:` bloğuyla `forgelm --config --output-format json` çalıştırıldığında +pipeline durum özeti üretilir. +Aynı durum `pipeline_state.json` dosyasına kaydedilir. Bu özette üst düzey `success` veya `error` sarmalayıcısı yoktur; +durum özeti üretilmeden yakalanan bir hata ise ortak `success: false` hata zarfını kullanabilir. + +| Anahtar | Tür | Notlar | +|---|---|---| +| `pipeline_run_id` | str | Pipeline koşusunun kimliği. | +| `pipeline_config_hash` | str | Pipeline yapılandırma baytlarının hash'i. | +| `forgelm_version` | str | Durumu üreten ForgeLM sürümü. | +| `started_at` | str | UTC başlangıç zaman damgası. | +| `finished_at` | str veya null | UTC bitiş zaman damgası; koşu sürerken null. | +| `final_status` | str | Pipeline durumu; değerleri aşağıda verilmiştir. | +| `stopped_at` | str veya null | Zincirin durduğu aşama; durmadıysa null. | +| `stages` | list[object] | Aşama başına durum, çıkış kodu, hata, metrik, model yolu ve zaman damgası kayıtları. | + +`final_status`, `in_progress`, `completed`, `stopped_at_stage` veya `gated_pending_approval` değerlerinden biridir. +Bir durum özetinde `.success` aramak yerine bu alanı süreç çıkış koduyla birlikte kontrol edin. +Aşama davranışı için [Çok aşamalı pipeline'lar](#/training/pipelines) sayfasına, +iç içe aşama alanları için +[PipelineStageState kaynağına](https://github.com/HodeTech/ForgeLM/blob/main/forgelm/cli/_pipeline.py) bakın. + ## `forgelm doctor` Ortam kontrolü. Bkz. [Doctor komutu](#/getting-started/first-run). diff --git a/pyproject.toml b/pyproject.toml index 99a53f58..c1866578 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -186,6 +186,8 @@ chat = [ # Developer tooling — test runner, coverage, linter/formatter. dev = [ + # Compile manual pages in the guide navigation regression tests. + "markdown>=3.5.0,<4.0.0", "pytest>=8.0.0", "pytest-cov>=5.0.0", # pytest-randomly shuffles test order per session so order-dependent diff --git a/site/js/guide.js b/site/js/guide.js index 9cfa90b6..084b069f 100644 --- a/site/js/guide.js +++ b/site/js/guide.js @@ -75,7 +75,7 @@ } function navigate(route, replace) { - state.route = route; + state.route = route.split('#')[0]; var newHash = '#/' + route; if (window.location.hash !== newHash) { if (replace) { @@ -249,6 +249,16 @@ + pager + editLink); + content.querySelectorAll('a[href^="#/"]').forEach(function (link) { + link.addEventListener('click', function (event) { + if (event.button > 0 || event.ctrlKey || event.metaKey || event.shiftKey || event.altKey) return; + var nextRoute = parseHash(link.getAttribute('href')); + if (!nextRoute) return; + event.preventDefault(); + navigate(nextRoute, false); + }); + }); + // Re-translate any chrome that data-i18n updated nodes inside the content. if (window.ForgeLMi18n && window.ForgeLMi18n.setLanguage) { window.ForgeLMi18n.setLanguage(state.lang); @@ -268,6 +278,14 @@ // After DOM updates, trigger scrollspy refresh. setTimeout(observeHeadings, 50); updateProgressBar(); + scrollToRouteAnchor(); + } + + function scrollToRouteAnchor() { + var anchor = window.location.hash.split('#')[2]; + if (!anchor) return; + var target = document.getElementById(anchor); + if (target) target.scrollIntoView({ block: 'start' }); } function renderNotFound(route) { diff --git a/tests/site/guide_navigation.test.cjs b/tests/site/guide_navigation.test.cjs new file mode 100644 index 00000000..d61bc957 --- /dev/null +++ b/tests/site/guide_navigation.test.cjs @@ -0,0 +1,135 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const vm = require('node:vm'); +const { execFileSync } = require('node:child_process'); +const { test } = require('node:test'); +const { root } = require('./wizard_harness.cjs'); + +const localPython = path.join(root, '.venv', 'bin', 'python'); +const python = process.env.FORGELM_TEST_PYTHON || (fs.existsSync(localPython) ? localPython : 'python3'); +const compiled = JSON.parse(execFileSync(python, ['-c', ` +import json +from html.parser import HTMLParser +from pathlib import Path +from tools.build_usermanuals import parse_front_matter, render_markdown + +class Elements(HTMLParser): + def __init__(self): + super().__init__() + self.ids = [] + self.links = [] + def handle_starttag(self, tag, attrs): + attrs = dict(attrs) + if 'id' in attrs: + self.ids.append(attrs['id']) + if tag == 'a' and 'href' in attrs: + self.links.append(attrs['href']) + +out = {} +for lang in ('en', 'tr'): + src = Path(f'docs/usermanuals/{lang}/reference/json-output.md').read_text(encoding='utf-8') + meta, body = parse_front_matter(src) + html, headings = render_markdown(body) + elements = Elements() + elements.feed(html) + out[lang] = dict(title=meta['title'], html=html, headings=headings, ids=elements.ids, links=elements.links) +print(json.dumps(out)) +`], { cwd: root, encoding: 'utf8' })); + +function loadGuide(lang, hash) { + const page = compiled[lang]; + const links = page.links.filter(href => href.startsWith('#/')).map(href => ({ + href, events: {}, + getAttribute(name) { assert.equal(name, 'href'); return href; }, + addEventListener(name, handler) { this.events[name] = handler; } + })); + const content = { + html: '', links, + querySelectorAll(selector) { assert.equal(selector, 'a[href^="#/"]'); return links; } + }; + const window = { + location: { hash }, + ForgeLMUserManualsIndex: { sections: [] }, + ForgeLMUserManuals: { [lang]: { 'reference/json-output': page } }, + scrollTo() { window.scrolled = null; } + }; + const ids = new Map(page.ids.map(id => [id, { scrollIntoView() { window.scrolled = id; } }])); + const document = { + addEventListener() {}, + getElementById(id) { return id === 'guide-content' ? content : ids.get(id) || null; }, + querySelectorAll() { return []; } + }; + const history = { + replaceState(_state, _title, url) { window.location.hash = url; }, + pushState(_state, _title, url) { window.location.hash = url; } + }; + let source = fs.readFileSync(path.join(root, 'site/js/guide.js'), 'utf8'); + const end = source.lastIndexOf('})();'); + assert.ok(end > 0, 'the guide is an isolated closure'); + source = source.slice(0, end) + ` + setHtml = function (element, html) { element.html = html; }; + window.guideUnderTest = { state: state, navigate: navigate, parseHash: parseHash }; + ` + source.slice(end); + const context = vm.createContext({ window, document, history, console, setTimeout() {} }); + vm.runInContext(source, context, { filename: 'guide.js' }); + const guide = window.guideUnderTest; + guide.state.lang = lang; + return { window, guide, content }; +} + +for (const lang of ['en', 'tr']) { + test(`${lang}: compiled section links retain their page and scroll target after navigation and reload`, async () => { + const page = compiled[lang]; + const links = page.links.filter(href => href.startsWith('#/reference/json-output#')); + assert.equal(links.length, 1, 'section links must include the SPA page route'); + const href = links[0]; + const anchor = href.split('#')[2]; + assert.ok(page.ids.includes(anchor), `the compiled HTML contains ${anchor}`); + + const first = loadGuide(lang, '#/reference/json-output'); + first.guide.navigate(first.guide.parseHash(first.window.location.hash), true); + await new Promise(setImmediate); + const link = first.content.links.find(link => link.href === href); + for (const click of [ + { button: 1 }, { button: 2 }, + { button: 0, ctrlKey: true }, { button: 0, metaKey: true }, + { button: 0, shiftKey: true }, { button: 0, altKey: true } + ]) { + let prevented = false; + link.events.click({ ...click, preventDefault() { prevented = true; } }); + await new Promise(setImmediate); + assert.equal(prevented, false, `preserve native behavior for ${JSON.stringify(click)}`); + assert.equal(first.window.location.hash, '#/reference/json-output'); + assert.equal(first.guide.state.route, 'reference/json-output'); + assert.equal(first.window.scrolled, null); + } + let prevented = false; + link.events.click({ button: 0, preventDefault() { prevented = true; } }); + await new Promise(setImmediate); + assert.ok(prevented, 'the real content click handler owns SPA navigation'); + assert.equal(first.guide.state.route, 'reference/json-output'); + assert.ok(first.content.html.includes(page.html), 'the real renderer receives the page, not a missing-page result'); + assert.equal(first.window.scrolled, anchor); + assert.equal(first.window.location.hash, href); + + first.window.scrolled = null; + link.events.click({ button: 0, preventDefault() {} }); + await new Promise(setImmediate); + assert.equal(first.window.scrolled, anchor, 'clicking the current fragment again still scrolls'); + + const reloaded = loadGuide(lang, first.window.location.hash); + reloaded.guide.navigate(reloaded.guide.parseHash(reloaded.window.location.hash), true); + await new Promise(setImmediate); + assert.equal(reloaded.guide.state.route, 'reference/json-output'); + assert.ok(reloaded.content.html.includes(page.html)); + assert.equal(reloaded.window.scrolled, anchor); + + reloaded.guide.navigate('reference/json-output', false); + await new Promise(setImmediate); + assert.equal(reloaded.window.location.hash, '#/reference/json-output'); + assert.equal(reloaded.window.scrolled, null, 'a route without a heading clears the scroll target'); + }); +} diff --git a/tests/test_cli.py b/tests/test_cli.py index b9fb69ac..e5a08fa0 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -518,3 +518,54 @@ def test_the_transformers_backend_still_loads(self, tmp_path, minimal_config): path = tmp_path / "c.yaml" path.write_text(yaml.dump(minimal_config(training={"rope_scaling": {"type": "linear", "factor": 2.0}}))) assert _load_config_or_exit(str(path), json_output=False).training.rope_scaling is not None + + +class TestCicdJsonOutputMode: + """Check training-result types and JSON-mode logging streams.""" + + @pytest.mark.parametrize(("success", "awaiting_approval"), [(True, False), (False, False), (True, True)]) + def test_training_envelope_success_is_boolean_and_has_no_verdict(self, capsys, success, awaiting_approval): + from forgelm.cli._result import _output_result + from forgelm.results import TrainResult + + _output_result(TrainResult(success=success, awaiting_approval=awaiting_approval), "json") + raw = capsys.readouterr().out + envelope = json.loads(raw) + assert "verdict" not in envelope + assert envelope["success"] is success + assert envelope["awaiting_approval"] is awaiting_approval + + def test_json_mode_logs_stay_plain_text_on_stderr(self, capsys): + import logging + + from forgelm.cli._logging import logger as cli_logger + + root = logging.getLogger() + saved_handlers = root.handlers[:] + saved_level = root.level + saved_cli_level = cli_logger.level + for handler in saved_handlers: + root.removeHandler(handler) + cli_logger.setLevel(logging.NOTSET) + try: + # A fresh process is what CI runs. ``basicConfig`` is a no-op once + # any handler exists, so drop pytest's capture handler first. + _setup_logging("DEBUG", json_format=True) + cli_logger.info("pipeline-started") + cli_logger.warning("gate-tripped") + captured = capsys.readouterr() + finally: + for handler in root.handlers[:]: + root.removeHandler(handler) + handler.close() + for handler in saved_handlers: + root.addHandler(handler) + root.setLevel(saved_level) + cli_logger.setLevel(saved_cli_level) + + assert captured.out == "" + assert "pipeline-started" not in captured.err + assert "gate-tripped" in captured.err + assert "[WARNING]" in captured.err + with pytest.raises(json.JSONDecodeError): + json.loads(captured.err.strip()) diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 696a2b56..e3faf342 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -20,9 +20,15 @@ from __future__ import annotations import bisect +import json +import os import re import shlex +import shutil +import subprocess +import sys from pathlib import Path +from typing import Tuple import pytest @@ -33,6 +39,203 @@ def _cli_md(lang: str) -> str: return (_REPO_ROOT / "docs" / "usermanuals" / lang / "reference" / "cli.md").read_text(encoding="utf-8") +def _cicd_gate_block(config: str, query: str) -> str: + """Keep the shared EN/TR gate identical in the complete-section snapshot.""" + return ( + f"```bash\nset -o pipefail\ncommand_status=(0 0)\n" + f"forgelm --config configs/{config}.yaml --output-format json \\\n" + f" | jq -e '{query}' || command_status=(\"${{PIPESTATUS[@]}}\")\n" + "forgelm_status=${command_status[0]}\njson_status=${command_status[1]}\n" + 'if [ "$forgelm_status" -eq 4 ]; then\n' + " printf 'Awaiting human approval\\n' >&2\n exit 4\n" + 'elif [ "$forgelm_status" -ne 0 ]; then\n exit "$forgelm_status"\n' + 'elif [ "$json_status" -ne 0 ]; then\n' + " printf 'JSON validation failed (jq exit %s)\\n' \"$json_status\" >&2\n" + " exit 1\nfi\nexit 0\n```" + ) + + +@pytest.mark.parametrize( + ("lang", "paragraphs"), + [ + ( + "en", + ( + "For a single-stage training run (a config without `pipeline:`), " + "the result envelope has a boolean `success` field:", + "The JSON envelope goes to stdout; human-readable logs remain plain text on stderr. " + "JSON mode suppresses logs below WARNING and does not mirror log events to stdout. " + "See [JSON output schemas](#/reference/json-output) for every field.", + "Printing `.success` is not a CI gate: `jq` normally exits 0 even for `false`. " + "Use `PIPESTATUS` to capture ForgeLM and `jq` results separately before another command overwrites them. " + "Only ForgeLM exit 4 means awaiting approval; `jq` can also exit 4 on empty input. " + "Preserve non-zero ForgeLM exits. If ForgeLM exits 0 but JSON validation fails, this wrapper exits 1. " + "See [Exit Codes](#/reference/exit-codes) for the ForgeLM contract.", + "A config with `pipeline:` emits a pipeline state with `final_status`, not a top-level `success`. " + "For a multi-stage run, require `completed`; `stopped_at_stage` and `gated_pending_approval` " + "must not pass:", + "Known limitation: training and pipeline runs can print non-JSON library output before the envelope " + "([#412](https://github.com/HodeTech/ForgeLM/issues/412)). The examples require clean JSON stdout; " + "a JSON parse error must fail the CI step.", + ), + ), + ( + "tr", + ( + "Tek aşamalı bir eğitim koşusunda (`pipeline:` içermeyen config), " + "sonuç zarfındaki `success` alanı boolean türündedir:", + "JSON zarfı stdout'a gider; okunabilir loglar stderr'de düz metin olarak kalır. " + "JSON modu WARNING altındaki logları bastırır ve log olaylarını stdout'a kopyalamaz. " + "Tüm alanlar için [JSON çıktı şemaları](#/reference/json-output) sayfasına bakın.", + "`.success` değerini yazdırmak bir CI kontrolü değildir: `jq`, `false` için de normalde 0 ile çıkar. " + "Başka bir komut üzerine yazmadan önce ForgeLM ve `jq` sonuçlarını `PIPESTATUS` ile ayrı yakalayın. " + "Yalnızca ForgeLM'in 4 ile çıkması onay beklendiğini gösterir; `jq` boş girdide de 4 ile çıkabilir. " + "ForgeLM'in sıfır dışındaki kodlarını koruyun. ForgeLM 0 ile çıkıp JSON doğrulaması başarısızsa " + "bu betik 1 ile çıkar. ForgeLM sözleşmesi için [Çıkış Kodları](#/reference/exit-codes) sayfasına bakın.", + "`pipeline:` içeren config, üst düzey `success` yerine `final_status` taşıyan bir pipeline durumu " + "üretir. Çok aşamalı bir koşuda `completed` şartını arayın; `stopped_at_stage` ve " + "`gated_pending_approval` başarılı sayılmamalıdır:", + "Bilinen sınırlama: eğitim ve pipeline koşuları, zarftan önce JSON olmayan kütüphane çıktısı " + "yazabilir ([#412](https://github.com/HodeTech/ForgeLM/issues/412)). Örnekler temiz JSON stdout " + "gerektirir; JSON ayrıştırma hatası CI adımını başarısız kılmalıdır.", + ), + ), + ], +) +def test_cicd_json_section_matches_result_contracts(lang: str, paragraphs: Tuple[str, ...]) -> None: + """Pin the complete JSON section so contradictory event-stream claims cannot be appended.""" + from forgelm.cli._pipeline import PipelineState, _serialise_state + from forgelm.cli._result import _build_result_json_envelope + from forgelm.results import TrainResult + + page = _REPO_ROOT / "docs" / "usermanuals" / lang / "operations" / "cicd.md" + heading = "## JSON output mode" if lang == "en" else "## JSON çıktı modu" + section = page.read_text(encoding="utf-8").partition(heading + "\n")[2].partition("\n## ")[0].strip() + blocks = ( + "```shell\n$ forgelm --config configs/run.yaml --output-format json | jq '.success'\ntrue\n```", + _cicd_gate_block("run", ".success == true"), + _cicd_gate_block("pipeline", '.final_status == "completed"'), + ) + expected = "\n\n".join( + (paragraphs[0], blocks[0], paragraphs[1], paragraphs[2], blocks[1], paragraphs[3], blocks[2], paragraphs[4]) + ) + assert " ".join(section.split()) == " ".join(expected.split()) + + training_envelope = _build_result_json_envelope(TrainResult(success=True)) + pipeline_envelope = _serialise_state( + PipelineState(pipeline_run_id="test", pipeline_config_hash="test", forgelm_version="test", started_at="test") + ) + for block, envelope in ( + (blocks[0], training_envelope), + (blocks[1], training_envelope), + (blocks[2], pipeline_envelope), + ): + tokens = shlex.split(block.split("|", 1)[1].splitlines()[0]) + query = next(token for token in tokens[tokens.index("jq") + 1 :] if not token.startswith("-")) + key = query.split(" == ", 1)[0].removeprefix(".") + assert key in envelope + assert training_envelope["success"] is True + assert "success" not in pipeline_envelope + + +_BASH = shutil.which("bash") if sys.platform != "win32" else None +_JQ = shutil.which("jq") + + +@pytest.mark.skipif(not _BASH or not _JQ, reason="needs POSIX Bash and jq to execute the documented CI examples") +@pytest.mark.parametrize("lang", ["en", "tr"]) +@pytest.mark.parametrize("mode", ["training", "pipeline"]) +@pytest.mark.parametrize( + "case", + ["success", "failure", "unsuccessful", "approval", "empty", "usage", "malformed", "producer_error", "missing"], +) +def test_cicd_bash_preserves_producer_status(lang, mode, case, tmp_path): + """Run the shipped gate under errexit using real jq and the real CLI usage-error path.""" + from forgelm.cli._pipeline import PipelineState, _serialise_state + from forgelm.cli._result import _build_result_json_envelope + from forgelm.results import TrainResult + + text = (_REPO_ROOT / "docs" / "usermanuals" / lang / "operations" / "cicd.md").read_text(encoding="utf-8") + heading = "## JSON output mode" if lang == "en" else "## JSON çıktı modu" + section = text.partition(heading + "\n")[2].partition("\n## ")[0] + blocks = [part.split("\n", 1)[1] for part in section.split("```")[1::2] if part.startswith("bash\n")] + script = blocks[0 if mode == "training" else 1] + producer_status = {"failure": 3, "approval": 4, "producer_error": 2}.get(case, 0) + if mode == "training": + envelope = _build_result_json_envelope( + TrainResult(success=case not in {"failure", "unsuccessful"}, awaiting_approval=case == "approval") + ) + else: + status = { + "failure": "stopped_at_stage", + "unsuccessful": "stopped_at_stage", + "approval": "gated_pending_approval", + }.get(case, "completed") + envelope = _serialise_state( + PipelineState( + pipeline_run_id="test", + pipeline_config_hash="test", + forgelm_version="test", + started_at="test", + final_status=status, + ) + ) + payload = {"empty": "", "usage": "", "malformed": "not JSON", "missing": "{}"}.get(case, json.dumps(envelope)) + producer = tmp_path / "forgelm" + producer.write_text( + '#!/bin/sh\nif [ "$FORGELM_TEST_CASE" = usage ]; then\n' + ' exec "$FORGELM_TEST_PYTHON" -m forgelm --definitely-invalid-argument\nfi\n' + 'printf "%s" "$FORGELM_TEST_JSON"\nexit "$FORGELM_TEST_STATUS"\n', + encoding="utf-8", + ) + producer.chmod(0o755) + env = dict( + os.environ, + PATH=str(tmp_path) + os.pathsep + os.environ["PATH"], + FORGELM_TEST_CASE=case, + FORGELM_TEST_PYTHON=sys.executable, + FORGELM_TEST_JSON=payload, + FORGELM_TEST_STATUS=str(producer_status), + ) + result = subprocess.run( + [_BASH, "-e", "-u", "-o", "pipefail", "-c", script], + cwd=_REPO_ROOT, + env=env, + capture_output=True, + text=True, + timeout=30, + ) + expected = {"success": 0, "failure": 3, "approval": 4, "usage": 2, "producer_error": 2}.get(case, 1) + assert result.returncode == expected, result.stderr + assert ("Awaiting human approval" in result.stderr) is (case == "approval") + + +@pytest.mark.parametrize("lang", ["en", "tr"]) +def test_json_reference_documents_pipeline_state_exception(lang: str) -> None: + """Keep the canonical wrapper convention consistent with pipeline serialization.""" + from typing import get_args + + from forgelm.cli._pipeline import FinalStatusLiteral, PipelineState, _serialise_state + + text = (_REPO_ROOT / "docs" / "usermanuals" / lang / "reference" / "json-output.md").read_text(encoding="utf-8") + common_heading = "## Common conventions" if lang == "en" else "## Ortak konvansiyonlar" + state_heading = "## Multi-stage pipeline state" if lang == "en" else "## Çok aşamalı pipeline durumu" + common = text.partition(common_heading + "\n")[2].partition("\n## ")[0] + section = text.partition(state_heading + "\n")[2].partition("\n## ")[0] + keys = {line.split("|")[1].strip().strip("`") for line in section.splitlines() if line.startswith("| `")} + envelope = _serialise_state( + PipelineState(pipeline_run_id="test", pipeline_config_hash="test", forgelm_version="test", started_at="test") + ) + assert keys == set(envelope) + assert "success" not in envelope + assert "`pipeline:`" in common and "`final_status`" in common + assert "#/operations/cicd" in common and "#412" in common + assert "Every envelope starts" not in common + assert "Her envelope" not in common + for status in get_args(FinalStatusLiteral): + assert f"`{status}`" in section + + @pytest.mark.parametrize("lang", ["en", "tr"]) class TestExitCodeTableConsistency: def test_ctrl_c_not_documented_as_130(self, lang: str) -> None: