Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,15 @@ All notable changes to ForgeLM are documented here.

### Fixed

- CI/CD examples capture ForgeLM and `jq` statuses separately: a JSON parser's exit 4 cannot be mistaken for an
approval hold, and a multi-stage approval hold retains ForgeLM's exit 4. The guide's JSON-reference section links
use the compiled heading IDs and preserve the page route and heading on reload
(F-PR1073-A1-01, F-PR1073-A2-01).
- CI/CD user-manual examples in English and Turkish read the training envelope's boolean `success` field,
keep plain-text stderr logs separate from the JSON result, and distinguish the multi-stage `final_status`
summary. The canonical JSON reference documents the pipeline-state exception and its actual fields in both languages.
Bash gate examples use `pipefail` and `jq -e` so failed or awaiting-approval runs do not pass
(#433, #216; CR-A04-008, DOC-D3B02-016). Non-JSON library output before the envelope remains tracked in #412.
- **`forgelm purge --row-id` warns about the weights a run actually leaves behind.** This restores the documented
contract that `data.erasure_warning_memorisation` fires when a trained model exists for the erased row. The check
used to look only for a literal `<--output-dir>/final_model/`, so a default invocation (`--output-dir` omitted,
Expand Down
7 changes: 7 additions & 0 deletions docs/standards/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,13 @@ tests/
- `conftest.py` holds shared fixtures only. Domain-specific helpers live in the test files that need them.
- A new feature PR adds the matching `test_*.py` in the **same** PR. No "tests in next PR."

## Documentation regressions

The CI/CD manual regression cases execute the shipped snippets with POSIX Bash and `jq`.
They are skipped with an explicit prerequisite reason on hosts without these tools; GitHub's Linux runners provide both.
Guide navigation regressions compile the EN/TR Markdown with the development-only `markdown` dependency and run
the shipped router in the Node VM harness, checking the compiled heading target and route after reload.

## Browser wizard regressions

Browser wizard regressions live in `tests/site/*.test.cjs`. Run
Expand Down
61 changes: 57 additions & 4 deletions docs/usermanuals/en/operations/cicd.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,14 +143,67 @@ pipeline {

## JSON output mode

For programmatic parsing of run outputs, use `--output-format json`:
For a single-stage training run (a config without `pipeline:`), the result envelope has a boolean `success` field:

```shell
$ forgelm --config configs/run.yaml --output-format json | jq '.verdict'
"success"
$ forgelm --config configs/run.yaml --output-format json | jq '.success'
Comment thread
sourcery-ai[bot] marked this conversation as resolved.
Resolved
true
```

Every event ForgeLM logs to stderr is also emitted as a JSON object on stdout, ready to pipe into your log aggregator or dashboard.
The JSON envelope goes to stdout; human-readable logs remain plain text on stderr.
JSON mode suppresses logs below WARNING and does not mirror log events to stdout.
See [JSON output schemas](#/reference/json-output) for every field.

Printing `.success` is not a CI gate: `jq` normally exits 0 even for `false`.
Use `PIPESTATUS` to capture ForgeLM and `jq` results separately before another command overwrites them.
Only ForgeLM exit 4 means awaiting approval; `jq` can also exit 4 on empty input.
Preserve non-zero ForgeLM exits. If ForgeLM exits 0 but JSON validation fails, this wrapper exits 1.
See [Exit Codes](#/reference/exit-codes) for the ForgeLM contract.

```bash
set -o pipefail
command_status=(0 0)
forgelm --config configs/run.yaml --output-format json \
| jq -e '.success == true' || command_status=("${PIPESTATUS[@]}")
forgelm_status=${command_status[0]}
json_status=${command_status[1]}
if [ "$forgelm_status" -eq 4 ]; then
printf 'Awaiting human approval\n' >&2
exit 4
elif [ "$forgelm_status" -ne 0 ]; then
exit "$forgelm_status"
elif [ "$json_status" -ne 0 ]; then
printf 'JSON validation failed (jq exit %s)\n' "$json_status" >&2
exit 1
fi
exit 0
```

A config with `pipeline:` emits a pipeline state with `final_status`, not a top-level `success`.
Comment thread
cemililik marked this conversation as resolved.
Resolved
For a multi-stage run, require `completed`; `stopped_at_stage` and `gated_pending_approval` must not pass:

```bash
set -o pipefail
command_status=(0 0)
forgelm --config configs/pipeline.yaml --output-format json \
| jq -e '.final_status == "completed"' || command_status=("${PIPESTATUS[@]}")
forgelm_status=${command_status[0]}
json_status=${command_status[1]}
if [ "$forgelm_status" -eq 4 ]; then
printf 'Awaiting human approval\n' >&2
exit 4
elif [ "$forgelm_status" -ne 0 ]; then
exit "$forgelm_status"
elif [ "$json_status" -ne 0 ]; then
printf 'JSON validation failed (jq exit %s)\n' "$json_status" >&2
exit 1
fi
exit 0
```

Known limitation: training and pipeline runs can print non-JSON library output before the envelope
([#412](https://github.com/HodeTech/ForgeLM/issues/412)). The examples require clean JSON stdout;
a JSON parse error must fail the CI step.

## Caching models

Expand Down
32 changes: 30 additions & 2 deletions docs/usermanuals/en/reference/json-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,40 @@ This page is the canonical reference. CI/CD pipelines that parse `forgelm` outpu

## Common conventions

- **stdout vs stderr.** The JSON envelope goes to **stdout**. Human-friendly logs (info / warning / error) go to **stderr**. Pipe `forgelm ... --output-format json | jq .` and read your operator-facing messages from `2>` separately.
- **Top-level wrapper.** Every envelope starts with `"success": true | false`. Consumers can branch on this single key before parsing the rest.
- **stdout vs stderr.** The JSON envelope goes to **stdout**; human-friendly logs go to **stderr**.
Training and pipeline runs can also print non-JSON library output before the envelope
([#412](https://github.com/HodeTech/ForgeLM/issues/412)); a parse error must fail the CI step.
See [CI/CD Pipelines](#/operations/cicd) for Bash gate examples.
- **Top-level wrapper.** Most envelopes start with `"success": true | false`.
Summaries for configs containing `pipeline:` are the exception: they use `final_status` instead.
See [Multi-stage pipeline state](#/reference/json-output#multi-stage-pipeline-state).
- **Error envelope.** When `success: false`, the envelope carries `"error": "<message>"` (string). Optional richer fields (`exit_code`, `error_type`, `details`) MAY be present per the [error-handling standard on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/standards/error-handling.md). Consumers that need certainty on these fields should also check the process exit code via `$?`.
- **Exit codes.** See [Exit Codes](#/reference/exit-codes). `success: false` always pairs with a non-zero exit. `success: true` pairs with exit `0` in the normal case, with **one exception**: a training run halted at the Article 14 human-approval gate emits `success: true` **and** `awaiting_approval: true` while exiting `4` (the model is staged, not yet promoted). Consumers that must distinguish "done" from "awaiting human approval" should read the `awaiting_approval` discriminator (or check `$?`), not `success` alone.
- **Argument-parsing errors are the exception.** The JSON-envelope guarantee begins **after** arguments parse successfully. A usage error detected at parse time (unknown flag, bad `--choice`, out-of-range validator value such as `--workers 0`) is reported by `argparse` itself: a human-readable `usage: ... error: ...` line on **stderr** and exit code `2`, with **no** JSON envelope on stdout. A consumer that reads stdout-JSON on every non-zero exit must treat empty stdout (paired with non-zero `$?`) as a parse/usage error and fall back to stderr for the message. The envelope contract applies only to errors raised once the dispatcher is running.

## Multi-stage pipeline state

`forgelm --config <pipeline.yaml> --output-format json` with a `pipeline:` block emits a pipeline-state summary.
The same state is saved in `pipeline_state.json`. This summary has no top-level `success` or `error` wrapper;
an error caught before a state summary is produced may instead use the common `success: false` error envelope.

| Key | Type | Notes |
|---|---|---|
| `pipeline_run_id` | str | Identifier of the pipeline run. |
| `pipeline_config_hash` | str | Hash of the pipeline configuration bytes. |
| `forgelm_version` | str | ForgeLM version that produced the state. |
| `started_at` | str | UTC start timestamp. |
| `finished_at` | str or null | UTC finish timestamp, or null while the run is in progress. |
| `final_status` | str | Pipeline status; see the values below. |
| `stopped_at` | str or null | Stage where the chain stopped, or null. |
| `stages` | list[object] | Per-stage snapshots: status, exit code, error, metrics, model paths and timestamps. |

`final_status` is one of `in_progress`, `completed`, `stopped_at_stage`, or `gated_pending_approval`.
Check this field together with the process exit code, rather than testing `.success` on a state summary.
See [Multi-stage pipelines](#/training/pipelines) for stage behavior and
the [PipelineStageState source](https://github.com/HodeTech/ForgeLM/blob/main/forgelm/cli/_pipeline.py)
for the nested stage fields.

## `forgelm doctor`

Environment check. See [Doctor command](#/getting-started/first-run).
Expand Down
62 changes: 58 additions & 4 deletions docs/usermanuals/tr/operations/cicd.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,14 +144,68 @@ pipeline {

## JSON çıktı modu

Koşu çıktısının programatik parse'ı için `--output-format json` kullanın:
Tek aşamalı bir eğitim koşusunda (`pipeline:` içermeyen config), sonuç zarfındaki `success` alanı boolean türündedir:

```shell
$ forgelm --config configs/run.yaml --output-format json | jq '.verdict'
"success"
$ forgelm --config configs/run.yaml --output-format json | jq '.success'
true
```

ForgeLM'in stderr'a yazdığı her olay stdout'ta da JSON nesnesi olarak çıkar — log aggregator'ınıza veya dashboard'unuza pipe'lamaya hazır.
JSON zarfı stdout'a gider; okunabilir loglar stderr'de düz metin olarak kalır.
JSON modu WARNING altındaki logları bastırır ve log olaylarını stdout'a kopyalamaz.
Tüm alanlar için [JSON çıktı şemaları](#/reference/json-output) sayfasına bakın.

`.success` değerini yazdırmak bir CI kontrolü değildir: `jq`, `false` için de normalde 0 ile çıkar.
Başka bir komut üzerine yazmadan önce ForgeLM ve `jq` sonuçlarını `PIPESTATUS` ile ayrı yakalayın.
Yalnızca ForgeLM'in 4 ile çıkması onay beklendiğini gösterir; `jq` boş girdide de 4 ile çıkabilir.
ForgeLM'in sıfır dışındaki kodlarını koruyun. ForgeLM 0 ile çıkıp JSON doğrulaması başarısızsa bu betik 1 ile çıkar.
ForgeLM sözleşmesi için [Çıkış Kodları](#/reference/exit-codes) sayfasına bakın.

```bash
set -o pipefail
command_status=(0 0)
forgelm --config configs/run.yaml --output-format json \
| jq -e '.success == true' || command_status=("${PIPESTATUS[@]}")
forgelm_status=${command_status[0]}
json_status=${command_status[1]}
if [ "$forgelm_status" -eq 4 ]; then
printf 'Awaiting human approval\n' >&2
exit 4
elif [ "$forgelm_status" -ne 0 ]; then
exit "$forgelm_status"
elif [ "$json_status" -ne 0 ]; then
printf 'JSON validation failed (jq exit %s)\n' "$json_status" >&2
exit 1
fi
exit 0
```

`pipeline:` içeren config, üst düzey `success` yerine `final_status` taşıyan bir pipeline durumu üretir.
Çok aşamalı bir koşuda `completed` şartını arayın;
`stopped_at_stage` ve `gated_pending_approval` başarılı sayılmamalıdır:

```bash
set -o pipefail
command_status=(0 0)
forgelm --config configs/pipeline.yaml --output-format json \
| jq -e '.final_status == "completed"' || command_status=("${PIPESTATUS[@]}")
forgelm_status=${command_status[0]}
json_status=${command_status[1]}
if [ "$forgelm_status" -eq 4 ]; then
printf 'Awaiting human approval\n' >&2
exit 4
elif [ "$forgelm_status" -ne 0 ]; then
exit "$forgelm_status"
elif [ "$json_status" -ne 0 ]; then
printf 'JSON validation failed (jq exit %s)\n' "$json_status" >&2
exit 1
fi
exit 0
```

Bilinen sınırlama: eğitim ve pipeline koşuları, zarftan önce JSON olmayan kütüphane çıktısı yazabilir
([#412](https://github.com/HodeTech/ForgeLM/issues/412)). Örnekler temiz JSON stdout gerektirir;
JSON ayrıştırma hatası CI adımını başarısız kılmalıdır.

## Model cache

Expand Down
33 changes: 31 additions & 2 deletions docs/usermanuals/tr/reference/json-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,41 @@ Bu sayfa kanonik referanstır. `forgelm` çıktısını parse eden CI/CD pipelin

## Ortak konvansiyonlar

- **stdout vs stderr.** JSON envelope **stdout**'a gider. İnsana okunabilir loglar (info / warning / error) **stderr**'e gider. `forgelm ... --output-format json | jq .` ile pipe edip operatör mesajlarını `2>` ile ayrı okuyun.
- **Üst seviye sarmalayıcı.** Her envelope `"success": true | false` ile başlar. Tüketiciler önce bu tek anahtarda branch açabilir.
- **stdout vs stderr.** JSON zarfı **stdout**'a, okunabilir loglar **stderr**'e gider.
Eğitim ve pipeline koşuları, zarftan önce JSON olmayan kütüphane çıktısı da yazabilir
([#412](https://github.com/HodeTech/ForgeLM/issues/412)); ayrıştırma hatası CI adımını başarısız kılmalıdır.
Bash kontrol örnekleri için [CI/CD Pipeline'ları](#/operations/cicd) sayfasına bakın.
- **Üst seviye sarmalayıcı.** Zarfların çoğu `"success": true | false` ile başlar.
`pipeline:` içeren config'lerin özetleri istisnadır: bunun yerine `final_status` kullanırlar.
Bkz. [Çok aşamalı pipeline durumu](#/reference/json-output#cok-asamal-pipeline-durumu).
- **Hata envelope'u.** `success: false` olduğunda envelope `"error": "<mesaj>"` (string) taşır. Opsiyonel zenginleştirilmiş alanlar (`exit_code`, `error_type`, `details`) [GitHub'daki error-handling standardı](https://github.com/HodeTech/ForgeLM/blob/main/docs/standards/error-handling.md) gereği var olabilir. Bu alanlardan kesinlik isteyen tüketiciler `$?` ile süreç exit kodunu da kontrol etmelidir.
- **Exit kodları.** Bkz. [Exit Kodları](#/reference/exit-codes). `success: false` her zaman non-zero bir exit ile eşleşir. `success: true` normal durumda exit `0` ile eşleşir, **tek istisna** dışında: Article 14 insan-onay gate'inde duraklayan bir eğitim çalışması `success: true` **ve** `awaiting_approval: true` emit ederken exit `4` ile çıkar (model stage'lenmiştir, henüz promote edilmemiştir). "Bitti" ile "insan onayı bekliyor"u ayırması gereken tüketiciler tek başına `success` yerine `awaiting_approval` discriminator'ını (veya `$?`'i) okumalıdır.
- **Argüman-ayrıştırma hataları istisnadır.** JSON-envelope garantisi argümanlar başarıyla ayrıştırıldıktan **sonra** başlar. Ayrıştırma anında saptanan bir kullanım hatası (bilinmeyen flag, hatalı `--choice`, `--workers 0` gibi aralık-dışı validator değeri) `argparse`'in kendisi tarafından raporlanır: **stderr**'e insana okunabilir bir `usage: ... error: ...` satırı ve exit kodu `2`, stdout'ta **hiçbir** JSON envelope olmadan. Her non-zero exit'te stdout-JSON okuyan bir tüketici, boş stdout'u (non-zero `$?` ile eşleşen) bir ayrıştırma/kullanım hatası olarak ele almalı ve mesaj için stderr'e dönmelidir. Envelope kontratı yalnızca dispatcher çalışmaya başladıktan sonra ortaya çıkan hatalar için geçerlidir.

## Çok aşamalı pipeline durumu

`pipeline:` bloğuyla `forgelm --config <pipeline.yaml> --output-format json` çalıştırıldığında
pipeline durum özeti üretilir.
Aynı durum `pipeline_state.json` dosyasına kaydedilir. Bu özette üst düzey `success` veya `error` sarmalayıcısı yoktur;
durum özeti üretilmeden yakalanan bir hata ise ortak `success: false` hata zarfını kullanabilir.

| Anahtar | Tür | Notlar |
|---|---|---|
| `pipeline_run_id` | str | Pipeline koşusunun kimliği. |
| `pipeline_config_hash` | str | Pipeline yapılandırma baytlarının hash'i. |
| `forgelm_version` | str | Durumu üreten ForgeLM sürümü. |
| `started_at` | str | UTC başlangıç zaman damgası. |
| `finished_at` | str veya null | UTC bitiş zaman damgası; koşu sürerken null. |
| `final_status` | str | Pipeline durumu; değerleri aşağıda verilmiştir. |
| `stopped_at` | str veya null | Zincirin durduğu aşama; durmadıysa null. |
| `stages` | list[object] | Aşama başına durum, çıkış kodu, hata, metrik, model yolu ve zaman damgası kayıtları. |

`final_status`, `in_progress`, `completed`, `stopped_at_stage` veya `gated_pending_approval` değerlerinden biridir.
Bir durum özetinde `.success` aramak yerine bu alanı süreç çıkış koduyla birlikte kontrol edin.
Aşama davranışı için [Çok aşamalı pipeline'lar](#/training/pipelines) sayfasına,
iç içe aşama alanları için
[PipelineStageState kaynağına](https://github.com/HodeTech/ForgeLM/blob/main/forgelm/cli/_pipeline.py) bakın.

## `forgelm doctor`

Ortam kontrolü. Bkz. [Doctor komutu](#/getting-started/first-run).
Expand Down
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,8 @@ chat = [

# Developer tooling — test runner, coverage, linter/formatter.
dev = [
# Compile manual pages in the guide navigation regression tests.
"markdown>=3.5.0,<4.0.0",
"pytest>=8.0.0",
"pytest-cov>=5.0.0",
# pytest-randomly shuffles test order per session so order-dependent
Expand Down
20 changes: 19 additions & 1 deletion site/js/guide.js
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@
}

function navigate(route, replace) {
state.route = route;
state.route = route.split('#')[0];
var newHash = '#/' + route;
if (window.location.hash !== newHash) {
if (replace) {
Expand Down Expand Up @@ -249,6 +249,16 @@
+ pager
+ editLink);

content.querySelectorAll('a[href^="#/"]').forEach(function (link) {
link.addEventListener('click', function (event) {
if (event.button > 0 || event.ctrlKey || event.metaKey || event.shiftKey || event.altKey) return;
var nextRoute = parseHash(link.getAttribute('href'));
if (!nextRoute) return;
event.preventDefault();
navigate(nextRoute, false);
});
});

// Re-translate any chrome that data-i18n updated nodes inside the content.
if (window.ForgeLMi18n && window.ForgeLMi18n.setLanguage) {
window.ForgeLMi18n.setLanguage(state.lang);
Expand All @@ -268,6 +278,14 @@
// After DOM updates, trigger scrollspy refresh.
setTimeout(observeHeadings, 50);
updateProgressBar();
scrollToRouteAnchor();
}

function scrollToRouteAnchor() {
var anchor = window.location.hash.split('#')[2];
if (!anchor) return;
var target = document.getElementById(anchor);
if (target) target.scrollIntoView({ block: 'start' });
}

function renderNotFound(route) {
Expand Down
Loading
Loading