From 2d3e2ee39e41bfb89a55d8f1b38626c2ff5e8c08 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sat, 22 Aug 2026 14:57:20 +0200 Subject: [PATCH 1/7] docs: align feed-directory contracts with registry sync --- AGENTS.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 86245d79..1d783489 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,8 +14,8 @@ What this repo owns: What this repo does not own: - runtime extractor behavior and CLI semantics (`html2rss/`) -- catalog metadata, YAML configs, or catalog serialization (`html2rss-configs/` → `Html2rss::Configs::Catalog`) -- catalog HTTP API (`html2rss-web/` → `GET /api/v1/configs`) +- feed YAML and signed registry bundle publishing (`html2rss-configs/` → `configs/`, `tool/registry-build`) +- catalog HTTP API and registry sync (`html2rss-web/` → `GET /api/v1/configs`, `Registry::Sync`) When docs describe behavior from other repos, treat those repos as source-of-truth and update docs to match them. @@ -30,9 +30,10 @@ Before substantial edits, state cross-repo context in your notes: Common contracts: - Feed Directory browse data comes from `{instance}/api/v1/configs` on a running `html2rss-web` instance (see OpenAPI in `html2rss-web`). +- Catalog wire rows include `id`, `path`, `directory`, `channel`, `parameters`, plus `source` (`registry` | `local`) and `registry` (registry id for bundle entries). The browse UI ignores `source`/`registry` today — wire parsing stays in `adapters/catalog-api.ts`. - Instance URL persistence: default public instance, `#!url=` hash deep link from the web app, browser localStorage, and filter state in URL query params (`q`, `topic`, `lang`, `sort`, `page`). - Deep link from `html2rss-web`: `https://html2rss.github.io/feed-directory/#!url={encodedInstanceUrl}` must keep working. -- Catalog metadata in YAML (`directory.title`, `directory.summary`, `directory.topics`) is authored in `html2rss-configs` only. +- Catalog metadata in YAML (`directory.*`, `registry.id`) is authored in `html2rss-configs` only; instances load verified bundles via registry sync. - Ruby gem docs should match `html2rss` behavior and CLI output. - Web application docs should match `html2rss-web` behavior and published OpenAPI. From 1d83efd3c3433370909650a1e9228c6ce4dc3fec Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sat, 22 Aug 2026 15:22:52 +0200 Subject: [PATCH 2/7] chore: format monitoring.mdx for Prettier lint Unblocks lint CI on docs PRs; formatting drift on main. --- .../web-application/reference/monitoring.mdx | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/content/docs/web-application/reference/monitoring.mdx b/src/content/docs/web-application/reference/monitoring.mdx index ddc31083..ebb0195c 100644 --- a/src/content/docs/web-application/reference/monitoring.mdx +++ b/src/content/docs/web-application/reference/monitoring.mdx @@ -7,11 +7,11 @@ import { Code } from "@astrojs/starlight/components"; ## Health Endpoints -| Endpoint | Auth | Purpose | -| --- | --- | --- | -| `GET /api/v1/health/live` | none | Process liveness | -| `GET /api/v1/health/ready` | none | Config readiness | -| `GET /api/v1/health` | bearer | Operator and uptime checks | +| Endpoint | Auth | Purpose | +| -------------------------- | ------ | -------------------------- | +| `GET /api/v1/health/live` | none | Process liveness | +| `GET /api/v1/health/ready` | none | Config readiness | +| `GET /api/v1/health` | bearer | Operator and uptime checks | Set `HEALTH_CHECK_TOKEN`, then call the authenticated endpoint: @@ -31,9 +31,9 @@ Set `SENTRY_DSN` to enable Issue capture in html2rss-web (Rack middleware and op Use separate Sentry projects for html2rss-web and botasaurus-scrape-api. Never share a DSN. -| Env var | Service | -| --- | --- | -| `SENTRY_DSN` | html2rss-web | +| Env var | Service | +| ----------------------- | --------------------- | +| `SENTRY_DSN` | html2rss-web | | `BOTASAURUS_SENTRY_DSN` | botasaurus-scrape-api | In `docker-compose.yml`, the botasaurus service maps `BOTASAURUS_SENTRY_DSN` into its own `SENTRY_DSN`, requires it at startup, and does not fall back to the web project's DSN. From 2a6bda2741950f858800a7880b79cefdc5e23dc7 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sat, 22 Aug 2026 17:16:08 +0200 Subject: [PATCH 3/7] docs: clarify local rows omit registry in catalog wire contract --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 1d783489..0091960b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ Before substantial edits, state cross-repo context in your notes: Common contracts: - Feed Directory browse data comes from `{instance}/api/v1/configs` on a running `html2rss-web` instance (see OpenAPI in `html2rss-web`). -- Catalog wire rows include `id`, `path`, `directory`, `channel`, `parameters`, plus `source` (`registry` | `local`) and `registry` (registry id for bundle entries). The browse UI ignores `source`/`registry` today — wire parsing stays in `adapters/catalog-api.ts`. +- Catalog wire rows include `id`, `path`, `directory`, `channel`, `parameters`, plus `source` (`registry` | `local`). Bundle entries also include `registry` (registry id); local rows omit `registry`. The browse UI ignores `source`/`registry` today — wire parsing stays in `adapters/catalog-api.ts`. - Instance URL persistence: default public instance, `#!url=` hash deep link from the web app, browser localStorage, and filter state in URL query params (`q`, `topic`, `lang`, `sort`, `page`). - Deep link from `html2rss-web`: `https://html2rss.github.io/feed-directory/#!url={encodedInstanceUrl}` must keep working. - Catalog metadata in YAML (`directory.*`, `registry.id`) is authored in `html2rss-configs` only; instances load verified bundles via registry sync. From bec4b59287044a1142ab4654137667b086499e3f Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sat, 22 Aug 2026 17:20:39 +0200 Subject: [PATCH 4/7] docs: align contributor paths with flat configs and registry.id Replace legacy lib/html2rss/configs links with configs/, document registry.id in contribution workflows, and use directory feed terminology. --- src/content/docs/creating-custom-feeds.mdx | 9 ++++++--- src/content/docs/feed-directory/index.mdx | 2 +- src/content/docs/get-involved/contributing.mdx | 4 ++-- src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx | 2 +- src/content/docs/ruby-gem/reference/mcp-server.mdx | 2 +- .../web-application/guides/use-the-feed-directory.mdx | 2 +- 6 files changed, 12 insertions(+), 9 deletions(-) diff --git a/src/content/docs/creating-custom-feeds.mdx b/src/content/docs/creating-custom-feeds.mdx index f9651a0c..7a731a44 100644 --- a/src/content/docs/creating-custom-feeds.mdx +++ b/src/content/docs/creating-custom-feeds.mdx @@ -189,14 +189,17 @@ there. **Help the community by sharing your config:** 1. Go to [html2rss-configs on GitHub](https://github.com/html2rss/html2rss-configs) -2. Click "Fork" → "Add file" → Create `domain.com/name.yml` under `lib/html2rss/configs/` -3. Include top-level `directory.topics`, `directory.title`, and mirror `channel.title` (required for Feed Directory configs). Optional `directory.summary` (max 160 characters). +2. Click "Fork" → "Add file" → Create `domain.com/name.yml` under `configs/` +3. Declare `registry.id` (unique feed identity), `directory.topics`, `directory.title`, and mirror `channel.title` (required for Feed Directory configs). Optional `directory.summary` (max 160 characters). 4. Paste your config → "Commit new file" → "Open pull request" Example catalog metadata: Date: Sat, 22 Aug 2026 17:23:04 +0200 Subject: [PATCH 5/7] docs: add directory.title requirement to contributing guide Align feed-config contribution steps with creating-custom-feeds for Feed Directory catalog metadata. --- src/content/docs/get-involved/contributing.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/get-involved/contributing.mdx b/src/content/docs/get-involved/contributing.mdx index 51cdb290..381db7e8 100644 --- a/src/content/docs/get-involved/contributing.mdx +++ b/src/content/docs/get-involved/contributing.mdx @@ -27,7 +27,7 @@ Are you missing an RSS feed for a website? You can create your own feed config a 1. Go to [html2rss-configs on GitHub](https://github.com/html2rss/html2rss-configs) 2. Click "Fork" → "Add file" → Create `domain.com/name.yml` under `configs/` -3. Declare `registry.id` (unique feed identity) and include top-level `directory.topics` (non-empty) from the controlled vocabulary (`tech`, `news`, `security`, `research`, … — see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config)) +3. Declare `registry.id`, `directory.topics` (non-empty), `directory.title`, and mirror `channel.title` (required for Feed Directory configs). Optional `directory.summary` (max 160 characters). See [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config) for topics and examples. 4. Prefer setting `channel.language` when the page language is clear 5. Paste your config → "Commit new file" → "Open pull request" From e658477b2cc5c5ac086e33c1cf077d08b2b1c972 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sat, 22 Aug 2026 17:26:18 +0200 Subject: [PATCH 6/7] docs: align Feed Directory metadata and terminology Unify registry.id / directory.* requirements across MCP and contributor guides, and replace stale "included feeds" / lowercase "feed directory" wording with consistent Feed Directory terminology. --- README.md | 2 +- src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx | 2 +- src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx | 2 +- src/content/docs/ruby-gem/reference/mcp-server.mdx | 2 +- src/content/docs/web-application/deployment.mdx | 2 +- src/content/docs/web-application/getting-started.mdx | 2 +- .../docs/web-application/guides/use-the-feed-directory.mdx | 6 +++--- src/content/docs/web-application/index.mdx | 2 +- 8 files changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 8d0a40cc..a7e35cb2 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ The Codespace provides a cloud development environment with Node.js pre-installe - **New to RSS?** → Start with the [web application](https://html2rss.github.io/web-application) - **Ruby Developer?** → Check out the [Ruby gem documentation](https://html2rss.github.io/ruby-gem) -- **Need a specific feed?** → Browse the [feed directory](https://html2rss.github.io/feed-directory) +- **Need a specific feed?** → Browse the [Feed Directory](https://html2rss.github.io/feed-directory) - **Want to contribute?** → See our [contributing guide](https://html2rss.github.io/get-involved/contributing) ## Contributing diff --git a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx index f1cd7957..601d00d0 100644 --- a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx +++ b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx @@ -94,7 +94,7 @@ When an agent is tasked with creating a durable YAML feed configuration: 1. **Capture:** Call `capture_config`. YAML is `payload.yaml` (`items` + `enhance: true`). Check `payload.articles_count` and `has_selectors`. 2. **Recon (optional):** Call `inspect_url` for `final_url`, `status`, `scheme_downgrade`, and `alternate_feeds` if the draft is weak. -3. **Rewrite:** If the destination is html2rss-configs, declare `registry.id`, add `directory.topics`, and set explicit channel `title`/`url`. Strive to keep `enhance: true` (set `false` only when chrome leaks into items). +3. **Rewrite:** If the destination is html2rss-configs, declare `registry.id`, `directory.topics` (non-empty), `directory.title`, and mirror `channel.title` (required for Feed Directory configs). Optional `directory.summary` (max 160 characters). Set explicit channel `url`. Strive to keep `enhance: true` (set `false` only when chrome leaks into items). 4. **Validate:** Pass `yaml` (or `config`) to `validate_config`. Exactly one of those arguments. 5. **Apply:** Call `apply_config`. `isError` plus `payload.item_count: 0` means the config is not shippable — channel title in the XML is not success. diff --git a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx index 8f11d655..98069240 100644 --- a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx @@ -100,7 +100,7 @@ MCP `capture_config` returns that YAML in `payload.yaml`. `validate_config` / `a 1. Validate: `html2rss validate my-feed.yml` 2. Render: `html2rss feed my-feed.yml` 3. Tighten the items selector, strategy, or `request.botasaurus` options if needed -4. For Feed Directory contributions, add `directory.topics` and keep `enhance: true` unless chrome leaks (see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config)) +4. For Feed Directory contributions, declare `registry.id`, `directory.topics` (non-empty), `directory.title`, and mirror `channel.title`; keep `enhance: true` unless chrome leaks (see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config)) ## Related diff --git a/src/content/docs/ruby-gem/reference/mcp-server.mdx b/src/content/docs/ruby-gem/reference/mcp-server.mdx index 0d6bb1d8..b0538ecb 100644 --- a/src/content/docs/ruby-gem/reference/mcp-server.mdx +++ b/src/content/docs/ruby-gem/reference/mcp-server.mdx @@ -165,7 +165,7 @@ Four-step workflow for a durable feed config: 3. Call `validate_config` with `yaml` (or `config`) — must not be `isError`. 4. Call `apply_config` — `isError` if zero items. Confirm `payload.item_count` before shipping. -If the destination is html2rss-configs, rewrite the draft with `registry.id`, `directory.topics`, and explicit channel `title`/`url`. +If the destination is html2rss-configs, rewrite the draft with `registry.id`, `directory.topics` (non-empty), `directory.title`, and mirror `channel.title` (required for Feed Directory configs). Optional `directory.summary` (max 160 characters). Set explicit channel `url`. - **Arguments:** - `url` _(string, required)_: Target URL to analyze. diff --git a/src/content/docs/web-application/deployment.mdx b/src/content/docs/web-application/deployment.mdx index 1024ee09..44d5d3a8 100644 --- a/src/content/docs/web-application/deployment.mdx +++ b/src/content/docs/web-application/deployment.mdx @@ -18,7 +18,7 @@ The examples use `html2rss/web:1`, the recommended major-version tag. Pin an exa There are two materially different deployment modes: - **Automatic generation enabled:** primary self-hosted workflow, requires `AUTO_SOURCE_ENABLED=true` and `HTML2RSS_ACCESS_TOKEN` -- **Included feeds fallback only:** lower-maintenance path when the Feed Directory already covers your needs +- **Feed Directory only:** lower-maintenance path when the Feed Directory already covers your needs If you do not need page-URL generation yet, keep `AUTO_SOURCE_ENABLED` off and ship the fallback mode only. diff --git a/src/content/docs/web-application/getting-started.mdx b/src/content/docs/web-application/getting-started.mdx index b047870a..78df5aa6 100644 --- a/src/content/docs/web-application/getting-started.mdx +++ b/src/content/docs/web-application/getting-started.mdx @@ -95,7 +95,7 @@ When output quality is poor, change the input URL first before assuming setup is ## Fallback Path -If the site you want is already in the feed directory, or you want a fast sample before creating another generated feed, use the feed directory on your own instance. +If the site you want is already in the Feed Directory, or you want a fast sample before creating another generated feed, use the Feed Directory on your own instance. - [Use the Feed Directory](/web-application/guides/use-the-feed-directory/) - [Feed Directory](/feed-directory/) diff --git a/src/content/docs/web-application/guides/use-the-feed-directory.mdx b/src/content/docs/web-application/guides/use-the-feed-directory.mdx index 23ac6a97..b34e4c74 100644 --- a/src/content/docs/web-application/guides/use-the-feed-directory.mdx +++ b/src/content/docs/web-application/guides/use-the-feed-directory.mdx @@ -7,7 +7,7 @@ description: "Use the built-in Feed Directory from your own html2rss-web instanc ## How to Use Them -1. **Find a feed** in the [Feed Directory](/feed-directory/) (loaded from the active instance catalog) +1. **Find a feed** in the [Feed Directory](/feed-directory/) (loaded from your active `html2rss-web` instance) 2. **Set the Instance URL** to your own `html2rss-web` address when it differs from the default 3. **Open the feed path** from your instance (for example `/anthropic.com/news.rss`) 4. **Add it to your feed reader** @@ -33,8 +33,8 @@ Just replace `localhost:4000` with your own `html2rss-web` address. Use a custom config when: -- the site you want is not in the directory -- the directory feed misses fields you care about +- the site you want is not in the Feed Directory +- the Feed Directory feed misses fields you care about - you need a more specific page, section, or output format Next steps: diff --git a/src/content/docs/web-application/index.mdx b/src/content/docs/web-application/index.mdx index 1d8cb2bf..7f8eca66 100644 --- a/src/content/docs/web-application/index.mdx +++ b/src/content/docs/web-application/index.mdx @@ -20,7 +20,7 @@ Start with **[Getting Started](/web-application/getting-started/)** to: - **Web interface:** direct feed creation from page URLs - **Access-controlled generation:** `POST /api/v1/feeds` expects the token configured on your instance -- **Included Feed Directory:** packaged fallback feeds you can use immediately from your own deployment +- **Feed Directory:** pre-built feeds synced from signed registry bundles on your own deployment - **Config-based extension path:** move to custom feeds when you need reviewable rules - **Caching and HTTP handling:** shipped as part of the deployment - **Generated API contract:** OpenAPI is published at `/openapi.yaml` From 8d08d4e1bcde89bcbc7ab3f4a3bc907a5afc4949 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 23 Aug 2026 12:31:06 +0200 Subject: [PATCH 7/7] docs(web-application): add custom feed registry guide --- .../guides/custom-feed-registry.mdx | 443 ++++++++++++++++++ src/content/docs/web-application/index.mdx | 1 + 2 files changed, 444 insertions(+) create mode 100644 src/content/docs/web-application/guides/custom-feed-registry.mdx diff --git a/src/content/docs/web-application/guides/custom-feed-registry.mdx b/src/content/docs/web-application/guides/custom-feed-registry.mdx new file mode 100644 index 00000000..be107d9e --- /dev/null +++ b/src/content/docs/web-application/guides/custom-feed-registry.mdx @@ -0,0 +1,443 @@ +--- +title: "Publish and Consume a Custom Feed Registry" +description: "Build, sign with Ed25519, host on GitHub Releases, and consume a custom registry.v1 feed bundle in html2rss-web." +--- + +A `registry.v1` bundle is a signed package of curated `html2rss` feed configurations. By publishing your own registry, your organization or community can distribute private or specialized RSS feed definitions that any `html2rss-web` instance can subscribe to, cryptographically verify, and serve. + +--- + +## Architecture & Trust Model + +A `registry.v1` feed registry uses an offline-first cryptographic trust model: + +```text +Publisher Repo GitHub Releases html2rss-web Instance ++-------------------------+ +--------------------------+ +-------------------------+ +| configs/**/*.yml | | registry-bundle.tar.gz | | config/registries.yml | +| manifest.json (SHA-256) | --CI--> | - manifest.json | ---> | - Pinned Public Key | +| manifest.sig (Ed25519) | | - manifest.sig | | - Sync & Catalog Policy | ++-------------------------+ | - configs/ | +-------------------------+ + +--------------------------+ | + v + Serves /.rss +``` + +### Bundle Contents + +Each release is packaged as a `registry-bundle.tar.gz` tarball containing: + +- **`configs/`**: Directory containing standard `html2rss` feed YAML configuration files. +- **`manifest.json`**: Index mapping every relative configuration path to its SHA-256 hex digest, along with metadata (`format: registry.v1`, `registry_id`, `version`, and `public_key_id`). +- **`manifest.sig`**: Strict Base64-encoded **Ed25519** digital signature computed over the canonical JSON representation of `manifest.json`. + +### Cryptographic Guarantees + +- **Integrity**: Every individual YAML file's SHA-256 hash is validated before loading. +- **Authenticity & Non-Repudiation**: The publisher signs the manifest using an Ed25519 private key. `html2rss-web` instances verify this signature against the publisher's pinned public key before activating feeds. +- **Atomic Rollback**: If a download, signature check, or YAML validation fails, `html2rss-web` keeps the previous active bundle running without downtime. + +--- + +## Prerequisites + +Before starting, ensure you have: + +- **OpenSSL 3.0+** (or 1.1.1+) installed locally for key generation. +- **Git** and a **GitHub repository** for hosting feed configs and automated releases. +- **Docker** or a running **`html2rss-web`** instance to consume the feeds. +- Zero Ruby knowledge required: the publisher CI uses standard POSIX shell and Python 3. + +--- + +## Step 1: Generate an Ed25519 Keypair + +Generate an Ed25519 private signing key and extract its public key in standard PEM format. + +### 1. Generate Private and Public Keys + +Run the following commands in your terminal: + +```bash +# 1. Generate an Ed25519 private key in PEM format +openssl genpkey -algorithm ED25519 -out registry-signing.pem + +# 2. Extract the public key in PEM format +openssl pkey -in registry-signing.pem -pubout -out registry-signing.pub + +# 3. Secure file permissions +chmod 600 registry-signing.pem +``` + +### 2. Configure Key Storage + +- **Private Key (`registry-signing.pem`)**: Store this secret securely. In your GitHub repository, go to **Settings** → **Secrets and variables** → **Actions** and add a secret named `REGISTRY_SIGNING_KEY` containing the entire contents of `registry-signing.pem`. +- **Public Key (`registry-signing.pub`)**: This key is public. You will share this PEM text with `html2rss-web` operators subscribing to your registry. + +--- + +## Step 2: Repository Structure & Feed Configuration + +Create a dedicated Git repository for your custom feeds with the following directory structure: + +```text +my-feed-registry/ +├── .github/ +│ └── workflows/ +│ └── release.yml +└── configs/ + └── example.com/ + └── engineering.yml +``` + +### Feed Configuration Requirements + +Every YAML file placed inside `configs/` must include explicit `registry` and `directory` blocks: + +```yaml +registry: + id: example.com/engineering + aliases: [] + +directory: + title: Example Corp — Engineering Blog + summary: Updates and technical deep dives from the Example engineering team. + topics: + - tech + - engineering + +channel: + title: Example Corp — Engineering Blog + url: https://example.com/blog + language: en + +selectors: + items: + selector: "article.post" + title: + selector: "h2.entry-title a" + url: + selector: "h2.entry-title a" + extractor: "href" + description: + selector: "p.summary" +``` + +### Key Configuration Rules + +- **`registry.id`**: Unique identifier slug for this feed (`[a-z0-9._/-]`). Consumers access the feed at `/.rss` (for example, `/example.com/engineering.rss`). +- **`registry.aliases`**: Optional list of previous IDs if a feed is renamed, ensuring existing subscribers continue to receive updates. +- **`directory.title`**: Human-readable name used when listing feeds in the catalog. Mirror this in `channel.title`. +- **`directory.topics`**: Array containing 1–2 topics from the controlled vocabulary: `sports`, `energy`, `tech`, `science`, `news`, `entertainment`, `jobs`, `finance`, `security`, `travel`, `environment`, `consumer`, `civic`, `product`, `research`. + +--- + +## Step 3: CI Release Workflow + +Automate the bundle compilation, canonical manifest generation, Ed25519 signing, and GitHub Release publication using GitHub Actions. + +Create `.github/workflows/release.yml` in your repository: + +```yaml +name: Release Feed Registry Bundle + +on: + push: + tags: + - "v*" + +permissions: + contents: write + +jobs: + release: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.x" + + - name: Build manifest and canonical bytes + env: + REGISTRY_ID: custom + REGISTRY_PUBLIC_KEY_ID: custom:key:2026 + TAG_VERSION: ${{ github.ref_name }} + run: | + python3 - << 'EOF' + import hashlib, json, os, sys + + registry_id = os.environ.get("REGISTRY_ID", "custom") + version = os.environ.get("TAG_VERSION", "1.0.0").lstrip("v") + public_key_id = os.environ.get("REGISTRY_PUBLIC_KEY_ID", "custom:key:2026") + + files = {} + configs_dir = "configs" + + if not os.path.isdir(configs_dir): + sys.exit("Error: configs/ directory not found.") + + for root, _, filenames in os.walk(configs_dir): + for filename in sorted(filenames): + if filename.endswith((".yml", ".yaml")): + full_path = os.path.join(root, filename) + rel_path = os.path.relpath(full_path).replace("\\", "/") + with open(full_path, "rb") as f: + digest = hashlib.sha256(f.read()).hexdigest() + files[rel_path] = digest + + if not files: + sys.exit("Error: No config files found in configs/.") + + manifest = { + "format": "registry.v1", + "registry_id": registry_id, + "version": version, + "public_key_id": public_key_id, + "files": dict(sorted(files.items())) + } + + def deep_sort(obj): + if isinstance(obj, dict): + return {k: deep_sort(v) for k, v in sorted(obj.items())} + if isinstance(obj, list): + return [deep_sort(x) for x in obj] + return obj + + # 1. Pretty manifest.json for human inspection + with open("manifest.json", "w", encoding="utf-8") as f: + json.dump(manifest, f, indent=2) + f.write("\n") + + # 2. Deterministic canonical bytes for Ed25519 signing + with open("canonical_manifest.json", "wb") as f: + canonical_bytes = json.dumps(deep_sort(manifest), separators=(",", ":"), ensure_ascii=False).encode("utf-8") + f.write(canonical_bytes) + + print(f"Manifest created with {len(files)} configs.") + EOF + + - name: Sign manifest with Ed25519 + env: + SIGNING_KEY: ${{ secrets.REGISTRY_SIGNING_KEY }} + run: | + # Write private key securely to a temporary file + echo "$SIGNING_KEY" > signing_key.pem + chmod 600 signing_key.pem + + # Compute Ed25519 signature over canonical JSON bytes + openssl pkeyutl -sign \ + -inkey signing_key.pem \ + -rawin \ + -in canonical_manifest.json \ + -out signature.raw + + # Encode signature to strict single-line Base64 + openssl base64 -A -in signature.raw -out manifest.sig + + # Clean up private key and raw signatures + rm -f signing_key.pem signature.raw canonical_manifest.json + + - name: Package registry bundle + run: | + mkdir -p dist + tar -czf dist/registry-bundle.tar.gz manifest.json manifest.sig configs + ls -lh dist/registry-bundle.tar.gz + + - name: Publish GitHub Release + uses: softprops/action-gh-release@v2 + with: + files: dist/registry-bundle.tar.gz + generate_release_notes: true +``` + +### Triggering a Release + +Commit your files, tag the commit with a version, and push the tag to GitHub: + +```bash +git add . +git commit -m "Add initial custom feed configs" +git tag v2026.08.23 +git push origin v2026.08.23 +``` + +The workflow executes, attaches `registry-bundle.tar.gz` to the GitHub Release, and makes it available for download. + +--- + +## Step 4: Consuming in `html2rss-web` + +To consume your new custom registry, configure your `html2rss-web` instance using `config/registries.yml`. + +### 1. Update `config/registries.yml` + +Create or edit `config/registries.yml` alongside your `html2rss-web` deployment: + +```yaml +# Feed lookup precedence (first match wins) +precedence: + - custom + - official + +registries: + custom: + sync: + # Direct HTTPS download URL for the release asset + url: https://github.com/my-org/my-feed-registry/releases/latest/download/registry-bundle.tar.gz + # Optional: pin to a specific release tag + # pin_version: v2026.08.23 + # Optional: cap maximum version to prevent unexpected upgrades + # max_version: v2026.08.23 + auto_promote: true # Automatically activate new bundles on sync + catalog: true # Include feeds in GET /api/v1/configs and Web Directory + public_key_id: custom:key:2026 # Must match public_key_id in manifest.json + public_key: | + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAG... (your registry-signing.pub contents here) ...= + -----END PUBLIC KEY----- + allowed_channel_domains: # Optional: security whitelist for scraped domains + - example.com + - my-org.com + + official: + sync: + channel: html2rss-official + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- +``` + +### Configuration Options Reference + +| Option | Type | Default | Description | +| ------------------------- | ------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | +| `precedence` | Array | `["official"]` | Evaluation order when looking up feeds by ID. | +| `sync.url` | String | — | HTTPS download URL for the `registry-bundle.tar.gz` archive. | +| `sync.pin_version` | String | — | Optional tag to lock sync to an exact release version. | +| `sync.max_version` | String | — | Maximum version cap; rejects any bundle with a higher version. | +| `auto_promote` | Boolean | `false` | When `true`, automatically swaps the active bundle upon verified sync. When `false`, stages to `.staging/` for manual approval. | +| `catalog` | Boolean | `true` | When `true`, feeds appear in `GET /api/v1/configs` and the web directory. When `false`, feeds are private and only accessible by direct URL. | +| `public_key_id` | String | — | Identifier of the key that must match `manifest.json`. | +| `public_key` | String | — | Full PEM text of the publisher's Ed25519 public key. | +| `allowed_channel_domains` | Array | `[]` | Optional domain whitelist. Rejects any config whose `channel.url` targets a host outside this list. | + +### 2. Mount in Docker Compose + +Mount your custom `config/registries.yml` into your `html2rss-web` container and set allowed download hostnames if hosting outside GitHub: + +```yaml +services: + html2rss-web: + image: html2rss/web:1 + restart: unless-stopped + ports: + - "4000:3000" + environment: + - HTML2RSS_SECRET_KEY=replace_with_openssl_rand_hex_32 + - REGISTRY_SYNC_ON_BOOT=true + - REGISTRY_SYNC_INTERVAL_HOURS=24 + # Optional: add custom CDN/host if not using github.com + # - REGISTRY_SYNC_ALLOWED_HOSTS=registry.example.com,cdn.example.com + volumes: + - ./config/registries.yml:/app/config/registries.yml:ro + - registry-data:/app/data/registries + +volumes: + registry-data: +``` + +--- + +## Step 5: Verify and Manage Runtime Sync + +`html2rss-web` includes built-in CLI commands to check status, trigger syncs, and promote staged bundles. + +### 1. Check Registry Status + +Inspect the state of all configured registries: + +```bash +# Inside container or via docker exec +docker compose exec html2rss-web bin/registry-sync --status +``` + +Example output: + +```text +registry mode version staged_version updated_at sync_url last_error +custom sync 2026.08.23 - 2026-08-23 12:00:00 UTC https://github.com/my-org/my-feed-registry/releases/latest/download/... - +official sync 2026.08.22 - 2026-08-23 11:30:00 UTC https://github.com/html2rss/html2rss-configs/releases/latest/download/... - +``` + +### 2. Manual Sync & Staging Promotion + +If you have `auto_promote: false`, bundles are verified and placed in staging. Promote them when ready: + +```bash +# 1. Test sync without modifying active feeds (dry run) +docker compose exec html2rss-web bin/registry-sync --registry custom --dry-run + +# 2. Fetch and verify the latest bundle (stages when auto_promote is false) +docker compose exec html2rss-web bin/registry-sync --registry custom + +# 3. Promote staged bundle to become active +docker compose exec html2rss-web bin/registry-sync --promote --registry custom +``` + +### 3. Verify Live Feeds & API Catalog + +- **Fetch the RSS feed**: + + ```bash + curl -i http://localhost:4000/example.com/engineering.rss + ``` + + Expect HTTP `200 OK` and valid RSS 2.0 XML. + +- **Check the Config Catalog**: + ```bash + curl -s http://localhost:4000/api/v1/configs | jq '.data.configs[] | select(.registry == "custom")' + ``` + +--- + +## Step 6: Key Rotation & Incident Response + +### Key Rotation Workflow + +When rotating signing keys: + +1. **Generate a new keypair** with a new identifier (for example, `custom:key:2027`). +2. **Update consumer `config/registries.yml`** files with the new `public_key` and `public_key_id`. +3. **Publish the new release** in your registry repository signed with the new private key. +4. Existing bundles signed with older keys continue to function until the next sync replaces them. + +### Incident Response Freeze + +If a downstream release is compromised or introduces unintended feed errors: + +1. **Cap the version**: Add `sync.max_version: ` to `config/registries.yml`. +2. **Disable automatic sync**: Set `REGISTRY_SYNC_INTERVAL_HOURS=0` in the environment to prevent periodic refreshes while investigating. +3. Because `html2rss-web` uses atomic swaps, any sync that fails signature checks or YAML validation automatically leaves the active, known-good bundle in place. + +--- + +## Step 7: Air-Gapped / Offline Deployment (`path` Mode) + +For secure environments without outbound Internet access, deploy pre-verified bundles via directory bind-mounts: + +```yaml +registries: + custom: + path: /app/mounted_registries/custom + catalog: true +``` + +Mount the folder containing `manifest.json` and `configs/` into `/app/mounted_registries/custom`. In `path` mode, `html2rss-web` validates file digests using `:integrity_only` mode without requiring network access. diff --git a/src/content/docs/web-application/index.mdx b/src/content/docs/web-application/index.mdx index 7f8eca66..3e8d1d1a 100644 --- a/src/content/docs/web-application/index.mdx +++ b/src/content/docs/web-application/index.mdx @@ -39,6 +39,7 @@ Once your instance is running, learn how to use its core features: - **[Use automatic feed generation](/web-application/guides/use-automatic-feed-generation/)**: create feeds from page URLs - **[Use the Feed Directory](/web-application/guides/use-the-feed-directory/)**: use the built-in Feed Directory as a fallback +- **[Publish and consume a custom feed registry](/web-application/guides/custom-feed-registry/)**: distribute private or curated feeds with cryptographic signing - **[Monitoring](/web-application/reference/monitoring/)**: keep your instance healthy ## For Integrations