Move .env ownership to this repo and default CDC to pgoutput

pgcopydb-helpers/AGENTS.md

@@ -6,14 +6,34 @@ This file provides guidance to AI coding assistants (Claude Code, Cursor, Copilo
 
 These scripts run on a **migration instance** (EC2 or GCP Compute) that sits between the source PostgreSQL database and the [PlanetScale for Postgres](https://planetscale.com/docs/postgres/) target. The instance has pgcopydb installed and network access to both databases.
 
-All scripts read connection strings from `~/.env`:
+All scripts read their configuration from `~/.env`. This repo owns the reference copy: `env-template`. The user copies it to `~/.env` and edits the values:
+
+```bash
+cp ~/env-template ~/.env
+chmod 600 ~/.env
+```
 
 ```bash
 export PGCOPYDB_SOURCE_PGURI='postgresql://user:pass@source-host:5432/dbname'
 export PGCOPYDB_TARGET_PGURI='postgresql://user:pass@target-host:5432/dbname'
-export SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...'  # optional, for Slack alerts
+
+export TABLE_JOBS=8                       # parallel COPY workers
+export INDEX_JOBS=6                       # parallel index build workers
+export SPLIT_TABLES_LARGER_THAN=50GB      # copy larger tables in parts
+export OUTPUT_PLUGIN=pgoutput             # logical decoding plugin for CDC
+export FILTER_FILE=~/filters.ini          # pgcopydb filter file
+
+#export SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...'  # optional
 ```
 
+Each script applies the same default if a variable is unset, with the form `TABLE_JOBS="${TABLE_JOBS:-8}"`. To add a new tunable, edit `env-template`, the scripts that use it, the Configuration table below, and the Script Configuration table in `README.md`.
+
+The `pgcopydb-templates/` templates no longer create `~/.env`. They copy the whole `pgcopydb-helpers/` directory to `/home/ubuntu/`, so `env-template` arrives with the scripts and the user makes `~/.env` from it. Do not add `.env` generation back to a template.
+
+`env-template` cannot be named `.env` in this repo. `.gitignore` blocks that name to keep credentials out of git, and the templates deploy with `cp -r pgcopydb-helpers/* /home/ubuntu/`, where the shell glob `*` does not match dotfiles.
+
+Every script that sources `~/.env` first checks that the file exists and, if not, prints the `cp ~/env-template ~/.env` command and exits 1. Keep that guard in any new script that reads `~/.env`.
+
 ## Script Reference
 
 Scripts are organized by migration phase: preparation, execution, monitoring, recovery, and cutover.
@@ -188,17 +208,16 @@ Starts a full `pgcopydb clone --follow` migration. Creates a new timestamped dir
 ~/run-migration.sh
 ```
 
-**Default configuration (edit the script to adjust):**
-- `TABLE_JOBS=16` — parallel COPY workers
-- `INDEX_JOBS=12` — parallel index creation workers
-- `--split-tables-larger-than 50GB` — splits large tables into parts
-- `--split-max-parts` matches TABLE_JOBS
-- `--plugin wal2json` — logical decoding plugin for CDC
-- `--filter ~/filters.ini`
+**Default configuration (all values come from `~/.env`):**
+- `TABLE_JOBS=8` — parallel COPY workers, also used for `--split-max-parts`
+- `INDEX_JOBS=6` — parallel index creation workers
+- `SPLIT_TABLES_LARGER_THAN=50GB` — splits large tables into parts
+- `OUTPUT_PLUGIN=pgoutput` — logical decoding plugin for CDC
+- `FILTER_FILE=~/filters.ini`
 
 **When to use:** Starting a fresh migration. For a COPY-only test (no CDC), remove the `--follow` and `--plugin` flags.
 
-**Requires:** `PGCOPYDB_SOURCE_PGURI`, `PGCOPYDB_TARGET_PGURI`, `~/filters.ini`
+**Requires:** `PGCOPYDB_SOURCE_PGURI`, `PGCOPYDB_TARGET_PGURI`, the file named by `FILTER_FILE`
 
 ---
 
@@ -350,7 +369,7 @@ Resumes a previously interrupted `pgcopydb clone --follow` migration. Backs up t
 MIGRATION_DIR=~/migration_YYYYMMDD-HHMMSS ~/resume-migration.sh          # specify explicitly
 ```
 
-**Important:** The script passes `--split-tables-larger-than` to match `run-migration.sh`. pgcopydb requires catalog consistency — if the original run used split tables, the resume must pass the same value.
+**Important:** The script reads `SPLIT_TABLES_LARGER_THAN` and `OUTPUT_PLUGIN` from the same `~/.env` as `run-migration.sh`. pgcopydb requires catalog consistency — do not change either value between the original run and the resume.
 
 **When to use:** After pgcopydb crashes, the instance reboots, or the migration is interrupted. To start completely over instead, run `~/target-clean.sh` + `~/drop-replication-slots.sh` first, then `~/start-migration-screen.sh`.
 
@@ -404,6 +423,7 @@ Cleans up pgcopydb replication artifacts on both source and target databases.
 
 **What it cleans:**
 - **Source:** Drops the logical replication slot (terminates active consumer if needed)
+- **Source:** Drops the publication of the same name. pgcopydb creates it only for the `pgoutput` plugin. The drop needs ownership of the publication; the script warns and continues if it fails
 - **Target:** Drops the replication origin and the `pgcopydb` sentinel schema
 
 **When to use:** After a migration completes or is abandoned. Replication slots that are not consumed will cause WAL to accumulate on the source until the disk fills up. Always clean up slots when done.
@@ -549,14 +569,26 @@ IF SOMETHING GOES WRONG:
 
 ## Configuration
 
-All scripts use variables at the top that can be adjusted per migration. See [Cluster configuration parameters](https://planetscale.com/docs/postgres/cluster-configuration/parameters) for understanding target-side capacity when tuning these values:
+Every tunable is set once in `~/.env` and picked up by every script that uses it. No script hardcodes these values. See [Cluster configuration parameters](https://planetscale.com/docs/postgres/cluster-configuration/parameters) for understanding target-side capacity when tuning them:
+
+| Variable | Default | pgcopydb option | Used in |
+|----------|---------|-----------------|---------|
+| `PGCOPYDB_SOURCE_PGURI` | none (required) | `--source` | all scripts |
+| `PGCOPYDB_TARGET_PGURI` | none (required) | `--target` | all scripts |
+| `TABLE_JOBS` | 8 | `--table-jobs`, `--split-max-parts` | run-migration.sh, resume-migration.sh, resume-cdc.sh |
+| `INDEX_JOBS` | 6 | `--index-jobs` | run-migration.sh, resume-migration.sh |
+| `SPLIT_TABLES_LARGER_THAN` | 50GB | `--split-tables-larger-than` | run-migration.sh, resume-migration.sh, resume-cdc.sh |
+| `OUTPUT_PLUGIN` | pgoutput | `--plugin` | run-migration.sh, resume-migration.sh, resume-cdc.sh |
+| `FILTER_FILE` | `~/filters.ini` | `--filter` | run-migration.sh, resume-migration.sh, resume-cdc.sh, preflight-check.sh, verify-migration.sh |
+| `SLACK_WEBHOOK_URL` | unset | — | slack-migration-alerts.sh |
+
+### `OUTPUT_PLUGIN`
+
+`pgoutput` is the default. It is part of PostgreSQL core, so the source server needs no extension. pgcopydb builds a publication from the table list in `FILTER_FILE` and drops it during `pgcopydb stream cleanup`. Compared to `wal2json` it sends about 4.5x less network volume and uses about 4x less CPU on the source.
+
+`wal2json` and `test_decoding` remain supported. Set `OUTPUT_PLUGIN=wal2json` to use the previous plugin; the source server must have the `wal2json` extension installed.
 
-| Variable | Default | Used in |
-|----------|---------|---------|
-| `TABLE_JOBS` | 16 | run-migration.sh, resume-migration.sh |
-| `INDEX_JOBS` | 12 | run-migration.sh, resume-migration.sh |
-| `FILTER_FILE` | ~/filters.ini | run-migration.sh, resume-migration.sh, resume-cdc.sh |
-| `--split-tables-larger-than` | 50GB | run-migration.sh, resume-migration.sh |
+`OUTPUT_PLUGIN` and `SPLIT_TABLES_LARGER_THAN` must not change between a run and its resume. pgcopydb requires catalog consistency.
 
 ## Critical Warnings
 

pgcopydb-helpers/README.md

@@ -113,12 +113,37 @@ SHOW wal_level;  -- should return 'logical'
 
 1. **Deploy these scripts** to the migration instance home directory (`~/`).
 
-2. **Create `~/.env`** with your connection strings:
+2. **Create `~/.env` from `~/env-template`** and edit the values:
+
+   ```bash
+   cp ~/env-template ~/.env
+   chmod 600 ~/.env
+   ```
+
+   `env-template` holds every setting the scripts read. Only the two connection
+   strings are required. The remaining settings have working defaults:
 
    ```bash
    export PGCOPYDB_SOURCE_PGURI='postgresql://user:pass@source-host:5432/dbname'
    export PGCOPYDB_TARGET_PGURI='postgresql://user:pass@target-host:5432/dbname'
-   export SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...'  # optional, for Slack alerts
+
+   export TABLE_JOBS=8                       # parallel COPY workers
+   export INDEX_JOBS=6                       # parallel index build workers
+   export SPLIT_TABLES_LARGER_THAN=50GB      # copy larger tables in parts
+   export OUTPUT_PLUGIN=pgoutput             # logical decoding plugin for CDC
+   export FILTER_FILE=~/filters.ini          # pgcopydb filter file
+
+   #export SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...'  # optional
+   ```
+
+   See [Script Configuration](#script-configuration) for how to tune each value.
+
+   The provisioning templates do not create `~/.env` for you. Every script stops
+   with this message until you create it:
+
+   ```
+   ERROR: ~/.env not found. Create it from the template:
+     cp ~/env-template ~/.env && chmod 600 ~/.env
    ```
 
 3. **Customize `~/filters.ini`** to exclude schemas, tables, and extensions that should not be migrated. See [Filter Configuration](#filter-configuration) below.
@@ -233,7 +258,7 @@ After the migration is complete (or abandoned), clean up replication artifacts:
 ~/drop-replication-slots.sh my_slot      # custom slot name
 ```
 
-This drops the replication slot on the source, the replication origin on the target, and the pgcopydb sentinel schema. **Always do this** — unconsumed replication slots cause WAL to accumulate on the source until the disk fills up.
+This drops the replication slot on the source, the replication origin on the target, and the pgcopydb sentinel schema. It also drops the publication that pgcopydb creates on the source for the `pgoutput` plugin, which carries the same name as the slot. **Always do this** — unconsumed replication slots cause WAL to accumulate on the source until the disk fills up.
 
 ## Recovery
 
@@ -253,7 +278,7 @@ If pgcopydb crashes, the instance reboots, or the migration is interrupted:
 MIGRATION_DIR=~/migration_YYYYMMDD-HHMMSS ~/resume-migration.sh              # or specify explicitly
 ```
 
-This backs up the SQLite catalog before resuming and uses `--not-consistent` to allow resuming from a mid-transaction state. The script passes `--split-tables-larger-than` to match `run-migration.sh` — pgcopydb requires catalog consistency, so the resume must use the same split value as the original run.
+This backs up the SQLite catalog before resuming and uses `--not-consistent` to allow resuming from a mid-transaction state. The script reads `SPLIT_TABLES_LARGER_THAN` and `OUTPUT_PLUGIN` from the same `~/.env` as `run-migration.sh` — pgcopydb requires catalog consistency, so do not change these values between the original run and the resume.
 
 If the initial COPY completed successfully but CDC was interrupted, you can resume only the CDC phase without re-attempting the clone:
 
@@ -274,7 +299,7 @@ To start completely over, wipe the target and clean up replication:
 
 ## Filter Configuration
 
-Every migration needs a `~/filters.ini` file to exclude objects that should not be copied. Use the filter to exclude source-specific schemas, tables, and extensions that are not needed on the target — particularly extensions not [supported by PlanetScale](https://planetscale.com/docs/postgres/extensions). The file uses pgcopydb's [filter syntax](https://github.com/planetscale/pgcopydb/blob/main/docs/ref/pgcopydb_filter.rst):
+Every migration needs a filter file to exclude objects that should not be copied. The scripts read the path from `FILTER_FILE` in `~/.env`, which defaults to `~/filters.ini`. Use the filter to exclude source-specific schemas, tables, and extensions that are not needed on the target — particularly extensions not [supported by PlanetScale](https://planetscale.com/docs/postgres/extensions). The file uses pgcopydb's [filter syntax](https://github.com/planetscale/pgcopydb/blob/main/docs/ref/pgcopydb_filter.rst):
 
 ```ini
 [exclude-schema]
@@ -352,16 +377,34 @@ google_ml_integration
 
 ## Script Configuration
 
-The migration scripts have tunable parameters at the top of each file:
+Every tunable setting lives in `~/.env`. Change a value once and all scripts use it. `env-template` in this directory is the reference copy.
+
+| Variable | Default | pgcopydb option | Description |
+|----------|---------|-----------------|-------------|
+| `PGCOPYDB_SOURCE_PGURI` | none (required) | `--source` | Source connection string |
+| `PGCOPYDB_TARGET_PGURI` | none (required) | `--target` | Target connection string |
+| `TABLE_JOBS` | 8 | `--table-jobs`, `--split-max-parts` | Parallel COPY workers |
+| `INDEX_JOBS` | 6 | `--index-jobs` | Parallel index build workers |
+| `SPLIT_TABLES_LARGER_THAN` | 50GB | `--split-tables-larger-than` | Size above which a table is copied in parts |
+| `OUTPUT_PLUGIN` | pgoutput | `--plugin` | Logical decoding plugin for CDC |
+| `FILTER_FILE` | `~/filters.ini` | `--filter` | pgcopydb filter file |
+| `SLACK_WEBHOOK_URL` | unset | — | Webhook for `slack-migration-alerts.sh` |
+
+Adjust `TABLE_JOBS` and `INDEX_JOBS` based on your instance size and database characteristics. More jobs require more CPU cores and memory. A good baseline for `TABLE_JOBS` is fewer than the vCPU count of whichever is smaller — the SOURCE or TARGET. `INDEX_JOBS` should be fewer than the vCPUs on the TARGET. Exceeding these numbers can overwhelm the SOURCE during the COPY phase or the TARGET during index rebuilding. See [Cluster configuration parameters](https://planetscale.com/docs/postgres/cluster-configuration/parameters) for understanding target-side capacity.
+
+`SPLIT_TABLES_LARGER_THAN` must not change between a run and its resume. pgcopydb requires catalog consistency, so `resume-migration.sh` and `resume-cdc.sh` must use the same value as the original `run-migration.sh`. Set 0 to disable splitting.
+
+### Output plugin
+
+`OUTPUT_PLUGIN` selects the logical decoding plugin that pgcopydb uses for CDC. The default is `pgoutput`:
+
+- `pgoutput` is part of PostgreSQL core, so the source server needs no extension.
+- On a mixed INSERT/UPDATE/DELETE workload it sends about 4.5x less network volume and uses about 4x less CPU on the source than `wal2json`.
+- pgcopydb builds a publication from the table list in `FILTER_FILE`, so the source server does the filtering. `pgcopydb stream cleanup` drops that publication.
 
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `TABLE_JOBS` | 16 | Parallel COPY workers |
-| `INDEX_JOBS` | 12 | Parallel index creation workers |
-| `--split-tables-larger-than` | 50GB | Threshold for splitting large tables into parts |
-| `--split-max-parts` | Same as TABLE_JOBS | Maximum number of parts per split table |
+Set `OUTPUT_PLUGIN=wal2json` to use the previous plugin. `wal2json` and `test_decoding` remain supported, but `wal2json` must be installed on the source server.
 
-Adjust these based on your instance size and database characteristics. More jobs require more CPU cores and memory. A good baseline for `TABLE_JOBS` is fewer than the vCPU count of whichever is smaller — the SOURCE or TARGET. `INDEX_JOBS` should be fewer than the vCPUs on the TARGET. Exceeding these numbers can overwhelm the SOURCE during the COPY phase or the TARGET during index rebuilding. See [Cluster configuration parameters](https://planetscale.com/docs/postgres/cluster-configuration/parameters) for understanding target-side capacity.
+Do not change `OUTPUT_PLUGIN` in the middle of a migration. A resume must use the same plugin as the original run.
 
 ## Troubleshooting
 
@@ -424,6 +467,7 @@ sqlite3 ~/migration_*/schema/filter.db "SELECT COUNT(*) FROM s_depend;"
 
 | Script | Phase | Description |
 |--------|-------|-------------|
+| `env-template` | Prepare | Reference `~/.env`. Copy to `~/.env` and edit before you start |
 | `compare-pg-params.sh` | Prepare | Compare PostgreSQL parameters between source and target |
 | `preflight-check.sh` | Prepare | Validate migration prerequisites (connectivity, WAL level, permissions, slots, extension compatibility) |
 | `fix-replica-identity.sh` | Prepare | Set REPLICA IDENTITY FULL on tables without primary keys |
@@ -439,7 +483,7 @@ sqlite3 ~/migration_*/schema/filter.db "SELECT COUNT(*) FROM s_depend;"
 | `resume-migration.sh` | Recovery | Resume an interrupted migration (full clone + CDC) |
 | `resume-cdc.sh` | Recovery | Resume only the CDC phase (skips clone) |
 | `target-clean.sh` | Recovery | Wipe target database for re-migration (prompts for confirmation) |
-| `drop-replication-slots.sh` | Cleanup | Remove replication slots and origins |
+| `drop-replication-slots.sh` | Cleanup | Remove replication slots, the pgoutput publication, and origins |
 | `stop-cdc.sh` | Cutover | Set CDC endpoint via SQLite to initiate cutover |
 | `verify-migration.sh` | Cutover | Verify schema and data consistency between source and target |
 

pgcopydb-helpers/check-cdc-status.sh

@@ -9,6 +9,11 @@
 set -euo pipefail
 
 # --- Load environment ---
+if [ ! -f ~/.env ]; then
+    echo "ERROR: ~/.env not found. Create it from the template:" >&2
+    echo "  cp ~/env-template ~/.env && chmod 600 ~/.env" >&2
+    exit 1
+fi
 set +u
 set -a
 source ~/.env

pgcopydb-helpers/check-copy-stall.sh

@@ -42,6 +42,11 @@ while [ $# -gt 0 ]; do
 done
 
 # --- Load environment ---
+if [ ! -f ~/.env ]; then
+    echo "ERROR: ~/.env not found. Create it from the template:" >&2
+    echo "  cp ~/env-template ~/.env && chmod 600 ~/.env" >&2
+    exit 1
+fi
 set +u
 set -a
 # shellcheck disable=SC1090

pgcopydb-helpers/check-migration-status.sh

@@ -21,6 +21,11 @@ echo -e "${BLUE}╚════════════════════
 echo ""
 
 # --- Load environment ---
+if [ ! -f ~/.env ]; then
+    echo "ERROR: ~/.env not found. Create it from the template:" >&2
+    echo "  cp ~/env-template ~/.env && chmod 600 ~/.env" >&2
+    exit 1
+fi
 set +u
 set -a
 source ~/.env

pgcopydb-helpers/compare-pg-params.sh

@@ -11,6 +11,11 @@
 set -euo pipefail
 
 # --- Load environment ---
+if [ ! -f ~/.env ]; then
+    echo "ERROR: ~/.env not found. Create it from the template:" >&2
+    echo "  cp ~/env-template ~/.env && chmod 600 ~/.env" >&2
+    exit 1
+fi
 set +u
 set -a
 source ~/.env