Skip to content

RDSC-5780: Document multiple pipelines in a Redis Cloud RDI workspace - #4326

Open
zhivko-kirishev-redis wants to merge 5 commits into
mainfrom
RDSC-5780-cloud-multi-pipeline-docs
Open

zhivko-kirishev-redis wants to merge 5 commits into
mainfrom
RDSC-5780-cloud-multi-pipeline-docs

Conversation

@zhivko-kirishev-redis

@zhivko-kirishev-redis zhivko-kirishev-redis commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Documents multiple pipelines in one Redis Cloud Data Integration workspace. The on-prem RDI 2.2.0 multi-pipeline docs are in #4270; this Cloud part is a separate PR so that each is easier to review.

Ticket: RDSC-5780

Changes

  • Create workspace (create-workspace.md): new Add pipelines to a workspace section with the steps (Add pipeline at the bottom of the workspace, the Create new pipeline dialog, the new Draft row). It also covers the one-setup-at-a-time rule, what the pipelines of a workspace share (CIDR, PrivateLink connections, Role ARNs, RDI database), and billing. The pipeline table description now covers the Status and Billing columns, and the delete steps mention deleting the workspace together with its last pipeline.
  • Create data pipeline (define.md): a Name the pipeline step before Settings, with the naming rules. The quick start and the source setup steps link to it.
  • View and edit and FAQ: actions apply to one pipeline, except flushing a target database that several pipelines share. View metrics endpoints gives the metrics paths of a named pipeline and says to add a pipeline label to the scrape job of each pipeline. New questions on choosing between a source and a pipeline, and on pipeline isolation. The billing FAQ says the workspace RPU is charged once per workspace and gives a two-pipeline example.
  • Overview, networking, and scaling: the pipelines of a workspace share its network range, VPC endpoints, and RDI database. This fixes the statement that each pipeline has its own RDI database. It also says that RDI spreads pipelines across shards approximately evenly, instead of promising one pipeline per shard.
  • CIDR allow list and maintenance windows: the target allow list note refers to the workspace CIDR that all pipelines share, and the maintenance note speaks of pipelines in the plural.
  • Screenshots: three images cropped from QA console captures in the style of the existing rdi-2-* images. They show the dialog for the first and second pipeline and a workspace with a running and a draft pipeline. They replace rdi-2-workspace-sources.png, whose Streaming status the console now shows as Running. The Workspace actions screenshot stays as it is on main, because Add pipeline is no longer in that menu.

Before merging

  • Multiple pipelines are enabled in the production console. The console gates them behind the enableRdiMultiplePipelines LaunchDarkly flag. These pages are not versioned, so they go live on merge.
  • Pipeline name length: the docs give 3 to 20 characters, the limit that RDSC-6270 (in review) sets. Until it merges, RDI core, cloud-ui, and cloud-rdi-infra allow 22 characters. The console also accepts 1 and 2 character names, which RDI rejects later. If RDSC-6270 doesn't reach production first, the docs need 22 again.
  • Dialog screenshots: rdi-2-create-pipeline.png and rdi-2-create-second-pipeline.png show the 22-character hint. Replace them once the console shows the RDSC-6270 limit. TODO comments mark them on the pages.
  • Target database per pipeline: a separate target database for each pipeline is the intended behavior, and the docs describe it. The workspace table and the database Data Integration tab still show a single workspace-level target database. That is a console bug, which is being fixed.
  • Add pipeline location: RDSC-6273 (merged in cloud-ui) moved it from the Workspace actions menu to a row at the bottom of each workspace card, and the steps now describe that. A screenshot of the new row is still needed (TODO in create-workspace.md). The empty-workspace image rdi-workspace-add-pipeline.png, used on four pages, may also be outdated.
  • Metrics URLs of named pipelines: for a named pipeline, Connect to Prometheus shows the processor URL of the default pipeline and no collector URLs (RDSC-6220, to do). The collector metrics paths of named pipelines aren't routed yet either (fixed by RDSC-6270). View metrics endpoints describes the intended paths.
  • Delete workspace is not blocked by existing pipelines. The docs keep the existing warning to delete the pipelines first.

Validation

  • Built the site locally with Hugo: no warnings for the Cloud RDI pages.
  • Checked that every link and anchor on the changed pages resolves, and that every referenced image exists. The only report was the pre-existing, client-rendered API reference anchor on the scaling page.
  • Checked the facts against cloud-ui, rdi-ui, cloud-rdi-service, and redis-data-integration code. Not checked against the production console.

🤖 Generated with Claude Code


Note

Low Risk
Documentation-only updates to Redis Cloud RDI guides; no application or infrastructure code changes.

Overview
Documents running multiple independent Data Integration pipelines in one Redis Cloud RDI workspace, including console flows, shared vs per-pipeline resources, and updated screenshots.

The create workspace guide gains Add pipelines to a workspace (naming rules, Workspace actions > Add pipeline, one draft at a time, shared CIDR/PrivateLink/Role ARNs/RDI database, per-pipeline billing). The workspace list now describes Status, Billing, and Running instead of older Streaming wording.

Create data pipeline, quick start, and setup add a Name the pipeline step before Settings. FAQ adds guidance on source vs pipeline, isolation (except shared-target flush), and RPU math when several pipelines are deployed. Scaling corrects that the RDI database is one per workspace (not per pipeline) and softens shard-per-pipeline guarantees. View/edit and related pages clarify per-pipeline actions, cross-pipeline flush impact, and optional workspace deletion with the last pipeline.

Reviewed by Cursor Bugbot for commit d4a385e. Bugbot is set up for automated code reviews on this repo. Configure here.

zhivko-kirishev-redis and others added 2 commits October 22, 2026 11:37
A Data Integration workspace can now run several pipelines. Document it
where Cloud users meet it:

- Create workspace: a new "Add pipelines to a workspace" section with
  the steps (Workspace actions > Add pipeline, the Create new pipeline
  dialog, the new Draft row), the one-setup-at-a-time rule, what the
  pipelines of a workspace share, and billing. The pipeline table
  description now covers the Status and Billing columns.
- Create data pipeline: a "Name the pipeline" step before Settings, with
  the naming rules; the quick start and the source setup steps link it.
- View and edit, FAQ: actions apply to one pipeline, except flushing a
  shared target database; deleting the last pipeline can delete the
  workspace; when to add a source or a pipeline; the workspace RPU is
  charged once, with a two-pipeline example.
- Overview, networking, scaling: pipelines share the workspace network
  range, VPC endpoints, and RDI database. Fix the claim that each
  pipeline has its own RDI database, and say that RDI spreads pipelines
  across shards approximately evenly rather than one per shard.

The three new screenshots are cropped from QA console captures in the
style of the existing rdi-2 images. HTML TODO comments mark where the
screenshots of a second pipeline go.

Constraint: the console gates multiple pipelines behind the enableRdiMultiplePipelines LaunchDarkly flag; publish only once it is on in production
Constraint: pipeline names follow RDI core (3 to 22 characters); the console accepts shorter names that RDI rejects later
Gaps: per-pipeline target databases are supported by the backend, but the console lists pipelines under one workspace-level target database; confirm with the product team
Gaps: cloud-ui branch RDSC-6273-add-pipeline-row (unmerged) moves Add pipeline out of the Workspace actions menu
Ticket: RDSC-5780
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the TODO placeholders of the Cloud workspace page with crops of
the new QA console captures, and use one image per UI element across
the page:

- rdi-2-workspace-pipelines.png: a workspace with a Running pipeline and
  a Draft second pipeline. It shows the pipeline table in both the
  workspace status section and the Add pipelines section, and replaces
  rdi-2-workspace-sources.png, whose Streaming status the console now
  shows as Running, and the single Draft row crop.
- rdi-2-workspace-actions.png: the Workspace actions menu of an active
  workspace, with Add pipeline and an enabled Delete workspace. It now
  serves all three menu references, replacing the menu without Add
  pipeline and the crop with a disabled Delete workspace.
- rdi-2-create-second-pipeline.png: the Create new pipeline dialog with
  the name of the second pipeline, used in the Add pipelines steps. The
  first-pipeline dialog stays in the quick start and Create data
  pipeline.

The new captures also confirm that Add pipeline is still in the
Workspace actions menu, so the step text stays.

Ticket: RDSC-5780
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Redis Memory

Found 7 related items from repository history:

Memory updated at 505785b

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Comment thread content/operate/rc/rdi/create-workspace.md
Comment thread content/operate/rc/rdi/create-workspace.md Outdated
Comment thread content/operate/rc/rdi/faq.md
@zhivko-kirishev-redis
zhivko-kirishev-redis marked this pull request as ready for review October 22, 2026 15:16
- Stop promising a separate target database per pipeline in the
  overview, the workspace page, the FAQ, and the quick start. The
  backend and the pipeline editor take a target per pipeline, but the
  workspace table and the database Data Integration tab use one
  workspace-level target, and the product team has not settled which
  behavior to document. The PR description tracks the question.
- Show the workspace pipelines image and the Workspace actions menu
  once each on the workspace page, and point back to them from the
  Add pipelines and Delete workspace sections.
- Say that the workspace charge continues while any pipeline of the
  workspace remains deployed.

Ticket: RDSC-5780
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert the target database part of the previous commit. A separate
target database per pipeline is the intended behavior. The console
showing one workspace-level target in the workspace table and the
database Data Integration tab is a bug that is being fixed. The overview,
the workspace page, the FAQ, and the quick start again say that each
pipeline has its own target and that separate pipelines can write to
different target databases.

The other review fixes of the previous commit stay: each image appears
once on the workspace page, and the billing FAQ covers several deployed
pipelines.

Constraint: each pipeline of a workspace has its own target database by design; do not drop this again because of the workspace-level Target name column
Ticket: RDSC-5780
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@nicoweidner nicoweidner left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few spots that should be changed (some because of in-flight PRs), and there are also two more spots that are formulated for a single pipeline:

  1. security/cidr-whitelist.md:53: "the RDI pipeline's CIDR" is now the workspace CIDR, which all pipelines share. The target allow list entry covers every pipeline.
  2. Low: _index.md:128-130 and subscriptions/maintenance/set-maintenance-windows.md:18 still say "your RDI data pipeline" in the singular.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're about to change the max length to 20 characters, so the pictures will need an update unfortunately 😬

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The console still shows the 22-character hint until RDSC-6270 lands, so these screenshots can't be retaken yet. 505785b marks each dialog screenshot with a TODO for replacement, and the PR description lists it under Before merging. Leaving this thread open until the new screenshots are in.

🤖 Addressed by Claude Code

Comment thread content/operate/rc/rdi/create-workspace.md Outdated
Comment thread content/operate/rc/rdi/define.md Outdated
Comment thread content/operate/rc/rdi/create-workspace.md Outdated
Comment thread content/operate/rc/rdi/view-edit.md
- Add pipeline: RDSC-6273 (cloud-ui #3314) moved it from the Workspace
  actions menu to a row at the bottom of each workspace card. Update the
  steps on the workspace page and in Create data pipeline. Restore the
  original menu screenshot from main, which shows only Workspace details
  and Delete workspace again. Mark the missing screenshot of the new row
  with a TODO.
- Pipeline names: RDSC-6270 lowers the maximum length to 20 characters,
  so the rules now say 3 to 20. The dialog screenshots still show the
  22 character hint and are marked with TODO comments for replacement.
- View metrics endpoints: the paths of a named pipeline start with its
  name. All pipelines report the same metric names and the endpoints
  add no pipeline label, so a pipeline label goes in each scrape job.
- Single-pipeline wording: the target allow list note in
  cidr-whitelist.md now refers to the workspace CIDR that all pipelines
  share, and the maintenance notes speak of pipelines in the plural.

Gaps: RDSC-6270 is not merged yet (RDI, cloud-ui, and cloud-rdi-infra still allow 22 characters), and RDSC-6220 still returns the default pipeline metrics URLs for named pipelines; both are in the PR description
Ticket: RDSC-5780
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants