Repository navigation
RDSC-5780: Document multiple pipelines in a Redis Cloud RDI workspace - #4326
zhivko-kirishev-redis wants to merge 5 commits into
Conversation
A Data Integration workspace can now run several pipelines. Document it where Cloud users meet it: - Create workspace: a new "Add pipelines to a workspace" section with the steps (Workspace actions > Add pipeline, the Create new pipeline dialog, the new Draft row), the one-setup-at-a-time rule, what the pipelines of a workspace share, and billing. The pipeline table description now covers the Status and Billing columns. - Create data pipeline: a "Name the pipeline" step before Settings, with the naming rules; the quick start and the source setup steps link it. - View and edit, FAQ: actions apply to one pipeline, except flushing a shared target database; deleting the last pipeline can delete the workspace; when to add a source or a pipeline; the workspace RPU is charged once, with a two-pipeline example. - Overview, networking, scaling: pipelines share the workspace network range, VPC endpoints, and RDI database. Fix the claim that each pipeline has its own RDI database, and say that RDI spreads pipelines across shards approximately evenly rather than one per shard. The three new screenshots are cropped from QA console captures in the style of the existing rdi-2 images. HTML TODO comments mark where the screenshots of a second pipeline go. Constraint: the console gates multiple pipelines behind the enableRdiMultiplePipelines LaunchDarkly flag; publish only once it is on in production Constraint: pipeline names follow RDI core (3 to 22 characters); the console accepts shorter names that RDI rejects later Gaps: per-pipeline target databases are supported by the backend, but the console lists pipelines under one workspace-level target database; confirm with the product team Gaps: cloud-ui branch RDSC-6273-add-pipeline-row (unmerged) moves Add pipeline out of the Workspace actions menu Ticket: RDSC-5780 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the TODO placeholders of the Cloud workspace page with crops of the new QA console captures, and use one image per UI element across the page: - rdi-2-workspace-pipelines.png: a workspace with a Running pipeline and a Draft second pipeline. It shows the pipeline table in both the workspace status section and the Add pipelines section, and replaces rdi-2-workspace-sources.png, whose Streaming status the console now shows as Running, and the single Draft row crop. - rdi-2-workspace-actions.png: the Workspace actions menu of an active workspace, with Add pipeline and an enabled Delete workspace. It now serves all three menu references, replacing the menu without Add pipeline and the crop with a disabled Delete workspace. - rdi-2-create-second-pipeline.png: the Create new pipeline dialog with the name of the second pipeline, used in the Add pipelines steps. The first-pipeline dialog stays in the quick start and Create data pipeline. The new captures also confirm that Add pipeline is still in the Workspace actions menu, so the step text stays. Ticket: RDSC-5780 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🧠 Redis MemoryFound 7 related items from repository history:
Memory updated at 505785b |
|
|
- Stop promising a separate target database per pipeline in the overview, the workspace page, the FAQ, and the quick start. The backend and the pipeline editor take a target per pipeline, but the workspace table and the database Data Integration tab use one workspace-level target, and the product team has not settled which behavior to document. The PR description tracks the question. - Show the workspace pipelines image and the Workspace actions menu once each on the workspace page, and point back to them from the Add pipelines and Delete workspace sections. - Say that the workspace charge continues while any pipeline of the workspace remains deployed. Ticket: RDSC-5780 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert the target database part of the previous commit. A separate target database per pipeline is the intended behavior. The console showing one workspace-level target in the workspace table and the database Data Integration tab is a bug that is being fixed. The overview, the workspace page, the FAQ, and the quick start again say that each pipeline has its own target and that separate pipelines can write to different target databases. The other review fixes of the previous commit stay: each image appears once on the workspace page, and the billing FAQ covers several deployed pipelines. Constraint: each pipeline of a workspace has its own target database by design; do not drop this again because of the workspace-level Target name column Ticket: RDSC-5780 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nicoweidner
left a comment
There was a problem hiding this comment.
There are a few spots that should be changed (some because of in-flight PRs), and there are also two more spots that are formulated for a single pipeline:
- security/cidr-whitelist.md:53: "the RDI pipeline's CIDR" is now the workspace CIDR, which all pipelines share. The target allow list entry covers every pipeline.
- Low: _index.md:128-130 and subscriptions/maintenance/set-maintenance-windows.md:18 still say "your RDI data pipeline" in the singular.
There was a problem hiding this comment.
We're about to change the max length to 20 characters, so the pictures will need an update unfortunately 😬
There was a problem hiding this comment.
The console still shows the 22-character hint until RDSC-6270 lands, so these screenshots can't be retaken yet. 505785b marks each dialog screenshot with a TODO for replacement, and the PR description lists it under Before merging. Leaving this thread open until the new screenshots are in.
🤖 Addressed by Claude Code
- Add pipeline: RDSC-6273 (cloud-ui #3314) moved it from the Workspace actions menu to a row at the bottom of each workspace card. Update the steps on the workspace page and in Create data pipeline. Restore the original menu screenshot from main, which shows only Workspace details and Delete workspace again. Mark the missing screenshot of the new row with a TODO. - Pipeline names: RDSC-6270 lowers the maximum length to 20 characters, so the rules now say 3 to 20. The dialog screenshots still show the 22 character hint and are marked with TODO comments for replacement. - View metrics endpoints: the paths of a named pipeline start with its name. All pipelines report the same metric names and the endpoints add no pipeline label, so a pipeline label goes in each scrape job. - Single-pipeline wording: the target allow list note in cidr-whitelist.md now refers to the workspace CIDR that all pipelines share, and the maintenance notes speak of pipelines in the plural. Gaps: RDSC-6270 is not merged yet (RDI, cloud-ui, and cloud-rdi-infra still allow 22 characters), and RDSC-6220 still returns the default pipeline metrics URLs for named pipelines; both are in the PR description Ticket: RDSC-5780 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Documents multiple pipelines in one Redis Cloud Data Integration workspace. The on-prem RDI 2.2.0 multi-pipeline docs are in #4270; this Cloud part is a separate PR so that each is easier to review.
Ticket: RDSC-5780
Changes
create-workspace.md): new Add pipelines to a workspace section with the steps (Add pipeline at the bottom of the workspace, the Create new pipeline dialog, the new Draft row). It also covers the one-setup-at-a-time rule, what the pipelines of a workspace share (CIDR, PrivateLink connections, Role ARNs, RDI database), and billing. The pipeline table description now covers the Status and Billing columns, and the delete steps mention deleting the workspace together with its last pipeline.define.md): a Name the pipeline step before Settings, with the naming rules. The quick start and the source setup steps link to it.pipelinelabel to the scrape job of each pipeline. New questions on choosing between a source and a pipeline, and on pipeline isolation. The billing FAQ says the workspace RPU is charged once per workspace and gives a two-pipeline example.rdi-2-*images. They show the dialog for the first and second pipeline and a workspace with a running and a draft pipeline. They replacerdi-2-workspace-sources.png, whose Streaming status the console now shows as Running. The Workspace actions screenshot stays as it is on main, because Add pipeline is no longer in that menu.Before merging
enableRdiMultiplePipelinesLaunchDarkly flag. These pages are not versioned, so they go live on merge.rdi-2-create-pipeline.pngandrdi-2-create-second-pipeline.pngshow the 22-character hint. Replace them once the console shows the RDSC-6270 limit. TODO comments mark them on the pages.create-workspace.md). The empty-workspace imagerdi-workspace-add-pipeline.png, used on four pages, may also be outdated.defaultpipeline and no collector URLs (RDSC-6220, to do). The collector metrics paths of named pipelines aren't routed yet either (fixed by RDSC-6270). View metrics endpoints describes the intended paths.Validation
🤖 Generated with Claude Code
Note
Low Risk
Documentation-only updates to Redis Cloud RDI guides; no application or infrastructure code changes.
Overview
Documents running multiple independent Data Integration pipelines in one Redis Cloud RDI workspace, including console flows, shared vs per-pipeline resources, and updated screenshots.
The create workspace guide gains Add pipelines to a workspace (naming rules, Workspace actions > Add pipeline, one draft at a time, shared CIDR/PrivateLink/Role ARNs/RDI database, per-pipeline billing). The workspace list now describes Status, Billing, and Running instead of older Streaming wording.
Create data pipeline, quick start, and setup add a Name the pipeline step before Settings. FAQ adds guidance on source vs pipeline, isolation (except shared-target flush), and RPU math when several pipelines are deployed. Scaling corrects that the RDI database is one per workspace (not per pipeline) and softens shard-per-pipeline guarantees. View/edit and related pages clarify per-pipeline actions, cross-pipeline flush impact, and optional workspace deletion with the last pipeline.
Reviewed by Cursor Bugbot for commit d4a385e. Bugbot is set up for automated code reviews on this repo. Configure here.