gsoc26: Declarative Workflow Engine (chaining download, deploy, delete)

[DhanashreePetare]

Pull Request

Description

Adds a declarative workflow engine: databusclient workflow run pipeline.yml runs a sequence of download/deploy/delete steps defined in YAML, with output chaining between steps (${steps.name.output_files} / ${steps.name.output_urls}) and per-step error handling (fail/continue/retry).

Related Issues
Issue #79

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• This change requires a documentation update
• Housekeeping

Checklist:

• My code follows the ruff code style of this project.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (if applicable)
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
  • poetry run pytest - all tests passed
  • poetry run ruff check - no linting errors

One open design question raised with mentor separately: deploying a file that was downloaded and converted (--format/--compression) during a workflow step has no clean path yet, since the converted file only exists locally and deploy needs a fetchable URL. Deploying the original, unconverted download URL (via the new output_urls step output) works today and is tested.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e4bb340c-666a-4b2d-b1c4-11846b0ce830

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Integer-Ctrl]

All workflows working (with own credentials)

[Integer-Ctrl]

each runnable directly

Not really since it uses your account :D

But okay. Maybe re-phrasing in the clean-up phase after this PR