Skip to content

HDDS-16712. Retry key allocation when an allocated pipeline disappears - #11752

Open
adwetojha wants to merge 1 commit into
apache:masterfrom
adwetojha:codex/HDDS-16712-client-retry
Open

adwetojha wants to merge 1 commit into
apache:masterfrom
adwetojha:codex/HDDS-16712-client-retry

Conversation

@adwetojha

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

During Datanode restart and re-registration, SCM may wait for an ALLOCATED pipeline that is removed before becoming OPEN. If no replacement container is available, block allocation fails.

Previously, WritableRatisContainerProvider wrapped this specific failure in a generic IOException. OM consequently returned INTERNAL_ERROR, which was not retried by the Ozone client, causing intermittent key creation failures.

This patch adds a narrowly scoped client-side retry for this transient allocation failure:

  • Preserves the missing-pipeline failure as SCMException.ResultCodes.PIPELINE_NOT_FOUND after the provider's final container lookup.
  • Maps this SCM result to the new OM status KEY_ALLOCATION_PIPELINE_NOT_FOUND.
  • Extends the existing bounded retry helper for CreateKey, CreateFile, and AllocateBlock, while preserving SCM safe-mode retries.
  • Does not retry unrelated INTERNAL_ERROR or KEY_ALLOCATION_ERROR responses.
  • Adds a client version check to return INTERNAL_ERROR to older clients that do not recognize the new status.
  • Adds regression tests covering SCM error propagation, OM error mapping, client retries, retry exhaustion, and backward compatibility.

The CreateKey regression test also verifies that this failure occurs during pre-execution, before OM Ratis submission, without creating successful open-key state.

The retry reuses the existing limit of 90 retries after the initial attempt (up to 91 submissions), with a one-second delay between attempts.

What is the link to the Apache JIRA?

https://issues.apache.org/jira/browse/HDDS-16712

How was this patch tested?

The following checks passed in local JDK 21 runs:

  • 264 tests passed: SCM provider, SCM RPC translator, client protocol translator, OM block allocation, CreateKey, client versioning, and result-code tests.
  • 225 tests passed: TestRpcClient, TestOMFileCreateRequest, and TestOMFileCreateRequestWithFSO.
  • 19 client protocol translator tests passed in a final rerun after the logging change. These tests were already included in the 264-test suite above.
  • Protobuf backward-compatibility check passed.
  • Checkstyle passed with zero violations.
  • git diff --check passed.

Live Kubernetes Datanode restart and rolling-upgrade tests have not been performed.

Generated-by: OpenAI Codex (GPT-6 Sol High)

Co-authored-by: OpenAI Codex (GPT-6 Sol High)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant