Repository navigation
fix(collections): Ensure safe collection creation - #1312
Conversation
… flight (#1311) Retried POST /collections requests each created a new job and vector store because the collection row is only inserted when the worker finishes. Store the requested name on collection_jobs with a partial unique index over in-flight CREATE jobs. A duplicate request now returns the existing job without enqueuing again, and a failed enqueue marks the job FAILED so the name is not held.
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one): 1
Please check the settings in the CodeRabbit UI or the
 |
github-actions
Bot
changed the title
OpenAPI changes ⚪ No API surface changesNote This PR does not modify the API contract.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
@vprashrex can you please codecov report. |
Add crud tests for create_or_get_in_flight (new, duplicate, insert race, re-raise, name reuse after finish, unnamed) and route tests for duplicate in-flight name, existing-collection 409 and enqueue failure.
| logger.info( | ||
| "[CollectionJobCrud.create_or_get_in_flight] Duplicate in-flight create | " | ||
| f"{{'project_id': '{self.project_id}', 'job_id': '{existing.id}'}}" | ||
| ) |
| logger.info( | ||
| "[CollectionJobCrud.create_or_get_in_flight] Collection job created | " | ||
| f"{{'collection_job_id': '{job.id}'}}" | ||
| ) |
|
🎉 This PR is included in version 1.9.0-main.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Issue
Closes #1311
Summary
POST /api/v1/collections, causing each retry to create a new job and vector store due to the uniquenameindex not being available until the worker finishes.collection_jobs.namefor jobs inPENDING/PROCESSING.FAILEDif enqueue fails, preventing unrun jobs from holding names.Checklist
Before submitting a pull request, please ensure that you mark these task.
fastapi run --reload app/main.pyordocker compose upin the repository root and test.Notes
A job stuck in
PROCESSING(worker lost) keeps its name reserved; there is no recovery for that today, only the stale-PENDINGalert. Existing duplicate vector stores from earlier retries are not cleaned up here.Original PR description
Issue
Closes #1311
Summary
Glific's client times out on
POST /api/v1/collectionsand retries. Each retry created a new job and vector store, because thecollectionrow (and its uniquenameindex) only exists once the worker finishes.collection_jobs.name(nullable) plus a partial unique indexuq_collection_jobs_project_id_name_in_flighton(project_id, name)for CREATE jobs inPENDING/PROCESSING. Migration087.CollectionJobCrud.create_or_get_in_flightreturns the existing job on a duplicate, and the route only enqueues when a new row was created.FAILED, so a job that never ran does not hold the name.create.mdupdated.Contract: request/response schemas, status codes and the 409 for an already-created collection are unchanged. A retry now gets the same
job_id(itsstatuscan bePROCESSING).callback_url/documentson a duplicate request are ignored. Requests withoutnameare not covered.Checklist
FAILED. Migration upgrade/downgrade verified.Notes
A job stuck in
PROCESSING(worker lost) keeps its name reserved; there is no recovery for that today, only the stale-PENDINGalert. Existing duplicate vector stores from earlier retries are not cleaned up here.