[Fix] Keep uploading when a write races the write checkpoint

[bean1352]

What's wrong

When the CRUD queue looks empty, the upload loop requests a write checkpoint. updateLocalTarget re-reads the queue inside a write transaction and returns false if it finds a row, but it also returns false when there is nothing left to upload. The loop treated both the same way and stopped with the row still in ps_crud, even though updateLocalTarget had already seen it and logged "New data uploaded since write checkpoint N - need new write checkpoint".

The sync core then refuses every checkpoint with "Could not apply checkpoint due to local data".

The fix

Ask the adapter whether CRUD is still pending before stopping, and go round again if it is. This covers both branches where updateLocalTarget reports a race, including sequence updated.

The retry is delayed by crudUploadThrottleMs.

Tests

One test makes the queue read miss the row once with CRUD notifications left working, which reproduces the sequence in the logs from the original report. The other keeps that read missing and checks the retry stays near the throttle rate. Each one fails without the part of the fix it covers.

Still open

Why nextCrudItem() misses a committed row on React Native. It reads through this.db.getOptional while updateLocalTarget checks inside writeTransaction, so a stale read snapshot in the connection pool is the obvious candidate.

AI disclosure

I used Opus 5 to help research the original bug report, narrow down where the fault could be, and apply the fix. I reviewed and supervised the work.

[changeset-bot]

🦋 Changeset detected

Latest commit: fd15846

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 7 packages

Name	Type
@powersync/shared-internals	Patch
@powersync/adapter-sql-js	Patch
@powersync/capacitor	Patch
@powersync/node	Patch
@powersync/react-native	Patch
@powersync/web	Patch
@powersync/diagnostics-app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[simolus3]

The loop can't tell those apart, so it breaks out while the row is still in ps_crud.

Shouldn't the new write also have triggered a crud trigger? If we change crudUploadLoop to acquire the wait notification promise before the rest of the upload, does that work?

  private async crudUploadLoop(signal: AbortSignal, options: ResolvedSyncOptions): Promise<void> {
    while (!signal.aborted) {
      const crudChangeNotification = this.crudUploadNotifier.waitForNotification(signal);
      await Promise.all([
        // Start the initial CRUD upload on connect. Then, keep polling until we're done.
        this._uploadAllCrud(signal, options),
        this.delayRetry(signal, options.crudUploadThrottleMs)
      ]);

      await crudChangeNotification;
    }
  }

[bean1352]

The loop can't tell those apart, so it breaks out while the row is still in ps_crud.

Shouldn't the new write also have triggered a crud trigger? If we change crudUploadLoop to acquire the wait notification promise before the rest of the upload, does that work?

  private async crudUploadLoop(signal: AbortSignal, options: ResolvedSyncOptions): Promise<void> {
    while (!signal.aborted) {
      const crudChangeNotification = this.crudUploadNotifier.waitForNotification(signal);
      await Promise.all([
        // Start the initial CRUD upload on connect. Then, keep polling until we're done.
        this._uploadAllCrud(signal, options),
        this.delayRetry(signal, options.crudUploadThrottleMs)
      ]);

      await crudChangeNotification;
    }
  }

As far as I can tell the write does trigger the notification (my mistake), so registering the wait earlier should not change anything. I tried your version against the new test and the upload still stops.

[simolus3]

As far as I can tell the write does trigger the notification (my mistake), so registering the wait earlier should not change anything.

Well if it triggered a notification, wouldn't there be a second iteration in crudUploadLoop that calls _uploadAllCrud again, even if the first call stopped prematurely?

I see how this is a race condition that is hard to test, but I don't fully understand how this is happening yet, I assume it is this?

1. In _uploadAllCrud(), nextCrudItem() eventually returns null.
2. We call updateLocalTarget(), which starts by reading the sequence.
3. Likely while we're in the callback to request a write checkpoint from the service, there's a concurrent local write to the database creating a new crud item.
4. updateLocalTarget detects this and returns false, logging that a new write checkpoint is necessary.
5. We emit an Upload complete, no write checkpoint needed. message (that's definitely wrong) and return from _uploadAllCrud.
6. And now, somehow crudUploadLoop doesn't catch the write from step 3 to invoke _uploadAllCrud again?

I think we should still figure out why step 6 behaves the way it does, the synthetic tests mocking the bucket storage don't look that helpful. We could add a test delaying the /write-checkpoint2.json request to specifically trigger a local write in that state, but I suspect it would get picked up by another iteration like it's supposed to.

[simolus3]

Can we make uploadLocalTarget return this information directly? Instead of just returning a boolean, maybe it could return a 'no_crud_sequence' | 'new_data' | 'updated_checkpoint' string where we would enter this branch on new_data? That avoids another database call since updateLocalTarget probes for the same thing.

[bean1352]

Yes, thats a better idea.

uploadLocalTarget returns the following type now:

export type UpdateLocalTargetResult = 'updated_checkpoint' | 'new_data' | 'sequence_changed' | 'no_crud_sequence';

I see now how the tests weren't very helpful. I still can't explain step 6 either. It's probably either the notification not firing for that write, or nextCrudItem() missing a row that already committed.