Defer persisting metadata of new clients until they come back

CLAUDE.md

@@ -23,7 +23,7 @@ PHP 8.1+ is required (CI runs 8.1 and 8.2). Use the `8.1`/`8.2` shell switchers
 - Dependency analysis: `vendor/bin/composer-dependency-analyser` (no composer script; see `composer-dependency-analyser.php`)
 - Rector (dry-run): `vendor/bin/rector process --dry-run` (config in `rector.php`, `withPhpSets(php81: true)`; not run in CI)
 
-CI (`.github/workflows/build.yaml`) additionally runs `composer validate --strict` and `composer normalize --dry-run` across the PHP 8.1/8.2 × Symfony 6.4/7.0 matrix.
+CI (`.github/workflows/build.yaml`) additionally runs `composer validate --strict` and `composer normalize --dry-run` across the PHP 8.1/8.2 × Symfony 6.4/7.0 matrix. Pull requests also run Roave's BackwardCompatibilityCheck against their base branch (`.github/workflows/backwards-compatibility-check.yaml`).
 
 > Local note (PHP 8.4): the transitive dev dep `thecodingmachine/safe` emits a flood of implicit-nullable `E_DEPRECATED` notices on PHP 8.4 (they don't exist on CI's 8.1/8.2). This is harmless for PHPStan/PHPUnit but breaks Infection's initial test run — run it with `vendor/bin/infection --initial-tests-php-options="-d error_reporting=24575"` locally on 8.4.
 
@@ -35,22 +35,33 @@ The core abstraction is `ClientContextInterface::getClient(): Client`. Controlle
 
 Three concerns are each built as a **Symfony service-decoration chain** in `config/services.xml`. Higher `decoration-priority` is applied first and ends up *innermost*; the public alias resolves to the outermost decorator. Understanding the resolution order is essential before changing any service wiring:
 
-- **Client context** — runtime order outermost→innermost: `CachedClientContext` (prio 32) → `CookieBasedClientContext` (prio 64) → `DefaultClientContext`.
+- **Client context** — runtime order outermost→innermost: `CachedClientContext` (prio 32) → `PendingMetadataClientContext` (prio 48) → `CookieBasedClientContext` (prio 64) → `DefaultClientContext`.
   - `Cached` memoizes the `Client` for the request. It implements `ResetInterface` and is tagged `kernel.reset`, so long-running runtimes (FrankenPHP worker mode, RoadRunner) don't hand one visitor's client to the next request; `tests/Functional/LongRunningProcessTest.php` covers this. Any new service that holds per-request state needs the same, unless it keys the state by the request object like `CachedCookieProvider`.
+  - `PendingMetadata`: if the main request carries pending metadata signed for the client's id and it has keys (not expired) that the client's metadata lacks (`toArray()` initializes the lazy ghost), returns a new `Client` whose `ChangeAwareMetadata` holds the client's metadata plus those keys with their expiry timestamps, marked dirty so the persister writes it. Keys the client has win, because a stored row was written from metadata that already held the pending metadata (e.g. by a concurrent request) or by the app outside the client's requests.
   - `CookieBased` reads the cookie; if present, builds `Client(cookieId, metadataProvider->getMetadata(...))`; otherwise delegates.
   - `Default` returns an anonymous `new Client(null, new ChangeAwareMetadata())`.
 - **Cookie provider** — `CachedCookieProvider` (prio 32, memoizes per `Request` in a `WeakMap` keyed by the request object, so it needs no reset; not `spl_object_hash()`, which PHP hands to a new request once the old one is freed) → `RequestBasedCookieProvider` (parses the cookie off the request). The test kernel keeps its requests alive, so only `CachedCookieProviderTest` can reproduce that handle reuse.
 - **Metadata provider** — `DoctrineOrmBasedMetadataProvider` (prio 64) → `EmptyMetadataProvider` (default). Doctrine loads from DB, falling back to the decorated provider when no row exists.
 
 ### Request lifecycle
 
-- `EventSubscriber/StoreCookieSubscriber` (on `kernel.response`): dispatches `Event/PreStoreCookieEvent` (an app can set `$store = false` to veto), then writes/refreshes the cookie with an updated `lastSeenAt`. The cookie is set `HttpOnly(false)` on purpose so client-side JS can read it.
+- `EventSubscriber/StoreCookieSubscriber` (on `kernel.response`, priority -64 so metadata set by default-priority response listeners is included, and before the profiler at -100): dispatches `Event/PreStoreCookieEvent` (an app can set `$store = false` to veto), then writes/refreshes the cookie with an updated `lastSeenAt`. The cookie is set `HttpOnly(false)` on purpose so client-side JS can read it. It also handles pending metadata (below).
 - `EventSubscriber/StoreMetadataSubscriber` (on `kernel.finish_request`): calls the metadata persister for the current client.
 
 ### Lazy metadata (why writes are cheap)
 
 `Client/ChangeAwareMetadata` extends `setono/client`'s `Metadata` and flips a `dirty` flag on `set`/`remove`. `Client/LazyChangeAwareMetadata` adds Symfony VarExporter's `LazyGhostTrait`, so `DoctrineOrmBasedMetadataProvider` returns a lazy ghost whose DB query only fires on first access. `DoctrineOrmBasedMetadataPersister::persist()` then **short-circuits** in two cases: the metadata is a lazy ghost that was never initialized (never read), or it is not dirty (never mutated). This is what makes "lazy loaded metadata" in the README real — no SELECT and no flush unless the app touched metadata.
 
+### Pending metadata (new clients)
+
+A *new* client is one whose request has no id cookie. Its metadata is not persisted until it comes back, so clients that never return (crawlers, cookie blockers) never produce a row (issue #8):
+
+- `StoreCookieSubscriber`, for a new client: dirty `ChangeAwareMetadata` goes to `PendingMetadataStorage/PendingMetadataStorageInterface::store()` and is then `markClean()`ed, so `StoreMetadataSubscriber` skips it. If `store()` returns `false` (too big / not JSON encodable) it stays dirty and is persisted right away; anything set after the subscriber ran makes it dirty again and is persisted right away too. On veto, a new client's metadata is `markClean()`ed (discarded), because the client never receives its id.
+- `StoreCookieSubscriber`, when the cookie is stored: always calls `remove()`, which clears the pending cookie if the request carried one (a `store()` in the same response replaces the clearing cookie).
+- `PendingMetadataClientContext` adopts the pending metadata on the return request (see above).
+- `CookieBasedPendingMetadataStorage` (default, alias `setono_client.pending_metadata_storage.default`): cookie `<cookie name>_pending_metadata`, value `<json>.<hmac-sha256 hex>` signed with `%kernel.secret%` over a context string + client id + JSON, `HttpOnly`, `Secure` on HTTPS, same expiration as the id cookie, max 2048 bytes URL-encoded. JSON flags mirror Doctrine's json type (`JSON_PRESERVE_ZERO_FRACTION`).
+- `tests/Functional/NewClientMetadataTest.php` shows the whole flow in the test app, whose `GoogleClickIdSubscriber` sets metadata on `kernel.response` before (`gclid`) and after (`late_gclid`) `StoreCookieSubscriber`. Functional tests that check what a request stored call `FunctionalTestCase::becomeReturningVisitor()` first, because a new visitor's metadata only reaches the database when it comes back.
+
 ### Entity mapping
 
 `Entity/Metadata` (table `setono_client__metadata`, string id `clientId`, nullable `json` `metadata`) is mapped in `config/doctrine-mapping/Metadata.orm.xml` as a **mapped-superclass**. `EventListener/Doctrine/ConvertToEntityListener` listens on Doctrine's `loadClassMetadata` and flips `isMappedSuperclass = false` on the bundle's own `Metadata`, so it becomes a concrete entity. Apps can swap in a custom class via the `metadata_class` config option; `SetonoClientExtension::load()` enforces that it implements `Entity/MetadataInterface`, stores its declared name (e.g. without a leading backslash) in the parameter, and the app is responsible for mapping it. When the custom class extends `Entity/Metadata`, the extension removes the listener: the bundle's class stays a mapped superclass and the app's entity inherits its fields (`tests/Functional/CustomMetadataClassTest.php` runs `MetadataTest` that way). A custom class that only implements the interface leaves the bundle's `Metadata` an entity, as before.
@@ -71,4 +82,5 @@ setono_client:
 
 - Strict types everywhere (`declare(strict_types=1)`); concrete classes are `final`, except the metadata/entity classes (`ChangeAwareMetadata`, `LazyChangeAwareMetadata`, `Entity\Metadata`) which are intentionally extensible.
 - New services go in `config/services.xml` using the `setono_client.<concern>.<variant>` id scheme, with a `setono_client.<concern>.default` alias and an interface alias for autowiring. To insert behavior into a chain, add a decorator with an appropriate `decoration-priority` rather than editing existing classes.
+- Public constructors are covered by the BC check, also of `final` classes. Like Symfony, add a new constructor argument at the end as an optional one and `trigger_deprecation('setono/client-bundle', '<next minor>', 'Not passing an instance of "%s" as "$arg" to "%s()" is deprecated, the argument will be required in <next major>.', …)` when it's missing (see `StoreCookieSubscriber`, where a missing pending metadata storage means the metadata of a new client is persisted right away).
 - Tests use `matthiasnoback/symfony-dependency-injection-test` (e.g. `AbstractExtensionTestCase`), `matthiasnoback/symfony-config-test` (`ConfigurationTestCaseTrait`) and Prophecy (`ProphecyTrait`), with the `@test` annotation and snake_case method-name style. Keep the suite at 100% MSI — when adding logic, add a test that kills the corresponding mutant.

README.md

@@ -10,7 +10,9 @@ Recognize returning visitors in your Symfony application and attach your own met
 The bundle assigns every visitor a stable **client id**, stored in a first‑party cookie (`setono_client_id` by
 default) that also remembers when the visitor was *first* and *last* seen. On top of that you can persist arbitrary
 **metadata** per client — for example the first Google click id (`gclid`) a visitor arrived with, an A/B test variant,
-or a referrer. Metadata is **lazy**: if you never read or write it, the bundle never touches the database.
+or a referrer. Metadata is **lazy**: if you never read or write it, the bundle never touches the database. And the
+metadata of a first‑time visitor is only written to the database once the visitor comes back, so crawlers don't fill
+your database.
 
 Typical use cases: first‑party analytics, marketing attribution, returning‑visitor personalization.
 
@@ -22,6 +24,7 @@ Typical use cases: first‑party analytics, marketing attribution, returning‑v
   - [Get the current client](#get-the-current-client)
   - [Read and write metadata](#read-and-write-metadata)
   - [Set metadata from a request](#set-metadata-from-a-request)
+  - [Metadata of new visitors](#metadata-of-new-visitors)
   - [Access the cookie directly](#access-the-cookie-directly)
   - [Skip storing the cookie (consent)](#skip-storing-the-cookie-consent)
 - [Configuration](#configuration)
@@ -112,7 +115,8 @@ final class SomeService
 ### Read and write metadata
 
 `$client->metadata` is a key/value store. Reading a key for the first time lazily loads the metadata from the
-database; writing marks it dirty so it is persisted at the end of the request.
+database; writing marks it dirty so it is persisted at the end of the request (for a first‑time visitor, once it comes
+back: see [Metadata of new visitors](#metadata-of-new-visitors)).
 
 ```php
 $metadata = $client->metadata;
@@ -164,6 +168,33 @@ final class GoogleClickIdSubscriber implements EventSubscriberInterface
 }
 ```
 
+### Metadata of new visitors
+
+A visitor without the cookie is a *new* client, and many new clients never come back: crawlers, link checkers, link
+preview fetchers and browsers that block cookies get a new client id on every request. So that they don't fill the
+`setono_client__metadata` table with rows nobody reads, the metadata of a new client is not written to the database
+right away. It travels to the browser in a second cookie, `setono_client_id_pending_metadata` (the cookie name plus
+`_pending_metadata`), and is persisted on the visitor's next request, i.e. once the visitor has shown that it keeps
+its cookies. A visitor that never comes back never produces a row.
+
+In the request where you set it, the metadata is available as usual, and TTLs carry over unchanged.
+
+Things to be aware of:
+
+- The pending metadata cookie is `HttpOnly` and signed with your `kernel.secret`, so JavaScript can't read it and the
+  visitor can neither forge nor change it. It is not encrypted, so visitors can see their own pending metadata in the
+  browser's developer tools until it is persisted. Changing `kernel.secret` discards pending metadata that hasn't been
+  persisted yet.
+- Metadata that doesn't fit in about 2 KB, or can't be JSON encoded, is persisted right away.
+- If the database already holds metadata for the visitor when it comes back, e.g. because a message handler stored
+  some, the pending metadata is merged into it. Where both hold a key, the database wins.
+- The metadata of a new client is captured when the cookie is stored (on `kernel.response` with priority `-64`). Set it
+  before that, e.g. on `kernel.request`, in a controller, or in a `kernel.response` listener with the default priority.
+  Metadata set later is persisted right away.
+- If you need the metadata of a new client in the database during its first request, replace the
+  `setono_client.pending_metadata_storage.default` alias with a `PendingMetadataStorageInterface` implementation whose
+  `store()` returns `false`.
+
 ### Access the cookie directly
 
 The cookie stores the client id plus the first/last seen timestamps. Read it through `CookieProviderInterface`:
@@ -222,6 +253,9 @@ final class CookieConsentSubscriber implements EventSubscriberInterface
 }
 ```
 
+If you skip the cookie for a new visitor, its metadata is not persisted either: without the cookie the visitor can't be
+recognized again, so nothing could ever read that metadata.
+
 ## Configuration
 
 All options are optional; the defaults are shown below:
@@ -230,6 +264,7 @@ All options are optional; the defaults are shown below:
 setono_client:
     cookie:
         # Name of the cookie that holds the client id.
+        # The pending metadata cookie is named after it: <name>_pending_metadata.
         # NOTE: changing this makes every visitor with the old cookie name look like a new client.
         name: setono_client_id
 
@@ -251,9 +286,14 @@ setono_client:
   A new client id is generated only when no valid cookie is present.
 - **Lazy metadata** — `$client->metadata` is a lazy ghost object. The database is queried only the first time you
   read a value, and a row is written only at the end of the request, and only if the metadata was actually changed.
+- **Pending metadata** — for a new client, `StoreCookieSubscriber` stores changed metadata in the pending metadata
+  cookie instead of the database. When the client comes back with it, the metadata is handed to the client and
+  persisted at the end of that request, and the pending metadata cookie is removed.
 - **Resolution chain** — `ClientContextInterface` is built from a chain of decorating services
-  (`CachedClientContext` → `CookieBasedClientContext` → `DefaultClientContext`). The cached layer memoizes the client
-  per request; the cookie‑based layer builds it from the cookie; the default layer creates a fresh, anonymous client.
+  (`CachedClientContext` → `PendingMetadataClientContext` → `CookieBasedClientContext` → `DefaultClientContext`). The
+  cached layer memoizes the client per request; the pending metadata layer hands a returning client the metadata it
+  was given as a new client; the cookie‑based layer builds the client from the cookie; the default layer creates a
+  fresh, anonymous client.
 
 ## Extending the bundle
 
@@ -265,6 +305,7 @@ Every moving part is a service behind an interface, so you can swap or decorate
 | Reading the cookie | `Setono\ClientBundle\CookieProvider\CookieProviderInterface` |
 | Loading metadata | `Setono\ClientBundle\MetadataProvider\MetadataProviderInterface` |
 | Persisting metadata | `Setono\ClientBundle\MetadataPersister\MetadataPersisterInterface` |
+| Holding the metadata of new clients | `Setono\ClientBundle\PendingMetadataStorage\PendingMetadataStorageInterface` |
 | Metadata entity | `Setono\ClientBundle\Entity\MetadataInterface` |
 
 To add behaviour, [decorate](https://symfony.com/doc/current/service_container/service_decoration.html) the relevant

composer.json

@@ -20,6 +20,7 @@
         "setono/doctrine-orm-trait": "^1.1",
         "symfony/config": "^6.4 || ^7.0",
         "symfony/dependency-injection": "^6.4 || ^7.0",
+        "symfony/deprecation-contracts": "^2.5 || ^3.0",
         "symfony/event-dispatcher": "^6.4 || ^7.0",
         "symfony/http-foundation": "^6.4 || ^7.0",
         "symfony/http-kernel": "^6.4 || ^7.0",

config/services.xml

@@ -17,6 +17,14 @@
             <argument type="service" id="setono_client.cookie_provider.default"/>
         </service>
 
+        <service id="setono_client.client_context.pending_metadata"
+                 class="Setono\ClientBundle\Context\PendingMetadataClientContext"
+                 decorates="setono_client.client_context.default" decoration-priority="48">
+            <argument type="service" id="setono_client.client_context.pending_metadata.inner"/>
+            <argument type="service" id="setono_client.pending_metadata_storage.default"/>
+            <argument type="service" id="request_stack"/>
+        </service>
+
         <service id="setono_client.client_context.cached"
                  class="Setono\ClientBundle\Context\CachedClientContext"
                  decorates="setono_client.client_context.default" decoration-priority="32">
@@ -64,6 +72,7 @@
             <argument type="service" id="event_dispatcher"/>
             <argument>%setono_client.cookie.name%</argument>
             <argument>%setono_client.cookie.expiration%</argument>
+            <argument type="service" id="setono_client.pending_metadata_storage.default"/>
 
             <tag name="kernel.event_subscriber"/>
         </service>
@@ -86,6 +95,20 @@
             <argument>%setono_client.metadata_class%</argument>
         </service>
 
+        <!-- Pending Metadata Storages -->
+        <service id="Setono\ClientBundle\PendingMetadataStorage\PendingMetadataStorageInterface"
+                 alias="setono_client.pending_metadata_storage.default"/>
+
+        <service id="setono_client.pending_metadata_storage.default"
+                 alias="setono_client.pending_metadata_storage.cookie_based"/>
+
+        <service id="setono_client.pending_metadata_storage.cookie_based"
+                 class="Setono\ClientBundle\PendingMetadataStorage\CookieBasedPendingMetadataStorage">
+            <argument>%setono_client.cookie.name%_pending_metadata</argument>
+            <argument>%setono_client.cookie.expiration%</argument>
+            <argument>%kernel.secret%</argument>
+        </service>
+
         <!-- Metadata Providers -->
         <service id="setono_client.metadata_provider.default" alias="setono_client.metadata_provider.empty"/>
 

src/Client/ChangeAwareMetadata.php

@@ -15,6 +15,22 @@ public function isDirty(): bool
         return $this->dirty;
     }
 
+    /**
+     * Use this when the metadata holds changes that were not made through this object, e.g. pending metadata of a new client
+     */
+    public function markDirty(): void
+    {
+        $this->dirty = true;
+    }
+
+    /**
+     * Use this when the changes must not be persisted, e.g. because they were stored as pending metadata of a new client
+     */
+    public function markClean(): void
+    {
+        $this->dirty = false;
+    }
+
     public function set(string $key, mixed $value, ?int $ttl = null): void
     {
         parent::set($key, $value, $ttl);