Repository navigation
Reduce first-query cost of the bundled Lovdata index #21
Description
Consumers query Norwegian legal metadata offline through Get-LovdataDocument, Find-LovdataDocument, and Get-LovdataLegalArea, all of which read the bundled index. Bundling the whole corpus rather than a small list was the main design risk, and the lazy-load design handles it well: nothing is read at import, so a caller using only Get-LovdataPublicDataset or Test-LovdataConnection never pays for the corpus, and the packaged artifact is trivial.
Request
User impact
The first index query — the moment a caller touches any of the three document commands — costs roughly 2.3 seconds and a one-time jump of about 254 MB resident memory, regardless of what the caller asked for. Import-LovdataIndexData reads every source file and materialises all 4,185 records as LovdataDocument instances up front, then caches them for the session. Subsequent queries are fast (~22 ms). A quarter of a gigabyte of live objects is material in CI runners, containers, and constrained hosts, and a caller querying only acts (nl) still pays for sf, del, ins, and stv.
Measurements taken by building the module locally and copying the index to the module root as Build-PSModuleBase does:
| Metric | Value |
|---|---|
| Index JSON on disk | 9.94 MB across 5 files |
| Built module folder | 10.03 MB |
| Packaged (zip/nupkg) | 1.21 MB |
Import-Module cold |
50 ms |
| Memory after import | 91 MB |
| First index query | 2,319 ms |
| Memory after first query | 345 MB |
| Subsequent query | 22 ms |
Find-LovdataDocument -Name 'veg' |
89 ms, 112 hits |
The nl source is 1.87 MB of the 9.94 MB total, so a caller who only needs acts currently loads roughly five times more data and memory than the query requires. Constructing 4,185 full class instances is what dominates the memory: a search returning 112 hits does not need 4,185 constructed objects to exist first.
Acceptance criteria
- A caller can restrict a query to the sources it needs, and doing so loads and constructs only those sources.
- The default behaviour is unchanged for callers that pass no source, so nothing regresses.
- First-query time and peak memory drop measurably for source-scoped queries, and ideally for the whole-index case as well.
- No change to the shipped index files or to the object shape callers already receive.
Regression
Not a regression. This is the behaviour since the index was introduced in #18. Deferred from the review of #20 to keep that correctness fix (per-reference LastChangedBy dates) landing clean on its own.
Technical decisions
Two independent improvements, either deliverable on its own.
1 — Load only the sources needed. Import-LovdataIndexData (in src/functions/private/Parsing/) takes no source parameter today; it globs LovdataIndex.*.json and reads all of them into one session cache. Add an optional Source parameter that selects which per-source files to read, defaulting to all so existing callers are unaffected. Thread an equivalent selector through the three public commands so a caller can scope a query. The cache becomes per-source rather than a single all-or-nothing blob. Open decision: whether the source selector is surfaced on the public commands as a parameter with a ValidateSet of the known bases (nl, sf, del, ins, stv) or discovered from the shipped files at runtime — discovery avoids hardcoding but loses tab-completion.
2 — Defer LovdataDocument construction. The loader currently calls [LovdataDocument]::new(...) for every record eagerly. Keep the deserialised records and construct a LovdataDocument only for the records a query actually returns. This preserves the [LovdataDocument] output contract the commands and tests assert while removing the up-front construction of thousands of objects that are never returned.
Breaking changes: None. Both changes are additive; default output type and default behaviour are preserved.
Test approach: Assert that a source-scoped query does not read or construct other sources (spy on file reads / count constructed instances), that the default path still returns every document, and that the output remains [LovdataDocument].
Implementation plan
Core changes
- Add an optional
Sourceparameter toImport-LovdataIndexDatainsrc/functions/private/Parsing/, defaulting to all sources, with a per-source cache - Surface a source selector on
Get-LovdataDocument,Find-LovdataDocument, andGet-LovdataLegalArea - Defer
LovdataDocumentconstruction to the records actually returned rather than constructing the whole index up front
Tests
- Add a test asserting a source-scoped query reads and constructs only that source
- Add a test asserting the default (no source) path still returns every document with type
LovdataDocument - Add a test asserting deferred construction does not change the returned object shape
Documentation
- Update the affected command help and
Documents.mdwith the source parameter - Add an example showing a source-scoped offline query
Activity
MariusStorhaug commentedon Aug 2, 2026
Measurements
Taken on a locally built module with the index copied to the module root as
Build-PSModuleBasedoes. Recording them here so they are not re-measured or guessed at later.nl759,sf3 426, plusdel,ins,stv)Import-ModulecoldFind-LovdataDocument -Name 'veg'Packaged size and import time are both fine and need no work. The index is genuinely lazy, so anyone using only the download or service commands never pays for it. This issue is only about the first query.
This is the common path, not a tail case
Framing it as a "first-use cost" understates it. For this module, querying the corpus is the primary reason to install it, so a user of the archive surface pays this once per session, every session. It is the common case rather than a rare one.
Where the cost actually is
The obvious assumption is that constructing 5 879
LovdataDocumentinstances is the whole problem. That was worth testing before committing to a refactor built on it, so the two stages were measured separately in a clean process:ConvertFrom-Jsononly, no class constructionLovdataDocumentconstruction on topClass construction is the larger share — 58% of the memory and 81% of the time. So deferring it is worth doing.
But deserialization alone is still 93.5 MB, which is 42% of the footprint. That is the floor: deferring class construction cannot take the module below roughly 93 MB and 440 ms without also changing the on-disk or in-memory shape. Anyone planning that refactor should size it against that ceiling rather than expecting the cost to disappear.
Proposed order
1. Source-scoped loading — cheap, do this first.
Import-LovdataIndexDatareads every source file regardless of what the caller asked for.nlis 1.87 MB of the 9.94 MB, so someone querying only acts currently pays roughly 5x what their query needs. Adding a source parameter, with the present behaviour as the default, is small, backwards-compatible, and independently useful.2. Defer class construction — the bigger win, but a real refactor. Keep the deserialized records and build a
LovdataDocumentonly for what is returned.Find-LovdataDocumentreturning 112 hits does not need 5 879 constructed objects to exist. Worth keeping separate from item 1 so the cheap change can land on its own.Item 2 is bounded by the 93.5 MB deserialization floor above. If that is still too much, the answer is a leaner on-disk shape or streaming, not a deeper object-construction change.
MariusStorhaug commentedon Aug 2, 2026
Resolving the open decision on how
Sourceshould be constrained: an argument completer, not aValidateSet.The choice was framed as tab-completion versus not hardcoding, but those are not actually in tension.
ValidateSetis what couples the two — it is the only one of the options that both completes and validates. An[ArgumentCompleter]gives completion from the shipped files at runtime, with no baked-in list.This is the established pattern in the ecosystem rather than a novel suggestion:
PSModule/ContextandPSModule/GitHubboth usecompleters.ps1files registeringRegister-ArgumentCompleterfor exactly this shape of value — GitHub has sixteen of them. The convention is acompleters.ps1alongside the commands in each public function group.Two reasons
ValidateSetis the wrong tool here specifically:ValidateSetwould have to be edited whenever Lovdata adds or renames one, and the daily updater would then be capable of producing an index the module refuses to read. The same reasoning already applies elsewhere in this module: the archive reader was deliberately written to enumerate the extracted folders rather than assumenl, because the regulations package uses different ones.ValidateSetrejects at bind time with a generic message. An unknown source is better handled by the loader returning nothing for it, or by naming the sources actually present, which tells the caller something the validator cannot.So: complete from the index files present on disk, and let the loader handle an unrecognised value with a message that names what is available.