feat(memory): add depth codecs and default to lossless JPEG XL - #3637
feat(memory): add depth codecs and default to lossless JPEG XL#3637TomCC7 wants to merge 7 commits into
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #3637 +/- ##
==========================================
+ Coverage 77.26% 77.36% +0.09%
==========================================
Files 1262 1274 +12
Lines 120302 121642 +1340
Branches 10600 10844 +244
==========================================
+ Hits 92953 94110 +1157
- Misses 24284 24469 +185
+ Partials 3065 3063 -2
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 24 files with indirect coverage changes 🚀 New features to boost your workflow:
|
TomCC7
changed the title
Greptile SummaryThis change adds lossless JPEG XL depth storage, an opt-in LERC codec, updated depth recorder defaults, benchmark tooling, tests, and documentation. Two issues need resolution before merging: benchmark dataset loading can execute code embedded in pickle files, and newly written JPEG XL depth records retain the legacy T-Rex validation blockedThe pickle failure path executed and showed code running before payload rejection, but the artifact-upload tool was unavailable. The missing tool prevented attaching the required published evidence reference for that result. Confidence Score: 2/5Not safe to merge until untrusted benchmark input is no longer executable and the persisted JPEG XL depth format has an explicit compatibility strategy. The JPEG XL reader incompatibility was reproduced with a real current encoding and the pre-change decode path. The pickle behavior was also exercised directly, but its evidence artifact could not be published because the upload tool was unavailable. Files Needing Attention: dimos/memory/codecs/tool_depth_benchmark.py and dimos/memory/codecs/jpeg.py
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
|
|
||
| def _pickle_frame(path: Path, source: Path) -> Image: | ||
| payload = pickle.loads(path.read_bytes()) |
There was a problem hiding this comment.
Benchmarking a directory or named dataset deserializes every discovered depth/*.pickle file with pickle.loads before checking whether the decoded value has an accepted type or shape. A crafted pickle can therefore execute its reconstruction hook with the benchmark process's privileges even though it is subsequently rejected as an invalid depth payload. Do not deserialize discovered dataset files with unrestricted pickle; use a non-executable frame format or a strictly allowlisted legacy decoder.
| if value.format in (ImageFormat.DEPTH, ImageFormat.DEPTH16): | ||
| return value.lcm_jpegxl_encode(effort=1) | ||
| return value.lcm_jpeg_encode(quality=self._quality) |
There was a problem hiding this comment.
JpegCodec still persists under the jpeg codec ID, but depth frames now carry jxl;depth or jxl;depth16 envelopes. A reader from before JPEG XL envelope dispatch still selects the legacy jpeg codec and raises ValueError: Unsupported encoding: jxl;depth16 for these newly written records. Version or distinguish this persisted format transition and provide an explicit compatibility or migration path so older readers do not treat JPEG XL depth data as legacy JPEG.
Artifacts
JPEG codec compatibility reproduction source
- This authored harness encodes a DEPTH16 frame through the real current codec and applies the exact pre-JPEG-XL decoder branch, Takeaway.
Base-parent JPEG codec and decoder capture
- This command capture shows the base-parent implementation before JPEG XL envelope dispatch existed, Takeaway.
Current depth payload through pre-change decoder
- This executed run proves the same newly encoded depth payload fails in the pre-change path and succeeds in the current path, Takeaway.
2 participants
Contribution path
Problem
The default
Imagecodec sends every image through ordinary lossy JPEG, whichcorrupts metric depth. Some recorders avoid that path with generic LZ4, but LZ4
leaves substantial size savings on the table and provides no bounded-error option.
Solution
JpegCodecandjpegcodec ID. Visual images retain theoriginal lossy JPEG bytes and LCM
Imageenvelope;DEPTH/float32andDEPTH16/uint16use lossless JPEG XL in that same envelope.lerccodec with a fixed 5 mm maximum error. It supportsfloat32 meters and uint16 millimeters while preserving metadata and validity masks.
grayscale remains explicitly lossless with
lz4+lcm.lcm,lz4+lcm,jpeg, andlerc.highlights, while retaining complete JSON and Markdown artifacts.
Benchmark workflow
The runner processes every frame in every detected depth stream through the
production codec objects. It measures encoded size, encode/decode wall and
process-CPU time, and metric fidelity, then writes
results.jsonandresults.md.After all streams finish, it prints one compact terminal report with wall-time,
fidelity, and exact-codec highlights. It fails if metadata changes, an exact codec
changes depth values, LERC exceeds 5 mm, the valid mask changes, or a stream
changes dtype, shape, or format.
Real-recording benchmark
How to test
Quick synthetic check:
Reproduce the three-recording run:
Validation performed:
487 passed; both executable documentation paths, Ruff, formatting, diff checks,and mypy completed without errors.
AI assistance
OpenAI Codex with GPT-5 handled the research, implementation, tests, benchmarks,
documentation, and PR draft under user direction.
Checklist