Splats

A Rust/Ratatui application that turns a directory of overlapping photographs and videos into a Gaussian splat and a standalone web viewer.

Current release: Linux x86-64 / NVIDIA alpha. The entire local pipeline is implemented and has been exercised on Fedora 44 with an RTX 4070 Ti. Processing backends are isolated behind versioned contracts so additional platforms and trainers can be added independently.

Quick start

Build/install using a current stable Rust toolchain:

cargo install --path crates/splats-cli --locked

splats init LivingRoom
# Put photographs and/or videos in ~/Splats/LivingRoom/input/
splats process LivingRoom
splats view LivingRoom

Existing projects with an input/ directory work without init.

~/Splats/LivingRoom/
├── input/                   original photos/videos
└── output/
    ├── scene.ply            full Gaussian PLY
    ├── scene.json           coordinate conventions and initial camera
    ├── training.json        training/evaluation metrics
    ├── report.json          export provenance
    ├── poster.webp          reference render
    └── web/                 upload this entire directory to a static HTTP host
        ├── index.html
        ├── index.js
        ├── index.css
        ├── index.sog
        ├── settings.json
        └── poster.webp

splats process presents a quality selector and a live dashboard. Press L for logs, Q to stop at a recoverable point, or Ctrl-C twice to force termination. If reconstruction finds separate substantial scenes, the TUI asks which to train. Browser preview is a localhost HTTP server; Ctrl-C stops that server.

First-run dependencies

The first processing run automatically installs the managed toolchain. You can install it ahead of time:

splats doctor
splats setup

This downloads several gigabytes of runtime packages. A working NVIDIA driver and network access are needed for initial setup. Subsequent processing uses local tools and cached packages. Source media stays local.

Managed components include Python 3.10, FFmpeg libraries through PyAV, Pillow/HEIF support, CUDA-enabled PyCOLMAP, PyTorch/gsplat, Node.js, and PlayCanvas SplatTransform. Precompiled gsplat wheels are used: users do not need a CUDA compiler. Reconstruction and training have separate environments because their CUDA runtime dependencies differ. Dependencies are pinned and hash-checked; Node and uv archives also have pinned checksums.

Configuration and cache

All application configuration, managed dependencies, cache, logs, and recovery state live under ~/.splats/:

~/.splats/
├── config.toml
├── state.sqlite3
├── toolchains/
├── workers/                  immutable, content-versioned worker bundles
├── projects/<project-id>/
│   ├── config.toml
│   ├── runs/<run-id>/         resolved plan, stage requests and logs
│   └── checkpoints/<key>/    full training resume state
├── cache/
│   ├── artifacts/<project-id>/<key>/
│   ├── downloads/
│   ├── uv/
│   ├── npm/
│   └── tmp/
└── locks/

splats config                # print effective global settings
splats config --init         # create ~/.splats/config.toml if absent

Example global configuration:

schema_version = 1
project_root = "~/Splats"
profile = "balanced"
checkpoint_every = 500
gpu = 0

Optional fields: max_images, max_resolution, iterations, max_gaussians, and component.

Precedence is defaults → global config → remembered project config → command-line arguments. Effective project choices are saved centrally. Project IDs derive from canonical paths, so a moved project currently registers as a new project.

SPLATS_HOME=/absolute/path overrides the application home for isolated installations and tests. --root /path overrides the capture-project root.

Commands

splats process LivingRoom --profile fast
splats process LivingRoom --yes --no-tui
splats process LivingRoom --json
splats process LivingRoom --dry-run
splats process LivingRoom --resume
splats process LivingRoom --restart-from train
splats process LivingRoom --component 0

splats list
splats status LivingRoom
splats logs LivingRoom --stage train
splats export LivingRoom
splats view LivingRoom --no-open --port 8080
splats cache status
splats cache prune --yes

Stage names are prepare, reconstruct, train, and export.

• Repeating process reuses checksum-validated completed stages.
• Training automatically loads a compatible full checkpoint.
• --resume additionally requires an unchanged plan and an interrupted/failed previous run.
• --restart-from discards the affected regenerable stages and their checkpoints.
• export requires compatible cached upstream artifacts and will not silently retrain.
• --dry-run hashes inputs and displays a plan without creating state or installing tools.
• --json emits newline-delimited events and never prompts. Required component choices become actionable errors.
• cache prune removes regenerable artifacts; it retains published outputs, toolchains, and resume checkpoints. It refuses to run during active processing.

A checkpoint also requires matching reconstructed cameras and images. Pruning its upstream artifacts may make that checkpoint unusable if reconstruction produces a different solution.

Initial profiles

Profile	Image budget	Long edge	Iterations	Maximum Gaussians
Fast	200	1,280	7,000	500,000
Balanced	500	1,600	15,000	1,000,000
High	1,000	1,920	30,000	2,000,000

These are starting profiles, not quality or timing guarantees. Override individual limits with --max-images, --max-resolution, --iterations, --max-gaussians, and --checkpoint-every.

What the pipeline does

1. Inspect: validate and hash source media, resolve configuration, acquire project/GPU locks.
2. Prepare: normalize EXIF orientation/color profiles, resize photos, sample sharp video frames within temporal windows, remove exact pixel duplicates, retain source/timestamp mappings.
3. Reconstruct: GPU SIFT, sequential or exhaustive matching, sparse COLMAP mapping, coverage checks, optional global-pair retry, undistortion, camera-axis/scale normalization, dataset export.
4. Train: gsplat MCMC training with bounded Gaussian count, L1/SSIM loss, contiguous evaluation holdout, full checkpoints and metrics. CUDA OOM triggers one half-resolution retry; the fallback is recorded and retained in checkpoints.
5. Export: validate Gaussian attributes, preserve canonical PLY, convert to SOG, generate a pinned static viewer with a reconstructed camera as its starting position.
6. Publish: copy a complete generation onto the destination filesystem and atomically exchange the output directory. Existing non-Splats output is protected.

Source files are never rewritten. Failed stages do not replace successful published output. A hard process kill can lose work since the last checkpoint, but completed stages remain reusable.

Capture support and current boundaries

• Photos: JPEG, PNG, HEIC/HEIF, WebP, TIFF.
• Video containers: MP4, MOV, MKV, AVI, M4V, WebM; decoding depends on the bundled FFmpeg codecs. H.264 capture is covered by automated tests.
• Captures should depict a mostly static scene with overlapping translated viewpoints.
• HDR video is detected and asks for SDR conversion; automatic HDR tone mapping is not implemented.
• Large unordered datasets currently use exhaustive matching. Retrieval-based matching, semantic masks, LOD streaming, and additional hardware backends are follow-up work.
• Up direction is estimated from camera orientations. Metric scale requires external information and is not inferred.
• Native cameras/scene artifacts are retained for diagnosis. A browser viewer is included; collision geometry and mesh generation are not part of this pipeline.
• Full training recovery is implemented. Reconstruction restarts an interrupted stage while reusing completed preparation.

See capture guidance, architecture and contracts, implementation roadmap, and development/testing.

Verification

CPU tests cover configuration precedence, input containment, cache corruption, atomic publication, worker protocol validation, cancellation, EXIF correction, and duplicate handling. The GPU test interrupts training after densification, resumes, and checks optimizer counters, scheduling, random states, and rendered-image agreement.

An original synthetic room capture is supplied as a generator rather than a large downloaded dataset. The full pipeline has reconstructed its camera poses, trained on the RTX 4070 Ti, and rendered the exported bundle in Chromium without external requests. Real-world capture quality still needs broader benchmarking.

License

Splats source is MIT licensed. Managed third-party components retain their own licenses; see THIRD_PARTY.md.