quantprint

Figure out what an LLM API is really serving. quantprint reads the top-k logprobs any endpoint already returns and tells you which model and which quantization produced them.

Providers can swap fp16 for int4, change a kernel, or route to a smaller model without telling anyone. Right now nobody outside the provider can prove it happened. quantprint is the audit tool for that gap.

The claim, and how we test it

Claim. Weight quantization and compute dtype leave a statistical fingerprint in the shape of the top-k next-token distribution, and that fingerprint is strong enough to identify the serving condition from a few dozen API calls.

Signal. Two numbers, both computed on prompts the classifier never saw during training:

1. Held-out accuracy over conditions like Llama-3.1-8B/fp16, /int8, /nf4. Chance is 1/N.
2. Samples-to-detect. Given a server that claims condition R, how many probe calls until we reject R at 150:1 odds, and how often we falsely reject when the server really is R.

If (1) lands near chance, or (2) shows a nonzero false-alarm rate on the diagonal, the claim is false and this README will say so.

Status

Pipeline verified on synthetic traces. No real-model numbers yet. The synthetic noise model is a guess, so the synthetic result (chance without pairing, 1.00 with pairing, 2 samples to detect) says the plumbing works and nothing more. The first real result comes from notebooks/colab_collect.ipynb.

One real finding so far. Per-response features are dominated by prompt content, so absolute statistics of the top-k distribution do not separate conditions. Delta features against a per-prompt reference profile do. That is why the probe set is fixed and versioned in prompts/probe_set.txt.

How it works collect.py local model, known (model, quant) -> runs/A_.jsonl ground truth, run A

collect.py same, second pass -> runs/B_.jsonl held-out run collect.py reference condition, third pass -> runs/REF_*.jsonl per-prompt reference profile features.py top-k logprobs -> margin/entropy/tailmass/gaps/grid, raw + delta vs profile train.py GBDT, GroupKFold by prompt, ablation per feature group evaluate.py samples-to-detect table over all (claimed, actual) pairs probe_api.py same probe set against any OpenAI-compatible endpoint detect.py sequential log-likelihood-ratio monitor with a per-sample evidence cap ledger.py weekly cron: probe providers, append verdicts to runs/ledger.md Holdout is by prompt, never by response. The reference profile comes from a separate run so the reference class cannot be identified by a zero delta.

Everything downstream of collect.py is CPU-only and runs anywhere. The schema is exactly what APIs expose: per token, k logprobs and the rank of the chosen token. Nothing learned locally depends on information a black box would withhold.

Reproduce

pip install -r requirements.txt
python -m pytest -q tests                      # pipeline tests on synthetic data

On a GPU, which is what the Colab notebook automates:

python -m quantprint.collect --model Qwen/Qwen2.5-3B-Instruct --quant nf4 --out runs/A_qwen3b_nf4.jsonl
python -m quantprint.train "runs/A_*.jsonl" --reference runs/REF_qwen3b_fp16.jsonl --ablate --out runs/clf.joblib
python -m quantprint.evaluate --train "runs/A_*.jsonl" --test "runs/B_*.jsonl" --reference-run runs/REF_qwen3b_fp16.jsonl

Experiments queued, in order

1. Qwen2.5-3B on a T4 at fp16, int8, nf4, greedy decoding. Does anything separate at all?
2. Ablate feature groups. Report which ones carry the signal, then check each for collection artifacts such as response length or EOS timing.
3. Add --temperature 0.7 runs. Greedy traces may come out byte-identical across runs A and B, which makes the holdout too easy. Sampled runs are the honest test.
4. fp16 versus bf16. If those separate, the method sees compute dtype and not only weight quantization.
5. Cross-model transfer: train on Qwen, test on Llama. Expect failure, and measure how bad.
6. Point the Llama profile at three commercial providers serving Llama-3.1-8B, then publish the ledger.

What would make this a company

The weekly ledger. A public, reproducible record of what each provider actually serves, updated by cron, with the probe set and classifier committed so anyone can rerun it. Inference is a commodity market with no independent quality auditing. This is the auditor.

Maintenance

.github/workflows/ci.yml runs the tests on every push. .github/workflows/weekly_probe.yml re-probes providers every Monday and commits the ledger. When a provider legitimately upgrades their model, the classifier needs a new reference run. The ledger row will show DRIFT with a suspect that matches no known quant of the claimed model, and that is your cue.