🛠️[v0.3.47] Release Note: Multi-Output Sampling, Pocket TTS and audio helper API Bindings, and Llama State Reset Improvements

[JamePeng]

0.3.47

Multi-Output Sampling, Pocket TTS and audio helper API Bindings, and Llama State Reset Improvements

Version 0.3.47 is a small iterative update focused on keeping the Python bindings synchronized with recent llama.cpp changes, especially around MTMD audio generation, sampler backends, and model state management.

Pocket TTS and MTMD Audio API Updates

This release continues the MTMD synchronization work with new Pocket TTS and audio helper bindings.

The updates include:

• Pocket TTS audio generation types and related fields
• updated generated-audio structures for the latest MTMD ABI
• default generation parameters
• experimental audio helper APIs
• multimodal chat capability detection

These changes mainly expose and synchronize the latest upstream MTMD interfaces at the binding level, preparing the Python side for future audio-generation workflows.

Multi-Output Backend Sampler Support

The sampler bindings have been updated for llama.cpp's newer multi-output backend behavior.

This includes:

• per-sequence output limits in context parameters
• updated sampler reset interfaces
• sampler state-copy support
• size_t alignment for sampler counts
• preservation of advanced context settings when rebuilding Llama instances

The related documentation also explains the ordered multi-output sampling behavior more clearly.

For custom samplers, the new backend hooks are explicitly disabled where they are not implemented. backend_reset and copy_state are set to NULL, while the CPU callback behavior and inherited backend semantics are documented more clearly for CustomSampler and ReasoningBudgetSampler.

More Complete Model State Reset

Llama.reset() now clears the native context state more completely.

In addition to synchronizing the Python-side state, reset now clears native context memory and invalidates hybrid checkpoints where required.

This improves reset behavior across:

• standard transformer models
• recurrent models
• hybrid architectures

The goal is to ensure that a reset actually returns the context to a clean state instead of leaving native state behind.

llama.cpp Synchronization

Updated llama.cpp to:

ggml-org/llama.cpp
commit ad1de39e0708e3ced9c71bb3c82d93a2c046a73f

The llama, MTMD, and GGML API bindings were synchronized on:

2026-08-13

Overall, 0.3.47 is mainly an API synchronization and compatibility release, extending the groundwork for audio generation and newer sampler workflows while improving state consistency in the existing Llama runtime.

Full comparison:

81190b0...9acda8b

JamePeng

[JamePeng]

😉Qwen 3.8 27B working fine with llama_multimodal.GenericMTMDChatHandler:

Exmple code(Qwen 3.8)

from llama_cpp import Llama
import base64
import os

# Model and multimodal projection paths
MODEL_PATH = r"./Qwen3.8-27B-Q5_K_M.gguf"
MMPROJ_PATH = r"./mmproj-Qwen3.8-27b-BF16.gguf"

# Initialize the Llama model with vision support
llm = Llama(
    model_path=MODEL_PATH,
    mmproj_path=MMPROJ_PATH,
    n_gpu_layers=-1,    # Use all available GPU layers
    n_ctx=10240,         # Context window size
    ctx_checkpoints=0,
    verbose=True,
    verbosity=2,
    chat_handler_kwargs={
        "extra_template_arguments": {
            "enable_thinking": True,
            "preserve_thinking": True,
            "reasoning_effort": "medium", # xhigh by default; supported levels are xhigh, medium, and low
        },
        "verbose": True,
    },
)

_IMAGE_MIME_TYPES = {
    # Most common formats
    '.png':  'image/png',
    '.jpg':  'image/jpeg',
    '.jpeg': 'image/jpeg',
    '.gif':  'image/gif',
    '.webp': 'image/webp',
    '.bmp':  'image/bmp',
}

def image_to_base64_data_uri(
    file_path: str,
    *,
    fallback_mime: str = "application/octet-stream"
) -> str:
    """
    Convert a local image file to a base64-encoded data URI with the correct MIME type.

    Supports 20+ image formats (PNG, JPEG, WebP, AVIF, HEIC, SVG, BMP, ICO, TIFF, etc.).

    Args:
        file_path: Path to the image file on disk.
        fallback_mime: MIME type used when the file extension is unknown.

    Returns:
        A valid data URI string (e.g., data:image/webp;base64,...).

    Raises:
        FileNotFoundError: If the file does not exist.
        OSError: If reading the file fails.
    """
    if not os.path.isfile(file_path):
        raise FileNotFoundError(f"Image file not found: {file_path}")

    extension = os.path.splitext(file_path)[1].lower()
    mime_type = _IMAGE_MIME_TYPES.get(extension, fallback_mime)

    if mime_type == fallback_mime:
        print(f"Warning: Unknown extension '{extension}' for '{file_path}'. "
              f"Using fallback MIME type: {fallback_mime}")

    try:
        with open(file_path, "rb") as img_file:
            encoded_data = base64.b64encode(img_file.read()).decode("utf-8")
    except OSError as e:
        raise OSError(f"Failed to read image file '{file_path}': {e}") from e

    return f"data:{mime_type};base64,{encoded_data}"


# ========================
# Main image processing & inference section
# ========================

# 1. List of image paths you want to analyze (supports mixed formats)
image_paths = [
    r"1.png", # image_path
]

# 2. Container for message content (each image + final text prompt)
user_content = []

# 3. Convert every image to a properly formatted data URI message
for path in image_paths:
    data_uri = image_to_base64_data_uri(path)
    user_content.append({
        "type": "image_url",
        "image_url": {"url": data_uri}
    })

# 4. Append the text instruction (appears after all images in the message)
user_content.append({
    "type": "text",
    "text": "Describe the contents of this image in detail."  # You can change the prompt as needed
})

# 5. Perform chat completion with vision
# Sampling Parameters: We suggest using the following sets of sampling parameters:
# Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
# Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
response = llm.create_chat_completion(
    messages=[
        {"role": "user", "content": user_content}
    ],
    temperature=1.0,
    top_p=0.95,
    top_k=20,
    min_p=0.0,
    present_penalty=0.0,
    repeat_penalty=1.0,
    max_tokens=8192,
)

# 6. Print the model's reply
print(response["choices"][0]["message"]["content"])

[allanmeng]

eval time = 14854.75 ms / 1242 runs (11.96 ms per token, 83.61 tokens per second)

Metric	0.3.45	0.3.47	Change
Generation speed	72.7 / 72.9 t/s	83.61 t/s	🚀 +15%
Per-token time	13.75 ms	11.96 ms	✅ Faster
Load time (warm)	4003 ms	3179 ms	✅ Faster
Image encoding	1752 ms	1765 ms	≈ Same
Total (warm)	33.28s	25.75s	✅ 7.5s faster

🔍 Key Takeaways

1. 83.61 t/s is ~15% faster than 0.3.45 — llama.cpp ad1de39e brings real performance gains (likely from new kernel optimizations or the sampler refactor)
2. Warm start 25.75s (was 33s) — ~23% overall improvement
3. The 141s cold start is a one-time cost (model loading + kernel JIT compile), unrelated to steady-state performance

✅ Verdict

0.3.47 is fully healthy — actually the fastest version we've measured.

---

🖥️ Test Environment

Item	Version
GPU	Intel Arc B580 (Battlemage)
Driver	Latest
OS	Windows 11 x64
Python	3.13.11
Intel oneAPI	2026.1
llama-cpp-python	0.3.47+sycl (self-built)
Model	Qwen3-VL (vision, GGUF)
ComfyUI	0.16.3
Build flags	-DGGML_SYCL=on -DGGML_SYCL_TARGET=INTEL -DCMAKE_C_COMPILER=icx

[JamePeng]

Starting with the next release, I will gradually roll out new LlamaSpecEngine code for speculative decoding features that have been developed and tested locally. For the MTP component, priority will be given to refining text-only capabilities, while optimization of multimodal features will be addressed subsequently.

0.3.48-preview 20260812：

Qwen3.5-9B-MTP:

0.3.48-preview 20260816

Qwen3.5-9B-MTP:

Qwen3.8-27B-Q5_K_M:

[allanmeng]

这个是什么测试环境？ 显卡和内存是？

[JamePeng]

RTX5090 32GB + 192GB DDR5

[JamePeng]

0.3.48-preview 20260819：

Increased throughput and optimization of decoding latency

Qwen3.5-9B-MTP:

Qwen3.8-27B-Q5_K_M: