​

NPU Acceleration

​

Supported NPUs

​

Apple Neural Engine (iOS/macOS)

• iPhone 12 and newer (A14 Bionic+)
• iPad Air (4th gen) and newer (M1+)
• iPad Pro (3rd gen) and newer (A12X+)
• Mac with Apple Silicon (M1, M2, M3, M4)
• Apple Watch Series 6+ (S6+)

• Vision models: LFM2-VL-450M, LFM2.5-VL-1.6B
• Speech models: whisper-tiny, whisper-base, whisper-small, whisper-medium, parakeet-ctc-0.6b, parakeet-ctc-1.1b, parakeet-tdt-0.6b-v3

• 5-11x faster prefill on iOS/macOS
• Vision: 0.2-0.3s first token latency (vs 2-5s CPU)
• Speech: 0.1-0.7s transcription latency (vs 1-10s CPU)

​

Qualcomm Hexagon DSP (Android)

• Snapdragon 8 Gen 1 and newer
• Snapdragon 7 Gen 1 and newer (select models)

​

MediaTek APU (Android)

• Dimensity 9000 series
• Dimensity 8000 series

​

Google Tensor (Android)

• Pixel 6, 7, 8, 9 series (TPU)

​

Performance Benchmarks

​

With Apple NPU

​

Without NPU (Android)

​

How NPU Acceleration Works

1. NPU for Prefill — Encoder and prefill transformer layers run on NPU
2. CPU/SIMD for Decode — Token-by-token generation uses ARM SIMD kernels
3. Zero-Copy Handoff — Seamless transitions between NPU and CPU

​

Why Hybrid?

• NPU excels at batch processing — Prefill processes many tokens at once
• CPU excels at autoregressive decode — Single token generation is memory-bound
• Best of both worlds — 5-11x faster prefill, no decode slowdown

​

Enabling NPU Backend

​

Automatic (Default)

1. Device has compatible NPU hardware
2. Model supports NPU acceleration
3. Cactus runtime includes NPU support

​

Checking NPU Availability

#include "cactus/npu/npu.h"

if (cactus::npu::is_npu_available()) {
    printf("NPU acceleration available\n");
} else {
    printf("NPU not available, using CPU fallback\n");
}

​

Loading NPU Prefill Model

#include "cactus/engine/engine.h"

cactus::Engine* engine = /* ... load model ... */;

// Load NPU-optimized prefill weights
if (engine->load_npu_prefill("./model/npu_prefill.mlmodelc")) {
    printf("NPU prefill loaded, chunk size: %zu\n", 
           engine->get_prefill_chunk_size());
} else {
    printf("NPU prefill not available, using CPU\n");
}

​

Disabling NPU Backend

​

Configuration Options

​

Prefill Chunk Size

// Get current chunk size
size_t chunk_size = engine->get_prefill_chunk_size();
printf("Prefill chunk size: %zu\n", chunk_size);

// Chunk size is determined by NPU model architecture
// Not configurable at runtime

• Memory usage — Larger chunks use more NPU memory
• Latency — Smaller chunks add overhead
• Throughput — 256 is optimal for most models

​

KV Cache Window

// Set sliding window cache (e.g., 2048 tokens)
engine->set_cache_window(2048, 4);  // window_size, sink_size

// NPU prefill will process chunks up to window limit

​

Platform Requirements

​

iOS Requirements

• iOS 14.0+ for Neural Engine support
• iOS 15.0+ recommended (better ANE APIs)
• A12 Bionic or newer (iPhone XS, XR, and later)

​

macOS Requirements

• macOS 11.0+ (Big Sur) for Apple Silicon
• M1 or newer (M1, M2, M3, M4)
• Intel Macs not supported (no Neural Engine)

​

Android Requirements (Coming Soon)

• Android API 29+ (Android 10) for Hexagon DSP
• Snapdragon 8 Gen 1+ or Dimensity 9000+
• NNAPI or QNN runtime installed

​

Model Compatibility

​

NPU-Accelerated Models

• LiquidAI/LFM2-VL-450M — Vision, txt & img embed, Apple NPU
• LiquidAI/LFM2.5-VL-1.6B — Vision, txt & img embed, Apple NPU

• openai/whisper-tiny — Transcription, speech embed, Apple NPU
• openai/whisper-base — Transcription, speech embed, Apple NPU
• openai/whisper-small — Transcription, speech embed, Apple NPU
• openai/whisper-medium — Transcription, speech embed, Apple NPU
• nvidia/parakeet-ctc-0.6b — Transcription, speech embed, Apple NPU
• nvidia/parakeet-ctc-1.1b — Transcription, speech embed, Apple NPU
• nvidia/parakeet-tdt-0.6b-v3 — Transcription, speech embed, Apple NPU

• Most LLMs use NPU for prefill (when available)
• Decode always uses CPU SIMD kernels

​

Models Without NPU Support

• google/gemma-3-* — CPU-only (for now)
• Qwen/Qwen3-* — CPU-only (for now)
• Embedding-only models — CPU-only
• VAD models — CPU-only

​

Troubleshooting

​

NPU Not Available

if (!cactus::npu::is_npu_available()) {
    // Device doesn't have compatible NPU, or
    // NPU support not compiled into runtime
}

• Older device (pre-A12 iPhone, Intel Mac)
• Android device (NPU coming Mar 2026)
• Simulator build (NPU only on physical devices)

​

Slow Despite NPU

1. Check model supports NPU — Not all models have NPU variants
2. Verify NPU prefill loaded — Call load_npu_prefill() explicitly
3. Measure properly — Use --benchmark flag for accurate timing
4. Thermal throttling — Device may throttle under sustained load

​

Memory Issues

• Vision models: +100-200MB for NPU weights
• Speech models: +50-100MB for NPU weights
• Text models: +20-50MB for NPU prefill cache

1. Reduce KV cache window size
2. Use smaller model variant
3. Disable NPU prefill (not recommended)

​

Roadmap

​

Q1 2026 (Shipped)

• ✅ Apple Neural Engine support (Jan 2026)
• ✅ Vision model NPU acceleration
• ✅ Speech model NPU acceleration
• ✅ 5-11x prefill speedup on iOS/macOS

​

Q1-Q2 2026 (Coming)

• 🚧 Qualcomm Hexagon DSP (Mar 2026)
• 🚧 Google Tensor TPU (Mar 2026)
• 📅 MediaTek APU (Apr 2026)
• 📅 Samsung Exynos NPU (Apr 2026)

​

Q2-Q3 2026 (Planned)

• 📅 Mac GPU acceleration (May 2026)
• 📅 Wearables optimizations (Jul 2026)
• 📅 Additional model NPU support

​

See Also

• Performance Tuning — Optimize KV cache and prefill
• Compatibility — Platform requirements
• Custom Models — NPU support in fine-tunes
• Cactus Roadmap — Upcoming features