Documentation Index

Fetch the complete documentation index at: https://mintlify.com/cactus-compute/cactus/llms.txt

Use this file to discover all available pages before exploring further.

​

Custom Models

​

Converting Models with LoRA

​

Basic Conversion

# Convert from local LoRA adapter
cactus convert Qwen/Qwen3-0.6B ./my-qwen3-0.6b --lora ./my-lora-adapter

# Convert from HuggingFace Hub
cactus convert google/gemma-3-270m-it ./my-gemma3 --lora username/my-lora-adapter

# With specific quantization
cactus convert LiquidAI/LFM2.5-1.2B-Instruct ./my-lfm --lora ./adapters/my-lora --precision INT8

​

Command Options

​

Model Format Requirements

​

Supported Base Models

• Gemma 3: google/gemma-3-270m-it, google/gemma-3-1b-it
• Qwen 3: Qwen/Qwen3-0.6B, Qwen/Qwen3-1.7B
• LFM 2/2.5: LiquidAI/LFM2-350M, LiquidAI/LFM2.5-1.2B-Instruct, LiquidAI/LFM2-8B-A1B
• SmolLM 2: Coming soon

​

LoRA Adapter Format

• Be trained with Unsloth, PEFT, or compatible LoRA libraries
• Target standard transformer modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Use rank (r) between 8-64 (recommended: 16-32 for mobile)
• Include a valid adapter_config.json file

​

Weight Quantization Options

​

INT4 (Default)

cactus convert Qwen/Qwen3-0.6B ./my-model --lora ./adapter --precision INT4

• Benefits: ~50% memory reduction vs INT8, fastest inference
• Trade-offs: Minimal quality loss with task-specific fine-tunes
• Best for: Production deployment on budget devices

​

INT8

cactus convert Qwen/Qwen3-0.6B ./my-model --lora ./adapter --precision INT8

• Benefits: Better quality retention than INT4
• Trade-offs: 2x memory usage vs INT4
• Best for: Quality-critical applications, mid-range devices
• Performance: 60-70 tok/s on iPhone 17 Pro, 13-18 tok/s on Pixel 6a

​

FP16

cactus convert Qwen/Qwen3-0.6B ./my-model --lora ./adapter --precision FP16

• Benefits: Full precision, no quality loss
• Trade-offs: 4x memory usage vs INT4, slower inference
• Best for: Development, benchmarking, high-end devices only

​

Testing Converted Models

​

Interactive Testing (Mac/Linux)

# Test your converted model locally
cactus run ./my-qwen3-0.6b

​

Benchmark Mode

# Run performance benchmarks
cactus test --model ./my-qwen3-0.6b --benchmark

• Prefill tokens per second (TPS)
• Decode tokens per second
• Time to first token
• RAM usage
• Model confidence scores

​

Testing on iOS Device

# Build and test on connected iPhone
cactus build --apple
cactus test --model ./my-model --ios

• Xcode installed
• iPhone connected via USB
• Device unlocked and trusted

​

Testing on Android Device

# Build and test on connected Android phone
cactus build --android
cactus test --model ./my-model --android

• Android SDK/NDK installed
• USB debugging enabled
• Device connected via ADB

​

Example: End-to-End Custom Model

​

1. Train LoRA Adapter (Colab/GPU)

from unsloth import FastLanguageModel
from trl import SFTTrainer

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-0.6B",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    use_gradient_checkpointing="unsloth",
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    max_seq_length=2048,
)

trainer.train()
model.save_pretrained("./task-specific-adapter")

​

2. Convert for Cactus

cactus convert Qwen/Qwen3-0.6B ./qwen3-task-specific \
  --lora ./task-specific-adapter \
  --precision INT8

​

3. Test Locally

cactus run ./qwen3-task-specific

​

4. Deploy to iOS

cactus build --apple
# Copy model folder to Xcode project
# Link cactus-ios.xcframework

let modelPath = Bundle.main.path(forResource: "qwen3-task-specific", ofType: nil)!
let model = cactus_init(modelPath, nil, false)

let messages = "[{\"role\":\"user\",\"content\":\"Your query\"}]"
var response = [CChar](repeating: 0, count: 4096)
cactus_complete(model, messages, &response, response.count, nil, nil, nil, nil)

print(String(cString: response))
cactus_destroy(model)

​

Performance Expectations

​

INT8 Qwen3-0.6B (Custom Fine-Tune)

​

INT8 Gemma3-270m (Task-Specific)

​

Troubleshooting

​

Conversion Fails

Error: Base model architecture mismatch

​

Poor Quality After Conversion

1. Try INT8 instead of INT4: --precision INT8
2. Verify adapter trained properly (check validation loss)
3. Test with different prompts and temperatures

​

Model Too Large for Device

1. Use INT4 quantization: --precision INT4
2. Try a smaller base model (e.g., Gemma3-270m instead of Qwen3-1.7B)
3. Reduce context window at runtime (see Performance Tuning)

​

See Also

• Fine-Tuning Guide — Training LoRA adapters with Unsloth
• Performance Tuning — Optimize runtime performance
• Compatibility — Weight versioning and breaking changes