Architecture Overview

​

System Architecture

┌──────────────────────────────────────────────────────────────────┐
│                       React Native UI Layer                       │
│            (Brutalist Design System - TypeScript/TSX)            │
├──────────────────────────────────────────────────────────────────┤
│                  TypeScript Services Layer                        │
│                                                                   │
│   Core Services (background-safe singletons):                    │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐│
│   │   llmService    │  │  whisperService │  │ hardware        ││
│   │  (llama.rn)     │  │  (whisper.rn)   │  │  (RAM/CPU info) ││
│   └─────────────────┘  └─────────────────┘  └─────────────────┘│
│                                                                   │
│   Orchestration Services (lifecycle-independent):                │
│   ┌───────────────────────┐  ┌───────────────────────┐          │
│   │  generationService    │  │ imageGenerationService│          │
│   │  (text, background)   │  │  (images, background) │          │
│   └───────────────────────┘  └───────────────────────┘          │
│                                                                   │
│   Management Services:                                           │
│   ┌────────────────────┐  ┌────────────────────┐                │
│   │activeModelService  │  │   modelManager     │                │
│   │(singleton, mem mgmt)│  │(download, storage) │                │
│   └────────────────────┘  └────────────────────┘                │
├──────────────────────────────────────────────────────────────────┤
│                 Native Module Bridge (JNI / ObjC)                 │
├──────────────────────────────────────────────────────────────────┤
│   Native Implementations:                                         │
│                                                                   │
│   Cross-platform:                                                 │
│   ┌──────────────┐  ┌──────────────┐                             │
│   │   llama.rn   │  │  whisper.rn  │                             │
│   │ (C++ native) │  │ (C++ native) │                             │
│   └──────────────┘  └──────────────┘                             │
│                                                                   │
│   Android:                           iOS:                         │
│   ┌──────────┐ ┌───────────────┐    ┌──────────────────────┐    │
│   │local-dream│ │DownloadManager│    │CoreMLDiffusionModule │    │
│   │(C++/MNN)  │ │   (Kotlin)    │    │(StableDiffusionPipe) │    │
│   └──────────┘ └───────────────┘    └──────────────────────┘    │
├──────────────────────────────────────────────────────────────────┤
│   Hardware Acceleration:                                          │
│   Android:                           iOS:                         │
│   ┌──────────────────┐              ┌──────────────────┐         │
│   │OpenCL (Adreno GPU)│              │ ANE (Neural Engine)│         │
│   │  Text LLMs only   │              │  Image gen + LLMs │         │
│   ├──────────────────┤              ├──────────────────┤         │
│   │   QNN (NPU)      │              │  Metal (GPU)     │         │
│   │  Image gen only  │              │  LLM inference   │         │
│   └──────────────────┘              └──────────────────┘         │
└──────────────────────────────────────────────────────────────────┘

​

Layer Breakdown

​

UI Layer (React Native)

• ChatInput — Message composition with attachment badges (src/components/ChatInput.tsx)
• ModelCard — Model display with actions (src/components/ModelCard.tsx)
• AnimatedPressable — Spring scale feedback + haptics (src/components/AnimatedPressable.tsx)
• AppSheet — Custom swipe-dismissible bottom sheets (src/components/AppSheet.tsx)

​

Services Layer (TypeScript)

​

Core Services

• Model loading with automatic context scaling based on device RAM
• Streaming token generation with 50ms batched UI updates
• Vision model support via mmproj (multimodal projector) files
• Tool calling detection from jinja chat templates
• KV cache management with quantization (f16/q8_0/q4_0)

​

Orchestration Services

• Maintains generation state independently of React component lifecycle
• Token batching: collects tokens and flushes to UI every 50ms
• Message queue: non-blocking input during active generation
• Tool loop integration (max 3 iterations, 5 tool calls)

class GenerationService {
  private state: GenerationState = { isGenerating: false, ... };
  private listeners: Set<GenerationListener> = new Set();

  subscribe(listener: GenerationListener): () => void {
    this.listeners.add(listener);
    listener(this.getState()); // Immediate state delivery
    return () => this.listeners.delete(listener);
  }

  private notifyListeners(): void {
    const state = this.getState();
    this.listeners.forEach(listener => listener(state));
  }
}

import { useEffect, useState } from 'react';
import { generationService } from '../services/generationService';

function ChatScreen() {
  const [genState, setGenState] = useState(generationService.getState());

  useEffect(() => {
    // Subscribe on mount, get current state immediately
    const unsubscribe = generationService.subscribe(setGenState);
    return unsubscribe; // Cleanup on unmount
  }, []);

  // Generation continues even if user navigates away
}

• Continues generation when screens unmount
• Real-time step progress (1-50 steps)
• Preview images every N steps
• Optional LLM-based prompt enhancement

​

Management Services

• Guards against concurrent loads with promise deduplication
• Pre-load memory checks (60% RAM budget enforcement)
• Automatic model unload when switching models
• Synchronization with native state on app resume

async loadTextModel(modelId: string, timeoutMs: number = 120000): Promise<void> {
  // Already loaded natively — ensure store reflects it
  if (this.loadedTextModelId === modelId && llmService.isModelLoaded()) {
    const store = useAppStore.getState();
    if (store.activeModelId !== modelId) { 
      store.setActiveModelId(modelId); 
    }
    return;
  }

  // Wait for in-flight load to complete
  if (this.textLoadPromise !== null) {
    await this.textLoadPromise;
    if (this.loadedTextModelId === modelId) {
      const store = useAppStore.getState();
      if (store.activeModelId !== modelId) { 
        store.setActiveModelId(modelId); 
      }
      return;
    }
  }

  // Proceed with load...
  this.loadingState.text = true;
  this.notifyListeners();
  this.textLoadPromise = doLoadTextModel({ /* ... */ });
  await this.textLoadPromise;
}

​

Native Bridge Layer

​

Native Module Layer

​

Cross-Platform

• Android: OpenCL (Adreno GPUs), NEON/i8mm SIMD
• iOS: Metal GPU, Neural Engine for vision models
• Multimodal support via mmproj (CLIP vision encoder)

​

Android-Only

• MNN backend: CPU inference (all ARM64 devices)
• QNN backend: Qualcomm NPU (Snapdragon 8 Gen 1+)
• Subprocess architecture: spawns HTTP server on localhost:18081
• Automatic backend selection with CPU fallback

• Background downloads with system notifications
• Progress polling (500ms intervals)
• Persistent download tracking via SharedPreferences
• Cleanup of completed/stale downloads

​

iOS-Only

• Neural Engine (ANE) + CPU compute units
• DPM-Solver scheduler for faster convergence
• Supports both SD 1.5/2.1 and SDXL models
• Palettized (6-bit) and full-precision (fp16) models

​

Technology Stack

​

Data Flow

​

Text Generation Flow

1. User Input → ChatScreen collects message + attachments
2. Service Call → generationService.generateResponse() or .generateWithTools()
3. Context Management → llmService passes all messages to llama.rn (no JS truncation)
4. Native Inference → llama.cpp streams tokens via callback
5. Token Batching → generationService buffers tokens, flushes every 50ms
6. UI Update → chatStore updates streaming message
7. Completion → Finalize message with metadata (tok/s, TTFT, generation time)

​

Vision Inference Flow

1. Image Attachment → User attaches photo from camera/library
2. mmproj Check → llmService verifies multimodal initialized
3. OAI Message Format → Convert to OpenAI-compatible message with image URIs
4. CLIP Encoding → Native CLIP processes image to embeddings
5. LLM Processing → llama.cpp merges text + vision embeddings
6. Response → Stream tokens as normal text generation

​

Image Generation Flow (Android)

1. Prompt Input → User sends text or enables image mode toggle
2. Intent Detection → Pattern matching or LLM-based classification
3. Prompt Enhancement → Optional: LLM expands prompt (“a dog” → detailed 75-word description)
4. Model Load → LocalDreamModule starts subprocess server (MNN or QNN)
5. HTTP Request → TypeScript sends POST to localhost:18081/generate
6. SSE Stream → Server sends progress events (step/totalSteps) + preview images
7. RGB → PNG → Native code decodes base64 RGB, converts to PNG
8. Gallery Save → Image stored in app files, added to gallery

​

Image Generation Flow (iOS)

1. Prompt Input → Same as Android
2. Pipeline Load → CoreMLDiffusionModule loads StableDiffusionPipeline
3. Generation → Native Swift calls pipeline.generateImages() with progress callback
4. ANE Acceleration → Neural Engine processes UNet denoising steps
5. PNG Save → CGImage converted to PNG, stored in documents
6. Gallery Save → Same as Android

​

Design Patterns

• Singleton services
• Background-safe orchestration
• Memory-first loading strategy
• Combined asset tracking (vision models + mmproj)
• State cleanup patterns