Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/software-mansion/react-native-executorch/llms.txt

Use this file to discover all available pages before exploring further.

Object detection identifies and localizes multiple objects within images, providing bounding boxes, class labels, and confidence scores for each detection. React Native ExecuTorch supports real-time detection with VisionCamera integration.

​
Quick Start

import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';

function ObjectDetector() {
  const { isReady, forward } = useObjectDetection({
    model: SSDLITE_320_MOBILENET_V3_LARGE,
  });

  const detectObjects = async (imageUri: string) => {
    const detections = await forward(imageUri, 0.7);
    // [{ bbox: { x1, y1, x2, y2 }, label: 'PERSON', score: 0.95 }, ...]
  };

  return <Button title="Detect" onPress={() => detectObjects(imageUri)} />;
}

​
Hook API

​
useObjectDetection<C>(props)

Manages an object detection model instance with type-safe labels.

​
Type Parameters

​
C
ObjectDetectionModelSources
Model configuration type that determines available labels

​
Parameters

​
model
C
required
Model configuration object
​
preventLoad
boolean
default:"false"
Prevent automatic model loading

​
Returns

​
error
RnExecutorchError | null
Error object if loading or inference fails
​
isReady
boolean
Whether the model is loaded and ready
​
isGenerating
boolean
Whether the model is currently processing
​
downloadProgress
number
Download progress (0-1)
​
forward
(input: string | PixelData, detectionThreshold?: number) => Promise<Detection[]>
Detect objects in an image. Returns array of detections with bboxes, labels, and scores.Parameters:
  • input: Image URI or PixelData object
  • detectionThreshold: Minimum confidence score (0-1), default 0.7
​
runOnFrame
((frame: Frame, detectionThreshold: number) => Detection[]) | null
Synchronous worklet function for VisionCamera frame processing. Available after model loads.

​
Available Models

​
SSDLITE_320_MOBILENET_V3_LARGE

Lightweight SSD detector optimized for mobile. 
import { SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';

const detector = useObjectDetection({
  model: SSDLITE_320_MOBILENET_V3_LARGE,
});
Specifications:
  • Architecture: SSDLite with MobileNetV3-Large backbone
  • Classes: 80 COCO classes (person, car, dog, etc.)
  • Input Size: 320x320
  • Inference Time: ~100-150ms
  • mAP: ~22%

​
RF_DETR_NANO

Real-time DETR-based detector with improved accuracy. 
import { RF_DETR_NANO } from 'react-native-executorch';

const detector = useObjectDetection({
  model: RF_DETR_NANO,
});
Specifications:
  • Architecture: Real-time DETR Nano
  • Classes: 80 COCO classes
  • Input Size: 640x640
  • Inference Time: ~150-200ms
  • mAP: ~35%

​
Detection Types

​
Detection Interface

interface Detection<L = CocoLabel> {
  bbox: Bbox;
  label: keyof L;
  score: number;
}

interface Bbox {
  x1: number; // Bottom-left x
  y1: number; // Bottom-left y
  x2: number; // Top-right x
  y2: number; // Top-right y
}

​
COCO Labels

All built-in models detect 80 COCO classes: 
import { CocoLabel } from 'react-native-executorch';

// Available labels:
CocoLabel.PERSON
CocoLabel.CAR
CocoLabel.DOG
CocoLabel.CAT
CocoLabel.BICYCLE
CocoLabel.BOTTLE
// ... and 74 more

​
Complete Example

import React, { useState } from 'react';
import { View, Image, StyleSheet, TouchableOpacity, Text } from 'react-native';
import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE, Detection } from 'react-native-executorch';
import { launchImageLibrary } from 'react-native-image-picker';
import Svg, { Rect, Text as SvgText } from 'react-native-svg';

function ObjectDetectionDemo() {
  const [imageUri, setImageUri] = useState<string | null>(null);
  const [imageDimensions, setImageDimensions] = useState({ width: 0, height: 0 });
  const [detections, setDetections] = useState<Detection[]>([]);

  const { isReady, isGenerating, error, forward } = useObjectDetection({
    model: SSDLITE_320_MOBILENET_V3_LARGE,
  });

  const selectAndDetect = async () => {
    const result = await launchImageLibrary({ mediaType: 'photo' });
    
    if (result.assets && result.assets[0]) {
      const asset = result.assets[0];
      setImageUri(asset.uri!);
      setImageDimensions({
        width: asset.width || 300,
        height: asset.height || 300,
      });

      try {
        const dets = await forward(asset.uri!, 0.5);
        setDetections(dets);
      } catch (err) {
        console.error('Detection failed:', err);
      }
    }
  };

  const renderBoundingBoxes = () => {
    if (!imageUri || detections.length === 0) return null;

    return (
      <Svg
        style={StyleSheet.absoluteFill}
        viewBox={`0 0 ${imageDimensions.width} ${imageDimensions.height}`}
      >
        {detections.map((detection, index) => {
          const { bbox, label, score } = detection;
          const width = bbox.x2 - bbox.x1;
          const height = bbox.y2 - bbox.y1;

          return (
            <React.Fragment key={index}>
              <Rect
                x={bbox.x1}
                y={bbox.y1}
                width={width}
                height={height}
                stroke="#00FF00"
                strokeWidth="3"
                fill="none"
              />
              <SvgText
                x={bbox.x1 + 5}
                y={bbox.y1 + 20}
                fill="#00FF00"
                fontSize="16"
                fontWeight="bold"
              >
                {label} {(score * 100).toFixed(0)}%
              </SvgText>
            </React.Fragment>
          );
        })}
      </Svg>
    );
  };

  if (error) {
    return <Text>Error: {error.message}</Text>;
  }

  if (!isReady) {
    return <Text>Loading model...</Text>;
  }

  return (
    <View style={styles.container}>
      <TouchableOpacity
        style={styles.button}
        onPress={selectAndDetect}
        disabled={isGenerating}
      >
        <Text style={styles.buttonText}>
          {isGenerating ? 'Detecting...' : 'Select & Detect Objects'}
        </Text>
      </TouchableOpacity>

      {imageUri && (
        <View style={styles.imageContainer}>
          <Image
            source={{ uri: imageUri }}
            style={[styles.image, { aspectRatio: imageDimensions.width / imageDimensions.height }]}
          />
          {renderBoundingBoxes()}
        </View>
      )}

      {detections.length > 0 && (
        <View style={styles.results}>
          <Text style={styles.title}>Detected {detections.length} objects:</Text>
          {detections.map((det, idx) => (
            <Text key={idx} style={styles.detection}>
              {det.label}: {(det.score * 100).toFixed(1)}%
            </Text>
          ))}
        </View>
      )}
    </View>
  );
}

const styles = StyleSheet.create({
  container: { flex: 1, padding: 20 },
  button: { backgroundColor: '#007AFF', padding: 15, borderRadius: 8 },
  buttonText: { color: 'white', fontSize: 16, textAlign: 'center' },
  imageContainer: { marginVertical: 20, position: 'relative' },
  image: { width: '100%', borderRadius: 8 },
  results: { padding: 15, backgroundColor: '#f5f5f5', borderRadius: 8 },
  title: { fontSize: 16, fontWeight: 'bold', marginBottom: 10 },
  detection: { fontSize: 14, paddingVertical: 4 },
});

export default ObjectDetectionDemo;

​
Real-Time Camera Detection

Integrate with VisionCamera for live object detection: 
import { Camera, useFrameProcessor } from 'react-native-vision-camera';
import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';
import { useSharedValue } from 'react-native-reanimated';

function LiveObjectDetection() {
  const detections = useSharedValue<Detection[]>([]);
  
  const { runOnFrame, isReady } = useObjectDetection({
    model: SSDLITE_320_MOBILENET_V3_LARGE,
  });

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';
      if (!runOnFrame) return;
      
      const detected = runOnFrame(frame, 0.6);
      detections.value = detected;
    },
    [runOnFrame]
  );

  if (!isReady) return <Text>Loading...</Text>;

  return (
    <>
      <Camera
        style={StyleSheet.absoluteFill}
        device={device}
        isActive={true}
        frameProcessor={frameProcessor}
      />
      <BoundingBoxOverlay detections={detections} />
    </>
  );
}

​
Use Cases

​
Object Counting

const countObjects = async (imageUri: string, targetLabel: string) => {
  const detections = await forward(imageUri, 0.6);
  return detections.filter(d => d.label === targetLabel).length;
};

// Count people in an image
const peopleCount = await countObjects(imageUri, 'PERSON');

​
Region of Interest

Detect objects only in specific image regions: 
const detectInRegion = async (
  imageUri: string,
  region: { x: number; y: number; width: number; height: number }
) => {
  const allDetections = await forward(imageUri, 0.5);
  
  return allDetections.filter(det => {
    const centerX = (det.bbox.x1 + det.bbox.x2) / 2;
    const centerY = (det.bbox.y1 + det.bbox.y2) / 2;
    
    return (
      centerX >= region.x &&
      centerX <= region.x + region.width &&
      centerY >= region.y &&
      centerY <= region.y + region.height
    );
  });
};

​
Object Tracking

Track objects across frames using IoU: 
const calculateIoU = (box1: Bbox, box2: Bbox): number => {
  const x1 = Math.max(box1.x1, box2.x1);
  const y1 = Math.max(box1.y1, box2.y1);
  const x2 = Math.min(box1.x2, box2.x2);
  const y2 = Math.min(box1.y2, box2.y2);
  
  const intersection = Math.max(0, x2 - x1) * Math.max(0, y2 - y1);
  const area1 = (box1.x2 - box1.x1) * (box1.y2 - box1.y1);
  const area2 = (box2.x2 - box2.x1) * (box2.y2 - box2.y1);
  const union = area1 + area2 - intersection;
  
  return intersection / union;
};

const trackObjects = (
  prevDetections: Detection[],
  currDetections: Detection[],
  iouThreshold: number = 0.5
) => {
  return currDetections.map(curr => {
    const match = prevDetections.find(prev => 
      prev.label === curr.label && 
      calculateIoU(prev.bbox, curr.bbox) > iouThreshold
    );
    
    return { ...curr, tracked: !!match };
  });
};

​
Performance Tips

​
Threshold Tuning

Adjust detection threshold based on use case: 
// High precision (fewer false positives)
const detections = await forward(imageUri, 0.8);

// High recall (catch more objects)
const detections = await forward(imageUri, 0.3);

// Balanced
const detections = await forward(imageUri, 0.5);

​
Model Selection

  • SSDLite: Faster, lower accuracy, good for real-time
  • RF-DETR Nano: Better accuracy, slightly slower

​
Frame Skipping

For camera processing, skip frames to reduce CPU load: 
const frameProcessor = useFrameProcessor(
  (frame) => {
    'worklet';
    if (!runOnFrame || frame.timestamp % 3 !== 0) return; // Process every 3rd frame
    
    const detections = runOnFrame(frame, 0.7);
    // ...
  },
  [runOnFrame]
);

​
Type Reference

import { ResourceSource, PixelData, Frame, LabelEnum } from 'react-native-executorch';

type ObjectDetectionModelSources =
  | { modelName: 'ssdlite-320-mobilenet-v3-large'; modelSource: ResourceSource }
  | { modelName: 'rf-detr-nano'; modelSource: ResourceSource };

interface ObjectDetectionProps<C extends ObjectDetectionModelSources> {
  model: C;
  preventLoad?: boolean;
}

interface ObjectDetectionType<L extends LabelEnum> {
  error: RnExecutorchError | null;
  isReady: boolean;
  isGenerating: boolean;
  downloadProgress: number;
  forward: (input: string | PixelData, detectionThreshold?: number) => Promise<Detection<L>[]>;
  runOnFrame: ((frame: Frame, detectionThreshold: number) => Detection<L>[]) | null;
}

Build docs developers (and LLMs) love

Get started for freeTalk to us