AI & Machine Learning Integration

Overview

PokéDex Fleek combines computer vision and machine learning to create an intelligent Pokémon detection system. The app uses TensorFlow.js for object classification and react-native-vision-camera for high-performance frame processing.

Unlike traditional Pokédex apps that rely on manual search, PokéDex Fleek acts as a field research tool that analyzes real-world objects through your camera.

Technology Stack

Component	Technology	Purpose
ML Framework	TensorFlow.js	Object detection and classification
Camera Library	react-native-vision-camera V4	Native camera access and frame capture
Frame Processing	react-native-worklets-core	60fps real-time processing
Color Analysis	react-native-image-colors	Dominant color extraction
Model Format	TensorFlow Lite	Optimized mobile inference

TensorFlow.js Role

Vision Pipeline

The AI detection system follows a multi-stage pipeline:

Current Implementation

The project is structured to support TensorFlow.js integration, with the foundation already in place:

Camera Setup

High-performance camera interface using Vision Camera V4 with frame processor support

Color Detection

Dominant color extraction using react-native-image-colors for instant analysis

Worklets Integration

Frame processors run on separate threads for 60fps performance

ML Roadmap

TensorFlow Lite models planned for object classification (Person, Animal, Plant, Object)

Camera Integration

Vision Camera Setup

The ScannerScreen uses react-native-vision-camera V4 for native camera access:

src/features/scanner/ScannerScreen.js

import React, { useEffect, useState, useRef } from 'react';
import { Camera, useCameraDevice, useFrameProcessor, useCodeScanner } 
    from 'react-native-vision-camera';
import { useIsFocused } from '@react-navigation/native';
import { runAtJS } from 'react-native-worklets-core';

export const ScannerScreen = ({ navigation }) => {
    const [hasPermission, setHasPermission] = useState(false);
    const [isProcessing, setIsProcessing] = useState(false);
    const isFocused = useIsFocused();
    const device = useCameraDevice('back');
    const cameraRef = useRef(null);

    // Request camera permissions
    useEffect(() => {
        (async () => {
            const status = await Camera.requestCameraPermission();
            setHasPermission(status === 'granted');
        })();
    }, []);

    // Camera component
    return (
        <Camera
            ref={cameraRef}
            device={device}
            isActive={isFocused}
            style={StyleSheet.absoluteFill}
            photo={true}
        />
    );
};

Key Features

Permission Handling

The app requests camera permissions on mount and handles denied/blocked states:

const status = await Camera.requestCameraPermission();
if (status !== 'granted') {
    // Show settings prompt
    Linking.openSettings();
}

Focus Management

Camera only activates when the screen is focused, saving battery:

const isFocused = useIsFocused();

<Camera
    isActive={isFocused}
    // Only processes frames when screen is visible
/>

Photo Capture

High-quality photo capture for color analysis:

const photo = await cameraRef.current.takePhoto({
    flash: 'off',
    skipMetadata: true  // Faster processing
});

Frame Processing Architecture

Worklets for Real-Time Analysis

Worklets are JavaScript functions that run on a separate thread, enabling 60fps frame processing without blocking the UI:

import { Worklets } from 'react-native-worklets-core';
import { runAtJS } from 'react-native-worklets-core';

const useFrameProcessor = useFrameProcessor((frame) => {
    'worklet';
    
    // This runs on a separate thread at 60fps
    const pixels = frame.toArrayBuffer();
    
    // Run TensorFlow inference
    const predictions = model.predict(pixels);
    
    // Send results back to JS thread
    runAtJS(handleDetection)(predictions);
});

Frame processors must be marked with 'worklet' directive and can only call other worklet functions. Use runAtJS to communicate with the main JavaScript thread.

Current Frame Capture Flow

While full frame processing is planned, the current implementation uses QR code scanning and manual photo capture:

src/features/scanner/ScannerScreen.js

const codeScanner = useCodeScanner({
    codeTypes: ['qr', 'ean-13'],
    onCodeScanned: async (codes) => {
        if (isProcessing || codes.length === 0) return;

        setIsProcessing(true);

        // Capture photo when QR code detected
        const photo = await cameraRef.current.takePhoto({
            flash: 'off',
            skipMetadata: true
        });

        // Extract dominant color
        const result = await ImageColors.getColors(photo.path, { 
            fallback: '#000000' 
        });
        const detectedHex = result.platform === 'android' 
            ? result.dominant 
            : result.background;
        
        // Convert to hue and match Pokémon
        const hue = hexToHue(detectedHex);
        const match = Object.values(POKEMON_DB).find(p =>
            hue >= p.colorRange.hueMin && hue <= p.colorRange.hueMax
        );

        if (match) {
            navigation.navigate('Details', { pokemonId: match.id });
        }
    }
});

Object Detection Strategy

Classification Categories

The planned TensorFlow model will classify objects into four categories:

Person

Human subjects detected via pose estimation or facial recognition

Animal

Living creatures with distinct shapes and movement patterns

Plant

Vegetation, flowers, and organic structures

Object

Inanimate items, tools, and manufactured goods

Detection + Color Matching

The AI system will combine object classification with color analysis for more accurate results:

// Pseudo-code for future implementation
const detectAndMatch = async (frame) => {
    // 1. Classify object type
    const classification = await tensorflowModel.classify(frame);
    // Output: { type: 'animal', confidence: 0.89 }
    
    // 2. Extract dominant color
    const dominantColor = await extractColor(frame);
    // Output: { hex: '#FF5733', hue: 9 }
    
    // 3. Filter Pokémon by category
    const candidates = POKEMON_DB.filter(p => 
        p.category === classification.type
    );
    
    // 4. Find color match within category
    const match = candidates.find(p =>
        dominantColor.hue >= p.colorRange.hueMin && 
        dominantColor.hue <= p.colorRange.hueMax
    );
    
    return match;
};

This two-stage approach (classification → color matching) dramatically reduces false positives. For example, a red apple won’t match Charmander because “Plant” ≠ “Fire-type Pokémon”.

Performance Considerations

Mobile Optimization

Running ML models on mobile devices requires careful optimization:

TensorFlow Lite Conversion

Full TensorFlow models are converted to TensorFlow Lite format, reducing size by 75% and inference time by 3-4x

Quantization

Model weights are quantized from 32-bit floats to 8-bit integers, trading minimal accuracy for massive speed gains

Thread Isolation

Frame processing runs on a dedicated thread via Worklets, keeping the UI at 60fps

Throttling

Inference runs every 5-10 frames instead of every frame, balancing responsiveness with battery life

Benchmarks (Projected)

Metric	Target	Notes
Inference Time	< 50ms	MobileNet V2 on modern devices
Frame Rate	30-60 fps	Depends on device GPU
Battery Impact	< 5% per min	With GPU acceleration
Model Size	< 10MB	TFLite quantized model

Future Roadmap

The ML integration is planned in phases:

Phase 1: Foundation (Current)

✅ Camera integration with Vision Camera V4
✅ Color extraction pipeline
✅ Worklets setup for frame processing
✅ Basic Pokémon matching algorithm

Phase 2: ML Integration (Planned)

⏳ TensorFlow Lite model integration
⏳ Object classification (Person/Animal/Plant/Object)
⏳ Frame processor implementation
⏳ Real-time inference pipeline

Phase 3: Advanced Features (Future)

🔮 Shape/silhouette detection
🔮 Multiple object tracking
🔮 AR overlay for detected Pokémon
🔮 Custom model training for Gen 2+ Pokémon

The roadmap is defined in the source README under “Roadmap de Desarrollo” and is actively being worked on by the development team.

Code Example: Manual Scan

Here’s the current implementation for manual photo capture and analysis:

src/features/scanner/ScannerScreen.js

const handleManualScan = async () => {
    if (isProcessing) return;
    Vibration.vibrate(70);  // Haptic feedback
    setIsProcessing(true);

    try {
        // Capture photo
        const photo = await cameraRef.current.takePhoto({ 
            flash: 'off', 
            skipMetadata: true 
        });
        
        // Extract dominant color
        const result = await ImageColors.getColors(photo.path, { 
            fallback: '#000000' 
        });
        const detectedHex = result.platform === 'android' 
            ? result.dominant 
            : result.background;
        const hue = hexToHue(detectedHex);

        // Find matching Pokémon
        const potentialMatches = Object.values(POKEMON_DB).filter(p =>
            hue >= p.colorRange.hueMin && hue <= p.colorRange.hueMax
        );

        if (potentialMatches.length > 0) {
            const randomIndex = Math.floor(Math.random() * potentialMatches.length);
            const match = potentialMatches[randomIndex];

            setDetectedPokemon(match);
            Vibration.vibrate([0, 100, 50, 100]);  // Success pattern

            setTimeout(() => {
                navigation.navigate('Details', { 
                    pokemonId: match.id, 
                    themeColor: match.uiTheme 
                });
                setIsProcessing(false);
                setDetectedPokemon(null);
            }, 500);
        } else {
            alert("No se detectó ADN Pokémon conocido en este objeto.");
            setIsProcessing(false);
        }
    } catch (error) {
        console.error(error);
        Vibration.vibrate(400);  // Error vibration
        setIsProcessing(false);
    }
};

User Experience Details

Haptic Feedback: Vibration confirms scan start and result
Loading State: Button shows ActivityIndicator while processing
Error Handling: Graceful fallback with user-friendly messages
Navigation: Auto-navigate to detail screen on successful match

Integration with Color Matching

The AI detection system enhances the color matching algorithm:

┌─────────────────┐
│  Camera Frame   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  TensorFlow.js  │ ← Object classification
│  Classification │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Color Extraction│ ← Dominant color via hexToHue
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ ColorService    │ ← Euclidean distance matching
│ findClosest()   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Pokémon Match  │ ← Result + navigation
└─────────────────┘

Resources & References

TensorFlow.js

Official documentation for TensorFlow.js

Vision Camera

react-native-vision-camera V4 docs

Worklets

react-native-worklets-core for frame processing

TFLite

TensorFlow Lite for mobile optimization

Next Steps

Color Matching

Learn how Euclidean distance powers Pokémon identification

Project Architecture

Explore the feature-based folder structure and navigation

Get Started

Core Concepts

Features

Development

Documentation Index

​Overview

​Technology Stack

​TensorFlow.js Role

​Vision Pipeline

​Current Implementation

Camera Setup

Color Detection

Worklets Integration

ML Roadmap

​Camera Integration

​Vision Camera Setup

​Key Features

​Frame Processing Architecture

​Worklets for Real-Time Analysis

​Current Frame Capture Flow

​Object Detection Strategy

​Classification Categories

Person

Animal

Plant

Object

​Detection + Color Matching

​Performance Considerations

​Mobile Optimization

​Benchmarks (Projected)

​Future Roadmap

​Phase 1: Foundation (Current)

​Phase 2: ML Integration (Planned)

​Phase 3: Advanced Features (Future)

​Code Example: Manual Scan

​User Experience Details

​Integration with Color Matching

​Resources & References

TensorFlow.js

Vision Camera

Worklets

TFLite

​Next Steps

Color Matching

Project Architecture

Build docs developers (and LLMs) love

Overview

Technology Stack

TensorFlow.js Role

Vision Pipeline

Current Implementation

Camera Integration

Vision Camera Setup

Key Features

Frame Processing Architecture

Worklets for Real-Time Analysis

Current Frame Capture Flow

Object Detection Strategy

Classification Categories

Detection + Color Matching

Performance Considerations

Mobile Optimization

Benchmarks (Projected)

Future Roadmap

Phase 1: Foundation (Current)

Phase 2: ML Integration (Planned)

Phase 3: Advanced Features (Future)

Code Example: Manual Scan

User Experience Details

Integration with Color Matching

Resources & References

Next Steps