Face Recognition

Overview

After detecting a face, Iris uses face recognition to determine identity by:

Extracting a unique embedding (128-dimensional vector) from the face
Comparing embeddings using cosine similarity
Matching against a threshold to determine if two faces belong to the same person

Model: face_recognition_sface_2021dec.onnx (December 2021 version)

SFace Model

SFace is a lightweight face recognition network trained on millions of facial images to learn discriminative features.

Key Characteristics

Input Size

112x112 pixels (aligned face crop)

Output

128-dimensional feature vector

Architecture

Convolutional neural network (CNN)

Training Data

MS-Celeb-1M, VGGFace2, Asian-Celeb

Why 128 Dimensions?

The embedding size balances:

Discriminative power: Enough dimensions to distinguish billions of faces
Computation speed: Small enough for fast cosine similarity calculations
Memory efficiency: Only 512 bytes per face (128 floats × 4 bytes)

Initialization

The recognizer is created when FaceEngine starts (face.rs:14-16):

face.rs

let recognizer = objdetect::FaceRecognizerSF::create(
    "face_recognition_sface_2021dec.onnx", 
    "",  // config file (empty = use defaults)
    0,   // backend_id (0 = default)
    0    // target_id (0 = CPU)
)?;

The recognizer runs on CPU by default. For GPU acceleration, set target_id to an appropriate value based on your OpenCV build.

Embedding Extraction

The process of converting a face image into a feature vector happens in face.rs:32-36:

Step 1: Face Alignment

face.rs

let mut aligned = Mat::default();
rec.align_crop(img, &face_data, &mut aligned)?;

Using the facial landmarks from detection (eye positions, nose, mouth), the face is:

Rotated to be upright
Scaled to 112x112 pixels
Cropped to include only the face region

This normalization ensures consistent input to the neural network.

Step 2: Feature Extraction

face.rs

let mut feature = Mat::default();
rec.feature(&aligned, &mut feature)?;
return Ok(Some(feature.clone()));

The aligned face is fed through the SFace CNN, producing a 128-dimensional vector where:

Each dimension captures abstract facial characteristics
Similar faces produce vectors that point in similar directions in 128D space
The vector is L2-normalized (unit length), enabling cosine similarity comparison

Embedding Properties

What Information Is Encoded?

The 128D embedding captures: ✅ Identity-specific features: Eye shape, nose structure, face geometry
✅ Unique patterns: Facial proportions, distinctive characteristics
✅ Invariant representations: Robust to lighting, expression, minor aging ❌ Not stored: Skin color, gender, exact age (these vary too much)
❌ Not reversible: Cannot reconstruct the original image from embedding

Embeddings are NOT encryption. They are feature representations optimized for comparison, not privacy.

Face Matching Algorithm

The comparison logic is in main.rs:103-111:

main.rs

if let Ok(Some(p_emb)) = get_embedding(&p_img, det, rec) {
    if let Ok(score) = rec.match_(&t_emb, &p_emb, objdetect::FaceRecognizerSF_DisType::FR_COSINE as i32) {
        if score > 0.363 {
            results.push(MatchResult {
                name: person.name,
                probability: (score.max(0.0) * 100.0).round(),
            });
        }
    }
}

Cosine Similarity

The match_() function computes:

cosine_similarity = (A · B) / (|A| × |B|)

Where:

A and B are the two face embeddings
· is the dot product
|A| is the vector magnitude (length)

Since embeddings are L2-normalized (unit vectors), this simplifies to:

cosine_similarity = A · B

Why Cosine Similarity?

Cosine similarity measures the angle between vectors, not their distance. This makes it:

Scale-invariant: Only direction matters, not magnitude
Fast to compute: Single dot product operation
Robust: Works well for normalized feature vectors

Alternatives like Euclidean distance are less reliable for high-dimensional spaces due to the “curse of dimensionality.”

The 0.363 Threshold

The critical line: if score > 0.363

What Does This Mean?

Score Range	Interpretation	Action
> 0.500	Very high confidence match	Same person
0.363 - 0.500	Likely match	Same person (threshold)
0.300 - 0.363	Uncertain	Different person
< 0.300	Low similarity	Different person

The 0.363 threshold is OpenCV’s recommended value for the SFace model, balancing false positives and false negatives.

Adjusting the Threshold

Stricter (Lower FPR)
Lenient (Lower FNR)
Default (Balanced)

if score > 0.450 {
    // Fewer false positives
    // But may miss valid matches
}

Use for high-security applications where false acceptance is costly.

if score > 0.300 {
    // More matches found
    // But more false positives
}

Use for search/recommendation systems where missing a match is worse.

if score > 0.363 {
    // Balanced accuracy
    // ~1% FPR, ~1% FNR
}

Recommended for most use cases.

Probability Calculation

The API returns a “probability” to clients (main.rs:107):

main.rs

probability: (score.max(0.0) * 100.0).round(),

This converts the 0-1 cosine similarity into a 0-100 percentage:

Score 0.363 → 36% probability
Score 0.500 → 50% probability
Score 0.800 → 80% probability
Score 0.950 → 95% probability

This is NOT a statistical probability! It’s a normalized similarity score. A “90% match” doesn’t mean 90% confidence—it means very high similarity.

Recognition Accuracy

Benchmark Performance

On standard datasets (LFW, CFP-FP):

True Positive Rate: 99.1% at threshold 0.363
False Positive Rate: 0.9% at threshold 0.363
Equal Error Rate (EER): ~1%

Real-World Factors

Accuracy degrades with: ❌ Aging: Faces change over 5+ years
❌ Occlusion: Masks, sunglasses, hats
❌ Lighting: Extreme shadows or backlighting
❌ Image quality: Low resolution, blur, compression artifacts
❌ Pose variation: Profiles vs. frontal views

For best results, use:

High-resolution images (at least 640px width)
Frontal faces (within ±30° yaw/pitch)
Good lighting (even illumination, avoid harsh shadows)
Minimal occlusion (no sunglasses, masks, or hands covering face)

Multi-Face Comparison

The API compares one target against multiple candidates (main.rs:92-113):

main.rs

let mut results = Vec::new();
for person in payload.people {
    if let Ok(p_img) = download_and_decode(&person.image_url).await {
        let mut guard = state.engine.lock().await;
        // ... extract embedding and compare ...
        if score > 0.363 {
            results.push(MatchResult {
                name: person.name,
                probability: (score.max(0.0) * 100.0).round(),
            });
        }
    }
}

results.sort_by(|a, b| b.probability.partial_cmp(&a.probability).unwrap());
Json(CompareResponse { matches: results })

Key Behaviors

Independent comparisons: Each candidate is evaluated separately
Parallel tolerance: Failed image downloads don’t stop processing
Threshold filtering: Only matches above 0.363 are returned
Sorted results: Best matches appear first in the response

Optimization Techniques

Mutex Locking Strategy

main.rs

let mut guard = state.engine.lock().await;
let (det, rec) = unsafe {
    (
        &mut *(guard.detector.as_raw_mut() as *mut objdetect::FaceDetectorYN),
        &mut *(guard.recognizer.as_raw_mut() as *mut objdetect::FaceRecognizerSF)
    )
};

Why Unsafe Pointers?

OpenCV’s Rust bindings require mutable access to internal state during inference. The unsafe block extracts raw pointers while the Mutex guard is held, ensuring:

Thread safety: Only one request processes recognition at a time
Memory safety: Pointers are valid only within the guard’s lifetime
Performance: Avoids unnecessary cloning of large model weights

This is a safe use of unsafe because the Mutex guarantees exclusive access.

Embedding Reuse

If you’re comparing one face against a large database:

Extract target embedding once
Compare against all stored embeddings
Sort by similarity score

This is exactly what Iris does—target embedding is computed before the loop in main.rs:73-85.

Security Considerations

Spoofing Attacks

SFace is vulnerable to: ⚠️ Photo attacks: Holding a printed photo of someone
⚠️ Screen attacks: Showing a face on a phone screen
⚠️ Deep fakes: AI-generated fake faces

For high-security applications, combine face recognition with liveness detection to prevent spoofing.

Privacy Implications

Embeddings cannot reconstruct original images, but:

They uniquely identify individuals
They can be used for tracking across databases
They should be treated as biometric data under GDPR/CCPA

Iris’s stateless design helps with privacy—no embeddings are stored permanently. See Stateless Design for details.

Comparison with Other Models

Model	Accuracy	Speed	Size	Best For
SFace	99.1%	⚡⚡⚡ Fast	27 MB	Production APIs, edge devices
ArcFace	99.8%	⚡⚡ Medium	90 MB	High accuracy needs
FaceNet	99.6%	⚡⚡ Medium	128 MB	Research, offline processing
DeepFace	97.4%	⚡ Slow	140 MB	Legacy applications

SFace offers the best speed/accuracy trade-off for real-time API deployments.

Get Started

Setup

Core Concepts

Security & Privacy

Face Recognition

Overview

SFace Model

Key Characteristics

Input Size

Output

Architecture

Training Data

Why 128 Dimensions?

Initialization

Embedding Extraction

Step 1: Face Alignment

Step 2: Feature Extraction

Embedding Properties

What Information Is Encoded?

Face Matching Algorithm

Cosine Similarity

The 0.363 Threshold

What Does This Mean?

Adjusting the Threshold

Probability Calculation

Recognition Accuracy

Benchmark Performance

Real-World Factors

Multi-Face Comparison

Key Behaviors

Optimization Techniques

Mutex Locking Strategy

Embedding Reuse

Security Considerations

Spoofing Attacks

Privacy Implications

Comparison with Other Models

Build docs developers (and LLMs) love

Get Started

Setup

Core Concepts

Security & Privacy

​Overview

​SFace Model

​Key Characteristics

Input Size

Output

Architecture

Training Data

​Why 128 Dimensions?

​Initialization

​Embedding Extraction

​Step 1: Face Alignment

​Step 2: Feature Extraction

​Embedding Properties

​What Information Is Encoded?

​Face Matching Algorithm

​Cosine Similarity

​The 0.363 Threshold

​What Does This Mean?

​Adjusting the Threshold

​Probability Calculation

​Recognition Accuracy

​Benchmark Performance

​Real-World Factors

​Multi-Face Comparison

​Key Behaviors

​Optimization Techniques

​Mutex Locking Strategy

​Embedding Reuse

​Security Considerations

​Spoofing Attacks

​Privacy Implications

​Comparison with Other Models

Build docs developers (and LLMs) love

Overview

SFace Model

Key Characteristics

Why 128 Dimensions?

Initialization

Embedding Extraction

Step 1: Face Alignment

Step 2: Feature Extraction

Embedding Properties

What Information Is Encoded?

Face Matching Algorithm

Cosine Similarity

The 0.363 Threshold

What Does This Mean?

Adjusting the Threshold

Probability Calculation

Recognition Accuracy

Benchmark Performance

Real-World Factors

Multi-Face Comparison

Key Behaviors

Optimization Techniques

Mutex Locking Strategy

Embedding Reuse

Security Considerations

Spoofing Attacks

Privacy Implications

Comparison with Other Models