Skip to main content
The visual_sync module synchronizes videos by correlating motion patterns across different camera views. Even though cameras may have different angles, the timing of motion events (walking, sitting, gestures) should be the same.

​
extract_motion_energy

Extract motion energy timeseries from a video file. 
from src.visual_sync import extract_motion_energy

motion, fps = extract_motion_energy(
    video_path="camera1.mp4",
    downsample=4,
    blur_size=5,
    center_crop=True,
    step=3
)

print(f"Extracted {len(motion)} motion samples at {fps:.2f} fps")

​
Parameters

​
video_path
str
required
Path to the video file to process
​
downsample
int
default:"4"
Spatial downsampling factor for faster processing. Higher values = faster but less detail.
​
blur_size
int
default:"5"
Gaussian blur kernel size to reduce noise. Must be odd. Set to 0 to disable blurring.
​
center_crop
bool
default:"True"
If True, use only center 50% of frame (0.25 to 0.75 in both dimensions). Helps focus on main subject area.
​
step
int
default:"3"
Process every Nth frame for speed. step=3 means process every 3rd frame.

​
Returns

​
motion_energy
np.ndarray
1D array of motion energy values (normalized 0-1) for each processed frame
​
effective_fps
float
Effective frame rate after skipping frames (original_fps / step)

​
Raises

  • ValueError - Cannot open the specified video file

​
smooth_motion_signal

Smooth the motion signal to reduce noise using a uniform filter. 
from src.visual_sync import smooth_motion_signal
import numpy as np

raw_signal = np.array([0.1, 0.9, 0.2, 0.8, ...])
fps = 10.0

smoothed = smooth_motion_signal(raw_signal, fps, window_sec=0.2)

​
Parameters

​
signal
np.ndarray
required
Raw motion energy signal to smooth
​
fps
float
required
Frame rate of the signal in Hz
​
window_sec
float
default:"0.2"
Smoothing window duration in seconds. Larger values = more smoothing.

​
Returns

​
smoothed_signal
np.ndarray
Smoothed motion energy signal with same length as input

​
correlate_motion_signals

Find time offset between two motion signals using cross-correlation. 
from src.visual_sync import correlate_motion_signals

sig1 = motion_signals['camera1.mp4']
sig2 = motion_signals['camera2.mp4']
fps = 10.0

offset, confidence = correlate_motion_signals(sig1, sig2, fps, max_offset_sec=20.0)
print(f"Camera 2 offset: {offset:+.2f}s, confidence: {confidence:.3f}")

​
Parameters

​
sig1
np.ndarray
required
First motion signal (reference)
​
sig2
np.ndarray
required
Second motion signal to align
​
fps
float
required
Frame rate of the signals in Hz
​
max_offset_sec
float
default:"20.0"
Maximum expected offset in seconds. Constrains the search range.

​
Returns

​
offset_seconds
float
Time offset in seconds to add to sig2 to align with sig1
​
confidence
float
Confidence score between 0 and 1. Values below 0.3 are considered low confidence.

​
visualize_motion_signals

Create a visualization plot of motion signals for debugging. 
from src.visual_sync import visualize_motion_signals

motion_signals = {
    'camera1.mp4': signal1,
    'camera2.mp4': signal2,
    'camera3.mp4': signal3
}

visualize_motion_signals(
    motion_signals=motion_signals,
    fps=10.0,
    output_path="./output/motion_signals.png"
)

​
Parameters

​
motion_signals
Dict[str, np.ndarray]
required
Dictionary mapping video names to motion signal arrays
​
fps
float
required
Frame rate of the signals for x-axis time scaling
​
output_path
str
required
Path where the plot image will be saved (PNG format)

​
Returns

None - saves plot to the specified output path

​
sync_videos_by_motion

Main entry point for visual/motion-based video synchronization. 
from src.visual_sync import sync_videos_by_motion

offsets = sync_videos_by_motion(
    video_dir="./videos",
    selected_files=['camera1.mp4', 'camera2.mp4', 'camera3.mp4'],
    max_offset_sec=20.0,
    output_dir="./output"
)

# Use these offsets with apply_video_offsets
for filename, offset in offsets.items():
    print(f"{filename}: {offset:+.2f}s")

​
Parameters

​
video_dir
str
required
Directory containing video files to synchronize
​
selected_files
List[str]
required
List of video filenames to synchronize (must be in video_dir)
​
max_offset_sec
float
default:"20.0"
Maximum expected offset between videos in seconds
​
output_dir
str | None
default:"None"
Optional directory to save debug plots. If provided, motion signal visualization will be saved.

​
Returns

​
offsets
Dict[str, float]
Dictionary mapping filename to offset in seconds. The first file is anchored at 0.0. Add these offsets to align the videos.

​
Algorithm Overview

The synchronization process consists of three steps:
  1. Motion Extraction: Extract motion energy timeseries from each video using frame differencing
  2. Pairwise Correlation: Cross-correlate all pairs of motion signals to find relative offsets
  3. Global Optimization: Find globally consistent offsets using weighted least-squares

​
Performance Notes

  • Videos are processed in parallel using ThreadPoolExecutor for speed
  • Motion signals are resampled to a common 10 fps for consistent correlation
  • The algorithm focuses on motion timing rather than visual appearance, making it robust to different camera angles and lighting conditions

Build docs developers (and LLMs) love

Get started for freeTalk to us