main.py

​

Overview

• YouTube video downloads
• Audio transcription with word-level timestamps
• Scene detection and strategic framing
• AI-powered viral clip identification
• Vertical video reframing with subject tracking

​

Key Functions

​

process_video_to_vertical

def process_video_to_vertical(input_video: str, final_output_video: str) -> bool

• input_video (str): Path to input video file
• final_output_video (str): Path for output vertical video

• bool: True if successful, False otherwise

1. Detects scenes using PySceneDetect
2. Analyzes each scene (single subject vs. group)
3. Applies appropriate framing strategy:
   • TRACK mode: MediaPipe face detection + YOLOv8 fallback with stabilization
   • GENERAL mode: Blurred background layout for groups/landscapes
4. Merges processed video with original audio

​

transcribe_video

def transcribe_video(video_path: str) -> dict

• video_path (str): Path to video file

• dict with keys:
  • text (str): Full transcript text
  • segments (list): Segment objects with word timestamps
  • language (str): Detected language code

{
  "text": "This is the full transcript...",
  "segments": [
    {
      "text": "This is the full transcript",
      "start": 0.0,
      "end": 2.5,
      "words": [
        {"word": "This", "start": 0.0, "end": 0.2, "probability": 0.99},
        {"word": "is", "start": 0.25, "end": 0.35, "probability": 0.98}
      ]
    }
  ],
  "language": "en"
}

​

get_viral_clips

def get_viral_clips(transcript_result: dict, video_duration: float) -> dict

• transcript_result (dict): Output from transcribe_video()
• video_duration (float): Total video duration in seconds

• dict containing:
  • shorts (list): Array of clip objects
  • cost_analysis (dict): Token usage and cost breakdown

{
  "start": 12.340,  # Absolute seconds from video start
  "end": 37.900,
  "video_description_for_tiktok": "...",
  "video_description_for_instagram": "...",
  "video_title_for_youtube_short": "...",
  "viral_hook_text": "POV: You realized..."  # Max 10 words
}

• Clips are 15-60 seconds each
• Returns 3-15 clips ranked by predicted performance
• Uses ABSOLUTE timestamps (not relative)

​

detect_scenes

def detect_scenes(video_path: str) -> tuple[list, float]

• video_path (str): Path to video file

• tuple of:
  • scene_list (list): List of (start_time, end_time) tuples
  • fps (float): Video framerate

​

download_youtube_video

def download_youtube_video(url: str, output_dir: str = ".") -> tuple[str, str]

• url (str): YouTube video URL
• output_dir (str): Directory to save video (default: current directory)

• tuple of:
  • downloaded_file (str): Path to downloaded MP4 file
  • sanitized_title (str): Sanitized video title

• Automatically selects H.264 codec for compatibility
• Uses multiple player clients (tv_embed, android) to bypass restrictions
• Supports cookie authentication via YOUTUBE_COOKIES env var
• Sanitizes filenames (removes special characters, limits length to 100 chars)

​

Core Classes

​

SmoothedCameraman

class SmoothedCameraman:
    def __init__(self, output_width: int, output_height: int, 
                 video_width: int, video_height: int)

• Safe zone logic: only moves when subject leaves center zone (25% of crop width)
• Adaptive speed: slow pan (3px/frame) or fast reframe (15px/frame)
• Prevents oscillation with overshoot detection

​

update_target

def update_target(self, face_box: list) -> None

​

get_crop_box

def get_crop_box(self, force_snap: bool = False) -> tuple[int, int, int, int]

• force_snap (bool): If True, immediately snap to target (used on scene changes)

​

SpeakerTracker

class SpeakerTracker:
    def __init__(self, stabilization_frames: int = 15, cooldown_frames: int = 30)

• Face ID assignment using proximity matching
• Score decay system (0.85x per frame)
• Hysteresis: 3x score bonus for current active speaker
• Cooldown period (30 frames) before switching to new speaker

​

get_target

def get_target(self, face_candidates: list, frame_number: int, width: int) -> Optional[list]

• face_candidates (list): List of dicts with {"box": [x,y,w,h], "score": float}
• frame_number (int): Current frame index
• width (int): Video width for distance calculations

• list: Bounding box [x, y, w, h] of target speaker, or None

​

Constants

​

ASPECT_RATIO

ASPECT_RATIO = 9 / 16  # 0.5625

​

GEMINI_PROMPT_TEMPLATE

• FFmpeg timestamp format requirements
• Clip duration constraints (15-60s)
• Natural cut point guidelines
• Output JSON schema with social media metadata

​

Dependencies

• faster-whisper: CPU-optimized transcription (base model, int8 quantization)
• google-genai: Gemini API client
• ultralytics: YOLOv8 person detection
• mediapipe: Face detection (BlazeFace)
• opencv-python (cv2): Video frame processing
• scenedetect: Scene boundary detection
• yt-dlp: YouTube video downloads
• ffmpeg: Video encoding/merging (subprocess)