Screenshot Capture & Storage

Overview

ContextFort automatically captures screenshots of every agent interaction, creating a complete visual audit trail. Screenshots are captured in pairs (before/after) for most actions, providing clear evidence of what the agent saw and what changed.

Privacy Notice: Screenshots may contain sensitive information (PII, PHI, financial data). All screenshots are stored locally on your machine and never transmitted externally.

When Screenshots are Captured

Action-Based Triggers

Click Events

Before: Immediate capture when agent clicks After: 300ms delay to show result

Double Click

Before: Immediate capture After: 300ms delay

Right Click

Before: Immediate capture After: 300ms delay

Input Events

Debounced: 1 second after last keystroke After Only: Shows final input value

Page Navigation: When agent navigates to a new URL, a “page_read” event is logged but no screenshot is captured (only metadata). This reduces storage usage while maintaining navigation history.

Screenshot Capture Implementation

Click Event Capture

chrome-extension/background.js

function onMessageScreenshotTrigger(message, tab) {
  let activation = activeAgentTabs.get(tab.id);
  if (!activation) return;

  if (message.action === 'click') {
    chrome.tabs.get(tab.id, (currentTab) => {
      if (chrome.runtime.lastError || !currentTab) return;

      // Capture BEFORE screenshot
      chrome.tabs.captureVisibleTab(tab.windowId, { format: 'png' }, (dataUrl1) => {
        if (chrome.runtime.lastError) return;

        saveEventData(dataUrl1, false, currentTab.url, currentTab.title, null)
          .then(actionId => {
            // Wait 300ms for page to update
            setTimeout(() => {
              chrome.tabs.get(tab.id, (resultTab) => {
                if (chrome.runtime.lastError || !resultTab) return;

                // Capture AFTER screenshot
                chrome.tabs.captureVisibleTab(tab.windowId, { format: 'png' }, (dataUrl2) => {
                  if (chrome.runtime.lastError) return;
                  saveEventData(dataUrl2, true, resultTab.url, resultTab.title, actionId);
                });
              });
            }, 300);
          });
      });
    });
  }
}

Why 300ms delay?
What if page takes longer?

Most web interactions complete within 300ms:

Button clicks trigger immediate UI changes
Modal dialogs appear instantly
Form validations show feedback quickly

300ms balances capturing the result without unnecessary delay.

Input Event Capture (Debounced)

chrome-extension/background.js

const inputDebounceTimers = new Map();
const INPUT_DEBOUNCE_MS = 1000;

function onMessageScreenshotTrigger(message, tab) {
  if (message.action === 'input') {
    let debounceState = inputDebounceTimers.get(tab.id);

    if (!debounceState) {
      debounceState = { timer: null, inputs: [], tabInfo: null };
      inputDebounceTimers.set(tab.id, debounceState);
    }

    // Clear existing timer
    if (debounceState.timer) {
      clearTimeout(debounceState.timer);
    }

    // Collect input value
    debounceState.inputs.push({
      element: message.element,
      inputValue: message.inputValue,
      timestamp: new Date().toISOString()
    });

    // Set new timer
    debounceState.timer = setTimeout(() => {
      const collectedInputs = debounceState.inputs;
      const inputValues = collectedInputs.map(i => i.inputValue);
      inputDebounceTimers.delete(tab.id);

      chrome.tabs.get(tab.id, (currentTab) => {
        if (chrome.runtime.lastError || !currentTab) return;

        // Wait 500ms for rendering
        setTimeout(() => {
          chrome.tabs.captureVisibleTab(tab.windowId, { format: 'png' }, (dataUrl) => {
            if (chrome.runtime.lastError) return;

            // Save with ALL input values
            const screenshotData = {
              id: Date.now() + Math.random(),
              sessionId: activation.sessionId,
              tabId: tab.id,
              url: currentTab.url,
              title: currentTab.title,
              reason: 'agent_event',
              timestamp: new Date().toISOString(),
              dataUrl: dataUrl,
              eventType: 'input',
              eventDetails: {
                inputValues: inputValues, // Array of all input values
                actionType: 'input_result'
              }
            };
            queuedStorageWrite(screenshotData, activation);
          });
        }, 500);
      });
    }, INPUT_DEBOUNCE_MS);
  }
}

Why debounce input events?

Without debouncing, typing “Hello World” would trigger 11 screenshots (one per keystroke). Debouncing ensures only one screenshot is captured after typing is complete.Debounce Timer: 1000ms (1 second) Additional Delay: 500ms for rendering Total Delay: 1.5 seconds after last keystroke

Why collect all input values?

The inputValues array captures the full typing sequence:

inputValues: ["H", "He", "Hel", "Hell", "Hello", "Hello ", "Hello W", ...]

This provides context for debugging agent behavior, showing the agent’s typing process.

Screenshot Data Structure

Full Data Format

interface Screenshot {
  id: number;              // Timestamp + random for uniqueness
  sessionId: number;       // References parent session
  tabId: number;           // Chrome tab ID
  url: string;             // Full URL at time of capture
  title: string;           // Page title at time of capture
  reason: string;          // "agent_event" or "page_read"
  timestamp: string;       // ISO 8601 timestamp
  dataUrl: string | null;  // Base64 PNG data URL
  eventType: string;       // "click" | "input" | "page_read"
  eventDetails: {
    element: ElementMetadata | null;
    coordinates: { x: number; y: number } | null;
    inputValue: string | null;
    inputValues?: string[];  // For debounced inputs
    actionType: string;      // "click" | "input_result" | "page_read"
    actionId?: number;       // Links before/after pairs
  };
}

interface ElementMetadata {
  tag: string;        // "BUTTON", "INPUT", "A", etc.
  id: string | null;
  className: string | null;
  text: string | null;      // First 50 chars of textContent
  type: string | null;      // "submit", "text", etc.
  name: string | null;
}

Example: Click Event

Before Screenshot
After Screenshot

{
  "id": 1709567890123.456,
  "sessionId": 1709567800000,
  "tabId": 1234,
  "url": "https://app.example.com/dashboard",
  "title": "Dashboard - Example App",
  "reason": "agent_event",
  "timestamp": "2025-02-28T12:34:50.123Z",
  "dataUrl": "data:image/png;base64,iVBORw0KGgoAAAANS...",
  "eventType": "click",
  "eventDetails": {
    "element": {
      "tag": "BUTTON",
      "id": "submit-button",
      "className": "btn btn-primary",
      "text": "Submit Form",
      "type": "submit",
      "name": null
    },
    "coordinates": { "x": 450, "y": 300 },
    "inputValue": null,
    "actionType": "click"
  }
}

{
  "id": 1709567890423.789,
  "sessionId": 1709567800000,
  "tabId": 1234,
  "url": "https://app.example.com/success",
  "title": "Success - Example App",
  "reason": "agent_event",
  "timestamp": "2025-02-28T12:34:50.423Z",
  "dataUrl": "data:image/png;base64,iVBORw0KGgoAAAANS...",
  "eventType": "click",
  "eventDetails": {
    "element": null,
    "coordinates": null,
    "inputValue": null,
    "actionType": "click_result",
    "actionId": 1709567890123.456
  }
}

The actionId field links the “after” screenshot to the “before” screenshot, allowing the UI to display them as a pair.

Example: Input Event

{
  "id": 1709567900500.123,
  "sessionId": 1709567800000,
  "tabId": 1234,
  "url": "https://mail.example.com/compose",
  "title": "Compose Email - Mail",
  "reason": "agent_event",
  "timestamp": "2025-02-28T12:35:00.500Z",
  "dataUrl": "data:image/png;base64,iVBORw0KGgoAAAANS...",
  "eventType": "input",
  "eventDetails": {
    "element": null,
    "coordinates": null,
    "inputValue": null,
    "inputValues": [
      "H",
      "He",
      "Hel",
      "Hell",
      "Hello",
      "Hello ",
      "Hello W",
      "Hello Wo",
      "Hello Wor",
      "Hello Worl",
      "Hello World"
    ],
    "actionType": "input_result"
  }
}

{
  "id": 1709567920000.456,
  "sessionId": 1709567800000,
  "tabId": 1234,
  "url": "https://docs.example.com/guide",
  "title": "User Guide - Docs",
  "reason": "page_read",
  "timestamp": "2025-02-28T12:35:20.000Z",
  "dataUrl": null,
  "eventType": "page_read",
  "eventDetails": {
    "element": null,
    "coordinates": null,
    "inputValue": null,
    "actionType": "page_read"
  }
}

Storage Management

Queued Storage Writes

To prevent storage bottlenecks and race conditions, screenshot writes are queued:

chrome-extension/background.js

const storageWriteQueue = [];
let isProcessingQueue = false;

async function queuedStorageWrite(screenshotData, activation) {
  return new Promise((resolve) => {
    storageWriteQueue.push({ screenshotData, activation, resolve });
    processStorageQueue();
  });
}

async function processStorageQueue() {
  if (isProcessingQueue || storageWriteQueue.length === 0) {
    return;
  }
  isProcessingQueue = true;

  while (storageWriteQueue.length > 0) {
    const { screenshotData, activation, resolve } = storageWriteQueue.shift();
    try {
      const result = await chrome.storage.local.get(['screenshots', 'sessions']);
      const screenshots = result.screenshots || [];
      const allSessions = result.sessions || [];

      screenshots.push(screenshotData);
      if (screenshots.length > 100) {
        screenshots.shift(); // FIFO - remove oldest
      }

      // Update session screenshot count
      const sessionIndex = allSessions.findIndex(s => s.id === activation.sessionId);
      if (sessionIndex !== -1) {
        allSessions[sessionIndex].screenshotCount = 
          (allSessions[sessionIndex].screenshotCount || 0) + 1;
        const groupId = activation.groupId;
        if (sessions.get(groupId)) {
          sessions.get(groupId).screenshotCount = allSessions[sessionIndex].screenshotCount;
        }
      }

      await chrome.storage.local.set({ screenshots: screenshots, sessions: allSessions });
      resolve(screenshotData.id);
    } catch (error) {
      console.error('[ContextFort] ❌ Storage write failed:', error);
      resolve(null);
    }
  }

  isProcessingQueue = false;
}

Retention Policy

Maximum Screenshots

100 screenshots stored at any time

FIFO Queue

Oldest screenshots deleted when limit reached

No Expiration

Screenshots never expire automatically

Manual Deletion

User controls all retention via Dashboard or uninstall

Storage Limits: Chrome’s local storage has a 10MB limit for extension data. 100 screenshots (each ~100-500KB) fit comfortably within this limit.

Security & Privacy

What’s in Screenshots

Sensitive Data
Storage Security
User Responsibility

Screenshots may contain:

PII: Names, email addresses, phone numbers
PHI: Medical records, health data
Financial: Credit card numbers, bank balances
Credentials: Visible passwords, API keys
Confidential: Trade secrets, internal documents

Data Deletion

Delete All Screenshots

Uninstall ContextFort extension:

Go to chrome://extensions/
Find ContextFort
Click “Remove”
All local data deleted immediately

Clear Storage

Keep extension but clear all data:

chrome.storage.local.clear(() => {
  console.log('All ContextFort data deleted');
});

Delete Specific Screenshots

Currently, you can delete all screenshots via Chrome storage. Individual screenshot deletion via the dashboard is not yet available.To delete all screenshots:

chrome.storage.local.remove(['screenshots'], () => {
  console.log('Screenshots deleted');
});

Performance Optimization

Capture Optimization

PNG Format

Screenshots use PNG format for lossless quality:

chrome.tabs.captureVisibleTab(windowId, { format: 'png' }, callback);

Alternatives considered:

JPEG: Lossy compression, not suitable for text-heavy screenshots
WebP: Better compression but lower browser support

Device Pixel Ratio

Coordinates are scaled for high-DPI displays:

chrome-extension/content.js

coordinates: {
  x: e.clientX * window.devicePixelRatio,
  y: e.clientY * window.devicePixelRatio
}

Capture Failures

If capture fails (tab closed, window minimized), errors are silently ignored:

chrome.tabs.captureVisibleTab(windowId, { format: 'png' }, (dataUrl) => {
  if (chrome.runtime.lastError) {
    console.error('Screenshot capture failed:', chrome.runtime.lastError);
    return; // Silently fail
  }
  saveEventData(dataUrl);
});

Viewing Screenshots

Screenshots can be viewed in the Visibility Dashboard:

Click ContextFort icon in Chrome toolbar
Navigate to “Visibility” page
Select a session to view all screenshots
Click screenshot to view full size
Download individual screenshots or entire session

Open Visibility Dashboard

View all captured screenshots organized by session

Export Screenshots

Export screenshots for offline review or audit:

// Export all screenshots for a session
chrome.storage.local.get(['screenshots'], (result) => {
  const screenshots = result.screenshots || [];
  const sessionScreenshots = screenshots.filter(s => s.sessionId === sessionId);
  
  // Convert to JSON for export
  const exportData = JSON.stringify(sessionScreenshots, null, 2);
  const blob = new Blob([exportData], { type: 'application/json' });
  const url = URL.createObjectURL(blob);
  
  // Trigger download
  chrome.downloads.download({
    url: url,
    filename: `contextfort-session-${sessionId}.json`,
    saveAs: true
  });
});

Export Format: Screenshots export as JSON with base64-encoded PNG data URLs. You can extract the data URLs and save as individual PNG files.

Next Steps

Visibility Dashboard

Learn about session tracking and monitoring

Session Isolation

Understand how screenshots relate to session swaps

Blocking Rules

See how blocking events are captured in screenshots

Data Policy

Read the full data policy and security details

​Overview

​When Screenshots are Captured

​Action-Based Triggers

Click Events

Double Click

Right Click

Input Events

​Navigation Events

​Screenshot Capture Implementation

​Click Event Capture

​Input Event Capture (Debounced)

​Screenshot Data Structure

​Full Data Format

​Example: Click Event

​Example: Input Event

​Example: Page Navigation

​Storage Management

​Queued Storage Writes

​Retention Policy

Maximum Screenshots

FIFO Queue

No Expiration

Manual Deletion

​Security & Privacy

​What’s in Screenshots

​Data Deletion

​Performance Optimization

​Capture Optimization

​Viewing Screenshots

Open Visibility Dashboard

​Export Screenshots

​Next Steps

Visibility Dashboard

Session Isolation

Blocking Rules

Data Policy

Overview

When Screenshots are Captured

Action-Based Triggers

Navigation Events

Screenshot Capture Implementation

Click Event Capture

Input Event Capture (Debounced)

Screenshot Data Structure

Full Data Format

Example: Click Event

Example: Input Event

Example: Page Navigation

Storage Management

Queued Storage Writes

Retention Policy

Security & Privacy

What’s in Screenshots

Data Deletion

Performance Optimization

Capture Optimization

Viewing Screenshots

Export Screenshots

Next Steps