AI Capabilities Overview

​

Core AI Features

​

Intelligent Chat Assistant

• Workspace Context: AI has access to all your notes, flashcards, PDFs, quizzes, and other content
• Multi-modal Understanding: Processes text, images, PDFs, videos, and audio transcripts
• Tool Calling: Executes specialized tools to perform actions on your behalf
• Web Search: Retrieves current information from the web when needed

​

Content Processing

• PDF Analysis: Extracts text and analyzes document structure using Azure Document AI (Mistral OCR)
• Image Understanding: Analyzes images and provides detailed descriptions
• Video Processing: Processes YouTube videos natively for content extraction
• Web Scraping: Fetches and analyzes web page content using Google URL Context API or Firecrawl

​

Learning Tools

• Flashcard Generation: Creates flashcard decks from any content
• Quiz Creation: Generates interactive quizzes with multiple-choice questions
• Note Taking: Assists in creating structured notes with markdown formatting
• Content Summarization: Distills key information from long documents

​

Architecture

​

Chat API Endpoint

{
  messages: UIMessage[];           // Conversation history
  system: string;                  // System prompt
  workspaceId?: string;           // Current workspace ID
  activeFolderId?: string;        // Active folder context
  modelId?: string;               // Model to use (default: gemini-2.5-flash)
  selectedCardsContext?: string;  // Pre-formatted selected cards
  metadata?: {
    custom?: {
      replySelections?: Array<{ text: string }>;
      blockNoteSelection?: { cardName: string; text: string };
    };
  };
}

• Automatic URL detection and processing
• Context injection from selected cards
• Token usage tracking and optimization
• Message pruning to manage context window
• Smooth streaming with word-based chunking

​

AI Gateway Integration

const model = gateway(modelId); // e.g., "google/gemini-2.5-flash"

• google/ - Google AI models (Gemini)
• anthropic/ - Anthropic models (Claude) experimental

​

Thinking Capabilities

thinkingConfig: {
  includeThoughts: true,
  thinkingLevel: "minimal" // For Gemini 3 Flash
}

​

Configuration

​

Required Environment Variables

# Google AI (required)
GOOGLE_GENERATIVE_AI_API_KEY=AIza...

# Optional: AI Gateway
AI_GATEWAY_API_KEY=your-gateway-key

# Optional: Firecrawl Web Scraping
FIRECRAWL_API_KEY=fc_...
SCRAPING_MODE=hybrid  # hybrid, firecrawl-only, google-only, direct-only

# Optional: Azure Document AI (PDF OCR)
AZURE_DOCUMENT_AI_API_KEY=your-api-key
AZURE_DOCUMENT_AI_ENDPOINT=your-endpoint-url
AZURE_DOCUMENT_AI_MODEL=mistral-document-ai-2512
OCR_INCLUDE_IMAGES=true  # Extract images from PDFs

​

Scraping Modes

• hybrid (default): Try Google Context → Firecrawl (if key exists) → Direct fetch
• firecrawl-only: Force Firecrawl for all scraping
• google-only: Force Google Context API only
• direct-only: Force direct HTTP fetch only

​

Performance Optimizations

​

Context Window Management

// Prune older messages to save tokens
convertedMessages = pruneMessages({
  messages: convertedMessages,
  reasoning: "before-last-message",
  toolCalls: "before-last-5-messages",
  emptyMessages: "remove",
});

​

Token Usage Tracking

onFinish: ({ usage, finishReason }) => {
  console.log({
    inputTokens: usage?.inputTokens,
    outputTokens: usage?.outputTokens,
    totalTokens: usage?.totalTokens,
    cachedInputTokens: usage?.cachedInputTokens,
    reasoningTokens: usage?.reasoningTokens,
    finishReason,
  });
}

​

Caching Strategy

1. First access triggers Azure Document AI OCR
2. Extracted content stored in database
3. Subsequent accesses use cached content
4. Set forceReprocess: true to bypass cache

​

Analytics Integration

const model = withTracing(
  gateway(modelId),
  posthogClient,
  {
    posthogDistinctId: userId || "anonymous",
    posthogProperties: {
      workspaceId,
      activeFolderId,
      modelId,
    },
  }
);

​

Error Handling

​

Timeout Protection

if (isTimeout) {
  return new Response(JSON.stringify({
    error: "Request timeout",
    message: "The request took too long to process...",
    code: "TIMEOUT",
  }), { status: 504 });
}

​

Graceful Degradation

• PDF OCR failures fall back to Gemini file analysis
• Web scraping failures try multiple methods
• Missing context returns helpful error messages

​

Next Steps