Skip to main content

Overview

The Image Analysis feature allows nurses to upload photos of handoff whiteboards, paper documents, or other medical images. Claude’s vision capabilities extract structured information from these images and integrate it seamlessly with electronic patient records.

Use Cases

Handoff Whiteboards

Capture information from patient room whiteboards with handwritten notes

Paper Documents

Digitize paper handoff sheets and forms

Vital Signs

Extract vitals from monitor screenshots or flowsheets

Lab Results

Parse printed lab reports and imaging results

Image Upload Workflow

In Patient Detail View

From ~/workspace/source/src/components/PatientDetail.jsx:220-284:
1

Select Image

Click “Upload Handoff Image” button or drag and drop an image file
2

Preview

Image preview appears with file information
3

Analyze

Click “Analyze & Add to Notes” to process with AI
4

Integration

Analysis is automatically integrated into handoff notes
5

Regeneration

System regenerates complete handoff notes with image data merged
const handleImageSelect = (e) => {
  const file = e.target.files[0];
  if (file && file.type.startsWith('image/')) {
    setSelectedImage(file);
    
    const reader = new FileReader();
    reader.onload = (e) => setImagePreview(e.target.result);
    reader.readAsDataURL(file);
  }
};

Standalone Image Analyzer

A dedicated image analyzer component is available at ~/workspace/source/src/components/ImageAnalyzer.jsx with:
  • Drag and drop support
  • Click to upload functionality
  • Real-time preview of uploaded images
  • Copy analysis to clipboard
const handleFileSelect = (e) => {
  const file = e.target.files[0];
  if (file && file.type.startsWith('image/')) {
    setSelectedFile(file);
    
    const reader = new FileReader();
    reader.onload = (e) => setPreview(e.target.result);
    reader.readAsDataURL(file);
  }
};

AI Analysis

What Claude Extracts

From ~/workspace/source/server/index.js:63-142, the AI analyzes images for:
  • Patient name and identifiers
  • Demographics (age, sex)
  • Chief complaint or admission reason
  • Current condition
  • Vital signs (temperature, heart rate, blood pressure, O2 saturation)
  • Trends in vital signs over time
  • Key lab results
  • Imaging findings
  • Trends and changes from previous results
  • Active medications
  • Recent medication changes
  • Ongoing treatments or interventions
  • Completed tasks during the shift
  • Outstanding tasks with priorities
  • Time-sensitive actions needed
  • Allergies
  • Fall risk
  • Code status (Full Code, DNR, DNI)
  • Isolation precautions
Task Status is Very Important - The system specifically looks for and emphasizes completed and outstanding tasks from handoff documents.

Claude Prompt

The AI receives this instruction at ~/workspace/source/server/index.js:99-126:
You are a clinical nursing assistant analyzing a handoff document or whiteboard. 
Please extract and summarize the following information:

1. Patient overview and Shift-Level Forecast
2. Current clinical status and vital signs
3. Key lab/imaging findings and trends
4. Active medications and treatments
5. TASK STATUS - Completed and outstanding tasks (VERY IMPORTANT)
6. Safety concerns (allergies, fall risk, code status, etc.)

WE CARE A LOT ABOUT TRENDS - If there are trends in vitals, labs, or 
condition changes, highlight them clearly.

Format the output as a clear, organized summary suitable for nurse handoff. 
If any information is unclear or not visible, note that. 
Keep the summary concise and focused on actionable items.

Integration with Patient Records

When an image is analyzed in the patient detail view at ~/workspace/source/src/components/PatientDetail.jsx:157-217, the system:
1

Extracts Data

Claude analyzes the image and returns structured text
2

Merges Information

Image data is combined with electronic patient record
3

Regenerates Notes

Complete handoff notes are regenerated with integrated data
4

Marks Updates

Information from the image is prefixed with IMAGE UPDATE:
const regenerateHandoffWithImage = async (imageAnalysisText) => {
  const response = await fetch(
    `http://localhost:3001/api/summarize-record/${aiProvider}`,
    {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        patientData: patientDataWithCurrentTasks,
        previousNotes: handoffNotes,
        imageAnalysis: imageAnalysisText,  // Image data merged here
      }),
    }
  );
};
The system prioritizes newer information from images when conflicts exist with the electronic record. These updates are clearly marked with IMAGE UPDATE: prefix.

API Endpoint

Analyze Image

curl -X POST http://localhost:3001/api/analyze-image/claude \
  -H "Content-Type: multipart/form-data" \
  -F "[email protected]"
Response:
{
  "summary": "## Patient Overview\n\nJohn Doe, 65M, Room 302A\n\n## Current Clinical Status\n\nVitals (18:00):\n- Temp: 98.6°F\n- HR: 72 bpm\n- BP: 132/78\n- O2 Sat: 96% on RA\n\n## Task Status\n\n**Completed:**\n✓ 20:00 - Medications administered\n✓ 21:00 - Vitals checked\n\n**Outstanding:**\n⚠ 00:00 - Lab draw (CRITICAL)\n⚠ 02:00 - Dressing change (HIGH)\n\n## Safety Concerns\n\n- Allergies: Penicillin\n- Code Status: Full Code\n- Fall Risk: Moderate",
  "provider": "claude"
}
From ~/workspace/source/server/index.js:63-142

File Upload Configuration

The backend uses Multer for file handling at ~/workspace/source/server/index.js:19-24:
const storage = multer.memoryStorage();
const upload = multer({
  storage,
  limits: { fileSize: 10 * 1024 * 1024 }, // 10MB limit
});
File Size Limit: Images must be under 10MB. The system stores images in memory during processing.

Supported Formats

The system accepts all standard image formats:
  • JPEG / JPG
  • PNG
  • GIF
  • WebP
  • BMP
  • TIFF
Validation occurs at ~/workspace/source/src/components/ImageAnalyzer.jsx:14-26:
if (file && file.type.startsWith('image/')) {
  // Process image
} else {
  setError('Please select a valid image file');
}

Best Practices

Good Lighting

Ensure whiteboards and documents are well-lit without glare

Clear Focus

Keep camera steady and ensure text is in focus

Full Frame

Capture the entire document or whiteboard in frame

High Resolution

Use highest quality camera setting available
For handwritten notes, ensure handwriting is legible. Claude can read most handwriting but neat, clear text produces better results.

Features

See uploaded image before analysis with option to clear and reupload
Analysis automatically merges with electronic records in patient detail view
Newer information from images is prioritized and clearly marked
One-click copy of analysis results to clipboard
Clear error messages for invalid files or analysis failures

Next Steps

Patient Handoff

Learn how image analysis integrates with handoff note generation

Claude Integration

Understand how Claude’s vision model processes images

Build docs developers (and LLMs) love