Skip to main content
The Roboflow plugin provides object detection capabilities using both cloud-hosted inference and local RF-DETR models.

​
Installation

uv add vision-agents-plugins-roboflow

​
Authentication

Set your API credentials in the environment: 
export ROBOFLOW_API_KEY=your_roboflow_api_key
export ROBOFLOW_API_URL=your_roboflow_api_url  # Optional, for cloud processor

​
Components

​
RoboflowCloudDetectionProcessor

Use Roboflow’s cloud inference API with pre-trained models from Roboflow Universe: 
from vision_agents.plugins import roboflow
from vision_agents.core import Agent

processor = roboflow.RoboflowCloudDetectionProcessor(
    api_key="your_api_key",
    api_url="https://detect.roboflow.com",
    model_id="football-players-detection-3zvbc/20",
    classes=["player"],
    conf_threshold=0.5,
    fps=3
)

agent = Agent(
    processors=[processor],
    llm=your_llm,
    # ... other config
)
​
model_id
string
required
Roboflow Universe model ID. Example: "football-players-detection-3zvbc/20". Browse models at Roboflow Universe
​
api_key
string
Roboflow API key. Defaults to ROBOFLOW_API_KEY environment variable
​
api_url
string
Roboflow API URL. Defaults to ROBOFLOW_API_URL environment variable
​
conf_threshold
float
default:"0.5"
Confidence threshold for detections (0.0 - 1.0)
​
fps
int
default:"5"
Frame processing rate (frames per second)
​
classes
list[string]
Optional list of class names to detect. If not specified, all classes are detected. Example: ["person", "sports ball"]
​
annotate
bool
default:"True"
Whether to annotate detected objects with boxes and labels
​
dim_background_factor
float
default:"0.0"
How much to dim the background around detected objects (0.0 - 1.0). Only effective when annotate=True

​
RoboflowLocalDetectionProcessor

Run RF-DETR models locally for offline inference: 
from vision_agents.plugins import roboflow

processor = roboflow.RoboflowLocalDetectionProcessor(
    model_id="rfdetr-seg-preview",
    conf_threshold=0.5,
    classes=["person"],
    fps=3
)

agent = Agent(
    processors=[processor],
    llm=your_llm,
    # ... other config
)
​
model_id
string
default:"rfdetr-seg-preview"
RF-DETR model identifier:
  • rfdetr-base
  • rfdetr-large
  • rfdetr-nano
  • rfdetr-small
  • rfdetr-medium
  • rfdetr-seg-preview (with segmentation)
​
conf_threshold
float
default:"0.5"
Confidence threshold for detections (0.0 - 1.0)
​
fps
int
default:"10"
Frame processing rate
​
classes
list[string]
Optional list of class names to detect
​
annotate
bool
default:"True"
Whether to annotate detected objects
​
model
RFDETRModel
Optional custom instance of rfdetr.RFDETR() class

​
Usage Examples

​
Cloud Detection with Specific Classes

from vision_agents.plugins import roboflow

processor = roboflow.RoboflowCloudDetectionProcessor(
    model_id="football-players-detection-3zvbc/20",
    classes=["player", "ball"],
    conf_threshold=0.6,
    fps=5,
    annotate=True,
    dim_background_factor=0.5  # Dim background by 50%
)

​
Local Detection with Custom Model

from vision_agents.plugins import roboflow
from rfdetr import RFDETR

# Custom model configuration
custom_model = RFDETR(model="rfdetr-large")

processor = roboflow.RoboflowLocalDetectionProcessor(
    model=custom_model,
    classes=["person", "car", "dog"],
    conf_threshold=0.4,
    fps=10
)

​
Complete Agent Example

from vision_agents.core import Agent, User
from vision_agents.plugins import roboflow, gemini, deepgram, elevenlabs, getstream

processor = roboflow.RoboflowCloudDetectionProcessor(
    model_id="coco/3",
    classes=["person"],
    conf_threshold=0.5,
    fps=3
)

agent = Agent(
    edge=getstream.Edge(),
    agent_user=User(name="Detection Agent"),
    instructions="Describe what you detect in the video.",
    processors=[processor],
    llm=gemini.VLM(),
    stt=deepgram.STT(),
    tts=elevenlabs.TTS()
)

​
Events

Listen to detection events: 
from vision_agents.plugins.roboflow import DetectionCompletedEvent

@agent.events.subscribe
async def on_detection(event: DetectionCompletedEvent):
    print(f"Detected {len(event.detections)} objects")
    for detection in event.detections:
        print(f"- {detection.class_name}: {detection.confidence:.2f}")

​
Cloud vs Local

​
Use Cloud When:

  • You want access to pre-trained models from Roboflow Universe
  • You don’t have GPU resources
  • You want managed infrastructure
  • You need custom-trained models from your Roboflow workspace

​
Use Local When:

  • You need offline inference
  • You have GPU resources available
  • You want to avoid API rate limits
  • You need maximum throughput

​
Available Models

​
Cloud Models

Browse thousands of pre-trained models at Roboflow Universe:
  • COCO objects
  • Sports detection
  • Medical imaging
  • Custom trained models

​
Local Models (RF-DETR)

ModelSizeSpeedAccuracy
rfdetr-nanoSmallestFastestGood
rfdetr-smallSmallFastBetter
rfdetr-baseMediumMediumHigh
rfdetr-mediumLargeSlowerHigher
rfdetr-largeLargestSlowestHighest
rfdetr-seg-previewMediumMediumHigh + Segmentation

​
Configuration

​
Environment Variables

ROBOFLOW_API_KEY=your_roboflow_api_key_here
ROBOFLOW_API_URL=your_roboflow_api_url_here

​
Performance Tuning

For Speed: 
processor = roboflow.RoboflowLocalDetectionProcessor(
    model_id="rfdetr-nano",
    fps=3,
    conf_threshold=0.6
)
For Accuracy: 
processor = roboflow.RoboflowCloudDetectionProcessor(
    model_id="your-custom-model/version",
    fps=1,
    conf_threshold=0.3
)

​
Dependencies

  • vision-agents - Core framework
  • numpy>=2.0.0 - Array operations
  • rfdetr>=1.3.0 - RF-DETR models for local detection
  • inference-sdk>=0.26.1 - Roboflow SDK for cloud inference

​
References

Build docs developers (and LLMs) love

Get started for freeTalk to us