Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/commaai/openpilot/llms.txt

Use this file to discover all available pages before exploring further.

openpilot is a driver assistance system built around a continuously running perception-and-control loop. Raw camera frames flow into a neural network called supercombo, whose outputs feed a planning layer, which then drives longitudinal and lateral actuator commands sent to the car over the CAN bus. Every component communicates asynchronously through cereal, a Cap’n Proto–based IPC framework that serializes messages into a shared-memory message queue called msgq.

​
Vision pipeline

The core perceptual work happens in selfdrive/modeld/modeld.py. The model process (modeld) receives frames from the road-facing camera and wide-angle road camera via VisionIPC shared memory, runs them through the supercombo neural network, and publishes structured outputs on the modelV2 socket. Supercombo is a single end-to-end network with two internally distinct subnetworks:
  • Vision encoder — processes pairs of NV12 camera frames (two consecutive frames to encode motion) warped to a canonical input shape. Features are extracted into a 512-dimensional feature vector (FEATURE_LEN = 512).
  • Driving policy — consumes the vision features plus recurrent state, desired behaviour signals (lane-change intent, traffic convention), and prior control state. It outputs a multi-hypothesis trajectory plan, lane-line and road-edge detections, lead-vehicle trajectories, FCW probabilities, and a desired curvature/acceleration action.
The model runs at 20 Hz (MODEL_RUN_FREQ = 20) while camera context is sampled at 5 Hz (MODEL_CONTEXT_FREQ = 5), giving the policy access to recent temporal context without running the full encoder every frame.

​
Control loop

Model outputs travel through a pipeline of processes that progressively refine intentions into low-level actuator commands.
1

Camera frames captured

camerad continuously reads frames from the road-facing and wide-road cameras and publishes them into VisionIPC shared-memory buffers. The driver-facing camera stream is handled separately for driver monitoring.
2

Neural network inference (modeld)

modeld dequeues the latest road-camera frames, warps them to the model’s input resolution, and runs supercombo inference. The modelV2 message is published containing the desired curvature, desired acceleration, lane lines, road edges, lead-vehicle predictions, and FCW output.
3

Longitudinal planning (plannerd)

plannerd subscribes to modelV2 and radarState. It combines the model’s desired acceleration with radar-derived lead-vehicle data to produce a safe longitudinal plan, including a target acceleration (aTarget) and stop flag. The result is published as longitudinalPlan.
4

State management (selfdrived)

selfdrived runs at 100 Hz and acts as the system supervisor. It monitors engagement state, driver monitoring alerts, panda safety model consistency, excessive actuation checks, and hardware health. It publishes selfdriveState (enabled/active flags and alert metadata) and onroadEvents.
5

Low-level control (controlsd)

controlsd subscribes to selfdriveState, modelV2, longitudinalPlan, and carState. It runs the lateral controller (PID, torque, or angle depending on the car) against the model’s desired curvature and the longitudinal PID against plannerd’s acceleration target. The combined actuator commands are published as carControl.
6

CAN output (card / panda)

The car-specific driver (card) translates carControl into CAN messages for the vehicle’s ADAS ECUs. These messages are transmitted over USB to the panda hardware interface, which places them on the vehicle’s CAN bus. Panda enforces independent hardware-level safety limits before any message is sent.

​
Driver monitoring

Parallel to the driving pipeline, dmonitoringmodeld processes frames from the driver-facing camera to estimate driver attention. It publishes driverMonitoringState, which selfdrived reads to issue distraction alerts or trigger disengagement lockout if the driver is persistently unresponsive. Driver monitoring is a required safety feature and cannot be disabled.

​
Inter-process messaging with cereal

All processes communicate exclusively through cereal sockets backed by msgq. Cereal uses Cap’n Proto schemas (defined in the cereal repository) to define strongly typed message structures for every signal—carState, modelV2, longitudinalPlan, carControl, and many others. Processes declare their publish and subscribe sockets at startup; SubMaster and PubMaster handle serialization, frequency checks, and liveness monitoring automatically. 
# Example from controlsd.py
self.sm = messaging.SubMaster([
  'liveDelay', 'liveParameters', 'modelV2', 'selfdriveState',
  'longitudinalPlan', 'carState', 'onroadEvents', ...
], poll='selfdriveState')
self.pm = messaging.PubMaster(['carControl', 'controlsState'])

​
openpilot ACC vs. stock ACC

openpilot replaces—rather than supplements—the car’s stock Lane Keep Assist and Automated Lane Centering. On supported cars, it also replaces the stock Adaptive Cruise Control. The key difference is the source of the control signal:
FeatureStock systemopenpilot
Lane centeringRule-based, camera-onlySupercombo neural network
Adaptive cruiseRadar-only, fixed-gap logicModel + radar fusion, learned behaviour
Following distanceFixed tiersContinuous, personality-adjustable
FCWStock system activeopenpilot FCW runs in addition to stock
openpilot accesses the car’s ADAS ECUs through the same CAN messages the stock system uses, meaning it operates within the vehicle’s existing safety envelope while providing higher-quality acceleration, braking, and steering inputs. All other stock safety features—Automatic Emergency Braking, blind spot warning, auto high-beam—remain active and unaffected.

Build docs developers (and LLMs) love

Get started for freeTalk to us