Skip to main content
Keras model parts provide neural network-based autopilots that predict steering and throttle from camera images.

​
Base Classes

​
KerasPilot

Abstract base class for all Keras-based pilots. Constructor: 
KerasPilot(interpreter=KerasInterpreter(), input_shape=(120, 160, 3))
​
interpreter
Interpreter
default:"KerasInterpreter()"
Model interpreter (Keras, TFLite, TensorRT)
​
input_shape
tuple
default:"(120, 160, 3)"
Input image shape (height, width, channels)
Methods:
​
run
(img_arr: np.ndarray, *other_arr) -> (angle, throttle)
Run inference on image and optional additional inputs. Returns steering angle and throttle.
​
load
(model_path: str) -> None
Load a trained model from file
​
train
(model_path, train_data, train_steps, ...) -> history
Train the model on provided dataset
​
compile
() -> None
Compile the model with optimizer and loss function

​
Linear Pilots

​
KerasLinear

Linear regression pilot with continuous outputs. Constructor: 
KerasLinear(interpreter=KerasInterpreter(), input_shape=(120, 160, 3), num_outputs=2)
​
num_outputs
int
default:"2"
Number of outputs (typically 2: angle and throttle)
Architecture:
  • 5 convolutional layers (24, 32, 64, 64, 64 filters)
  • Dropout layers for regularization
  • 2 dense layers (100, 50 neurons)
  • Linear output layer per output
Loss Function: Mean Squared Error (MSE) Usage Example: 
from donkeycar.parts.keras import KerasLinear

kl = KerasLinear(input_shape=(120, 160, 3))
kl.load('models/pilot.h5')

V.add(kl,
      inputs=['cam/image_array'],
      outputs=['pilot/angle', 'pilot/throttle'])

​
KerasCategorical

Categorical classification pilot that bins outputs. Constructor: 
KerasCategorical(interpreter=KerasInterpreter(), input_shape=(120, 160, 3),
                 throttle_range=0.5)
​
throttle_range
float
default:"0.5"
Maximum throttle range (0 to throttle_range)
Architecture:
  • 5 convolutional layers
  • Dropout layers
  • 2 dense layers
  • Softmax outputs: 15 bins for angle, 20 bins for throttle
Loss Function: Categorical Cross-Entropy Usage Example: 
from donkeycar.parts.keras import KerasCategorical

kl = KerasCategorical(throttle_range=0.5)
kl.load('models/pilot.h5')

V.add(kl,
      inputs=['cam/image_array'],
      outputs=['pilot/angle', 'pilot/throttle'])

​
Recurrent Pilots

​
KerasMemory

Pilot with memory of previous steering/throttle commands. Constructor: 
KerasMemory(interpreter=KerasInterpreter(), input_shape=(120, 160, 3),
            mem_length=3, mem_depth=0, mem_start_speed=0.0)
​
mem_length
int
default:"3"
Number of previous commands to remember
​
mem_depth
int
default:"0"
Number of dense layers for memory processing
​
mem_start_speed
float
default:"0.0"
Initial throttle value for memory
Architecture:
  • Standard CNN layers for image
  • Additional dense layers for memory input
  • Concatenation of image and memory features
  • Linear outputs
Returns: Smoother steering outputs by considering recent control history Usage Example: 
from donkeycar.parts.keras import KerasMemory

kl = KerasMemory(mem_length=3, mem_depth=2)
kl.load('models/pilot.h5')

V.add(kl,
      inputs=['cam/image_array'],
      outputs=['pilot/angle', 'pilot/throttle'])

​
KerasLSTM

Long Short-Term Memory (LSTM) pilot for temporal sequences. Constructor: 
KerasLSTM(interpreter=KerasInterpreter(), input_shape=(120, 160, 3),
          seq_length=3, num_outputs=2)
​
seq_length
int
default:"3"
Number of frames in sequence
​
num_outputs
int
default:"2"
Number of outputs
Architecture:
  • Time-distributed convolutional layers
  • 2 LSTM layers (128 units each)
  • Dense layers
  • Linear output
Usage Example: 
from donkeycar.parts.keras import KerasLSTM

kl = KerasLSTM(seq_length=3)
kl.load('models/pilot.h5')

V.add(kl,
      inputs=['cam/image_array'],
      outputs=['pilot/angle', 'pilot/throttle'])

​
Keras3D_CNN

3D Convolutional Neural Network for temporal sequences. Constructor: 
Keras3D_CNN(interpreter=KerasInterpreter(), input_shape=(120, 160, 3),
            seq_length=20, num_outputs=2)
​
seq_length
int
default:"20"
Number of frames in sequence
Architecture:
  • 4 Conv3D layers with max pooling
  • Flatten and dense layers
  • Batch normalization and dropout

​
Multi-Input Pilots

​
KerasIMU

Pilot that uses camera image + IMU sensor data. Constructor: 
KerasIMU(interpreter=KerasInterpreter(), input_shape=(120, 160, 3),
         num_outputs=2, num_imu_inputs=6)
​
num_imu_inputs
int
default:"6"
Number of IMU inputs (accel_xyz + gyro_xyz)
Architecture:
  • CNN branch for image processing
  • Dense branch for IMU data
  • Concatenated features
  • Linear outputs
Usage Example: 
from donkeycar.parts.keras import KerasIMU

kl = KerasIMU(num_imu_inputs=6)
kl.load('models/pilot.h5')

V.add(kl,
      inputs=['cam/image_array', 'imu_array'],
      outputs=['pilot/angle', 'pilot/throttle'])

​
KerasBehavioral

Behavioral cloning with behavior state input. Constructor: 
KerasBehavioral(interpreter=KerasInterpreter(), input_shape=(120, 160, 3),
                throttle_range=0.5, num_behavior_inputs=2)
​
num_behavior_inputs
int
default:"2"
Number of behavior state inputs
Architecture:
  • CNN branch for image
  • Dense branch for behavior state
  • Categorical outputs (like KerasCategorical)
Usage Example: 
from donkeycar.parts.keras import KerasBehavioral

kl = KerasBehavioral(num_behavior_inputs=2)
kl.load('models/pilot.h5')

V.add(kl,
      inputs=['cam/image_array', 'behavior/state'],
      outputs=['pilot/angle', 'pilot/throttle'])

​
KerasLocalizer

Pilot with track localization output. Constructor: 
KerasLocalizer(interpreter=KerasInterpreter(), input_shape=(120, 160, 3),
               num_locations=8)
​
num_locations
int
default:"8"
Number of track location categories
Outputs:
  • Steering angle
  • Throttle
  • Track location (0 to num_locations-1)

​
Training

​
Training a Model

from donkeycar.parts.keras import KerasLinear

kl = KerasLinear()
kl.compile()

history = kl.train(
    model_path='models/pilot.h5',
    train_data=train_dataset,
    train_steps=steps,
    batch_size=128,
    validation_data=val_dataset,
    validation_steps=val_steps,
    epochs=100,
    verbose=1
)

​
Optimizer Configuration

kl.set_optimizer('adam', rate=0.001, decay=0.0)
Supported Optimizers:
  • 'adam': Adam optimizer
  • 'sgd': Stochastic Gradient Descent
  • 'rmsprop': RMSProp

​
Configuration

Typical configuration in myconfig.py: 
# Model Settings
DEFAULT_MODEL_TYPE = 'linear'
BATCH_SIZE = 128
TRAIN_TEST_SPLIT = 0.8

# Image Settings  
IMAGE_W = 160
IMAGE_H = 120
IMAGE_DEPTH = 3

# Training
MAX_EPOCHS = 100
MIN_DELTA = 0.0005
PATIENCE = 5

​
Integration Example

from donkeycar.parts.keras import KerasLinear

# Create and load pilot
kl = KerasLinear()
kl.load(cfg.MODEL_PATH)

# Add to vehicle
V.add(kl,
      inputs=['cam/image_array'],
      outputs=['pilot/angle', 'pilot/throttle'],
      run_condition='pilot_mode')

Build docs developers (and LLMs) love

Get started for freeTalk to us