Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/abelperezr/nokia-bng-lab/llms.txt

Use this file to discover all available pages before exploring further.

This guide covers common operational issues and provides step-by-step troubleshooting procedures for the Nokia BNG lab.

​
Container Status Issues

​
Check Overall Lab Status

1

Verify Lab Deployment

sudo containerlab inspect -t lab.yml
This shows all containers, their status, and IP addresses.
2

Check Container Health

docker ps -a | grep clab-lab
Look for:
  • Status: Should be “Up” for all containers
  • Restart count: Should be 0 or low
3

Identify Failed Containers

# Show only stopped containers
docker ps -a --filter "status=exited" | grep clab-lab

# Show containers that are restarting
docker ps -a --filter "status=restarting" | grep clab-lab
Symptoms: Container immediately exits or restarts continuouslyDiagnosis:
# View container logs
docker logs clab-lab-<container-name>

# Check last 50 lines with timestamps
docker logs --tail 50 -t clab-lab-<container-name>

# Follow logs in real-time
docker logs -f clab-lab-<container-name>
Common Causes:
  1. Missing or invalid license file (Nokia devices)
  2. Configuration file errors
  3. Port conflicts
  4. Insufficient system resources
Solutions:
# Restart specific container
docker restart clab-lab-<container-name>

# Force recreate container
sudo containerlab destroy -t lab.yml --cleanup
sudo containerlab deploy -t lab.yml

# View resource consumption
docker stats clab-lab-bng1

# Check all containers
docker stats $(docker ps --format '{{.Names}}' | grep clab-lab)
Nokia SR OS containers typically require:
  • CPU: 2+ cores
  • Memory: 4-8 GB per BNG instance
  • Storage: 10+ GB available

​
Connectivity Issues

​
Management Network Connectivity

1

Verify Management Network

docker network inspect lab
Confirm:
  • Subnet: 10.77.1.0/24
  • All containers are attached
2

Test Inter-Container Connectivity

# Ping from BNG1 to Grafana
docker exec clab-lab-bng1 ping -c 3 10.77.1.14

# Ping from Grafana to Prometheus
docker exec clab-lab-grafana ping -c 3 10.77.1.13

# Ping from gNMIc to BNG devices
docker exec clab-lab-gnmic ping -c 3 10.77.1.2
docker exec clab-lab-gnmic ping -c 3 10.77.1.3
3

Verify DNS Resolution

# Check if container names resolve
docker exec clab-lab-grafana ping -c 2 prometheus
docker exec clab-lab-prometheus ping -c 2 gnmic
Symptoms: Cannot reach Grafana (localhost:3030) or SSH (localhost:56661)Diagnosis:
# Check if ports are listening
netstat -tuln | grep -E '(3030|9090|56661|56664)'

# Or using ss
ss -tuln | grep -E '(3030|9090|56661|56664)'

# Verify Docker port mappings
docker port clab-lab-grafana
docker port clab-lab-bng1
Solutions:
  1. Check firewall rules: sudo iptables -L -n | grep <port>
  2. Verify no other service is using the port: sudo lsof -i :<port>
  3. Restart container: docker restart clab-lab-<container>
  4. Check Docker iptables: sudo iptables -t nat -L -n
Between BNG and ONT:
# Verify link status on BNG1
docker exec -it clab-lab-bng1 sr_cli "show port"

# Check OLT interfaces
docker exec -it clab-lab-olt sr_cli "show port"

# Verify ONT1 interface
docker exec clab-lab-ont1 ip link show
Check VLAN configuration:
# ONT1 should have VLAN 150
docker exec clab-lab-ont1 ip -d link show eth1

# Verify service configuration on BNG
docker exec -it clab-lab-bng1 sr_cli "show service id 150"

​
License Problems

Nokia devices will not function without valid licenses. This is the most common deployment issue.

​
Verify License Status

1

Check License File Exists

# Verify license file in configs directory
ls -lh configs/license/SR_SIM_license.txt

# Check file is not empty
wc -l configs/license/SR_SIM_license.txt
2

Verify License in Container

# BNG1 license status
docker exec -it clab-lab-bng1 sr_cli "show system license"

# Check license details
docker exec -it clab-lab-bng1 sr_cli "show system license detail"
3

Check License Expiry

docker exec -it clab-lab-bng1 sr_cli \
  "show system license | match \"Expiry Date\""
Symptoms: Device starts but shows “License not installed” or similar errorDiagnosis:
# Check if license was mounted correctly
docker exec clab-lab-bng1 ls -l /configs/license/

# Verify lab.yml has correct license path
grep -A 2 "license:" lab.yml
Solution:
# Ensure license file exists
mkdir -p configs/license
cp /path/to/your/SR_SIM_license.txt configs/license/

# Verify permissions
chmod 644 configs/license/SR_SIM_license.txt

# Redeploy lab
sudo containerlab destroy -t lab.yml
sudo containerlab deploy -t lab.yml
Symptoms: Device starts but features are disabledSolution:
  1. Obtain a new license from Nokia
  2. Replace the old license file: 
    cp /path/to/new_license.txt configs/license/SR_SIM_license.txt
  3. Restart affected containers: 
    docker restart clab-lab-bng1 clab-lab-bng2 clab-lab-switch clab-lab-olt
  4. Verify new license: 
    docker exec -it clab-lab-bng1 sr_cli "show system license"
Symptoms: License is valid but features are missingCheck Required License Features:
  • SR-7 platform support (for BNG1/BNG2)
  • IXR-ec platform support (for Switch/OLT)
  • ISA (Integrated Service Adapter) support for BNG
  • Required capacity limits
# View licensed features
docker exec -it clab-lab-bng1 sr_cli "show system license detail"

​
Authentication Failures

​
RADIUS Authentication Issues

1

Verify RADIUS Service

# Check RADIUS container is running
docker ps | grep radius

# Test RADIUS locally
docker exec clab-lab-radius radiusd -X
2

Test User Authentication

# Test PPPoE user credentials
docker exec clab-lab-radius radtest \
  test@test.com testlab123 \
  localhost 0 testing123

# Should return "Access-Accept"
3

Check RADIUS Logs

# View authentication attempts
docker exec clab-lab-radius tail -100 /var/log/radius/radius.log

# Follow logs in real-time
docker exec clab-lab-radius tail -f /var/log/radius/radius.log
Symptoms: Subscriber authentication fails, Access-Reject in logsCommon Causes:
  1. Incorrect username/password
  2. User not in authorize file
  3. Wrong shared secret between BNG and RADIUS
Diagnosis:
# Verify user exists in database
docker exec clab-lab-radius cat /etc/raddb/mods-config/files/authorize | grep test

# Check BNG RADIUS configuration
docker exec -it clab-lab-bng1 sr_cli "show system security radius"

# Verify shared secret in clients.conf
docker exec clab-lab-radius cat /etc/raddb/clients.conf
Solution:
# Edit authorize file to add/fix user
docker exec -it clab-lab-radius vi /etc/raddb/mods-config/files/authorize

# Restart RADIUS to apply changes
docker restart clab-lab-radius
Symptoms: RADIUS timeout, no response from serverDiagnosis:
# Check connectivity from BNG to RADIUS
docker exec clab-lab-bng1 ping -c 3 10.77.1.10

# Check RADIUS process is running
docker exec clab-lab-radius ps aux | grep radiusd

# Check RADIUS is listening on port 1812
docker exec clab-lab-radius netstat -uln | grep 1812
Solution:
# Restart RADIUS service
docker restart clab-lab-radius

# Check startup script executed
docker logs clab-lab-radius | grep "client.sh"

​
SSH Authentication Issues

For Nokia Devices (admin/lab123):
# Verify SSH port mapping
docker port clab-lab-bng1 | grep 22

# Try with verbose output
ssh -v admin@localhost -p 56661

# Check if SSH is running in container
docker exec clab-lab-bng1 netstat -tln | grep :22
For Linux Containers (user/test for ONTs):
# Verify ONT1 SSH access
ssh -o StrictHostKeyChecking=no user@localhost -p 56673

# Or use direct shell access
docker exec -it clab-lab-ont1 /bin/bash

​
Telemetry Issues

​
gNMIc Not Collecting Metrics

1

Check gNMIc Status

# Verify container is running
docker ps | grep gnmic

# Check logs for errors
docker logs --tail 100 clab-lab-gnmic
2

Verify Device Discovery

# Check Docker socket is accessible
docker exec clab-lab-gnmic ls -l /var/run/docker.sock

# Test container discovery
docker exec clab-lab-gnmic docker ps | grep clab-lab
3

Test gRPC Connectivity

# Test connection to BNG1
docker exec clab-lab-gnmic gnmic -a 10.77.1.2:57400 \
  -u admin -p lab123 --insecure capabilities

# Test subscription
docker exec clab-lab-gnmic gnmic -a 10.77.1.2:57400 \
  -u admin -p lab123 --insecure \
  subscribe --path "/state/system/cpu"
Symptoms: “context deadline exceeded” or timeout errorsDiagnosis:
# Check gRPC port is accessible from gNMIc container
docker exec clab-lab-gnmic nc -zv 10.77.1.2 57400

# Verify device is listening on gRPC port
docker exec clab-lab-bng1 netstat -tln | grep 57400
Solution:
  1. Verify credentials are correct: admin/lab123
  2. Check skip-verify: true or insecure: true in config
  3. Ensure device has finished booting
  4. Restart gNMIc: docker restart clab-lab-gnmic
Diagnosis Chain:
# 1. Check gNMIc metrics endpoint
curl http://localhost:9273/metrics | grep -i system_cpu

# 2. Verify Prometheus can reach gNMIc
docker exec clab-lab-prometheus wget -qO- http://gnmic:9273/metrics | head

# 3. Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq

# 4. Query Prometheus for metrics
curl -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=system_cpu_total' | jq
Common Issues:
  • gNMIc not exporting metrics (check logs)
  • Prometheus not scraping (check targets page)
  • Metric name mismatch (check processors in gNMIc config)
  • Time range issue (metrics may be too old or in future)

​
Grafana Dashboard Issues

Checklist:
# 1. Verify Grafana is running
docker ps | grep grafana

# 2. Check Prometheus data source
curl -u admin:admin http://localhost:3030/api/datasources | jq

# 3. Test data source from Grafana
curl -X POST -u admin:admin \
  http://localhost:3030/api/datasources/proxy/1/api/v1/query \
  -H "Content-Type: application/json" \
  -d '{"query":"up"}'
Solutions:
  1. Verify time range is appropriate (last 6h, 24h)
  2. Check query in Prometheus UI first
  3. Verify dashboard variables are set correctly
  4. Refresh dashboard or clear browser cache

​
Common Containerlab Issues

​
Lab Won’t Deploy

Symptoms: Port conflict during deployment
# Find process using the port
sudo lsof -i :56661
sudo lsof -i :3030

# Kill the process or change port in lab.yml
sudo kill <PID>
Symptoms: Cannot pull or find specified image
# List available images
docker images | grep nokia
docker images | grep ont-ds

# Pull missing images
docker pull ghcr.io/abelperezr/ont-ds:0.2
docker pull ghcr.io/openconfig/gnmic:latest
docker pull prom/prometheus
docker pull grafana/grafana:10.3.5
Nokia SR-SIM images must be built locally. They are not publicly available.
Symptoms: Stale network from previous deployment
# Clean up old lab
sudo containerlab destroy -t lab.yml --cleanup

# Remove stale network
docker network rm lab

# Redeploy
sudo containerlab deploy -t lab.yml

​
Lab Won’t Destroy

# Force stop all lab containers
docker stop $(docker ps -q --filter "name=clab-lab")

# Force remove
docker rm -f $(docker ps -aq --filter "name=clab-lab")

# Clean up networks
docker network prune -f

​
Log Locations

​
Containerlab Logs

# Containerlab deployment logs
sudo journalctl -u docker | grep containerlab

# Or check syslog
sudo tail -f /var/log/syslog | grep containerlab

​
Device Logs

# BNG1 system logs
docker exec -it clab-lab-bng1 sr_cli "show log log-id 99"

# Event logs
docker exec -it clab-lab-bng1 sr_cli "show log event-triggered log-id 99"

# Debug logs (if enabled)
docker exec -it clab-lab-bng1 sr_cli "show debug"

# gNMIc
docker logs clab-lab-gnmic 2>&1 | tee gnmic.log

# Prometheus
docker logs clab-lab-prometheus 2>&1 | tee prometheus.log

# Grafana
docker logs clab-lab-grafana 2>&1 | tee grafana.log

# RADIUS
docker exec clab-lab-radius cat /var/log/radius/radius.log

​
Debug Commands

​
Network Debugging

# Capture packets on container interface
docker exec clab-lab-bng1 tcpdump -i eth1 -w /tmp/capture.pcap

# Copy capture file to host
docker cp clab-lab-bng1:/tmp/capture.pcap ./

# View interface details
docker exec clab-lab-ont1 ip addr show
docker exec clab-lab-ont1 ip route show

# Check ARP tables
docker exec clab-lab-bng1 sr_cli "show router arp"

# Trace route
docker exec clab-lab-ont1 traceroute 172.19.1.1

​
Service Debugging

# View active subscribers
docker exec -it clab-lab-bng1 sr_cli \
  "show service active-subscribers"

# Check DHCP bindings
docker exec -it clab-lab-bng1 sr_cli \
  "show router dhcp bindings"

# PPPoE sessions
docker exec -it clab-lab-bng1 sr_cli \
  "show service id 150 pppoe session"

# Debug RADIUS
docker exec -it clab-lab-bng1 sr_cli \
  "debug router radius all"

# Enable gNMIc debug
docker exec -it clab-lab-gnmic sh -c \
  'gnmic --config /gnmic-config.yml --log --debug subscribe'

# Enable Prometheus debug
# Edit prometheus.yml and add: --log.level=debug to cmd

# RADIUS debug mode
docker exec -it clab-lab-radius radiusd -X

​
Getting Help

When reporting issues, include:
  1. Output of sudo containerlab inspect -t lab.yml
  2. Container logs: docker logs clab-lab-<device>
  3. Error messages (full text)
  4. Steps to reproduce
  5. System information: uname -a, docker version
  6. Lab topology file (lab.yml)
Before seeking help:
  1. Check license is valid and not expired
  2. Verify all containers are running: docker ps
  3. Review recent changes to configuration files
  4. Try redeploying: sudo containerlab destroy && sudo containerlab deploy
  5. Check system resources: free -h, df -h

​
Recovery Procedures

​
Complete Lab Reset

# 1. Destroy existing lab
sudo containerlab destroy -t lab.yml --cleanup

# 2. Remove all containers and networks
docker stop $(docker ps -q --filter "name=clab-lab")
docker rm $(docker ps -aq --filter "name=clab-lab")
docker network rm lab

# 3. Clean up resources
docker system prune -f

# 4. Redeploy from scratch
sudo containerlab deploy -t lab.yml

# 5. Wait for devices to boot (2-3 minutes)
sleep 180

# 6. Verify deployment
sudo containerlab inspect -t lab.yml

​
Backup Configuration

# Backup BNG configuration
docker exec -it clab-lab-bng1 sr_cli \
  "admin save" > backup-bng1-$(date +%Y%m%d).cfg

# Backup telemetry data (if needed)
docker exec clab-lab-prometheus tar czf /tmp/prometheus-data.tar.gz /prometheus
docker cp clab-lab-prometheus:/tmp/prometheus-data.tar.gz ./

Build docs developers (and LLMs) love

Get started for freeTalk to us