Monitoring & Observability

The Nokia BNG lab includes a comprehensive telemetry stack based on gNMI, Prometheus, and Grafana for real-time monitoring and observability.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                   Telemetry Stack Flow                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Nokia Devices (BNG, Switch, OLT, TX)                      │
│          │                                                  │
│          │ gRPC/gNMI (port 57400)                         │
│          ▼                                                  │
│     gNMIc Collector (10.77.1.12:9273)                      │
│          │                                                  │
│          │ Prometheus Exposition                           │
│          ▼                                                  │
│   Prometheus TSDB (10.77.1.13:9090)                        │
│          │                                                  │
│          │ PromQL Queries                                  │
│          ▼                                                  │
│   Grafana Dashboards (10.77.1.14:3000)                     │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Grafana Dashboard Access

Access Grafana UI

Open your browser and navigate to:

http://localhost:3030

Credentials:

Username: admin
Password: admin

Verify Data Source

Navigate to Configuration → Data Sources to verify Prometheus connection:

Name: Prometheus
Type: prometheus
URL: http://prometheus:9090
UID: prometheus
Status: Should show green checkmark

Access Pre-configured Dashboards

Dashboards are auto-provisioned from /var/lib/grafana/dashboards:

Nokia SROS System Metrics
Interface Statistics
BNG Subscriber Sessions
Network Instance Status

Grafana is configured with anonymous access enabled (Editor role), allowing dashboard viewing without authentication.

Prometheus Metrics

Access Prometheus UI

http://localhost:9090

Key Prometheus Endpoints

Targets: http://localhost:9090/targets
Configuration: http://localhost:9090/config
Service Discovery: http://localhost:9090/service-discovery
Alerts: http://localhost:9090/alerts

Prometheus Configuration

The Prometheus server scrapes metrics from gNMIc:

# From configs/prometheus/prometheus.yml
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "gnmic"
    static_configs:
      - targets: ["gnmic:9273"]

All metrics are scraped at 5-second intervals. Adjust this if you experience performance issues or need different granularity.

Verify Metric Collection

Check gNMIc Metrics Endpoint

curl http://localhost:9273/metrics

You should see Prometheus-formatted metrics from all Nokia devices.

Query Metrics in Prometheus

Navigate to http://localhost:9090/graph and try:

# CPU usage across all devices
system_cpu_total

# Interface statistics
port_statistics_in_octets

# Operational state
port_oper_state

Check Target Health

Visit http://localhost:9090/targets and verify:

Target: gnmic:9273
State: UP
Last Scrape: < 5s ago

gNMIc Telemetry Collector

Configuration Overview

The gNMIc collector automatically discovers and subscribes to Nokia devices using Docker labels.

Device Discovery Configuration

# From configs/gnmic/config.yml
loader:
  type: docker
  address: unix:///run/docker.sock
  filters:
    # SR Linux nodes
    - containers:
        - label: clab-node-kind=nokia_srlinux
      network:
        label: containerlab
      port: "57400"
      config:
        username: admin
        password: lab123
        skip-verify: true
        encoding: proto
    
    # SR OS nodes (BNG, Switch, OLT)
    - containers:
        - label: clab-node-kind=nokia_srsim
      network:
        label: containerlab
      port: "57400"
      config:
        username: admin
        password: lab123
        insecure: true
        encoding: json

Active Subscriptions

SR Linux Subscriptions

Sample Interval: 5 seconds

srl_platform: CPU and memory usage
srl_apps: Application management
srl_if_stats: Interface statistics and operational state
srl_if_lag_stats: LAG member statistics
srl_net_instance: Network instance state and route tables
srl_bgp_stats: BGP protocol statistics
srl_event_handler_stats: Event handler metrics

SR OS Subscriptions (BNG/Switch/OLT)

Sample Interval: 5 seconds (10s for VPLS SAPs)

sros_ports_stats: Port operational state and statistics
sros_router_bgp: BGP statistics and routes per family
sros_router_interface: IPv4/IPv6 interface statistics
sros_router_isis: IS-IS protocol statistics
sros_router_route_table: Route table statistics
sros_system: CPU and memory pool usage
sros_service_stats: VPLS/VPRN service operational state
sros_ludb: Local user database (subscriber info)
sros_vpls_sap_all: VPLS SAP statistics
sros_temperature_stats: Hardware temperature sensors
sros_fan_stats: Chassis fan speeds

View gNMIc Logs

# Real-time log streaming
docker logs -f clab-lab-gnmic

# Last 100 lines
docker logs --tail 100 clab-lab-gnmic

# Logs with timestamps
docker logs -t clab-lab-gnmic

The gNMIc collector logs all subscription activities, connection status, and metric processing. Use these logs to debug telemetry issues.

Key Metrics to Monitor

System Health Metrics

CPU Usage

# SR OS CPU utilization (1-second sample)
system_cpu_total{source="bng1"}

# SR Linux CPU usage
platform_control_cpu_total{source="tx"}

Alert if CPU usage exceeds 80% for more than 5 minutes.

Memory Usage

# SR OS memory pools
system_memory_pools_summary_total

# SR Linux memory
platform_control_memory_physical
platform_control_memory_utilized

Interface Status

# Port operational state (1=up, 0=down)
port_oper_state

# SR Linux interface state
interface_oper_state

Critical interfaces should be monitored with alerts for state changes.

BNG-Specific Metrics

Subscriber Sessions

# Local user database entries
subscriber_mgmt_local_user_db_ipoe_host_session_count
subscriber_mgmt_local_user_db_ppp_session_count

# Session statistics by type
rate(subscriber_mgmt_local_user_db_ipoe_sessions_created[5m])

VPLS Service Status

# Service operational state
service_vpls_oper_state{service_name="subscriber-vlan-150"}

# SAP statistics
service_vpls_sap_stats_ingress_octets
service_vpls_sap_stats_egress_octets

Network Performance Metrics

Interface Traffic

# Ingress traffic rate (bytes/sec)
rate(port_statistics_in_octets[1m])

# Egress traffic rate
rate(port_statistics_out_octets[1m])

# Packet rates
rate(port_statistics_in_packets[1m])
rate(port_statistics_out_packets[1m])

Interface Errors

# Input errors
rate(port_statistics_in_errors[5m])

# Output errors
rate(port_statistics_out_errors[5m])

# Discards
rate(port_statistics_in_discards[5m])

Any non-zero error rate should be investigated immediately.

BGP Sessions

# BGP established sessions
router_bgp_statistics_established_sessions

# Routes per address family
router_bgp_statistics_routes_per_family_active_routes{family="ipv4"}

RADIUS Accounting Logs

Access RADIUS Logs

View Authentication Logs

docker exec clab-lab-radius tail -f /var/log/radius/radius.log

View Accounting Logs

docker exec clab-lab-radius tail -f /var/log/radius/radacct/*/*

Search for Specific User

docker exec clab-lab-radius grep "test@test.com" /var/log/radius/radius.log

RADIUS Log Locations

Main Log: /var/log/radius/radius.log
Accounting: /var/log/radius/radacct/
Configuration: /etc/raddb/

# View active RADIUS sessions
docker exec clab-lab-radius radclient localhost status testing123

# Debug mode (verbose logging)
docker exec clab-lab-radius radiusd -X

Device Health Monitoring

Temperature Monitoring

# Card temperature sensors
card_hardware_data_temperature_current

# MDA temperature
card_mda_hardware_data_temperature_current

# Control module temperature
chassis_chassis_control_module_hardware_data_temperature_current

Temperature thresholds:

Normal: < 65°C
Warning: 65-75°C
Critical: > 75°C

Fan Speed Monitoring

# Fan speeds (RPM)
chassis_fan_speed_current

Fan speed should remain consistent. Sudden drops may indicate hardware issues.

Container Logs and Monitoring

View Container Logs

Network Devices

# BNG logs
docker logs clab-lab-bng1
docker logs clab-lab-bng2

# Switch and OLT
docker logs clab-lab-switch
docker logs clab-lab-olt

# Transit router
docker logs clab-lab-tx

Telemetry Stack

# gNMIc collector
docker logs -f clab-lab-gnmic

# Prometheus
docker logs clab-lab-prometheus

# Grafana
docker logs clab-lab-grafana

Services

# RADIUS server
docker logs clab-lab-radius

# Subscriber devices
docker logs clab-lab-ont1
docker logs clab-lab-ont2

Container Resource Usage

# Real-time stats for all containers
docker stats

# Stats for specific container
docker stats clab-lab-bng1

# One-time snapshot
docker stats --no-stream

Alerting Best Practices

Recommended Alerts:

Device Down: port_oper_state == 0 on critical links
High CPU: system_cpu_total > 80 for 5 minutes
Memory Exhaustion: Memory utilization > 90%
Interface Errors: Non-zero error rates
BGP Session Down: Loss of established BGP peers
Subscriber Session Failures: Failed authentication attempts
Temperature Alert: Hardware temperature > 70°C
License Expiry: Nokia license approaching expiration

Troubleshooting Monitoring Issues

No Metrics in Prometheus

Check gNMIc is running: docker ps | grep gnmic
Verify gNMIc metrics endpoint: curl http://localhost:9273/metrics
Check Prometheus targets: http://localhost:9090/targets
Review gNMIc logs: docker logs clab-lab-gnmic

Grafana Shows No Data

Verify Prometheus data source connection in Grafana
Check time range in dashboard (default: last 6 hours)
Run test query in Prometheus UI first
Ensure dashboards are using correct metric names

gNMIc Connection Failures

Verify device gRPC port is accessible: netstat -tuln | grep 57400
Check credentials: admin/lab123
Confirm Docker socket is mounted: docker exec clab-lab-gnmic ls -l /var/run/docker.sock
Review device labels: docker inspect clab-lab-bng1 | grep clab-node-kind

Performance Optimization

Reduce Resource Usage:

Increase scrape interval from 5s to 10s or 15s
Reduce retention period in Prometheus
Disable unused subscriptions in gNMIc config
Limit metric cardinality by filtering unnecessary labels

Get Started

Deployment

Architecture

Configuration

Operations

Monitoring & Observability

Architecture Overview

Grafana Dashboard Access

Prometheus Metrics

Access Prometheus UI

Prometheus Configuration

Verify Metric Collection

gNMIc Telemetry Collector

Configuration Overview

Active Subscriptions

View gNMIc Logs

Key Metrics to Monitor

System Health Metrics

BNG-Specific Metrics

Network Performance Metrics

RADIUS Accounting Logs

Access RADIUS Logs

Device Health Monitoring

Temperature Monitoring

Fan Speed Monitoring

Container Logs and Monitoring

View Container Logs

Container Resource Usage

Alerting Best Practices

Troubleshooting Monitoring Issues

Performance Optimization

Build docs developers (and LLMs) love

Get Started

Deployment

Architecture

Configuration

Operations

Documentation Index

​Architecture Overview

​Grafana Dashboard Access

​Prometheus Metrics

​Access Prometheus UI

​Prometheus Configuration

​Verify Metric Collection

​gNMIc Telemetry Collector

​Configuration Overview

​Active Subscriptions

​View gNMIc Logs

​Key Metrics to Monitor

​System Health Metrics

​BNG-Specific Metrics

​Network Performance Metrics

​RADIUS Accounting Logs

​Access RADIUS Logs

​Device Health Monitoring

​Temperature Monitoring

​Fan Speed Monitoring

​Container Logs and Monitoring

​View Container Logs

​Container Resource Usage

​Alerting Best Practices

​Troubleshooting Monitoring Issues

​Performance Optimization

Build docs developers (and LLMs) love

Architecture Overview

Grafana Dashboard Access

Prometheus Metrics

Access Prometheus UI

Prometheus Configuration

Verify Metric Collection

gNMIc Telemetry Collector

Configuration Overview

Active Subscriptions

View gNMIc Logs

Key Metrics to Monitor

System Health Metrics

BNG-Specific Metrics

Network Performance Metrics

RADIUS Accounting Logs

Access RADIUS Logs

Device Health Monitoring

Temperature Monitoring

Fan Speed Monitoring

Container Logs and Monitoring

View Container Logs

Container Resource Usage

Alerting Best Practices

Troubleshooting Monitoring Issues

Performance Optimization