QUIC Tile Internals

The QUIC tile provides a TPU (Transaction Processing Unit) server that handles incoming transactions clients request to be included in blocks. It supports both TPU/UDP and TPU/QUIC protocols.

Overview

The QUIC tile acts as a plain Tango producer:

Defragments multi-packet TPU streams from QUIC
Each frag_meta refers to a complete transaction
Requires dcache MTU at least as large as max serialized transaction size
Does not service network devices directly (relies on net tiles)
Arbitrary number of QUIC tiles can run
Each UDP flow must stick to one QUIC tile

Architecture

Tile Structure

The QUIC tile consists of:

┌──────────────┐
│   Net Tile   │
│  (AF_XDP)    │
└──────┬───────┘
       │ UDP/IP packets
       v
┌──────────────┐
│  QUIC Tile   │
│              │
│ - UDP fast   │
│ - QUIC slow  │
│ - Reassembly │
└──────┬───────┘
       │ Complete transactions
       v
┌──────────────┐
│ Verify Tile  │
└──────────────┘

Output Indices

#define OUT_IDX_VERIFY 0
#define OUT_IDX_NET    1

The QUIC tile has two output paths:

VERIFY: Complete transactions sent to verify tile
NET: Outgoing QUIC packets sent back to net tile

Transaction Processing

UDP Fast Path

Legacy TPU/UDP transactions are processed via legacy_stream_notify:

UDP transactions must fit in one packet (no fragmentation)
Entire packet received when notify is called
Published directly to verify tile
Simple, fast path with minimal overhead

static void
legacy_stream_notify( fd_quic_ctx_t * ctx,
                      uchar *         packet,
                      ulong           packet_sz,
                      uint            ipv4 ) {
  
  long                tspub    = ctx->now;
  fd_tpu_reasm_t *    reasm    = ctx->reasm;
  fd_stem_context_t * stem     = ctx->stem;
  fd_frag_meta_t *    mcache   = stem->mcaches[0];
  void *              base     = ctx->verify_out_mem;
  ulong               seq      = stem->seqs[0];

  int err = fd_tpu_reasm_publish_fast( 
    reasm, packet, packet_sz, mcache, 
    base, seq, tspub, ipv4, FD_TXN_M_TPU_SOURCE_UDP
  );
  
  if( FD_LIKELY( err==FD_TPU_REASM_SUCCESS ) ) {
    fd_stem_advance( stem, 0UL );
    ctx->metrics.txns_received_udp++;
  }
}

QUIC Path

QUIC protocol handling involves:

Packet Reception: IP packets received from net tile
QUIC Processing: fd_quic_process_packet() handles QUIC protocol
Stream Reassembly: Multi-packet transactions are reassembled
Publishing: Complete transactions sent to verify tile

The QUIC tile always publishes messages downstream, even without credits available. It ignores flow control of the downstream verify tile, enabling the QUIC tile to publish as fast as possible.

Stream Reception

The quic_stream_rx callback handles incoming QUIC stream data:

Fast Path (Single Packet)

When offset==0 and fin flag is set:

if( offset==0UL && fin ) {
  // Transaction fits in single QUIC packet
  if( FD_UNLIKELY( data_sz<FD_TXN_MIN_SERIALIZED_SZ ) ) {
    ctx->metrics.quic_txn_too_small++;
    return FD_QUIC_SUCCESS; // drop
  }
  if( FD_UNLIKELY( oversz ) ) {
    ctx->metrics.quic_txn_too_large++;
    return FD_QUIC_SUCCESS; // drop
  }
  // Publish directly
}

Slow Path (Fragmented)

Multi-packet transactions go through reassembly:

Fragments tracked per connection and stream
Reassembly buffer manages incomplete transactions
Published when complete transaction received

Connection Management

QUIC Limits

The tile configures QUIC with:

fd_quic_limits_t limits = {
  .conn_cnt      = tile->quic.max_concurrent_connections,
  .handshake_cnt = tile->quic.max_concurrent_handshakes,
  .conn_id_cnt                 = FD_QUIC_MIN_CONN_ID_CNT,
  .inflight_frame_cnt          = 64UL * max_concurrent_connections,
  .min_inflight_frame_cnt_conn = 32UL
};

Connection Finalization

When connections close, the quic_conn_final callback:

Counts abandoned reassembly streams
Updates metrics for active streams
Tracks abandoned transactions

static void
quic_conn_final( fd_quic_conn_t * conn,
                 void *           quic_ctx ) {
  fd_quic_ctx_t * ctx = quic_ctx;
  long abandon_cnt = fd_long_max( conn->srx->rx_streams_active, 0L );
  ctx->metrics.reasm_active    -= abandon_cnt;
  ctx->metrics.reasm_abandoned += (ulong)abandon_cnt;
}

Packet Filtering

Before Fragment Processing

The before_frag callback filters incoming packets:

static int
before_frag( fd_quic_ctx_t * ctx,
             ulong           in_idx,
             ulong           seq,
             ulong           sig ) {
  // Check protocol
  ulong proto = fd_disco_netmux_sig_proto( sig );
  if( FD_UNLIKELY( proto!=DST_PROTO_TPU_UDP && 
                   proto!=DST_PROTO_TPU_QUIC ) ) 
    return 1;

  // Round-robin load balancing
  ulong hash = fd_disco_netmux_sig_hash( sig );
  if( FD_UNLIKELY( (hash % ctx->round_robin_cnt) != ctx->round_robin_id ) ) 
    return 1;

  return 0;
}

Filtering criteria:

Protocol: Only TPU_UDP or TPU_QUIC
Load balancing: Round-robin based on hash

During Fragment Processing

The during_frag callback copies packet data:

static void
during_frag( fd_quic_ctx_t * ctx,
             ulong           in_idx,
             ulong           seq,
             ulong           sig,
             ulong           chunk,
             ulong           sz,
             ulong           ctl ) {
  void const * src = fd_net_rx_translate_frag( 
    &ctx->net_in_bounds[ in_idx ], chunk, ctl, sz
  );
  
  // FIXME: This copy could be eliminated by combining 
  // with decrypt operation
  fd_memcpy( ctx->buffer, src, sz );
}

The copy operation could potentially be optimized by combining it with the QUIC decrypt operation.

Transaction Validation

Size Checks

Both UDP and QUIC paths validate transaction sizes: Minimum size:

if( FD_UNLIKELY( data_sz<FD_TXN_MIN_SERIALIZED_SZ ) ) {
  ctx->metrics.udp_pkt_too_small++;
  return;
}

Maximum size:

if( FD_UNLIKELY( data_sz>FD_TPU_MTU ) ) {
  ctx->metrics.udp_pkt_too_large++;
  return;
}

Network Header Handling

For UDP packets:

ulong network_hdr_sz = fd_disco_netmux_sig_hdr_sz( sig );
if( FD_UNLIKELY( sz<=network_hdr_sz ) ) {
  ctx->metrics.udp_pkt_too_small++;
  return;
}
ulong data_sz = sz - network_hdr_sz;

For QUIC packets:

if( FD_UNLIKELY( sz<sizeof(fd_eth_hdr_t) ) ) 
  FD_LOG_ERR(( "QUIC packet too small" ));
uchar * ip_pkt = ctx->buffer + sizeof(fd_eth_hdr_t);
ulong   ip_sz  = sz - sizeof(fd_eth_hdr_t);

Metrics

The QUIC tile tracks comprehensive metrics:

Transaction Metrics

TXNS_RECEIVED_UDP - UDP transactions received
TXNS_RECEIVED_QUIC_FAST - Single-packet QUIC transactions
TXNS_RECEIVED_QUIC_FRAG - Multi-packet QUIC transactions
TXNS_OVERRUN - Reassembly buffer overruns
TXNS_ABANDONED - Abandoned incomplete transactions

Connection Metrics

CONNECTIONS_CREATED - New connections established
CONNECTIONS_CLOSED - Clean connection closes
CONNECTIONS_ABORTED - Aborted connections
CONNECTIONS_TIMED_OUT - Connection timeouts
CONNECTIONS_RETRIED - Retry connections

Packet Metrics

RECEIVED_PACKETS - Total packets received
RECEIVED_BYTES - Total bytes received
SENT_PACKETS - Total packets sent
SENT_BYTES - Total bytes sent
PKT_CRYPTO_FAILED - Decryption failures
PKT_NO_CONN - Packets for unknown connections

Frame Metrics

RECEIVED_FRAMES - Frames received by type
FRAME_FAIL_PARSE - Frame parsing failures
ACK_TX - ACK frames transmitted

static inline void
metrics_write( fd_quic_ctx_t * ctx ) {
  FD_MCNT_SET( QUIC, TXNS_RECEIVED_UDP,       
               ctx->metrics.txns_received_udp );
  FD_MCNT_SET( QUIC, TXNS_RECEIVED_QUIC_FAST, 
               ctx->metrics.txns_received_quic_fast );
  FD_MCNT_SET( QUIC, TXNS_RECEIVED_QUIC_FRAG, 
               ctx->metrics.txns_received_quic_frag );
  
  FD_MGAUGE_SET( QUIC, TXN_REASMS_ACTIVE,       
                 (ulong)fd_long_max( ctx->metrics.reasm_active, 0L ) );
  
  FD_MCNT_SET( QUIC, RECEIVED_PACKETS, 
               ctx->quic->metrics.net_rx_pkt_cnt );
  FD_MCNT_SET( QUIC, SENT_PACKETS,     
               ctx->quic->metrics.net_tx_pkt_cnt );
  
  // ... additional metrics
}

Performance Characteristics

Busy Polling

The before_credit callback services QUIC without sleeping:

static inline void
before_credit( fd_quic_ctx_t *     ctx,
               fd_stem_context_t * stem,
               int *               charge_busy ) {
  ctx->stem = stem;
  long now = fd_clock_now( ctx->clock );
  ctx->now = now;
  *charge_busy = fd_quic_service( ctx->quic, now );
}

Continuously polls for QUIC events
No blocking or sleeping
Updates current time for QUIC protocol
Services connections and handles retransmissions

Memory Layout

Scratch memory footprint includes:

static inline ulong
scratch_footprint( fd_topo_tile_t const * tile ) {
  ulong out_depth = tile->quic.out_depth;
  ulong reasm_max = tile->quic.reasm_cnt;
  fd_quic_limits_t limits = quic_limits( tile );
  
  ulong l = FD_LAYOUT_INIT;
  l = FD_LAYOUT_APPEND( l, alignof(fd_quic_ctx_t), 
                        sizeof(fd_quic_ctx_t) );
  l = FD_LAYOUT_APPEND( l, fd_quic_align(), 
                        fd_quic_footprint( &limits ) );
  l = FD_LAYOUT_APPEND( l, fd_tpu_reasm_align(), 
                        fd_tpu_reasm_footprint( out_depth, reasm_max ) );
  return FD_LAYOUT_FINI( l, scratch_align() );
}

Configuration

Tile Configuration

Key configuration parameters:

max_concurrent_connections - Maximum simultaneous QUIC connections
max_concurrent_handshakes - Maximum simultaneous handshakes
out_depth - Output queue depth
reasm_cnt - Maximum concurrent reassemblies
round_robin_cnt - Number of QUIC tiles for load balancing
round_robin_id - This tile’s ID in round-robin pool

Keylog Support

For debugging, QUIC supports keylog output:

#define FD_QUIC_KEYLOG_FLUSH_INTERVAL_NS ((long)100e6)

Keylog allows decrypting QUIC traffic in Wireshark for analysis.

Development

Contributing

Internals

Overview

Architecture

Tile Structure

Output Indices

Transaction Processing

UDP Fast Path

QUIC Path

Stream Reception

Fast Path (Single Packet)

Slow Path (Fragmented)

Connection Management

QUIC Limits

Connection Finalization

Packet Filtering

Before Fragment Processing

During Fragment Processing

Transaction Validation

Size Checks

Network Header Handling

Metrics

Transaction Metrics

Connection Metrics

Packet Metrics

Frame Metrics

Performance Characteristics

Busy Polling

Memory Layout

Configuration

Tile Configuration

Keylog Support

​Overview

​Architecture

​Tile Structure

​Output Indices

​Transaction Processing

​UDP Fast Path

​QUIC Path

​Stream Reception

​Fast Path (Single Packet)

​Slow Path (Fragmented)

​Connection Management

​QUIC Limits

​Connection Finalization

​Packet Filtering

​Before Fragment Processing

​During Fragment Processing

​Transaction Validation

​Size Checks

​Network Header Handling

​Metrics

​Transaction Metrics

​Connection Metrics

​Packet Metrics

​Frame Metrics

​Performance Characteristics

​Busy Polling

​Memory Layout

​Configuration

​Tile Configuration

​Keylog Support

Overview

Architecture

Tile Structure

Output Indices

Transaction Processing

UDP Fast Path

QUIC Path

Stream Reception

Fast Path (Single Packet)

Slow Path (Fragmented)

Connection Management

QUIC Limits

Connection Finalization

Packet Filtering

Before Fragment Processing

During Fragment Processing

Transaction Validation

Size Checks

Network Header Handling

Metrics

Transaction Metrics

Connection Metrics

Packet Metrics

Frame Metrics

Performance Characteristics

Busy Polling

Memory Layout

Configuration

Tile Configuration

Keylog Support