Verify Tile Internals

The verify tile performs cryptographic signature verification and deduplication on incoming transactions before they proceed to the dedup and pack tiles.

Overview

The verify tile:

Verifies Ed25519 signatures on transactions
Performs high-availability (HA) deduplication
Validates transaction parsing
Filters gossip votes
Handles bundle transaction flows
Publishes verified transactions to downstream tiles

Input Sources

The verify tile accepts transactions from multiple sources:

#define IN_KIND_QUIC   (0UL)  // From QUIC tile (UDP/QUIC)
#define IN_KIND_BUNDLE (1UL)  // From bundle tile
#define IN_KIND_GOSSIP (2UL)  // From gossip (votes)
#define IN_KIND_TXSEND (3UL)  // From transaction send

Each input source has different processing characteristics and routing rules.

Architecture

Tile Structure

┌─────────────┐
│ QUIC Tile   │───┐
└─────────────┘   │
                  │
┌─────────────┐   │
│ Bundle Tile │───┤
└─────────────┘   │
                  ├──► ┌──────────────┐
┌─────────────┐   │    │ Verify Tile  │
│ Gossip Tile │───┤    │              │
└─────────────┘   │    │ - Parse txn  │
                  │    │ - Verify sig │
┌─────────────┐   │    │ - Dedup      │
│ Txsend Tile │───┘    │ - Filter     │
└─────────────┘        └──────┬───────┘
                              │
                              v
                       ┌──────────────┐
                       │  Dedup Tile  │
                       └──────────────┘

Round-Robin Load Balancing

Multiple verify tiles share load via round-robin:

ctx->round_robin_cnt = fd_topo_tile_name_cnt( topo, tile->name );
ctx->round_robin_idx = tile->kind_id;

Transactions are distributed based on:

Sequence number for QUIC and gossip packets
Tile 0 only for bundles (prevents interleaving)

Transaction Processing Pipeline

1. Before Fragment (Filtering)

The before_frag callback determines if this tile should process the transaction:

static int
before_frag( fd_verify_ctx_t * ctx,
             ulong             in_idx,
             ulong             seq,
             ulong             sig ) {
  // Bundle packets need special handling
  int is_bundle_packet = (ctx->in_kind[in_idx]==IN_KIND_BUNDLE && !sig);

  if( FD_LIKELY( is_bundle_packet || 
                 ctx->in_kind[in_idx]==IN_KIND_QUIC ) ) {
    // Round-robin for QUIC and bundle packets
    return (seq % ctx->round_robin_cnt) != ctx->round_robin_idx;
  } else if( FD_LIKELY( ctx->in_kind[in_idx]==IN_KIND_BUNDLE ) ) {
    // All bundles go to verify:0
    return ctx->round_robin_idx!=0UL;
  } else if( FD_LIKELY( ctx->in_kind[in_idx]==IN_KIND_GOSSIP ) ) {
    // Only process vote transactions from gossip
    return (seq % ctx->round_robin_cnt) != ctx->round_robin_idx ||
           sig!=FD_GOSSIP_UPDATE_TAG_VOTE;
  }
  return 0;
}

Bundle transactions must go through verify:0 to prevent interleaving of bundle streams, which would break atomicity guarantees.

2. During Fragment (Copy)

The during_frag callback copies transaction data:

static inline void
during_frag( fd_verify_ctx_t * ctx,
             ulong             in_idx,
             ulong             seq,
             ulong             sig,
             ulong             chunk,
             ulong             sz,
             ulong             ctl ) {

  ulong in_kind = ctx->in_kind[ in_idx ];
  
  if( FD_UNLIKELY( in_kind==IN_KIND_BUNDLE || 
                   in_kind==IN_KIND_QUIC || 
                   in_kind==IN_KIND_TXSEND ) ) {
    // Validate chunk is in valid range
    if( FD_UNLIKELY( chunk<ctx->in[in_idx].chunk0 || 
                     chunk>ctx->in[in_idx].wmark || 
                     sz>FD_TPU_RAW_MTU ) )
      FD_LOG_ERR(( "chunk %lu %lu corrupt", chunk, sz ));

    uchar * src = fd_chunk_to_laddr( ctx->in[in_idx].mem, chunk );
    uchar * dst = fd_chunk_to_laddr( ctx->out_mem, ctx->out_chunk );
    fd_memcpy( dst, src, sz );

    // Validate payload size
    fd_txn_m_t const * txnm = (fd_txn_m_t const *)dst;
    if( FD_UNLIKELY( txnm->payload_sz>FD_TPU_MTU ) ) {
      FD_LOG_ERR(( "txn payload size %hu exceeds max %lu", 
                   txnm->payload_sz, FD_TPU_MTU ));
    }
  }
}

3. After Fragment (Verify)

The after_frag callback performs the core verification:

static inline void
after_frag( fd_verify_ctx_t *   ctx,
            ulong               in_idx,
            ulong               seq,
            ulong               sig,
            ulong               sz,
            ulong               tsorig,
            ulong               _tspub,
            fd_stem_context_t * stem ) {

  // Parse transaction
  fd_txn_m_t * txnm = (fd_txn_m_t *)fd_chunk_to_laddr( 
    ctx->out_mem, ctx->out_chunk
  );
  fd_txn_t * txnt = fd_txn_m_txn_t( txnm );
  txnm->txn_t_sz = (ushort)fd_txn_parse( 
    fd_txn_m_payload(txnm), txnm->payload_sz, txnt, NULL
  );

  int is_bundle = !!txnm->block_engine.bundle_id;

  // Track bundle state
  if( FD_UNLIKELY( is_bundle & 
                   (txnm->block_engine.bundle_id!=ctx->bundle_id) ) ) {
    ctx->bundle_failed = 0;
    ctx->bundle_id     = txnm->block_engine.bundle_id;
  }

  // Fail entire bundle if any transaction failed
  if( FD_UNLIKELY( is_bundle & (!!ctx->bundle_failed) ) ) {
    ctx->metrics.verify_tile_result[
      FD_METRICS_ENUM_VERIFY_TILE_RESULT_V_BUNDLE_PEER_FAILURE_IDX
    ]++;
    return;
  }

  // Verify parse succeeded
  if( FD_UNLIKELY( !txnm->txn_t_sz ) ) {
    if( FD_UNLIKELY( is_bundle ) ) ctx->bundle_failed = 1;
    ctx->metrics.verify_tile_result[
      FD_METRICS_ENUM_VERIFY_TILE_RESULT_V_PARSE_FAILURE_IDX
    ]++;
    return;
  }

  // Verify signatures and check for duplicates
  ulong _txn_sig;
  int res = fd_txn_verify( 
    ctx, fd_txn_m_payload(txnm), txnm->payload_sz, 
    txnt, !is_bundle, &_txn_sig
  );
  
  if( FD_UNLIKELY( res!=FD_TXN_VERIFY_SUCCESS ) ) {
    if( FD_UNLIKELY( is_bundle ) ) ctx->bundle_failed = 1;

    if( FD_LIKELY( res==FD_TXN_VERIFY_DEDUP ) )
      ctx->metrics.verify_tile_result[
        FD_METRICS_ENUM_VERIFY_TILE_RESULT_V_DEDUP_FAILURE_IDX
      ]++;
    else
      ctx->metrics.verify_tile_result[
        FD_METRICS_ENUM_VERIFY_TILE_RESULT_V_VERIFY_FAILURE_IDX
      ]++;
    return;
  }

  // Publish verified transaction
  ulong realized_sz = fd_txn_m_realized_footprint( txnm, 1, 0 );
  ulong tspub = (ulong)fd_frag_meta_ts_comp( fd_tickcount() );
  fd_stem_publish( stem, 0UL, 0UL, ctx->out_chunk, 
                   realized_sz, 0UL, tsorig, tspub );
  ctx->out_chunk = fd_dcache_compact_next( 
    ctx->out_chunk, realized_sz, ctx->out_chunk0, ctx->out_wmark
  );

  ctx->metrics.verify_tile_result[
    FD_METRICS_ENUM_VERIFY_TILE_RESULT_V_SUCCESS_IDX
  ]++;
}

Bundle Handling

Bundles require special handling for atomicity:

Bundle State Machine

if( FD_UNLIKELY( is_bundle & 
                 (txnm->block_engine.bundle_id!=ctx->bundle_id) ) ) {
  ctx->bundle_failed = 0;
  ctx->bundle_id     = txnm->block_engine.bundle_id;
}

Each bundle has a unique ID
Track current bundle ID and failure state
If any transaction in bundle fails, mark entire bundle as failed

Bundle Deduplication

Bundle transactions are exempt from normal HA dedup checks. This allows users to send the same transaction both as part of a bundle (with a tip) and via the normal path (without a tip). The one with the tip will be packed. The dedup tile still performs full-bundle dedup to drop identical bundles.

int res = fd_txn_verify( 
  ctx, fd_txn_m_payload(txnm), txnm->payload_sz, 
  txnt, !is_bundle,  // Skip HA dedup for bundles
  &_txn_sig
);

Gossip Vote Handling

Gossip votes have a special code path:

else if( FD_UNLIKELY( ctx->in_kind[in_idx]==IN_KIND_GOSSIP ) ) {
  fd_gossip_update_message_t const * msg = 
    fd_chunk_to_laddr_const( ctx->in[in_idx].mem, chunk );
  fd_txn_m_t * dst = fd_chunk_to_laddr( ctx->out_mem, ctx->out_chunk );

  dst->payload_sz = (ushort)msg->vote->value->transaction_len;
  dst->block_engine.bundle_id = 0UL;
  dst->source_ipv4 = msg->vote->socket->is_ipv6 ? 0U : 
                     msg->vote->socket->ip4;
  dst->source_tpu = FD_TXN_M_TPU_SOURCE_GOSSIP;
  fd_memcpy( fd_txn_m_payload(dst), 
             msg->vote->value->transaction, 
             msg->vote->value->transaction_len );
}

Gossip votes are:

Extracted from gossip update messages
Marked with FD_TXN_M_TPU_SOURCE_GOSSIP
Processed through normal verification pipeline

Signature Verification

SHA-512 Context Pool

Each verify tile maintains SHA-512 contexts for signature verification:

for ( ulong i=0; i<FD_TXN_ACTUAL_SIG_MAX; i++ ) {
  fd_sha512_t * sha = fd_sha512_join( 
    fd_sha512_new( FD_SCRATCH_ALLOC_APPEND( 
      l, alignof(fd_sha512_t), sizeof(fd_sha512_t)
    ))
  );
  if( FD_UNLIKELY( !sha ) ) FD_LOG_ERR(( "fd_sha512_join failed" ));
  ctx->sha[i] = sha;
}

Pre-allocated SHA-512 contexts for each possible signature
Avoids allocation overhead during verification
Maximum of FD_TXN_ACTUAL_SIG_MAX signatures per transaction

Transaction Cache (tcache)

The verify tile uses a transaction cache for deduplication:

fd_tcache_t * tcache = fd_tcache_join( 
  fd_tcache_new( 
    FD_SCRATCH_ALLOC_APPEND( l, FD_TCACHE_ALIGN, 
                             FD_TCACHE_FOOTPRINT( tile->verify.tcache_depth, 0UL ) ), 
    tile->verify.tcache_depth, 0UL 
  )
);

ctx->tcache_depth   = fd_tcache_depth( tcache );
ctx->tcache_map_cnt = fd_tcache_map_cnt( tcache );
ctx->tcache_sync    = fd_tcache_oldest_laddr( tcache );
ctx->tcache_ring    = fd_tcache_ring_laddr( tcache );
ctx->tcache_map     = fd_tcache_map_laddr( tcache );

The tcache stores recent transaction signatures to detect duplicates.

Memory Management

Input Memory

Each input link has associated memory regions:

for( ulong i=0UL; i<tile->in_cnt; i++ ) {
  fd_topo_link_t * link = &topo->links[ tile->in_link_id[i] ];

  fd_topo_wksp_t * link_wksp = 
    &topo->workspaces[ topo->objs[link->dcache_obj_id].wksp_id ];
  ctx->in[i].mem = link_wksp->wksp;
  ctx->in[i].chunk0 = fd_dcache_compact_chunk0( ctx->in[i].mem, 
                                                 link->dcache );
  ctx->in[i].wmark  = fd_dcache_compact_wmark( ctx->in[i].mem, 
                                               link->dcache, link->mtu );
}

Output Memory

Verified transactions are copied to output workspace:

ctx->out_mem    = topo->workspaces[
  topo->objs[topo->links[tile->out_link_id[0]].dcache_obj_id].wksp_id
].wksp;
ctx->out_chunk0 = fd_dcache_compact_chunk0( 
  ctx->out_mem, topo->links[tile->out_link_id[0]].dcache
);
ctx->out_wmark  = fd_dcache_compact_wmark( 
  ctx->out_mem, topo->links[tile->out_link_id[0]].dcache, 
  topo->links[tile->out_link_id[0]].mtu
);
ctx->out_chunk  = ctx->out_chunk0;

Metrics

The verify tile tracks verification results:

static inline void
metrics_write( fd_verify_ctx_t * ctx ) {
  FD_MCNT_ENUM_COPY( VERIFY, TRANSACTION_RESULT, 
                     ctx->metrics.verify_tile_result );
  FD_MCNT_SET( VERIFY, GOSSIPED_VOTES_RECEIVED,  
               ctx->metrics.gossiped_votes_cnt );
}

Result Counters

V_SUCCESS_IDX - Successfully verified transactions
V_PARSE_FAILURE_IDX - Transaction parsing failures
V_VERIFY_FAILURE_IDX - Signature verification failures
V_DEDUP_FAILURE_IDX - Duplicate transactions detected
V_BUNDLE_PEER_FAILURE_IDX - Bundle failed due to peer transaction
GOSSIPED_VOTES_RECEIVED - Votes received via gossip

Security

Seccomp Filtering

The verify tile uses seccomp to restrict system calls:

static ulong
populate_allowed_seccomp( fd_topo_t const *      topo,
                          fd_topo_tile_t const * tile,
                          ulong                  out_cnt,
                          struct sock_filter *   out ) {
  populate_sock_filter_policy_fd_verify_tile( 
    out_cnt, out, (uint)fd_log_private_logfile_fd()
  );
  return sock_filter_policy_fd_verify_tile_instr_cnt;
}

File Descriptor Restrictions

Limited file descriptors allowed:

static ulong
populate_allowed_fds( fd_topo_t const *      topo,
                      fd_topo_tile_t const * tile,
                      ulong                  out_fds_cnt,
                      int *                  out_fds ) {
  ulong out_cnt = 0UL;
  out_fds[ out_cnt++ ] = 2; /* stderr */
  if( FD_LIKELY( -1!=fd_log_private_logfile_fd() ) )
    out_fds[ out_cnt++ ] = fd_log_private_logfile_fd();
  return out_cnt;
}

Only stderr and the log file are accessible.

Initialization

Privileged Init

Generates secure random seed for hashmap:

static void
privileged_init( fd_topo_t *      topo,
                 fd_topo_tile_t * tile ) {
  void * scratch = fd_topo_obj_laddr( topo, tile->tile_obj_id );
  FD_SCRATCH_ALLOC_INIT( l, scratch );
  fd_verify_ctx_t * ctx = FD_SCRATCH_ALLOC_APPEND( 
    l, alignof(fd_verify_ctx_t), sizeof(fd_verify_ctx_t)
  );
  FD_TEST( fd_rng_secure( &ctx->hashmap_seed, 8U ) );
}

Unprivileged Init

Sets up data structures:

Transaction cache (tcache)
SHA-512 contexts
Round-robin configuration
Input/output memory regions
Bundle state tracking

Performance Characteristics

Burst Size

#define STEM_BURST (1UL)

Processes one transaction at a time for:

Simpler error handling
More predictable latency
Easier bundle state tracking

Memory Footprint

static ulong
scratch_footprint( fd_topo_tile_t const * tile ) {
  ulong l = FD_LAYOUT_INIT;
  l = FD_LAYOUT_APPEND( l, alignof(fd_verify_ctx_t), 
                        sizeof(fd_verify_ctx_t) );
  l = FD_LAYOUT_APPEND( l, fd_tcache_align(), 
                        fd_tcache_footprint( tile->verify.tcache_depth, 0UL ) );
  for( ulong i=0; i<FD_TXN_ACTUAL_SIG_MAX; i++ ) {
    l = FD_LAYOUT_APPEND( l, fd_sha512_align(), fd_sha512_footprint() );
  }
  return FD_LAYOUT_FINI( l, scratch_align() );
}

Footprint includes:

Context structure
Transaction cache
SHA-512 contexts for signature verification

Development

Contributing

Internals

Verify Tile Internals

Overview

Input Sources

Architecture

Tile Structure

Round-Robin Load Balancing

Transaction Processing Pipeline

1. Before Fragment (Filtering)

2. During Fragment (Copy)

3. After Fragment (Verify)

Bundle Handling

Bundle State Machine

Bundle Deduplication

Gossip Vote Handling

Signature Verification

SHA-512 Context Pool

Transaction Cache (tcache)

Memory Management

Input Memory

Output Memory

Metrics

Result Counters

Security

Seccomp Filtering

File Descriptor Restrictions

Initialization

Privileged Init

Unprivileged Init

Performance Characteristics

Burst Size

Memory Footprint

​Overview

​Input Sources

​Architecture

​Tile Structure

​Round-Robin Load Balancing

​Transaction Processing Pipeline

​1. Before Fragment (Filtering)

​2. During Fragment (Copy)

​3. After Fragment (Verify)

​Bundle Handling

​Bundle State Machine

​Bundle Deduplication

​Gossip Vote Handling

​Signature Verification

​SHA-512 Context Pool

​Transaction Cache (tcache)

​Memory Management

​Input Memory

​Output Memory

​Metrics

​Result Counters

​Security

​Seccomp Filtering

​File Descriptor Restrictions

​Initialization

​Privileged Init

​Unprivileged Init

​Performance Characteristics

​Burst Size

​Memory Footprint

Overview

Input Sources

Architecture

Tile Structure

Round-Robin Load Balancing

Transaction Processing Pipeline

1. Before Fragment (Filtering)

2. During Fragment (Copy)

3. After Fragment (Verify)

Bundle Handling

Bundle State Machine

Bundle Deduplication

Gossip Vote Handling

Signature Verification

SHA-512 Context Pool

Transaction Cache (tcache)

Memory Management

Input Memory

Output Memory

Metrics

Result Counters

Security

Seccomp Filtering

File Descriptor Restrictions

Initialization

Privileged Init

Unprivileged Init

Performance Characteristics

Burst Size

Memory Footprint