Skip to content
yanox BUILD SHARED TRUST

Restart, catch-up and snapshots

A member that restarts, falls behind, or joins from a snapshot never trusts the data it is given. It re-checks every block and certificate, and it re-executes every block it applies.

Catch-up

Section titled “Catch-up”

Gossip never re-delivers a missed proposal, so a member that falls behind cannot recover from live rounds. Instead, every 5 seconds it asks one connected peer for the next range of up to 50 blocks and checks each one in order:

Catching up, restarting, restoringIllustrative data

What you're seeing: Member C rejoins a three-member ledger with a threshold of 2. Every block it receives is checked, never trusted. Try a “What if” for a lying peer, a restart, and a snapshot restore.

  1. Fall behind Member C was offline while A and B, a threshold of 2 of 3, finalized blocks 101 to 140. Gossip never delivers a missed proposal again, so C cannot recover from the live round.
  2. Request a batch Every 5 seconds the lagging member asks one connected peer for the next range of up to 50 blocks, tip + 1 to tip + 50, over the dedicated block-range sync protocol.
  3. Check each block It checks each block in order, with the same rules as live consensus. It does not re-check message expiry: those messages were finalized before they expired. A Cardano reference ahead of its own L1 view pauses the batch rather than failing it.
    • passes: Height is the local tip + 1, and the block links to the local tip hash
    • passes: The proposer was a member, and the right leader for that height and view
    • passes: The messages root recomputes, and the consensus context matches
    • passes: The Cardano reference and observations agree with the member's own L1 view
    • passes: Every message is signed by a member at that height
    • not reached: The finality certificate has a threshold of valid member signatures
    • not reached: Re-executing the block gives a byte-identical state root
  4. Verify the certificate The finality certificate is verified in full: every signer must be a member at that height, and valid signatures must reach that height's threshold. The peer is not trusted.
    • passes: Height is the local tip + 1, and the block links to the local tip hash
    • passes: The proposer was a member, and the right leader for that height and view
    • passes: The messages root recomputes, and the consensus context matches
    • passes: The Cardano reference and observations agree with the member's own L1 view
    • passes: Every message is signed by a member at that height
    • passes: The finality certificate has a threshold of valid member signatures
    • not reached: Re-executing the block gives a byte-identical state root
  5. Re-execute The member applies the block itself and requires a byte-identical state root, then commits the block, its certificate, and its state in one atomic batch.
    • passes: Height is the local tip + 1, and the block links to the local tip hash
    • passes: The proposer was a member, and the right leader for that height and view
    • passes: The messages root recomputes, and the consensus context matches
    • passes: The Cardano reference and observations agree with the member's own L1 view
    • passes: Every message is signed by a member at that height
    • passes: The finality certificate has a threshold of valid member signatures
    • passes: Re-executing the block gives a byte-identical state root
  6. Rejoin consensus It repeats batch by batch until it reaches the peers' tip, then votes in live rounds again. If a peer stays ahead and the member makes no progress for 60 seconds, the node logs that it appears stalled, raises AppChainStalledEvent, and status shows stalled.

What if: A peer sends a forged root

A peer serves block 101 with a state root the block does not produce.

  1. Fall behind Member C was offline while A and B, a threshold of 2 of 3, finalized blocks 101 to 140. Gossip never delivers a missed proposal again, so C cannot recover from the live round.
  2. Request a batch Every 5 seconds the lagging member asks one connected peer for the next range of up to 50 blocks, tip + 1 to tip + 50, over the dedicated block-range sync protocol.
  3. Root mismatch Even if every earlier check passes, for example because faulty members signed the block, re-executing it gives a different root. The block is rejected and the batch stops; nothing is applied. A member never adopts a root it did not compute itself.
    • passes: Height is the local tip + 1, and the block links to the local tip hash
    • passes: The proposer was a member, and the right leader for that height and view
    • passes: The messages root recomputes, and the consensus context matches
    • passes: The Cardano reference and observations agree with the member's own L1 view
    • passes: Every message is signed by a member at that height
    • passes: The finality certificate has a threshold of valid member signatures
    • fails: Re-executing the block gives a byte-identical state root

What if: A block without a valid certificate

A peer serves a block whose certificate does not reach the threshold.

  1. Fall behind Member C was offline while A and B, a threshold of 2 of 3, finalized blocks 101 to 140. Gossip never delivers a missed proposal again, so C cannot recover from the live round.
  2. Request a batch Every 5 seconds the lagging member asks one connected peer for the next range of up to 50 blocks, tip + 1 to tip + 50, over the dedicated block-range sync protocol.
  3. Certificate rejected Too few valid member signatures at that height: the certificate check fails, and the block is rejected before it is executed.
    • passes: Height is the local tip + 1, and the block links to the local tip hash
    • passes: The proposer was a member, and the right leader for that height and view
    • passes: The messages root recomputes, and the consensus context matches
    • passes: The Cardano reference and observations agree with the member's own L1 view
    • passes: Every message is signed by a member at that height
    • fails: The finality certificate has a threshold of valid member signatures
    • not reached: Re-executing the block gives a byte-identical state root

What if: Restart

Member C is stopped and started again.

  1. What survives Persisted with each block: blocks and certificates, tip metadata, the state trie and root, message and query indexes, per-sender floors, per-view prepare locks, prepared certificates with their proposals, and membership epochs. Lost by design: the pending message pool, gossip state, and pending anchor rounds.
    On restart
    KeptLost
    blocks, certificates, state, indexespending messages
    vote locks, prepared certificatesgossip state
    membership epochs, sender floorspending anchor rounds
  2. Startup checks Startup re-verifies rather than trusts: the committed root must equal the tip block's root, the tip block must re-hash to the stored hash, and its certificate must carry a threshold of valid member signatures. Then catch-up fetches what was missed.
    • passes: Committed root equals the tip block's root
    • passes: Persisted membership epochs override static configuration
    • passes: Tip block re-hashes, and its certificate meets the threshold
    • passes: Own sender sequence floor restored

What if: Restore a snapshot

Member C starts from a member-signed snapshot instead of from height 1.

  1. Take a snapshot Member A writes a RocksDB checkpoint with a manifest it signs: chain id, tip height and block hash, state root, state identity, a hash of the membership epochs, and a SHA-256 of every file.
    ./yano.sh appchain state snapshot --url <api-base> --chain <id> --path <fresh server directory>
  2. Verify before opening Before the database is opened, C checks that the manifest was signed by a member and that every file hash matches. After opening, the tip, block hash, state root, identity, and membership must match the manifest, and the tip certificate must reach the threshold. Then C catches up the rest.
    • passes: Manifest signed by a configured member
    • passes: Every file hash matches the manifest
    • passes: Tip, root, identity, and membership match
    • passes: Tip certificate meets the threshold

What if: An edited snapshot

Someone changed a file in the checkpoint after it was signed.

  1. Take a snapshot Member A writes and signs a snapshot as before, and the copy is changed in transit.
  2. Refuse to start The file hashes no longer match the signed manifest, so the node refuses to start before it opens the database. A snapshot whose contents merely agree with themselves is also refused: the tip certificate must carry a threshold of member signatures.
    • passes: Manifest signed by a configured member
    • fails: Every file hash matches the manifest
    • not reached: Tip, root, identity, and membership match
    • not reached: Tip certificate meets the threshold
  • member node
  • authoritative store
  • verified and committed
  • rejected

A failed check rejects the block and stops the batch; nothing from it is applied. A Cardano reference ahead of the member’s own view of Cardano pauses the batch instead, until its view catches up. Message expiry is not checked again, because those messages were finalized before they expired.

If a peer stays ahead and the member makes no progress for 60 seconds, the node logs that the chain appears stalled, raises AppChainStalledEvent, and reports stalled in its status. A member stalls this way when it cannot apply the certified blocks, for example because its state machine computes a different root. See troubleshooting.

Restart

Section titled “Restart”

Everything a member needs to resume is written with each block, in one atomic batch:

Kept across a restartLost by design
Blocks and their finality certificates, tip metadataThe pending message pool
The state trie and state rootGossip deduplication and counters
Message and query indexes, per-sender sequence floorsPending anchor rounds
Prepare locks, prepared certificates and their proposalsActive round aggregation, rebuilt from persisted votes
Membership epochs and pending governance

A message that was accepted (202) but never reached another member is lost if its member restarts first. Submit it again after checking that it was not finalized.

At startup the member re-verifies rather than trusts:

  1. If it is restoring a snapshot, the snapshot manifest’s signature and file hashes, before the database is opened.
  2. The committed state root equals the root recorded in the tip block.
  3. Persisted membership epochs override the static configuration.
  4. The tip block re-hashes to the stored hash, and its certificate carries a threshold of valid member signatures at that height.
  5. The member’s own sender sequence floor is restored, so it never reuses a finalized sequence number.

Prepare locks survive a restart, so a member never votes for two different blocks in one view, even across a crash.

Snapshots

Section titled “Snapshots”

A snapshot is a cheap copy of one ledger’s database, taken with hard links, plus a manifest signed by the member that took it. The manifest binds the chain id, the tip height and block hash, the state root, the state identity, a hash of the membership epochs, and a SHA-256 of every file.

Terminal window
./yano.sh appchain state snapshot --url http://127.0.0.1:7070/api/v1 --chain orders-chain \
  --path /var/lib/yano/snapshots/orders-chain-1   # a fresh directory on the node

To restore, copy the snapshot directory to <yano.app-chain.storage.path>/<chain-id>/ on the new or recovering member and start it. The node then refuses to start unless:

  • the manifest was signed by a configured member, and every file hash matches, checked before the database is opened;
  • the restored tip height, block hash, state root, state identity, and membership epochs match the manifest; and
  • the tip block’s certificate carries a threshold of valid member signatures. A snapshot whose contents merely agree with themselves is rejected.

A restore does not replay history: it trusts the certified tip and the signed manifest, then catches up the blocks after the snapshot height with full verification. For onboarding a new member, this saves catching up from height 1.

Snapshots of a ledger’s database are different from authenticated snapshots, which are period datasets, such as epoch stake, proved under their own roots.

Next: Keys and trust.