Goal: add or remove a member of a running application, recover from a
lost key, host another application on the same nodes, and upgrade a node.
Before you start: a live application from
Set up an application, with every existing operator able to
run one command on their own node.
Time: a few minutes per operator; each change waits for the chain to
make about ten blocks.
Outcome: the new member finalizes blocks with the others, and no chain
was reset.
Members are fixed at seal, but a governed application (the default) can
change them later. Every existing operator runs approve once; it does the
whole quorum-safe procedure on every chain of the application.
Add a member, step by stepIllustrative data
What you're seeing: dave joins the running supply application, which alice, bob and carol run with two signatures per block. The “What if” scenario shows a lost key.
RequestThe new operator, dave, runs request against the sealed file on his own machine. It makes his key and a member card. He sends the card to every existing operator.
./yano.sh appchain request supply.chain
ApproveEach existing operator runs approve once, in any order. It is safe to run again: it skips what is already done and waits for what is not.
./yano.sh appchain approve dave.member
Raise the thresholdTwo of three is a majority; two of four is not. So approve first raises the threshold to three and waits until enough operators approve and the chain makes ten more blocks. It refuses if fewer than three members are online, so growth cannot stall the chain.
AddThen it submits the member change and waits again until it is active. A chain makes blocks only when it has messages; on an idle chain, approve offers to send small heartbeat messages, or you send ordinary traffic.
RestartEach operator's approve adds dave as a peer in its own node configuration and restarts that node once.
Join and startDave runs join with the same sealed file and then start. His node catches up from the other members.
RemoveCarol lost her node directory. The other operators each run approve --remove carol. The chain keeps finalizing with alice and bob.
./yano.sh appchain approve --remove carol
RequestCarol makes a fresh node directory and runs request supply.chain, which creates a new key and card.
./yano.sh appchain request supply.chain
ApproveThe others approve the new card as for any new member, and carol joins and starts. With fixed sequencing (the default) the lead's key is the exception: the runtime refuses to remove the fixed proposer, so losing it means sealing a new application.
The new operator, dave, needs the same Yano X release and the sealed
supply.chain file. On his machine:
Terminal window
./yano.shappchainrequestsupply.chain
✓ You should see✓ Wrote ./dave.member (key id ee22fe112c2e69cb, …)
and Next: send dave.member to the existing operators.
Each existing operator, with their node running, then runs:
Terminal window
./yano.shappchainapprovedave.member
✓ You should see
Adding dave (key ee22fe112c2e69cb, 127.0.0.1:15450) to supply
orders: first raising the threshold from 2 to 3 (4 members)
orders: waiting for threshold 3 (it activates once enough operators approve and each chain makes 10 more blocks)
✓ orders: dave is a member
Restarting this node to connect to dave …
Check the key id against the one dave reads out to you before you approve.
Why the threshold rises. Two of three is a majority; two of four is not.
So while a three-member application grows, it runs at three of four for a
while, and one offline member stops it. approve refuses to raise the
threshold unless enough members are online.
Why it waits. A change activates ten blocks after enough operators
approve, and a chain only makes blocks when it has messages. On an idle
chain running ordered-log (the audit-log recipe), kv-registry
(owned-registry), or the stock approvals machine, approve offers to send
small heartbeat messages (--heartbeat in scripts); on an ordered log each
one is a permanent entry. Other chains need ordinary traffic.
Safe to run again. If approve is interrupted or times out, run it
again: it skips what is done and reports what is still waiting.
Approvals expire after 600 blocks.
When every operator has approved, dave joins and starts:
Terminal window
./yano.shappchainjoinsupply.chain
./yano.shappchainstart
./yano.shappchainstatussupply
join says you join later as dave.
✓ You should see four members online and members agree for every
chain.
approve lowers the threshold if needed, removes carol on every chain, drops
her as a peer, and restarts the operator’s node once. It lowers the threshold
only when the old one would be unsafe, so it does not return to the sealed
policy: after growing to four members (three signatures) and removing one, the
chain runs at three of three, and one offline member stops it. status shows
the current members N, T signatures. With sequencing: fixed (the default) the lead cannot be removed: the runtime refuses to
remove the fixed proposer. An application sealed with sequencing: rotating
has no fixed proposer, so its lead can be removed like any other member
(on a devnet the lead’s node still produces the Cardano devnet).
If carol loses her node directory, remove her first, then add her again:
The other operators each run approve --remove carol. The chain keeps
finalizing without her.
Carol makes a fresh node directory and runs request supply.chain under a
new name, for example --name carol-2, which creates a new key and
card. The sealed roster keeps the name carol for her old key, so
request refuses to reuse it.
The others run approve carol-2.member; carol runs join supply.chain and
start.
Adding first would raise the threshold to a level the lost key can never help
reach. With fixed sequencing the lead’s key is the exception: losing it
means sealing a new application, so the lead must back up their node
directory.
An operator can run several applications on one node. Each one is a separate
setup: the coordinator runs new and seal for it, and each operator runs
request and join with the new files. Keep each application’s files in a
folder of its own: the coordinator creates the draft with new documents --output documents-app, and each operator runs request with --output <folder>, so the new cards neither overwrite nor mix with the first
application’s cards. In an existing node directory, request reuses your
member key and addresses. Adding a chain to a running node
still needs a stop and start:
Terminal window
./yano.shappchainjoindocuments.chain
./yano.shappchainstop
./yano.shappchainstart
join refuses a chain id the node already runs, and node-wide settings that
differ from those of the applications it already hosts. On a devnet, the
applications on one node must share the same lead.
A chain’s state machines are fixed at its genesis. To add a new chain to an
existing application, seal a new application with the same operators.
start checks the plugin bundles against the ones the node accepted at
join. After installing a newer Yano X release, it refuses with
Plugin bundles changed since this node joined supply. Agree the upgrade with
the other operators first, then accept the new bundles:
Terminal window
./yano.shappchainjoinsupply.chain--accept-bundles
./yano.shappchainstart
join --accept-bundles lists the changed bundles and records the new set.
Once every node runs again, the Plugins: line of status must show the same
catalog fingerprint on every member.
Changing what a state machine computes is a consensus change, not a rolling
upgrade; see consensus rules.