Skip to content

5. Operate, diagnose and evolve a workflow

A validated YAML document is the beginning of deployment qualification, not its end. This chapter turns the first workflow into an operational checklist. Keep the complete language reference and upgrade preflight guide nearby.

Distinguish admission from business success

Section titled “Distinguish admission from business success”
ObservationMeaningNext action
HTTP 400 application rejectionThe submitted command failed ingress admission; it was not pooled. For a static admission rule, the body’s details names the rule, its deny code and, for a batch, the deciding writeCorrect its encoding, size or other stated invalidity and submit again
HTTP 503 admission unavailableThe admission callback could not completeInvestigate node warnings and plugin health; it is not business acceptance
HTTP 202The node accepted the submission into its processing pathWait for finalization, then inspect the receipt
Finalized accepted receiptThis source’s cascade committedInspect the resulting business state; external delivery may still be pending
Finalized rejected receiptThis source’s business writes and effects did not commitRead the code and failed step, correct the cause and submit a fresh signed message if appropriate
Finalized ADMISSION_RULE_* receiptAn admission rule refused a step; the whole cascade rolled backRead the failed step’s rule trace: the rule, clause and deny code say which requirement did not hold

Early admission cannot know all future state or remaining block capacity. A well-formed command can therefore still fail during execution. The host does not invalidate the whole finalized block because one cascade was rejected. Its terminal receipt remains, while the source’s business changes are rolled back.

Do not retry a finalized rejection using the same message ID: replay returns the original receipt. A fresh envelope/ID is a new attempt, not a bypass of business idempotency, authorization or one-use approval consumption. A capacity failure may succeed in a later block; a bad signature or invalid action needs a real correction.

Start from what the submission returned and follow the receipt. Each answer says where the outcome is recorded and whether a new message can help. The rejection-code table has every code.

Where did my command stop?Decision aid

What you're seeing: start from what your submission returned, then follow the receipt. Each answer says where the outcome is recorded and whether sending a new message can help.

What did the submission return?

  • HTTP 400 with a code

    Which code did the 400 carry?

    A 400 means the ingress member refused the message before pooling it. There is no receipt.

    • UNKNOWN_BINDING_SOURCE
      The topic is not a component’s topic

      Every component listens on its own ingress topic, <id>.command.v1 unless the document sets topic. A message on any other topic is refused.

      Recorded in
      Nothing: the message was not pooled.
      Will a retry help?
      Yes, on the right topic.
    • MALFORMED_SOURCE_COMMAND
      The body did not decode

      The component’s codec could not decode the body. Encode commands with the machine’s public codec, such as KvRegistryContract.put, not with arbitrary JSON.

      Recorded in
      Nothing: the message was not pooled.
      Will a retry help?
      Yes, once the encoding is fixed.
    • COMMAND_PAYLOAD_TOO_LARGE or COMMAND_WORK_EXCEEDED
      The command cannot fit

      Either a subscribed baseline event would exceed the event limit, or the command’s mandatory work exceeds the per-cascade allowance. Both limits are committed in the profile.

      Recorded in
      Nothing: the message was not pooled.
      Will a retry help?
      Only with a smaller command, or after a qualified profile change.
    • ADMISSION_RULE_* or EXPRESSION_CAPACITY_EXCEEDED
      A static admission rule refused it

      A rule that reads only the command, parameters, configuration and write content is checked at ingress too. The 400 body’s details names the rule, its deny code and, for a batch, the deciding write. EXPRESSION_CAPACITY_EXCEEDED means evaluating those rules ran out of work.

      Recorded in
      The HTTP response only.
      Will a retry help?
      Only with a command the rule allows.
    • Another code
      The machine refused it at admission

      The target machine’s own stateless admission check refused the command. Look the code up in that machine’s reference.

      Recorded in
      Nothing: the message was not pooled.
      Will a retry help?
      Yes, once the command is corrected.
  • HTTP 429
    The pending pool is full

    The ingress member’s pending pool is full, so the message was not kept or relayed.

    Recorded in
    Nothing: the message was not accepted.
    Will a retry help?
    Yes, after the backpressure clears.
  • HTTP 503
    Admission is unavailable

    The application’s admission check could not run, the chain is stopped or paused, or this node is not yet an active member (a joiner whose membership epoch is scheduled). This is not a business outcome.

    Recorded in
    The node’s logs and health, and memberActiveForNextBlock in its status.
    Will a retry help?
    Once the node is healthy again, or once it is an active member.
  • HTTP 202 and a message id

    What does the query composite/binding-receipt-v1/<message-id> return?

    Query with the source message id and empty parameters. A 202 only means one member queued the message.

    • An empty payload
      No receipt yet

      An empty payload means no receipt at the height the node answered from. The message may still be pending. It may also have expired, or been lost when its only holder restarted. Absence is not a rejection.

      Recorded in
      Nothing yet.
      Will a retry help?
      Not blindly. Poll with backoff first. A resubmission is a new message with a new id.
    • A receipt with status ACCEPTED

      Did the follow-up you expected happen?

      • Yes. I want to confirm it.
        Every step committed

        In an accepted receipt, steps that read PLANNED committed. Prove the receipt with the key from bindings receipt-key. External effects are delivered later and separately.

        Recorded in
        The receipt and the component state.
        Will a retry help?
        Not needed.
      • No. Its binding shows a failedClause from 0 to 7.
        A false condition skipped the binding

        The clause at that index was false, so the binding derived nothing. That is not an error: the rest of the cascade still committed.

        Recorded in
        [bindingId, failedClause] in the producing step’s conditions.
        Will a retry help?
        Only if the data changes so the condition holds.
      • No. Its binding is not in any step’s conditions.
        No event selected the binding

        A binding runs only when its source component emits its event. Check the component and the event id, not just the topic. Bindings that were never considered are not listed.

        Recorded in
        Its absence from every step’s conditions.
        Will a retry help?
        Not until the source emits that event.
      • No, and the source command changed nothing either.
        Accepted, but nothing changed

        Accepted means the cascade committed, not that state changed. A machine can approve a no-op, such as a second vote from the same member. Read the component’s state.

        Recorded in
        The component state, not the receipt.
        Will a retry help?
        Only with a command that changes something.
    • A receipt with status REJECTED

      Which kind of code is the receipt’s code?

      Read failedStepOrdinal first: it names the step that failed. Nothing in a rejected cascade committed.

      • An admission rule: ADMISSION_RULE_*
        An admission rule refused a step

        DENIED: a clause was false. ERROR: a clause or read could not be evaluated, for example an unguarded absent read. INPUT: a kernel value broke its declaration. The whole cascade rolled back.

        Recorded in
        rules of the failed step: [heldCount, [ruleId, failedClause, denyCode, writeIndex]].
        Will a retry help?
        Only once the input satisfies the rule.
      • The target refused: ADMISSION, MALFORMED_DERIVED_COMMAND or the machine’s own code
        The target refused the command

        The target machine’s codec, admission or decision refused the command. For a derived command, check what the mapping built.

        Recorded in
        The failed step’s code.
        Will a retry help?
        Yes, once the input or the mapping is corrected.
      • An evaluation error: EXPRESSION_*, MAPPING_*, FUNCTION_*, LOOKUP_KEY_*, EVENT_TYPE_ERROR or INVALID_UNICODE
        A condition or mapping could not be evaluated

        Unlike a false condition, an evaluation error rejects the cascade. For a condition, the binding’s failedClause points at the clause.

        Recorded in
        The failed step’s code, and its conditions.
        Will a retry help?
        Not with the same data and document.
      • A limit or budget: LIMIT_*, *_CAPACITY_EXCEEDED, *_WORK_EXCEEDED, *_TOO_LARGE, EFFECT_*
        A limit or budget ran out

        Cascade limits such as LIMIT_DEPTH and LIMIT_FANOUT fail the same way every time. Block budgets such as CAPACITY_EXCEEDED and CRYPTO_WORK_EXCEEDED reset each block. EXPRESSION_CAPACITY_EXCEEDED can be either. Work already spent is not refunded.

        Recorded in
        The receipt’s code.
        Will a retry help?
        For a block budget, a new message in a later block can succeed.
      • A conflict: REPLAY_OR_CONFLICT or CONSUMPTION_CONFLICT
        A replay or consumption conflict

        REPLAY_OR_CONFLICT: the source id could not be claimed. CONSUMPTION_CONFLICT: two steps of one cascade used the same one-use key.

        Recorded in
        The receipt’s code.
        Will a retry help?
        Replaying the same id returns this receipt. A new message is a new attempt.
      • A plugin contract violation: RESERVED_*, STATE_KEY_LIMIT, EVENT_MISSING_FIELD, UNDECLARED_WORK_REFERENCE
        A kernel broke its plugin contract

        A kernel wrote a reserved key or event, exceeded the state-key bound, used undeclared work, or emitted an event without a required field.

        Recorded in
        The failed step’s code.
        Will a retry help?
        No. Report it to the plugin’s maintainer.

Retain the submitted source message ID in your application. Query composite/binding-receipt-v1/<source-message-id-hex> through the ordinary chain query API. The canonical receipt contains the overall result, visited steps, emitted events and condition results. A planned step inside a rejected receipt is not a committed operation.

To discover its physical authenticated key without scanning state:

Terminal window
./yano.sh appchain bindings receipt-key "$SOURCE_MESSAGE_ID"

This offline command returns stateKeyHex and the receipt/key query paths. Alternatively query composite/binding-receipt-key-v1/<source-message-id-hex> with empty parameters. Its generic query response’s payloadHex is already the raw physical key—do not decode it as a second CBOR value.

Key discovery works even if no receipt exists. Request a state proof for the key and verify its presence/value against independently trusted chain identity, root and finality. It is not enough to trust the same server’s claimed root. An app-final proof is not automatically a Cardano-anchored proof; see anchors and verification.

  1. Confirm the source finalized on the expected chain and profile.
  2. Inspect the receipt’s overall status before looking at individual steps.
  3. Check the source instance and event ID, not just the command topic.
  4. Inspect the first false/error failedClause (zero-based). -1 usually means all clauses matched, but a pre-clause budget failure also uses it; read the rejection code and step together. Unvisited bindings have no invented trace.
  5. Verify mapping types and target admission/authorization. A successful source transition alone does not guarantee its derived target succeeds; an admission rule on the target may refuse it, and each step’s rules trace names that rule.
  6. Check event, work, fan-out, receipt and effect bounds. The rejection-code table distinguishes them.

Reproduce with the exact document/IR, plugin catalog and explicit context, using appropriate pre-block state, message order and height. For multi-block rehearsals, carry postState with --prior-result and advance exactly one height each time. Use messages: [] for intervening empty blocks. The approval tutorial demonstrates this progression.

Continuation files are editable, unauthenticated test fixtures. They are not node backups or trusted state exports and must not be imported into retained stores. Dry-run does not calculate the predecessor root for your next fixture.

The 64-KiB host message limit is not a promise that every 64-KiB command can produce every event or fan-out. Events add metadata; copied values, hashes, expressions and derived dispatch consume bounded work.

The default per-cascade work allowance is 1,048,576 units; the shared block allowance is 33,554,432. These are deterministic accounting units, not CPU milliseconds. For the reference source.v1 near-limit raw-body tee, 65,369 body bytes cost 392,650 cascade units and 196,278 shared units. The binding-work allowance alone could cover 170 such cascades; host block-byte/message-count limits and other work may permit fewer. In particular, chapter 1’s 1-MiB block limit cannot fit 170 such commands. This exact body boundary depends on metadata length; do not generalize it to every topic or sender layout.

An original source event with no subscribers does not incur shared event-decode work. An unbound source is not rejected merely because binding work is exhausted. Derived dispatch and preparation still consume shared work, even for an unbound target. Rejected attempts do not refund work, and the block budget resets for the next block.

Before fixing your profile, test typical and maximum payloads, longest paths, fan-out, false conditions, failing targets and crowded blocks. Include the trace itself: receipts are bounded too. Increasing one limit cannot establish that all other limits accommodate the workload. Limits are part of the committed program, not node-local knobs you can raise on a busy server.

A cascade can atomically commit an effect intent with its state changes. The external HTTP service, object store or other receiver does not participate in that state transaction. Delivery is at-least-once; implement receiver-side idempotency using the effect identity, observe outcomes and plan retries.

The procurement recipe’s test effect is demonstration plumbing, not a production webhook integration. Select a real installed executor, configure its security and credentials outside consensus data, and qualify failure/restart behavior. Dry-run prints intents; it never proves external delivery. See webhook effects.

Changing YAML is not hot reload. Sealing commits component settings and defaults, routes, quotas, generation heights and the binding program, including admission rules, their parameters and attachments, into a profile. Machine application versions and query subjects also affect identity. Even a seemingly additive bundle update can make a retained profile impossible to reconstruct.

For a chain created in governed mode, evolution requires an executable catalog that reproduces the necessary historical profiles, qualified target semantics, and the existing proposal/approval/readiness/activation protocol. Changed bindings require a new workflow generation; changed component configuration requires a new component generation. Changed rules, parameters or attachments change the binding program and need a new workflow generation too, for example an ADR-015 proposal that tightens a limit. Neither migrates incompatible stored genesis or state automatically.

Use the read-only preflight against candidate bundles:

Terminal window
./yano.sh appchain bindings profile-check \
--profiles retained-profiles.json --context context.json \
--plugins-directory /absolute/path/to/candidate/plugins

Here retained-profiles.json is a nonempty JSON array of canonical profile hex strings, genesis first, including the historical/target profiles being checked. Multiple profiles require the appropriate governed membership context. Follow the full preflight contract before constructing these inputs.

A running member reports the profile it currently executes through the composite/active-profile-v1 query. For the Chapter 4 devnet application (records-audit on port 7370), this writes a one-profile array:

Terminal window
curl -fsS -X POST \
"http://127.0.0.1:7370/api/v1/app-chain/chains/records-audit/query/composite/active-profile-v1" \
-H 'Content-Type: application/json' -d '{"paramsHex":""}' \
| jq '[.payloadHex]' > retained-profiles.json

Add an X-API-Key header if your node requires one for queries. The answer is one member’s unproven view of the current profile only: compare it across members and against your deployment records, and add the genesis and any earlier profiles from those records yourself.

profile-check does not cross-check the context against the profiles or a deployed chain. It uses context.json only to construct the candidate catalog, so a context with another chain’s identity can still pass. Take the context from the same trusted deployment records as the profiles.

Success means byte-exact profile reconstruction only. It does not prove replay, binary semantic equivalence, migration safety, historical proof correctness or effect-result compatibility. The current stock provider executes version 1.2.0, and IR or receipts written before ADR-031.3 (versions 1.1.0 and 1.0.0) fail decode with a “predates ADR-031.3” error. ADR-031.4 then changed the rule layout in place without changing that version: IR whose admission rules were written before it, and receipts that record a rule failure, fail decode with a “predates ADR-031.4” error. Re-create such experimental chains from their YAML; do not upgrade them in place or rewrite their identity markers. The upgrade reference steps through what profile-check does.

  • Archive the exact host/X artifacts, plugin catalog, YAML, IR, canonical profiles, sealed <app>.chain file and deployment identity; keep key custody separate.
  • Test accepted and rejected cascades, authorization failures, replay and independent receipt proofs with your real command codecs.
  • Compare finalized heights, roots and profile identities across members; exercise catch-up, restart and historical reads.
  • Qualify expected peak payloads and throughput, including rejected attempts.
  • Check product boundaries: derived writes do not automatically appear in every legacy projection, and derived audit commands are not independently signed message-attestation certificates.
  • Keep backups and rehearse upgrades without modifying retained production state.

Return to the learning path, continue with the retained-chain upgrade reference, or author the same documents with guided forms in chapter 6.