> For the complete documentation index, see [llms.txt](https://docs.zetrix.com/zetrix-l2-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zetrix.com/zetrix-l2-documentation/architecture/system-components.md).

# System Components

Each component below documents its **Purpose, Responsibilities, Inputs, Outputs, Dependencies, Internal workflow, Interaction with other components, Deployment, HA, Scalability, Security, Monitoring, Failure scenarios, Recovery, Performance, and Best practices**, per the platform documentation standard.

***

## Z2 Node

### Purpose

The Z2 Node is the execution engine of the L2. It maintains the L2 ledger, runs the EVM, holds the mempool, serves JSON-RPC to clients, and participates in **QBFT** consensus for block production.

### Responsibilities

* Accept and validate incoming transactions; maintain the mempool.
* Execute transactions on the EVM and produce blocks.
* Participate in QBFT consensus with peer nodes.
* Serve JSON-RPC / WebSocket RPC to wallets, dApps, Explorer, and tooling.
* Maintain and serve chain state and historical data.

### Inputs / Outputs

| Inputs                                                                           | Outputs                                   |
| -------------------------------------------------------------------------------- | ----------------------------------------- |
| Signed transactions (RPC), peer blocks/consensus messages, deposits from Relayer | Blocks, state, RPC responses, events/logs |

### Dependencies

* Peer Z2 Nodes (QBFT quorum).
* Sequencer (batch ordering upstream/downstream depending on flow).
* Relayer (L1→L2 deposit injection).

### Network Topology & Consensus (QBFT)

Z2 execution nodes run **go-quorum with QBFT (Byzantine Fault Tolerant)** consensus to produce the canonical block stream, and a **second, independent EVM (Besu) re-executes** the batched blocks for **cross-client** verification (see [Z2 Reexecutor](#z2-reexecutor)). QBFT is a proof-of-authority BFT protocol tolerating up to `f` faulty nodes in a network of `3f + 1`.

**Current deployment: 5 go-quorum QBFT nodes.** With `n = 5`, the network tolerates `f = 1` Byzantine/faulty node (`3(1)+1 = 4 ≤ 5`), requiring a quorum of `ceil(2n/3) = 4` nodes to finalize a block.

> \[!NOTE] **Deliberate two-client design.** The canonical block stream is produced by **go-quorum (QBFT)**; verification re-execution runs on **Besu**. Using two different clients means a client-specific bug in one is caught by the other — a stronger guarantee than single-client replay.

```mermaid
flowchart LR
    N1[go-quorum Node 1] --- N2[go-quorum Node 2]
    N2 --- N3[go-quorum Node 3]
    N3 --- N4[go-quorum Node 4]
    N4 --- N5[go-quorum Node 5]
    N5 --- N1
    N1 --- N3
    N2 --- N4
    N3 --- N5
    N4 --- N1
    N5 --- N2
```

### Synchronization

* New nodes sync from peers (full sync of blocks and state).
* QBFT provides immediate finality per block (no probabilistic reorg under honest quorum).

### RPC Services

* `eth_*` JSON-RPC methods (EVM standard).
* WebSocket subscriptions (`eth_subscribe`) for new heads, logs, pending tx.
* Served behind the AWS NLB (see [Networking](/zetrix-l2-documentation/network-and-deployment/networking.md#networking)).

### Deployment

* 5 go-quorum QBFT nodes (Testnet); Besu runs as the independent Reexecutor.
* Each node exposes RPC and P2P ports (see [Appendix ports](/zetrix-l2-documentation/reference/appendix.md#ports-reference)).

### High Availability

* 5-node QBFT quorum survives loss of 1 node with no downtime.
* RPC fronted by NLB with health checks; unhealthy nodes are removed from rotation.

### Scalability

* **Read scaling:** add non-validating RPC/full nodes behind the load balancer.
* **Write scaling:** bounded by QBFT quorum; vertical scaling of validator nodes and mempool tuning.

### Security

* P2P restricted to known validator peers.
* RPC exposure limited to safe method sets on public endpoints; admin/debug namespaces restricted to internal networks.

### Monitoring

* Block height, peer count, QBFT round changes, mempool size, RPC latency/error rate.

### Failure Scenarios & Recovery

| Failure         | Effect                      | Recovery                               |
| --------------- | --------------------------- | -------------------------------------- |
| 1 node down     | No downtime (quorum intact) | Restart / redeploy node; resync        |
| 2+ nodes down   | Consensus halts             | Restore quorum; investigate root cause |
| Corrupted state | Node cannot validate        | Resync from healthy peers / snapshot   |

### Performance Considerations

* I/O-bound on state DB — prefer NVMe SSD.
* Tune mempool and RPC worker pools for expected TPS.

### Best Practices

* Keep an odd, `3f+1`-shaped validator set.
* Separate public RPC nodes from consensus validators.
* Automate snapshot backups of chain data.

***

## Z2 Sequencer

### Purpose

The Sequencer orders incoming L2 transactions, drives block production, assembles them into **batches**, and posts each ordered batch to the L1 **Sequencer Inbox** — with the batch data made available via the DAC (AnyTrust), or as L1 calldata on fallback.

### Responsibilities

* Receive ordered transactions and determine canonical ordering; drive block production.
* Create batches (compress and package transactions + metadata).
* Post the ordered batch to the **L1 Sequencer Inbox**.
* Make batch data available: the **DAC** attests it in the normal path; **fall back to posting full batch data as L1 calldata** if the committee can't reach its attestation threshold.
* **Self-healing batch posting** — retry/recover posting without manual intervention.

> \[!NOTE] AnyTrust: the Sequencer keeps data off-chain via the DAC in the normal path but **falls back to L1 calldata** so data is never silently lost. See [Data Availability Committee (DAC) — AnyTrust](/zetrix-l2-documentation/architecture/trust-model.md#data-availability-committee-dac--anytrust).

### Deployment (clarification)

* **3-instance cluster with active/standby failover and self-healing batch posting** (matches the "3 servers" figure).

### Batch Creation

A batch bundles a contiguous range of L2 transactions with metadata (batch number, parent reference, state root, timestamp). Batching amortizes L1 posting cost across many transactions.

### Batch Submission & Interaction with L1

```mermaid
sequenceDiagram
    participant Seq as Sequencer
    participant L1 as Zetrix L1 (inbox)
    participant DAC as DAC
    participant Prop as Proposer
    participant Comm as Committee
    Seq->>Seq: order tx, build batch
    Seq->>L1: post batch commitment
    Seq->>DAC: publish tx data
    Prop->>L1: propose assertion (reads inbox + DAC)
    Comm->>DAC: fetch data, re-execute
    Comm->>L1: sign assertion (3-of-5) → soft confirmation
```

### Inputs / Outputs

| Inputs                          | Outputs                                         |
| ------------------------------- | ----------------------------------------------- |
| Ordered transactions from nodes | Batches, batch metadata, L1 submission payloads |

### Dependencies

* Z2 Nodes (transaction source).
* Committee (validation).
* Proposer / L1 (settlement).
* Refueler (gas top-up of Sequencer EOA).

### Deployment

* **Current deployment: 3 servers.** Active/standby configuration for HA; only one Sequencer is canonical at a time.

### High Availability

* 3 servers enable active + warm-standby.
* Leader election / failover promotes a standby if the active Sequencer fails.

> \[!WARNING] Two active Sequencers producing divergent orderings is a **split-brain** hazard. Failover must guarantee a single canonical Sequencer at any time (fencing / lease-based leadership).

### Scalability

* Vertical scaling for ordering throughput.
* Batch size / frequency tuning to balance latency vs. L1 cost.

### Security

* Sequencer signing key must be protected (HSM / KMS).
* Restricted network access; only authenticated pipeline peers.

### Monitoring

* Batch production rate, batch size, submission latency, L1 posting success, Sequencer EOA balance.

### Failure Scenarios & Recovery

| Failure                | Effect           | Recovery                                |
| ---------------------- | ---------------- | --------------------------------------- |
| Active Sequencer crash | Ordering pauses  | Promote standby; resume from last batch |
| L1 submission failure  | Batches queue    | Retry with backoff; alert; drain queue  |
| Low gas on EOA         | Submissions fail | Refueler auto top-up; alert             |

### Performance Considerations

* Batch cadence directly affects soft-confirmation latency and L1 cost.

### Best Practices

* Idempotent batch submission with sequence numbers.
* Monitored, automated failover with fencing.

***

## Z2 Validator

### Purpose

Validators form the **Committee** that validates batches, produces signatures for **soft confirmation**, and — via the **Aggregator** role — condenses Committee attestations into compact proofs.

> \[!NOTE] In the current deployment, each Committee server runs a **Z2 Validator + Z2 Reexecutor** pair together, operating in the **Committee and Aggregator** role. The Reexecutor provides the deterministic replay the Validator relies on before signing.

### Committee Role

* Fetch batch data from the DAC and independently re-execute (via Reexecutor).
* **Verify and sign the assertion created by the Proposer** — ordering, execution, resulting state root.
* Require **3-of-5** signatures to soft-confirm.
* **Manage the DAC** (temporary L2 data availability) mapped to the Sequencer's commitment.

### Aggregator Role

* Collect Committee signatures.
* Aggregate the **3-of-5** signatures into a single compact attestation.
* Provide the aggregated attestation downstream (to L1 / clients).

### Deployment Note

* **5 committee bundles, each deployed on a separate server; 3-of-5 required for voting.**

### Validation Process

```mermaid
flowchart TD
    A["Proposer proposes<br/>assertion"] --> B["Fetch batch data<br/>from DAC"]
    B --> C["Re-execute via<br/>Reexecutor"]
    C --> D{"State root matches<br/>assertion?"}
    D -->|Yes| E["Sign assertion"]
    D -->|No| F["Reject / flag"]
    E --> G["Send signature<br/>to Aggregator"]
    G --> H{"3-of-5 collected?"}
    H -->|Yes| I["Aggregated signature<br/>→ soft confirmation"]
    H -->|No| J["Wait for 3-of-5"]
```

### Signing the Assertion & Soft Confirmation

Each Validator signs the **Proposer's assertion** only after fetching the data from the DAC and independently re-executing it. The **aggregated 3-of-5** signature from the Committee constitutes the **soft confirmation** delivered to users.

### Inputs / Outputs

| Inputs                 | Outputs                                                             |
| ---------------------- | ------------------------------------------------------------------- |
| Batches from Sequencer | Per-validator signatures; aggregated attestation; soft confirmation |

### Dependencies

* Sequencer (batch source).
* Reexecutors (deterministic verification support).
* Refueler (gas top-up of Committee EOAs).

### Deployment

* **Current deployment: 5 servers**, each running a **Z2 Validator + Z2 Reexecutor** pair in the **Committee and Aggregator** role.

### High Availability

* 5-member Committee tolerates member outages while retaining quorum.
* Aggregator role can be re-assigned if the current Aggregator fails.

### Scalability

* Add Committee members to increase decentralization (trade-off: signature/coordination overhead — mitigated by aggregation).

### Security

* Each Validator holds an independent signing key (HSM/KMS recommended).
* Independent validation prevents blind co-signing.

### Monitoring

* Signature participation rate, validation latency, disagreement/flag events, member liveness, EOA balances.

### Failure Scenarios & Recovery

| Failure          | Effect                | Recovery                                      |
| ---------------- | --------------------- | --------------------------------------------- |
| 1–2 members down | Quorum may hold       | Restart members; investigate                  |
| Quorum lost      | No soft confirmations | Restore members; fall back to L1 finality     |
| Aggregator down  | No aggregated sig     | Reassign Aggregator or submit individual sigs |

### Performance Considerations

* Validation cost dominated by re-execution; benefits from Reexecutor offloading.

### Best Practices

* Keep validation logic byte-for-byte identical to node execution.
* Diverse hosting to avoid correlated failure.

***

## Z2 Reexecutor

### Purpose

The Reexecutor is a **second, independent EVM** that deterministically replays the batched blocks to verify the resulting state — providing **cross-client verification** of the canonical block stream and underpinning the Validator/Committee and fraud-detection process.

> \[!NOTE] The independent replay runs on **Besu**, deliberately a **different client** from the canonical execution client (**go-quorum**, see [Z2 Node](#z2-node)). Cross-client re-execution catches client-specific bugs that same-client replay would miss.

### Responsibilities

* Replay each transaction in a batch against the prior state.
* Produce the resulting state root deterministically.
* Compare against the claimed/asserted state root.
* Report verification results to Validators.

### Transaction Replay & Deterministic Execution

Given identical batch data, prior state, and EVM rules, re-execution is deterministic — the same inputs must yield the same state root. This determinism is the foundation of fraud detection.

```mermaid
flowchart LR
    IN["Batch + prior state"] --> EXEC["Deterministic<br/>EVM replay"]
    EXEC --> ROOT["Computed<br/>state root"]
    ROOT --> CMP{"== claimed root?"}
    CMP -->|yes| OK["Verified"]
    CMP -->|no| BAD["Discrepancy —<br/>flag fraud"]
```

### Verification & Interaction with Validator

The Reexecutor is the "second opinion" for Validators: Validators rely on Reexecutor output to sign with confidence, and Watchers run the same replay independently to detect fraud.

### Inputs / Outputs

| Inputs                  | Outputs                                   |
| ----------------------- | ----------------------------------------- |
| Batch data, prior state | Computed state root, verification verdict |

### Dependencies

* Batch/state data availability.
* Validators (consumer of verdicts).

### Deployment

* Deployed alongside Validators; also packaged as a standalone Docker image for external Watchers (`zetrixchain/z2-reexecutor:latest`).

### High Availability

* Stateless w\.r.t. long-term storage (recomputes from inputs) — trivially horizontally scalable.

### Scalability

* Scale horizontally: run many Reexecutor instances in parallel across batches.

### Security

* Must use the exact EVM ruleset/version as production nodes to avoid false positives/negatives.

### Monitoring

* Replay throughput, discrepancy count, replay latency, version/ruleset hash.

### Failure Scenarios & Recovery

| Failure          | Effect                          | Recovery                        |
| ---------------- | ------------------------------- | ------------------------------- |
| Reexecutor down  | Reduced verification redundancy | Restart; scale replicas         |
| Ruleset mismatch | False discrepancies             | Pin exact EVM version; redeploy |

### Performance Considerations

* CPU-bound; benefits from parallelism across batches.

### Best Practices

* Pin and verify the EVM ruleset hash across all Reexecutors.
* Run independent Reexecutors for Watchers (do not share infra with Validators).

***

## Main Proposer Validator

### Purpose

The Main Proposer Validator proposes and submits **assertions** (L2 state commitments) to Zetrix L1, bridging verified L2 state to L1 settlement. It runs as a **Z2 Validator + Z2 Reexecutor** pair operating in the **Proposer and Watcher** role.

> \[!NOTE] The Main Proposer node also runs the internal **Watcher** role: it independently re-executes (via its co-located Reexecutor) and can dispute mismatched data. This is the same role that external parties run (see [Watcher](#watcher)) — the internal Proposer is simply the first, always-on watcher. The 1-of-N safety assumption still depends on **independent external** watchers.

### Assertion Proposal & Submission

```mermaid
sequenceDiagram
    participant L1 as Zetrix L1 (inbox)
    participant DAC as DAC
    participant Prop as Main Proposer
    participant Comm as Committee
    Prop->>L1: read batch commitment
    Prop->>DAC: read batch data
    Prop->>Prop: compute resulting state root
    Prop->>L1: propose assertion (root + commitment ref)
    Comm->>L1: verify & sign (3-of-5) → soft confirmation
    Note over L1: challenge window opens
```

### Responsibilities

* Read the batch commitment (L1 inbox) and data (DAC), and compute the resulting state.
* **Propose** the assertion to the L1 contracts (the Committee then soft-confirms it).
* Run an internal **Watcher** role (re-execute and dispute).
* Track challenge windows and finalization status.

### Inputs / Outputs

| Inputs                                 | Outputs                   |
| -------------------------------------- | ------------------------- |
| Verified state roots, batch references | L1 assertion transactions |

### Dependencies

* Reexecutors/Validators (verified input).
* Zetrix L1 (settlement target).
* Refueler (gas top-up of Proposer EOA).

### Deployment

* **Current deployment: 1 server** (Z2 Validator + Z2 Reexecutor, Proposer + Watcher role).

> \[!WARNING] A single Proposer is a **liveness** single point (assertions stall if it is down) but **not** a safety single point — an invalid assertion is caught by Watchers. Production should add standby Proposers with failover.

### High Availability

* Recommend warm standby Proposer with leader election for production.

### Scalability

* Single logical Proposer suffices; throughput bound by L1 posting cadence, not Proposer count.

### Security

* Proposer key is high-value — protect with HSM/KMS.
* Monitor Proposer EOA gas balance (Refueler-managed).

### Monitoring

* Assertion submission rate/success, L1 confirmation, challenge status, EOA balance.

### Failure Scenarios & Recovery

| Failure            | Effect                              | Recovery                     |
| ------------------ | ----------------------------------- | ---------------------------- |
| Proposer down      | Assertions stall (finality delayed) | Failover to standby; restart |
| L1 posting failure | Assertions queue                    | Retry/backoff; alert         |
| Low gas            | Submissions fail                    | Refueler top-up              |

### Best Practices

* Add standby Proposer before mainnet.
* Alert aggressively on assertion staleness.

***

## Watcher

### Purpose

Watchers provide the **safety** guarantee of the optimistic model: they independently re-execute batches, compare against on-chain assertions, and **dispute** invalid assertions during the challenge window.

### Independent Verification

A Watcher runs its own Validator + Reexecutor stack, reads the L1 assertion, **fetches the batch data from the DAC**, re-executes, and compares state roots — trusting nothing from the Sequencer, Committee, or Proposer.

### Incentive

External users can participate as Watchers to detect any mismatch posted by the main validator, and **earn a reward from a successful dispute** by being an honest validator. This reward is what makes the 1-of-N honest assumption economically self-sustaining.

### Dispute Process

```mermaid
flowchart TD
    A["Read L1 assertion"] --> B["Fetch batch data<br/>from DAC"]
    B --> C["Independently<br/>re-execute"]
    C --> D{"Root matches?"}
    D -->|yes| E["No action"]
    D -->|no| F["Submit dispute<br/>to L1"]
    F --> G["L1 dispute<br/>game resolves"]
    G --> H["Honest watcher<br/>earns reward"]
```

### External / Third-Party Watcher Deployment

Anyone can operate a Watcher — this is by design and strengthens the network's safety. Third parties run the published Docker images with no need for permission.

### Docker Deployment

**Available images:**

* `zetrixchain/z2-validator:latest`
* `zetrixchain/z2-reexecutor:latest`

Example (illustrative) compose fragment:

```yaml
# docker-compose.watcher.yml
services:
  reexecutor:
    image: zetrixchain/z2-reexecutor:latest
    environment:
      - L1_RPC=https://<zetrix-l1-rpc>
      - L2_RPC=https://z2-test-node.zetrix.com   # production: https://z2-node.zetrix.com
      - MODE=watcher
    restart: unless-stopped

  validator:
    image: zetrixchain/z2-validator:latest
    environment:
      - L1_RPC=https://<zetrix-l1-rpc>
      - L2_RPC=https://z2-test-node.zetrix.com   # production: https://z2-node.zetrix.com
      - REEXECUTOR_URL=http://reexecutor:8547
      - ROLE=watcher            # verify-and-dispute only
    depends_on:
      - reexecutor
    restart: unless-stopped
```

```bash
docker pull zetrixchain/z2-validator:latest
docker pull zetrixchain/z2-reexecutor:latest
docker compose -f docker-compose.watcher.yml up -d
```

> \[!NOTE] Configuration keys above are illustrative placeholders. Consult the component's `--help` / environment reference in the image for the authoritative variable names before production use.

### How Third Parties Operate a Watcher

1. Pull the two images.
2. Configure L1 and L2 RPC endpoints and a dispute-capable L1 account (for gas to submit disputes).
3. Run in `watcher` mode (verify only; no batch signing authority required for safety).
4. Monitor discrepancy alerts; the stack disputes automatically or notifies operators to dispute.

### Inputs / Outputs

| Inputs                       | Outputs                                        |
| ---------------------------- | ---------------------------------------------- |
| L1 assertions, L2 batch data | Verification verdicts; disputes (fraud proofs) |

### Dependencies

* L1 RPC, L2 RPC (or batch data source), gas-funded L1 account for disputes.

### High Availability

* Run multiple independent Watchers (different operators, regions, infra) — the more, the safer.

### Security

* Watcher must use the exact EVM ruleset to avoid false disputes.
* Protect the dispute account key; keep it funded.

### Monitoring

* Assertions checked, discrepancies found, disputes submitted, lag behind chain head.

### Failure Scenarios & Recovery

| Failure           | Effect                    | Recovery                                |
| ----------------- | ------------------------- | --------------------------------------- |
| Watcher down      | Reduced safety redundancy | Restart; encourage more operators       |
| All Watchers down | Safety assumption at risk | Multiple independent operators mitigate |

### Best Practices

* **Decentralize Watchers** across independent parties.
* Keep dispute account funded and monitored.

***

## Z2 Refueler

### Purpose

The Refueler monitors critical operational **EOAs** (externally owned accounts) and automatically tops them up with gas so that the Sequencer, Committee, and Proposer never stall for lack of funds.

### EOA Monitoring & Automatic Top-Up

```mermaid
flowchart TD
    R[Refueler] -->|poll balance| S[Sequencer EOA]
    R -->|poll balance| C[Committee EOAs]
    R -->|poll balance| P[Proposer EOA]
    R -->|balance < threshold| T[Send top-up tx]
    T --> S
    T --> C
    T --> P
```

### Accounts Monitored

* Sequencer
* Committee
* Main Proposer Validator

### Inputs / Outputs

| Inputs                   | Outputs             |
| ------------------------ | ------------------- |
| EOA balances, thresholds | Top-up transactions |

### Dependencies

* A funded treasury/source account.
* Chain RPC to read balances and send top-ups.

### Deployment

* **Current deployment: 1 server.**

### High Availability

* Warm standby recommended; monitored source balance.

### Scalability

* Trivial (low request volume); polling interval tunable.

### Security

* Holds a funded source key — protect with HSM/KMS.
* Enforce per-tx and per-interval caps to limit blast radius if compromised.

> \[!WARNING] The Refueler's source key can move funds. Apply strict spend limits, allowlisted destinations (only the monitored EOAs), and alerting on anomalous top-ups.

### Monitoring

* Source balance, top-up frequency/amount per EOA, failed top-ups.

### Failure Scenarios & Recovery

| Failure         | Effect                             | Recovery                        |
| --------------- | ---------------------------------- | ------------------------------- |
| Refueler down   | EOAs may deplete → pipeline stalls | Standby; manual top-up; alert   |
| Source depleted | Cannot top up                      | Replenish treasury; alert early |

### Best Practices

* Alert on source balance well before depletion.
* Allowlist destinations; cap amounts.

***

## Z2 Explorer

### Purpose

The Explorer is the public block explorer for Z2 — a web UI and API for inspecting blocks, transactions, accounts, tokens, and smart contracts.

### Architecture

```mermaid
flowchart LR
    NODE[Z2 Node RPC] --> IDX[Indexer]
    IDX --> DB[(Explorer DB)]
    DB --> API[Explorer API]
    API --> UI[Explorer Web UI]
    UI --> USER([Users])
```

### Features

| Feature         | Description                                   |
| --------------- | --------------------------------------------- |
| Search          | By block, tx hash, address, token, contract   |
| Blocks          | Height, timestamp, proposer, tx count, gas    |
| Transactions    | Status, from/to, value, gas, logs, receipts   |
| Accounts        | Balances, nonce, tx history                   |
| Tokens          | ZTP20/token metadata, holders, transfers      |
| Smart contracts | Bytecode, verified source, read/write methods |

### Inputs / Outputs

| Inputs                 | Outputs                            |
| ---------------------- | ---------------------------------- |
| Node RPC, chain events | Indexed data, web UI, explorer API |

### Dependencies

* Z2 Nodes (RPC), indexer, database.

### Deployment

* Indexer + database + API + UI tier; can be scaled independently of nodes.

### High Availability

* Redundant API/UI instances behind a load balancer; replicated database.

### Scalability

* Read-heavy — scale API/UI horizontally; use read replicas and caching.

### Security

* Read-only against the chain; sanitize inputs; rate-limit API.

### Monitoring

* Indexer lag behind chain head, query latency, error rate.

### Failure Scenarios & Recovery

| Failure     | Effect               | Recovery                         |
| ----------- | -------------------- | -------------------------------- |
| Indexer lag | Stale explorer data  | Scale indexer; catch up          |
| DB down     | Explorer unavailable | Failover replica; restore backup |

### Best Practices

* Alert on indexer lag; cache hot queries; verify contract source publicly.

***

## Z2 Monitoring

### Purpose

Provide platform-wide observability: metrics, logs, health monitoring, performance analysis, alerting, and dashboards across all Z2 components.

### Architecture

```mermaid
flowchart LR
    subgraph Targets
        N[Nodes]
        S[Sequencer]
        V[Validators]
        P[Proposer]
        R[Refueler/Relayer]
    end
    Targets -->|/metrics| PROM[Prometheus]
    Targets -->|logs| LOKI[Log store]
    PROM --> GRAF[Grafana Dashboards]
    LOKI --> GRAF
    PROM --> ALERT[Alertmanager]
    ALERT --> NOTIFY[Email / Slack / PagerDuty]
```

### Capabilities

| Capability           | Description                               |
| -------------------- | ----------------------------------------- |
| Metrics              | Prometheus time-series for all components |
| Logs                 | Centralized aggregation and search        |
| Health monitoring    | Liveness/readiness of every service       |
| Performance analysis | Latency, throughput, resource trends      |
| Alerting             | Threshold- and rate-based alerts          |
| Dashboards           | Grafana per-component and platform views  |

### Inputs / Outputs

| Inputs                  | Outputs                     |
| ----------------------- | --------------------------- |
| Metrics endpoints, logs | Dashboards, alerts, reports |

### Dependencies

* All components exporting `/metrics` and logs.

### Deployment

* Prometheus + Grafana + Alertmanager + log store.

### High Availability

* Redundant Prometheus (HA pairs / remote-write), replicated Grafana, durable alerting.

### Scalability

* Federation / remote-write for large fleets; retention tiers.

### Security

* Protect dashboards behind auth (`z2-admin.zetrix.com`); restrict scrape endpoints.

### Failure Scenarios & Recovery

| Failure           | Effect            | Recovery             |
| ----------------- | ----------------- | -------------------- |
| Prometheus down   | No metrics/alerts | HA pair; restore     |
| Alertmanager down | Missed alerts     | Redundant AM cluster |

### Best Practices

* Alert on the alerting pipeline itself (dead-man's-switch).
* Version-control dashboards and alert rules.

*(Detailed alert thresholds are in* [*Monitoring*](#monitoring)*.)*

***

## Z2 Relayer

### Purpose

The Relayer synchronizes state from L1 to L2, primarily driving the **deposit** workflow: assets locked/registered on Zetrix L1 are relayed to L2 so users receive their L2 balance.

### Deposit Workflow (L1 → L2)

```mermaid
sequenceDiagram
    participant User
    participant L1 as Zetrix L1 (Bridge)
    participant Rel as Relayer
    participant N as Z2 Node
    User->>L1: deposit ZETRIX / ZTP-20 / ZTP-721 (lock)
    L1-->>Rel: enqueue delayed message
    Rel->>Rel: observe & confirm
    Rel->>N: deliver → credit on L2
    N-->>User: L2 balance available (seconds)
```

### Supported Assets

* **Native ZETRIX** (bridged to L2 as Wrapped Zetrix — see [Asset Flow](/zetrix-l2-documentation/usage/asset-flow.md#asset-flow)).
* **ZTP20** tokens.

### Inputs / Outputs

| Inputs            | Outputs                |
| ----------------- | ---------------------- |
| L1 deposit events | L2 credit transactions |

### Dependencies

* L1 RPC / bridge contract events; Z2 Nodes.

### Deployment

* Runs as an operational service (co-located with infra services in Testnet).

### High Availability

* Redundant Relayer instances with idempotent event processing (dedupe by L1 event id).

### Scalability

* Event-driven; scales with L1 event volume; parallelize by asset/queue.

### Security

* Must not double-credit — enforce exactly-once processing keyed on L1 event.
* Wait for sufficient L1 confirmations before crediting.

> \[!WARNING] Deposit relaying is trust-sensitive: replay or double-processing of L1 events would mint unbacked L2 balances. Enforce idempotency and confirmation thresholds.

### Monitoring

* Deposit lag, events processed, failed/duplicate events, confirmation depth.

### Failure Scenarios & Recovery

| Failure      | Effect           | Recovery                                    |
| ------------ | ---------------- | ------------------------------------------- |
| Relayer down | Deposits delayed | Standby; catch up from last processed event |
| Missed event | Deposit stuck    | Reprocess from L1 event log                 |

### Best Practices

* Idempotent, checkpointed event processing.
* Confirmation-depth threshold before crediting.

***

## Faucet

### Purpose

The Faucet distributes **test gas** on Testnet so developers and users can obtain funds to exercise the network without real value.

### Gas Distribution & Rate Limiting

```mermaid
flowchart TD
    U([User]) -->|request| F[Faucet]
    F --> RL{Rate limit OK?}
    RL -->|no| REJ[Reject / retry-after]
    RL -->|yes| CHK{Anti-abuse checks}
    CHK -->|fail| REJ
    CHK -->|pass| SEND[Send test gas]
    SEND --> U
```

### Security & Anti-Abuse

* Rate limiting per address / IP.
* Optional captcha or auth (see prohibited actions caveat: automated captcha solving is not a platform feature).
* Caps on amount per request and per period.

### Current Implementation

* Web-based faucet served at `https://z2-test-faucet.zetrix.com`; production at `https://z2-faucet.zetrix.com`.
* Distributes the L2 native gas unit (`ZETRIX2` symbol) so users can pay for transactions on Z2.
* Fixed gas amount per request with per-address/IP rate limits.

### Future Improvements

* Stronger anti-Sybil (proof-of-work, social auth).
* Dynamic amounts based on demand.
* Abuse analytics and dynamic blocklists.

### Inputs / Outputs

| Inputs            | Outputs           |
| ----------------- | ----------------- |
| Address + request | Test-gas transfer |

### Dependencies

* Funded faucet EOA; Z2 RPC.

### Deployment

* Web service + faucet EOA (Testnet only).

### Security Considerations

* Protect the faucet EOA key; monitor drain rate.

### Failure Scenarios & Recovery

| Failure        | Effect      | Recovery                       |
| -------------- | ----------- | ------------------------------ |
| Faucet drained | No test gas | Replenish; tighten limits      |
| Abuse spike    | Rapid drain | Rate-limit; blocklist; captcha |

### Best Practices

* Conservative default limits; monitor drain; alert on anomalies.

> \[!NOTE] The Faucet is a **Testnet-only** convenience and has no role on Mainnet.

***

[← Trust Model](/zetrix-l2-documentation/architecture/trust-model.md) · [Index](/zetrix-l2-documentation/readme.md) · [Z2 Zetrix MCP →](/zetrix-l2-documentation/integration/zetrix-mcp.md)