Cluster Architecture

The Monibuca V6 cluster solution is based on the Origin-Edge model. Nodes communicate via the QUIC protocol for low-latency, high-reliability communication, supporting automatic node discovery, intelligent stream forwarding, and load balancing.

Architecture Overview

flowchart TB
  DNS["DNS/LB<br/>User Entry Point"]
  DNS --> EBJ["Edge-BJ<br/>Beijing"]
  DNS --> ESH["Edge-SH<br/>Shanghai"]
  DNS --> EGZ["Edge-GZ<br/>Guangzhou"]
  subgraph OC["Origin Cluster"]
    O1["Origin-1"]
    O2["Origin-2"]
    O3["Origin-3"]
  end
  EBJ -->|QUIC Relay| OC
  ESH -->|QUIC Relay| OC
  EGZ -->|QUIC Relay| OC

Role Definitions

Role	Responsibility
Origin	Receives direct ingest from publishers; serves as the stream source. Provides stream data to Edge nodes
Edge	Receives pull requests from viewers. Automatically pulls from Origin when a stream is not available locally

Roles Are Not Fixed

Each Monibuca node can be both an Origin and an Edge — the role is determined by where the stream is actually published. The same node may be an Origin for stream A (locally published) and an Edge for stream B (requires pulling from origin).

QUIC Communication

Cluster nodes communicate using the QUIC protocol, leveraging its inherent advantages:

Why QUIC

Feature	TCP	QUIC
Connection establishment	1-3 RTT (including TLS)	0-1 RTT
Head-of-line blocking	Entire connection blocked	Only individual stream blocked
Multiplexing	Requires HTTP/2	Native support
Connection migration	Not supported	Supported (survives IP changes)
Congestion control	Shared across connection	Independent per stream

Communication Layers

flowchart TB
  App["Application Layer<br/>HTTP /cluster/api control"]
  Relay["Relay Layer<br/>Audio/Video Data Transfer"]
  Quic["QUIC Transport<br/>quinn Connection Mgmt"]
  Tls["TLS 1.3<br/>Self-signed / Auto Certs"]
  App --> Relay --> Quic --> Tls

• HTTP control plane: Discovery, heartbeats (POST /cluster/api/heartbeat), node/session APIs; use cluster.sync.address / seed_servers on the same port as global.http.listenaddr (default 8180)
• Relay Layer: High-speed audio/video frame data transfer, transmitted directly over QUIC streams
• Transport Layer: QUIC connection management based on the quinn library, with automatic certificate generation

Global gRPC is not cluster control

global.tcp.listenaddr (e.g. :50051) is the main Monibuca gRPC listener. It is not used for cluster heartbeats or seeds. Do not put :50051 in sync.address or seed_servers.

Node Discovery and Health Checks

Seed Node Discovery

When a new node starts, it connects to the seed nodes configured in seed_servers to obtain cluster topology information:

sequenceDiagram
  participant N as New Node
  participant S as Seed Node
  participant O as Each Node
  N->>S: I am edge-3, please tell me who is in the cluster
  S->>N: origin-1, edge-1, edge-2, ...
  N->>O: Establish QUIC connections

Heartbeat Mechanism

sequenceDiagram
  participant A as Node A
  participant B as Node B
  A->>B: heartbeat (every 5s)
  B->>A: heartbeat_ack

Each heartbeat carries node summary information:

• CPU usage
• Memory usage
• Bandwidth usage
• Current stream count
• Subscriber count

Failure Detection

Three-level failure detection based on heartbeat timeout:

State	Condition	Behavior
Healthy	Heartbeat normal	Participates in cluster normally
Suspect	suspect_threshold consecutive missed responses	Marked as suspect, weight reduced
Offline	offline_threshold consecutive missed responses	Marked as offline, sessions cleaned up

When a node is marked as Offline, the following actions are triggered:

1. SessionRegistry clears all stream registrations for that node
2. RelayManager disconnects all Relay connections to that node
3. AllocationManager stops assigning requests to that node

Stream Relay

Pull-from-Origin Flow

sequenceDiagram
  participant V as Viewer
  participant E as Edge
  participant Reg as SessionRegistry
  participant O as Origin-1
  V->>E: Request live/camera01
  Note over E: Stream not available locally, query SessionRegistry
  Reg->>E: Stream is on Origin-1
  E->>O: Establish QUIC Relay connection
  O->>E: Transfer audio/video data via QUIC
  E->>V: Distribute to local subscribers

Relay Lifecycle

1. Establishment: Edge detects no local stream and initiates a Relay request to Origin via QUIC
2. Transfer: RingBuffer data from Origin is transmitted to Edge over QUIC streams
3. Health Check: Relay connection status is periodically checked (every health_check_interval seconds)
4. Release: After the last subscriber on the Edge leaves, the Relay is released after waiting release_delay seconds

Failure Recovery

When a Relay connection is broken:

1. RelayManager detects the connection anomaly
2. Waits retry_delay seconds before retrying
3. Retries up to max_retry_attempts times
4. If the Origin node is offline, queries SessionRegistry for a new Origin
5. After all retries fail, the stream on the Edge is marked as unavailable

Load Balancing

Allocation Strategy

AllocationManager selects the optimal node for new requests via POST /cluster/api/allocate/publish|play, considering:

1. Node health status: Only selects nodes in Online state
2. Load metrics: Weighted scoring over CPU usage, memory, bandwidth, stream count, and latency

allocate/play also attaches origin hints from the session catalog (origin_server_id / origin_addr / origin_quic_addr / relay_required) so the caller knows whether playing on the chosen node will trigger an Edge relay pull from origin.

Advisory APIs

Allocation and redirect are advisory control-plane APIs: they return decision JSON only and never redirect clients or establish relays themselves. Callers must point the client at the chosen node’s media endpoint; when the client actually plays on an Edge, the Edge auto-triggers ensure_relay via its subscribe hooks.

Redirect Decision

When the local node’s load exceeds the thresholds, GET /cluster/api/redirect/decision suggests a better node (should_redirect / target / candidates, plus the same origin hint fields):

sequenceDiagram
  participant C as Caller
  participant E as Edge-BJ
  C->>E: GET /cluster/api/redirect/decision?stream_path=live/camera01
  Note over E: CPU 90% exceeds threshold 85%
  E->>C: should_redirect=true, target=edge-sh
  Note over C: Point viewer at Edge-SH media endpoint

Redirect threshold configuration:

cluster:
  routing:
    cpu_threshold: 85.0           # CPU usage threshold
    bandwidth_threshold: 8000.0   # Bandwidth threshold (Mbps)
    subscriber_threshold: 2000    # Subscriber count threshold

Deployment Modes

Single Origin + Multiple Edges

The most common deployment mode, suitable for small to medium scale:

# Origin configuration
cluster:
  sync:
    server_id: "origin-1"
    address: "10.0.1.1:8180"
    seed_servers: ["10.0.1.1:8180"]

# Edge configuration
cluster:
  sync:
    server_id: "edge-bj-1"
    address: "10.0.2.1:8180"
    seed_servers: ["10.0.1.1:8180"]  # Points to Origin

Multiple Origins + Multiple Edges

Large-scale deployment with multiple Origins sharing the ingest load:

# Origin-1
cluster:
  sync:
    server_id: "origin-1"
    address: "10.0.1.1:8180"
    seed_servers: ["10.0.1.1:8180", "10.0.1.2:8180"]

# Origin-2
cluster:
  sync:
    server_id: "origin-2"
    address: "10.0.1.2:8180"
    seed_servers: ["10.0.1.1:8180", "10.0.1.2:8180"]

# Edge
cluster:
  sync:
    server_id: "edge-1"
    address: "10.0.2.1:8180"
    seed_servers: ["10.0.1.1:8180", "10.0.1.2:8180"]

Cascaded Edges

Edge nodes can also serve as upstream for other Edges, forming a multi-tier cascade:

flowchart LR
  Ingest["Ingest"] --> Origin --> L1["Edge-L1<br/>Regional Hub"] --> L2["Edge-L2<br/>City Node"] --> Viewer["Viewer"]

Suitable for large-scale nationwide distribution scenarios.

Operations Commands

Cluster HTTP APIs live under /cluster/api/ on global.http.listenaddr (not gRPC global.tcp).

Local node status

GET /cluster/api/status/local

Node catalog

GET /cluster/api/servers

Session catalog

GET /cluster/api/sessions

Local relay sessions

GET /cluster/api/relay/sessions

The Admin Cluster page derives alerts and Origin→relay topology from current snapshots of these endpoints (30s polling); it is not a historical event store.

See the Cluster plugin doc for the full API list.