Skip to content

Explanation

Context

Why Proxmox VE works the way it does: the platform stack, the cluster file system and corosync design that everything else stands on, how HA and the new CRS scheduler actually decide placements, the storage and backup pipelines, the SDN stack, identity and firewall, and the threat model of a hypervisor whose management plane is part of the product.

Platform Architecture

PVE is not a hypervisor kernel — it is a Debian distribution whose packages assemble into a hypervisor appliance. Each node runs the same set:

  • Debian 13 Trixie base with Proxmox's own kernel (7.0 is the default in 9.2), QEMU 11.0 for KVM virtual machines, LXC 7.0 for system containers. Since 2026-08 the same stack also ships for arm64.
  • Management plane: pveproxy (web UI + REST API on TCP 8006; Perl API server, with new components increasingly written in Rust), pvedaemon (privileged API worker), pvestatd (status broadcasting), pvescheduler (backup and replication jobs), and spiceproxy (SPICE consoles on TCP 3128; noVNC consoles are tunnelled through pveproxy as WebSockets).
  • Guest abstraction: VMs are QEMU processes with config files in /etc/pve/qemu-server/<vmid>.conf; containers are LXC with /etc/pve/lxc/<vmid>.conf. Same tooling (qm vs pct), same storage layer (pvesm), same backup pipeline (vzdump), same API — the unification is PVE's core design decision.
flowchart TB
    subgraph Node["PVE node (repeated N times)"]
        GUI["pveproxy :8006<br/>GUI + REST API"]
        DM["pvedaemon / pvestatd / pvescheduler"]
        QEMU["QEMU/KVM VMs"]
        LXC["LXC containers"]
        STOR["Storage plugins (pvesm)<br/>ZFS / Ceph / LVM / dir / NFS / PBS"]
        CFSCONF["/etc/pve (pmxcfs)"]
        GUI --> DM --> QEMU & LXC
        DM --> STOR
        CFSCONF -.config source.-> DM
    end
    subgraph Cluster["Cluster layer"]
        CORO["corosync 3<br/>UDP 5405-5412, up to 8 links"]
        HA["pve-ha-lrm + pve-ha-crm<br/>watchdog fencing"]
        CRS["Cluster Resource Scheduler<br/>dynamic load balancing (9.2)"]
    end
    CFSCONF <-->|"real-time config replication"| CORO
    CORO --> HA --> CRS

The Cluster File System: pmxcfs

Every design decision in a PVE cluster flows from one choice: cluster configuration lives in a database-backed FUSE filesystem, /etc/pve, replicated in real time by corosync to every node. Consequences:

  • One source of truth per cluster. Edit a VM config on any node; all nodes see it instantly. No config drift, no per-node "vCenter sync".
  • Strong consistency checks — duplicate VM IDs are impossible cluster-wide; a distributed locking mechanism serializes cluster-wide operations.
  • Quorum-gated writes. pmxcfs is read-only when a node loses quorum — a partitioned node cannot start or change guests, trading availability for split-brain safety. This is why two-node clusters need a QDevice (a third corosync vote on separate hardware) and why pvecm expected 1 exists as a conscious override for recovery, not for normal operation.
  • Size bound: the database is capped at 128 MiB (RAM copy) — several thousand VM configs fit; unbounded metadata does not.

Corosync requirements follow from the design: UDP 5405-5412 between all nodes, synchronized clocks, SSH (TCP 22) for node joins, a dedicated NIC recommended (low latency matters more than bandwidth), up to 8 redundant links, and a known corosync-over-bonds failure mode — a single logical link on a bond can mask a partition, so true redundancy means a second link on separate physical networking.

HA, Fencing, and the Cluster Resource Scheduler

HA in PVE is watchdog fencing, not just restart-on-fail. Every HA-managed node holds a hardware watchdog (or software fallback); pve-ha-lrm (per node) must continuously prove liveness. If a node stops feeding the watchdog — kernel hang, power loss, network partition confirmed by corosync — the watchdog resets the box, guaranteeing a dead node cannot keep a VM running while the cluster restarts it elsewhere. pve-ha-crm (one elected master) then restarts the node's HA services per their configured states.

The sequence below shows the recovery path when a node running HA guests loses quorum (the fence-before-restart order is what prevents two copies of a VM writing to the same disk):

sequenceDiagram
    participant N2 as node2 pve-ha-lrm
    participant WD as node2 watchdog
    participant CS as corosync
    participant CRM as pve-ha-crm master
    participant CRS as CRS scheduler
    participant N3 as node3 pve-ha-lrm
    N2->>CS: loses cluster communication
    CS-->>CRM: membership change, node2 not quorate
    N2->>WD: stops updating watchdog, pmxcfs read-only
    WD->>N2: hardware reset after timeout
    CRM->>CRM: waits until node2 fence is certain, marks node2 fenced
    CRM->>CRS: select target for vm:101 honoring HA rules
    CRS-->>CRM: node3
    CRM->>N3: start vm:101
    N3-->>CRM: service started

The Cluster Resource Scheduler (CRS) chooses target nodes whenever the HA stack must place a resource: recovery after fencing, HA rule changes (node affinity and resource affinity/anti-affinity rules, which replaced HA groups in 9.0), and optionally on start. It has three modes: basic (count of active guests, the default), static (configured CPU/memory, since 7.3), and — new in 9.2 — dynamic, which uses measured CPU and memory usage. With 9.2's automatic rebalancing enabled, the CRM continuously migrates HA guests to reduce cluster-wide imbalance (threshold tunable, per-resource opt-out). The roadmap extends balancing to non-HA guests.

Maintenance got safer too: ha-manager crm-command disarm-ha freeze|ignore releases all CRM and LRM watchdogs cluster-wide without changing resource state, so planned switch or corosync network work cannot trigger mass fencing; arm-ha restores protection. The cost is explicit: while disarmed, nothing is recovered.

Storage Stack

One abstraction (/etc/pve/storage.cfg) fronts every storage type — local ZFS pools, directory/BTRFS, NFS/CIFS, LVM/LVM-thin, iSCSI/FC, Ceph RBD/CephFS, ZFS-over-iSCSI, and Proxmox Backup Server. The full capability matrix lives in Reference. The design-relevant points:

  • ZFS is the first-class local filesystem: zvols for VM disks, native snapshots, compression, RAID-Z, and PVE's storage replication (pvesr) — scheduled, incremental, snapshot-based replication of guest volumes between nodes for cheap DR without shared storage.
  • Ceph is the hyperconverged answer: pveceph installs and integrates Ceph (Tentacle 20.2 is the default for new installs on 9.2; Squid 19.2 remains available; the Proxmox repository lists Squid EOL as 2026-09 while upstream Ceph targets 2026-10-31) so RBD pools and CephFS appear as first-class storages; every node is MON/MGR/OSD capable. RAM budget: ~1 GB per TB of used storage for both ZFS and Ceph.
  • Snapshots diverge by storage: native on ZFS/LVM-thin/Ceph/RBD; on file-based storages historically via qcow2 internals (slow — minutes on large disks, blocking the VM). PVE 9 introduced snapshots-as-volume-chains (tech preview; roadmap promotes it to GA on LVM-thick, Directory, NFS, CIFS) where each snapshot is a separate volume layered in a chain — online removal of the top-most snapshot included, and the foundation for extending replication beyond ZFS.

Backup Pipeline

vzdump is the single backup engine behind both legacy file backups and PBS:

  • Modes: stop (downtime, most consistent), suspend (containers: rsync, brief suspend, second rsync), snapshot (live; VMs use QEMU's own backup job with guest-agent fs-freeze, containers use storage snapshots — the default for modern setups).
  • Backup fleecing (VMs): when the guest overwrites a block that has not been backed up yet, the old data is first copied to a fleecing image on fast local storage, so guest writes are not throttled by a slow backup target.
  • Proxmox Backup Server replaces the tar/vzdump-file world with a deduplicated, compressed, optionally client-side-encrypted chunk store: incremental-forever backups (dirty bitmaps for running VMs), cross-guest dedup per datastore, Zstd compression, integrity via checksummed chunks and verify jobs, live restore (boot the guest while its disk still restores), single-file restore, tape support, and S3-compatible object-storage datastores (tech preview in 4.0, officially supported since PBS 4.2, 2026-04-29, which also added parallel sync jobs and on-the-fly encryption/decryption in push/pull sync).

SDN, Identity, and Firewall

  • SDN (core generally available since 8.1, still gaining features): zones (Simple, VLAN, QinQ, VXLAN, EVPN) contain VNets with subnet/IPAM management; 9.0 introduced fabrics (OpenFabric, OSPF) for routed underlays, and 9.2 added WireGuard and BGP fabrics, route maps and prefix lists for BGP/EVPN filtering, route redistribution, and IPv6 underlay support. Roadmap items include DHCP IPAM stabilization, more IPAM/DNS integrations, and topology visualizations (see the PVE roadmap).
  • Identity: authentication realms — Linux PAM, Proxmox VE auth server (internal password store), LDAP/Active Directory, OpenID Connect — compose with RBAC (roles bound to paths like /vms/101 or pools) and API tokens (scoped, separable from user password, ideal for automation). TOTP/WebAuthn two-factor is built in.
  • Firewall: distributed, stateful, three levels (datacenter, node, guest) plus VNet-level rules, with security groups, IP sets, and logging; the default pve-firewall compiles to iptables, while the nftables-based proxmox-firewall (Rust) is still tech preview as of 9.2 (introduced in 8.2) and is required for VNet-level firewalling.

Engineering Direction

Two multi-year migrations shape the codebase: backend components are moving from Perl to Rust through the perlmod interop layer (new services like PBS and PDM are Rust-first), and the ExtJS web interface is moving to the Rust/Yew-based Proxmox widget toolkit that already powers the PVE mobile UI, PDM, and Mail Gateway quarantine. The "Cattle and Pets" project targets large estates: cluster-wide operations center, semi-automated ordered Ceph upgrades, bulk guest operations, large-cluster GUI performance.

Proxmox Datacenter Manager (PDM) is the multi-cluster layer that sits above this: a separate Rust/Yew appliance that talks to PVE clusters and PBS instances ("remotes") over their APIs, giving one view of resources, cross-cluster live migration, and update/SDN overviews. PDM 1.0 shipped 2025-12-04; PDM 1.1 (2026-05-28) added automated-installation workflows (PDM as answer-file server for new hosts), a central subscription-key registry, Ceph cluster monitoring, and snapshot management. PDM has no subscription of its own; its enterprise repo unlocks when at least 80% of managed remote nodes hold Basic or higher subscriptions.

flowchart LR
    PDM["Proxmox Datacenter Manager<br/>(Rust/Yew, remotes view)"]
    subgraph C1["PVE cluster A"]
        A1["pveproxy API :8006"]
    end
    subgraph C2["PVE cluster B"]
        B1["pveproxy API :8006"]
    end
    PBS1["Proxmox Backup Server<br/>(datastores, S3 backend)"]
    PDM -->|"API token"| A1
    PDM -->|"API token"| B1
    PDM -->|"API token"| PBS1
    A1 -->|"vzdump backups"| PBS1
    B1 -->|"vzdump backups"| PBS1
    A1 <-.->|"remote migration via PDM"| B1

Threat Model

Surface Reality Control
Management plane (8006, SSH 22) The hypervisor IS the management product — console access is root-equivalent Dedicated mgmt VLAN, PAM/PVE realms + TOTP/WebAuthn 2FA, RBAC least-privilege, API tokens instead of passwords for automation
Cluster bus (corosync 5405-5412) A node joining the corosync ring can influence config replication Isolated cluster NIC/VLAN; corosync traffic is authenticated/encrypted with the cluster key (/etc/corosync/authkey); restrict the ring to cluster nodes
Guest escape QEMU/LXC share the host kernel (LXC especially) KVM for hard multi-tenancy; LXC for trusted workloads; apply AppArmor (LXC default), keep kernel/QEMU patched via subscription
Backups PBS chunks are the crown jewels; theft of unencrypted chunks exposes data Enable client-side encryption (key held by PVE, not PBS); protect PBS separately; test restore-path key custody
Quorum attacks / misoperation pvecm expected N overrides split-brain protection; HA fencing can power-cycle nodes Treat expected-quorum overrides as break-glass; watchdog discipline; disarm-ha only for planned windows
Supply chain Debian + Proxmox repos; no-subscription repo is less tested Enterprise repo (any subscription incl. Community tier) for tested rollouts; offline updates for air-gapped sites
Lifecycle PVE 8 left support in 2026-08; Ceph Squid repos reach EOL 2026-09 (upstream Ceph target 2026-10-31) Track the support table in Reference; run pve8to9 and upgrade

Guest Storage and Backup Data Flow

flowchart LR
    G["Guest (VM / CT)<br/>virtio-scsi-single + iothread<br/>discard=on, ssd=1"]
    subgraph NodeS["Node storage layer"]
        QM["qm / pct volume layer<br/>(qemu-server config in /etc/pve)"]
        PLUG["pvesm storage plugins<br/>storage.cfg per cluster"]
        QM --> PLUG
    end
    subgraph Backends
        Z["ZFS zvol<br/>+ pvesr replication"]
        C["Ceph RBD<br/>hyperconverged"]
        F["file-based<br/>(qcow2 / volume chain)"]
    end
    subgraph Backup["Backup pipeline (vzdump)"]
        SNAP["snapshot mode<br/>(storage snapshot or chain)"]
        FLEECE["fleecing image<br/>(absorbs guest writes)"]
        PBS["PBS chunk store<br/>dedup + zstd + encryption"]
    end
    G --> QM
    PLUG --> Z & C & F
    SNAP --> FLEECE --> PBS
    Z & C & F -.snapshot source.- SNAP

Three consequences worth internalizing: the config file in /etc/pve is the single authority for what a guest's disks are (fixing "wrong disk" problems means fixing the config, not the hypervisor); snapshot semantics differ per backend, which is why 9.x is investing in volume chains to make file-based storages behave like ZFS/Ceph; and the fleecing image is what makes PBS backups invisible to busy guests — without it, a backup window is an I/O ceiling.

Live Migration

Migration moves a running guest between nodes: QEMU iterates memory while tracking dirty pages, copies them to the target, then pauses the source and transfers the residual delta — sub-second stop time in healthy clusters. Design constraints follow directly:

  • Same architecture (hard), same CPU vendor (recommended). The guest sees a vCPU model; if the target lacks a flag the guest already uses, it faults. x86 to arm64 is impossible. Intel to AMD "might work" with generic models (kvm64, x86-64-v2-AES) but is not guaranteed, so the docs say to test first. The GUI default for new VMs is x86-64-v2-AES; custom CPU models (managed cluster-wide in the 9.2 GUI) pin a portable feature set across heterogeneous nodes.
  • Shared storage vs local storage. With shared storage (Ceph, NFS, FC) only RAM/CPU state moves. With local storages, --with-local-disks adds a storage-migration phase (NBD copy of disks while running) — workable, but it is a data copy and slows with disk size.
  • Passed-through devices block migration unless the resource mapping is marked live-migration-capable (experimental; recent NVIDIA GPUs only).
  • HA interplay: HA-managed guests migrate through the CRS; the 9.2 load balancer uses the same machinery continuously, so a healthy cluster is always gently re-balancing by choice.

Performance Levers (and why they exist)

  • virtio-scsi-single + iothread + discard=on + ssd=1 — the standard disk quartet on ZFS/Ceph: one request queue per disk that can thread independently, TRIM passthrough so thin layers reclaim space, and SSD semantics so guest schedulers don't over-optimize.
  • CPU type: host exposes all host features (fastest, blocks cross-vendor migration); x86-64-vX baselines for portability; custom models for the middle ground.
  • Ballooning trades guest RAM against host pressure — set a realistic minimum, since ballooning shrinks guest-visible memory under host contention.
  • Multiqueue on virtio-net for high-PPS guests (queues should roughly match guest vCPUs).
  • NUMA + CPU pinning for latency-sensitive or large VMs on multi-socket hosts; taskset-style vCPU pinning trades scheduler flexibility for cache locality.
  • pveperf is the shipped first-guess tool: if fsync latency is poor, no guest tuning will save you — fix the storage layer first.

Container or VM? The Trust Boundary

LXC containers are kernel-shared: they boot in milliseconds, cost ~nothing, and suit trusted workloads (services, CI runners, homelab apps) — but a container escape lands on the hypervisor kernel, and the mitigation (unprivileged mapping, AppArmor, seccomp, no unnecessary nesting) is a hardening posture, not a boundary. KVM guests have a hardware boundary. The operational rule: default to unprivileged LXC for infrastructure you control; choose KVM whenever the workload is untrusted, licensed-per-VM, or needs its own kernel (SELinux policies, custom sysctls, out-of-tree modules).

Why It Won the VMware Exodus

The 2024 Broadcom licensing shock made thousands of vSphere estates re-platform. PVE's win was combination, not individual superiority: a native ESXi import wizard (added in 8.1-era updates, 2024), per-socket pricing independent of cores, all-features-free AGPL, and a data path (PBS) that replaced third-party backup for many. The remaining enterprise objections are being closed one by one: multi-site management arrived with PDM 1.0 (Dec 2025) and 1.1 (May 2026), and global 24/7 support for Premium subscribers starts 2026-10-19 alongside a North American subsidiary. Very large estates still lack some operational polish the "Cattle and Pets" roadmap targets.

Sources