Jun 2026 · 4 min
Slashing is a culture problem
Why most validator failures begin in process, not in code.
Cosmos SDK chains punish a validator for two things: going offline and signing twice. Both are handled by the x/slashing module, and the parameters are on-chain, so you can read them before you join a network:
gaiad q slashing paramsOn Cosmos Hub at the time of writing, the window is 10,000 blocks, a validator must sign at least 5% of them, downtime costs 0.01% of stake plus a 10-minute jail, and a double-sign costs 5% of stake and a permanent tombstone. Other chains set their own values. Some are much stricter on downtime.
The gap between those two numbers is the whole point. Downtime is cheap. Double-signing ends the validator. Every operating decision should be read against that asymmetry.
Double-signs are rarely a software bug
CometBFT does not sign two different blocks at the same height on its own. When a double-sign happens, the cause is almost always two processes holding the same consensus key at the same time. The usual ways that happens:
- A failover is done by hand while the old node is still running, or comes back after a reboot because its service is still enabled.
- A server is rebuilt from a backup or snapshot that includes priv_validator_key.json, and the original machine is still up.
- priv_validator_state.json is copied from an older backup, so the signer no longer remembers the last height and round it signed.
- An automated failover (keepalived, a script, an orchestrator) promotes a second node during a network partition, while the first one is still signing.
None of these are code problems. They are process problems: an unclear runbook, a step done from memory at 3 a.m., a backup policy that nobody checked against the key.
Controls that actually help
- One key, one signer. Keep the consensus key off the validator node and use a remote signer such as TMKMS or Horcrux. Horcrux threshold signing means a single misbehaving host cannot produce a valid signature alone.
- Treat priv_validator_state.json as part of the key. When the key moves, the state moves with it. Never restore an older copy.
- Stop, disable and verify before you start. The old signer is stopped, its service disabled and the key removed, and someone confirms it is no longer signing on an explorer, before the new one is started.
- No automatic failover for the signer. Automatic restart of the full node is fine. Automatic promotion of a second signer is how operators get tombstoned.
- Exclude keys from generic backups and snapshots. Keys are backed up separately, encrypted, with a short list of people who can restore them.
- Two people for key moves and chain upgrades: one runs the steps, one reads the runbook and checks each step.
Prefer downtime when in doubt
If you are not sure whether the old signer is dead, stay down. Missing a few hundred blocks on most chains costs nothing, or at worst a small slash and an unjail transaction. Guessing wrong on a double-sign costs delegators 5% and costs the operator the validator.
The same rule applies to upgrades. Use the halt height or Cosmovisor, verify binary checksums, and run the upgrade on testnet first. A late upgrade is downtime. Starting with the wrong binary or a hand-patched data directory risks an app-hash mismatch, and the fix for that is usually a resync, not a shortcut.
Downtime still matters to delegators
The 0.01% downtime slash is small, but a jailed validator earns nothing until it unjails, and delegators notice. Missed-block alerts should page someone well before the jail threshold, not show up on a dashboard the next morning.
Write up near misses too: the restore that almost included the key, the failover that almost ran twice. Those are cheaper lessons than the real thing, and they are the part of the culture that keeps the slashing count at zero.