Troubleshooting

A pragmatic fix-guide for the most common problems. For each symptom: brief explanation → diagnostics → fix.

General: reading logs

journalctl -u dvb-warppool -f --since "5 min ago"

For Docker:

docker logs -f dvb-warppool --since 5m

Temporarily raise the log level:

RUST_LOG=warppool=debug,info dvb-warppool-daemon

Targeted per crate:

RUST_LOG=warppool_stratum_v1=trace,warppool_bitcoin_rpc=debug,info dvb-warppool-daemon

RPC

getblockchaininfo failed: HTTP 401 — credentials rejected

Cookie or user/pass mismatch.

Diagnosis:

# Cookie auth:
cat ~/.bitcoin/.cookie       # should be "user:hexhash"
# user/pass auth:
grep -E "^rpc(user|password)" ~/.bitcoin/bitcoin.conf

Fix:

• Cookie mode: make sure node.rpc_cookie_path in config.toml points to the correct file. Bitcoin Core has its own cookie files for regtest/testnet (~/.bitcoin/regtest/.cookie, ~/.bitcoin/testnet3/.cookie). The setup wizard typically shows the right path.
• user/pass mode: in secrets.toml, rpc_user + rpc_pass must match the values in bitcoin.conf. In secrets.toml, not config.toml — otherwise the daemon complains at startup.
• On permission errors: bitcoind runs as a different user; the cookie is 0600 and unreadable for your warppool user. The setup wizard catches this and suggests chmod g+r + group-add.

getblocktemplate failed: -8 ...

Bitcoin Core is reachable, but says no to the template.

Possible causes:

• IBD running: warppool_rpc_ibd == 1. Wait until sync completes.
• Wallet locked: getblocktemplate without -walletbroadcast=0 needs an unlocked wallet in Core. Unusual in a pool context, though — we use GBT without a wallet.
• Too few peers: getpeerinfo shows 0 or 1. Core then refuses to mine to avoid ending up on a minority fork. Fix: wait, or use -minimumchainwork=0 for tests.

RPC latency high (Prometheus shows p99 > 1s)

Diagnosis:

• Are many RPCs running in parallel? The daemon normally does ~1-5/s in steady state. If p99 is high: is Bitcoin Core on a slow disk or swapping?
• tx-index=1 setup on HDD: makes GBT slow. SSD recommended.
• Pruned node + old block request: doesn’t happen on the normal pool path, but backup/restore workflows can trigger it.

Fix: usually a disk bottleneck. iotop / iostat shows this quickly.

ZMQ

Daemon only uses poll-loop, no ZMQ

Symptom: job refresh runs poll-only because no ZMQ watcher is active — the [node] zmq_hashblock_addr in config.toml is empty or missing (it is not an env var). If a configured channel later drops, the log shows:

zmq channel closed — falling back to polling only

Fix: set in config.toml:

[node]
zmq_hashblock_addr = "tcp://127.0.0.1:28332"

And in bitcoin.conf:

zmqpubhashblock=tcp://127.0.0.1:28332

Bitcoin Core must be restarted after this change.

ZMQ configured but new blocks still arrive late

Symptom: the dashboard shows the health warning bitcoin_zmq_hashblock_silent, and the log has new tip discovered by poll although zmq hashblock watchers are configured lines. zmq_hashblock_addr is set, the watcher connects, but no hashblock event ever arrives — the pool only learns about new blocks from the poll tick, up to job_refresh_secs late.

Causes, in the order we have seen them: the address points at a port bitcoind does not publish zmqpubhashblock on (compare bitcoin-cli getzmqnotifications), bitcoind publishes on 127.0.0.1 while the pool runs in a container or on another host, a firewall between the two, or bitcoind restarted on a new container IP. Fix the address, restart the daemon, and the counter resets with the first real ZMQ event. Details of the detection are in Setup Health-Checks. The IPC template source is not affected (it pushes tips itself).

ZMQ connection refused / timeout

Bitcoin Core isn’t listening on the ZMQ port. Diagnosis:

ss -tlnp | grep 28332
# Expected output: bitcoind listens on 0.0.0.0:28332 or 127.0.0.1:28332

If nothing: the bitcoin.conf line is missing or Bitcoin Core wasn’t restarted.

If 0.0.0.0: ZMQ is exposed — often necessary with a Docker bridge (daemon and bitcoind in different containers). On a direct host tcp://127.0.0.1:28332 should be enough.

Stratum / miner connections

Workers reconnect-loop every few seconds

Symptom: many session ended with shares_accepted = 0 in the logs. Many miner-disconnect events in the notifier (or, with Phase 15 debounce, one every 30s).

Common causes:

• Wrong worker address: a V1 miner submits an address the ShareValidator doesn’t accept. Logs show mining.authorize with a bad address.
• VarDiff too aggressive: the initial diff was too high, the miner can’t produce shares within the target_seconds_per_share window, retargets down quickly, and some firmwares don’t like rapid retargets. Fix: a higher min_diff or a lower initial_diff in [vardiff].
• Auth rate limit kicks in: at >auths_per_sec auth attempts per IP you get rejected. In the log: auth rate-limited. For legitimate cases with multiple workers behind NAT: raise the rate limit.

Miner can’t connect at all

# Is the pool port reachable?
nc -v pool.local 3333
# TLS port?
openssl s_client -connect pool.local:3334 -showcerts

If local is OK but external isn’t: firewall / NAT. The UPnP wizard in setup (Phase 11) can open this automatically, but only if the FritzBox / router allows UPnP. Some ISP routers have UPnP off by default.

External miner suddenly can’t connect (auto-banned)

A public IP that repeatedly trips the connect rate limit is auto-banned and dropped before the handshake (default: 10 strikes / 60 s → escalating 5 min … 1 h). Loopback and private LAN are never auto-banned. Check the ban state via /metrics (warppool_auto_bans_active, warppool_manual_bans) or the Admin → Profile panel. Clear a manual ban with POST /api/admin/ratelimit/unban; relax auto-ban via [ratelimit].autoban_* (or autoban_enabled = false).

Dashboard shows “API unreachable → HTTP 429”

The HTTP API is per-IP rate-limited (api_rpm, profile-driven; over the limit → HTTP 429 + Retry-After). Loopback and private LAN are exempt (the LAN exemption was widened in v1.4.1 — before that, reaching the dashboard over the box’s LAN IP with several tabs + polling could trip it). If you still see it from a genuinely remote IP, switch to a larger profile (bigger api_rpm) or put a reverse proxy with its own limits in front.

Miner is on the wrong protocol port

Each protocol has its own endpoint: Stratum V1 plain on 3333, V1-over-TLS on 3334, Stratum V2 on 34254 (defaults; the connect modal in the UI shows the exact host:port for all three). A miner in SV2 mode pointed at 3333 fails its NOISE handshake against the line-based V1 listener; a V1 miner on 34254 dies in the encrypted framing. Since v1.0.22 both listeners detect this mismatch and log a warning naming the right port — grep -i "point it at" in the pool log finds it. (Real-world example: #60, a Bitaxe in SV2 mode configured with the V1 port.)

Sv2: standard or extended channel?

Use an extended channel when you connect to a pool you do not run yourself — including this one. It is a real setting, not something the firmware decides for you: in AxeOS it appears as SV2 Channel Type once the pool’s protocol is set to V2 (that is why you won’t see it while the pool is still on V1). Not every firmware leaves the choice open: a Bitaxe Max (BM1397) cannot use standard channels at all, and NerdQAxe firmware currently opens an extended channel regardless — the config field exists, but the code that reads it is commented out (main/stratum/stratum_task_v2.cpp, “Extended Channel only for now”).

The reason is verification, not throughput. On an extended channel the pool hands over the coinbase halves and the merkle path, so your miner can check for itself what it is hashing and that the reward output is the one you configured. On a standard channel it receives a finished merkle root and has to take the pool’s word for it. AxeOS puts the same advice in its own tooltip: extended for external pools, standard only with a locally deployed Sv2 node or JD client.

The counter-argument is real but smaller: extended costs bigger jobs and per-job merkle work on a small MCU, and a single ASIC does not need the extra search space — nonce plus BIP-320 version rolling is already far more than its hash rate can exhaust. That is why an aggregator wants extended for a different reason entirely (splitting one connection across many downstream miners — which is why our own V1↔V2 translator opens an extended channel upstream), and why a device mining to its own local node can reasonably stay on standard.

This pool serves both kinds fully and needs no operator configuration for either. Each channel gets its own extranonce prefix (since v1.25.1 — before that, the first channel of every connection was handed the same one, so several directly connected Sv2 miners searched the same space), and VarDiff converges per channel for both kinds (since v1.20.1). If two of your Sv2 miners show a different difficulty, that is convergence state, not a quality difference — see VarDiff.

The wire-level detail is in Architecture → Standard vs. extended channels.

Sv2: debugging a client’s handshake against this pool

Every stage of the NOISE handshake and the first frames after it are logged at debug level under the warppool_stratum_v2 target (the default RUST_LOG=info,warppool=debug already includes it). For a client that cannot get past the handshake, look for these lines in order:

Note that a client-side Noise(Aead) right after act2 almost always means the client encrypted header and payload as one chunk or prefixed a length. If the frame sizes are right (22 + N+16 bytes) and the server still reports the first frame as undecryptable, the transport key is wrong rather than the framing — seen in the wild as the two cipher states swapped after the Noise split (the initiator must send with the first HKDF output, k1). The reference implementation for comparison is SRI’s noise_sv2, which this pool uses unchanged.

To pin this down without a running pool, the crate ships a one-shot probe that runs the real responder handshake and reports per AEAD chunk:

cargo run -p warppool-stratum-v2 --example noise_probe          # binds 127.0.0.1:34255

It prints a fresh authority pubkey to give to the client under test. A header-chunk failure means the key (nonce 0 is all zeros in every encoding); a payload-only failure means the nonce (endianness or not advancing per chunk).

Sv2 miner shows low-difficulty rejects that look wrong

Both stratum crates carry a dedicated diagnostic log target for the low-difficulty share verdict. Enable it without touching anything else:

RUST_LOG=info,warppool_stratum_v1::lowdiff_diag=debug,warppool_stratum_v2::lowdiff_diag=debug

Each rejected share then logs the target it was checked against, the share hash, and a re-check against the recent VarDiff history (the ring of targets the session handed out). A share that passes the re-check was rejected by a target the miner never received — report that, it is a pool bug. A share that fails every historical target really was below difficulty.

TLS: handshake fails with BadCertificate

Symptom: the miner works on port 3333 but not on the TLS port; the daemon log shows, once per reconnect attempt:

WARN tls handshake failed peer=192.168.x.y:nnnnn error=received fatal alert: BadCertificate

The alert comes from the miner: its firmware verifies the server certificate and rejects the pool’s self-signed one. What works depends entirely on what the firmware offers (ESP-Miner family naming):

A bundled-CA-only firmware insists on a certificate signed by a public CA — that is the whole point of the check, so there is no self-signed workaround. Realistic options:

1. Stay on plain :3333 for that miner. In a home LAN this is a perfectly reasonable trade-off (TLS on Stratum mainly guards against on-path tampering; the share data itself is not secret).
2. Serve a CA-signed certificate: if you own a domain, issue e.g. a Let’s Encrypt certificate for pool.example.com (DNS-01 challenge — the host doesn’t need to be publicly reachable), point the DNS record at the pool’s LAN IP, set [stratum] tls_cert_path/tls_key_path to the issued files and connect the miner to pool.example.com:3334. Renewal (90 days for LE) is your responsibility; the daemon picks up the renewed files on its own (tls_reload_secs, default 5 min) and the connect modal then drops the self-signed warning — firmware with a bundled CA store needs nothing extra.
3. Ask the vendor for a firmware update — ESP-Miner mainline already ships the “TLS (Custom CA certificate)” mode, so forks usually pick it up over time.

A plaintext miner pointed at the TLS port produces a different error (tls handshake failed ... received corrupt message-style) — that is a forgotten SSL toggle, not a certificate problem.

Notifier doesn’t fire

notifier sink skipped: env var foo not set

At daemon start, the env vars for the sink aren’t present. With systemd:

[Service]
EnvironmentFile=/etc/default/dvb-warppool

With /etc/default/dvb-warppool as a chmod-600 file:

TELEGRAM_BOT_TOKEN=123:abc...
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
POOL_SMTP_PASSWORD=...

For Docker: via -e KEY=val or an env-file in docker-compose.yml.

Test button shows err

Click Test in the admin UI → err badge. The specific error shows as a tooltip on the badge (hover). Common cases:

• Telegram: bot not found — wrong bot_token. chat not found — wrong chat_id or the bot was removed from the chat.
• Discord: Unknown Webhook — webhook was deleted.
• Slack: invalid_payload — very rare; usually a typo in the webhook URL.
• Email: Could not resolve host — wrong SMTP host. authentication failed — wrong password env. unable to connect: connection refused — wrong port or firewall.

Spam from miner-disconnect events

Per-worker debounce is 30s (WARPPOOL_DISCONNECT_DEBOUNCE_SECS). If that’s too noisy: set the env var to 300 (5min).

If a single worker keeps disconnecting: that’s a real problem on the miner side — check the logs on the miner hardware (the Bitaxe web interface has Logs; Antminer has SSH access).

Auto-update

Update fails: download/verify: sha256 mismatch

The release asset was changed between when you saw the release and downloaded it (rare), or you passed the wrong sha256. For POST /api/admin/update, always take the sha256 from the release notes file or from the forge API itself — never type it manually.

Update applied but daemon keeps restarting (StartLimitBurst)

The Phase 8f rollback hook fired. The systemd journal shows:

OnFailure=dvb-warppool-rollback.service triggered

rollback.sh copies the .backup back and runs systemctl restart. If the backup is also broken: install the old version manually or roll back from a snapshot.

cosign verify-blob failed

When cosign_verify=true is set in the update request: the signature doesn’t match the asset. Possible causes:

• The asset is authentic but the signature file is stale
• The asset has been tampered with

The default is cosign_verify=false — only use cosign if you’ve explicitly configured the trust anchor (public key or Sigstore root).

Setup wizard

“Cookie not readable”

Most likely a permission issue. The setup wizard shows the exact path + suggested fix:

sudo usermod -a -G bitcoin warppool   # one-time
sudo chmod g+r ~/.bitcoin/.cookie     # after every Core restart

Cleaner: configure bitcoind with rpcauth=warppool:saltedhash and use user/pass mode instead of the cookie.

UPnP test “no gateway found”

The router has UPnP off, or the daemon is bound to the wrong interface. Some ISP routers (Vodafone, some FritzBoxes with a locked-down user account) don’t allow UPnP from the outside.

Fix:

• Enable UPnP in the router web interface
• Alternatively: set port forwarding manually (3333/3334)

bitcoin.conf snippet suggests recommendations for keys that are already set

If the setup wizard says “add zmqpubhashblock=...” even though it’s already there — usually the wrong conf path. Check bitcoin-cli -datadir=… getrpcinfo to see the datadir that’s actually in use.

API / UI

/api/admin/* returns 401

Admin auth is configured but the token is missing from the request:

curl -H "Authorization: Bearer wpat_…" http://pool.local:18334/api/admin/notifier/sinks

Generate a token with dvb-warppool-cli token-create -n my-script — the raw token is shown only once.

/api/admin/* returns 503 “auth disabled”

secrets.toml has no admin_password_hash or jwt_secret. Fix:

dvb-warppool-cli hash-password

and paste the output into secrets.toml.

UI doesn’t load — Failed to fetch /api/...

Direct --ui-dir mode or dev mode? In dev (pnpm dev), Vite runs with a proxy /api → http://127.0.0.1:18334. In production: the UI is served directly by the daemon via --ui-dir. Check the daemon startup log:

serving static UI from daemon ui_dir=/usr/share/dvb-warppool/ui

If not present: the --ui-dir flag is missing or the directory doesn’t exist.

Announcement banner doesn’t show up

The marquee announcement (Admin → Banner, since v1.21.0) is stored as two pool_settings rows — text plus an RFC3339 expiry — and served in the banner field of GET /api/overview. It is suppressed when the text is empty, the expiry is missing/unparseable, or the expiry is already in the past — in that case the field is omitted from the response entirely, not sent as an empty string. So:

curl -s http://pool.local:18334/api/overview | grep -o '"banner":[^}]*}'

If that prints nothing, the announcement has expired (or was cleared) — re-post it via Admin → Banner or POST /api/admin/banner with {"text": "...", "duration_secs": 3600}. Constraints the API enforces: at most 200 characters, no control characters, and duration_secs between 60 (1 minute) and 7776000 (90 days). Ending it early is DELETE /api/admin/banner; both writes land in the audit log (banner.set / banner.clear).

Vendor probes (Phase 22)

Miner is discovered via mDNS but no telemetry

Possible causes:

vendor=avalon and vendor=avalonq are separate adapters — one matches MTavg/MTmax/GHS av, the other THSspd/TMax/Cur_Load. If you add a Q with vendor=avalon, lots of None values come back.

Manual add command

curl -X POST -H "Authorization: Bearer wpat_..." \
  -H "Content-Type: application/json" \
  -d '{"host":"192.168.1.42:4028","vendor":"avalonq","label":"avalonq-1"}' \
  http://pool.local:18334/api/admin/miners

Port 4028 is the CGMiner default. Some vendors reconfigure it — in that case, pass the correct port number.

Last resort

Daemon panicked and won’t come up

Look at the logs. Common cases:

• DB migration failed: sqlx::migrate throws on an inconsistent schema
• Config file broken: TOML parse error with line number
• Port taken: another pool / service on 3333

Rebuild storage DB from scratch

systemctl stop dvb-warppool
mv /var/lib/dvb-warppool/warppool.db /var/lib/dvb-warppool/warppool.db.broken
systemctl start dvb-warppool   # daemon creates a fresh DB and runs migrations

You will lose:

• VarDiff snapshots per worker (every miner starts at initial_diff)
• Aggregated hashrate history (raw shares > 1h are evicted anyway)
• Audit log

Blocks-found is NOT the source of truth in the DB — those live in the Bitcoin blockchain itself. The block list is lazily re-populated on the first request via getblock RPCs.

See also

• Setup health checks — preventive diagnostics during setup
• Auto-update — update path + rollback
• Configuration Reference