What are the failure modes and design pitfalls when deploying MQTT?

A handful recur. The broker is a central single point of failure and a scaling chokepoint, if it goes down, everyone is cut off, so production deployments cluster or run the broker in HA and monitor its load, and a careless wildcard subscriber (#) can drown itself or the broker in traffic. Keepalive must be set shorter than the NAT/firewall idle timeout on the path, or an idle connection is silently dropped and the device appears connected but isn't until it notices and reconnects; tuning keepalive trades responsiveness against battery/data. Topic design is effectively permanent: the hierarchy is your routing scheme and your API, so renaming topics after devices ship is painful and must be planned up front with a clear, stable namespace. QoS misunderstandings bite, assuming it's end-to-end, or ignoring that QoS 1 duplicates so handlers must be idempotent. Security is opt-in: shipping plaintext MQTT exposes data and credentials, so TLS plus authentication and broker ACLs are required. And reconnection logic matters: devices must handle broker restarts and network drops with backoff, re-subscribe on reconnect (unless using persistent sessions), and combine retained status with LWT so presence stays accurate across reconnects. The throughline is that MQTT's simplicity pushes responsibility onto your design, broker availability, keepalive vs NAT, topic namespace, QoS semantics, security, and robust reconnection are all things you must get right operationally.