WebSockets

The client sends an HTTP/1.1 GET request with four key headers: Upgrade: websocket, Connection: Upgrade, Sec-WebSocket-Key: <base64 random 16 bytes>, and Sec-WebSocket-Version: 13. The server responds with 101 Switching Protocols and a Sec-WebSocket-Accept header computed as base64(sha1(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11")). The GUID is fixed in the spec — it's there to prevent a misconfigured HTTP server from accidentally accepting a WebSocket upgrade. After 101, the TCP connection is no longer used for HTTP. Both sides switch to the WebSocket framing protocol on the same socket. No new connection is created — the upgrade repurposes the existing TCP stream. The handshake being HTTP is intentional: it lets WebSockets work through firewalls on port 443 that allow HTTPS, and it gives intermediaries (proxies, load balancers) a familiar signal to route the connection correctly.

Each WebSocket message is split into one or more frames. A frame has a header containing: FIN bit (1 = this is the last frame of the message), RSV bits (reserved for extensions), a 4-bit opcode, a MASK bit, and a 7-bit payload length (extended to 16 or 64 bits for larger payloads). For masked frames, a 4-byte masking key follows the length, and the payload is XORed byte-by-byte with the key cycling across all 4 bytes. Why masking? A browser is a shared environment — any website can open a WebSocket to any server. Without masking, a malicious page could send carefully crafted binary frames that look like valid HTTP requests to an intermediate proxy. If a proxy caches those "responses," a subsequent request from a legitimate user could be served poisoned data. Masking makes the wire bytes unpredictable, breaking this attack. Server-to-client frames are not masked because servers are trusted and don't operate in the shared-origin browser environment.

The browser's WebSocket constructor does not accept custom headers — there is no option to pass Authorization: Bearer <token>. This is a spec limitation. Three patterns, ordered by preference: Cookie auth — if the user is already authenticated via cookie on the same origin, the browser automatically includes cookies in the upgrade request. The server validates the session cookie. No special handling needed. Works only for same-origin. Token in the first message — connect without authentication, then immediately send an auth message as the first WebSocket frame. The server enforces a strict timeout (e.g., 5s): if no valid auth message arrives, it closes with 1008 Policy Violation. Prevents anonymous connections from sitting open. Short-lived token in query string — wss://host/ws?token=<jwt>. Tokens appear in server access logs and browser history. Mitigate by generating single-use, short-TTL tokens on a REST endpoint immediately before connecting. The token exchange endpoint is authenticated via cookie or normal Authorization header.

An unexpected drop fires onerror (if there's a transport error) immediately followed by onclose with code 1006 (abnormal closure — no Close frame was sent, the TCP connection just died). The browser does not automatically reconnect. A robust reconnection implementation: 1. Exponential backoff with jitter — first retry at ~1s, double each attempt up to a cap (~30s), add random jitter (±25%) to prevent thundering herds when a server restarts and many clients reconnect simultaneously. 2. Reconnect on 1006, 1001, 1011 — all indicate unclean closure. Don't reconnect on 1000 (normal close) or 1008 (policy violation) — those are intentional. 3. Re-authenticate on reconnect — the new connection has no memory of the old one; session state must be re-established. 4. Re-subscribe to channels — if using pub/sub, the server has no record of what the new connection was subscribed to; the client must re-declare subscriptions in onopen. 5. Track missed messages — if ordering matters, send a last_received_seq to the server on reconnect so it can replay what was missed.

TCP does not detect a dead peer without sending data. A client that loses network connectivity (mobile backgrounded, cable unplugged) does not send a Close frame — the server's socket stays ESTABLISHED indefinitely with no data flowing. The solution is application-level heartbeats via Ping/Pong: The server sends a Ping frame every N seconds (e.g., 30s). The client must respond with a Pong. If no Pong arrives within a timeout (e.g., 10s), the server closes the socket and cleans up the connection record. Implementation: most server frameworks (Spring WebSocket, Netty, Node.js ws) handle Ping/Pong automatically if configured. The key parameters are pingInterval and pingTimeout. Set the interval shorter than the load balancer's idle timeout — typically 25–45s if the LB drops connections after 60s. Also monitor OS-level socket state — ss -tnp | grep CLOSE_WAIT — CLOSE_WAIT sockets that are not being closed indicate the server is not handling Close frames in application code.

The fundamental problem: a WebSocket connection is pinned to one server process. A simple round-robin load balancer will route a client's reconnection to a different server, which knows nothing about that client's subscriptions or session state. Sticky sessions (session affinity): the load balancer routes connections from the same client to the same server, using IP hash or a cookie set during the handshake. This solves routing — but doesn't help with cross-server broadcasting. Cross-server messaging with a pub/sub bus: when server A needs to send a message to a user whose connection is on server B, server A publishes to a Redis channel (or Kafka topic). Server B is subscribed to that channel and delivers the message to the right connection. This is the standard pattern for chat, presence, and any broadcast/multicast use case. Stateless design where possible: move connection state (subscriptions, user metadata) out of process memory into a fast shared store (Redis). On reconnect, the new server reloads state from Redis — sticky sessions become a performance optimization, not a correctness requirement. At very high scale, sticky sessions become a load distribution problem: a "hot" server serves more connections than others, and a server restart sends all its connections reconnecting simultaneously (thundering herd). Counter with gradual shutdowns, draining connections over minutes, and circuit breakers on reconnection.

WebSocket has no built-in flow control. The server can write faster than the client can read. Unacknowledged data accumulates in the TCP send buffer; once full, the OS write call blocks (or throws in non-blocking mode). Left unhandled, one slow client can stall a server thread or exhaust buffer memory. Detection: server-side frameworks expose buffer state. In Netty: channel.isWritable() returns false when the high-water mark is breached. In Spring WebFlux: use reactive streams backpressure — Flux.sink with overflow strategies. In Node.js ws: listen for drain events after send() returns false. Strategies by use case: - Drop non-critical messages: for live tickers, dashboards, or telemetry — if the client is lagging, drop the oldest pending message and send only the latest. The client sees slightly stale data, not a crash. - Rate-limit per connection: enforce a per-session send queue with a max depth. When the queue is full, either drop or close the connection with a warning. - Close slow clients: for real-time systems where staleness is harmful (e.g., game state), disconnect clients that can't keep up. They reconnect with a fresh state snapshot. - Apply backpressure upstream: in reactive pipelines, propagate backpressure back to the data source — stop reading from Kafka/DB until the client catches up.

Server-Sent Events (SSE) when: the server pushes data to the client, the client sends infrequently (via separate HTTP calls), and you want simplicity. SSE is an HTTP chunked response with text/event-stream content type. The browser auto-reconnects, sends Last-Event-ID so the server can resume, and it works through HTTP/2 multiplexing (no sticky sessions). Proxy support is excellent — it looks like a slow HTTP response. WebSockets when: the client sends data frequently enough that opening a new HTTP request per send is too expensive (chat input, game controls, collaborative editing ops), or when you need full-duplex with sub-50ms latency in both directions. Long polling when: both WebSockets and SSE are blocked by network policy (rare now, but real in enterprise environments), or when you need a simpler fallback for old clients. Long polling is inefficient — it creates a new TCP connection per message without keep-alive, and each response triggers a new request with full HTTP headers. When the choice is wrong: using WebSockets for a notification system where the server pushes events every few minutes and the client never sends — SSE would be simpler, more scalable (no sticky sessions), and auto-reconnecting. Using SSE for a collaborative editor — the round-trip latency of a separate POST for each keystroke vs. a WebSocket frame is significant, and the implementation is more complex.

Core insight: collaborative editing is a distributed state synchronization problem, not a messaging problem. Two users editing the same document simultaneously must reach the same final state regardless of network order. The algorithm choice (OT or CRDT) drives most of the architecture. Algorithm layer: Operational Transformation (OT) — used by Google Docs — requires a central server to order all operations for a document. CRDTs (Yjs, Automerge) are peer-to-peer friendly and eventually consistent without a coordinator, at the cost of more complex merge semantics. For a centralized system with a server component, OT is more predictable; for multi-leader or offline-first, CRDTs are superior. Server topology: shard documents across servers by document ID. All connections to the same document route to the same server (sticky by document ID, not user ID). Each document's editing session is handled by a single process — this avoids the complexity of distributed OT. At 500k concurrent users, assume most documents have few active editors — a document server handles hundreds to thousands of active documents. Persistence layer: operations are appended to a log (Kafka topic or Postgres table) per document, in order. The current document state is a snapshot plus replayed operations since the snapshot. New collaborators receive the snapshot + delta, not the full history replay. Presence and cursors: a separate lightweight pub/sub channel per document (Redis Pub/Sub) broadcasts cursor positions and selection state. These are ephemeral (not persisted) and can be dropped under back-pressure without data loss. Conflict resolution: operations are transformed relative to concurrent ops by the server before acknowledgment. The client applies operations optimistically (for feel) and reconciles when the server ack arrives with the transformed operation. Scale path: 500k users across thousands of documents. Each active document shard handles ~50-200 concurrent editors. Shard servers are stateful within a session — crash recovery replays from the operation log. Global state (document metadata, user identity) lives in shared stores (Postgres, Redis) accessible to all shards.

A naive restart drops all connections on the restarting server simultaneously, triggering 40k clients to reconnect with exponential backoff — a thundering herd that can overload the reconnect target servers. Graceful drain protocol: 1. Signal the server to stop accepting new connections — remove it from the load balancer's target group (or mark it unhealthy in health checks). New connections stop routing here within one health check interval (5–15s). 2. Send Close frames to existing connections — iterate all open connections and send Close with code 1001 (Going Away) and a Retry-After-equivalent reconnect hint in the reason string (or via a custom message before closing). This signals clients to reconnect immediately rather than backing off. 3. Reconnect hint includes server exclusion — pass a flag in the close reason that tells the client to avoid this server's address for N minutes. Implement in the load balancer by removing it from rotation before draining starts. 4. Set a drain deadline — some clients will not respond to Close frames (backgrounded mobile, misbehaving clients). Force-close all remaining connections after a timeout (e.g., 60s). Accept that a small percentage will see 1006. 5. Stagger the rolling restart — restart one server at a time with a cooldown (wait until reconnect load is absorbed before starting the next drain). Monitor aggregate connection count and reconnection rate as gating signals. Reconnection amplification: each closed connection generates at least one reconnect request. 40k connections draining ≈ 40k reconnects in ~10s. Design load balancer and server capacity for 2× normal connection rate during maintenance windows.