Microservices Patterns

Closed (normal operation): requests pass through to the downstream service. Failures are counted. Transition to Open when: failure rate exceeds the threshold (e.g., 50% of calls in a 10-request sliding window), or slow call rate exceeds threshold (e.g., 80% of calls exceed 2s). Open (fail fast): all requests immediately return a failure or fallback without calling the downstream. A timer starts. Transition to Half-Open after the wait duration (e.g., 30s). Half-Open (probe): a limited number of probe requests are allowed through. If they succeed → transitions back to Closed. If they fail → returns to Open. Configuration knobs (Resilience4j): - failureRateThreshold: % failures to open (default 50) - slidingWindowSize: number of calls in the window - waitDurationInOpenState: how long to stay open before probing - permittedNumberOfCallsInHalfOpenState: probe count Fallback: always pair with a fallback — return cached data, a default, or a degraded response so the caller doesn't surface a raw failure.

A saga is a sequence of local transactions coordinated via events or commands. Each step performs a local transaction and either publishes an event (choreography) or sends a command to the next participant (orchestration). On failure, previously completed steps are undone via compensating transactions. Why not 2PC (two-phase commit)? 2PC requires all participants to hold locks during the prepare phase, blocking other operations. In a microservices architecture spanning multiple independent services and databases, 2PC is impractical — it requires all participants to be available simultaneously, introduces tight coupling, and kills availability (if one participant is down, the whole transaction blocks). 2PC violates the "each service owns its data" principle. When to use sagas: - Any multi-service operation that must be atomic from a business perspective (create order + reserve inventory + charge payment) - When strong consistency across services is impossible or undesirable - When compensating transactions can faithfully undo business effects When sagas are hard: - Compensations that can't be undone (you can't "unsend" an email — use a flag) - Long-running sagas with many steps and many failure paths - Maintaining correct ordering of compensations when multiple steps fail

API Gateway: north-south traffic (external clients → internal services). Handles: authentication, rate limiting, routing, SSL termination, request/response transformation, documentation, API versioning. Operates at L7 (HTTP application layer). One gateway per product or organization. Service Mesh: east-west traffic (service → service inside the cluster). Handles: mTLS, retries, timeouts, circuit breaking, load balancing, distributed tracing, traffic shaping (canary). Operates at L4/L7 via sidecar proxies. Applied uniformly across all services without app code changes. Overlap: both can do load balancing, retries, and observability at their respective layers. But they serve fundamentally different traffic directions. When you need both: large orgs with many internal services. The API gateway secures and routes external traffic; the service mesh provides internal zero-trust networking and consistent observability without per-service implementation. When a gateway is enough: early stage, few services, internal calls are simple. Add a mesh when internal security (mTLS) and consistent cross-service observability become pain points.

Problem: after a service writes to its DB, it must publish an event. If the app publishes to the broker and then the process crashes before the DB commits (or vice versa), you get inconsistency: DB updated but no event, or event published but DB rolled back. Outbox pattern: 1. Within a single local DB transaction, write both the business record AND an outbox_events row: {id, aggregate_type, aggregate_id, event_type, payload, created_at} 2. Transaction commits atomically — both or neither 3. A separate relay process reads unpublished outbox rows and publishes them to the broker (Kafka, SQS) 4. On successful publish, mark the row as published (or delete it) Relay options: - Polling: relay polls outbox_events WHERE published = false every N seconds. Simple, slightly delayed. - CDC (Debezium): captures DB changes via transaction log (binlog/WAL) and streams them to Kafka without polling. Near-real-time, lower DB load.

At-least-once: if the relay crashes after publishing but before marking as published, it republishes. Consumers must be idempotent (deduplicate by event ID).

Distributed tracing connects a single user request across all the services it touches into a single trace, composed of spans (one per operation). How it works: 1. The first service (or API gateway) generates a trace-id and a span-id 2. These are propagated via HTTP headers: traceparent (W3C standard), or X-B3-TraceId/X-B3-SpanId (Zipkin B3 format) 3. Each downstream service reads the incoming headers, creates a child span (inheriting the trace-id), does its work, and reports the span to a collector 4. The tracing backend (Jaeger, Tempo, Zipkin) assembles spans into a trace waterfall Instrumentation: - Auto-instrumentation: agent-based (Java agent, OpenTelemetry SDK) — instruments HTTP clients, DB drivers, message consumers automatically - Manual spans: add spans around business-critical operations not covered by auto-instrumentation

OpenTelemetry is now the standard: vendor-neutral SDK that exports to any backend (Jaeger, Zipkin, Tempo, Datadog, Honeycomb). Sampling: never trace 100% in production. Use adaptive sampling: trace 100% of errors, 1–5% of successful requests. Head-based or tail-based sampling.

Named after the fig tree that grows around a host tree, gradually replacing it. Instead of a risky big-bang rewrite, you incrementally extract functionality from the monolith while keeping it running. Steps: 1. Put an API gateway (or routing layer) in front of the monolith 2. Choose the first capability to extract — prefer low-coupling, high-value, or high-change-frequency modules. Avoid shared DB tables. 3. Build the new microservice for that capability 4. Run it in shadow mode: route production traffic to both; compare responses; fix discrepancies 5. Gradually shift traffic: 1% → 10% → 50% → 100% (canary via feature flag or gateway weight) 6. Remove the corresponding code from the monolith 7. Repeat Key enablers: the gateway provides a stable external URL regardless of where the code lives. The monolith and new service coexist; no forced cutover. Pitfall: don't extract services before untangling the shared database. Extract the data ownership first (create a separate schema or sidecar DB) before splitting the service boundary.

A saga step failure leaves some local transactions committed and some not. Consistency is restored via compensating transactions that undo the committed steps. Compensation requirements: - Every saga step must have a corresponding compensating transaction defined upfront - Compensations must be idempotent — they may be retried on failure - Compensations must be semantically correct: undoing "charge $100" is "refund $100", not just "delete the charge record" - Some operations can't be undone (sending an email). Use a pivot transaction: schedule the irreversible action only after all preceding steps succeed, or use a flag to suppress the action if the saga is later rolled back

Failure handling in orchestration: - Orchestrator detects failure (timeout, explicit failure response) - Triggers compensations in reverse order - Compensations are retried with backoff on failure; if all retries fail, saga enters a "stuck" state requiring manual intervention — alert on this

Semantic lock (countermeasure): mark the resource as "pending" during the saga. Prevents other transactions from reading a stale committed intermediate state.

The bulkhead isolates resource pools (threads, connections, semaphores) per downstream dependency. Without it, a slow downstream A blocks threads → shared thread pool exhausted → requests to healthy downstream B also queue → entire service fails. Thread pool isolation: Each downstream gets a dedicated thread pool (e.g., 10 threads for Payment service, 10 for Inventory service). A slow Payment call blocks only its 10 threads; Inventory calls still have their pool. Total threads = sum of pools + core app threads. Semaphore isolation: Instead of separate threads, limit concurrent calls via a semaphore (counter). Cheaper (no thread switching), but if the downstream blocks a thread, that thread is still occupied — you just cap concurrent callers. Good for fast, in-process calls. In practice (Resilience4j): java BulkheadConfig config = BulkheadConfig.custom() .maxConcurrentCalls(10) .maxWaitDuration(Duration.ofMillis(500)) .build(); Infrastructure level: Kubernetes resource requests/limits — each pod gets bounded CPU/memory so a runaway service can't starve neighbors. Pair bulkhead with circuit breaker: bulkhead limits load; circuit breaker stops calls to failed services. Together they contain failures within defined boundaries.

mTLS (Mutual TLS): Both client and server present certificates. A trusted CA (Vault PKI, cert-manager) issues short-lived certificates to each service. Each service verifies the other's cert. No shared secret — each service has its own cert/key pair. Cert rotation is automatic. In a service mesh (Istio), this happens at the sidecar level without app code changes. JWT with service identity: Services obtain a short-lived JWT from an identity provider (Vault, Keycloak) using their service account credentials. Include service identity as a claim. Downstream services verify the JWT signature (public key). Tokens expire quickly (5–15 min); services refresh automatically. Kubernetes ServiceAccount tokens: Kubernetes injects a ServiceAccount token into each pod. Services can use this to authenticate to other Kubernetes-native services. Bound tokens include namespace and service account claims — downstream can authorize based on these. Best practice: use mTLS at the transport layer (mesh handles it) for encryption and identity; combine with AuthorizationPolicy (which service can call what) enforced by the mesh control plane. This gives you encryption, authentication, and authorization without app code changes.

CQRS (Command Query Responsibility Segregation) separates write operations (commands that change state) from read operations (queries that return data). The write model handles validation and business rules; the read model is a denormalized projection optimized for specific query patterns. Why separate them? - Write model enforces invariants (requires consistency); read model optimizes for query performance (can be denormalized, cached, eventually consistent) - Read and write loads are typically asymmetric (10:1 reads:writes) — scale independently - Read projections can be tailored per consumer (dashboard view, search index, mobile summary) without changing the write model

Implementation patterns: - Simple CQRS (no event sourcing): same DB, separate code paths. Write path updates normalized tables; read path queries denormalized views or materialized views. - Full CQRS + Event Sourcing: commands produce events; events update separate read stores (Elasticsearch, Redis, read DB). Read stores are eventually consistent.

When NOT to use: - Simple CRUD applications — adds complexity without benefit - When strong consistency between write and read is required - Small teams without the operational capacity to maintain dual stores CQRS is a pattern, not a requirement. Apply it to specific aggregates/domains with high read/write asymmetry, not globally across the entire system.

Flow (Orchestration with Temporal): CreateOrderSaga: 1. orders-service: create order (status=PENDING) 2. inventory-service: reserve items 3. payment-service: charge customer 4. orders-service: confirm order (status=CONFIRMED) 5. notification-service: send confirmation (async, fire-and-forget) Compensations (reverse order on failure): - Payment fails: release inventory → cancel order - Inventory fails: cancel order (no payment taken yet) - Confirmation fails: release payment → release inventory → cancel order Implementation details: - Orchestrator: Temporal workflow. Each activity is idempotent (activity ID as idempotency key). Temporal handles retries, timeouts, and state persistence. - Each service exposes: reserve(), release() (compensation), charge(), refund(). All idempotent by saga step ID. - Order status machine: PENDING → INVENTORY_RESERVED → PAYMENT_CHARGED → CONFIRMED | INVENTORY_FAILED → CANCELLED | PAYMENT_FAILED → INVENTORY_RELEASED → CANCELLED - Semantic lock: order in PENDING blocks concurrent saga on same order. Observability: - Trace ID propagated through all saga steps - Saga state persisted in Temporal — visible in Temporal UI - Metrics: saga success rate, step failure rate, compensation rate, saga duration P99 - Alert: sagas stuck in non-terminal state > 5 min → manual intervention queue

Goal: route a small percentage of traffic to a new version; compare error rate, latency, and business metrics; roll forward or roll back automatically. Istio-based canary: yaml # VirtualService: 5% to v2, 95% to v1 - destination: { host: orders-service, subset: v1 } weight: 95 - destination: { host: orders-service, subset: v2 } weight: 5 --- # DestinationRule: defines v1 and v2 subsets by pod label subsets: - name: v1 labels: { version: v1 } - name: v2 labels: { version: v2 } Progressive delivery automation (Flagger/Argo Rollouts): - Start at 5% → auto-promote to 10% → 20% → 50% → 100% on success - Rollback trigger: error rate > 1% OR P99 latency > SLO threshold - Analysis period: 5 minutes at each step before promoting - Metrics from Prometheus/Istio telemetry — no manual intervention needed What to measure per canary step: - HTTP 5xx error rate (primary signal) - P99 request latency - Business metrics (order conversion rate, payment success rate via custom Prometheus metrics) - DB error rate, downstream dependency error rate Header-based routing: before percentage rollout, route internal testers to v2 via X-Canary: true header. Validates v2 with real data before any external traffic. Database migration: canary deployments require backward-compatible DB changes. v1 and v2 must read/write the same schema. Use expand-contract for schema changes.

Kubernetes resource model: every container declares requests (guaranteed) and limits (maximum). The scheduler places pods on nodes with sufficient requested resources. requests = what the container needs under normal load. limits = cap to prevent runaway consumption. Setting requests correctly: - Profile under realistic load (load test): P95 CPU and memory during peak - Set requests to P95 steady-state, limits to 2× requests (headroom for bursts) - For memory: set requests == limits to avoid OOMKilled evictions under pressure (memory is not compressible like CPU)

Bulkheads at cluster level: - Namespace resource quotas: cap total CPU/memory per team namespace. Prevents one team's runaway service from exhausting cluster capacity. - LimitRange: sets default requests/limits for pods that don't specify them. - Priority classes: critical services (auth, payment) get higher priority — scheduler evicts lower-priority pods first under pressure.

Autoscaling: - HPA (Horizontal Pod Autoscaler): scale pods on CPU, memory, or custom metrics (Kafka consumer lag, queue depth via KEDA) - VPA (Vertical Pod Autoscaler): recommends correct requests/limits based on observed usage - Cluster Autoscaler: adds/removes nodes based on pending pod pressure Capacity planning process: - Baseline: measure CPU/memory/RPS per service under current load - Forecast: project traffic growth (30/60/90 day) - Load test: validate the new service handles projected peak × 1.5 safety margin - Review: revisit resource settings quarterly; shrink over-provisioned services

Schema evolution strategies: Backward-compatible (producer adds): - Add optional fields: consumers using older schema ignore unknown fields (if consumer is lenient — Avro, Protobuf handle this natively; JSON requires lenient deserialization) - Avro/Protobuf evolution rules: add fields with defaults; never remove or rename Breaking changes (producer removes/renames): - Dual-write transition: producer publishes both v1 and v2 events simultaneously. Consumers migrate to v2 at their own pace. Producer drops v1 after all consumers migrate (tracked via consumer group lag metrics). - Event versioning: include schema_version: "v2" in event headers. Consumers route by version — handle v1 and v2 with separate handlers during transition. - Schema registry (Confluent): enforces compatibility rules (BACKWARD, FORWARD, FULL) before a schema can be registered. CI validates new schema against registry before deploy.

Consumer-driven contract testing (Pact): - Consumers publish their expected event schema as a pact - Producer CI runs against the pact broker; build fails if new event schema breaks a consumer contract - Catches breaking changes before they reach production Pitfall: never share a code library of event classes between services — it couples service deployments. Each service owns its own event schema representation. Use a schema registry for the canonical schema, not a shared library.

Conway's Law: your architecture mirrors your communication structure. A monolith built by one team; microservices built by many teams. The implication: design team structure and service boundaries together, not separately. Team Topologies framework: - Stream-aligned teams: each team owns a product domain end-to-end (Orders, Payments, Catalog). Services align to team boundaries. - Platform team: owns shared infrastructure (auth, observability, CI/CD, service mesh). Other teams consume it as self-service. - Enabling teams: temporary; help stream teams adopt new patterns. - Complicated-subsystem teams: own high-expertise components (ML pipeline, real-time pricing engine).

Service granularity principles: - One service per team (not per class or DB table). If two services are always deployed together, they're one service owned by one team. - Service size: small enough to be understood by the team, large enough to be worth the operational overhead. "How long to onboard a new engineer?" is a good proxy. - Start coarser, split later. Merging overly-split services is much harder than splitting an oversized one.

Wrong granularity signals: - Teams routinely coordinate multi-service deployments → boundaries are wrong - Teams step on each other's changes in the same service → split needed - A service has very low traffic but high operational cost → merge - Services that can't be tested independently → excessive coupling My framing for the interview: microservices are an organizational scaling pattern as much as a technical one. You don't adopt microservices because it's better architecture; you adopt it because your team structure demands independent deployment velocity. If you have 5 engineers, you almost certainly shouldn't have 20 services.

Immediate diagnosis: - Pull stuck sagas from Temporal/Step Functions dashboard. Group by: which step fails, which service, what error type (timeout? business exception? infra error?) - Correlate with: deployment times (did failure rate increase after a deploy?), infrastructure events (DB slow query, network blip), specific customers/order types - Check compensation success rate: are compensations themselves failing? Double-failure is the worst case — payment charged, inventory not reserved, compensation can't refund

Root cause categories: - Flaky downstream (payment processor 2% timeout): add retry with backoff inside the saga step; ensure idempotency key on processor call so retries don't double-charge - Non-idempotent step being retried: audit every step — does retrying produce duplicate side effects? Fix: idempotency key per saga step ID - Timeout too tight: step timeout < downstream P99. Instrument downstream P99 per saga step; set timeout at P99 + 20% buffer - Data race in semantic lock: two sagas on the same order_id competing. Fix: advisory lock or optimistic concurrency (version field) on the order record

Operational fixes (immediate): - Manual resolution workflow: ops console to inspect stuck sagas, see exact state, trigger retry or rollback. This unblocks customers now. - Alert: stuck saga > 10 min → PagerDuty. SLA: resolve or escalate within 30 min. Systemic fixes (weeks): - Saga observability dashboard: success rate, step failure rate, median/P99 duration, compensation rate — per saga type, per step - Chaos engineering: inject failures at each saga step; validate compensations work; validate stuck saga detection and alerting fire correctly - Backpressure: if saga failure rate > 5%, pause new saga creation (circuit breaker at saga entry point) rather than accumulating more stuck sagas