JVM Tuning

Generational GC is based on the weak generational hypothesis: most objects die young. In a typical web application, objects created to serve a request (request objects, DTOs, parsed JSON) become garbage as soon as the request completes — usually within milliseconds. Very few objects (caches, connection pools, application state) live long. Generations: - Young Gen (Eden + Survivor S0/S1): new objects allocated in Eden. When Eden fills: Minor GC copies live objects to a Survivor space; dead objects are reclaimed. Very fast because most objects are already dead — GC only copies the few live objects (copying collection: cost proportional to live objects, not heap size). - Old Gen (Tenured): objects surviving MaxTenuringThreshold Minor GCs are promoted. Major GC collects Old Gen — slower because more objects survive, and the heap is larger.

Why it works: - Minor GC is fast (milliseconds) and frequent. - Major GC is slow but rare — only triggered when Old Gen fills up. - Total GC overhead is low because most garbage is collected cheaply in Young Gen. Tuning implication: if Minor GC is frequent and Old Gen grows quickly, objects are being promoted too aggressively. Symptoms: frequent Minor GCs followed by a rapid Full GC. Fix: increase Young Gen size (-Xmn or -XX:NewRatio) or reduce promotion rate by fixing allocation patterns.

Step 1: Confirm it's a leak (not just a large live set). Monitor heap usage over time. If heap grows steadily and Full GCs can no longer reclaim it (Old Gen keeps growing after each GC), that's a leak. jstat -gcutil <pid> 5000 shows Old Gen % trending up. Step 2: Take a heap dump. bash jmap -dump:live,format=b,file=heap.hprof <pid> live forces a Full GC first — shows only truly reachable objects. Step 3: Analyze with Eclipse MAT. - Leak Suspects Report: MAT auto-identifies objects with large retained heap - Dominator tree: shows which objects retain the most memory (dominate the GC graph) - Path to GC roots: for a suspicious object, show the reference chain keeping it alive Common causes by what MAT shows: - Large HashMap or ArrayList in a static field → static collection leak - Many instances of the same class → object pool or cache not bounded - ClassLoader instances accumulating → classloader leak (web app redeploy) - Many byte[] or char[] → String interning or large byte buffer accumulation Step 4: Fix. - Use WeakHashMap for caches (entries evicted when key is no longer strongly referenced) - Add removeEventListener / unsubscribe calls - Call ThreadLocal.remove() at end of each task in thread pools - Use bounded caches (Caffeine with maximumSize)

G1 (Garbage First) divides the heap into equal-sized regions (1MB–32MB). Each region can be Eden, Survivor, Old, or Humongous (for objects > 50% of a region). G1 selects which regions to collect based on garbage density (most garbage first — hence "Garbage First"), aiming to meet the pause time target. Key G1 behaviors: - Mixed GC: after a concurrent Old Gen marking cycle, G1 collects both Young regions and select Old regions in a single pause (mixed collection) - Humongous objects: large objects allocated directly in Old Gen; frequent humongous allocations trigger eager GC

Tuning: -XX:+UseG1GC # (default in Java 9+) -XX:MaxGCPauseMillis=100 # Pause target (200ms default). G1 adapts. -XX:G1HeapRegionSize=16m # For heaps > 8GB, set explicitly -XX:G1NewSizePercent=20 # Min Young Gen % (default 5) -XX:G1MaxNewSizePercent=40 # Max Young Gen % (default 60) -XX:ConcGCThreads=4 # Concurrent marking threads -XX:InitiatingHeapOccupancyPercent=45 # % Old Gen full before concurrent marking starts Common G1 issues: - Frequent Evacuation Failures (GC can't find free regions) → heap too small or humongous allocations - To-space exhaustion → increase heap, check for memory leaks - Long mixed GC pauses → reduce -XX:G1MixedGCLiveThresholdPercent

JFR is a built-in profiler with < 2% overhead — safe to run in production continuously. Start recording on a running JVM: bash jcmd <pid> JFR.start duration=120s filename=/tmp/app.jfr settings=profile # Or via JVM flag at startup: -XX:StartFlightRecording=duration=0,filename=/tmp/app.jfr,settings=profile Key event categories to analyze in JMC (JDK Mission Control): GC: pause durations, allocation rates, promotion rates. See if GC pauses correlate with application latency spikes. Method Profiling: CPU flame graph showing hot methods. Find: CPU hotspots in application code, unexpected library code consuming CPU, String.format() in hot paths. Lock Contention: java.util.concurrent.Lock and synchronized contention. Which locks block which threads? How long? Thread sleep/wait: threads spending time in TIMED_WAITING — why? Blocking I/O? Excessive Thread.sleep()? I/O: file reads, socket reads/writes, their latencies. Find unexpected synchronous I/O in hot paths. Allocation profiling: jfr settings=profile enables allocation profiling. Find which code paths allocate the most objects — high allocation rate = frequent Minor GC. async-profiler is an alternative for CPU and allocation profiling with lower overhead and flame graph output directly: ./profiler.sh -d 60 -f profile.html <pid>.

-Xms: initial heap size. The JVM starts with this much heap allocated. If set lower than -Xmx, the JVM may grow the heap dynamically (heap expansion can cause a Full GC pause). -Xmx: maximum heap size. The JVM will never exceed this. If exceeded, OutOfMemoryError: Java heap space is thrown. Setting -Xms == -Xmx (e.g., -Xms4g -Xmx4g): recommended in production. The JVM pre-allocates the full heap at startup. Eliminates heap expansion GC pauses. Gives a predictable, stable memory footprint. -XX:MaxRAMPercentage (Java 10+): sets -Xmx as a percentage of available RAM. The JVM reads the container cgroup memory limit (or physical RAM if no limit): -XX:MaxRAMPercentage=75.0 # heap = 75% of container memory limit In containers: always use MaxRAMPercentage instead of hardcoded -Xmx. A Kubernetes pod with memory: 2Gi limit on one node may be rescheduled to a pod with memory: 4Gi in the future. -Xmx1536m is now wrong. -XX:MaxRAMPercentage=75.0 adapts automatically. Recommended container config: -XX:InitialRAMPercentage=50.0 # start with 50% allocated -XX:MaxRAMPercentage=75.0 # cap at 75% Reserve 20–25% for: Metaspace, Code Cache, thread stacks, JVM overhead, off-heap buffers.

Detection: Thread dump analysis (jstack <pid> or kill -3 <pid>): Look for threads in BLOCKED state — they're waiting for a monitor lock. "http-nio-8080-exec-5" BLOCKED on lock held by "http-nio-8080-exec-1" at com.example.OrderService.processOrder(OrderService.java:142) waiting for <0x00000006cd45b890> (a java.util.HashMap) The lock holder and the contending threads are both visible. The object being locked (java.util.HashMap in the example) points to the root cause. JFR lock contention: JFR records lock events with duration and thread. JMC shows a breakdown of where threads spent time blocked. Common fixes: Replace synchronized collections: Collections.synchronizedMap(new HashMap<>()) → new ConcurrentHashMap<>() (fine-grained segment locking, no global lock). Reduce lock scope: hold locks for the minimum required time. Move non-critical code outside the synchronized block. ReadWriteLock: for read-heavy maps: ReentrantReadWriteLock. Multiple readers hold the read lock simultaneously; writers get exclusive access. Lock-free structures: AtomicLong, AtomicReference, LongAdder (better than AtomicLong under contention for counters — uses striped counters internally). Virtual Threads (Java 21): virtual threads block their carrier thread when they enter a synchronized block (thread pinning). Use ReentrantLock instead of synchronized in virtual thread code to avoid pinning.

Each OOM type has a different cause: OutOfMemoryError: Java heap space Heap is full. Objects can't be allocated. Either: (1) legitimate large live set — increase -Xmx; (2) memory leak — heap dump + MAT analysis; (3) allocation spike — JFR allocation profiling. OutOfMemoryError: GC overhead limit exceeded JVM spent > 98% of time in GC reclaiming < 2% of heap (consecutive collections). Nearly always a memory leak. Disable this check with -XX:-UseGCOverheadLimit (buys time but doesn't fix the leak). Heap dump + analyze. OutOfMemoryError: Metaspace Too many classes loaded. Causes: (1) excessive dynamic class generation (cglib proxies, Groovy scripts, bytecode generation frameworks creating unique classes per call); (2) classloader leaks in web app redeploys. Increase -XX:MaxMetaspaceSize temporarily; fix the classloader leak or class generation issue permanently. OutOfMemoryError: Direct buffer memory Off-heap DirectByteBuffer memory exhausted. Caused by: NIO networking, mapped files, Netty buffers not released. Increase with -XX:MaxDirectMemorySize. Profile with JFR native memory tracking. Check for unreleased buffers. OutOfMemoryError: unable to create native thread OS can't create more threads. Either: process has hit ulimit -u (thread limit per user); or the 32-bit address space can't fit more thread stacks. Reduce thread count or stack size (-Xss256k). In containers: check pids.max cgroup limit. Migrate to Virtual Threads.

Platform threads (before Java 21): each Java thread = one OS thread. OS threads are expensive (~1MB stack, kernel scheduling). Practical limit: thousands. For I/O-bound services: threads block waiting for I/O → you need large thread pools to handle many concurrent requests → high memory consumption. Virtual threads: JVM-managed, extremely lightweight (~few KB). Millions can exist. When a virtual thread blocks (I/O, sleep, lock), it unmounts from its carrier (OS) thread. The carrier thread runs other virtual threads. When I/O completes, the virtual thread remounts on any available carrier. Impact on code: write synchronous, blocking code. Get async performance automatically. No need for reactive programming (WebFlux, RxJava) just to handle concurrency. java // Spring Boot 3.2+ with virtual threads enabled: spring.threads.virtual.enabled=true // Tomcat now uses virtual threads for request handling // Synchronous JDBC calls are fine — virtual threads handle blocking Impact on sizing: don't pool virtual threads — create one per task. Executors.newVirtualThreadPerTaskExecutor(). Thread pool tuning for I/O-bound services becomes unnecessary. What changes: - Thread pool sizing: no longer critical for I/O-bound services - Memory: dramatically lower per-connection overhead - CPU: carrier thread count = number of CPU cores (default); this is still the parallelism limit for CPU-bound work

What doesn't change: - CPU-bound work still limited by CPU cores - Avoid synchronized in hot paths — use ReentrantLock to prevent carrier thread pinning - Database connection pools still needed (limit connections to DB, not threads)

Symptom pattern: regular, periodic spikes → classic GC pause signature. Irregular spikes → more likely lock contention or I/O. Step 1: Correlate with GC. Enable GC logging if not already on: -Xlog:gc*:file=gc.log:time,uptime. Check if pause timestamps in the GC log align with latency spikes in APM (Datadog trace waterfall will show a "wall" of requests finishing at the same time after a pause). Step 2: Identify GC type and duration. Parse gc.log (GCEasy.io). What kind of pauses? Major/Full GC at 30-60s intervals? This means Old Gen is filling up every 30-60s → too much promotion from Young Gen. Diagnosis path A — Old Gen filling too fast (high allocation rate): JFR allocation profiling: which code paths allocate the most? Common culprits: string concatenation in hot loops, large collections created per request, excessive object creation in JSON serialization. Fix: optimize allocation-heavy paths. Diagnosis path B — Young Gen too small (objects promoted early): jstat -gcutil shows Young Gen % hitting 100% frequently. Increase Young Gen size via -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=60. More objects die in Young Gen. Diagnosis path C — Wrong GC algorithm: If using Parallel GC (default pre-Java 9): switch to G1 or ZGC. G1 with -XX:MaxGCPauseMillis=50 targets shorter, more frequent pauses instead of one long Full GC every 30-60s. ZGC for sub-millisecond pauses. Resolution: after tuning, rerun load test. Confirm P99 latency spikes disappear. Metrics to watch: GC pause duration, GC frequency, heap utilization post-GC.

Total JVM process memory is NOT just the heap: Total memory = heap + metaspace + code cache + thread stacks + native overhead Measure each component: - Heap: set with -XX:MaxRAMPercentage=70. Measure actual peak post-GC heap usage under load (this is the live set). Heap should be 2-3× the live set. - Metaspace: jcmd <pid> VM.metaspace. Default uncapped — cap with -XX:MaxMetaspaceSize=256m (adjust based on class count; Spring Boot with many features can need 200-400MB). - Code Cache: jcmd <pid> VM.flags | grep CodeCache. Default 240-256MB in Java 11+. Usually fine; check if code cache flush events appear in logs. - Thread stacks: thread_count × stack_size. For 200 threads with 512KB stacks = 100MB. ps -o nlwp <pid> shows thread count. Reduce with -Xss256k if needed. - Native overhead: typically 50-100MB for JVM internals, JNI, DirectByteBuffers. Sizing formula: Container memory limit = heap_max + metaspace_max + code_cache + thread_stacks + 100MB_overhead Example: 1536MB (heap) + 256MB (meta) + 256MB (code) + 100MB (threads) + 100MB = ~2.2GB → Set container limit: 2.5GB, heap MaxRAMPercentage: 65% In practice: 1. Run load test in staging with native memory tracking: -XX:NativeMemoryTracking=summary 2. jcmd <pid> VM.native_memory summary after warmup under load 3. Sum all components + 20% headroom = container memory limit 4. Set MaxRAMPercentage so heap fits within that budget

Preconditions check: - Java 21+ (Virtual Threads GA) - Spring Boot 3.2+ (native VT support) - Identify: does the service use synchronized blocks heavily? (pinning risk) - Check: what connection pools does it use? (HikariCP 5.1+ is VT-compatible; older versions pin) Migration steps: Step 1 — Enable VT for Tomcat only: yaml spring.threads.virtual.enabled=true This makes Tomcat use virtual threads for HTTP request handling. No code changes needed. Step 2 — Identify pinning risks: -Djdk.tracePinnedThreads=full Log statements like Thread[#42,ForkJoinPool...] pinned appear when a virtual thread hits a synchronized block. Common culprits: JDBC drivers (fix: upgrade or use R2DBC), old Spring Security synchronized blocks (fixed in modern versions). Step 3 — Fix critical pinning hot paths: Replace synchronized(lock) { ... } with ReentrantLock: java private final ReentrantLock lock = new ReentrantLock(); lock.lock(); try { ... } finally { lock.unlock(); } Step 4 — Canary in staging with load test: Run the same load test with VT enabled. Compare: P50/P99 latency, throughput, memory usage, thread count (now virtual threads, not platform threads). JFR: check virtual thread mount/unmount events, look for unexpected pinning. Step 5 — Tune connection pool: With VT, thousands of virtual threads can concurrently request DB connections. The DB connection pool (HikariCP) is still bounded (e.g., 10 connections). Virtual threads queue waiting for connections — this is correct behavior. Pool size doesn't need to match thread count; it should match DB concurrency capacity.

The problem at scale: 200 services, each with different heap configs, GC algorithms, thread pool sizes — most set by copying a template from 2019. Incidents caused by GC pauses, OOM kills, and thread exhaustion are common but not systematically tracked. Standardize baseline JVM config via a platform library: Publish a Java agent or configuration library that applies sensible defaults: - G1 GC with ZGC opt-in flag per service - MaxRAMPercentage=70 (container-aware heap sizing) - GC logging always on (-Xlog:gc*) - JFR continuous recording enabled - HeapDumpOnOutOfMemoryError with a defined path - MaxMetaspaceSize=512m (cap prevents runaway class loading) Services include one dependency / set one env var to get all of these. No JVM expertise required from service teams for baseline correctness. Observability: GC metrics in Datadog/Prometheus: All services export JVM metrics via Micrometer (Spring Boot default): jvm.gc.pause (pause duration), jvm.memory.used, jvm.threads.live, jvm.gc.memory.promoted. Platform team provides: standard JVM dashboard per service, alert template for OOM rate > 0/hour, GC pause P99 > 500ms, Old Gen utilization > 90%. Teams opt in to alerts; platform team audits that all production services have alerts. Continuous heap sizing validation: Post-deploy analysis job: after each production deployment, capture jstat -gcutil for 30 minutes. Report actual vs configured heap utilization. Flag services where Xmx is > 3× live set (over-provisioned) or Old Gen > 80% (under-provisioned). Monthly report: top 10 over- and under-provisioned services. Teams fix within one sprint. Performance regression testing in CI: Golden path: each service runs a 10-minute load test in CI. Collect: P99 latency, GC pause P99, heap utilization, allocation rate. Compare to previous run baseline. Alert if P99 latency increases > 10% or GC pause P99 increases > 20%. This catches GC regressions before production. Virtual Thread migration program: Identify I/O-bound services (high thread count, low CPU utilization). Prioritize for VT migration. Platform team provides a migration guide, pinning detection tooling, and a 2-week office hours engagement per team. Track: service count migrated, memory reduction per service (typical: 30-50% reduction in container memory).