Java in Docker and Kubernetes: avoid OOMKilled

The pod restarts. kubectl get pod shows OOMKilled, and the exit code is 137. But the logs show no OutOfMemoryError, and the heap was far from full. The JVM did not run out of memory: the container did. The Linux kernel killed the process because it went over its limit.

This post shows what the JVM sees in a container, what adds to the heap, how to reproduce the kill, and how to set the memory so that everything fits. All the examples are in the containers/ folder of the public java-gc-lab repository. They use the official eclipse-temurin images: no image to build.

The Docker commands of this post

If you do not use Docker much, here are the options used below.

Option	What it does
`docker run <image> <command>`	Starts a container from the image and runs the command in it.
`--rm`	Deletes the container when it stops.
`--name over-limit`	Gives the container a name, for the next commands.
`--memory=512m`	Memory limit of the container. This is the limit the JVM reads.
`--memory-swap=512m`	Limit for memory + swap. Same value as `--memory`: no swap.
`--cpus=2`	CPU limit: the same as 2 CPUs of time.
`-v "$PWD/containers":/app`	Mounts the local folder `containers/` as `/app` in the container.
`-d`	Runs the container in the background.
`docker inspect <name>`	Shows the state of a stopped container: exit code, `OOMKilled`.
`docker exec <name> <command>`	Runs a command in a running container.
`docker logs <name>`	Shows what the container printed.

Watch out for swap: with --memory alone, Docker also gives the same amount of swap. java -XshowSettings:system shows it: “Memory Limit: 1.00G”, “Memory & Swap Limit: 2.00G”. The container then uses swap instead of being killed at the limit. To get what happens in Kubernetes, I set --memory-swap to the same value.

What the JVM sees in a container

On Linux, the JVM reads the limits of the container (the cgroups): the memory and the number of CPUs. This is the option -XX:+UseContainerSupport, on by default. The JVM uses these limits to size the heap, choose the GC and set the number of threads.

ShowLimits.java prints what the JVM chose:

docker run --rm --memory=1g --cpus=2 -v "$PWD/containers":/app eclipse-temurin:25 java /app/ShowLimits.java

CPUs: 2
Max heap: 256 MiB
GC: Copy, MarkSweepCompact

“Copy, MarkSweepCompact” is the Serial GC. With other limits, the same results on Java 17, 21 and 25:

`--memory`	`--cpus`	Max heap	GC
`256m`	`1`	126 MiB	Serial
`1g`	`2`	256 MiB	Serial
`1791m`	`2`	448 MiB	Serial
`1792m`	`2`	448 MiB	G1
`4g`	`2`	1024 MiB	G1

The JVM gives these values itself, with no program:

docker run --rm --memory=1g --cpus=2 eclipse-temurin:25 java -XX:+PrintFlagsFinal -version | grep -E " MaxHeapSize | UseSerialGC | UseG1GC "
docker run --rm --memory=1g --cpus=2 eclipse-temurin:25 java -XshowSettings:system -version

The first one prints the computed options (MaxHeapSize = 268435456, UseSerialGC = true). The second one shows the limits read from the container: “Provider: cgroupv2”, “Effective CPU Count: 2”, “Memory Limit: 1.00G”.

The default heap: 25 %, or 50 % for a small heap

Without -Xmx, the maximum heap is 25 % of the memory the JVM sees: this is -XX:MaxRAMPercentage, 25 by default. In a 1 GB container, that is 256 MB. The other 75 % are not all free: the rest of the JVM needs them, as we will see below.

For a small heap, another option takes over: -XX:MinRAMPercentage, 50 by default. The documentation talks about a heap of “approximately 125 MB”. This is what the table shows: with 256 MB, the heap is 126 MB, not 64.

You set both options at startup:

java -XX:MaxRAMPercentage=50 -jar app.jar

A percentage follows the container limit: if the limit goes from 1 to 2 GB, the heap follows. A fixed -Xmx does not change (see the JVM options that matter).

Serial or G1: a question of CPUs and memory

The table has a surprise: with 2 CPUs and 1 GB, the JVM picks Serial, not G1. Oracle’s GC guide gives the rule: G1 on a “server-class” machine, Serial otherwise. A machine is “server-class” if the JVM detects “two or more processors and physical memory larger than or equal to 1792 MB”. The tests confirm it: Serial at 1791 MB, G1 at 1792 MB.

Serial stops the application during each GC, with one thread. For a service with little memory, this is not always a problem. But you must know it, and choose it. To force G1: -XX:+UseG1GC.

The number of CPUs also changes the number of GC threads. With 2 CPUs, G1 has 2 parallel threads and 1 concurrent thread. The option -XX:ActiveProcessorCount=<n> replaces the number of CPUs that the JVM uses for these sizes. The documentation says it can be useful “when running multiple Java processes in docker containers”.

What about the CPU request of Kubernetes? In a pod with requests.cpu: 1 and no CPU limit, the JVM sees all the CPUs of the machine: 12 on my machine. The same with the Docker option --cpu-shares (the cgroup parameter “cpu.shares”). A limit does reduce the number of CPUs the JVM sees: with limits.cpu: "2" (or --cpus=2), it sees 2.

Before Java 19: the JVM also used “cpu.shares” to count the CPUs, and could see fewer than available. Java 19 ignores it (JDK-8281181). In my tests, Temurin 17.0.20 also ignores it.

What adds to the heap

The container limit covers the whole process, not only the heap:

the Metaspace: the description of the loaded classes;
the thread stacks: by default 1024 KB per thread on Linux/x64, and 2048 KB on Linux/Aarch64 (for example Docker on an Apple Silicon Mac);
the code cache: the code compiled by the JIT;
the direct memory: the ByteBuffer.allocateDirect buffers. Without -XX:MaxDirectMemorySize, the JVM chooses the limit itself;
the GC data and other internal areas.

A heap at 75 % of the limit leaves 25 % for all of this. If the rest needs more, the container is killed, even with a half-empty heap.

Reproduce the kill

OverLimit.java keeps 250 MB in the heap, then takes direct memory by blocks of 10 MB, with no end. A 512 MB container with no swap, max heap at 75 % (384 MB):

docker run --name over-limit --memory=512m --memory-swap=512m -v "$PWD/containers":/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 /app/OverLimit.java

Heap: 250 MB used
Direct memory: 10 MB
...
Direct memory: 170 MB

The output just stops. No OutOfMemoryError, no stack trace: the process was killed. docker inspect confirms it:

docker inspect over-limit --format "exit={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}"

exit=137 OOMKilled=true

137 is 128 + 9: the process got signal 9 (SIGKILL), which cannot be caught. The JVM could not write anything, and -XX:+HeapDumpOnOutOfMemoryError does not help here: there was no OutOfMemoryError. Depending on the version, the kill comes between 170 and 190 MB of direct memory.

See where the memory goes with NMT

Native Memory Tracking (NMT) counts the memory of the JVM by area. I run it again with NMT, and the argument 100 stops the direct memory at 100 MB; the program then waits 30 seconds:

docker run -d --name over-limit --memory=512m --memory-swap=512m -v "$PWD/containers":/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 -XX:NativeMemoryTracking=summary /app/OverLimit.java 100
docker exec over-limit jcmd 1 VM.native_memory summary scale=MB

Total: reserved=1875MB, committed=533MB
-                 Java Heap (reserved=384MB, committed=384MB)
-                     Class (reserved=1024MB, committed=2MB)
-                    Thread (reserved=26MB, committed=1MB)
-                      Code (reserved=245MB, committed=9MB)
-                     Other (reserved=100MB, committed=100MB)
-                 Metaspace (reserved=64MB, committed=9MB)

(I removed some lines.) 1 is the process id of Java in the container. The heap takes 384 MB, and the 100 MB of direct memory are in “Other”. This small program has few classes and threads; a real application has more Metaspace, more threads and more compiled code. docker stats --no-stream over-limit shows the memory the container really uses: about 430 MiB in my runs, out of 512.

Better than a kill: a Java error

With a limit on direct memory, the JVM stops before the container, with a clear message:

docker run --rm --memory=512m --memory-swap=512m -v "$PWD/containers":/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 -XX:MaxDirectMemorySize=100m /app/OverLimit.java

Exception in thread "main" java.lang.OutOfMemoryError: Cannot reserve 10485760 bytes of direct buffer memory (allocated: 104857600, limit: 104857600)

An error with a stack trace tells what is full. A kill tells nothing. The other OutOfMemoryError messages are in the post about OutOfMemoryError.

Choose `MaxRAMPercentage`

There is no good percentage for all applications. The rule is simple: the maximum heap plus all the rest must fit in the limit. You measure the rest with NMT, on the application under load.

In the example, the program needs 250 MB of heap and 200 MB of direct memory. With 768 MB and 50 %:

docker run --name over-limit --memory=768m --memory-swap=768m -v "$PWD/containers":/app eclipse-temurin:25 java -XX:MaxRAMPercentage=50 /app/OverLimit.java 200

Done: 250 MB heap, 200 MB direct, waiting 30 s

docker inspect gives exit=0 OOMKilled=false. The heap can grow to 384 MB (50 % of 768), and 384 MB stay for the direct memory and the rest of the JVM. The more memory an application uses outside the heap (threads, direct memory, many classes), the lower the percentage must be.

In Kubernetes

In a pod, each container has requests and limits. The Kubernetes documentation says that the request is used to choose the node for the pod; the limit is enforced by the kernel. For memory, a container that goes over its limit can be killed (OOM kill). For CPU, it is slowed down.

So the memory limit is the one the JVM reads. Without limits.memory, the JVM sees the memory of the node: in a pod with only requests.memory: 1Gi, the max heap is 1984 MiB on my machine (25 % of the 7.7 GB of the Docker VM), much more than the request.

For the example, the Java file goes into the pod with a ConfigMap: no image to build. The manifest pod-before.yaml is the killed case:

      command: ["java", "-XX:MaxRAMPercentage=75", "/app/OverLimit.java"]
      resources:
        requests:
          memory: "512Mi"
          cpu: "1"
        limits:
          memory: "512Mi"

The kubectl commands used:

Command	What it does
`kubectl create configmap over-limit-code --from-file=containers/OverLimit.java`	Stores the Java file in the cluster; the pod mounts it in `/app`.
`kubectl apply -f containers/pod-before.yaml`	Creates the pod described in the file.
`kubectl get pod over-limit`	Shows the status of the pod.
`kubectl describe pod over-limit`	Shows the details: state, reason, exit code, limits.
`kubectl logs over-limit`	Shows what the program printed.
`kubectl delete pod over-limit`	Deletes the pod.

kubectl create configmap over-limit-code --from-file=containers/OverLimit.java
kubectl apply -f containers/pod-before.yaml
kubectl get pod over-limit

NAME         READY   STATUS      RESTARTS   AGE
over-limit   0/1     OOMKilled   0          7s

kubectl describe pod over-limit

    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137

pod-after.yaml uses the size that fits: limits.memory: 768Mi, -XX:MaxRAMPercentage=50. After kubectl delete pod over-limit and kubectl apply -f containers/pod-after.yaml, the pod ends as Completed, exit code 0.

What has changed

What you still read	What is true today	Since
`-XX:+UseCGroupMemoryLimitForHeap` so that Java sees the limit	Nothing to add: the JVM reads the limits itself (`UseContainerSupport`); the old option is refused	JDK 10, and Java 8u191; option removed in Java 11
`-XX:MaxRAMFraction=4`	`-XX:MaxRAMPercentage=25` (and `InitialRAMPercentage`, `MinRAMPercentage`)	JDK 10
“cpu.shares” changes the number of CPUs Java sees	“cpu.shares” is ignored	Java 19 (Temurin 17.0.20 too, in my tests)

Tested with Docker Desktop 29.8 (Linux VM, aarch64, cgroup v2), its Kubernetes (v1.36), and the eclipse-temurin images 17.0.20, 21.0.12 and 25.0.4.

Sources: the documentation of the java command (JDK 25), Oracle’s GC guide, part “Ergonomics”, the release notes of JDK 10, Java 8u191 and JDK 19, and the Kubernetes documentation (Resource Management for Pods and Containers, Assign Memory Resources to Containers and Pods).