在 Docker 容器中运行的 JVM 的驻留集大小 (RSS) 和 Java 总提交内存 (NMT) 之间的差异

2022-01-16 00:00:00 docker memory jvm linux java

场景:

我有一个在 docker 容器中运行的 JVM.我使用两个工具做了一些内存分析:1) top 2) Java Native Memory Tracking.这些数字看起来令人困惑，我试图找出导致差异的原因.

问题:

Java 进程的 RSS 报告为 1272MB，Java 总内存报告为 790.55 MB.我如何解释剩余的内存 1272 - 790.55 = 481.44 MB 去了哪里?

为什么即使在查看

Java NMT

Docker 内存统计数据

图表

我的 docker 容器运行时间超过 48 小时.现在，当我看到一个包含以下内容的图表时:

  1. 分配给 docker 容器的总内存 = 2 GB
  2. Java 最大堆 = 1 GB
  3. 总提交 (JVM) = 始终小于 800 MB
  4. 使用的堆 (JVM) = 始终小于 200 MB
  5. 未使用堆 (JVM) = 始终小于 100 MB.
  6. RSS = 大约 1.1 GB.

那么，1.1 GB (RSS) 到 800 MB(Java 提交的总内存)之间的内存是什么?

解决方案

你在Mikhail Krestjaninoff 分析 Docker 容器中的 java 内存使用情况:

(需要明确的是，三年后的 2019 年 5 月，情况确实有所改善8u212)

<块引用>

Resident Set Size 是进程当前分配和使用的物理内存量(没有换出页面).它包括代码、数据和共享库(在每个使用它们的进程中都计算在内)

为什么 docker stats info 与 ps 数据不同?

第一个问题的答案很简单 - Docker 有一个 bug(或一个特性 - 取决于你的心情):它包括文件缓存到总内存使用信息中.所以，我们可以避免这个指标并使用关于 RSS 的 ps 信息.

好吧，但是为什么 RSS 比 Xmx 高?

理论上，如果是 java 应用程序

RSS = 堆大小 + MetaSpace + OffHeap 大小

<块引用>

其中 OffHeap 由线程堆栈、直接缓冲区、映射文件(库和 jar)和 JVM 代码组成

自 JDK 1.8.40 我们有 本机内存跟踪器！

如您所见，我已经在 J​​VM 中添加了 -XX:NativeMemoryTracking=summary 属性，所以我们可以从命令行调用它:

docker exec my-app jcmd 1 VM.native_memory 摘要

(这就是OP所做的)

<块引用>

不要担心未知"部分 - 似乎 NMT 是一个不成熟的工具，无法处理 CMS GC(当您使用另一个 GC 时，此部分会消失).

请记住，NMT 显示已提交"内存，而不是驻留"(您可以通过 ps 命令获得).换句话说，一个内存页可以在不考虑驻留的情况下提交(直到它被直接访问).

这意味着非堆区域(堆始终预初始化)的 NMT 结果可能大于 RSS 值.

(这就是为什么 JVM 报告比 linux 进程驻留集大小更多的提交内存?"来的地方中)

<块引用>

因此，尽管我们将 jvm 堆限制设置为 256m，但我们的应用程序消耗了 367M.其他"的 164M 主要用于存储类元数据、编译代码、线程和 GC 数据.

前三个点对于应用程序来说通常是常量，因此唯一会随着堆大小增加的是 GC 数据.
这种依赖性是线性的，但k"系数 (y = kx + b) 远小于 1.

<小时>

更一般地说，这似乎是 issue 15020 报告了类似的问题从 docker 1.7 开始

<块引用>

我正在运行一个简单的 Scala (JVM) 应用程序，它将大量数据加载到内存中和从内存中加载出来.
我将 JVM 设置为 8G 堆 (-Xmx8G).我有一台 132G 内存的机器，它不能处理超过 7-8 个容器，因为它们的增长远远超过了我对 JVM 施加的 8G 限制.

(docker stat 是 之前被报告为误导，因为它显然将文件缓存包含在总内存使用信息中)

<块引用>

docker stat 显示每个容器它本身使用的内存比 JVM 应该使用的要多得多.例如:

CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O戴夫-1 3.55% 10.61 GB/135.3 GB 7.85% 7.132 MB/959.9 MB性能-1 3.63% 16.51 GB/135.3 GB 12.21% 30.71 MB/5.115 GB

<块引用>

似乎 JVM 正在向操作系统请求内存，该内存是在容器内分配的，并且 JVM 在其 GC 运行时正在释放内存，但容器不会将内存释放回主操作系统.所以...内存泄漏.

Scenario:

I have a JVM running in a docker container. I did some memory analysis using two tools: 1) top 2) Java Native Memory Tracking. The numbers look confusing and I am trying to find whats causing the differences.

Question:

The RSS is reported as 1272MB for the Java process and the Total Java Memory is reported as 790.55 MB. How can I explain where did the rest of the memory 1272 - 790.55 = 481.44 MB go?

Why I want to keep this issue open even after looking at this question on SO:

I did see the answer and the explanation makes sense. However, after getting output from Java NMT and pmap -x , I am still not able to concretely map which java memory addresses are actually resident and physically mapped. I need some concrete explanation (with detailed steps) to find whats causing this difference between RSS and Java Total committed memory.

Top Output

Java NMT

Docker memory stats

Graphs

I have a docker container running for most than 48 hours. Now, when I see a graph which contains:

  1. Total memory given to the docker container = 2 GB
  2. Java Max Heap = 1 GB
  3. Total committed (JVM) = always less than 800 MB
  4. Heap Used (JVM) = always less than 200 MB
  5. Non Heap Used (JVM) = always less than 100 MB.
  6. RSS = around 1.1 GB.

So, whats eating the memory between 1.1 GB (RSS) and 800 MB (Java Total committed memory)?

解决方案

You have some clue in " Analyzing java memory usage in a Docker container" from Mikhail Krestjaninoff:

(And to be clear, in May 2019, three years later, the situation does improves with openJDK 8u212 )

Resident Set Size is the amount of physical memory currently allocated and used by a process (without swapped out pages). It includes the code, data and shared libraries (which are counted in every process which uses them)

Why does docker stats info differ from the ps data?

Answer for the first question is very simple - Docker has a bug (or a feature - depends on your mood): it includes file caches into the total memory usage info. So, we can just avoid this metric and use ps info about RSS.

Well, ok - but why is RSS higher than Xmx?

Theoretically, in case of a java application

RSS = Heap size + MetaSpace + OffHeap size

where OffHeap consists of thread stacks, direct buffers, mapped files (libraries and jars) and JVM code itse

Since JDK 1.8.40 we have Native Memory Tracker!

As you can see, I’ve already added -XX:NativeMemoryTracking=summary property to the JVM, so we can just invoke it from the command line:

docker exec my-app jcmd 1 VM.native_memory summary

(This is what the OP did)

Don’t worry about the "Unknown" section - seems that NMT is an immature tool and can’t deal with CMS GC (this section disappears when you use an another GC).

Keep in mind, that NMT displays "committed" memory, not "resident" (which you get through the ps command). In other words, a memory page can be committed without considering as a resident (until it directly accessed).

That means that NMT results for non-heap areas (heap is always preinitialized) might be bigger than RSS values.

(that is where "Why does a JVM report more committed memory than the linux process resident set size?" comes in)

As a result, despite the fact that we set the jvm heap limit to 256m, our application consumes 367M. The "other" 164M are mostly used for storing class metadata, compiled code, threads and GC data.

First three points are often constants for an application, so the only thing which increases with the heap size is GC data.
This dependency is linear, but the "k" coefficient (y = kx + b) is much less then 1.

More generally, this seems to be followed by issue 15020 which reports a similar issue since docker 1.7

I'm running a simple Scala (JVM) application which loads a lot of data into and out of memory.
I set the JVM to 8G heap (-Xmx8G). I have a machine with 132G memory, and it can't handle more than 7-8 containers because they grow well past the 8G limit I imposed on the JVM.

(docker stat was reported as misleading before, as it apparently includes file caches into the total memory usage info)

docker stat shows that each container itself is using much more memory than the JVM is supposed to be using. For instance:

CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
dave-1 3.55% 10.61 GB/135.3 GB 7.85% 7.132 MB/959.9 MB
perf-1 3.63% 16.51 GB/135.3 GB 12.21% 30.71 MB/5.115 GB

It almost seems that the JVM is asking the OS for memory, which is allocated within the container, and the JVM is freeing memory as its GC runs, but the container doesn't release the memory back to the main OS. So... memory leak.

相关文章