创建 < 时的 Java `OutOfMemoryError`100 个线程

2022-01-16 00:00:00 threadpool out-of-memory jvm java openjdk

由于这个错误，我已经阅读和测试了一天多的时间.

我在一个名为 Listener 的类中有一些 Java 代码，看起来像这样

ExecutorService 执行器 = Executors.newFixedThreadPool(NTHREADS);布尔监听 = true;整数计数 = 0；而(听){可运行的工人；尝试 {worker = new ServerThread(serverSocket.accept());//这是第 254 行executor.execute(worker);计数++；logger.info("{} 线程启动", count);} 捕捉(异常 e1){//...}}

我一直在调整 JVM 设置 -Xmx(从 1 到 15G)和 -Xss(从 104k 到 512M).服务器有 24 GB 的 RAM，但还必须运行支持该程序的数据库.

创建 2-20 个线程后(程序中的其他地方也存在几十个)，我得到了错误

线程Thread-0"java.lang.OutOfMemoryError 中的异常:无法创建新的本机线程在 java.lang.Thread.start0(本机方法)在 java.lang.Thread.start(Thread.java:657)在 java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)在 java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325)在 xxx.Listener.run(Listener.java:254)

$java -version 产生:

java 版本1.6.0_24"OpenJDK 运行时环境 (IcedTea6 1.11.1) (fedora-65.1.11.1.fc16-x86_64)OpenJDK 64 位服务器 VM(内部版本 20.0-b12，混合模式)

发生这种情况时，系统上总是有大量可用内存，其他程序继续正常执行.是什么导致 Java 认为它没有更多内存可用于新线程?

更新:也许这比我想象的要大 - 当我使用 ^C:

时，我设法得到了这个错误(只有一次)

OpenJDK 64 位服务器 VM 警告:异常 java.lang.OutOfMemoryError 发生调度信号 SIGINT 到处理程序 - 可能需要强制终止 VM

当我试图杀死客户端时发生了同样的情况(也是用Java编写并在同一台服务器上运行，它是一个读取文件并通过套接字将其发送到服务器的单线程)，所以肯定有超出 JVM 的限制导致一个干扰另一个，但我无法想象如果我仍然有空闲内存并且根本不使用交换会怎样?服务器 -Xmx1G -Xss104k 客户端 -Xmx10M

更新 2:放弃 perl Forks::Super 库并从 bash 运行客户端让我在服务器因 OOME 崩溃之前获得多达 34 个线程，因此运行多个客户端肯定会对服务器产生影响，但在同时我应该仍然能够一次运行超过 34 个(如果计算客户端，则为 68 个)java 线程.哪些系统资源阻止了更多线程的创建(即我应该在哪里寻找猪)?当所有东西(客户端、服务器、GC...)同时耗尽内存时，top 会这样说我的 CPU 和内存使用情况:

Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0% 0.0%si, 0.0%st内存:总计 24681040k，已使用 1029420k，空闲 23651620k，30648k 缓冲区交换:总共 26836988k，使用 0k，免费 26836988k，缓存 453620k

UPDATE3: 下面的 hs_error 日志是否表明我的 java 不是 64 位?

# 内存不足，Java 运行时环境无法继续.# 无法创建 GC 线程.系统资源不足.# 可能的原因:# 系统的物理 RAM 或交换空间不足# 在 32 位模式下，达到了进程大小限制# 可能的解决方案:# 减少系统内存负载# 增加物理内存或交换空间# 检查交换后备存储是否已满# 在 64 位操作系统上使用 64 位 Java# 减小 Java 堆大小 (-Xmx/-Xms)# 减少Java线程数# 减少 Java 线程堆栈大小 (-Xss)# 使用 -XX:ReservedCodeCacheSize= 设置更大的代码缓存# 此输出文件可能被截断或不完整.## JRE 版本:6.0_24-b24# Java VM:OpenJDK 64-Bit Server VM(20.0-b12混合模式linux-amd64压缩oops)# 衍生产品:IcedTea6 1.11.1# 分发:Fedora release 16 (Verne)，包 fedora-65.1.11.1.fc16-x86_64

解决方案

你可以通过max user processes来限制，知道你的限制使用:

ulimit -u

更改限制:

在 /etc/security/limits.conf 中设置:

用户软 nproc [your_val]用户硬 nproc [your_val]

如果还不够，您可能需要添加一些其他配置，请参阅此链接..p>

注意:OP 在 fedora 和 centos 中发现了这个错误报告，它解释了编辑/etc/security/limits.conf的限制.

I've been reading and testing and banging my head on the wall for over a day because of this error.

I have some Java code in a class called Listener that looks like this

ExecutorService executor = Executors.newFixedThreadPool(NTHREADS);
boolean listening = true;
int count = 0;
while (listening) {
    Runnable worker;
    try {
        worker = new ServerThread(serverSocket.accept()); // this is line 254
        executor.execute(worker);
        count++;
        logger.info("{} threads started", count);
    } catch (Exception e1){
        //...
    }
}

I have been tweaking the JVM settings -Xmx (anywhere from 1 to 15G) and -Xss (anywhere from 104k to 512M). The server has 24 GB of RAM, but must also run the database that supports the program.

After 2-20 threads are created (a few dozen exist elsewhere in the program as well), I get the error

Exception in thread "Thread-0" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325)
at xxx.Listener.run(Listener.java:254)

$java -version yields:

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.1) (fedora-65.1.11.1.fc16-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

There is always a large amount of free memory on the system when this happens, and other programs continue to execute fine. What is causing Java to think it has no more memory for new threads?

UPDATE: Perhaps this is bigger than I thought- I managed to get this error (only one time) when I used ^C:

OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGINT to handler- the VM may need to be forcibly terminated

and the same happened when I tried to kill the client (also written in Java and running on the same server, it is a single thread that reads a file and sends it to the server over the socket), so there is definitely a limit beyond the JVM causing one to interfere with the other, but I can't imagine what if I still have free memory and am not using swap at all? Server -Xmx1G -Xss104k Client -Xmx10M

UPDATE2: Abandoning the perl Forks::Super library and running the clients from bash let me get up to 34 threads before the server crashed with OOME, so running multiple clients definitely had an impact on the server, but at the same time I should still be able to run more than 34 (68 if one counts the clients) java threads at a time. Which system resources are blocking the creation of more threads (i.e. where should I look to find the hog)? When everything (clients, server, GC...) runs out of memory at the same time, top says this about my CPU and memory usage:

Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  24681040k total,  1029420k used, 23651620k free,    30648k buffers
Swap: 26836988k total,        0k used, 26836988k free,   453620k cached

UPDATE3: Does the hs_error log below indicate that my java is not 64 bit?

# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# JRE version: 6.0_24-b24
# Java VM: OpenJDK 64-Bit Server VM (20.0-b12 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea6 1.11.1
# Distribution: Fedora release 16 (Verne), package fedora-65.1.11.1.fc16-x86_64

解决方案

You can be limit by max user processes, to know your limit use :

ulimit -u

To change the limit :

In /etc/security/limits.conf set :

user soft nproc [your_val] 
user hard nproc [your_val]

You may have to add some other config if it's not enough see this link.

Note : The OP found this bug report in fedora and centos which explains the limitations of editing /etc/security/limits.conf.

相关文章

创建 &lt; 时的 Java `OutOfMemoryError`100 个线程

创建 < 时的 Java `OutOfMemoryError`100 个线程