高负载下的 502 网关错误 (nginx/php-fpm)
我在一个相当繁忙的互联网站点工作，该站点经常会出现非常大的流量高峰.在这些高峰期间，每秒请求数百页，这会产生随机的 502 网关错误.
I work for a rather busy internet site that is often gets very large spikes of traffic. During these spikes hundreds of pages per second are requested and this produces random 502 gateway errors.
现在我们在具有 4 个 SAS 15k 驱动器 (raid10) 的机器上运行 Nginx (1.0.10) 和 PHP-FPM，该驱动器具有 16 核 CPU 和 24GB DDR3 内存.我们还使用最新的 Xcache 版本.数据库位于另一台机器上，但是这台机器的负载非常低，没有问题.
Now we run Nginx (1.0.10) and PHP-FPM on a machine with 4x SAS 15k drives (raid10) with a 16 core CPU and 24GB of DDR3 ram. Also we make use of the latest Xcache version. The DB is located on another machine, but this machine's load is very low, and has no issues.
在正常负载下，一切运行完美，系统负载低于 1，PHP-FPM 状态报告从未真正同时显示超过 10 个活动进程.始终有大约 10GB 的内存可用.在正常负载下，机器每秒处理大约 100 次浏览量.
Under normal load everything runs perfect, system load is below 1, and PHP-FPM status report never really shows more than 10 active processes at one time. There is always about 10GB of ram still available. Under normal load the machine handles about 100 pageviews per second.
当出现巨大的流量高峰时，问题就出现了，并且每秒数百次页面浏览量是从机器请求的.我注意到 FPM 的状态报告随后会显示多达 50 个活动进程，但这仍然远低于我们配置的 300 个最大连接数.在这些高峰期间，Nginx 状态报告多达 5000 个活动连接，而不是正常的平均 1000 个.
The problem arises when huge spikes of traffic arrive, and hundreds of page-views per second are requested from the machine. I notice that FPM's status report then shows up to 50 active processes, but that is still way below the 300 max connections that we have configured. During these spikes Nginx status reports up to 5000 active connections instead of the normal average of 1000.
操作系统信息:CentOS 5.7 版(最终版)
OS Info: CentOS release 5.7 (Final)
CPU:Intel(R) Xeon(R) CPU E5620 @ 2.40GH(16 核)
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GH (16 cores)
php-fpm.conf
daemonize = yes
listen = /tmp/fpm.sock
pm = static
pm.max_children = 300
pm.max_requests = 1000
我没有设置 rlimit_files，因为据我所知，如果你不这样做，它应该使用系统默认值.
I have not setup rlimit_files, because as far as I know it should use the system default if you don't.
fastcgi_params(仅向标准文件添加值)
fastcgi_params (only added values to standard file)
fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;
fastcgi_pass unix:/tmp/fpm.sock;
nginx.conf
worker_processes 8;
worker_connections 16384;
sendfile on;
tcp_nopush on;
keepalive_timeout 4;
Nginx 通过 Unix Socket 连接到 FPM.
Nginx connects to FPM via Unix Socket.
sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 1
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.eth0.rp_filter=1
net.ipv4.conf.lo.rp_filter=1
net.ipv4.ip_conntrack_max = 100000
limits.conf
* soft nofile 65536
* hard nofile 65536
这些是以下命令的结果:
These are the results for the following commands:
ulimit -n
65536
ulimit -Sn
65536
ulimit -Hn
65536
cat /proc/sys/fs/file-max
2390143
问题:如果 PHP-FPM 没有用完连接，负载仍然很低，并且有足够的 RAM 可用，那么在高流量期间可能会导致这些随机 502 网关错误的瓶颈是什么?
Question: If PHP-FPM is not running out of connections, the load is still low, and there is plenty of RAM available, what bottleneck could be causing these random 502 gateway errors during high traffic?
注意:默认情况下，这台机器的 ulimit 是 1024，因为我将它更改为 65536，所以我没有完全重启机器，因为它是生产机器，这意味着停机时间太长.
推荐答案
这应该解决它...
你有:fastcgi_buffers 4 256k;
将其更改为:fastcgi_buffers 256 16k;//总共 4096k
Change it to: fastcgi_buffers 256 16k; // 4096k total
同时设置 fastcgi_max_temp_file_size 0，这将在回复开始超过您的 fastcgi 缓冲区时禁用缓冲到磁盘.
Also set fastcgi_max_temp_file_size 0, that will disable buffering to disk if replies start to exceeed your fastcgi buffers.
相关文章