如何避免【Errno 12】使用子进程模块导致的Cannot allocate memory错误

2022-01-18 00:00:00 python subprocess networking memory paramiko

问题描述

完整的工作测试用例

当然，根据您在本地和远程计算机上的内存，您的数组大小会有所不同.

Of course depending on your memory on the local and remote machines your array sizes will be different.

z1 = numpy.random.rand(300000000,2);
for i in range(1000):
  print('*******************************************
'); 
  direct_output = subprocess.check_output('ssh blah@blah "ls /"', shell=True);
  direct_output = 'a'*1200000; 
  a2 = direct_output*10;
  print(len(direct_output));

当前用例

如果它有助于我的用例如下:

In case it helps my use case is as follows:

我发出数据库查询然后将结果表存储在远程机器上.然后我想通过网络传输它们并进行分析.到目前为止，我一直在 python 中执行以下操作:

I issue db queries then store the resulting tables on the remote machine. I then want to transfer them across a network and do analysis. Thus far I have been doing something like the following in python:

#run a bunch of queries before hand with the results in remote files

....
counter = 0
mergedDataFrame = None
while NotDone:
  output = subprocess.check_output('ssh blah@blah cat /data/file%08d'%(counter))
  data = pandas.read_csv(...)
  #do lots of analysis, append, merge, numpy stuff etc...
  mergedDataFrame = pandas.merge(...)
  counter += 1

有时我在 check_output 命令中收到以下错误:[Errno 12] 无法分配内存

At some point I receive the following error at the check_output command: [Errno 12] Cannot allocate memory

背景

感谢以下问题，我想我知道出了什么问题.发布了许多解决方案，我正在尝试确定哪些解决方案可以避免 [Errno 12] Cannot allocate memory 与使用 fork/clone 的子进程实现相关的错误.

Thanks to the below questions I think I have an idea of what is wrong. There are a number of solutions posted, and I am trying to determine which of the solutions will avoid the [Errno 12] Cannot allocate memory error associated with the subprocess implementation using fork/clone.

Python subprocess.Popen "OSError: [Errno 12] 无法分配内存" 这给出了基本诊断并提出了一些解决方法，例如生成单独的脚本等......

Python subprocess.Popen "OSError: [Errno 12] Cannot allocate memory" This gives the underlying diagnosis and suggests some workaround like spawning separate script etc...

了解 Python fork 和内存分配错误 建议使用 rfoo规避 fork/clone 的子进程限制以及生成子进程和复制内存等...这似乎暗示了客户端-服务器模型

Understanding Python fork and memory allocation errors Suggests using rfoo to circumvent the subprocess limitation of fork/clone and spawning child process and copy memory etc... This seems to imply a client-server model

什么是最简单的 SSH 方式使用 Python? ，但由于内存限制和 fork/clone 实现，我有无法使用子进程的额外限制?解决方案建议使用 paramiko 或基于它构建的东西，其他人建议使用 subprocess (我发现这在我的情况下不起作用).

What is the simplest way to SSH using Python? , but I have the additional constraints that I cannot use subprocess due to memory limitations and fork/clone implementation? The solutions suggests using paramiko or something built on top of it, others suggest subprocess (which I have found will not work in my case).

还有其他类似的问题，但答案经常谈到文件描述符是罪魁祸首(在这种情况下他们不是)，向系统添加更多 RAM(我不能这样做)，升级到 x64(我已经在 x64).一些暗示 ENOMEM 的问题.一些答案提到尝试确定 subprocess.Popen(在我的情况下为 check_output)是否没有正确清理进程，但看起来 S. Lott 和其他人同意子进程代码本身正在正确清理.

There were other similar questions but the answers often talked about file descriptors being the culprit (in this case they are not), adding more RAM to the system ( I cannot do this), upgrading to x64 ( I already am on x64). Some hint at the problem of ENOMEM. A few answers mention trying to determine if the subprocess.Popen (in my case check_output) is not properly cleaning the processes, but it looks like S. Lott and others agree that the subprocess code itself is properly cleaning up.

  • 使用 subprocess.Popen 的 Python 内存分配错误
  • Python IOError 无法分配内存虽然有很多
  • 无法在 Popen 命令上分配内存
  • Python subprocess.Popen 错误与 OSError: [Errno 12] 一段时间后无法分配内存

我在github上搜索了源代码https://github.com/paramiko/paramiko/search?q=Popen&type=Code 并且它似乎在 proxy.py 文件中使用了子进程.

I have searched through the source code on github https://github.com/paramiko/paramiko/search?q=Popen&type=Code and it appears to use subprocess in the proxy.py file.

实际问题

这是否意味着最终 paramiko 正在使用上述 Popen 解决方案，当 python 内存占用增加并且由于克隆/分叉实现而重复 Popen 调用时会出现问题?

Does this mean that ultimately paramiko is using the Popen solution described above that will have problems when the python memory footprint grows and repeated Popen calls are made due to the clone/fork implementation?

如果 paramiko 不起作用，是否有另一种方法可以通过仅客户端解决方案来做我正在寻找的事情?还是需要客户端/服务器/套接字解决方案?如果是这样，rfoo、tornado 或 zeromq、http 传输中的任何一个都可以在这里工作吗?

If paramiko will not work is there another way to do what I am looking for with a client side only solution? Or will a client/server/socket solution be needed? If so will any of rfoo, tornado, or zeromq, http transfers work here?

注意事项我正在运行 64 位 linux 8GB 主内存.我不想追求购买更多 RAM 的选择.

Notes I am running 64bit linux 8GB main memory. I do not want to pursue the options of buying more RAM.

解决方案

如果内存不足，您可能需要增加交换内存.或者您可能根本没有启用交换.在 Ubuntu(它也应该适用于其他发行版)中，您可以通过以下方式检查您的交换:

If you are running out of memory, you may want to increase your swap memory. Or you might have no swap enabled at all. In Ubuntu (it should work for other distributions as well) you can check your swap by:

$sudo swapon -s

如果它为空，则表示您没有启用任何交换.添加 1GB 交换:

if it is empty it means you don't have any swap enabled. To add a 1GB swap:

$sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024k
$sudo mkswap /swapfile
$sudo swapon /swapfile

将以下行添加到 fstab 以使交换永久化.

Add the following line to the fstab to make the swap permanent.

$sudo vim /etc/fstab

     /swapfile       none    swap    sw      0       0

来源和更多信息可以在这里.

Source and more information can be found here.

相关文章