使用 subprocess 模块是否会释放 python GIL?
问题描述
当通过Python的subprocess模块调用一个耗时较长的linux二进制文件时，是否会释放GIL?
When calling a linux binary which takes a relatively long time through Python's subprocess module, does this release the GIL?
我想并行化一些从命令行调用二进制程序的代码.使用线程(通过 threading 和 multiprocessing.pool.ThreadPool)还是 multiprocessing 更好?我的假设是，如果 subprocess 释放 GIL，那么选择 threading 选项会更好.
I want to parallelise some code which calls a binary program from the command line. Is it better to use threads (through threading and a multiprocessing.pool.ThreadPool) or multiprocessing? My assumption is that if subprocess releases the GIL then choosing the threading option is better.
解决方案
当通过Python的
subprocess模块调用一个耗时较长的linux二进制文件时，是否会释放GIL?
When calling a linux binary which takes a relatively long time through Python's
subprocessmodule, does this release the GIL?
是的，它在调用过程中释放了全局解释器锁(GIL).
Yes, it releases the Global Interpreter Lock (GIL) in the calling process.
您可能知道，在 POSIX 平台上，subprocess 在来自 fork、execve 和 execve 的原始"组件之上提供了便利的接口code>waitpid.
As you are likely aware, on POSIX platforms subprocess offers convenience interfaces atop the "raw" components from fork, execve, and waitpid.
通过检查 CPython 2.7.9 源代码，fork 和 execve not 发布 GIL.但是，这些调用不会阻塞，所以我们不希望 GIL 被释放.
By inspection of the CPython 2.7.9 sources, fork and execve do not release the GIL. However, those calls do not block, so we'd not expect the GIL to be released.
waitpid 当然 确实 阻塞，但我们看到它的实现确实放弃了使用 ALLOW_THREADS 宏的 GIL:
waitpid of course does block, but we see it's implementation does give up the GIL using the ALLOW_THREADS macros:
static PyObject *
posix_waitpid(PyObject *self, PyObject *args)
{
....
Py_BEGIN_ALLOW_THREADS
pid = waitpid(pid, &status, options);
Py_END_ALLOW_THREADS
....
这也可以通过调用一些长时间运行的程序来测试，例如 sleep 来自一个演示多线程 python 脚本.
This could also be tested by calling out to some long running program like sleep from a demonstration multithreaded python script.
相关文章