Python的Multiprocessing之进程通信
问题描述
我了解了 Python 多进程的 Pipes/Queues/Shared ctypes Objects/Managers，我想将它们与 Linux 的匿名管道、命名管道、共享内存、套接字等进行比较.我现在有以下问题
I've learned about Python multiprocess's Pipes/Queues/Shared ctypes Objects/Managers, and I want to compare them with Linux's anonymous pipes, named pipes, shared memory, socket, and so on. I now have the following questions
Python 多处理的管道和队列模块是基于匿名管道的.是否提供命名管道?
The pipes and queue modules of Python's multiprocessing are based on anonymous pipes. Does it provide named pipes?
Python multiprocessing.sharedctypes 是否支持独立进程沟通?我认为它只支持父子进程或兄弟进程通信.
Does Python multiprocessing.sharedctypes support independent process communication? I think it only supports father and child process or brotherly process communication.
其中哪些仅用于亲子鉴定过程中或兄弟情谊，可以在独立进程之间进行通信还是不同的主机?
Which of them are only used in the process of paternity or brotherhood, which can be communicated between independent processes or different hosts?
它们各自的特点是什么，应该如何选择?
What are their respective characteristics, how should I choose them?
提前致谢.
解决方案
您的问题相当广泛，大部分答案都可以在 multiprocessing 模块文档中找到.
Your question is quite broad and most of the answers can be found in the multiprocessing module documentation.
下面是一个简短的回答.
Here follows a somewhat short answer.
- 多处理侦听器和客户端 允许选择命名管道作为传输介质.
来自 文档:
- The multiprocessing Listeners and Clients allow to choose named pipes as transport medium.
From the documentation:
multiprocessing.sharedctypes 模块提供了从共享内存中分配 ctypes 对象的函数，这些对象可以被子进程继承.
The multiprocessing.sharedctypes module provides functions for allocating ctypes objects from shared memory which can be inherited by child processes.
您不能跨没有父/子关系的进程使用 multiprocessing.sharedctypes 功能.
You cannot use multiprocessing.sharedctypes functionalities across processes which don't have parent/child relationship.
Python multiprocessing 模块最初是通过 threading API 实现的.到那时，它支持的功能有所增长，但核心思想保持不变.multiprocessing 模块旨在处理 Python 进程系列.对于任何其他用途，subprocess 模块是更好的选择.
Python multiprocessing module was initially implemented over the threading APIs. By the time, it grew in features it supports but the core idea remains the same. The multiprocessing module is intended to deal with Python process families. For any other use, the subprocess module is a better option.
对于跨多个主机分配任务和作业，有更好的解决方案来抽象低级基础架构.您可以查看 Python 项目，例如 Celery 或 Luigi 或更复杂的基础架构，例如 Apache Mesos.
For distribution of tasks and jobs across multiple hosts, there are far better solutions abstracting the low level infrastructure. You can take a look at Python projects such as Celery or Luigi or more complex infrastructures such as Apache Mesos.
相关文章