将 Python Twisted 与多处理混合使用?
问题描述
我需要用 Python 编写一个类似代理的程序，工作流程与 Web 代理非常相似.该程序位于客户端和服务器之间，接收客户端向服务器发送的请求，处理请求，然后将其发送到原始服务器.当然使用的协议是私有协议，使用TCP.
I need to write a proxy like program in Python, the work flow is very similar to a web proxy. The program sits in between the client and the server, incept requests sent by the client to the server, process the request, then send it to the original server. Of course the protocol used is a private protocol uses TCP.
为了尽量减少工作量，我想使用 Python Twisted 来处理请求接收(部分充当服务器)和重新发送(部分充当客户端).
To minimize the effort, I want to use Python Twisted to handle the request receiving (the part acts as a server) and resending (the part acts as a client).
为了最大化性能，我想使用 python 多处理(线程有 GIL 限制)将程序分成三个部分(进程).第一个进程运行 Twisted 接收请求，将请求放入队列，并立即向原始客户端返回成功.第二个进程从队列中获取请求，进一步处理请求并将其放入另一个队列.第三个进程从第二个队列中获取请求并将其发送到原始服务器.
To maximum the performance, I want to use python multiprocessing (threading has the GIL limit) to separate the program into three parts (processes). The first process runs Twisted to receive requests, put the request in a queue, and return success immediately to the original client. The second process take request from the queue, process the request further and put it to another queue. The 3rd process take request from the 2nd queue and send it to the original server.
我是 Python Twisted 的新手，我知道它是事件驱动的，我还听说最好不要将 Twisted 与线程或多处理混合使用.所以我不知道这种方式是否合适或者仅使用Twisted是否有更优雅的方式?
I was a new comer to Python Twisted, I know it is event driven, I also heard it's better to not mix Twisted with threading or multiprocessing. So I don't know whether this way is appropriate or is there a more elegant way by just using Twisted?
解决方案
Twisted 有自己的事件驱动运行子进程的方式(在我的谦虚但正确的观点中)比 multiprocessing 模块.核心 API 是 spawnProcess，但是像 ampoule 提供更高级别的包装器.
Twisted has its own event-driven way of running subprocesses which is (in my humble, but correct, opinion) better than the multiprocessing module. The core API is spawnProcess, but tools like ampoule provide higher-level wrappers over it.
如果您使用 spawnProcess，您将能够像处理 Twisted 中的任何其他事件一样处理子流程的输出；如果你使用multiprocessing，你需要开发你自己的基于队列的方法，以某种方式从子进程获取输出到Twisted mainloop，因为正常的callFromThread 线程可能使用的 API 在另一个进程中不起作用.根据你如何称呼它，它要么尝试腌制反应器，要么只是在子进程中使用不同的非工作反应器；无论哪种方式，它都会永远失去你的电话.
If you use spawnProcess, you will be able to handle output from subprocesses in the same way you'd handle any other event in Twisted; if you use multiprocessing, you'll need to develop your own queue-based way of getting output from a subprocess into the Twisted mainloop somehow, since the normal callFromThread API that a thread might use won't work from another process. Depending on how you call it, it will either try to pickle the reactor, or just use a different non-working reactor in the subprocess; either way it will lose your call forever.
相关文章