与 celery 正在进行的任务交互
问题描述
我们有一个基于 rabbitMQ 和 Celery 的分布式架构.我们可以毫无问题地并行启动多个任务.扩展性好.
We have a distributed architecture based on rabbitMQ and Celery.
We can launch in parallel multiple tasks without any issue. The scalability is good.
现在我们需要远程控制任务:暂停、恢复、取消.我们找到的唯一解决方案是在 Celery 任务中对另一个任务进行 RPC 调用，该任务在 DB 请求后回复命令.Celery 任务和 RPC 任务不在同一台机器上，只有 RPC 任务可以访问数据库.
Now we need to control the task remotely: PAUSE, RESUME, CANCEL. The only solution we found is to make in the Celery task a RPC call to another task that replies the command after a DB request. The Celery task and RPC task are not on the same machine and only the RPC task has access to the DB.
您对如何改进它并轻松与正在进行的任务进行沟通有什么建议吗?谢谢
Do you have any advice how to improve it and easily communicate with an ongoing task? Thank you
事实上，我们想做如下图所示的事情.Blue 配置或 Orange 配置很容易，但我们不知道如何同时进行.Worker 订阅了一个通用的 Jobs queue，每个 worker 都有自己的 Admin queue 在交换中声明.
In fact we would like to do something like in the picture below. It's easy to do the Blue configuration or the Orange, but we don't know how to do both simultaneously.
Workers are subscribing to a common Jobs queue and each worker has its own Admin queue declared on an exchange.
如果 Celery 无法做到这一点，我愿意接受其他框架(如 python-rq)的解决方案.
IF this is not possible with Celery, I'am open to a solution with other frameworks like python-rq.
解决方案
看起来像 控制总线模式.
为了更好的可扩展性和减少 RPC 调用，我建议反转逻辑.PAUSE, RESUME, CANCEL 命令在状态改变发生时通过控制总线推送到 Celery 任务.Celery 应用程序会将 Celery 应用程序的当前状态存储在存储中(可能在内存中，在文件系统中......).如果即使在应用程序停止/启动后也必须保持任务状态，这将涉及更多工作以保持两个应用程序同步(例如启动时同步).
For a better scalability and in order to reduce the RPC call, I recommend to reverse the logic. The PAUSE, RESUME, CANCEL command are push to the Celery tasks through a control bus when the state change occurs. The Celery app will store the current state of the Celery app in a store (could be in memory, on the filesystem..). If task states must be kept even after a stop/start of the app, It will involve more work in order to keep both app synchronized (eg. synchronization at startup).
相关文章