区分 celery、kombu、PyAMQP 和 RabbitMQ/ironMQ

2022-01-11 00:00:00 python heroku celery kombu rabbitmq

问题描述

我想将图像上传到 S3 服务器，但在上传之前我想生成 3 种不同大小的缩略图，并且我希望它在请求/响应周期之外完成，因此我使用的是 celery.我已经阅读了文档，这是我所理解的.如果我错了，请纠正我.

I want to upload images to S3 server, but before uploading I want to generate thumbnails of 3 different sizes, and I want it to be done out of request/response cycle hence I am using celery. I have read the docs, here is what I have understood. Please correct me if I am wrong.

  1. Celery 可帮助您在请求响应周期之外管理任务队列.
  2. 还有一种叫做胡萝卜/kombu 的东西——它是一个 django 中间件，用于打包通过 celery 创建的任务.
  3. 然后是第三层 PyAMQP，它促进胡萝卜与代理的通信.例如.RabbitMQ、AmazonSQS、ironMQ 等
  4. Broker 位于不同的服务器上，为您处理事务.

现在我的理解是 - 如果多个用户同时上传图片，celery 会排队调整大小，而调整大小实际上会发生在 ironMQ 服务器上，因为它在 heroku 上提供了一个很酷的插件.

Now my understanding is - if multiple users upload image at the same time, celery will queue the resizing, and the resizing will actually happen at the ironMQ server, since it offers a cool addon on heroku.

现在的疑惑:

  1. 但是图片调整大小后，ironMQ会推送到S3服务器，还是会在处理完成后通知..我不清楚.

  1. But what after the image is resized, will ironMQ push it to the S3 server, or will it notify once the process is completed.. i am not clear about it.

芹菜和昆布/胡萝卜有什么区别，你能形象地解释一下吗?

What is the difference between celery and kombu/carrot, could you explain vividly.

解决方案

IronMQ 不会为您处理您的任务；它只是作为 Celery 的后端来跟踪需要执行的工作.

IronMQ does not process your tasks for you; it simply serves as the backend for Celery to keep track of what jobs need to be performed.

所以，这就是发生的事情.假设你有两台服务器，你的 web 服务器和你的 Celery 服务器.您的 Web 服务器负责处理请求，您的 Celery 服务器创建缩略图并将其上传到 S3.典型请求如下所示:

So, here's what happens. Assume you have two servers, your web server and your Celery server. Your web server is responsible for handling requests, your Celery server creates the thumbnails and uploads them to S3. Here's what a typical request looks like:

  1. 您的用户将图像上传到您的网络服务器.
  2. 您将该图像存储在某个地方——我个人建议您将其放在 S3 上，但您也可以将其存储在例如 IronCache，base64 编码.关键是把它放在你的 Celery 服务器可以访问的地方.
  3. 你在 Celery 上排队，将图像的位置传递给你的 Celery 服务器.
  4. 您的 Celery 服务器会下载图像、生成缩略图并将其上传到 S3.然后它将 S3 URL 存储在作业结果中.
  5. 您的网络服务器会等待作业完成，然后才能访问结果.或者，您可以让 Celery 服务器将结果存储在数据库本身中.关键是 Celery 服务器完成了繁重的工作(生成缩略图)，并且在执行时不会阻止请求循环.
  1. Your user uploads the image to your web server.
  2. You store that image somewhere--I'd recommend putting it on S3 right then, personally, but you could also store it in, for example IronCache, base64-encoded. The point is to put it somewhere your Celery server can access it.
  3. You queue up a job on Celery, passing the location of the image to your Celery server.
  4. Your Celery server downloads the image, generates your thumbnails, and uploads them to S3. It then stores the S3 URLs in the job results.
  5. Your web server waits until the job finishes, then has access to the results. Alternatively, you could have your Celery server store the results in the database itself. The point is that the Celery server does the heavy lifting (generating the thumbnails) and does not hold up the request loop while it does.

我写了一个在 Heroku 上使用 IronMQ 的示例.你可以在这里看到它:http://iron-celery-demo.herokuapp.com.您可以在 Github 上查看示例的源代码 和 阅读教程，它解释得非常透彻和步骤-一步一步如何在 Heroku 上部署 Celery.

I wrote an example for using IronMQ on Heroku. You can see it here: http://iron-celery-demo.herokuapp.com. You can see the source for the example on Github and read the tutorial, which explains pretty thoroughly and step-by-step how to deploy Celery on Heroku.

清理 AMQP 的东西:

To clear up the AMQP stuff:

  • IronMQ 是 Iron.io 开发的基于云的消息队列服务.
  • AMQP 是一种开放的消息传递规范
  • RabbitMQ 是 AMQP 规范最流行的实现(据我所知).
  • PyAMQP 是一个 Python 库，可让 Python 客户端与任何 AMQP 实现进行通信，包括 RabbitMQ

IronMQ 和 RabbitMQ/AMQP 的最大区别之一是 IronMQ 是托管和管理的，因此您不必自己托管服务器并担心正常运行时间.该规范在差异化方面提供了更多信息，并且存在潜在的差异，但 Celery 将其中的大部分抽象出来.因为您使用的是 Celery，所以您可能会注意到的唯一区别是 IronMQ 是托管的，因此您不必站起来管理自己的服务器.

One of the biggest differences between IronMQ and RabbitMQ/AMQP is that IronMQ is hosted and managed, so you don't have to host the server yourself and worry about uptime. The spec offers a bunch more in terms of differentiation, and there are underlying differences, but Celery abstracts most of those away. Because you're using Celery, the only difference you're liable to notice is that IronMQ is hosted, so you don't have to stand up and manage your own server.

全面披露:我受雇于 IronMQ 背后的公司 Iron.io.

Full disclosure: I am employed by Iron.io, the company behind IronMQ.

相关文章