如何在没有线程或任务队列的情况下在 Flask 中运行后台作业

2022-01-21 00:00:00 python multithreading flask queue celery

问题描述

我正在使用 Flask-restplus 构建 REST API.我的一个端点获取从客户端上传的文件并运行一些分析.该作业最多使用 30 秒.我不希望这项工作阻止主进程.因此端点将立即返回 200 或 201 的响应，作业仍然可以运行.结果将保存到数据库中，稍后检索.

I am building REST API with Flask-restplus. One of my endpoints takes a file uploaded from client and run some analysis. The job uses up to 30 seconds. I don't want the job to block the main process. So the endpoint will return a response with 200 or 201 right away, the job can still be running. Results will be saved to database which will be retrieved later.

对于长期运行的工作，我似乎有两种选择.

It seems I have two options for long-running jobs.

  1. 线程
  2. 任务队列

线程相对简单.但问题是，Flask 应用程序的线程数是有限制的.在独立的 Python 应用程序中，我可以为线程使用队列.但这是 REST api，每个请求调用都是独立的.我不知道是否有办法为此维护一个全局队列.所以如果请求超过线程限制，它将无法接受更多的请求.

Threading is relatively simpler. But problem is, there is a limit of thread numbers for Flask app. In a standalone Python app, I could use a queue for the threads. But this is REST api, each request call is independent. I don't know if there is a way to maintain a global queue for that. So if the requests exceed the thread limit, it won't be able to take more requests.

带有 Celery 和 Redis 的任务队列可能是更好的选择.但这只是概念验证，时间线有点紧.使用 Flask 设置 Celery、Redis 并不容易，我在 Windows 的开发机器上遇到了很多麻烦.它将部署在有点复杂的 AWS 上.

Task-queue with Celery and Redis is probably better option. But this is just a proof of concept thing, and time line is kind of tight. Setting up Celery, Redis with Flask is not easy, I am having lots of trouble on my dev machine which is a Windows. It will be deployed on AWS which is kind of complex.

我想知道这种情况是否有第三种选择?

I wonder if there is a third option for this case?

解决方案

我强烈推荐使用 Celery 正如你在帖子中已经提到的那样.它正是为这个用例而构建的.他们的文档内容丰富，网上不乏可以帮助您快速启动和运行的示例.

I would HIGHLY recommend using Celery as you have already mentioned in your post. It is built exactly for this use case. Their docs are really informative and there are no shortage of examples online that can get you up and running quickly.

另外，我想说 THIS 将是您开始的绝佳第一个资源与.

Additionally, I would say THIS would be an excellent first resource for you to start with.

相关文章