Python CGI 队列

2022-01-21 00:00:00 python queue cgi

问题描述

我正在使用 Python 开发一个相当简单的 CGI.我即将将它放入 Django 等.整体设置是非常标准的服务器端(即计算在服务器上完成):

I'm working on a fairly simple CGI with Python. I'm about to put it into Django, etc. The overall setup is pretty standard server side (i.e. computation is done on the server):

  1. 用户上传数据文件并点击运行"按钮
  2. 服务器在后台并行分叉作业，使用大量 RAM 和处理器能力.大约 5-10 分钟后(平均用例)，程序终止，创建了一个输出文件和一些 .png 图形文件.
  3. 服务器显示带有数字和一些摘要文本的网页
  1. User uploads data files and clicks "Run" button
  2. Server forks jobs in parallel behind the scenes, using lots of RAM and processor power. ~5-10 minutes later (average use case), the program terminates, having created a file of its output and some .png figure files.
  3. Server displays web page with figures and some summary text

我认为不会有成百上千的人同时使用它；但是，因为正在进行的计算需要相当多的 RAM 和处理器能力(每个实例都使用 Python 的 Pool 分叉出 CPU 最密集的任务).

I don't think there are going to be hundreds or thousands of people using this at once; however, because the computation going on takes a fair amount of RAM and processor power (each instance forks the most CPU-intensive task using Python's Pool).

我想知道您是否知道使用排队系统是否值得麻烦.我遇到了一个名为 beanstalkc 的 Python 模块，但在页面上它说它是一个内存中"排队系统.

I wondered if you know whether it would be worth the trouble to use a queueing system. I came across a Python module called beanstalkc, but on the page it said it was an "in-memory" queueing system.

在这种情况下，内存中"是什么意思?我担心内存，而不仅仅是 CPU 时间，因此我想确保一次只运行一个作业(或保存在 RAM 中，无论它是否接收 CPU 时间).

What does "in-memory" mean in this context? I worry about memory, not just CPU time, and so I want to ensure that only one job runs (or is held in RAM, whether it receives CPU time or not) at a time.

另外，我试图决定是否

  • 结果页面(由 CGI 提供)应该告诉您它在队列中的位置(直到它运行并显示实际结果页面)

    或

  • 用户应将他们的电子邮件地址提交给 CGI，CGI 将在完成后通过电子邮件将指向结果页面的链接发送给他们.
  • the result page (served by the CGI) should tell you it's position in the queue (until it runs and then displays the actual results page)

    OR

  • the user should submit their email address to the CGI, which will email them the link to the results page when it is complete.

对于此类问题，您认为适合轻型交通 CGI 的设计方法是什么?非常感谢您的建议.

What do you think is the appropriate design methodology for a light traffic CGI for a problem of this sort? Advice is much appreciated.

解决方案

一定要用celery.您可以运行 amqp 服务器，或者我认为您可以将数据库作为消息队列进行起诉.它允许您在后台运行任务，并且可以根据需要使用多台工作机器进行处理.如果您使用 django-celery

Definitely use celery. You can run an amqp server or I think you can sue the database as a queue for the messages. It allows you to run tasks in the background and it can use multiple worker machines to do the processing if you want. It can also do cron jobs that are database based if you use django-celery

后台运行任务就这么简单:

It's as simple as this to run a task in the background:

@task
def add(x, y):
    return x + y

在一个项目中，我将工作分配到 4 台机器上，而且效果很棒.

In a project I have it's distributing the work over 4 machines and it works great.

相关文章