如何在 python 3 中将队列与并发未来的 ThreadPoolExecutor 一起使用?

2022-01-21 00:00:00 python python-3.x multithreading concurrent.futures queue

问题描述

我正在使用简单的线程模块来执行并发作业.现在我想利用并发期货模块.有人能给我举一个使用队列和并发库的例子吗?

I am using simple threading modules to do concurrent jobs. Now I would like to take advantages of concurrent futures modules. Can some put me a example of using a queue with concurrent library?

我收到 TypeError: 'Queue' object is not iterable我不知道如何迭代队列

I am getting TypeError: 'Queue' object is not iterable I dont know how to iterate queues

代码片段:

 def run(item):
      self.__log.info(str(item))
      return True
<queue filled here>

with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:
        furtureIteams = { executor.submit(run, item): item for item in list(queue)}
        for future in concurrent.futures.as_completed(furtureIteams):
            f = furtureIteams[future]
            print(f)

解决方案

我会建议这样的事情:

def run(queue):
      item = queue.get()
      self.__log.info(str(item))
      return True
<queue filled here>
workerThreadsToStart = 10
with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:
        furtureIteams = { executor.submit(run, queue): index for intex in range(workerThreadsToStart)}
        for future in concurrent.futures.as_completed(furtureIteams):
            f = furtureIteams[future]
            print(f)

您将遇到的问题是，队列被认为是无止境的，并且作为一种媒介来解耦将某些内容放入队列的线程和将项目从队列中取出的线程.

The problem you will run in is that a queue is thought to be endless and as a medium to decouple the threads that put something into the queue and threads that get items out of the queue.

当

  1. 您的商品数量有限或
  2. 您一次计算所有项目

然后并行处理它们，队列没有意义.在这些情况下，ThreadPoolExecutor 会使队列过时.

and afterwards process them in parallel, a queue makes no sense. A ThreadPoolExecutor makes a queue obsolete in these cases.

我查看了 ThreadPoolExecutor 源代码:

I had a look at the ThreadPoolExecutor source:

def submit(self, fn, *args, **kwargs): # line 94
    self._work_queue.put(w) # line 102

里面使用了一个队列.

相关文章