如何对请求使用线程处理？

问题描述

您好，我正在使用请求模块，我想提高速度，因为我有很多URL，所以我想我可以使用线程来获得更快的速度。以下是我的代码：

import requests

urls = ["http://www.google.com", "http://www.apple.com", "http://www.microsoft.com", "http://www.amazon.com", "http://www.facebook.com"]
for url in urls:
    reponse = requests.get(url)
    value = reponse.json()

但我不知道如何使用线程处理请求...

您能帮帮我吗？

谢谢！

---

解决方案

只需从bashrc添加，您也可以将其用于请求。 不需要使用urllib.Request方法。

类似于：

from concurrent import futures

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']
with futures.ThreadPoolExecutor(max_workers=5) as executor: ## you can increase the amount of workers, it would increase the amount of thread created
    res = executor.map(requests.get,URLS)
responses = list(res) ## the future is returning a generator. You may want to turn it to list.

不过，我喜欢做的是创建一个函数，直接从响应中返回json(如果您想要擦除文本，则直接返回文本)。 并在线程池中使用该函数

import requests
from concurrent import futures
URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

def getData(url):
   res = requests.get(url)
   try:
       return res.json()
   except:
       return res.text
with futures.ThreadPoolExecutor(max_workers=5) as executor:
    res = executor.map(getData,URLS)
responses = list(res) ## your list will already be pre-formated