超出最大递归深度.多处理和 bs4

2022-01-12 00:00:00 python beautifulsoup multiprocessing parsing

问题描述

我正在尝试让解析器使用 beautifulSoup 和多处理.我有一个错误:

I'm trying to make a parser use beautifulSoup and multiprocessing. I have an error:

RecursionError: 超出最大递归深度

RecursionError: maximum recursion depth exceeded

我的代码是:

import bs4, requests, time
from multiprocessing.pool import Pool

html = requests.get('https://www.avito.ru/moskva/avtomobili/bmw/x6?sgtd=5&radius=0')
soup = bs4.BeautifulSoup(html.text, "html.parser")

divList = soup.find_all("div", {'class': 'item_table-header'})


def new_check():
    with Pool() as pool:
        pool.map(get_info, divList)

def get_info(each):
   pass

if __name__ == '__main__':
    new_check()

为什么会出现此错误以及如何解决?

Why I get this error and how I can fix it?

更新:所有错误文本都是

Traceback (most recent call last):
  File "C:/Users/eugen/PycharmProjects/avito/main.py", line 73, in <module> new_check()
  File "C:/Users/eugen/PycharmProjects/avito/main.py", line 67, in new_check
    pool.map(get_info, divList)
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 644, in get
    raise self._value
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 424, in _handle_tasks
    put(task)
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingconnection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingeduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
RecursionError: maximum recursion depth exceeded

解决方案

当你使用 multiprocessing 时，你传递给 worker 的所有东西都必须是 腌制.

When you use multiprocessing, everything you pass to a worker has to be pickled.

很遗憾，很多 BeautifulSoup 树无法腌制.

Unfortunately, many BeautifulSoup trees can't be pickled.

这有几个不同的原因.其中一些是已修复的错误，因此您可以尝试确保您拥有最新的 bs4 版本，而有些则特定于不同的解析器或树构建器......但很有可能没有这样的事情会有帮助的.

There are a few different reasons for this. Some of them are bugs that have since been fixed, so you could try making sure you have the latest bs4 version, and some are specific to different parsers or tree builders… but there's a good chance nothing like this will help.

但根本问题是树中的许多元素都包含对树其余部分的引用.

But the fundamental problem is that many elements in the tree contain references to the rest of the tree.

有时，这会导致实际的无限循环，因为循环引用对于其循环引用检测来说过于间接.但这通常是一个可以修复的错误.

Occasionally, this leads to an actual infinite loop, because the circular references are too indirect for its circular reference detection. But that's usually a bug that gets fixed.

但是，更重要的是，即使循环不是无限，它仍然可以从树的其余部分拖入 1000 多个元素，这已经足以导致递归错误.

But, even more importantly, even when the loop isn't infinite, it can still drag in more than 1000 elements from all over the rest of the tree, and that's already enough to cause a RecursionError.

我认为后者就是这里发生的事情.如果我使用您的代码并尝试腌制 divList[0]，它会失败.(如果我提高递归限制并计算帧数，它需要 23080 的深度，这远远超过默认的 1000.)但是如果我采用完全相同的 div 并解析分开，它成功没有问题.

And I think the latter is what's happening here. If I take your code and try to pickle divList[0], it fails. (If I bump the recursion limit way up and count the frames, it needs a depth of 23080, which is way, way past the default of 1000.) But if I take that exact same div and parse it separately, it succeeds with no problem.

所以，一种可能性是只做 sys.setrecursionlimit(25000).这将解决这个确切页面的问题，但是稍微不同的页面可能需要更多.(另外，将递归限制设置得那么高通常不是一个好主意——不是因为浪费了内存，而是因为这意味着实际的无限递归需要 25 倍的时间和 25 倍的资源浪费来检测.)

So, one possibility is to just do sys.setrecursionlimit(25000). That will solve the problem for this exact page, but a slightly different page might need even more than that. (Plus, it's usually not a great idea to set the recursion limit that high—not so much because of the wasted memory, but because it means actual infinite recursion takes 25x as long, and 25x as much wasted resources, to detect.)

另一个技巧是编写修剪树"的代码，在您腌制之前/之后消除 div 中的任何向上链接.这是一个很好的解决方案，但它可能需要大量工作，并且需要深入了解 BeautifulSoup 的工作原理，我怀疑你是否愿意这样做.

Another trick is to write code that "prunes the tree", eliminating any upward links from the div before/as you pickle it. This is a great solution, except that it might be a lot of work, and requires diving into the internals of how BeautifulSoup works, which I doubt you want to do.

最简单的解决方法有点笨拙，但是……您可以将汤转换为字符串，将其传递给孩子，然后让孩子重新解析它:

The easiest workaround is a bit clunky, but… you can convert the soup to a string, pass that to the child, and have the child re-parse it:

def new_check():
    divTexts = [str(div) for div in divList]
    with Pool() as pool:
        pool.map(get_info, divTexts)

def get_info(each):
    div = BeautifulSoup(each, 'html.parser')

if __name__ == '__main__':
    new_check()

这样做的性能成本可能无关紧要；更大的担忧是，如果您的 HTML 不完美，则转换为字符串并重新解析它可能不是完美的往返.所以，我建议你先做一些没有多处理的测试，以确保这不会影响结果.

The performance cost for doing this is probably not going to matter; the bigger worry is that if you had imperfect HTML, converting to a string and re-parsing it might not be a perfect round trip. So, I'd suggest that you do some tests without multiprocessing first to make sure this doesn't affect the results.

相关文章