Python + MongoDB - 光标迭代太慢

2022-01-24 00:00:00 python performance mongodb iteration database-cursor

问题描述

我实际上正在从事一个搜索引擎项目.我们正在使用 python + mongoDb.

I'm actually working in a search engine project. We are working with python + mongoDb.

在对 mongo db 执行 find() 命令后，我有一个 pymongo 光标.pymongo 游标有大约 20k 个结果.

I have a pymongo cursor after excecuting a find() command to the mongo db. The pymongo cursor has around 20k results.

我注意到 pymongo 光标上的迭代与普通迭代相比非常慢，例如相同大小的列表.

I have noticed that the iteration over the pymongo cursor is really slow compared with a normal iteration over for example a list of the same size.

我做了一个小基准测试:

I did a little benchmark:

  • 迭代 20k 字符串列表:0.001492 秒
  • 在 pymongo 光标上迭代，结果为 20k:1.445343 秒

差别真的很大.这么多结果可能不是问题，但如果我有数百万个结果，时间将是不可接受的.

The difference is really a lot. Maybe not a problem with this amounts of results, but if I have millions of results the time would be unacceptable.

有没有人知道为什么 pymongo 游标太慢而无法迭代?知道如何在更短的时间内迭代光标吗?

Has anyone got an idea of why pymongo cursors are too slow to iterate? Any idea of how can I iterate the cursor in less time?

一些额外的信息:

  • Python v2.6
  • PyMongo v1.9
  • MongoDB v1.6 32 位

解决方案

记住 pymongo 驱动程序不会一次性返回所有 20k 结果.当您迭代时，它正在对 mongodb 后端进行网络调用以获取更多项目.当然它不会像字符串列表那么快.但是，我建议尝试调整光标 batch_size 如api文档:

Remember the pymongo driver is not giving you back all 20k results at once. It is making network calls to the mongodb backend for more items as you iterate. Of course it wont be as fast as a list of strings. However, I'd suggest trying to adjust the cursor batch_size as outlined in the api docs:

相关文章