python中一个好的持久同步队列
问题描述
我不会立即关心 fifo 或 filo 选项，但将来可能会很好..
I don't immediately care about fifo or filo options, but it might be nice in the future..
我正在寻找一种快速而简单的方法，可以在磁盘上存储(最多一个数据或数千万个条目)，可以由多个进程获取和放置.这些条目只是简单的 40 字节字符串，而不是 python 对象.并不真正需要 shelve 的所有功能.
What I'm looking for a is a nice fast simple way to store (at most a gig of data or tens of millions of entries) on disk that can be get and put by multiple processes. The entries are just simple 40 byte strings, not python objects. Don't really need all the functionality of shelve.
我看过这个 http://code.activestate.com/lists/python-list/310105/看起来很简单.需要升级到新的Queue版本.
I've seen this http://code.activestate.com/lists/python-list/310105/ It looks simple. It needs to be upgraded to the new Queue version.
想知道有没有更好的?我担心如果电源中断，整个腌制文件会损坏，而不仅仅是一条记录.
Wondering if there's something better? I'm concerned that in the event of a power interruption, the entire pickled file becomes corrupt instead of just one record.
解决方案
尝试使用 Celery.它不是纯 Python，因为它使用 RabbitMQ 作为后端，但它是可靠的、持久的和分布式的，总而言之，从长远来看比使用文件或数据库要好得多.
Try using Celery. It's not pure python, as it uses RabbitMQ as a backend, but it's reliable, persistent and distributed, and, all in all, far better then using files or database in the long run.
相关文章