MapReduce 结果似乎限制为 100?

2022-01-13 00:00:00 python mongodb pymongo mapreduce

问题描述

我在 MongoDB 和 python 中使用 Map Reduce，但遇到了一个奇怪的限制.我只是想计算书"记录的数量.它在少于 100 条记录时有效，但当超过 100 条记录时，由于某种原因计数会重置.

I'm playing around with Map Reduce in MongoDB and python and I've run into a strange limitation. I'm just trying to count the number of "book" records. It works when there are less than 100 records but when it goes over 100 records the count resets for some reason.

这是我的 MR 代码和一些示例输出:

Here is my MR code and some sample outputs:

var M = function () {
book = this.book;
emit(book, {count : 1});
}

var R = function (key, values) {
var sum = 0;
values.forEach(function(x) {
sum += 1;
});
var result = {
count : sum 
};
return result;
}

记录数为99时的MR输出:

MR output when record count is 99:

{u'_id': u'superiors', u'value': {u'count': 99}}

记录数为101时的MR输出:

MR output when record count is 101:

{u'_id': u'superiors', u'value': {u'count': 2.0}}

有什么想法吗?

解决方案

你的 reduce 函数应该是对 count 值求和，而不仅仅是添加 1 每个值.否则，一个 reduce 的输出不能被正确地用作另一个 reduce 的输入.试试这个:

Your reduce function should be summing up the count values, not just adding 1 for each value. Otherwise the output of a reduce can't properly be used as input back into another reduce. Try this instead:

var R = function (key, values) {
  var sum = 0;
  values.forEach(function(x) {
    sum += x.count;
  });
  var result = {
    count : sum 
  };
  return result;
}

相关文章