在 MongoDB Map Reduce 函数中查询

2022-01-13 00:00:00 mongodb mapreduce java

我已经将大约 250k 条推文流式传输并保存到 MongoDB 中，正如您所见，我正在根据推文中出现的单词或关键字检索它.

I have streamed and saved about 250k tweets into MongoDB and here, I am retrieving it, as you can see, based on a word, or keyword, present in the tweet.

Mongo mongo = new Mongo("localhost", 27017);
DB db = mongo.getDB("TwitterData");
DBCollection collection = db.getCollection("publicTweets");
BasicDBObject fields = new BasicDBObject().append("tweet", 1).append("_id", 0);
BasicDBObject query = new BasicDBObject("tweet", new BasicDBObject("$regex", "autobiography"));
DBCursor cur=collection.find(query,fields);

我想做的是使用 Map-Reduce 并根据关键字，对其进行分类并将其传递给 reduce 函数以计算每个类别下的推文数量，有点像你看到的 这里.在示例中，他计算的是页数，因为它是一个简单的数字.我想做类似的事情:

What I would like to do is to use Map-Reduce and based on the keyword, categorize it and pass it to the reduce function to count the number of tweets under each category, kinda like what you can see here. In the example, he's counting the number of pages as it is a simple number. I wanna do something like:

"if (this.tweet.contains("kword1")) "+
"category = 'kword1 tweets'; " + 
"else if (this.tweet.contains("kword2")) " + 
"category = 'kword2 tweets';

然后使用reduce函数获取计数，就像在示例程序中一样.

and then use the reduce function to get the count, just like in the sample program.

我知道语法不正确，但这正是我想做的.有没有办法实现它?谢谢！

I know that the syntax is incorrect, but that's pretty much what I would like to do. Is there any way of achieving it? Thanks!

PS:哦，我正在用 Java 编写代码.因此，Java 语法将受到高度赞赏.谢谢！

PS: Oh, and I'm coding in Java. So the Java syntax would be highly appreciated. Thank you!

发布的代码的输出是这样的:

The output of the code posted is something like this:

{ "tweet" : "An autobiography is a book that reveals nothing bad about its writer except his memory."}
{ "tweet" : "I refuse to read anything that's not real the only thing I've read since biff books is Jordan's autobiography #lol"}
{ "tweet" : "well we've had the 2012 publication of Ashley's Good Books, I predict 2013 will be seeing an autobiography ;)"}

当然，这适用于所有带有自传"一词的推文.我想在 map 函数中使用它，将其归类为自传推文"(以及其他关键字)，然后将其发送到 reduce 函数以计算所有内容并返回带有单词 in 的推文数量它.

This of course, is for all tweets with the word "autobiography". What I'd like is to use this in the map function, categorize it as a "autobiography tweet" (and other keywords too), and then send it to the reduce function to count everything and return the number of tweets with the word in it.

类似:

{"_id" : "Autobiography Tweets" , "value" : { "publicTweets" : 3.0}}
{"_id" : "Biography Tweets" , "value" : { "publicTweets" : 15.0}}

推荐答案

您可能想尝试以下方法:

You might want to try the following:

    String map = "function() { " +
                 "    var regex1 = new RegExp('autobiography', 'i'); " +
                 "    var regex2 = new RegExp('book', 'i'); " +
                 "    if (regex1.test(this.tweet) ) " +
                 "         emit('Autobiography Tweet', 1); " +
                 "    else if (regex2.test(this.tweet) ) " +
                 "         emit('Book Tweet', 1); " +
                 "    else " +
                 "       emit('Uncategorized Tweet', 1); " +
                 "}";

    String reduce = "function(key, values) { " +
                    "    return Array.sum(values); " +
                    "}";

    MapReduceCommand cmd = new MapReduceCommand(collection, map, reduce,
             null, MapReduceCommand.OutputType.INLINE, null);
    MapReduceOutput out = collection.mapReduce(cmd);

    try {
        for (DBObject o : out.results()) {

            System.out.println(o.toString());

       }
    } catch (Exception e) {
        e.printStackTrace();
    }

相关文章