Hadoop 和 Python:禁用排序
问题描述
我已经意识到，当使用 Python 代码运行 Hadoop 时，映射器或化简器(不确定是哪个)在 reducer.py 打印输出之前对我的输出进行排序.目前它似乎是按字母数字排序的.我想知道是否有办法完全禁用它.我想根据 mapper.py 的打印顺序输出程序.我在 Java 中找到了答案，但在 Python 中没有找到答案.我需要修改 mapper.py 还是命令行参数?
I've realized that when running Hadoop with Python code, either the mapper or reducer (not sure which) is sorting my output before it's printed out by reducer.py. Currently it seems to be sorted alphanumerically. I am wondering if there is a way to completely disable this. I would like the output of the program based off of the order in which it's printed from mapper.py. I've found answers in Java but none for Python. Would I need to modify mapper.py or perhaps the command line arguments?
解决方案
否，如上所述 这里:
No, as stated here:
如果你的reduce任务数不为0，hadoop框架会对你的结果进行排序.没有办法解决.
If your number of reduce tasks is not 0, the hadoop framework will sort your results. There is no way around it.
相关文章