org.apache.hadoop.mapreduce.counters.LimitExceededException:计数器太多:121 max=120

2022-01-13 00:00:00 hadoop mapreduce apache java oozie

我正在运行一个有少量计数器和多输出的 hadoop 作业(来自 oozie).

I'm running a hadoop job ( from oozie ） that has few counters, and multioutput.

我收到如下错误:org.apache.hadoop.mapreduce.counters.LimitExceededException:计数器太多:121 max=120

I get error like: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120

然后我删除了所有具有计数器的代码，并将 mout.setCountersEnabled 设置为 false.并且还在 hadoop 配置中将最大计数器设置为 240.

Then I removed all the code that has counters, and also set mout.setCountersEnabled to false. And also set the max counters to 240 in hadoop config.

现在我仍然遇到同样的错误org.apache.hadoop.mapreduce.counters.LimitExceededException:计数器太多:241 max=240

Now I still get the same error org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 241 max=240

我该如何解决这个问题?是否有任何隐藏计数器存在的可能性?我怎样才能弄清楚之前有哪些计数器超过 240 ?(在我可以打印任何东西之前，这个过程似乎已经停止了?)

How can I solve this problem? Is there any possibility that any hidden counters exists? How can I make clear what counters there before exceeds 240 ? (The process looks like stopped before I can print anything? )

谢谢，新松

推荐答案

我用下面的方法解决了这个问题:vi $HADOOP_HOME/conf/mapred-site.xml

I solved the problem use the following method: vi $HADOOP_HOME/conf/mapred-site.xml

<property>
    <name>mapreduce.job.counters.limit</name>
    <!--<value>120</value>-->
   <value>20000</value>
    <description>Limit on the number of counters allowed per job. The default value is 200.</description>
</property>

相关文章