如何在 Reducer 中访问 Mapper Counter 值?

2022-01-14 00:00:00 hadoop mapreduce java

我想访问 reducer 中的 myCounter.my 值:

I want to acces the myCounter.my value in reducer :

public static class Map extends Mapper<LongWritable, Text, ImmutableBytesWritable, ImmutableBytesWritable>
{
    public static enum myCounter{my};

    @Override
    public void map(LongWritable key, Text value, Context context) 
    {
        context.getCounter(myCounter.my).increment(1);
        context.write( new ImmutableBytesWritable ( ),new ImmutableBytesWritable() );
    }
}


public static class Reduce extends Reducer<ImmutableBytesWritable, ImmutableBytesWritable, Text, Text>
{
    @Override
    public void reduce(ImmutableBytesWritable key,Iterable<ImmutableBytesWritable> result,Context context)
    {

    }
}

从reducer访问映射器的计数器(对于旧API给定)如何使它适用于新的 API?

Accessing a mapper's counter from a reducer(for old API is given ) how to make it work for new API ?

或者

我想知道映射器输出的总数?有没有更好的办法?(我无法访问 Reducer 中的计数器:

I want to know the total number of mapper output ? Is there any better way ? (i am not able to access counter in Reducer:

组名->org.apache.hadoop.mapred.Task$Counter 计数器名->MAP_OUTPUT_RECORDS)

谢谢

推荐答案

您可以通过作业对象访问计数器，使其适用于新 API.

You to make it work for new API by accessing the counters via job object.

Configuration conf = context.getConfiguration();
Cluster cluster = new Cluster(conf);
Job currentJob = cluster.getJob(context.getJobID());
long val=currentJob.getCounters().findCounter(myCounter.my).getValue();

相关文章