从 windows eclipse 运行 Mapreduce(yarn) 时出错

2022-01-13 00:00:00 hadoop mapreduce java eclipse hadoop-yarn

我正在从我的 Eclipse 中运行 WordCount 程序.我尝试使用 Hadoop1.x 运行良好.在 hadoop2.x 上运行时遇到问题

I am running a WordCount program from my eclipse. I tried with Hadoop1.x it is running fine. Facing issue while running on hadoop2.x

我试过了1)将所有 xml 添加到我的类路径中.2)也尝试了conf.set()，在conf对象中设置xml属性.

i tried 1)added all xml into my classpath. 2)also tried conf.set(), setting xml properties in conf object.

还在日志中显示:-没有可用于容器 container_1394042163908_0573_01_000001 的日志

Also in logs it says :-No logs available for container container_1394042163908_0573_01_000001

  Application application_1394042163908_0573 failed 2 times due to AM Container for      appattempt_1394042163908_0573_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerEx    ecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)`enter code here`
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
.Failing this attempt.. Failing the application.

推荐答案

我已经解决了这个问题，有一个jira可以解决它:https://issues.apache.org/jira/browse/MAPREDUCE-5655

I have figured out the issue, there is a jira for it:https://issues.apache.org/jira/browse/MAPREDUCE-5655

我刚刚添加了 YARNRunner.java &MRApps.java 到我的项目.它还需要在 windows 框上的 mapred-site.xml 中添加以下属性，以便作业启动器知道作业运行器将是一个 linux:

I just added YARNRunner.java & MRApps.java to my project . It also reqires to add the following property to mapred-site.xml on the windows box, so that the job launcher knows, that the job runner will be a linux:

<property>
<name>mapred.remote.os</name>
<value>Linux</value>
<description>Remote MapReduce framework's OS, can be either Linux or Windows</description>
</property>

而且 MapReduce 现在运行良好.

And MapReduce is running fine now.

同时将 org.apache.hadoop.util.Shell.java 复制到您的项目中.

Also Copy org.apache.hadoop.util.Shell.java into your project.

您可以注释掉下面这行，以消除 winutils.exe 错误.
throw new IOException("在 Hadoop 二进制文件中找不到可执行文件 " + fullExeName + ".");

You can comment out the below line,to remove the winutils.exe Error.
throw new IOException("Could not locate executable " + fullExeName + " in the Hadoop binaries.");

相关文章