Giraph 源码编译常见错误

2022-03-03 00:00:00 专区版本指定错误解决方法

Giraph 源码编译命令：

giraph_basedir $ mvn -Phadoop_yarn -Dhadoop.version={指定hadoop版本} -DskipTests clean package

giraph_basedir $ mvn -Phadoop_2 -Dhadoop.version={指定hadoop版本} -DskipTests clean package

如果一切正常，在giraph_basedir/giraph-dist目录下已经生成目标 jar

Giraph 编译中常见错误

1. hadoop版本问题

[ERROR] /home/cyf/giraph-release/giraph-core/target/munged/main/org/apache/giraph/job/GiraphJob.java:[213,10] 错误: setPingInterval(Configuration,int)在Client中不是公共的; 无法从外部程序包中对其进行访问

[INFO] 1 error

[INFO] -------------------------------------------------------------

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Giraph Parent ............................... SUCCESS [ 3.658 s]

[INFO] Apache Giraph Core ................................. FAILURE [ 5.584 s]

[INFO] Apache Giraph Blocks Framework ..................... SKIPPED

[INFO] Apache Giraph Examples ............................. SKIPPED

[INFO] Apache Giraph Distribution ......................... SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 9.558 s

[INFO] Finished at: 2016-08-28T20:10:25+08:00

[INFO] Final Memory: 41M/709M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :giraph-core

解决方法：高版本的 Hadoop 的 hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java 文件中的 setPingInterval(Configuration,int) 方法的访问权限已经从public 变为 default，包外已经不能直接访问。只能降低Hadoop版本，2.5.2，2.6.3，2.7.2 还都是public 类型，以上版本都变成了default。

2. SASL_PROPS 变量问题

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:

[ERROR] /home/cyf/giraph-release/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,67] 错误: 找不到符号

[ERROR] 符号: 变量 SASL_PROPS

[ERROR] 位置: 类 SaslRpcServer

[ERROR] /home/cyf/giraph-release/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,61] 错误: 找不到符号

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :giraph-core

解决方法：

打开pom.xml文件，定位到profile为hadoop_yarn的位置

<profile>
      <id>hadoop_yarn</id>
      <modules>
        <module>giraph-dist</module>
      </modules>
      <properties>
        <hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
        <munge.symbols>
PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols>
        <!-- TODO: add these checks eventually -->
        <project.enforcer.skip>true</project.enforcer.skip>
        <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
        <giraph.maven.duplicate.finder.skip>true</giraph.maven.duplicate.finder.skip>
      </properties>
      <dependencies>
        <!-- sorted lexicographically -->
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-common</artifactId>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-mapreduce-client-common</artifactId>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-yarn-common</artifactId>
          <version>${hadoop.version}</version>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-yarn-server-common</artifactId>
          <version>${hadoop.version}</version>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
          <version>${hadoop.version}</version>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-yarn-server-nodemanager</artifactId>
          <version>${hadoop.version}</version>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-yarn-server-tests</artifactId>
          <version>${hadoop.version}</version>
          <type>test-jar</type>
        </dependency>
      </dependencies>
    </profile>

将文件中红色部分删除即可。

3. 强制遵循编码规范

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.15:check (default) on project giraph-core: You have 1 Checkstyle violation. -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :giraph-core

解决方法：使用了mvn install 命令进行编译，改成mvn package就行了。

giraph_basedir $ mvn -Phadoop_yarn -Dhadoop.version={指定hadoop版本} -DskipTests clean package

4. 没有指定hbase等的版本？

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Giraph Parent ............................... SUCCESS [ 3.750 s]

[INFO] Apache Giraph Core ................................. SUCCESS [ 49.851 s]

[INFO] Apache Giraph Blocks Framework ..................... SUCCESS [ 21.132 s]

[INFO] Apache Giraph Examples ............................. SUCCESS [ 19.574 s]

[INFO] Apache Giraph Distribution ......................... FAILURE [ 0.024 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 01:34 min

[INFO] Finished at: 2016-08-28T19:54:07+08:00

[INFO] Final Memory: 48M/766M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal on project giraph-dist: Could not resolve dependencies for project org.apache.giraph:giraph-dist:pom:1.2.0-RC0: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project giraph-dist: Could not resolve dependencies for project org.apache.giraph:giraph-dist:pom:1.2.0-RC0: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)

at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)

at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:245)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)

at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)

at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)

at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project org.apache.giraph:giraph-dist:pom:1.2.0-RC0: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:211)

at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)

... 23 more

Caused by: org.eclipse.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)

at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:205)

... 24 more

Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: org.apache.giraph:giraph-hbase:jar:1.2.0-RC0, org.apache.giraph:giraph-accumulo:jar:1.2.0-RC0, org.apache.giraph:giraph-hcatalog:jar:1.2.0-RC0, org.apache.giraph:giraph-gora:jar:1.2.0-RC0, org.apache.giraph:giraph-kibble:jar:1.2.0-RC0, org.apache.giraph:giraph-rexster-io:jar:1.2.0-RC0: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)

... 25 more

Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Failure to find org.apache.giraph:giraph-hbase:jar:1.2.0-RC0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

at org.eclipse.aether.internal.impl.DefaultUpdateCheckManager.newException(DefaultUpdateCheckManager.java:231)

at org.eclipse.aether.internal.impl.DefaultUpdateCheckManager.checkArtifact(DefaultUpdateCheckManager.java:206)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.gatherDownloads(DefaultArtifactResolver.java:585)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:503)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:421)

... 27 more

[ERROR]

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :giraph-dist

解决方法：

好像没有影响，打开giraph-dist目录，目标 jar 已经生成。不过按上面的命令好像不会出现这个问题，不知道当时当时怎么搞的遇到了这个错误。

相关文章

Giraph 源码编译常见错误

Giraph 源码编译命令：

Giraph 编译中常见错误