怎么在CentOS7中编译Hadoop2.x

这篇文章主要介绍"怎么在CentOS7中编译Hadoop2.x"，在日常操作中，相信很多人在怎么在CentOS7中编译Hadoop2.x问题上存在疑惑，小编查阅了各式资料，整理出简单好用的操作方法，希望对大家解答"怎么在CentOS7中编译Hadoop2.x"的疑惑有所帮助！接下来，请跟着小编一起来学习吧！

环境

VirtualBox-5.0, CentOS-7.1, Hadoop-2.7.1

1、关于Hadoop本地库

1.1、可能存在的问题

Hadoop发行包自带的库文件可能与本地操作系统不兼容。特别是在Hadoop-2.5.1之前的版本，自带的本地库都是32位的，64位操作系统无法使用，只能使用java自带的，而java自带的速度会比较慢。

查看本地库版本

[wukong@bd01 native]$ pwd/home/wukong/local/hadoop-2.7.1/lib/native[wukong@bd01 native]$ lltotal 4928-rw-r--r-- 1 wukong wukong 1383476 Jun 29 14:15 libhadoop.a-rw-r--r-- 1 wukong wukong 1634592 Jun 29 14:15 libhadooppipes.alrwxrwxrwx 1 wukong wukong      18 Jun 29 14:15 libhadoop.so -> libhadoop.so.1.0.0-rwxr-xr-x 1 wukong wukong  807942 Jun 29 14:15 libhadoop.so.1.0.0-rw-r--r-- 1 wukong wukong  476666 Jun 29 14:15 libhadooputils.a-rw-r--r-- 1 wukong wukong  447388 Jun 29 14:15 libhdfs.alrwxrwxrwx 1 wukong wukong      16 Jun 29 14:15 libhdfs.so -> libhdfs.so.0.0.0-rwxr-xr-x 1 wukong wukong  282388 Jun 29 14:15 libhdfs.so.0.0.0[wukong@bd01 native]$ file ./libhadoop.so.1.0.0./libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xd3720bf1ab8325bdde6e129bc386a86c888accc9, not stripped[wukong@bd01 native]$

1.2、可能存在的WARN级别日志

启停hdfs/mapreduce程序后可能出现的告警日志

2015-08-01 18:15:57,940 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2、编译过程

2.1、安装编译相关的包（需联网）

[root@bd01 ~]# yum install svn[root@bd01 ~]# yum install autoconfautomake libtool cmake[root@bd01 ~]# yum install ncurses-devel[root@bd01 ~]# yum install openssl-devel[root@bd01 ~]# yum install gcc*

2.2、安装maven

# 到apache官网下载最新稳定版wget # 解压tar -xzvf apache-maven-3.3.3-bin.tar.gz  # 放到/opt下面mv apache-maven-3.3.3 /opt# 配置环境变量export JAVA_HOME=/opt/jdk1.7.0_79export M2_HOME=/opt/apache-maven-3.3.3export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$PATH:$JAVA_HOME/bin:$M2_HOME/binexport MAVEN_OPTS=-Xmx512m# 更新环境变量[root@bd01 apache-maven-3.3.3]# source /etc/profile# 检查是否配置成功[root@bd01 apache-maven-3.3.3]# echo $PATH/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jdk1.7.0_79/bin:/home/wukong/local/hadoop-2.7.1/bin:/home/wukong/local/hadoop-2.7.1/sbin:/opt/jdk1.7.0_79/bin:/opt/apache-maven-3.3.3/bin[root@bd01 apache-maven-3.3.3]# mvn -versionApache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T19:57:37+08:00)Maven home: /opt/apache-maven-3.3.3Java version: 1.7.0_79, vendor: Oracle CorporationJava home: /opt/jdk1.7.0_79/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "3.10.0-229.7.2.el7.x86_64", arch: "amd64", family: "unix"[root@bd01 apache-maven-3.3.3]# exitexit[wukong@bd01 pack]$ mvn -version-bash: mvn: command not found[wukong@bd01 pack]$ source /etc/profile[wukong@bd01 pack]$ mvn -version       Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T19:57:37+08:00)Maven home: /opt/apache-maven-3.3.3Java version: 1.7.0_79, vendor: Oracle CorporationJava home: /opt/jdk1.7.0_79/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "3.10.0-229.7.2.el7.x86_64", arch: "amd64", family: "unix"[wukong@bd01 pack]$

注意用户权限

root@bd01 apache-maven-3.3.3]# pwd/opt/apache-maven-3.3.3[root@bd01 apache-maven-3.3.3]# cd ..[root@bd01 opt]# lltotal 4drwxr-xr-x 6 wukong wukong   92 Aug  2 15:11 apache-maven-3.3.3drwxr-xr-x 8     10    143 4096 Apr 11 02:53 jdk1.7.0_79[root@bd01 opt]# cd apache-maven-3.3.3/[root@bd01 apache-maven-3.3.3]# lltotal 32drwxr-xr-x 2 wukong wukong    91 Aug  2 15:11 bindrwxr-xr-x 2 wukong wukong    41 Aug  2 15:11 bootdrwxr-xr-x 3 wukong wukong    60 Apr 22 19:55 confdrwxr-xr-x 3 wukong wukong  4096 Aug  2 15:11 lib-rwxr-xr-x 1 wukong wukong 19091 Apr 22 19:58 LICENSE-rwxr-xr-x 1 wukong wukong   182 Apr 22 19:58 NOTICE-rwxr-xr-x 1 wukong wukong  2541 Apr 22 19:55 README.txt[root@bd01 apache-maven-3.3.3]#

2.3、安装protobuf

# 下载[wukong@bd01 pack]$ wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz# 解压[wukong@bd01 pack]$ tar -xzvf ./protobuf-2.5.0.tar.gz # 编译及安装[wukong@bd01 pack]$ cd protobuf-2.5.0[wukong@bd01 protobuf-2.5.0]$ ./configure # 以下三步时间比较长，而且有warning[wukong@bd01 protobuf-2.5.0]$ make[wukong@bd01 protobuf-2.5.0]$ make check# make install时要使用root账户，否则会因为权限的问题报错[root@bd01 protobuf-2.5.0]$ make install# 验证[root@bd01 lib]# protoc --versionlibprotoc 2.5.0[wukong@bd01 ~]$ protoc --versionlibprotoc 2.5.0

make check后可能会得到类似以下结果，检查看看是否有error，没有就ok

PASS: protobuf-testPASS: protobuf-lazy-descriptor-testPASS: protobuf-lite-testPASS: google/protobuf/compiler/zip_output_unittest.shPASS: google/protobuf/io/gzip_stream_unittest.shmake[5]: Entering directory `/home/wukong/pack/protobuf-2.5.0/src'make  all-ammake[6]: Entering directory `/home/wukong/pack/protobuf-2.5.0/src'make[6]: Nothing to be done for `all-am'.make[6]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src'make[5]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src'============================================================================Testsuite summary for Protocol Buffers 2.5.0============================================================================# TOTAL: 5# PASS:  5# SKIP:  0# XFAIL: 0# FAIL:  0# XPASS: 0# ERROR: 0============================================================================make[4]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src'make[3]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src'make[2]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src'make[1]: Leaving directory `/home/wukong/pack/protobuf-2.5.0/src'

注意，和hadoop-2.7.1对应的protobuf版本是2.5.0，如果版本不对，会报错大致如下：

[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 47.107 s[INFO] Finished at: 2015-08-02T16:59:47+08:00[INFO] Final Memory: 53M/135M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.1:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 2.6.1', expected version is '2.5.0' -> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn  -rf :hadoop-common

2.4、重新编译本地库

2.4.1、获取hadoop源代码

wget或者svn或者直接去官网下

[wukong@bd01 pack]$ lltotal 233916-rw-rw-r--  1 wukong wukong   8042383 Apr 28 23:12 apache-maven-3.3.3-bin.tar.gzdrwxr-xr-x 15 wukong wukong      4096 Jun 29 14:04 hadoop-2.7.1-src-rw-rw-r--  1 wukong wukong  18226773 Jul 30 22:57 hadoop-2.7.1-src.tar.gz-rw-rw-r--  1 wukong wukong 210606807 Jul 28 09:49 hadoop-2.7.1.tar.gzdrwxr-xr-x 10 wukong wukong      4096 Aug  2 15:28 protobuf-2.6.1-rw-rw-r--  1 wukong wukong   2641426 Oct 23  2014 protobuf-2.6.1.tar.gz

2.4.1、解压

[wukong@bd01 pack]# tar -xzvf hadoop-2.7.1-src.tar.gz [wukong@bd01 pack]# cd hadoop-2.7.1-src[wukong@bd01 hadoop-2.7.1-src]$ pwd/home/wukong/pack/hadoop-2.7.1-src[wukong@bd01 hadoop-2.7.1-src]$ lltotal 76-rw-r--r--  1 wukong wukong 13110 Jun 29 14:03 BUILDING.txtdrwxr-xr-x  2 wukong wukong  4096 Aug  2 16:06 dev-supportdrwxr-xr-x  3 wukong wukong    30 Aug  2 16:06 hadoop-assembliesdrwxr-xr-x  2 wukong wukong    20 Aug  2 16:06 hadoop-clientdrwxr-xr-x 10 wukong wukong  4096 Aug  2 16:06 hadoop-common-projectdrwxr-xr-x  2 wukong wukong    20 Aug  2 16:07 hadoop-distdrwxr-xr-x  6 wukong wukong   103 Aug  2 16:06 hadoop-hdfs-projectdrwxr-xr-x  9 wukong wukong  4096 Aug  2 16:06 hadoop-mapreduce-projectdrwxr-xr-x  3 wukong wukong    30 Aug  2 16:06 hadoop-maven-pluginsdrwxr-xr-x  2 wukong wukong    20 Aug  2 16:06 hadoop-miniclusterdrwxr-xr-x  3 wukong wukong    30 Aug  2 16:07 hadoop-projectdrwxr-xr-x  2 wukong wukong    37 Aug  2 16:06 hadoop-project-distdrwxr-xr-x 16 wukong wukong  4096 Aug  2 16:06 hadoop-toolsdrwxr-xr-x  3 wukong wukong    56 Aug  2 16:06 hadoop-yarn-project-rw-r--r--  1 wukong wukong 15429 Jun 29 14:03 LICENSE.txt-rw-r--r--  1 wukong wukong   101 Jun 29 14:03 NOTICE.txt-rw-r--r--  1 wukong wukong 18893 Jun 29 14:03 pom.xml-rw-r--r--  1 wukong wukong  1366 Jun 29 14:03 README.txt[wukong@bd01 hadoop-2.7.1-src]$

2.4.2、编译hadoop源代码

# 非常漫长非常漫长的等待[wukong@bd01 hadoop-2.7.1-src]$ mvn clean package -Pdist,native -DskipTests -Dtar# 最终编译结果[INFO] ------------------------------------------------------------------------[INFO] Reactor Summary:[INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [  7.458 s][INFO] Apache Hadoop Project POM .......................... SUCCESS [  5.576 s][INFO] Apache Hadoop Annotations .......................... SUCCESS [  6.810 s][INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.343 s][INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  3.293 s][INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  5.637 s][INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  6.570 s][INFO] Apache Hadoop Auth ................................. SUCCESS [ 14.101 s][INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.555 s][INFO] Apache Hadoop Common ............................... SUCCESS [03:54 min][INFO] Apache Hadoop NFS .................................. SUCCESS [ 19.327 s][INFO] Apache Hadoop KMS .................................. SUCCESS [ 22.779 s][INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.053 s][INFO] Apache Hadoop HDFS ................................. SUCCESS [08:56 min][INFO] Apache Hadoop HttpFS ............................... SUCCESS [02:19 min][INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [01:36 min][INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  9.181 s][INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.065 s][INFO] hadoop-yarn ........................................ SUCCESS [  0.076 s][INFO] hadoop-yarn-api .................................... SUCCESS [02:27 min][INFO] hadoop-yarn-common ................................. SUCCESS [03:22 min][INFO] hadoop-yarn-server ................................. SUCCESS [  0.564 s][INFO] hadoop-yarn-server-common .......................... SUCCESS [ 22.455 s][INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 40.593 s][INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  8.402 s][INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 20.187 s][INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 41.425 s][INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 13.118 s][INFO] hadoop-yarn-client ................................. SUCCESS [ 16.388 s][INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  7.473 s][INFO] hadoop-yarn-applications ........................... SUCCESS [  0.205 s][INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  5.137 s][INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  3.379 s][INFO] hadoop-yarn-site ................................... SUCCESS [  0.192 s][INFO] hadoop-yarn-registry ............................... SUCCESS [ 11.037 s][INFO] hadoop-yarn-project ................................ SUCCESS [  5.957 s][INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.084 s][INFO] hadoop-mapreduce-client-core ....................... SUCCESS [01:05 min][INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 44.305 s][INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  8.763 s][INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 20.268 s][INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 17.986 s][INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 24.200 s][INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  3.840 s][INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 12.104 s][INFO] hadoop-mapreduce ................................... SUCCESS [  4.579 s][INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 21.775 s][INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [01:16 min][INFO] Apache Hadoop Archives ............................. SUCCESS [ 11.316 s][INFO] Apache Hadoop Rumen ................................ SUCCESS [ 27.178 s][INFO] Apache Hadoop Gridmix .............................. SUCCESS [  8.318 s][INFO] Apache Hadoop Data Join ............................ SUCCESS [  5.153 s][INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  5.326 s][INFO] Apache Hadoop Extras ............................... SUCCESS [  7.849 s][INFO] Apache Hadoop Pipes ................................ SUCCESS [ 27.793 s][INFO] Apache Hadoop OpenStack support .................... SUCCESS [  9.929 s][INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 57.588 s][INFO] Apache Hadoop Azure support ........................ SUCCESS [ 21.386 s][INFO] Apache Hadoop Client ............................... SUCCESS [ 20.907 s][INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 20.113 s][INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 39.010 s][INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 15.505 s][INFO] Apache Hadoop Tools ................................ SUCCESS [  0.051 s][INFO] Apache Hadoop Distribution ......................... SUCCESS [07:26 min][INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 44:52 min[INFO] Finished at: 2015-08-03T00:37:02+08:00[INFO] Final Memory: 134M/492M[INFO] ------------------------------------------------------------------------

2.4.2、编译后的成果

这里只看了native库文件

[wukong@bd01 native]$ pwd/home/wukong/pack/hadoop-2.7.1-src/hadoop-dist/target/hadoop-2.7.1/lib/native[wukong@bd01 native]$ lltotal 4644-rw-rw-r-- 1 wukong wukong 1235998 Aug  3 00:30 libhadoop.a-rw-rw-r-- 1 wukong wukong 1606616 Aug  3 00:31 libhadooppipes.alrwxrwxrwx 1 wukong wukong      18 Aug  3 00:30 libhadoop.so -> libhadoop.so.1.0.0-rwxrwxr-x 1 wukong wukong  724522 Aug  3 00:30 libhadoop.so.1.0.0-rw-rw-r-- 1 wukong wukong  471882 Aug  3 00:31 libhadooputils.a-rw-rw-r-- 1 wukong wukong  433780 Aug  3 00:30 libhdfs.alrwxrwxrwx 1 wukong wukong      16 Aug  3 00:30 libhdfs.so -> libhdfs.so.0.0.0-rwxrwxr-x 1 wukong wukong  272202 Aug  3 00:30 libhdfs.so.0.0.0[wukong@bd01 native]$

可以把库文件拷贝到hadoop的相关目录中，或者直接使用用编译好的hadoop

到此，关于"怎么在CentOS7中编译Hadoop2.x"的学习就结束了，希望能够解决大家的疑惑。理论与实践的搭配能更好的帮助大家学习，快去试试吧！若想继续学习更多相关知识，请继续关注网站，小编会继续努力为大家带来更多实用的文章！