centos7 安装 hadoop-cdh-2.5 on mesos

1.环境说明

master 192.168.0.223 mesos-master

slave 192.168.0.225 mesos-salve

2.环境准备

关闭防火墙

关闭selinux

两台机器修改主机名master/slave

设置hosts，能互相解析

3.master和slave配置ssh互信

这里配置hadoop用户双机互信，因为hadoop用hadoop用户启动

master   yum  -y install  sshpass  ssh-keygen 一路回车  ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@192.168.0.220slave  yum  -y install  sshpass  ssh-keygen 一路回车  ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@192.168.0.201测试ssh对方主机，不提示输入密码则OK

4.安装JDK

tar zxvf jdk-8u65-linux-x64.tar.gzmv jdk1.8.0_65  /usr/jdk

4.1设置环境变量

export JAVA_HOME=/usr/jdkexport JRE_HOME=/usr/jdk/jreexport CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin执行 source  /etc/profile

4.2测试JDK

java -version #出现版本信息

5.安装mesos主从，看其他博客

安装完成，会在/usr/local/lib下生成一个libmesos.so文件

6.安装和配置Hadoop

master和slave

tar zxvf hadoop-2.5.0-cdh6.4.8.tar.gzmv hadoop-2.5.0-cdh6.4.8  /usr/hadoopcd  /usr/hadoopmkdir -p tmpcd /usr/hadoop/mv bin bin-mapreduce2/ ln -s bin-mapreduce1  binmv example  example-mapreduce2 ln -s example-mapreduce1  examplecd etc/mv hadoop hadoop-mapreduce2ln -s hadoop-mapreduce1 hadoop

7.添加hadoop环境变量

vim /etc/profile export HADOOP_HOME=/usr/hadoop export PATH=$PATH:$HADOOP_HOME:$HADOOP_HOME/binsource /etc/profile

8.获取hadoop on mesos的jar包

yum -y install maven openjdk-7-jdk git git clone cd hadoop mvn package   #获取jar包,jar包会在target下

9.把获取的jar包放到hadoop安装目录下

master和slave

cp hadoop/target/hadoop-mesos-0.1.0.jar  /usr/hadoop/share/hadoop/common/lib/

10.配置hadoop on mesos

master和slave

vim  /usr/hadoop/etc/hadoop/mapred.site.xml  mapred.job.tracker  localhost:9001  mapred.jobtracker.taskScheduler  org.apache.hadoop.mapred.MesosScheduler  mapred.mesos.taskScheduler  org.apache.hadoop.mapred.JobQueueTaskScheduler  mapred.mesos.master  zk://192.168.0.223  mapred.mesos.executor.uri  hdfs://localhost:9000/hadoop-2.5.0-cdh6.2.0.tar.gz

11.给hadoop用户权限

master和slave

chown -R  hadoop:hadoop /usr/hadoop

12.在master上启动jobtracker，连接mesos

su hadoop MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so hadoop jobtracker

13.测试

输入 192.168.0.223:5050 看看框架里有没有hadoop

----------------------------------------------------------------------------------------

二、搭建完HDFS，搭建hadoop on mesos

按照搭建hadoop文档配置完HDFS，需要配置core-site.xml，hdfs-site.xml，mapred-site.xml

mv bin  bin-mapreduce2ln -s   bin-mapreduce1  bin#不用移动hadoop，搭建hadoop on  mesos 的时候直接在hadoop里修改

2.移动hdfs命令和start-dfs.sh

cd /usr/hadoop/bin-mapreduce2 cp  hdfs  /usr/hadoop/bin-mapreduce1cd /usr/hadoop/sbincp start-dfs.sh /sur/hadoop/bin-mapreduce1

3.搭建hadoop on mesos

修改mapred-site.xml配置文件

  mapred.job.tracker  localhost:9002  #改为9002 避免和hdfs-site.xml中的端口冲突  mapred.jobtracker.taskScheduler  org.apache.hadoop.mapred.MesosScheduler  mapred.mesos.taskScheduler  org.apache.hadoop.mapred.JobQueueTaskScheduler  mapred.mesos.master  zk://192.168.0.223  mapred.mesos.executor.uri  hdfs://localhost:9000/hadoop-2.5.0-cdh6.2.0.tar.gz

4.格式化和启动HDFS

hdfs namenode  -foramt start-dfs.sh

5.启动hadoop on mesos

su hadoop MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so hadoop jobtracker