千家信息网

hadoop 2伪分布式如何搭建

发表于:2025-02-06 作者:千家信息网编辑
千家信息网最后更新 2025年02月06日,小编给大家分享一下hadoop 2伪分布式如何搭建,希望大家阅读完这篇文章之后都有所收获,下面让我们一起去探讨吧!单点伪分布式1.下载hadoop2.安装jdk,设置环境变量export JAVA_H
千家信息网最后更新 2025年02月06日hadoop 2伪分布式如何搭建

小编给大家分享一下hadoop 2伪分布式如何搭建,希望大家阅读完这篇文章之后都有所收获,下面让我们一起去探讨吧!

单点伪分布式

1.下载hadoop

2.安装jdk,设置环境变量

export JAVA_HOME=/usr/local/java/jdk1.7.0_79

export JRE_HOME=$JAVA_HOME/jre

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=./:/$JAVA_HOME/lib:$JAVA_HOME/jre/lib

[root@iZ281cu2lqjZ etc]# source /etc/profile

3.创建用于和组

groupadd hadoop

useradd -g hadoop yarn

useradd -g hadoop hdfs

useradd -g hadoop mapred

4.创建数据和日志目录

mkdir -p /var/data/hadoop/hdfs/nn

mkdir -p /var/data/hadoop/hdfs/snn

mkdir -p /var/data/hadoop/hdfs/dn

chown hdfs:hadoop /var/data/hadoop/hdfs -R

mkdir -p /var/log/hadoop/yarn

chown yarn:hadoop /var/log/hadoop/yarn -R

进入yarnhadoop的目录中

mkdir logs

chmod g+w logs

chown yarn:hadoop . -R

5.配置core-site.xml

fs.default.name

hdfs://localhost:9000

hadoop.http.staticuser.user

hdfs

fs.default.name 是Namenode指定了主机名和请求端口号;

hadoop.http.staticuser.user 是hdfs的默认用户名

6.配置hdfs.site.xml

dfs.replication

1 ---默认为3

dfs.namenode.name.dir

file:/var/data/hadoop/hdfs/nn

fs.checkpoint.dir

file:/var/data/hadoop/hdfs/snn

fs.checkpoint.edit.dir

file:/var/data/hadoop/hdfs/snn

dfs.datanode.data.dir

file:/var/data/hadoop/hdfs/db

7.配置mapred-site.xml

mapreduce.framework.name

yarn

为mapreduce制定框架名,使用 yarn

8 配置yarn-site.xml

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class

org.apache.hadoop.mapred.ShuffleHandler

主要是配置shuffle,默认是没有配置shuffle

9.调整JAVA堆大小

hadoop-env.sh

HADOOP_HEAPSIZE="500"

yarn-env.sh

YARN_HEAPSIZE=500

10.格式化HDFS

切换到hdfs用户,进入到hadoop bin目录执行

./hdfss namenode -format

碰到问题

11../hdfs namenode -format

12.[hdfs@localhost sbin]$ ./hadoop-daemon.sh start namenode

starting namenode, logging to /opt/yarn/hadoop-2.7.1/logs/hadoop-hdfs-namenode-localhost.out

[hdfs@localhost sbin]$ ./hadoop-daemon.sh start secondarynamenode

starting secondarynamenode, logging to /opt/yarn/hadoop-2.7.1/logs/hadoop-hdfs-secondarynamenode-localhost.out

[hdfs@localhost sbin]$ ./hadoop-daemon.sh start datanode

starting datanode, logging to /opt/yarn/hadoop-2.7.1/logs/hadoop-hdfs-datanode-localhost.out

使用JPS可以检查进程

结果:

[hdfs@localhost sbin]$ jps

3915 SecondaryNameNode

3969 DataNode

3833 NameNode

4047 Jps

12.启动yarn

[yarn@localhost sbin]$ ./yarn-daemon.sh start nodemanager

/opt/yarn/hadoop-2.7.1/etc/hadoop/yarn-env.sh: line 121: unexpected EOF while looking for matching `"'

/opt/yarn/hadoop-2.7.1/etc/hadoop/yarn-env.sh: line 124: syntax error: unexpected end of file

starting nodemanager, logging to /opt/yarn/hadoop-2.7.1/logs/yarn-yarn-nodemanager-localhost.out

/opt/yarn/hadoop-2.7.1/etc/hadoop/yarn-env.sh: line 121: unexpected EOF while looking for matching `"'

/opt/yarn/hadoop-2.7.1/etc/hadoop/yarn-env.sh: line 124: syntax error: unexpected end of file

[yarn@localhost sbin]$ jps

4132 ResourceManager

4567 Jps

4456 NodeManager

13 验证

访问: ip:50070

ip:8088

最后可以运行hadoop包中的例子进行检验。这就是伪分布式简单的安装步骤。

问题:如果遇到50070端口不能访问就是 yarn没有成功启动。

去sbin启动yarn ./start-yarn.sh时报

localhost: Error: JAVA_HOME is not set and could not be found.

需要修改 hadoop-env.xml中java_home设置为绝对路径。

重启yarn 解决问题。

配置hbase:

修改hbase-env.sh

修改# export JAVA_HOME=/usr/local/java/jdk1.7.0_79

修改 hbase-site.xml

hbase.rootdir

hdfs://localhost:9000/hbase

dfs.replication

1

hbase.cluster.distributed

true

启动hbase:

[root@iZ281cu2lqjZ bin]# ./start-hbase.sh

root@localhost's password:

localhost: starting zookeeper, logging to /usr/local/hbase/hbase-1.1.4/bin/../logs/hbase-root-zookeeper-iZ281cu2lqjZ.out

starting master, logging to /usr/local/hbase/hbase-1.1.4/bin/../logs/hbase-root-master-iZ281cu2lqjZ.out

starting regionserver, logging to /usr/local/hbase/hbase-1.1.4/bin/../logs/hbase-root-1-regionserver-iZ281cu2lqjZ.out

[root@iZ281cu2lqjZ bin]# jps

1597 DataNode

3180 ResourceManager

3463 NodeManager

1462 NameNode

8680 HRegionServer

1543 SecondaryNameNode

8536 HQuorumPeer

8597 HMaster

8729 Jps

完成

看完了这篇文章,相信你对"hadoop 2伪分布式如何搭建"有了一定的了解,如果想了解更多相关知识,欢迎关注行业资讯频道,感谢各位的阅读!

0