导航：首页 > 服务器 >

hadoop-006完全分布式问题有哪些

发表于：2025-02-07 作者：千家信息网编辑

千家信息网最后更新 2025年02月07日，小编给大家分享一下hadoop-006完全分布式问题有哪些，希望大家阅读完这篇文章之后都有所收获，下面让我们一起去探讨吧！一、运行mapreduce 任务，但是在yarn里看不到任务。bin/yarn

千家信息网最后更新 2025年02月07日hadoop-006完全分布式问题有哪些

小编给大家分享一下hadoop-006完全分布式问题有哪些，希望大家阅读完这篇文章之后都有所收获，下面让我们一起去探讨吧！

一、运行mapreduce 任务，但是在yarn里看不到任务。

bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /test /out1

我配置的resourceManager 为 http://192.168.31.136:8088

原因是在 mapred-site.xml 没有配置

mapreduce.framework.name
yarn

二、配置好问题一后，运行mr任务提示The auxService:mapreduce_shuffle does not exist

原因是在yarn-site.xml 没有配置yarn.nodemanager.aux-services 节点

yarn.nodemanager.aux-services
mapreduce_shuffle

报错如下

16/11/29 23:10:45 INFO mapreduce.Job: Task Id : attempt_1480432102879_0001_m_000000_2, Status : FAILED
Container launch failed for container_e02_1480432102879_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

-----------------------------------------------------------------------------------------------------------------

配置文件记录

已提前搭建好三个节点的zookeeper 小集群，配置可以实现 HDFS　ＨＡ和　ＹＡＲＮ　ＨＡ

一、hadoop-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

二、yarn-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

三、core-site.xml

fs.defaultFS
hdfs://mycluster

dfs.journalnode.edits.dir
/home/jxlgzwh/hadoop-2.7.2/data/jn

hadoop.tmp.dir
/home/jxlgzwh/hadoop-2.7.2/data/tmp

ha.zookeeper.quorum
master:2181,slave01:2181,slave02:2181

四、hdfs-site.xml

dfs.nameservices
mycluster

dfs.ha.namenodes.mycluster
nn1,nn2

dfs.namenode.rpc-address.mycluster.nn1
master:8020

dfs.namenode.rpc-address.mycluster.nn2
slave01:8020

dfs.namenode.http-address.mycluster.nn1
master:50070

dfs.namenode.http-address.mycluster.nn2
slave01:50070

dfs.namenode.shared.edits.dir
qjournal://master:8485;slave01:8485;slave02:8485/mycluster

dfs.client.failover.proxy.provider.mycluster
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.ha.fencing.methods
sshfence

dfs.ha.fencing.ssh.private-key-files
/home/jxlgzwh/.ssh/id_dsa

dfs.replication
3

dfs.permissions.enabled
false

dfs.ha.automatic-failover.enabled
true

五、mapred-site.xml

mapreduce.framework.name
yarn

六、slaves

192.168.31.136
192.168.31.130
192.168.31.229

七、yarn-site.xml

yarn.resourcemanager.ha.enabled
true

yarn.resourcemanager.cluster-id
cluster1

yarn.resourcemanager.ha.rm-ids
rm1,rm2

yarn.resourcemanager.hostname.rm1
master

yarn.resourcemanager.hostname.rm2
slave01

yarn.resourcemanager.webapp.address.rm1
master:8088

yarn.resourcemanager.webapp.address.rm2
slave01:8088

yarn.resourcemanager.zk-address
master:2181,slave01:2181,slave02:2181

yarn.resourcemanager.recovery.enabled
true

yarn.resourcemanager.zk-state-store.parent-path
/rmstore

yarn.resourcemanager.store.class
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

yarn.nodemanager.recovery.enabled
true

yarn.nodemanager.address
0.0.0.0:45454

yarn.nodemanager.recovery.dir
/home/jxlgzwh/hadoop-2.7.2/data/tmp/yarn-nm-recovery

yarn.nodemanager.aux-services
mapreduce_shuffle