hadoop2.7.0集群在使用中遇到的bug及解决办法是什么

hadoop2.7.0集群在使用中遇到的bug及解决办法是什么，很多新手对此不是很清楚，为了帮助大家解决这个难题，下面小编将为大家详细讲解，有这方面需求的人可以来学习下，希望你能有所收获。

hadoop环境是2.7.0的集群环境，使用sqoop 1.4.6执行从mysql向hive的数据导入。

执行过程中报错，如下方的日志信息。但是查询hive中的数据，发现实际数据已经过来了，但因为mysql的数据表较多，不能一一对应的比对一下。所以为了确保同步数据成功，需要重新的正确的执行一次。

15/09/28 10:22:01 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /data/hadoop/share/hadoop/mapreduceNote: /tmp/sqoop-hadoop/compile/60bb7ee51d4794512d28b8efc4029fbc/QueryResult.java uses or overrides a deprecated API.Note: Recompile with -Xlint:deprecation for details.15/09/28 10:22:06 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/60bb7ee51d4794512d28b8efc4029fbc/QueryResult.jar15/09/28 10:22:09 INFO tool.ImportTool: Destination directory /tmp/wfpuser_t0301 is not present, hence not deleting.15/09/28 10:22:09 INFO mapreduce.ImportJobBase: Beginning query import.15/09/28 10:22:09 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address15/09/28 10:22:09 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar15/09/28 10:22:09 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps15/09/28 10:22:09 INFO client.RMProxy: Connecting to ResourceManager at zhebuduan-bd-3/192.168.1.113:803215/09/28 10:22:15 INFO db.DBInputFormat: Using read commited transaction isolation15/09/28 10:22:15 INFO mapreduce.JobSubmitter: number of splits:115/09/28 10:22:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1443364253801_031015/09/28 10:22:17 INFO impl.YarnClientImpl: Submitted application application_1443364253801_031015/09/28 10:22:18 INFO mapreduce.Job: The url to track the job: http://zhebuduan-bd-3:8088/proxy/application_1443364253801_0310/15/09/28 10:22:18 INFO mapreduce.Job: Running job: job_1443364253801_031015/09/28 10:22:31 INFO mapreduce.Job: Job job_1443364253801_0310 running in uber mode : false15/09/28 10:22:31 INFO mapreduce.Job:  map 0% reduce 0/09/28 10:22:34 INFO mapreduce.Job: Task Id : attempt_1443364253801_0310_m_000000_0, Status : FAILEDRename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/60java.io.IOException: Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/60        at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735)        at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236)        at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678)        at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958)        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)        at java.util.concurrent.FutureTask.run(FutureTask.java:262)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)        at java.util.concurrent.FutureTask.run(FutureTask.java:262)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)        at java.lang.Thread.run(Thread.java:745)15/09/28 10:22:38 INFO mapreduce.Job: Task Id : attempt_1443364253801_0310_m_000000_1, Status : FAILEDRename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/62java.io.IOException: Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/62        at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735)        at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236)        at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678)        at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958)        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)        at java.util.concurrent.FutureTask.run(FutureTask.java:262)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)        at java.util.concurrent.FutureTask.run(FutureTask.java:262)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)        at java.lang.Thread.run(Thread.java:745)15/09/28 10:22:42 INFO mapreduce.Job:  map 100% reduce 0/09/28 10:22:42 INFO mapreduce.Job: Task Id : attempt_1443364253801_0310_m_000000_2, Status : FAILEDRename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/64java.io.IOException: Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/64        at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735)        at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236)        at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678)        at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958)        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)        at java.util.concurrent.FutureTask.run(FutureTask.java:262)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)        at java.util.concurrent.FutureTask.run(FutureTask.java:262)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)        at java.lang.Thread.run(Thread.java:745)15/09/28 10:22:43 INFO mapreduce.Job:  map 0% reduce 0/09/28 10:23:00 INFO mapreduce.Job:  map 100% reduce 0/09/28 10:23:00 INFO mapreduce.Job: Job job_1443364253801_0310 completed successfully15/09/28 10:23:00 INFO mapreduce.Job: Counters: 31        File System Counters                FILE: Number of bytes read=0                FILE: Number of bytes written=140349                FILE: Number of read operations=0                FILE: Number of large read operations=0                FILE: Number of write operations=0                HDFS: Number of bytes read=87                HDFS: Number of bytes written=3712573                HDFS: Number of read operations=4                HDFS: Number of large read operations=0                HDFS: Number of write operations=2        Job Counters                 Failed map tasks=3                Launched map tasks=4                Other local map tasks=4                Total time spent by all maps in occupied slots (ms)=20017                Total time spent by all reduces in occupied slots (ms)=0                Total time spent by all map tasks (ms)=20017                Total vcore-seconds taken by all map tasks=20017                Total megabyte-seconds taken by all map tasks=20497408        Map-Reduce Framework                Map input records=12661                Map output records=12661                Input split bytes=87                Spilled Records=0                Failed Shuffles=0                Merged Map outputs=0                GC time elapsed (ms)=177                CPU time spent (ms)=8810                Physical memory (bytes) snapshot=175165440                Virtual memory (bytes) snapshot=880988160                Total committed heap usage (bytes)=197132288        File Input Format Counters                 Bytes Read=0        File Output Format Counters                 Bytes Written=3712573

在网上查了一下问题原因，最终找到一个解决办法，删除/data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache目录下的缓存文件，直接进入该目录，执行rm -rf *，本来想备份一下，tar命令打包怎么也执行不完，所以直接删掉了。注意在集群关闭的时候执行。启动集群后，操作不再报错。

但是还有个问题，执行hadoop dfsadmin -report后，提示datanode节点的状态为

Decommission Status : NormalConfigured Capacity: 1055816155136 (983.31 GB)DFS Used: 267768670295 (249.38 GB)Non DFS Used: 59758983081 (55.65 GB)DFS Remaining: 728288501760 (678.27 GB)DFS Used%: 25.36%DFS Remaining%: 68.98%Configured Cache Capacity: 0 (0 B)Cache Used: 0 (0 B)Cache Remaining: 0 (0 B)Cache Used%: 100.00%Cache Remaining%: 0.00%Xceivers: 10Last contact: Mon Sep 28 15:43:20 CST 2015

cache used和cache remaining都是0了，这个不知道怎么搞定，虽然当前没看出什么问题，看着膈应啊。谁知道帮个忙告诉我吧~

我在官网找到了这个bug，在2.7.1版本中已经修复了这个bug，对集群进行升级。

看完上述内容是否对您有帮助呢？如果还想对相关知识有进一步的了解或阅读更多相关文章，请关注行业资讯频道，感谢您对的支持。