问题

  多次进行hdfs namenode -format 导致 hadoop/dfs/data/current/VERSION 不一致。

 启动hadoop 集群，发现只有主节点及SecondaryNameNode启动，datanode均为启动。

2019-12-13 14:33:31,532 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 1000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2019-12-13 14:33:31,560 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2019-12-13 14:33:31,632 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
2019-12-13 14:33:31,645 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2019-12-13 14:33:31,656 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2019-12-13 14:33:31,682 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to hadoop-master/10.0.1.73:9000 starting to offer service
2019-12-13 14:33:31,694 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-12-13 14:33:31,697 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2019-12-13 14:33:31,869 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to hadoop-master/10.0.1.73:9000
2019-12-13 14:33:31,871 INFO org.apache.hadoop.hdfs.server.common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1)
2019-12-13 14:33:31,875 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /usr/local/hadoop-2.10.0/data/dfs/data/in_use.lock acquired by nodename 6898@hadoop-slave03
2019-12-13 14:33:31,876 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/usr/local/hadoop-2.10.0/data/dfs/data/
java.io.IOException: Incompatible clusterIDs in /usr/local/hadoop-2.10.0/data/dfs/data: namenode clusterID = CID-b8fab296-6358-4add-a9b7-f7c666976286; datanode clusterID = CID-74e2011a-5429-4635-b86b-1e30728895f3
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:768)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:293)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:409)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:388)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:564)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1659)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1620)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:388)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:282)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:821)
        at java.lang.Thread.run(Thread.java:745)
2019-12-13 14:33:31,881 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid 7eaeeba7-caef-4aa4-961f-1a4f038ca61d) service to hadoop-master/10.0.1.73:9000. Exiting.
java.io.IOException: All specified directories have failed to load.
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:565)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1659)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1620)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:388)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:282)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:821)
        at java.lang.Thread.run(Thread.java:745)
2019-12-13 14:33:31,881 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid 7eaeeba7-caef-4aa4-961f-1a4f038ca61d) service to hadoop-master/10.0.1.73:9000
2019-12-13 14:33:31,882 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid 7eaeeba7-caef-4aa4-961f-1a4f038ca61d)
2019-12-13 14:33:33,882 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2019-12-13 14:33:33,885 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at hadoop-slave03/10.0.1.75
************************************************************/

解决

删除data的current。

再次启动dfs发现问题解决。

[root@hadoop-slave ~]# cd /usr/local/hadoop-2.10.0/
[root@hadoop-slave hadoop-2.10.0]# cd data/dfs/data/
[root@hadoop-slave data]# pwd
/usr/local/hadoop-2.10.0/data/dfs/data
[root@hadoop-slave data]# ls
current
[root@hadoop-slave data]# rm -rf current
[root@hadoop-slave data]#