메뉴 건너뛰기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


./hadoop-daemon.sh start namenode로 namenode기동시 EditLog의 custerId, namespaceId가 달라서 오류가 발생하면
"bin/hadoop namenode -recover"를 실행하면 문제를 해결해준다.

recover를 실행하면 아래의 짊문을 하는데 s를 선택한다.
Enter 'c' to continue, skipping the bad section in the log
Enter 's' to stop reading the edit log here, abandoning any later edits
Enter 'q' to quit without saving
Enter 'a' to always select the first choice in the future without prompting. (c/s/q/a)



-------------------------오류내용-----------------
16/09/24 10:57:03 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
16/09/24 10:57:03 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in 0 seconds.
16/09/24 10:57:03 INFO namenode.FSImage: Loaded image for txid 0 from /engine/bigdata/hadoop/dfs/namenode/current/fsimage_                                              0000000000000000000
16/09/24 10:57:03 INFO namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@830730                                              f8 expecting start txid #1
16/09/24 10:57:03 INFO namenode.FSImage: Start loading edits file http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTx                                              Id=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP04:8480/getJournal?jid=myc                                              luster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
16/09/24 10:57:03 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP03:8480/getJournal?jid=mycluster                                              &segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP04:8480/getJourn                                              al?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction                                               ID 1
16/09/24 10:57:03 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP03:8480/getJournal?jid=mycluster                                              &segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:03 ERROR namenode.EditLogInputStream: caught exception initializing http://IUDGTMP03:8480/getJournal?jid=my                                              cluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://IUDGTMP03:8480/getJournal?j                                              id=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35 failed with status                                               code 403
Response message:
This node has namespaceId '671989181 and clusterId 'CID-8395f825-0009-4668-862b-3a9e8d184b54' but the requesting node expe                                              cted '1812369513' and 'CID-732a108d-6a42-44e4-9b2a-313750cc4c35'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:468)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(AccessController.java:377)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java                                              :455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:03 ERROR namenode.EditLogInputStream: Got error reading edit log input stream http://IUDGTMP03:8480/getJour                                              nal?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35; failing over                                               to edit log http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6                                              a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txi                                              d 0; expected file to go up to 2
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:03 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP04:8480/getJournal?jid=mycluster                                              &segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:03 ERROR namenode.EditLogInputStream: caught exception initializing http://IUDGTMP04:8480/getJournal?jid=my                                              cluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://IUDGTMP04:8480/getJournal?j                                              id=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35 failed with status                                               code 403
Response message:
This node has namespaceId '671989181 and clusterId 'CID-8395f825-0009-4668-862b-3a9e8d184b54' but the requesting node expe                                              cted '1812369513' and 'CID-732a108d-6a42-44e4-9b2a-313750cc4c35'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:468)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(AccessController.java:377)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java                                              :455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:03 ERROR namenode.FSImage: Error replaying edit log at offset 0.  Expected transaction ID was 1
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txi                                              d 0; expected file to go up to 2
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:03 ERROR namenode.MetaRecoveryContext: We failed to read txId 1

Enter 'c' to continue, skipping the bad section in the log
Enter 's' to stop reading the edit log here, abandoning any later edits
Enter 'q' to quit without saving
Enter 'a' to always select the first choice in the future without prompting. (c/s/q/a)
s
16/09/24 10:57:52 WARN namenode.MetaRecoveryContext: Stopped reading edit log at 0/-1
16/09/24 10:57:52 INFO namenode.FSImage: Edits file http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACI                  D-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6                  a42-44e4-9b2a-313750cc4c35 of size -1 edits # 0 loaded in 49 seconds
16/09/24 10:57:52 INFO namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@fc20cc3e expecting start txid #3
16/09/24 10:57:52 INFO namenode.FSImage: Start loading edits file http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A18123                  69513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A1812369513%3A0%3A                  CID-732a108d-6a42-44e4-9b2a-313750cc4c35
16/09/24 10:57:52 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-                  63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A1812369                  513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:52 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-                  63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:52 ERROR namenode.EditLogInputStream: caught exception initializing http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storag                  eInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&s                  torageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35 failed with status code 403
Response message:
This node has namespaceId '671989181 and clusterId 'CID-8395f825-0009-4668-862b-3a9e8d184b54' but the requesting node expected '1812369513' and 'CID-7                  32a108d-6a42-44e4-9b2a-313750cc4c35'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:468)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(AccessController.java:377)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:52 ERROR namenode.EditLogInputStream: Got error reading edit log input stream http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxI                  d=3&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35; failing over to edit log http://IUDGTMP03:8480/getJournal?jid=myclust                  er&segmentTxId=3&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up                   to 127
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:52 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-                  63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:52 ERROR namenode.EditLogInputStream: caught exception initializing http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storag                  eInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&s                  torageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35 failed with status code 403
Response message:
This node has namespaceId '671989181 and clusterId 'CID-8395f825-0009-4668-862b-3a9e8d184b54' but the requesting node expected '1812369513' and 'CID-7                  32a108d-6a42-44e4-9b2a-313750cc4c35'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:468)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(AccessController.java:377)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:52 ERROR namenode.FSImage: Error replaying edit log at offset 0.  Expected transaction ID was 3
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up                   to 127
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:52 ERROR namenode.MetaRecoveryContext: We failed to read txId 3

Enter 'c' to continue, skipping the bad section in the log
Enter 's' to stop reading the edit log here, abandoning any later edits
Enter 'q' to quit without saving
Enter 'a' to always select the first choice in the future without prompting. (c/s/q/a)
s
16/09/24 10:57:54 WARN namenode.MetaRecoveryContext: Stopped reading edit log at 0/-1
16/09/24 10:57:54 INFO namenode.FSImage: Edits file http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A1812369513%3A0%3ACI                  D-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6                  a42-44e4-9b2a-313750cc4c35 of size -1 edits # 0 loaded in 1 seconds
16/09/24 10:57:54 INFO namenode.FSNamesystem: Need to save fs image? false (staleImage=true, haEnabled=true, isRollingUpgrade=false)
16/09/24 10:57:54 INFO namenode.NameCache: initialized with 0 entries 0 lookups
16/09/24 10:57:54 INFO namenode.FSNamesystem: Finished loading FSImage in 52039 msecs
16/09/24 10:57:54 INFO namenode.FSImage: Save namespace ...
16/09/24 10:57:54 INFO namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 0
16/09/24 10:57:55 INFO namenode.MetaRecoveryContext: RECOVERY COMPLETE
16/09/24 10:57:55 INFO namenode.FSNamesystem: Stopping services started for active state
16/09/24 10:57:55 INFO namenode.FSNamesystem: Stopping services started for standby state
16/09/24 10:57:55 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at IUDGTMP02/XXX.XXX.XXX.32
************************************************************/

번호 제목 글쓴이 날짜 조회 수
45 hadoop에서 yarn jar ..를 이용하여 appliction을 실행하여 정상적으로 수행되었으나 yarn UI의 어플리케이션 목록에 나타나지 않는 문제 총관리자 2017.05.02 28
44 hadoop에서 yarn jar ..를 이용하여 appliction을 실행하여 정상적으로 수행되었으나 yarn UI의 어플리케이션 목록에 나타나지 않는 문제 총관리자 2017.05.02 10
43 Cleaning up the staging area file시 'cannot access' 혹은 'Directory is not writable' 발생시 조치사항 총관리자 2017.05.02 173
42 Ubuntu 16.04 LTS에 4대에 Hadoop 2.8.0설치 총관리자 2017.05.01 452
41 Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 TaskAttempt killed because it ran on unusable node 오류시 조치방법 총관리자 2017.04.06 297
40 서버중 slave,worker,regionserver만 재기동해야 할때 필요한 기동스크립트및 사용방법 총관리자 2017.02.03 132
39 HDFS상의 /tmp폴더에 Permission denied오류가 발생시 조치사항 총관리자 2017.01.25 168
38 Mountable HDFS on CentOS 6.x(hadoop 2.7.2의 nfs기능을 이용) 총관리자 2016.11.24 76
» ./hadoop-daemon.sh start namenode로 namenode기동시 EditLog의 custerId, namespaceId가 달라서 발생하는 오류 해결방법 총관리자 2016.09.24 93
36 hadoop 어플리케이션을 사용하는 사용자 변경시 바꿔줘야 하는 부분 총관리자 2016.09.23 50
35 format된 namenode를 다른 서버에서 다시 format했을때 오류내용 총관리자 2016.09.22 88
34 AIX 7.1에 Hadoop설치(정리중#2) 총관리자 2016.09.20 156
33 AIX 7.1에 Hadoop설치(정리중) 총관리자 2016.09.12 202
32 hadoop클러스터를 구성하던 서버중 HA를 담당하는 서버의 hostname등이 변경되어 문제가 발생했을때 조치사항 총관리자 2016.07.29 305
31 Journal Storage Directory /data/hadoop/journal/data/mycluster not formatted 오류시 조치사항 총관리자 2016.07.29 1428
30 Job이 끝난 log을 볼수 있도록 설정하기 총관리자 2016.05.30 200
29 Hadoop 완벽 가이드 정리된 링크 총관리자 2016.04.19 165
28 Incompatible clusterIDs오류 원인및 해결방법 총관리자 2016.04.01 186
27 namenode오류 복구시 사용하는 명령 총관리자 2016.04.01 174
26 "java.net.NoRouteToHostException: 호스트로 갈 루트가 없음" 오류시 확인및 조치할 사항 총관리자 2016.04.01 2194

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로