메뉴 건너뛰기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


mapred-site.xml.template파일에 mapred-site.xml에 설정해야 하는 내용을 입력후 "yarn jar $HOME/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount in out"를 실행했을때 console상에 application_id도 보이지 않고 8088포트의 UI상에 어플리케이션목록도 보이지 않지만 아래 "비정상적으로 실행되는 모습"의 로그를 찍으면서 실행이 되어 정상적인(?) 결과가 만들어진다.

이때는 mapred-site.xml.template을 mapred-site.xml로 복사하고 각 서버에 scp로 복사해준후 stop-yarn.sh->start-yarn.sh으로 재기동해주면 정상적으로 실행된다.(mapred의 default값을 사용해서 오류가 발생하지는 않은거 같음)

(제대로 설정하고 실행했을때 하고 mapred 정보를 default로 사용하고 실행했을때 결과를 비교하면 개수의 차이가 난다)


*참고

mapred-site.xml.template:    <name>mapreduce.jobhistory.address</name>
mapred-site.xml.template:    <name>mapreduce.jobhistory.webapp.address</name>


----------------------------hadoop-mapreduce-examples-2.8.0.jar의 wordcount가 비정상적으로 실행되는 모습-----------
gooper@gsda1:/home/gooper$ yarn jar $HOME/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar wordcount in out
17/05/02 02:15:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/02 02:15:51 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/05/02 02:15:51 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/05/02 02:15:51 INFO input.FileInputFormat: Total input files to process : 1
17/05/02 02:15:51 INFO mapreduce.JobSubmitter: number of splits:1
17/05/02 02:15:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local27049618_0001
17/05/02 02:15:51 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/05/02 02:15:51 INFO mapreduce.Job: Running job: job_local27049618_0001
17/05/02 02:15:52 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/05/02 02:15:52 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/05/02 02:15:52 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/05/02 02:15:52 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/05/02 02:15:52 INFO mapred.LocalJobRunner: Waiting for map tasks
17/05/02 02:15:52 INFO mapred.LocalJobRunner: Starting task: attempt_local27049618_0001_m_000000_0
17/05/02 02:15:52 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/05/02 02:15:52 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/05/02 02:15:52 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/05/02 02:15:52 INFO mapred.MapTask: Processing split: hdfs://mycluster/user/gooper/in/test.tar.gz:0+429928937
17/05/02 02:15:52 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/05/02 02:15:52 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/05/02 02:15:52 INFO mapred.MapTask: soft limit at 83886080
17/05/02 02:15:52 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/05/02 02:15:52 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/05/02 02:15:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/05/02 02:15:52 INFO compress.CodecPool: Got brand-new decompressor [.gz]
17/05/02 02:15:52 INFO mapreduce.Job: Job job_local27049618_0001 running in uber mode : false
17/05/02 02:15:53 INFO mapreduce.Job:  map 0% reduce 0%
17/05/02 02:15:53 INFO mapred.MapTask: Spilling map output
17/05/02 02:15:53 INFO mapred.MapTask: bufstart = 0; bufend = 52740114; bufvoid = 104857600
17/05/02 02:15:53 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 18427892(73711568); length = 7786505/6553600
17/05/02 02:15:53 INFO mapred.MapTask: (EQUATOR) 60543426 kvi 15135852(60543408)
17/05/02 02:15:57 INFO mapred.MapTask: Finished spill 0
17/05/02 02:15:57 INFO mapred.MapTask: (RESET) equator 60543426 kv 15135852(60543408) kvi 13185036(52740144)
17/05/02 02:15:58 INFO mapred.MapTask: Spilling map output
17/05/02 02:15:58 INFO mapred.MapTask: bufstart = 60543426; bufend = 3432096; bufvoid = 104857594
17/05/02 02:15:58 INFO mapred.MapTask: kvstart = 15135852(60543408); kvend = 6100900(24403600); length = 9034953/6553600
17/05/02 02:15:58 INFO mapred.MapTask: (EQUATOR) 12035776 kvi 3008940(12035760)
17/05/02 02:16:00 INFO mapred.MapTask: Finished spill 1
17/05/02 02:16:00 INFO mapred.MapTask: (RESET) equator 12035776 kv 3008940(12035760) kvi 960640(3842560)
17/05/02 02:16:01 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:01 INFO mapred.MapTask: bufstart = 12035776; bufend = 62517813; bufvoid = 104857600
17/05/02 02:16:01 INFO mapred.MapTask: kvstart = 3008940(12035760); kvend = 20872332(83489328); length = 8351009/6553600
17/05/02 02:16:01 INFO mapred.MapTask: (EQUATOR) 71121493 kvi 17780368(71121472)
17/05/02 02:16:04 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:04 INFO mapred.MapTask: Finished spill 2
17/05/02 02:16:04 INFO mapred.MapTask: (RESET) equator 71121493 kv 17780368(71121472) kvi 15629460(62517840)
17/05/02 02:16:05 INFO mapreduce.Job:  map 2% reduce 0%
17/05/02 02:16:05 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:05 INFO mapred.MapTask: bufstart = 71121493; bufend = 13238215; bufvoid = 104857597
17/05/02 02:16:05 INFO mapred.MapTask: kvstart = 17780368(71121472); kvend = 8552436(34209744); length = 9227933/6553600
17/05/02 02:16:05 INFO mapred.MapTask: (EQUATOR) 21841911 kvi 5460472(21841888)
17/05/02 02:16:08 INFO mapred.MapTask: Finished spill 3
17/05/02 02:16:08 INFO mapred.MapTask: (RESET) equator 21841911 kv 5460472(21841888) kvi 3309560(13238240)
17/05/02 02:16:09 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:09 INFO mapred.MapTask: bufstart = 21841911; bufend = 69241483; bufvoid = 104857600
17/05/02 02:16:09 INFO mapred.MapTask: kvstart = 5460472(21841888); kvend = 22553248(90212992); length = 9121625/6553600
17/05/02 02:16:09 INFO mapred.MapTask: (EQUATOR) 78071579 kvi 19517888(78071552)
17/05/02 02:16:10 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:11 INFO mapreduce.Job:  map 3% reduce 0%
17/05/02 02:16:12 INFO mapred.MapTask: Finished spill 4
17/05/02 02:16:12 INFO mapred.MapTask: (RESET) equator 78071579 kv 19517888(78071552) kvi 17310376(69241504)
17/05/02 02:16:13 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:13 INFO mapred.MapTask: bufstart = 78071579; bufend = 21381477; bufvoid = 104857598
17/05/02 02:16:13 INFO mapred.MapTask: kvstart = 19517888(78071552); kvend = 10588176(42352704); length = 8929713/6553600
17/05/02 02:16:13 INFO mapred.MapTask: (EQUATOR) 30211445 kvi 7552856(30211424)
17/05/02 02:16:16 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:17 INFO mapreduce.Job:  map 4% reduce 0%
17/05/02 02:16:17 INFO mapred.MapTask: Finished spill 5
17/05/02 02:16:17 INFO mapred.MapTask: (RESET) equator 30211445 kv 7552856(30211424) kvi 5345376(21381504)
17/05/02 02:16:18 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:18 INFO mapred.MapTask: bufstart = 30211445; bufend = 79118571; bufvoid = 104857600
17/05/02 02:16:18 INFO mapred.MapTask: kvstart = 7552856(30211424); kvend = 25022496(100089984); length = 8744761/6553600
17/05/02 02:16:18 INFO mapred.MapTask: (EQUATOR) 87948619 kvi 21987148(87948592)
17/05/02 02:16:20 INFO mapred.MapTask: Finished spill 6
17/05/02 02:16:20 INFO mapred.MapTask: (RESET) equator 87948619 kv 21987148(87948592) kvi 19802232(79208928)
17/05/02 02:16:21 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:21 INFO mapred.MapTask: bufstart = 87948619; bufend = 35109798; bufvoid = 104857594
17/05/02 02:16:21 INFO mapred.MapTask: kvstart = 21987148(87948592); kvend = 14020324(56081296); length = 7966825/6553600
17/05/02 02:16:21 INFO mapred.MapTask: (EQUATOR) 43939894 kvi 10984968(43939872)
17/05/02 02:16:22 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:23 INFO mapreduce.Job:  map 6% reduce 0%
17/05/02 02:16:24 INFO mapred.MapTask: Finished spill 7
17/05/02 02:16:24 INFO mapred.MapTask: (RESET) equator 43939894 kv 10984968(43939872) kvi 8946776(35787104)
17/05/02 02:16:25 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:25 INFO mapred.MapTask: bufstart = 43939894; bufend = 95632156; bufvoid = 104857600
17/05/02 02:16:25 INFO mapred.MapTask: kvstart = 10984968(43939872); kvend = 2936520(11746080); length = 8048449/6553600
17/05/02 02:16:25 INFO mapred.MapTask: (EQUATOR) 104235836 kvi 26058952(104235808)
17/05/02 02:16:27 INFO mapred.MapTask: Finished spill 8
17/05/02 02:16:27 INFO mapred.MapTask: (RESET) equator 104235836 kv 26058952(104235808) kvi 24064864(96259456)
17/05/02 02:16:28 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:28 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:28 INFO mapred.MapTask: bufstart = 104235836; bufend = 49957365; bufvoid = 104857597
17/05/02 02:16:28 INFO mapred.MapTask: kvstart = 26058952(104235808); kvend = 17732188(70928752); length = 8326765/6553600
17/05/02 02:16:28 INFO mapred.MapTask: (EQUATOR) 58560997 kvi 14640244(58560976)
17/05/02 02:16:29 INFO mapreduce.Job:  map 7% reduce 0%
17/05/02 02:16:31 INFO mapred.MapTask: Finished spill 9
17/05/02 02:16:31 INFO mapred.MapTask: (RESET) equator 58560997 kv 14640244(58560976) kvi 12489348(49957392)
17/05/02 02:16:32 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:32 INFO mapred.MapTask: bufstart = 58560997; bufend = 993947; bufvoid = 104857600
17/05/02 02:16:32 INFO mapred.MapTask: kvstart = 14640244(58560976); kvend = 5491360(21965440); length = 9148885/6553600
17/05/02 02:16:32 INFO mapred.MapTask: (EQUATOR) 9824027 kvi 2456000(9824000)
17/05/02 02:16:34 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:35 INFO mapreduce.Job:  map 8% reduce 0%
17/05/02 02:16:35 INFO mapred.MapTask: Finished spill 10
17/05/02 02:16:35 INFO mapred.MapTask: (RESET) equator 9824027 kv 2456000(9824000) kvi 248492(993968)
17/05/02 02:16:36 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:36 INFO mapred.MapTask: bufstart = 9824027; bufend = 57213578; bufvoid = 104857600
17/05/02 02:16:36 INFO mapred.MapTask: kvstart = 2456000(9824000); kvend = 19546276(78185104); length = 9124125/6553600
17/05/02 02:16:36 INFO mapred.MapTask: (EQUATOR) 66043674 kvi 16510912(66043648)
17/05/02 02:16:39 INFO mapred.MapTask: Finished spill 11
17/05/02 02:16:39 INFO mapred.MapTask: (RESET) equator 66043674 kv 16510912(66043648) kvi 14303400(57213600)
17/05/02 02:16:39 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:39 INFO mapred.MapTask: bufstart = 66043674; bufend = 9261331; bufvoid = 104857600
17/05/02 02:16:39 INFO mapred.MapTask: kvstart = 16510912(66043648); kvend = 7558216(30232864); length = 8952697/6553600
17/05/02 02:16:39 INFO mapred.MapTask: (EQUATOR) 18091443 kvi 4522856(18091424)
17/05/02 02:16:40 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:41 INFO mapreduce.Job:  map 9% reduce 0%
17/05/02 02:16:42 INFO mapred.MapTask: Finished spill 12
17/05/02 02:16:42 INFO mapred.MapTask: (RESET) equator 18091443 kv 4522856(18091424) kvi 2315340(9261360)
17/05/02 02:16:43 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:43 INFO mapred.MapTask: bufstart = 18091443; bufend = 65402617; bufvoid = 104857600
17/05/02 02:16:43 INFO mapred.MapTask: kvstart = 4522856(18091424); kvend = 21593536(86374144); length = 9143721/6553600
17/05/02 02:16:43 INFO mapred.MapTask: (EQUATOR) 74232713 kvi 18558172(74232688)
17/05/02 02:16:46 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:46 INFO mapred.MapTask: Finished spill 13
17/05/02 02:16:46 INFO mapred.MapTask: (RESET) equator 74232713 kv 18558172(74232688) kvi 16350660(65402640)
17/05/02 02:16:47 INFO mapreduce.Job:  map 10% reduce 0%
17/05/02 02:16:47 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:47 INFO mapred.MapTask: bufstart = 74232713; bufend = 19284541; bufvoid = 104857573
17/05/02 02:16:47 INFO mapred.MapTask: kvstart = 18558172(74232688); kvend = 10064016(40256064); length = 8494157/6553600
17/05/02 02:16:47 INFO mapred.MapTask: (EQUATOR) 28114637 kvi 7028652(28114608)
17/05/02 02:16:49 INFO mapred.MapTask: Finished spill 14
17/05/02 02:16:49 INFO mapred.MapTask: (RESET) equator 28114637 kv 7028652(28114608) kvi 4840528(19362112)
17/05/02 02:16:50 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:50 INFO mapred.MapTask: bufstart = 28114637; bufend = 76839484; bufvoid = 104857600
17/05/02 02:16:50 INFO mapred.MapTask: kvstart = 7028652(28114608); kvend = 24452752(97811008); length = 8790301/6553600
17/05/02 02:16:50 INFO mapred.MapTask: (EQUATOR) 85669580 kvi 21417388(85669552)
17/05/02 02:16:52 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:53 INFO mapreduce.Job:  map 11% reduce 0%
17/05/02 02:16:53 INFO mapred.MapTask: Finished spill 15
17/05/02 02:16:53 INFO mapred.MapTask: (RESET) equator 85669580 kv 21417388(85669552) kvi 19226844(76907376)
17/05/02 02:16:54 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:54 INFO mapred.MapTask: bufstart = 85669580; bufend = 29529421; bufvoid = 104857583
17/05/02 02:16:54 INFO mapred.MapTask: kvstart = 21417388(85669552); kvend = 12625232(50500928); length = 8792157/6553600
17/05/02 02:16:54 INFO mapred.MapTask: (EQUATOR) 38359517 kvi 9589872(38359488)
17/05/02 02:16:56 INFO mapred.MapTask: Finished spill 16
17/05/02 02:16:56 INFO mapred.MapTask: (RESET) equator 38359517 kv 9589872(38359488) kvi 7393336(29573344)
17/05/02 02:16:57 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:57 INFO mapred.MapTask: bufstart = 38359517; bufend = 87069447; bufvoid = 104857600
17/05/02 02:16:57 INFO mapred.MapTask: kvstart = 9589872(38359488); kvend = 795844(3183376); length = 8794029/6553600
17/05/02 02:16:57 INFO mapred.MapTask: (EQUATOR) 95899543 kvi 23974880(95899520)
17/05/02 02:16:58 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:59 INFO mapreduce.Job:  map 13% reduce 0%
17/05/02 02:17:00 INFO mapred.MapTask: Finished spill 17
17/05/02 02:17:00 INFO mapred.MapTask: (RESET) equator 95899543 kv 23974880(95899520) kvi 21786320(87145280)
17/05/02 02:17:01 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:01 INFO mapred.MapTask: bufstart = 95899543; bufend = 39764665; bufvoid = 104857578
17/05/02 02:17:01 INFO mapred.MapTask: kvstart = 23974880(95899520); kvend = 15184040(60736160); length = 8790841/6553600
17/05/02 02:17:01 INFO mapred.MapTask: (EQUATOR) 48594761 kvi 12148684(48594736)
17/05/02 02:17:03 INFO mapred.MapTask: Finished spill 18
17/05/02 02:17:03 INFO mapred.MapTask: (RESET) equator 48594761 kv 12148684(48594736) kvi 9957740(39830960)
17/05/02 02:17:04 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:04 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:04 INFO mapred.MapTask: bufstart = 48594761; bufend = 97311190; bufvoid = 104857600
17/05/02 02:17:04 INFO mapred.MapTask: kvstart = 12148684(48594736); kvend = 3356280(13425120); length = 8792405/6553600
17/05/02 02:17:04 INFO mapred.MapTask: (EQUATOR) 1283686 kvi 320916(1283664)
17/05/02 02:17:05 INFO mapreduce.Job:  map 14% reduce 0%
17/05/02 02:17:07 INFO mapred.MapTask: Finished spill 19
17/05/02 02:17:07 INFO mapred.MapTask: (RESET) equator 1283686 kv 320916(1283664) kvi 24351672(97406688)
17/05/02 02:17:08 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:08 INFO mapred.MapTask: bufstart = 1283686; bufend = 50012840; bufvoid = 104857600
17/05/02 02:17:08 INFO mapred.MapTask: kvstart = 320916(1283664); kvend = 17746092(70984368); length = 8789225/6553600
17/05/02 02:17:08 INFO mapred.MapTask: (EQUATOR) 58842936 kvi 14710728(58842912)
17/05/02 02:17:10 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:10 INFO mapred.MapTask: Finished spill 20
17/05/02 02:17:10 INFO mapred.MapTask: (RESET) equator 58842936 kv 14710728(58842912) kvi 12514108(50056432)
17/05/02 02:17:11 INFO mapreduce.Job:  map 15% reduce 0%
17/05/02 02:17:11 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:11 INFO mapred.MapTask: bufstart = 58842936; bufend = 2688313; bufvoid = 104857591
17/05/02 02:17:11 INFO mapred.MapTask: kvstart = 14710728(58842912); kvend = 5914952(23659808); length = 8795777/6553600
17/05/02 02:17:11 INFO mapred.MapTask: (EQUATOR) 11518409 kvi 2879596(11518384)
17/05/02 02:17:14 INFO mapred.MapTask: Finished spill 21
17/05/02 02:17:14 INFO mapred.MapTask: (RESET) equator 11518409 kv 2879596(11518384) kvi 689616(2758464)
17/05/02 02:17:15 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:15 INFO mapred.MapTask: bufstart = 11518409; bufend = 60239193; bufvoid = 104857600
17/05/02 02:17:15 INFO mapred.MapTask: kvstart = 2879596(11518384); kvend = 20302680(81210720); length = 8791317/6553600
17/05/02 02:17:15 INFO mapred.MapTask: (EQUATOR) 69069289 kvi 17267316(69069264)
17/05/02 02:17:16 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:17 INFO mapreduce.Job:  map 16% reduce 0%
17/05/02 02:17:17 INFO mapred.MapTask: Finished spill 22
17/05/02 02:17:17 INFO mapred.MapTask: (RESET) equator 69069289 kv 17267316(69069264) kvi 15078212(60312848)
17/05/02 02:17:18 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:18 INFO mapred.MapTask: bufstart = 69069289; bufend = 12524402; bufvoid = 104857588
17/05/02 02:17:18 INFO mapred.MapTask: kvstart = 17267316(69069264); kvend = 8373980(33495920); length = 8893337/6553600
17/05/02 02:17:18 INFO mapred.MapTask: (EQUATOR) 21354498 kvi 5338620(21354480)
17/05/02 02:17:21 INFO mapred.MapTask: Finished spill 23
17/05/02 02:17:21 INFO mapred.MapTask: (RESET) equator 21354498 kv 5338620(21354480) kvi 3131108(12524432)
17/05/02 02:17:22 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:22 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:22 INFO mapred.MapTask: bufstart = 21354498; bufend = 69067391; bufvoid = 104857600
17/05/02 02:17:22 INFO mapred.MapTask: kvstart = 5338620(21354480); kvend = 22509724(90038896); length = 9043297/6553600
17/05/02 02:17:22 INFO mapred.MapTask: (EQUATOR) 77897487 kvi 19474364(77897456)
17/05/02 02:17:23 INFO mapreduce.Job:  map 17% reduce 0%
17/05/02 02:17:25 INFO mapred.MapTask: Finished spill 24
17/05/02 02:17:25 INFO mapred.MapTask: (RESET) equator 77897487 kv 19474364(77897456) kvi 17266852(69067408)
17/05/02 02:17:26 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:26 INFO mapred.MapTask: bufstart = 77897487; bufend = 20766799; bufvoid = 104857600
17/05/02 02:17:26 INFO mapred.MapTask: kvstart = 19474364(77897456); kvend = 10434576(41738304); length = 9039789/6553600
17/05/02 02:17:26 INFO mapred.MapTask: (EQUATOR) 29596895 kvi 7399216(29596864)
17/05/02 02:17:28 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:28 INFO mapred.MapTask: Finished spill 25
17/05/02 02:17:28 INFO mapred.MapTask: (RESET) equator 29596895 kv 7399216(29596864) kvi 5191704(20766816)
17/05/02 02:17:29 INFO mapreduce.Job:  map 18% reduce 0%
17/05/02 02:17:29 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:29 INFO mapred.MapTask: bufstart = 29596895; bufend = 77323888; bufvoid = 104857600
17/05/02 02:17:29 INFO mapred.MapTask: kvstart = 7399216(29596864); kvend = 24573856(98295424); length = 9039761/6553600
17/05/02 02:17:29 INFO mapred.MapTask: (EQUATOR) 86154000 kvi 21538496(86153984)
17/05/02 02:17:32 INFO mapred.MapTask: Finished spill 26
17/05/02 02:17:32 INFO mapred.MapTask: (RESET) equator 86154000 kv 21538496(86153984) kvi 19330976(77323904)
17/05/02 02:17:33 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:33 INFO mapred.MapTask: bufstart = 86154000; bufend = 29021185; bufvoid = 104857589
17/05/02 02:17:33 INFO mapred.MapTask: kvstart = 21538496(86153984); kvend = 12498172(49992688); length = 9040325/6553600
17/05/02 02:17:33 INFO mapred.MapTask: (EQUATOR) 37851281 kvi 9462816(37851264)
17/05/02 02:17:34 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:35 INFO mapreduce.Job:  map 20% reduce 0%
17/05/02 02:17:36 INFO mapred.MapTask: Finished spill 27
17/05/02 02:17:36 INFO mapred.MapTask: (RESET) equator 37851281 kv 9462816(37851264) kvi 7255304(29021216)
17/05/02 02:17:37 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:37 INFO mapred.MapTask: bufstart = 37851281; bufend = 85575673; bufvoid = 104857600
17/05/02 02:17:37 INFO mapred.MapTask: kvstart = 9462816(37851264); kvend = 422400(1689600); length = 9040417/6553600
17/05/02 02:17:37 INFO mapred.MapTask: (EQUATOR) 94405769 kvi 23601436(94405744)
17/05/02 02:17:40 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:41 INFO mapreduce.Job:  map 21% reduce 0%
17/05/02 02:17:41 INFO mapred.MapTask: Finished spill 28
17/05/02 02:17:41 INFO mapred.MapTask: (RESET) equator 94405769 kv 23601436(94405744) kvi 21393924(85575696)
17/05/02 02:17:42 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:42 INFO mapred.MapTask: bufstart = 94405769; bufend = 37275243; bufvoid = 104857594
17/05/02 02:17:42 INFO mapred.MapTask: kvstart = 23601436(94405744); kvend = 14561684(58246736); length = 9039753/6553600
17/05/02 02:17:42 INFO mapred.MapTask: (EQUATOR) 46105323 kvi 11526324(46105296)
17/05/02 02:17:45 INFO mapred.MapTask: Finished spill 29
17/05/02 02:17:45 INFO mapred.MapTask: (RESET) equator 46105323 kv 11526324(46105296) kvi 9318816(37275264)
17/05/02 02:17:46 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:46 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:46 INFO mapred.MapTask: bufstart = 46105323; bufend = 95419978; bufvoid = 104857600
17/05/02 02:17:46 INFO mapred.MapTask: kvstart = 11526324(46105296); kvend = 2883320(11533280); length = 8643005/6553600
17/05/02 02:17:46 INFO mapred.MapTask: (EQUATOR) 104249818 kvi 26062448(104249792)
17/05/02 02:17:47 INFO mapreduce.Job:  map 22% reduce 0%
17/05/02 02:17:48 INFO mapred.MapTask: Finished spill 30
17/05/02 02:17:48 INFO mapred.MapTask: (RESET) equator 104249818 kv 26062448(104249792) kvi 24250516(97002064)
17/05/02 02:17:49 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:49 INFO mapred.MapTask: bufstart = 104249818; bufend = 52633620; bufvoid = 104857575
17/05/02 02:17:49 INFO mapred.MapTask: kvstart = 26062448(104249792); kvend = 18401272(73605088); length = 7661177/6553600
17/05/02 02:17:49 INFO mapred.MapTask: (EQUATOR) 61463700 kvi 15365920(61463680)
17/05/02 02:17:52 INFO mapred.MapTask: Finished spill 31
17/05/02 02:17:52 INFO mapred.MapTask: (RESET) equator 61463700 kv 15365920(61463680) kvi 13712356(54849424)
17/05/02 02:17:52 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:52 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:52 INFO mapred.MapTask: bufstart = 61463700; bufend = 10607990; bufvoid = 104857567
17/05/02 02:17:52 INFO mapred.MapTask: kvstart = 15365920(61463680); kvend = 7894880(31579520); length = 7471041/6553600
17/05/02 02:17:52 INFO mapred.MapTask: (EQUATOR) 19438086 kvi 4859516(19438064)
17/05/02 02:17:53 INFO mapreduce.Job:  map 23% reduce 0%
17/05/02 02:17:55 INFO mapred.MapTask: Finished spill 32
17/05/02 02:17:55 INFO mapred.MapTask: (RESET) equator 19438086 kv 4859516(19438064) kvi 3088584(12354336)
17/05/02 02:17:56 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:56 INFO mapred.MapTask: bufstart = 19438086; bufend = 72862821; bufvoid = 104857600
17/05/02 02:17:56 INFO mapred.MapTask: kvstart = 4859516(19438064); kvend = 23458564(93834256); length = 7615353/6553600
17/05/02 02:17:56 INFO mapred.MapTask: (EQUATOR) 81692885 kvi 20423216(81692864)
17/05/02 02:17:58 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:58 INFO mapred.MapTask: Finished spill 33
17/05/02 02:17:58 INFO mapred.MapTask: (RESET) equator 81692885 kv 20423216(81692864) kvi 18215712(72862848)
17/05/02 02:17:59 INFO mapreduce.Job:  map 24% reduce 0%
17/05/02 02:17:59 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:59 INFO mapred.MapTask: bufstart = 81692885; bufend = 25358610; bufvoid = 104857600
17/05/02 02:17:59 INFO mapred.MapTask: kvstart = 20423216(81692864); kvend = 11582532(46330128); length = 8840685/6553600
17/05/02 02:17:59 INFO mapred.MapTask: (EQUATOR) 34188706 kvi 8547172(34188688)
17/05/02 02:18:02 INFO mapred.MapTask: Finished spill 34
17/05/02 02:18:02 INFO mapred.MapTask: (RESET) equator 34188706 kv 8547172(34188688) kvi 6350792(25403168)
17/05/02 02:18:03 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:03 INFO mapred.MapTask: bufstart = 34188706; bufend = 82904433; bufvoid = 104857600
17/05/02 02:18:03 INFO mapred.MapTask: kvstart = 8547172(34188688); kvend = 25968976(103875904); length = 8792597/6553600
17/05/02 02:18:03 INFO mapred.MapTask: (EQUATOR) 91734513 kvi 22933624(91734496)
17/05/02 02:18:04 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:05 INFO mapreduce.Job:  map 25% reduce 0%
17/05/02 02:18:06 INFO mapred.MapTask: Finished spill 35
17/05/02 02:18:06 INFO mapred.MapTask: (RESET) equator 91734513 kv 22933624(91734496) kvi 20743016(82972064)
17/05/02 02:18:07 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:07 INFO mapred.MapTask: bufstart = 91734513; bufend = 35593868; bufvoid = 104857589
17/05/02 02:18:07 INFO mapred.MapTask: kvstart = 22933624(91734496); kvend = 14141336(56565344); length = 8792289/6553600
17/05/02 02:18:07 INFO mapred.MapTask: (EQUATOR) 44423948 kvi 11105980(44423920)
17/05/02 02:18:09 INFO mapred.MapTask: Finished spill 36
17/05/02 02:18:09 INFO mapred.MapTask: (RESET) equator 44423948 kv 11105980(44423920) kvi 8915204(35660816)
17/05/02 02:18:10 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:10 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:10 INFO mapred.MapTask: bufstart = 44423948; bufend = 93140843; bufvoid = 104857600
17/05/02 02:18:10 INFO mapred.MapTask: kvstart = 11105980(44423920); kvend = 2313684(9254736); length = 8792297/6553600
17/05/02 02:18:10 INFO mapred.MapTask: (EQUATOR) 101970923 kvi 25492724(101970896)
17/05/02 02:18:11 INFO mapreduce.Job:  map 26% reduce 0%
17/05/02 02:18:13 INFO mapred.MapTask: Finished spill 37
17/05/02 02:18:13 INFO mapred.MapTask: (RESET) equator 101970923 kv 25492724(101970896) kvi 23308368(93233472)
17/05/02 02:18:14 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:14 INFO mapred.MapTask: bufstart = 101970923; bufend = 45840856; bufvoid = 104857578
17/05/02 02:18:14 INFO mapred.MapTask: kvstart = 25492724(101970896); kvend = 16703088(66812352); length = 8789637/6553600
17/05/02 02:18:14 INFO mapred.MapTask: (EQUATOR) 54670952 kvi 13667732(54670928)
17/05/02 02:18:16 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:17 INFO mapred.MapTask: Finished spill 38
17/05/02 02:18:17 INFO mapreduce.Job:  map 27% reduce 0%
17/05/02 02:18:17 INFO mapred.MapTask: (RESET) equator 54670952 kv 13667732(54670928) kvi 11471596(45886384)
17/05/02 02:18:18 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:18 INFO mapred.MapTask: bufstart = 54670952; bufend = 103375426; bufvoid = 104857600
17/05/02 02:18:18 INFO mapred.MapTask: kvstart = 13667732(54670928); kvend = 4872332(19489328); length = 8795401/6553600
17/05/02 02:18:18 INFO mapred.MapTask: (EQUATOR) 7347922 kvi 1836976(7347904)
17/05/02 02:18:21 INFO mapred.MapTask: Finished spill 39
17/05/02 02:18:21 INFO mapred.MapTask: (RESET) equator 7347922 kv 1836976(7347904) kvi 25861664(103446656)
17/05/02 02:18:22 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:22 INFO mapred.MapTask: bufstart = 7347922; bufend = 56068845; bufvoid = 104857600
17/05/02 02:18:22 INFO mapred.MapTask: kvstart = 1836976(7347904); kvend = 19260092(77040368); length = 8791285/6553600
17/05/02 02:18:22 INFO mapred.MapTask: (EQUATOR) 64898941 kvi 16224728(64898912)
17/05/02 02:18:22 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:23 INFO mapreduce.Job:  map 29% reduce 0%
17/05/02 02:18:24 INFO mapred.MapTask: Finished spill 40
17/05/02 02:18:24 INFO mapred.MapTask: (RESET) equator 64898941 kv 16224728(64898912) kvi 14035056(56140224)
17/05/02 02:18:25 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:25 INFO mapred.MapTask: bufstart = 64898941; bufend = 8762723; bufvoid = 104857599
17/05/02 02:18:25 INFO mapred.MapTask: kvstart = 16224728(64898912); kvend = 7433564(29734256); length = 8791165/6553600
17/05/02 02:18:25 INFO mapred.MapTask: (EQUATOR) 17592835 kvi 4398204(17592816)
17/05/02 02:18:28 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:28 INFO mapred.MapTask: Finished spill 41
17/05/02 02:18:28 INFO mapred.MapTask: (RESET) equator 17592835 kv 4398204(17592816) kvi 2201516(8806064)
17/05/02 02:18:29 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:29 INFO mapred.MapTask: bufstart = 17592835; bufend = 66294882; bufvoid = 104857600
17/05/02 02:18:29 INFO mapred.MapTask: kvstart = 4398204(17592816); kvend = 21816600(87266400); length = 8796005/6553600
17/05/02 02:18:29 INFO mapred.MapTask: (EQUATOR) 75124978 kvi 18781240(75124960)
17/05/02 02:18:32 INFO mapred.MapTask: Finished spill 42
17/05/02 02:18:32 INFO mapred.MapTask: (RESET) equator 75124978 kv 18781240(75124960) kvi 16573728(66294912)
17/05/02 02:18:33 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:33 INFO mapred.MapTask: bufstart = 75124978; bufend = 17865211; bufvoid = 104857599
17/05/02 02:18:33 INFO mapred.MapTask: kvstart = 18781240(75124960); kvend = 9709184(38836736); length = 9072057/6553600
17/05/02 02:18:33 INFO mapred.MapTask: (EQUATOR) 26695307 kvi 6673820(26695280)
17/05/02 02:18:34 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:35 INFO mapreduce.Job:  map 31% reduce 0%
17/05/02 02:18:36 INFO mapred.MapTask: Finished spill 43
17/05/02 02:18:36 INFO mapred.MapTask: (RESET) equator 26695307 kv 6673820(26695280) kvi 4466308(17865232)
17/05/02 02:18:37 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:37 INFO mapred.MapTask: bufstart = 26695307; bufend = 74423017; bufvoid = 104857600
17/05/02 02:18:37 INFO mapred.MapTask: kvstart = 6673820(26695280); kvend = 23848632(95394528); length = 9039589/6553600
17/05/02 02:18:37 INFO mapred.MapTask: (EQUATOR) 83253113 kvi 20813272(83253088)
17/05/02 02:18:40 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:40 INFO mapred.MapTask: Finished spill 44
17/05/02 02:18:40 INFO mapred.MapTask: (RESET) equator 83253113 kv 20813272(83253088) kvi 18605760(74423040)
17/05/02 02:18:41 INFO mapreduce.Job:  map 32% reduce 0%
17/05/02 02:18:41 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:41 INFO mapred.MapTask: bufstart = 83253113; bufend = 26121631; bufvoid = 104857582
17/05/02 02:18:41 INFO mapred.MapTask: kvstart = 20813272(83253088); kvend = 11773288(47093152); length = 9039985/6553600
17/05/02 02:18:41 INFO mapred.MapTask: (EQUATOR) 34951727 kvi 8737924(34951696)
17/05/02 02:18:44 INFO mapred.MapTask: Finished spill 45
17/05/02 02:18:44 INFO mapred.MapTask: (RESET) equator 34951727 kv 8737924(34951696) kvi 6530412(26121648)
17/05/02 02:18:46 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:46 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:46 INFO mapred.MapTask: bufstart = 34951727; bufend = 82681865; bufvoid = 104857600
17/05/02 02:18:46 INFO mapred.MapTask: kvstart = 8737924(34951696); kvend = 25913348(103653392); length = 9038977/6553600
17/05/02 02:18:46 INFO mapred.MapTask: (EQUATOR) 91511961 kvi 22877984(91511936)
17/05/02 02:18:47 INFO mapreduce.Job:  map 33% reduce 0%
17/05/02 02:18:49 INFO mapred.MapTask: Finished spill 46
17/05/02 02:18:49 INFO mapred.MapTask: (RESET) equator 91511961 kv 22877984(91511936) kvi 20670472(82681888)
17/05/02 02:18:50 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:50 INFO mapred.MapTask: bufstart = 91511961; bufend = 34367055; bufvoid = 104857586
17/05/02 02:18:50 INFO mapred.MapTask: kvstart = 22877984(91511936); kvend = 13834644(55338576); length = 9043341/6553600
17/05/02 02:18:50 INFO mapred.MapTask: (EQUATOR) 43197151 kvi 10799280(43197120)
17/05/02 02:18:52 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:52 INFO mapred.MapTask: Finished spill 47
17/05/02 02:18:52 INFO mapred.MapTask: (RESET) equator 43197151 kv 10799280(43197120) kvi 8591768(34367072)
17/05/02 02:18:53 INFO mapreduce.Job:  map 34% reduce 0%
17/05/02 02:18:53 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:53 INFO mapred.MapTask: bufstart = 43197151; bufend = 90923425; bufvoid = 104857600
17/05/02 02:18:53 INFO mapred.MapTask: kvstart = 10799280(43197120); kvend = 1759332(7037328); length = 9039949/6553600
17/05/02 02:18:53 INFO mapred.MapTask: (EQUATOR) 99753521 kvi 24938376(99753504)
17/05/02 02:18:56 INFO mapred.MapTask: Finished spill 48
17/05/02 02:18:56 INFO mapred.MapTask: (RESET) equator 99753521 kv 24938376(99753504) kvi 22730864(90923456)
17/05/02 02:18:57 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:57 INFO mapred.MapTask: bufstart = 99753521; bufend = 46056088; bufvoid = 104857599
17/05/02 02:18:57 INFO mapred.MapTask: kvstart = 24938376(99753504); kvend = 16756904(67027616); length = 8181473/6553600
17/05/02 02:18:57 INFO mapred.MapTask: (EQUATOR) 54886184 kvi 13721540(54886160)
17/05/02 02:18:58 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:59 INFO mapreduce.Job:  map 36% reduce 0%
17/05/02 02:19:00 INFO mapred.MapTask: Finished spill 49
17/05/02 02:19:00 INFO mapred.MapTask: (RESET) equator 54886184 kv 13721540(54886160) kvi 11514028(46056112)
17/05/02 02:19:02 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:02 INFO mapred.MapTask: bufstart = 54886184; bufend = 104834095; bufvoid = 104857600
17/05/02 02:19:02 INFO mapred.MapTask: kvstart = 13721540(54886160); kvend = 5236996(20947984); length = 8484545/6553600
17/05/02 02:19:02 INFO mapred.MapTask: (EQUATOR) 8806575 kvi 2201636(8806544)
17/05/02 02:19:04 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:05 INFO mapreduce.Job:  map 39% reduce 0%
17/05/02 02:19:05 INFO mapred.MapTask: Finished spill 50
17/05/02 02:19:05 INFO mapred.MapTask: (RESET) equator 8806575 kv 2201636(8806544) kvi 1710860(6843440)
17/05/02 02:19:07 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:07 INFO mapred.MapTask: bufstart = 8806575; bufend = 79077092; bufvoid = 104857600
17/05/02 02:19:07 INFO mapred.MapTask: kvstart = 2201636(8806544); kvend = 25012144(100048576); length = 3403893/6553600
17/05/02 02:19:07 INFO mapred.MapTask: (EQUATOR) 87907172 kvi 21976788(87907152)
17/05/02 02:19:09 INFO mapred.MapTask: Finished spill 51
17/05/02 02:19:09 INFO mapred.MapTask: (RESET) equator 87907172 kv 21976788(87907152) kvi 21502184(86008736)
17/05/02 02:19:10 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:11 INFO mapreduce.Job:  map 45% reduce 0%
17/05/02 02:19:12 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:12 INFO mapred.MapTask: bufstart = 87907172; bufend = 54619440; bufvoid = 104857420
17/05/02 02:19:12 INFO mapred.MapTask: kvstart = 21976788(87907152); kvend = 18897724(75590896); length = 3079065/6553600
17/05/02 02:19:12 INFO mapred.MapTask: (EQUATOR) 63223104 kvi 15805772(63223088)
17/05/02 02:19:14 INFO mapred.MapTask: Finished spill 52
17/05/02 02:19:14 INFO mapred.MapTask: (RESET) equator 63223104 kv 15805772(63223088) kvi 15333264(61333056)
17/05/02 02:19:16 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:17 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:17 INFO mapred.MapTask: bufstart = 63223104; bufend = 30462156; bufvoid = 104857596
17/05/02 02:19:17 INFO mapred.MapTask: kvstart = 15805772(63223088); kvend = 12858420(51433680); length = 2947353/6553600
17/05/02 02:19:17 INFO mapred.MapTask: (EQUATOR) 39065836 kvi 9766452(39065808)
17/05/02 02:19:17 INFO mapreduce.Job:  map 52% reduce 0%
17/05/02 02:19:19 INFO mapred.MapTask: Finished spill 53
17/05/02 02:19:19 INFO mapred.MapTask: (RESET) equator 39065836 kv 9766452(39065808) kvi 9279284(37117136)
17/05/02 02:19:21 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:21 INFO mapred.MapTask: bufstart = 39065836; bufend = 4362548; bufvoid = 104857567
17/05/02 02:19:21 INFO mapred.MapTask: kvstart = 9766452(39065808); kvend = 6333508(25334032); length = 3432945/6553600
17/05/02 02:19:21 INFO mapred.MapTask: (EQUATOR) 12751124 kvi 3187776(12751104)
17/05/02 02:19:22 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:23 INFO mapreduce.Job:  map 60% reduce 0%
17/05/02 02:19:24 INFO mapred.MapTask: Finished spill 54
17/05/02 02:19:24 INFO mapred.MapTask: (RESET) equator 12751124 kv 3187776(12751104) kvi 2686436(10745744)
17/05/02 02:19:26 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:26 INFO mapred.MapTask: bufstart = 12751124; bufend = 83577219; bufvoid = 104857600
17/05/02 02:19:26 INFO mapred.MapTask: kvstart = 3187776(12751104); kvend = 26137180(104548720); length = 3264997/6553600
17/05/02 02:19:26 INFO mapred.MapTask: (EQUATOR) 91965811 kvi 22991448(91965792)
17/05/02 02:19:28 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:28 INFO mapred.MapTask: Finished spill 55
17/05/02 02:19:28 INFO mapred.MapTask: (RESET) equator 91965811 kv 22991448(91965792) kvi 22491844(89967376)
17/05/02 02:19:29 INFO mapreduce.Job:  map 66% reduce 0%
17/05/02 02:19:29 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:29 INFO mapred.MapTask: Starting flush of map output
17/05/02 02:19:29 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:29 INFO mapred.MapTask: bufstart = 91965811; bufend = 8890854; bufvoid = 104857558
17/05/02 02:19:29 INFO mapred.MapTask: kvstart = 22991448(91965792); kvend = 22136352(88545408); length = 855097/6553600
17/05/02 02:19:29 INFO mapred.MapTask: Finished spill 56
17/05/02 02:19:29 INFO mapred.Merger: Merging 57 sorted segments
17/05/02 02:19:29 INFO mapred.Merger: Merging 3 intermediate segments out of a total of 57
17/05/02 02:19:29 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 55
17/05/02 02:19:30 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 46
17/05/02 02:19:30 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 37
17/05/02 02:19:31 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 28
17/05/02 02:19:32 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 19
17/05/02 02:19:33 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 569297651 bytes
17/05/02 02:19:34 INFO mapred.LocalJobRunner: map > sort >
17/05/02 02:19:35 INFO mapreduce.Job:  map 78% reduce 0%
17/05/02 02:19:40 INFO mapred.LocalJobRunner: map > sort >
17/05/02 02:19:40 INFO mapred.Task: Task:attempt_local27049618_0001_m_000000_0 is done. And is in the process of committing
17/05/02 02:19:40 INFO mapred.LocalJobRunner: map > sort
17/05/02 02:19:40 INFO mapred.Task: Task 'attempt_local27049618_0001_m_000000_0' done.
17/05/02 02:19:40 INFO mapred.LocalJobRunner: Finishing task: attempt_local27049618_0001_m_000000_0
17/05/02 02:19:40 INFO mapred.LocalJobRunner: map task executor complete.
17/05/02 02:19:40 INFO mapred.LocalJobRunner: Waiting for reduce tasks
17/05/02 02:19:40 INFO mapred.LocalJobRunner: Starting task: attempt_local27049618_0001_r_000000_0
17/05/02 02:19:40 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/05/02 02:19:40 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/05/02 02:19:40 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/05/02 02:19:40 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6f996faf
17/05/02 02:19:40 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=652528832, maxSingleShuffleLimit=163132208, mergeThreshold=430669056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
17/05/02 02:19:40 INFO reduce.EventFetcher: attempt_local27049618_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
17/05/02 02:19:40 INFO reduce.MergeManagerImpl: attempt_local27049618_0001_m_000000_0: Shuffling to disk since 278465134 is greater than maxSingleShuffleLimit (163132208)
17/05/02 02:19:40 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local27049618_0001_m_000000_0 decomp: 278465134 len: 278465138 to DISK
17/05/02 02:19:41 INFO mapreduce.Job:  map 100% reduce 0%
17/05/02 02:19:41 INFO reduce.OnDiskMapOutput: Read 278465138 bytes from map-output for attempt_local27049618_0001_m_000000_0
17/05/02 02:19:41 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
17/05/02 02:19:41 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/05/02 02:19:41 INFO reduce.MergeManagerImpl: finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
17/05/02 02:19:41 INFO reduce.MergeManagerImpl: Merging 1 files, 278465138 bytes from disk
17/05/02 02:19:41 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
17/05/02 02:19:41 INFO mapred.Merger: Merging 1 sorted segments
17/05/02 02:19:41 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 278465130 bytes
17/05/02 02:19:41 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/05/02 02:19:41 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
17/05/02 02:19:50 INFO mapred.Task: Task:attempt_local27049618_0001_r_000000_0 is done. And is in the process of committing
17/05/02 02:19:50 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/05/02 02:19:50 INFO mapred.Task: Task attempt_local27049618_0001_r_000000_0 is allowed to commit now
17/05/02 02:19:50 INFO output.FileOutputCommitter: Saved output of task 'attempt_local27049618_0001_r_000000_0' to hdfs://mycluster/user/gooper/out/_temporary/0/task_local27049618_0001_r_000000
17/05/02 02:19:50 INFO mapred.LocalJobRunner: reduce > reduce
17/05/02 02:19:50 INFO mapred.Task: Task 'attempt_local27049618_0001_r_000000_0' done.
17/05/02 02:19:50 INFO mapred.LocalJobRunner: Finishing task: attempt_local27049618_0001_r_000000_0
17/05/02 02:19:50 INFO mapred.LocalJobRunner: reduce task executor complete.
17/05/02 02:19:51 INFO mapreduce.Job:  map 100% reduce 100%
17/05/02 02:19:51 INFO mapreduce.Job: Job job_local27049618_0001 completed successfully
17/05/02 02:19:51 INFO mapreduce.Job: Counters: 35
        File System Counters
                FILE: Number of bytes read=2201991558
                FILE: Number of bytes written=2481118468
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=859857874
                HDFS: Number of bytes written=266794847
                HDFS: Number of read operations=13
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=4
        Map-Reduce Framework
                Map input records=25042379
                Map output records=115592557
                Map output bytes=2873343481
                Map output materialized bytes=278465138
                Input split bytes=108
                Combine input records=122449942
                Combine output records=9417480
                Reduce input groups=2560095
                Reduce shuffle bytes=278465138
                Reduce input records=2560095
                Reduce output records=2560095
                Spilled Records=16617705
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=761
                Total committed heap usage (bytes)=542638080
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=429928937
        File Output Format Counters
                Bytes Written=266794847


-----------------hadoop-mapreduce-examples-2.8.0.jar의 wordcount가 정상적으로 실행되는 모습-----------

root@gsda1:~/hadoop/sbin# yarn jar $HOME/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar wordcount in out3
17/05/02 04:53:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/02 04:54:00 INFO input.FileInputFormat: Total input files to process : 3
17/05/02 04:54:00 INFO mapreduce.JobSubmitter: number of splits:3
17/05/02 04:54:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1493638357554_0001
17/05/02 04:54:01 INFO impl.YarnClientImpl: Submitted application application_1493638357554_0001
17/05/02 04:54:01 INFO mapreduce.Job: The url to track the job: http://gsda2:8088/proxy/application_1493638357554_0001/
17/05/02 04:54:01 INFO mapreduce.Job: Running job: job_1493638357554_0001
17/05/02 04:54:10 INFO mapreduce.Job: Job job_1493638357554_0001 running in uber mode : false
17/05/02 04:54:10 INFO mapreduce.Job:  map 0% reduce 0%
17/05/02 04:54:15 INFO mapreduce.Job:  map 33% reduce 0%
17/05/02 04:54:18 INFO mapreduce.Job:  map 67% reduce 0%
17/05/02 04:54:33 INFO mapreduce.Job:  map 67% reduce 22%
17/05/02 04:54:34 INFO mapreduce.Job:  map 68% reduce 22%
17/05/02 04:54:52 INFO mapreduce.Job:  map 69% reduce 22%
17/05/02 04:55:10 INFO mapreduce.Job:  map 70% reduce 22%
17/05/02 04:55:29 INFO mapreduce.Job:  map 71% reduce 22%
17/05/02 04:55:41 INFO mapreduce.Job:  map 72% reduce 22%
17/05/02 04:55:59 INFO mapreduce.Job:  map 73% reduce 22%
17/05/02 04:56:17 INFO mapreduce.Job:  map 74% reduce 22%
17/05/02 04:56:35 INFO mapreduce.Job:  map 75% reduce 22%
17/05/02 04:56:53 INFO mapreduce.Job:  map 76% reduce 22%
17/05/02 04:57:11 INFO mapreduce.Job:  map 77% reduce 22%
17/05/02 04:57:23 INFO mapreduce.Job:  map 78% reduce 22%
17/05/02 04:57:35 INFO mapreduce.Job:  map 79% reduce 22%
17/05/02 04:57:41 INFO mapreduce.Job:  map 80% reduce 22%
17/05/02 04:57:47 INFO mapreduce.Job:  map 82% reduce 22%
17/05/02 04:57:53 INFO mapreduce.Job:  map 85% reduce 22%
17/05/02 04:57:59 INFO mapreduce.Job:  map 87% reduce 22%
17/05/02 04:58:05 INFO mapreduce.Job:  map 89% reduce 22%
17/05/02 04:58:11 INFO mapreduce.Job:  map 94% reduce 22%
17/05/02 04:58:17 INFO mapreduce.Job:  map 100% reduce 22%
17/05/02 04:58:21 INFO mapreduce.Job:  map 100% reduce 71%
17/05/02 04:58:27 INFO mapreduce.Job:  map 100% reduce 97%
17/05/02 04:58:28 INFO mapreduce.Job:  map 100% reduce 100%
17/05/02 04:58:28 INFO mapreduce.Job: Job job_1493638357554_0001 completed successfully
17/05/02 04:58:28 INFO mapreduce.Job: Counters: 50
        File System Counters
                FILE: Number of bytes read=1108076402
                FILE: Number of bytes written=1394490956
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=433965095
                HDFS: Number of bytes written=266795644
                HDFS: Number of read operations=12
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Killed map tasks=1
                Launched map tasks=4
                Launched reduce tasks=1
                Data-local map tasks=4
                Total time spent by all maps in occupied slots (ms)=483097
                Total time spent by all reduces in occupied slots (ms)=250461
                Total time spent by all map tasks (ms)=483097
                Total time spent by all reduce tasks (ms)=250461
                Total vcore-milliseconds taken by all map tasks=483097
                Total vcore-milliseconds taken by all reduce tasks=250461
                Total megabyte-milliseconds taken by all map tasks=1978765312
                Total megabyte-milliseconds taken by all reduce tasks=1025888256
        Map-Reduce Framework
                Map input records=25071914
                Map output records=115663168
                Map output bytes=2880593627
                Map output materialized bytes=285847897
                Input split bytes=362
                Combine input records=122520553
                Combine output records=9485298
                Reduce input groups=2560097
                Reduce shuffle bytes=285847897
                Reduce input records=2627913
                Reduce output records=2560097
                Spilled Records=16753341
                Shuffled Maps =3
                Failed Shuffles=0
                Merged Map outputs=3
                GC time elapsed (ms)=1042
                CPU time spent (ms)=292030
                Physical memory (bytes) snapshot=2016387072
                Virtual memory (bytes) snapshot=21047717888
                Total committed heap usage (bytes)=1123024896
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=433964733
        File Output Format Counters
                Bytes Written=266795644

번호 제목 글쓴이 날짜 조회 수
» hadoop에서 yarn jar ..를 이용하여 appliction을 실행하여 정상적(?)으로 수행되었으나 yarn UI의 어플리케이션 목록에 나타나지 않는 문제 총관리자 2017.05.02 77
46 hadoop에서 yarn jar ..를 이용하여 appliction을 실행하여 정상적으로 수행되었으나 yarn UI의 어플리케이션 목록에 나타나지 않는 문제 총관리자 2017.05.02 117
45 hadoop에서 yarn jar ..를 이용하여 appliction을 실행하여 정상적으로 수행되었으나 yarn UI의 어플리케이션 목록에 나타나지 않는 문제 총관리자 2017.05.02 51
44 hadoop에서 yarn jar ..를 이용하여 appliction을 실행하여 정상적으로 수행되었으나 yarn UI의 어플리케이션 목록에 나타나지 않는 문제 총관리자 2017.05.02 24
43 Cleaning up the staging area file시 'cannot access' 혹은 'Directory is not writable' 발생시 조치사항 총관리자 2017.05.02 332
42 Ubuntu 16.04 LTS에 4대에 Hadoop 2.8.0설치 총관리자 2017.05.01 520
41 Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 TaskAttempt killed because it ran on unusable node 오류시 조치방법 총관리자 2017.04.06 324
40 서버중 slave,worker,regionserver만 재기동해야 할때 필요한 기동스크립트및 사용방법 총관리자 2017.02.03 143
39 HDFS상의 /tmp폴더에 Permission denied오류가 발생시 조치사항 총관리자 2017.01.25 358
38 Mountable HDFS on CentOS 6.x(hadoop 2.7.2의 nfs기능을 이용) 총관리자 2016.11.24 173
37 ./hadoop-daemon.sh start namenode로 namenode기동시 EditLog의 custerId, namespaceId가 달라서 발생하는 오류 해결방법 총관리자 2016.09.24 118
36 hadoop 어플리케이션을 사용하는 사용자 변경시 바꿔줘야 하는 부분 총관리자 2016.09.23 68
35 format된 namenode를 다른 서버에서 다시 format했을때 오류내용 총관리자 2016.09.22 155
34 AIX 7.1에 Hadoop설치(정리중#2) 총관리자 2016.09.20 214
33 AIX 7.1에 Hadoop설치(정리중) 총관리자 2016.09.12 236
32 hadoop클러스터를 구성하던 서버중 HA를 담당하는 서버의 hostname등이 변경되어 문제가 발생했을때 조치사항 총관리자 2016.07.29 362
31 Journal Storage Directory /data/hadoop/journal/data/mycluster not formatted 오류시 조치사항 총관리자 2016.07.29 1518
30 Job이 끝난 log을 볼수 있도록 설정하기 총관리자 2016.05.30 403
29 Hadoop 완벽 가이드 정리된 링크 총관리자 2016.04.19 205
28 Incompatible clusterIDs오류 원인및 해결방법 총관리자 2016.04.01 481

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로