메뉴 건너뛰기

검색창 닫기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


Hadoop hadoop에서 yarn jar ..를 이용하여 appliction을 실행하여 정상적으로 수행되었으나 yarn UI의 어플리케이션 목록에 나타나지 않는 문제

총관리자 2017.05.02 14:05 조회 수 : 51

yarn jar $HOME/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount in out



----------------------------hadoop-mapreduce-examples-2.8.0.jar의 wordcount가 비정상적으로 실행되는 모습-----------
gooper@gsda1:/home/gooper$ yarn jar $HOME/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar wordcount in out
17/05/02 02:15:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/02 02:15:51 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/05/02 02:15:51 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/05/02 02:15:51 INFO input.FileInputFormat: Total input files to process : 1
17/05/02 02:15:51 INFO mapreduce.JobSubmitter: number of splits:1
17/05/02 02:15:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local27049618_0001
17/05/02 02:15:51 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/05/02 02:15:51 INFO mapreduce.Job: Running job: job_local27049618_0001
17/05/02 02:15:52 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/05/02 02:15:52 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/05/02 02:15:52 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/05/02 02:15:52 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/05/02 02:15:52 INFO mapred.LocalJobRunner: Waiting for map tasks
17/05/02 02:15:52 INFO mapred.LocalJobRunner: Starting task: attempt_local27049618_0001_m_000000_0
17/05/02 02:15:52 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/05/02 02:15:52 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/05/02 02:15:52 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/05/02 02:15:52 INFO mapred.MapTask: Processing split: hdfs://mycluster/user/gooper/in/test.tar.gz:0+429928937
17/05/02 02:15:52 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/05/02 02:15:52 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/05/02 02:15:52 INFO mapred.MapTask: soft limit at 83886080
17/05/02 02:15:52 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/05/02 02:15:52 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/05/02 02:15:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/05/02 02:15:52 INFO compress.CodecPool: Got brand-new decompressor [.gz]
17/05/02 02:15:52 INFO mapreduce.Job: Job job_local27049618_0001 running in uber mode : false
17/05/02 02:15:53 INFO mapreduce.Job:  map 0% reduce 0%
17/05/02 02:15:53 INFO mapred.MapTask: Spilling map output
17/05/02 02:15:53 INFO mapred.MapTask: bufstart = 0; bufend = 52740114; bufvoid = 104857600
17/05/02 02:15:53 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 18427892(73711568); length = 7786505/6553600
17/05/02 02:15:53 INFO mapred.MapTask: (EQUATOR) 60543426 kvi 15135852(60543408)
17/05/02 02:15:57 INFO mapred.MapTask: Finished spill 0
17/05/02 02:15:57 INFO mapred.MapTask: (RESET) equator 60543426 kv 15135852(60543408) kvi 13185036(52740144)
17/05/02 02:15:58 INFO mapred.MapTask: Spilling map output
17/05/02 02:15:58 INFO mapred.MapTask: bufstart = 60543426; bufend = 3432096; bufvoid = 104857594
17/05/02 02:15:58 INFO mapred.MapTask: kvstart = 15135852(60543408); kvend = 6100900(24403600); length = 9034953/6553600
17/05/02 02:15:58 INFO mapred.MapTask: (EQUATOR) 12035776 kvi 3008940(12035760)
17/05/02 02:16:00 INFO mapred.MapTask: Finished spill 1
17/05/02 02:16:00 INFO mapred.MapTask: (RESET) equator 12035776 kv 3008940(12035760) kvi 960640(3842560)
17/05/02 02:16:01 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:01 INFO mapred.MapTask: bufstart = 12035776; bufend = 62517813; bufvoid = 104857600
17/05/02 02:16:01 INFO mapred.MapTask: kvstart = 3008940(12035760); kvend = 20872332(83489328); length = 8351009/6553600
17/05/02 02:16:01 INFO mapred.MapTask: (EQUATOR) 71121493 kvi 17780368(71121472)
17/05/02 02:16:04 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:04 INFO mapred.MapTask: Finished spill 2
17/05/02 02:16:04 INFO mapred.MapTask: (RESET) equator 71121493 kv 17780368(71121472) kvi 15629460(62517840)
17/05/02 02:16:05 INFO mapreduce.Job:  map 2% reduce 0%
17/05/02 02:16:05 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:05 INFO mapred.MapTask: bufstart = 71121493; bufend = 13238215; bufvoid = 104857597
17/05/02 02:16:05 INFO mapred.MapTask: kvstart = 17780368(71121472); kvend = 8552436(34209744); length = 9227933/6553600
17/05/02 02:16:05 INFO mapred.MapTask: (EQUATOR) 21841911 kvi 5460472(21841888)
17/05/02 02:16:08 INFO mapred.MapTask: Finished spill 3
17/05/02 02:16:08 INFO mapred.MapTask: (RESET) equator 21841911 kv 5460472(21841888) kvi 3309560(13238240)
17/05/02 02:16:09 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:09 INFO mapred.MapTask: bufstart = 21841911; bufend = 69241483; bufvoid = 104857600
17/05/02 02:16:09 INFO mapred.MapTask: kvstart = 5460472(21841888); kvend = 22553248(90212992); length = 9121625/6553600
17/05/02 02:16:09 INFO mapred.MapTask: (EQUATOR) 78071579 kvi 19517888(78071552)
17/05/02 02:16:10 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:11 INFO mapreduce.Job:  map 3% reduce 0%
17/05/02 02:16:12 INFO mapred.MapTask: Finished spill 4
17/05/02 02:16:12 INFO mapred.MapTask: (RESET) equator 78071579 kv 19517888(78071552) kvi 17310376(69241504)
17/05/02 02:16:13 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:13 INFO mapred.MapTask: bufstart = 78071579; bufend = 21381477; bufvoid = 104857598
17/05/02 02:16:13 INFO mapred.MapTask: kvstart = 19517888(78071552); kvend = 10588176(42352704); length = 8929713/6553600
17/05/02 02:16:13 INFO mapred.MapTask: (EQUATOR) 30211445 kvi 7552856(30211424)
17/05/02 02:16:16 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:17 INFO mapreduce.Job:  map 4% reduce 0%
17/05/02 02:16:17 INFO mapred.MapTask: Finished spill 5
17/05/02 02:16:17 INFO mapred.MapTask: (RESET) equator 30211445 kv 7552856(30211424) kvi 5345376(21381504)
17/05/02 02:16:18 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:18 INFO mapred.MapTask: bufstart = 30211445; bufend = 79118571; bufvoid = 104857600
17/05/02 02:16:18 INFO mapred.MapTask: kvstart = 7552856(30211424); kvend = 25022496(100089984); length = 8744761/6553600
17/05/02 02:16:18 INFO mapred.MapTask: (EQUATOR) 87948619 kvi 21987148(87948592)
17/05/02 02:16:20 INFO mapred.MapTask: Finished spill 6
17/05/02 02:16:20 INFO mapred.MapTask: (RESET) equator 87948619 kv 21987148(87948592) kvi 19802232(79208928)
17/05/02 02:16:21 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:21 INFO mapred.MapTask: bufstart = 87948619; bufend = 35109798; bufvoid = 104857594
17/05/02 02:16:21 INFO mapred.MapTask: kvstart = 21987148(87948592); kvend = 14020324(56081296); length = 7966825/6553600
17/05/02 02:16:21 INFO mapred.MapTask: (EQUATOR) 43939894 kvi 10984968(43939872)
17/05/02 02:16:22 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:23 INFO mapreduce.Job:  map 6% reduce 0%
17/05/02 02:16:24 INFO mapred.MapTask: Finished spill 7
17/05/02 02:16:24 INFO mapred.MapTask: (RESET) equator 43939894 kv 10984968(43939872) kvi 8946776(35787104)
17/05/02 02:16:25 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:25 INFO mapred.MapTask: bufstart = 43939894; bufend = 95632156; bufvoid = 104857600
17/05/02 02:16:25 INFO mapred.MapTask: kvstart = 10984968(43939872); kvend = 2936520(11746080); length = 8048449/6553600
17/05/02 02:16:25 INFO mapred.MapTask: (EQUATOR) 104235836 kvi 26058952(104235808)
17/05/02 02:16:27 INFO mapred.MapTask: Finished spill 8
17/05/02 02:16:27 INFO mapred.MapTask: (RESET) equator 104235836 kv 26058952(104235808) kvi 24064864(96259456)
17/05/02 02:16:28 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:28 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:28 INFO mapred.MapTask: bufstart = 104235836; bufend = 49957365; bufvoid = 104857597
17/05/02 02:16:28 INFO mapred.MapTask: kvstart = 26058952(104235808); kvend = 17732188(70928752); length = 8326765/6553600
17/05/02 02:16:28 INFO mapred.MapTask: (EQUATOR) 58560997 kvi 14640244(58560976)
17/05/02 02:16:29 INFO mapreduce.Job:  map 7% reduce 0%
17/05/02 02:16:31 INFO mapred.MapTask: Finished spill 9
17/05/02 02:16:31 INFO mapred.MapTask: (RESET) equator 58560997 kv 14640244(58560976) kvi 12489348(49957392)
17/05/02 02:16:32 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:32 INFO mapred.MapTask: bufstart = 58560997; bufend = 993947; bufvoid = 104857600
17/05/02 02:16:32 INFO mapred.MapTask: kvstart = 14640244(58560976); kvend = 5491360(21965440); length = 9148885/6553600
17/05/02 02:16:32 INFO mapred.MapTask: (EQUATOR) 9824027 kvi 2456000(9824000)
17/05/02 02:16:34 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:35 INFO mapreduce.Job:  map 8% reduce 0%
17/05/02 02:16:35 INFO mapred.MapTask: Finished spill 10
17/05/02 02:16:35 INFO mapred.MapTask: (RESET) equator 9824027 kv 2456000(9824000) kvi 248492(993968)
17/05/02 02:16:36 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:36 INFO mapred.MapTask: bufstart = 9824027; bufend = 57213578; bufvoid = 104857600
17/05/02 02:16:36 INFO mapred.MapTask: kvstart = 2456000(9824000); kvend = 19546276(78185104); length = 9124125/6553600
17/05/02 02:16:36 INFO mapred.MapTask: (EQUATOR) 66043674 kvi 16510912(66043648)
17/05/02 02:16:39 INFO mapred.MapTask: Finished spill 11
17/05/02 02:16:39 INFO mapred.MapTask: (RESET) equator 66043674 kv 16510912(66043648) kvi 14303400(57213600)
17/05/02 02:16:39 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:39 INFO mapred.MapTask: bufstart = 66043674; bufend = 9261331; bufvoid = 104857600
17/05/02 02:16:39 INFO mapred.MapTask: kvstart = 16510912(66043648); kvend = 7558216(30232864); length = 8952697/6553600
17/05/02 02:16:39 INFO mapred.MapTask: (EQUATOR) 18091443 kvi 4522856(18091424)
17/05/02 02:16:40 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:41 INFO mapreduce.Job:  map 9% reduce 0%
17/05/02 02:16:42 INFO mapred.MapTask: Finished spill 12
17/05/02 02:16:42 INFO mapred.MapTask: (RESET) equator 18091443 kv 4522856(18091424) kvi 2315340(9261360)
17/05/02 02:16:43 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:43 INFO mapred.MapTask: bufstart = 18091443; bufend = 65402617; bufvoid = 104857600
17/05/02 02:16:43 INFO mapred.MapTask: kvstart = 4522856(18091424); kvend = 21593536(86374144); length = 9143721/6553600
17/05/02 02:16:43 INFO mapred.MapTask: (EQUATOR) 74232713 kvi 18558172(74232688)
17/05/02 02:16:46 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:46 INFO mapred.MapTask: Finished spill 13
17/05/02 02:16:46 INFO mapred.MapTask: (RESET) equator 74232713 kv 18558172(74232688) kvi 16350660(65402640)
17/05/02 02:16:47 INFO mapreduce.Job:  map 10% reduce 0%
17/05/02 02:16:47 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:47 INFO mapred.MapTask: bufstart = 74232713; bufend = 19284541; bufvoid = 104857573
17/05/02 02:16:47 INFO mapred.MapTask: kvstart = 18558172(74232688); kvend = 10064016(40256064); length = 8494157/6553600
17/05/02 02:16:47 INFO mapred.MapTask: (EQUATOR) 28114637 kvi 7028652(28114608)
17/05/02 02:16:49 INFO mapred.MapTask: Finished spill 14
17/05/02 02:16:49 INFO mapred.MapTask: (RESET) equator 28114637 kv 7028652(28114608) kvi 4840528(19362112)
17/05/02 02:16:50 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:50 INFO mapred.MapTask: bufstart = 28114637; bufend = 76839484; bufvoid = 104857600
17/05/02 02:16:50 INFO mapred.MapTask: kvstart = 7028652(28114608); kvend = 24452752(97811008); length = 8790301/6553600
17/05/02 02:16:50 INFO mapred.MapTask: (EQUATOR) 85669580 kvi 21417388(85669552)
17/05/02 02:16:52 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:53 INFO mapreduce.Job:  map 11% reduce 0%
17/05/02 02:16:53 INFO mapred.MapTask: Finished spill 15
17/05/02 02:16:53 INFO mapred.MapTask: (RESET) equator 85669580 kv 21417388(85669552) kvi 19226844(76907376)
17/05/02 02:16:54 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:54 INFO mapred.MapTask: bufstart = 85669580; bufend = 29529421; bufvoid = 104857583
17/05/02 02:16:54 INFO mapred.MapTask: kvstart = 21417388(85669552); kvend = 12625232(50500928); length = 8792157/6553600
17/05/02 02:16:54 INFO mapred.MapTask: (EQUATOR) 38359517 kvi 9589872(38359488)
17/05/02 02:16:56 INFO mapred.MapTask: Finished spill 16
17/05/02 02:16:56 INFO mapred.MapTask: (RESET) equator 38359517 kv 9589872(38359488) kvi 7393336(29573344)
17/05/02 02:16:57 INFO mapred.MapTask: Spilling map output
17/05/02 02:16:57 INFO mapred.MapTask: bufstart = 38359517; bufend = 87069447; bufvoid = 104857600
17/05/02 02:16:57 INFO mapred.MapTask: kvstart = 9589872(38359488); kvend = 795844(3183376); length = 8794029/6553600
17/05/02 02:16:57 INFO mapred.MapTask: (EQUATOR) 95899543 kvi 23974880(95899520)
17/05/02 02:16:58 INFO mapred.LocalJobRunner: map > map
17/05/02 02:16:59 INFO mapreduce.Job:  map 13% reduce 0%
17/05/02 02:17:00 INFO mapred.MapTask: Finished spill 17
17/05/02 02:17:00 INFO mapred.MapTask: (RESET) equator 95899543 kv 23974880(95899520) kvi 21786320(87145280)
17/05/02 02:17:01 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:01 INFO mapred.MapTask: bufstart = 95899543; bufend = 39764665; bufvoid = 104857578
17/05/02 02:17:01 INFO mapred.MapTask: kvstart = 23974880(95899520); kvend = 15184040(60736160); length = 8790841/6553600
17/05/02 02:17:01 INFO mapred.MapTask: (EQUATOR) 48594761 kvi 12148684(48594736)
17/05/02 02:17:03 INFO mapred.MapTask: Finished spill 18
17/05/02 02:17:03 INFO mapred.MapTask: (RESET) equator 48594761 kv 12148684(48594736) kvi 9957740(39830960)
17/05/02 02:17:04 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:04 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:04 INFO mapred.MapTask: bufstart = 48594761; bufend = 97311190; bufvoid = 104857600
17/05/02 02:17:04 INFO mapred.MapTask: kvstart = 12148684(48594736); kvend = 3356280(13425120); length = 8792405/6553600
17/05/02 02:17:04 INFO mapred.MapTask: (EQUATOR) 1283686 kvi 320916(1283664)
17/05/02 02:17:05 INFO mapreduce.Job:  map 14% reduce 0%
17/05/02 02:17:07 INFO mapred.MapTask: Finished spill 19
17/05/02 02:17:07 INFO mapred.MapTask: (RESET) equator 1283686 kv 320916(1283664) kvi 24351672(97406688)
17/05/02 02:17:08 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:08 INFO mapred.MapTask: bufstart = 1283686; bufend = 50012840; bufvoid = 104857600
17/05/02 02:17:08 INFO mapred.MapTask: kvstart = 320916(1283664); kvend = 17746092(70984368); length = 8789225/6553600
17/05/02 02:17:08 INFO mapred.MapTask: (EQUATOR) 58842936 kvi 14710728(58842912)
17/05/02 02:17:10 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:10 INFO mapred.MapTask: Finished spill 20
17/05/02 02:17:10 INFO mapred.MapTask: (RESET) equator 58842936 kv 14710728(58842912) kvi 12514108(50056432)
17/05/02 02:17:11 INFO mapreduce.Job:  map 15% reduce 0%
17/05/02 02:17:11 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:11 INFO mapred.MapTask: bufstart = 58842936; bufend = 2688313; bufvoid = 104857591
17/05/02 02:17:11 INFO mapred.MapTask: kvstart = 14710728(58842912); kvend = 5914952(23659808); length = 8795777/6553600
17/05/02 02:17:11 INFO mapred.MapTask: (EQUATOR) 11518409 kvi 2879596(11518384)
17/05/02 02:17:14 INFO mapred.MapTask: Finished spill 21
17/05/02 02:17:14 INFO mapred.MapTask: (RESET) equator 11518409 kv 2879596(11518384) kvi 689616(2758464)
17/05/02 02:17:15 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:15 INFO mapred.MapTask: bufstart = 11518409; bufend = 60239193; bufvoid = 104857600
17/05/02 02:17:15 INFO mapred.MapTask: kvstart = 2879596(11518384); kvend = 20302680(81210720); length = 8791317/6553600
17/05/02 02:17:15 INFO mapred.MapTask: (EQUATOR) 69069289 kvi 17267316(69069264)
17/05/02 02:17:16 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:17 INFO mapreduce.Job:  map 16% reduce 0%
17/05/02 02:17:17 INFO mapred.MapTask: Finished spill 22
17/05/02 02:17:17 INFO mapred.MapTask: (RESET) equator 69069289 kv 17267316(69069264) kvi 15078212(60312848)
17/05/02 02:17:18 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:18 INFO mapred.MapTask: bufstart = 69069289; bufend = 12524402; bufvoid = 104857588
17/05/02 02:17:18 INFO mapred.MapTask: kvstart = 17267316(69069264); kvend = 8373980(33495920); length = 8893337/6553600
17/05/02 02:17:18 INFO mapred.MapTask: (EQUATOR) 21354498 kvi 5338620(21354480)
17/05/02 02:17:21 INFO mapred.MapTask: Finished spill 23
17/05/02 02:17:21 INFO mapred.MapTask: (RESET) equator 21354498 kv 5338620(21354480) kvi 3131108(12524432)
17/05/02 02:17:22 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:22 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:22 INFO mapred.MapTask: bufstart = 21354498; bufend = 69067391; bufvoid = 104857600
17/05/02 02:17:22 INFO mapred.MapTask: kvstart = 5338620(21354480); kvend = 22509724(90038896); length = 9043297/6553600
17/05/02 02:17:22 INFO mapred.MapTask: (EQUATOR) 77897487 kvi 19474364(77897456)
17/05/02 02:17:23 INFO mapreduce.Job:  map 17% reduce 0%
17/05/02 02:17:25 INFO mapred.MapTask: Finished spill 24
17/05/02 02:17:25 INFO mapred.MapTask: (RESET) equator 77897487 kv 19474364(77897456) kvi 17266852(69067408)
17/05/02 02:17:26 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:26 INFO mapred.MapTask: bufstart = 77897487; bufend = 20766799; bufvoid = 104857600
17/05/02 02:17:26 INFO mapred.MapTask: kvstart = 19474364(77897456); kvend = 10434576(41738304); length = 9039789/6553600
17/05/02 02:17:26 INFO mapred.MapTask: (EQUATOR) 29596895 kvi 7399216(29596864)
17/05/02 02:17:28 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:28 INFO mapred.MapTask: Finished spill 25
17/05/02 02:17:28 INFO mapred.MapTask: (RESET) equator 29596895 kv 7399216(29596864) kvi 5191704(20766816)
17/05/02 02:17:29 INFO mapreduce.Job:  map 18% reduce 0%
17/05/02 02:17:29 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:29 INFO mapred.MapTask: bufstart = 29596895; bufend = 77323888; bufvoid = 104857600
17/05/02 02:17:29 INFO mapred.MapTask: kvstart = 7399216(29596864); kvend = 24573856(98295424); length = 9039761/6553600
17/05/02 02:17:29 INFO mapred.MapTask: (EQUATOR) 86154000 kvi 21538496(86153984)
17/05/02 02:17:32 INFO mapred.MapTask: Finished spill 26
17/05/02 02:17:32 INFO mapred.MapTask: (RESET) equator 86154000 kv 21538496(86153984) kvi 19330976(77323904)
17/05/02 02:17:33 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:33 INFO mapred.MapTask: bufstart = 86154000; bufend = 29021185; bufvoid = 104857589
17/05/02 02:17:33 INFO mapred.MapTask: kvstart = 21538496(86153984); kvend = 12498172(49992688); length = 9040325/6553600
17/05/02 02:17:33 INFO mapred.MapTask: (EQUATOR) 37851281 kvi 9462816(37851264)
17/05/02 02:17:34 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:35 INFO mapreduce.Job:  map 20% reduce 0%
17/05/02 02:17:36 INFO mapred.MapTask: Finished spill 27
17/05/02 02:17:36 INFO mapred.MapTask: (RESET) equator 37851281 kv 9462816(37851264) kvi 7255304(29021216)
17/05/02 02:17:37 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:37 INFO mapred.MapTask: bufstart = 37851281; bufend = 85575673; bufvoid = 104857600
17/05/02 02:17:37 INFO mapred.MapTask: kvstart = 9462816(37851264); kvend = 422400(1689600); length = 9040417/6553600
17/05/02 02:17:37 INFO mapred.MapTask: (EQUATOR) 94405769 kvi 23601436(94405744)
17/05/02 02:17:40 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:41 INFO mapreduce.Job:  map 21% reduce 0%
17/05/02 02:17:41 INFO mapred.MapTask: Finished spill 28
17/05/02 02:17:41 INFO mapred.MapTask: (RESET) equator 94405769 kv 23601436(94405744) kvi 21393924(85575696)
17/05/02 02:17:42 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:42 INFO mapred.MapTask: bufstart = 94405769; bufend = 37275243; bufvoid = 104857594
17/05/02 02:17:42 INFO mapred.MapTask: kvstart = 23601436(94405744); kvend = 14561684(58246736); length = 9039753/6553600
17/05/02 02:17:42 INFO mapred.MapTask: (EQUATOR) 46105323 kvi 11526324(46105296)
17/05/02 02:17:45 INFO mapred.MapTask: Finished spill 29
17/05/02 02:17:45 INFO mapred.MapTask: (RESET) equator 46105323 kv 11526324(46105296) kvi 9318816(37275264)
17/05/02 02:17:46 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:46 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:46 INFO mapred.MapTask: bufstart = 46105323; bufend = 95419978; bufvoid = 104857600
17/05/02 02:17:46 INFO mapred.MapTask: kvstart = 11526324(46105296); kvend = 2883320(11533280); length = 8643005/6553600
17/05/02 02:17:46 INFO mapred.MapTask: (EQUATOR) 104249818 kvi 26062448(104249792)
17/05/02 02:17:47 INFO mapreduce.Job:  map 22% reduce 0%
17/05/02 02:17:48 INFO mapred.MapTask: Finished spill 30
17/05/02 02:17:48 INFO mapred.MapTask: (RESET) equator 104249818 kv 26062448(104249792) kvi 24250516(97002064)
17/05/02 02:17:49 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:49 INFO mapred.MapTask: bufstart = 104249818; bufend = 52633620; bufvoid = 104857575
17/05/02 02:17:49 INFO mapred.MapTask: kvstart = 26062448(104249792); kvend = 18401272(73605088); length = 7661177/6553600
17/05/02 02:17:49 INFO mapred.MapTask: (EQUATOR) 61463700 kvi 15365920(61463680)
17/05/02 02:17:52 INFO mapred.MapTask: Finished spill 31
17/05/02 02:17:52 INFO mapred.MapTask: (RESET) equator 61463700 kv 15365920(61463680) kvi 13712356(54849424)
17/05/02 02:17:52 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:52 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:52 INFO mapred.MapTask: bufstart = 61463700; bufend = 10607990; bufvoid = 104857567
17/05/02 02:17:52 INFO mapred.MapTask: kvstart = 15365920(61463680); kvend = 7894880(31579520); length = 7471041/6553600
17/05/02 02:17:52 INFO mapred.MapTask: (EQUATOR) 19438086 kvi 4859516(19438064)
17/05/02 02:17:53 INFO mapreduce.Job:  map 23% reduce 0%
17/05/02 02:17:55 INFO mapred.MapTask: Finished spill 32
17/05/02 02:17:55 INFO mapred.MapTask: (RESET) equator 19438086 kv 4859516(19438064) kvi 3088584(12354336)
17/05/02 02:17:56 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:56 INFO mapred.MapTask: bufstart = 19438086; bufend = 72862821; bufvoid = 104857600
17/05/02 02:17:56 INFO mapred.MapTask: kvstart = 4859516(19438064); kvend = 23458564(93834256); length = 7615353/6553600
17/05/02 02:17:56 INFO mapred.MapTask: (EQUATOR) 81692885 kvi 20423216(81692864)
17/05/02 02:17:58 INFO mapred.LocalJobRunner: map > map
17/05/02 02:17:58 INFO mapred.MapTask: Finished spill 33
17/05/02 02:17:58 INFO mapred.MapTask: (RESET) equator 81692885 kv 20423216(81692864) kvi 18215712(72862848)
17/05/02 02:17:59 INFO mapreduce.Job:  map 24% reduce 0%
17/05/02 02:17:59 INFO mapred.MapTask: Spilling map output
17/05/02 02:17:59 INFO mapred.MapTask: bufstart = 81692885; bufend = 25358610; bufvoid = 104857600
17/05/02 02:17:59 INFO mapred.MapTask: kvstart = 20423216(81692864); kvend = 11582532(46330128); length = 8840685/6553600
17/05/02 02:17:59 INFO mapred.MapTask: (EQUATOR) 34188706 kvi 8547172(34188688)
17/05/02 02:18:02 INFO mapred.MapTask: Finished spill 34
17/05/02 02:18:02 INFO mapred.MapTask: (RESET) equator 34188706 kv 8547172(34188688) kvi 6350792(25403168)
17/05/02 02:18:03 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:03 INFO mapred.MapTask: bufstart = 34188706; bufend = 82904433; bufvoid = 104857600
17/05/02 02:18:03 INFO mapred.MapTask: kvstart = 8547172(34188688); kvend = 25968976(103875904); length = 8792597/6553600
17/05/02 02:18:03 INFO mapred.MapTask: (EQUATOR) 91734513 kvi 22933624(91734496)
17/05/02 02:18:04 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:05 INFO mapreduce.Job:  map 25% reduce 0%
17/05/02 02:18:06 INFO mapred.MapTask: Finished spill 35
17/05/02 02:18:06 INFO mapred.MapTask: (RESET) equator 91734513 kv 22933624(91734496) kvi 20743016(82972064)
17/05/02 02:18:07 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:07 INFO mapred.MapTask: bufstart = 91734513; bufend = 35593868; bufvoid = 104857589
17/05/02 02:18:07 INFO mapred.MapTask: kvstart = 22933624(91734496); kvend = 14141336(56565344); length = 8792289/6553600
17/05/02 02:18:07 INFO mapred.MapTask: (EQUATOR) 44423948 kvi 11105980(44423920)
17/05/02 02:18:09 INFO mapred.MapTask: Finished spill 36
17/05/02 02:18:09 INFO mapred.MapTask: (RESET) equator 44423948 kv 11105980(44423920) kvi 8915204(35660816)
17/05/02 02:18:10 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:10 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:10 INFO mapred.MapTask: bufstart = 44423948; bufend = 93140843; bufvoid = 104857600
17/05/02 02:18:10 INFO mapred.MapTask: kvstart = 11105980(44423920); kvend = 2313684(9254736); length = 8792297/6553600
17/05/02 02:18:10 INFO mapred.MapTask: (EQUATOR) 101970923 kvi 25492724(101970896)
17/05/02 02:18:11 INFO mapreduce.Job:  map 26% reduce 0%
17/05/02 02:18:13 INFO mapred.MapTask: Finished spill 37
17/05/02 02:18:13 INFO mapred.MapTask: (RESET) equator 101970923 kv 25492724(101970896) kvi 23308368(93233472)
17/05/02 02:18:14 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:14 INFO mapred.MapTask: bufstart = 101970923; bufend = 45840856; bufvoid = 104857578
17/05/02 02:18:14 INFO mapred.MapTask: kvstart = 25492724(101970896); kvend = 16703088(66812352); length = 8789637/6553600
17/05/02 02:18:14 INFO mapred.MapTask: (EQUATOR) 54670952 kvi 13667732(54670928)
17/05/02 02:18:16 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:17 INFO mapred.MapTask: Finished spill 38
17/05/02 02:18:17 INFO mapreduce.Job:  map 27% reduce 0%
17/05/02 02:18:17 INFO mapred.MapTask: (RESET) equator 54670952 kv 13667732(54670928) kvi 11471596(45886384)
17/05/02 02:18:18 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:18 INFO mapred.MapTask: bufstart = 54670952; bufend = 103375426; bufvoid = 104857600
17/05/02 02:18:18 INFO mapred.MapTask: kvstart = 13667732(54670928); kvend = 4872332(19489328); length = 8795401/6553600
17/05/02 02:18:18 INFO mapred.MapTask: (EQUATOR) 7347922 kvi 1836976(7347904)
17/05/02 02:18:21 INFO mapred.MapTask: Finished spill 39
17/05/02 02:18:21 INFO mapred.MapTask: (RESET) equator 7347922 kv 1836976(7347904) kvi 25861664(103446656)
17/05/02 02:18:22 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:22 INFO mapred.MapTask: bufstart = 7347922; bufend = 56068845; bufvoid = 104857600
17/05/02 02:18:22 INFO mapred.MapTask: kvstart = 1836976(7347904); kvend = 19260092(77040368); length = 8791285/6553600
17/05/02 02:18:22 INFO mapred.MapTask: (EQUATOR) 64898941 kvi 16224728(64898912)
17/05/02 02:18:22 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:23 INFO mapreduce.Job:  map 29% reduce 0%
17/05/02 02:18:24 INFO mapred.MapTask: Finished spill 40
17/05/02 02:18:24 INFO mapred.MapTask: (RESET) equator 64898941 kv 16224728(64898912) kvi 14035056(56140224)
17/05/02 02:18:25 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:25 INFO mapred.MapTask: bufstart = 64898941; bufend = 8762723; bufvoid = 104857599
17/05/02 02:18:25 INFO mapred.MapTask: kvstart = 16224728(64898912); kvend = 7433564(29734256); length = 8791165/6553600
17/05/02 02:18:25 INFO mapred.MapTask: (EQUATOR) 17592835 kvi 4398204(17592816)
17/05/02 02:18:28 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:28 INFO mapred.MapTask: Finished spill 41
17/05/02 02:18:28 INFO mapred.MapTask: (RESET) equator 17592835 kv 4398204(17592816) kvi 2201516(8806064)
17/05/02 02:18:29 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:29 INFO mapred.MapTask: bufstart = 17592835; bufend = 66294882; bufvoid = 104857600
17/05/02 02:18:29 INFO mapred.MapTask: kvstart = 4398204(17592816); kvend = 21816600(87266400); length = 8796005/6553600
17/05/02 02:18:29 INFO mapred.MapTask: (EQUATOR) 75124978 kvi 18781240(75124960)
17/05/02 02:18:32 INFO mapred.MapTask: Finished spill 42
17/05/02 02:18:32 INFO mapred.MapTask: (RESET) equator 75124978 kv 18781240(75124960) kvi 16573728(66294912)
17/05/02 02:18:33 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:33 INFO mapred.MapTask: bufstart = 75124978; bufend = 17865211; bufvoid = 104857599
17/05/02 02:18:33 INFO mapred.MapTask: kvstart = 18781240(75124960); kvend = 9709184(38836736); length = 9072057/6553600
17/05/02 02:18:33 INFO mapred.MapTask: (EQUATOR) 26695307 kvi 6673820(26695280)
17/05/02 02:18:34 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:35 INFO mapreduce.Job:  map 31% reduce 0%
17/05/02 02:18:36 INFO mapred.MapTask: Finished spill 43
17/05/02 02:18:36 INFO mapred.MapTask: (RESET) equator 26695307 kv 6673820(26695280) kvi 4466308(17865232)
17/05/02 02:18:37 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:37 INFO mapred.MapTask: bufstart = 26695307; bufend = 74423017; bufvoid = 104857600
17/05/02 02:18:37 INFO mapred.MapTask: kvstart = 6673820(26695280); kvend = 23848632(95394528); length = 9039589/6553600
17/05/02 02:18:37 INFO mapred.MapTask: (EQUATOR) 83253113 kvi 20813272(83253088)
17/05/02 02:18:40 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:40 INFO mapred.MapTask: Finished spill 44
17/05/02 02:18:40 INFO mapred.MapTask: (RESET) equator 83253113 kv 20813272(83253088) kvi 18605760(74423040)
17/05/02 02:18:41 INFO mapreduce.Job:  map 32% reduce 0%
17/05/02 02:18:41 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:41 INFO mapred.MapTask: bufstart = 83253113; bufend = 26121631; bufvoid = 104857582
17/05/02 02:18:41 INFO mapred.MapTask: kvstart = 20813272(83253088); kvend = 11773288(47093152); length = 9039985/6553600
17/05/02 02:18:41 INFO mapred.MapTask: (EQUATOR) 34951727 kvi 8737924(34951696)
17/05/02 02:18:44 INFO mapred.MapTask: Finished spill 45
17/05/02 02:18:44 INFO mapred.MapTask: (RESET) equator 34951727 kv 8737924(34951696) kvi 6530412(26121648)
17/05/02 02:18:46 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:46 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:46 INFO mapred.MapTask: bufstart = 34951727; bufend = 82681865; bufvoid = 104857600
17/05/02 02:18:46 INFO mapred.MapTask: kvstart = 8737924(34951696); kvend = 25913348(103653392); length = 9038977/6553600
17/05/02 02:18:46 INFO mapred.MapTask: (EQUATOR) 91511961 kvi 22877984(91511936)
17/05/02 02:18:47 INFO mapreduce.Job:  map 33% reduce 0%
17/05/02 02:18:49 INFO mapred.MapTask: Finished spill 46
17/05/02 02:18:49 INFO mapred.MapTask: (RESET) equator 91511961 kv 22877984(91511936) kvi 20670472(82681888)
17/05/02 02:18:50 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:50 INFO mapred.MapTask: bufstart = 91511961; bufend = 34367055; bufvoid = 104857586
17/05/02 02:18:50 INFO mapred.MapTask: kvstart = 22877984(91511936); kvend = 13834644(55338576); length = 9043341/6553600
17/05/02 02:18:50 INFO mapred.MapTask: (EQUATOR) 43197151 kvi 10799280(43197120)
17/05/02 02:18:52 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:52 INFO mapred.MapTask: Finished spill 47
17/05/02 02:18:52 INFO mapred.MapTask: (RESET) equator 43197151 kv 10799280(43197120) kvi 8591768(34367072)
17/05/02 02:18:53 INFO mapreduce.Job:  map 34% reduce 0%
17/05/02 02:18:53 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:53 INFO mapred.MapTask: bufstart = 43197151; bufend = 90923425; bufvoid = 104857600
17/05/02 02:18:53 INFO mapred.MapTask: kvstart = 10799280(43197120); kvend = 1759332(7037328); length = 9039949/6553600
17/05/02 02:18:53 INFO mapred.MapTask: (EQUATOR) 99753521 kvi 24938376(99753504)
17/05/02 02:18:56 INFO mapred.MapTask: Finished spill 48
17/05/02 02:18:56 INFO mapred.MapTask: (RESET) equator 99753521 kv 24938376(99753504) kvi 22730864(90923456)
17/05/02 02:18:57 INFO mapred.MapTask: Spilling map output
17/05/02 02:18:57 INFO mapred.MapTask: bufstart = 99753521; bufend = 46056088; bufvoid = 104857599
17/05/02 02:18:57 INFO mapred.MapTask: kvstart = 24938376(99753504); kvend = 16756904(67027616); length = 8181473/6553600
17/05/02 02:18:57 INFO mapred.MapTask: (EQUATOR) 54886184 kvi 13721540(54886160)
17/05/02 02:18:58 INFO mapred.LocalJobRunner: map > map
17/05/02 02:18:59 INFO mapreduce.Job:  map 36% reduce 0%
17/05/02 02:19:00 INFO mapred.MapTask: Finished spill 49
17/05/02 02:19:00 INFO mapred.MapTask: (RESET) equator 54886184 kv 13721540(54886160) kvi 11514028(46056112)
17/05/02 02:19:02 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:02 INFO mapred.MapTask: bufstart = 54886184; bufend = 104834095; bufvoid = 104857600
17/05/02 02:19:02 INFO mapred.MapTask: kvstart = 13721540(54886160); kvend = 5236996(20947984); length = 8484545/6553600
17/05/02 02:19:02 INFO mapred.MapTask: (EQUATOR) 8806575 kvi 2201636(8806544)
17/05/02 02:19:04 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:05 INFO mapreduce.Job:  map 39% reduce 0%
17/05/02 02:19:05 INFO mapred.MapTask: Finished spill 50
17/05/02 02:19:05 INFO mapred.MapTask: (RESET) equator 8806575 kv 2201636(8806544) kvi 1710860(6843440)
17/05/02 02:19:07 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:07 INFO mapred.MapTask: bufstart = 8806575; bufend = 79077092; bufvoid = 104857600
17/05/02 02:19:07 INFO mapred.MapTask: kvstart = 2201636(8806544); kvend = 25012144(100048576); length = 3403893/6553600
17/05/02 02:19:07 INFO mapred.MapTask: (EQUATOR) 87907172 kvi 21976788(87907152)
17/05/02 02:19:09 INFO mapred.MapTask: Finished spill 51
17/05/02 02:19:09 INFO mapred.MapTask: (RESET) equator 87907172 kv 21976788(87907152) kvi 21502184(86008736)
17/05/02 02:19:10 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:11 INFO mapreduce.Job:  map 45% reduce 0%
17/05/02 02:19:12 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:12 INFO mapred.MapTask: bufstart = 87907172; bufend = 54619440; bufvoid = 104857420
17/05/02 02:19:12 INFO mapred.MapTask: kvstart = 21976788(87907152); kvend = 18897724(75590896); length = 3079065/6553600
17/05/02 02:19:12 INFO mapred.MapTask: (EQUATOR) 63223104 kvi 15805772(63223088)
17/05/02 02:19:14 INFO mapred.MapTask: Finished spill 52
17/05/02 02:19:14 INFO mapred.MapTask: (RESET) equator 63223104 kv 15805772(63223088) kvi 15333264(61333056)
17/05/02 02:19:16 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:17 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:17 INFO mapred.MapTask: bufstart = 63223104; bufend = 30462156; bufvoid = 104857596
17/05/02 02:19:17 INFO mapred.MapTask: kvstart = 15805772(63223088); kvend = 12858420(51433680); length = 2947353/6553600
17/05/02 02:19:17 INFO mapred.MapTask: (EQUATOR) 39065836 kvi 9766452(39065808)
17/05/02 02:19:17 INFO mapreduce.Job:  map 52% reduce 0%
17/05/02 02:19:19 INFO mapred.MapTask: Finished spill 53
17/05/02 02:19:19 INFO mapred.MapTask: (RESET) equator 39065836 kv 9766452(39065808) kvi 9279284(37117136)
17/05/02 02:19:21 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:21 INFO mapred.MapTask: bufstart = 39065836; bufend = 4362548; bufvoid = 104857567
17/05/02 02:19:21 INFO mapred.MapTask: kvstart = 9766452(39065808); kvend = 6333508(25334032); length = 3432945/6553600
17/05/02 02:19:21 INFO mapred.MapTask: (EQUATOR) 12751124 kvi 3187776(12751104)
17/05/02 02:19:22 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:23 INFO mapreduce.Job:  map 60% reduce 0%
17/05/02 02:19:24 INFO mapred.MapTask: Finished spill 54
17/05/02 02:19:24 INFO mapred.MapTask: (RESET) equator 12751124 kv 3187776(12751104) kvi 2686436(10745744)
17/05/02 02:19:26 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:26 INFO mapred.MapTask: bufstart = 12751124; bufend = 83577219; bufvoid = 104857600
17/05/02 02:19:26 INFO mapred.MapTask: kvstart = 3187776(12751104); kvend = 26137180(104548720); length = 3264997/6553600
17/05/02 02:19:26 INFO mapred.MapTask: (EQUATOR) 91965811 kvi 22991448(91965792)
17/05/02 02:19:28 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:28 INFO mapred.MapTask: Finished spill 55
17/05/02 02:19:28 INFO mapred.MapTask: (RESET) equator 91965811 kv 22991448(91965792) kvi 22491844(89967376)
17/05/02 02:19:29 INFO mapreduce.Job:  map 66% reduce 0%
17/05/02 02:19:29 INFO mapred.LocalJobRunner: map > map
17/05/02 02:19:29 INFO mapred.MapTask: Starting flush of map output
17/05/02 02:19:29 INFO mapred.MapTask: Spilling map output
17/05/02 02:19:29 INFO mapred.MapTask: bufstart = 91965811; bufend = 8890854; bufvoid = 104857558
17/05/02 02:19:29 INFO mapred.MapTask: kvstart = 22991448(91965792); kvend = 22136352(88545408); length = 855097/6553600
17/05/02 02:19:29 INFO mapred.MapTask: Finished spill 56
17/05/02 02:19:29 INFO mapred.Merger: Merging 57 sorted segments
17/05/02 02:19:29 INFO mapred.Merger: Merging 3 intermediate segments out of a total of 57
17/05/02 02:19:29 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 55
17/05/02 02:19:30 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 46
17/05/02 02:19:30 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 37
17/05/02 02:19:31 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 28
17/05/02 02:19:32 INFO mapred.Merger: Merging 10 intermediate segments out of a total of 19
17/05/02 02:19:33 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 569297651 bytes
17/05/02 02:19:34 INFO mapred.LocalJobRunner: map > sort >
17/05/02 02:19:35 INFO mapreduce.Job:  map 78% reduce 0%
17/05/02 02:19:40 INFO mapred.LocalJobRunner: map > sort >
17/05/02 02:19:40 INFO mapred.Task: Task:attempt_local27049618_0001_m_000000_0 is done. And is in the process of committing
17/05/02 02:19:40 INFO mapred.LocalJobRunner: map > sort
17/05/02 02:19:40 INFO mapred.Task: Task 'attempt_local27049618_0001_m_000000_0' done.
17/05/02 02:19:40 INFO mapred.LocalJobRunner: Finishing task: attempt_local27049618_0001_m_000000_0
17/05/02 02:19:40 INFO mapred.LocalJobRunner: map task executor complete.
17/05/02 02:19:40 INFO mapred.LocalJobRunner: Waiting for reduce tasks
17/05/02 02:19:40 INFO mapred.LocalJobRunner: Starting task: attempt_local27049618_0001_r_000000_0
17/05/02 02:19:40 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/05/02 02:19:40 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/05/02 02:19:40 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/05/02 02:19:40 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6f996faf
17/05/02 02:19:40 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=652528832, maxSingleShuffleLimit=163132208, mergeThreshold=430669056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
17/05/02 02:19:40 INFO reduce.EventFetcher: attempt_local27049618_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
17/05/02 02:19:40 INFO reduce.MergeManagerImpl: attempt_local27049618_0001_m_000000_0: Shuffling to disk since 278465134 is greater than maxSingleShuffleLimit (163132208)
17/05/02 02:19:40 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local27049618_0001_m_000000_0 decomp: 278465134 len: 278465138 to DISK
17/05/02 02:19:41 INFO mapreduce.Job:  map 100% reduce 0%
17/05/02 02:19:41 INFO reduce.OnDiskMapOutput: Read 278465138 bytes from map-output for attempt_local27049618_0001_m_000000_0
17/05/02 02:19:41 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
17/05/02 02:19:41 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/05/02 02:19:41 INFO reduce.MergeManagerImpl: finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
17/05/02 02:19:41 INFO reduce.MergeManagerImpl: Merging 1 files, 278465138 bytes from disk
17/05/02 02:19:41 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
17/05/02 02:19:41 INFO mapred.Merger: Merging 1 sorted segments
17/05/02 02:19:41 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 278465130 bytes
17/05/02 02:19:41 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/05/02 02:19:41 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
17/05/02 02:19:50 INFO mapred.Task: Task:attempt_local27049618_0001_r_000000_0 is done. And is in the process of committing
17/05/02 02:19:50 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/05/02 02:19:50 INFO mapred.Task: Task attempt_local27049618_0001_r_000000_0 is allowed to commit now
17/05/02 02:19:50 INFO output.FileOutputCommitter: Saved output of task 'attempt_local27049618_0001_r_000000_0' to hdfs://mycluster/user/gooper/out/_temporary/0/task_local27049618_0001_r_000000
17/05/02 02:19:50 INFO mapred.LocalJobRunner: reduce > reduce
17/05/02 02:19:50 INFO mapred.Task: Task 'attempt_local27049618_0001_r_000000_0' done.
17/05/02 02:19:50 INFO mapred.LocalJobRunner: Finishing task: attempt_local27049618_0001_r_000000_0
17/05/02 02:19:50 INFO mapred.LocalJobRunner: reduce task executor complete.
17/05/02 02:19:51 INFO mapreduce.Job:  map 100% reduce 100%
17/05/02 02:19:51 INFO mapreduce.Job: Job job_local27049618_0001 completed successfully
17/05/02 02:19:51 INFO mapreduce.Job: Counters: 35
        File System Counters
                FILE: Number of bytes read=2201991558
                FILE: Number of bytes written=2481118468
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=859857874
                HDFS: Number of bytes written=266794847
                HDFS: Number of read operations=13
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=4
        Map-Reduce Framework
                Map input records=25042379
                Map output records=115592557
                Map output bytes=2873343481
                Map output materialized bytes=278465138
                Input split bytes=108
                Combine input records=122449942
                Combine output records=9417480
                Reduce input groups=2560095
                Reduce shuffle bytes=278465138
                Reduce input records=2560095
                Reduce output records=2560095
                Spilled Records=16617705
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=761
                Total committed heap usage (bytes)=542638080
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=429928937
        File Output Format Counters
                Bytes Written=266794847


-----------------hadoop-mapreduce-examples-2.8.0.jar의 wordcount가 정상적으로 실행되는 모습-----------

root@gsda1:~/hadoop/sbin# yarn jar $HOME/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar wordcount in out3
17/05/02 04:53:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/02 04:54:00 INFO input.FileInputFormat: Total input files to process : 3
17/05/02 04:54:00 INFO mapreduce.JobSubmitter: number of splits:3
17/05/02 04:54:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1493638357554_0001
17/05/02 04:54:01 INFO impl.YarnClientImpl: Submitted application application_1493638357554_0001
17/05/02 04:54:01 INFO mapreduce.Job: The url to track the job: http://gsda2:8088/proxy/application_1493638357554_0001/
17/05/02 04:54:01 INFO mapreduce.Job: Running job: job_1493638357554_0001
17/05/02 04:54:10 INFO mapreduce.Job: Job job_1493638357554_0001 running in uber mode : false
17/05/02 04:54:10 INFO mapreduce.Job:  map 0% reduce 0%
17/05/02 04:54:15 INFO mapreduce.Job:  map 33% reduce 0%
17/05/02 04:54:18 INFO mapreduce.Job:  map 67% reduce 0%
17/05/02 04:54:33 INFO mapreduce.Job:  map 67% reduce 22%
17/05/02 04:54:34 INFO mapreduce.Job:  map 68% reduce 22%
17/05/02 04:54:52 INFO mapreduce.Job:  map 69% reduce 22%
17/05/02 04:55:10 INFO mapreduce.Job:  map 70% reduce 22%
17/05/02 04:55:29 INFO mapreduce.Job:  map 71% reduce 22%
17/05/02 04:55:41 INFO mapreduce.Job:  map 72% reduce 22%
17/05/02 04:55:59 INFO mapreduce.Job:  map 73% reduce 22%
17/05/02 04:56:17 INFO mapreduce.Job:  map 74% reduce 22%
17/05/02 04:56:35 INFO mapreduce.Job:  map 75% reduce 22%
17/05/02 04:56:53 INFO mapreduce.Job:  map 76% reduce 22%
17/05/02 04:57:11 INFO mapreduce.Job:  map 77% reduce 22%
17/05/02 04:57:23 INFO mapreduce.Job:  map 78% reduce 22%
17/05/02 04:57:35 INFO mapreduce.Job:  map 79% reduce 22%
17/05/02 04:57:41 INFO mapreduce.Job:  map 80% reduce 22%
17/05/02 04:57:47 INFO mapreduce.Job:  map 82% reduce 22%
17/05/02 04:57:53 INFO mapreduce.Job:  map 85% reduce 22%
17/05/02 04:57:59 INFO mapreduce.Job:  map 87% reduce 22%
17/05/02 04:58:05 INFO mapreduce.Job:  map 89% reduce 22%
17/05/02 04:58:11 INFO mapreduce.Job:  map 94% reduce 22%
17/05/02 04:58:17 INFO mapreduce.Job:  map 100% reduce 22%
17/05/02 04:58:21 INFO mapreduce.Job:  map 100% reduce 71%
17/05/02 04:58:27 INFO mapreduce.Job:  map 100% reduce 97%
17/05/02 04:58:28 INFO mapreduce.Job:  map 100% reduce 100%
17/05/02 04:58:28 INFO mapreduce.Job: Job job_1493638357554_0001 completed successfully
17/05/02 04:58:28 INFO mapreduce.Job: Counters: 50
        File System Counters
                FILE: Number of bytes read=1108076402
                FILE: Number of bytes written=1394490956
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=433965095
                HDFS: Number of bytes written=266795644
                HDFS: Number of read operations=12
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Killed map tasks=1
                Launched map tasks=4
                Launched reduce tasks=1
                Data-local map tasks=4
                Total time spent by all maps in occupied slots (ms)=483097
                Total time spent by all reduces in occupied slots (ms)=250461
                Total time spent by all map tasks (ms)=483097
                Total time spent by all reduce tasks (ms)=250461
                Total vcore-milliseconds taken by all map tasks=483097
                Total vcore-milliseconds taken by all reduce tasks=250461
                Total megabyte-milliseconds taken by all map tasks=1978765312
                Total megabyte-milliseconds taken by all reduce tasks=1025888256
        Map-Reduce Framework
                Map input records=25071914
                Map output records=115663168
                Map output bytes=2880593627
                Map output materialized bytes=285847897
                Input split bytes=362
                Combine input records=122520553
                Combine output records=9485298
                Reduce input groups=2560097
                Reduce shuffle bytes=285847897
                Reduce input records=2627913
                Reduce output records=2560097
                Spilled Records=16753341
                Shuffled Maps =3
                Failed Shuffles=0
                Merged Map outputs=3
                GC time elapsed (ms)=1042
                CPU time spent (ms)=292030
                Physical memory (bytes) snapshot=2016387072
                Virtual memory (bytes) snapshot=21047717888
                Total committed heap usage (bytes)=1123024896
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=433964733
        File Output Format Counters
                Bytes Written=266795644

이 게시물을
목록
번호 제목 글쓴이 날짜 조회 수
129 [CDP7.1.7, Hive Replication]Hive Replication진행중 "The following columns have types incompatible with the existing columns in their respective positions " 오류 gooper 2023.12.27 7
128 [CDP7.1.7]Oozie job에서 ERROR: Kudu error(s) reported, first error: Timed out: Failed to write batch of 774 ops to tablet 8003f9a064bf4be5890a178439b2ba91가 발생하면서 쿼리가 실패하는 경우 gooper 2024.01.05 7
127 [CDP7.1.7]impala-shell수행시 간헐적으로 "-k requires a valid kerberos ticket but no valid kerberos ticket found." 오류 gooper 2023.11.16 11
126 임시 테이블에서 데이터를 읽어서 partitioned table에 입력하는 impala SQL문 예시 gooper 2023.11.10 15
125 [impala]insert into db명.table명 select a, b from db명.table명 쿼리 수행시 "Memory limit exceeded: Failed to allocate memory for Parquet page index"오류 조치 방법 gooper 2023.05.31 21
124 not leader of this config: current role FOLLOWER 오류 발생시 확인방법 총관리자 2022.01.17 23
123 kudu table와 impala(hive) table정보가 틀어져서 테이블을 읽지 못하는 경우(Error Loading Metadata) 조치방법 gooper 2023.11.10 24
122 [CDP7.1.7]Impala Query의 Memory Spilled 양은 ScratchFileUsedBytes값을 누적해서 구할 수 있다. gooper 2022.07.29 28
121 [Cloudera 6.3.4, Kudu]]Service Monitor에서 사용하는 metric중에 일부를 blacklist로 설정하여 모니터링 정보 수집 제외하는 방법 gooper 2022.07.08 31
120 Failed to write to server: (no server available): 총관리자 2022.01.17 32
119 AnalysisException: Incomplatible return type 'DECIMAL(38,0)' and 'DECIMAL(38,5)' of exprs가 발생시 조치 총관리자 2021.07.26 33
118 [KUDU] kudu tablet server여러가지 원인에 의해서 corrupted상태가 된 경우 복구방법 gooper 2023.03.28 34
117 spark에서 hive table을 읽어 출력하는 예제 소스 총관리자 2017.03.09 35
116 [TLS/SSL]Kudu Tablet Server설정 총관리자 2022.05.13 35
115 spark에서 hive table을 읽어 출력하는 예제 소스 총관리자 2017.03.09 37
114 [CDP7.1.7]impala-shell을 이용하여 kudu table에 insert/update수행시 발생하는 오류(Transport endpoint is not connected (error 107)) 발생시 확인할 내용 gooper 2023.11.30 40
113 spark 온라인 책자링크 (제목 : mastering-apache-spark) 총관리자 2016.05.25 48
112 [Impala jdbc]CDP7.1.7환경에서 java프로그램을 이용하여 kerberized impala cluster에 접근하여 SQL을 수행하는 방법 gooper 2023.08.22 53
111 [hive] hive.tbls테이블의 owner컬럼값은 hadoop.security.auth_to_local에 의해서 filtering된다. 총관리자 2022.04.14 55
110 [TLS/SSL]Kudu Master 설정하기 총관리자 2022.05.13 61
쓰기 태그

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로