Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다.
문의사항은 gooper@gooper.com로 메일을
보내주세요.
-------- WF(명칭 : wf_work_all)작업 순서및 내용 ------------
1번. sqoop : action_id->30d9, sqoop작업
2번. shell : action_id->5673, log_process.sh
3번. wf#1 : action_id->0de8, wf_work_compute_template
--> shell: action_id->5362, shell script
4번. wf#2 : action_id->2229, wf_work_compute_template
--> shell, action_id->fc71, shell script
---------------위의 WF에서 1,2번에 대한 내용만 아래에 설명함----------------------------------
<<1번 sqoop type의 wf job 수행시 작업 진행과정>>
1. oozie:launcher -> oozie:action 순으로 각각의 application을 생성하여 작업되며 application id도 각각 다르게 부여됨
2-1. oozie:launcher(application_1646905237381_141158, oozie:launcher:T=sqoop:W=wf_work_all:A=sqoop-30d9:ID=0067471-220310190308179-oozie-oozi-W, type : Oozie Launcher)
application_1646905237381_141158 ->
appattempt_1646905237381_141158_000001 (from YARN-> Application list ->Application Details) ->
container_e228_1646905237381_141158_01_000001 (application_id와 같은 값을 이용한 id, 1st Attempt)
2-2. oozie:action ( application_1646905237381_141163, oozie:action:T=sqoop:W=wf_work_all:A=sqoop-30d9:ID=0067471-220310190308179-oozie-oozi-W, type : MAPREDUCE)
application_1646905237381_141163 ->
appattempt_1646905237381_141163_000001 (from YARN-> Application list ->Application Details) ->
job_1646905237381_141163 (Hadoop Job IDs executed by Sqoop) ->
task_1646905237381_141163_m_000000 (MAP task) ->
attempt_1646905237381_141163_m_000000_0 (1st Attempt) ->
container_e228_1646905237381_141163_01_000002
<<2번 shell type의 wf job 수행시 작업 진행과정>>
1. oozie:launcher를 띄워서 바로 shellscript를 수행하며 별도의 job_은 수행되지 않음
2-1. oozie:launcher(application_1646905237381_141188, oozie:launcher:T=shell:W=wf_work_all:A=shell-5673:ID=0067471-220310190308179-oozie-oozi-W, type : Oozie Launcher)
application_1646905237381_141188 ->
appattempt_1646905237381_141188_000001 (from YARN-> Application list ->Application Details) ->
container_e228_1646905237381_141188_01_000001