메뉴 건너뛰기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


아래와 같은 오류가 발생하면 yarn-site.xml의 yarn.scheduler.maximum-allocation-mb 와 yarn.nodemanager.resource.memory-mb값을 Required executor memory (14336+1433 MB)이상으로 잡아줘야 정상적으로 수행된다.


-------------------------------오류내용----------------------------------------------------------
Caused by: java.lang.IllegalArgumentException: Required executor memory (14336+1433 MB) is above the max threshold (12288 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
        at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:283)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:139)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로