메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


Setting up RStudio Server to run with Apache Spark

https://markobigdata.com/2016/03/27/setting-up-rstudio-server-to-run-with-apache-spark/



Zeppelin supports Spark, PySpark, Spark R, Spark SQL with dependency loader.

Python is supported with Matplotlib, Conda, Pandas SQL and PySpark integrations.

http://zeppelin.apache.org/


Jupyter kernels

https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

Here is a list of available kernels. If you are writing your own kernel, feel free to add it to the table!


NameJupyter/IPython VersionLanguage(s) Version3rd party dependenciesExample NotebooksNotes
Coarray-FortranJupyter 4.0Fortran 2008/2015GFortran >= 7.1, OpenCoarraysMPICH >= 3.2DemoBinder demoDocker image
sparkmagicJupyter >=4.0Pyspark (Python 2 & 3), Spark (Scala), SparkR (R)LivyNotebooksDocker ImagesThis kernels are implemented via the magics machinery of the ipython kernel to use Spark via Livy
sas_kernelJupyter 4.0python >= 3.3SAS 9.4 or higher
IPyKernelJupyter 4.0python 2.7, >= 3.3pyzmq
IJuliajulia >= 0.3
IHaskellghc >= 7.6
IRubyruby >= 2.1
IJavascriptnodejs >= 0.10
jpCoffeescriptcoffeescript >= 1.7
ICSharpJupyter 4.0C# 4.0+scriptcs
IRKernelIPython 3.0R 3.2rzmq
SageMathJupyter 4Anymany
pari_jupyterJupyter 4PARI/GP >= 2.9
IFSharpIPython 2.0F#Features
lgoJupyter >= 4, JupyterLabGo >= 1.8ZeroMQ (4.x)ExampleDocker image
gopherlabJupyter 4.1, JupyterLabGo >= 1.6ZeroMQ (4.x)examplesDeprecated, use gophernotes
GophernotesJupyter 4, JupyterLab, nteractGo >= 1.9ZeroMQ 4.x.xexamplesdocker image
IGoGo >= 1.4
IScalaScala
Jupyter-scalaIPython>=3.0Scala>=2.10example
IErlangIPython 2.3Erlangrebar
ITorchIPython >= 2.2 and <= 5.xTorch 7 (LuaJIT)
IElixirJupyter < 6.0Elixir < 1.5Erlang OTP <= 19.3, RebarexampleIElixir Notebook in Docker
ierlJupyter >= 4.0Erlang 19 or 20, Elixir 1.4 or 1.5, LFE 1.2Erlang, (optional) Elixir
IAldorIPython >= 1Aldor
IOCamlIPython >= 1.1OCaml >= 4.01opam
OCaml-JupyterJupyter >= 4.0OCaml >= 4.02opamExampleDocker image
IForthIPython >= 3Forth
peforthIPython 6/Jupyter 5ForthExamplepython debugger in FORTH syntax
IPerlPerl 5
Perl6Jupyter >= 4Perl 6.czeromq 4
IPerl6Perl 6
Jupyter-Perl6JupyterPerl 6.CRakudo Perl 6
IPHPIPython >= 2PHP >= 5.4composerDEPRECATED, use Jupyter-PHP
Jupyter-PHPJupyter 4.0PHP >= 7.0.0composer, php-zmq
IOctaveJupyterOctaveExampleMetaKernel
IScilabJupyterScilabExampleMetaKernel
MATLAB KernelJupyterMatlabpymatbridgeExampleMetaKernel
BashIPython >= 3bashWrapper
PowerShellIPython >= 3WindowsWrapper, Based on Bash Kernel
CloJupyterJupyterClojure >= 1.7
CLJ-JupyterJupyterClojureAbandoned as of 2017-02-12
jupyter-kernel-jsr223Jupyter>=4.0Clojure 1.8clojure-jrs223, Java>=7Java based JSR223 compliant
Hy KernelJupyterHyTutorialtreats Hy as Python pre-processor
Calysto HyJupyterHyTutorialbased on MetaKernel (magics, shell, parallel, etc.)
Redis KernelIPython >= 3redisWrapper
joveio.js
jp-babelJupyterBabel
ICalicoIPython >= 2multipleIndex
IMathicsMathics
IWolframWolfram MathematicaWolfram Mathematica(R), MetakernelMetaKernel
Lua KernelLua
IPurescriptPurescript
IPyLuaLuaFork of Lua Kernel
Calysto SchemeSchemeReference GuideMetaKernel
Calysto ProcessingProcessing.js >= 2MetaKernel
idl_kernelIDLIDL seem to have a built-in kernelstarting with version 8.5
Mochi KernelMochi
Lua (used in Splash)Lua
Apache Toree (formerly Spark Kernel)JupyterScala, Python, RSpark >= 1.5Example
Skulpt Python KernelSkulpt PythonExamplesMetaKernel
MetaKernel BashbashMetaKernel
MetaKernel PythonpythonMetaKernel
IVisualVPythonBall-in-Box
IBrainfuckBrainfuckDemoWrapper
KDB+/Q Kernel (IKdbQ)IPython >= 3.1Qqzmq, qcrypt
KDB+/Q Kernel (KdbQ Kernel)JupyterQ
ICryptolCryptolCVC4
clingJupyter 4C++Example
XonshXonshExampleMetaKernel
PrologPrologMetaKernel
cl-jupyterJupyterCommon LispQuicklispAbout
Maxima-JupyterJupyterMaximaQuicklisp
Calysto LC3Assembly Language for the Little Computer 3
YacasYACAS
IJythonJython 2.7
ROOTJupyterC++/pythonROOT >= 6.05
Gnuplot KernelGnuplotExampleMetaKernel
TclJupyterTcl 8.5Based on Bash Kernel
JJupyterJ 805Examples
JythonJupyter>=4.0Jython>=2.7.0Java>=7Java based JSR223 compliant
CJupyterCgcc
TaQLJupyterTaQLpython-casacoreTaQL tutorial
CoconutJupyterCoconut
SPARQLJupyter 4Python 2.7 or >=3.4rdflibSPARQLWrapperExamplesOptional GraphVizdependency
AIML chatbotJupyter 4Python 2.7pyAIMLExamples
IArmJupyter 4ARMv6 THUMBExamplesBased off of the ARM Cortex M0+ CPU
SoSJupyter 4Python >=3.4ExamplesWorkflow system, Multi-Kernel support
jupyter-nodejsJupyter, iPython 3.xNodeJS, Babel, ClojurescriptExamples
PikeIPython >= 3Pike >= 7.8Wrapper, Based on Bash Kernel
ITypeScriptTypescript >= 2.0Node.js >= 0.10.0
imatlabipykernel >= 4.1MATLAB >= 2016b
jupyter-kotlinJupyterKotlin 1.1-M04 EAPJava >= 8
jupyter_kernel_singularJupyterSingular 4.1.0DemoOptional PySingular for better performance, surf for images, details
spylon-kernelipykernel >=4.5python >= 3.5, scala >= 2.11Apache Spark >=2.0ExampleMetaKernel
mit-scheme-kernelJupyter 4.0MIT Scheme 9.2
elm-kernelJupyterExamples
SciJava Jupyter KernelJupyter 4.3.0Java + 9 scripting languagesJavaExamples
IsbtJupyter 4.3.0sbt >= 1.0.0sbtexample
BeakerXGroovy, Java, Scala, Clojure, Kotlin, SQLexampledocker image
MicroPythonJupyterESP8266/ESP32USB or Webrepldeveloper notebooksrelies on the micro-controller's paste-mode
IJavaJupyterJava 9Java JDK >= 9Binder online demoBased on the new JShell tool
GuileJupyter 5.2Guile 2.0guile-json, openssl
번호 제목 날짜 조회 수
710 [Ranger]RangerAdminRESTClient Error gertting pplicies; Received NULL response!!, secureMode=true, user=rangerkms/node01.gooper.com@ GOOPER.COM (auth:KERBEROS), serviceName=cm_kms 2023.06.27 3518
709 [CDP7.1.3]Ranger WebUI에서 Error! Connection refused: Please check the KMS provider URL and whether the Ranager KMS is running발생시 조치 방법 2023.06.07 4689
708 [impala]insert into db명.table명 select a, b from db명.table명 쿼리 수행시 "Memory limit exceeded: Failed to allocate memory for Parquet page index"오류 조치 방법 2023.05.31 6326
707 Impala Admission Control 설정시 쿼리가 사용하는 메모리 사용량 판단 방법 2023.05.19 4842
706 [Atlas Server]org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=atlas/node01.gooper.com@GOOPER.COM, scope=default:atlas_janus, params=[table=default:atlas_janus,], action-CREATE)] 2023.05.15 4520
705 [Solr in Cloudera]Solr Data Directory변경 방법/절차 2023.04.21 4394
704 [Ranger]계정에 admin권한(grant, create등)의 권한 부여 방법 2023.04.18 4298
703 [DataNode]org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hdfs/datanode03@GOOPER.COM from keytab hdfs.keytab오류 2023.04.18 9824
702 [KUDU] kudu tablet server여러가지 원인에 의해서 corrupted상태가 된 경우 복구방법 2023.03.28 4883
701 Hadoop Clsuter에 이미 포함된 host의 hostname변경시 처리 절차 2023.03.24 4446
700 [Kudu]ERROR: Unable to advance iterator for node with id '2' for Kudu table 'impala::core.pm0_abdasubjct': Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) 2023.03.16 5211
699 hive의 메타정보 테이블을 MariaDB로 사용하는 경우 table comment나 column comment에 한글 입력시 깨지는 경우 utf8로 바꾸는 방법. 2023.03.10 4810
698 [Hive canary]Hive에 Metastore canary red alert및 hive log파일에 Duplicate entry '123456' for key 'NOTIFICATION_LOG_EVENT_ID'가 발생시 조치사항 2023.03.10 4779
697 호출 url현황 2023.02.21 2344
696 ./gradlew :composeDown 및 ./gradlew :composeUp 를 성공했을때의 메세지 2023.02.20 4414
695 [ftgo_application]Unable to infer base url오류 발생시 조치방법 2023.02.20 4826
694 [Impala TLS/SSL이슈]RangerAdminRESTClient.java:151] Failed to get response, Error is: TrustManager is not specified 2023.02.02 4339
693 [Kerberos]병렬 kinit 호출시 cache파일이 손상되어 Bad format in credentials cache 혹은 No credentials cache found 혹은 Internal credentials cache error 오류 발생시 2023.01.20 4584
692 [HA구성 이슈]oozie 2대를 L4로 HA구성했을때 발생하는 이슈 2023.01.17 4903
691 [Impala 3.2버젼]compute incremental stats db명.테이블명 수행시 ERROR: AnalysisException: Incremental stats size estimate exceeds 2000.00MB. 오류 발생원인및 조치방안 2022.11.30 4845
위로