메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


Setting up RStudio Server to run with Apache Spark

https://markobigdata.com/2016/03/27/setting-up-rstudio-server-to-run-with-apache-spark/



Zeppelin supports Spark, PySpark, Spark R, Spark SQL with dependency loader.

Python is supported with Matplotlib, Conda, Pandas SQL and PySpark integrations.

http://zeppelin.apache.org/


Jupyter kernels

https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

Here is a list of available kernels. If you are writing your own kernel, feel free to add it to the table!


NameJupyter/IPython VersionLanguage(s) Version3rd party dependenciesExample NotebooksNotes
Coarray-FortranJupyter 4.0Fortran 2008/2015GFortran >= 7.1, OpenCoarraysMPICH >= 3.2DemoBinder demoDocker image
sparkmagicJupyter >=4.0Pyspark (Python 2 & 3), Spark (Scala), SparkR (R)LivyNotebooksDocker ImagesThis kernels are implemented via the magics machinery of the ipython kernel to use Spark via Livy
sas_kernelJupyter 4.0python >= 3.3SAS 9.4 or higher
IPyKernelJupyter 4.0python 2.7, >= 3.3pyzmq
IJuliajulia >= 0.3
IHaskellghc >= 7.6
IRubyruby >= 2.1
IJavascriptnodejs >= 0.10
jpCoffeescriptcoffeescript >= 1.7
ICSharpJupyter 4.0C# 4.0+scriptcs
IRKernelIPython 3.0R 3.2rzmq
SageMathJupyter 4Anymany
pari_jupyterJupyter 4PARI/GP >= 2.9
IFSharpIPython 2.0F#Features
lgoJupyter >= 4, JupyterLabGo >= 1.8ZeroMQ (4.x)ExampleDocker image
gopherlabJupyter 4.1, JupyterLabGo >= 1.6ZeroMQ (4.x)examplesDeprecated, use gophernotes
GophernotesJupyter 4, JupyterLab, nteractGo >= 1.9ZeroMQ 4.x.xexamplesdocker image
IGoGo >= 1.4
IScalaScala
Jupyter-scalaIPython>=3.0Scala>=2.10example
IErlangIPython 2.3Erlangrebar
ITorchIPython >= 2.2 and <= 5.xTorch 7 (LuaJIT)
IElixirJupyter < 6.0Elixir < 1.5Erlang OTP <= 19.3, RebarexampleIElixir Notebook in Docker
ierlJupyter >= 4.0Erlang 19 or 20, Elixir 1.4 or 1.5, LFE 1.2Erlang, (optional) Elixir
IAldorIPython >= 1Aldor
IOCamlIPython >= 1.1OCaml >= 4.01opam
OCaml-JupyterJupyter >= 4.0OCaml >= 4.02opamExampleDocker image
IForthIPython >= 3Forth
peforthIPython 6/Jupyter 5ForthExamplepython debugger in FORTH syntax
IPerlPerl 5
Perl6Jupyter >= 4Perl 6.czeromq 4
IPerl6Perl 6
Jupyter-Perl6JupyterPerl 6.CRakudo Perl 6
IPHPIPython >= 2PHP >= 5.4composerDEPRECATED, use Jupyter-PHP
Jupyter-PHPJupyter 4.0PHP >= 7.0.0composer, php-zmq
IOctaveJupyterOctaveExampleMetaKernel
IScilabJupyterScilabExampleMetaKernel
MATLAB KernelJupyterMatlabpymatbridgeExampleMetaKernel
BashIPython >= 3bashWrapper
PowerShellIPython >= 3WindowsWrapper, Based on Bash Kernel
CloJupyterJupyterClojure >= 1.7
CLJ-JupyterJupyterClojureAbandoned as of 2017-02-12
jupyter-kernel-jsr223Jupyter>=4.0Clojure 1.8clojure-jrs223, Java>=7Java based JSR223 compliant
Hy KernelJupyterHyTutorialtreats Hy as Python pre-processor
Calysto HyJupyterHyTutorialbased on MetaKernel (magics, shell, parallel, etc.)
Redis KernelIPython >= 3redisWrapper
joveio.js
jp-babelJupyterBabel
ICalicoIPython >= 2multipleIndex
IMathicsMathics
IWolframWolfram MathematicaWolfram Mathematica(R), MetakernelMetaKernel
Lua KernelLua
IPurescriptPurescript
IPyLuaLuaFork of Lua Kernel
Calysto SchemeSchemeReference GuideMetaKernel
Calysto ProcessingProcessing.js >= 2MetaKernel
idl_kernelIDLIDL seem to have a built-in kernelstarting with version 8.5
Mochi KernelMochi
Lua (used in Splash)Lua
Apache Toree (formerly Spark Kernel)JupyterScala, Python, RSpark >= 1.5Example
Skulpt Python KernelSkulpt PythonExamplesMetaKernel
MetaKernel BashbashMetaKernel
MetaKernel PythonpythonMetaKernel
IVisualVPythonBall-in-Box
IBrainfuckBrainfuckDemoWrapper
KDB+/Q Kernel (IKdbQ)IPython >= 3.1Qqzmq, qcrypt
KDB+/Q Kernel (KdbQ Kernel)JupyterQ
ICryptolCryptolCVC4
clingJupyter 4C++Example
XonshXonshExampleMetaKernel
PrologPrologMetaKernel
cl-jupyterJupyterCommon LispQuicklispAbout
Maxima-JupyterJupyterMaximaQuicklisp
Calysto LC3Assembly Language for the Little Computer 3
YacasYACAS
IJythonJython 2.7
ROOTJupyterC++/pythonROOT >= 6.05
Gnuplot KernelGnuplotExampleMetaKernel
TclJupyterTcl 8.5Based on Bash Kernel
JJupyterJ 805Examples
JythonJupyter>=4.0Jython>=2.7.0Java>=7Java based JSR223 compliant
CJupyterCgcc
TaQLJupyterTaQLpython-casacoreTaQL tutorial
CoconutJupyterCoconut
SPARQLJupyter 4Python 2.7 or >=3.4rdflibSPARQLWrapperExamplesOptional GraphVizdependency
AIML chatbotJupyter 4Python 2.7pyAIMLExamples
IArmJupyter 4ARMv6 THUMBExamplesBased off of the ARM Cortex M0+ CPU
SoSJupyter 4Python >=3.4ExamplesWorkflow system, Multi-Kernel support
jupyter-nodejsJupyter, iPython 3.xNodeJS, Babel, ClojurescriptExamples
PikeIPython >= 3Pike >= 7.8Wrapper, Based on Bash Kernel
ITypeScriptTypescript >= 2.0Node.js >= 0.10.0
imatlabipykernel >= 4.1MATLAB >= 2016b
jupyter-kotlinJupyterKotlin 1.1-M04 EAPJava >= 8
jupyter_kernel_singularJupyterSingular 4.1.0DemoOptional PySingular for better performance, surf for images, details
spylon-kernelipykernel >=4.5python >= 3.5, scala >= 2.11Apache Spark >=2.0ExampleMetaKernel
mit-scheme-kernelJupyter 4.0MIT Scheme 9.2
elm-kernelJupyterExamples
SciJava Jupyter KernelJupyter 4.3.0Java + 9 scripting languagesJavaExamples
IsbtJupyter 4.3.0sbt >= 1.0.0sbtexample
BeakerXGroovy, Java, Scala, Clojure, Kotlin, SQLexampledocker image
MicroPythonJupyterESP8266/ESP32USB or Webrepldeveloper notebooksrelies on the micro-controller's paste-mode
IJavaJupyterJava 9Java JDK >= 9Binder online demoBased on the new JShell tool
GuileJupyter 5.2Guile 2.0guile-json, openssl
번호 제목 날짜 조회 수
650 [Hue metadata]Oracle에 있는 Hue 메타정보 테이블을 이용하여 coordinator와 workflow관계 목록을 추출하는 방법 2023.08.22 2860
649 [SparkR]SparkR 설치 사용기 1 - Installation Guide On Yarn Cluster & Mesos Cluster & Stand Alone Cluster file 2016.11.04 2862
648 S2RDF 테스트(벤치마크 테스트를 기준으로 python, scala소스가 만들어져서 기능은 파악되지 못함) [1] file 2016.05.27 2864
647 Could not configure server becase SASL configuration did not allow the Zookeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Checksum failed 2019.05.18 2870
646 hbase startrow와 endrow를 지정하여 검색하기 샘플 2016.12.07 2876
645 anaconda3(v5.4)를 이용하여 tensorflow설치후 ipython프로그램을 실행하여 import할때 오류발생시 조치 2018.07.27 2879
644 S2RDF를 이용한 다른 버젼의 github링크 2016.12.02 2881
643 서버 5대에 solr 5.5.0 설치하고 index data를 HDFS에 저장/search하도록 설치/설정하는 방법(SolrCloud) 2016.04.08 2907
642 elasticsearch 1.3.0에서 rdf및 hadoop plugin설치 2016.04.06 2919
641 elasticsearch에서 모든 인덱스(색인)을 삭제하는 방법 2019.06.09 2919
640 초기 오류(java.lang.NoSuchMethodError)에 따른 후속 작업에서 오류(java.lang.NoClassDefFoundError)가 발생되는 상황(quartz에서 주기적으로 작업시) 2016.08.29 2922
639 python 2.6.6에서 print 'A=' 형태의 사용이 python 3.5.1에서 오류(SyntaxError: Missing parentheses in call to 'print') 발생함.. 2016.05.27 2928
638 천문학적, 기후학적, 기상학적, 생물학적, 농사계절 구분 2015.12.16 2931
637 jena의 data폴더를 hadoop nfs를 이용하여 HDFS상의 폴더에 마운트 시키고 fuseki를 통하여 inert를 시도했을때 transaction 오류 발생 2016.12.02 2940
636 oneM2M Specification(Draft Release 3, 2, 1), Draft Technical Reports 2017.10.25 2940
635 [Impala] alter table구문수행시 "WARNINGS: Impala does not have READ_WRITE access to path 'hdfs://nameservice1/DATA/Temp/DB/source/table01_ccd'" 발생시 조치 2024.04.26 2940
634 lagom-windows용 build.sbt파일 내용 2017.10.12 2946
633 Incompatible clusterIDs오류 원인및 해결방법 2016.04.01 2948
632 Elastic Search For Hadoop 2.2.0설치하기(5대 클러스터링) 2016.04.04 2952
631 spark-shell실행시 "A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection."오류가 발생하는 경우 해결방법 2016.05.20 2953
위로