메뉴 건너뛰기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


Setting up RStudio Server to run with Apache Spark

https://markobigdata.com/2016/03/27/setting-up-rstudio-server-to-run-with-apache-spark/



Zeppelin supports Spark, PySpark, Spark R, Spark SQL with dependency loader.

Python is supported with Matplotlib, Conda, Pandas SQL and PySpark integrations.

http://zeppelin.apache.org/


Jupyter kernels

https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

Here is a list of available kernels. If you are writing your own kernel, feel free to add it to the table!


NameJupyter/IPython VersionLanguage(s) Version3rd party dependenciesExample NotebooksNotes
Coarray-FortranJupyter 4.0Fortran 2008/2015GFortran >= 7.1, OpenCoarraysMPICH >= 3.2DemoBinder demoDocker image
sparkmagicJupyter >=4.0Pyspark (Python 2 & 3), Spark (Scala), SparkR (R)LivyNotebooksDocker ImagesThis kernels are implemented via the magics machinery of the ipython kernel to use Spark via Livy
sas_kernelJupyter 4.0python >= 3.3SAS 9.4 or higher
IPyKernelJupyter 4.0python 2.7, >= 3.3pyzmq
IJuliajulia >= 0.3
IHaskellghc >= 7.6
IRubyruby >= 2.1
IJavascriptnodejs >= 0.10
jpCoffeescriptcoffeescript >= 1.7
ICSharpJupyter 4.0C# 4.0+scriptcs
IRKernelIPython 3.0R 3.2rzmq
SageMathJupyter 4Anymany
pari_jupyterJupyter 4PARI/GP >= 2.9
IFSharpIPython 2.0F#Features
lgoJupyter >= 4, JupyterLabGo >= 1.8ZeroMQ (4.x)ExampleDocker image
gopherlabJupyter 4.1, JupyterLabGo >= 1.6ZeroMQ (4.x)examplesDeprecated, use gophernotes
GophernotesJupyter 4, JupyterLab, nteractGo >= 1.9ZeroMQ 4.x.xexamplesdocker image
IGoGo >= 1.4
IScalaScala
Jupyter-scalaIPython>=3.0Scala>=2.10example
IErlangIPython 2.3Erlangrebar
ITorchIPython >= 2.2 and <= 5.xTorch 7 (LuaJIT)
IElixirJupyter < 6.0Elixir < 1.5Erlang OTP <= 19.3, RebarexampleIElixir Notebook in Docker
ierlJupyter >= 4.0Erlang 19 or 20, Elixir 1.4 or 1.5, LFE 1.2Erlang, (optional) Elixir
IAldorIPython >= 1Aldor
IOCamlIPython >= 1.1OCaml >= 4.01opam
OCaml-JupyterJupyter >= 4.0OCaml >= 4.02opamExampleDocker image
IForthIPython >= 3Forth
peforthIPython 6/Jupyter 5ForthExamplepython debugger in FORTH syntax
IPerlPerl 5
Perl6Jupyter >= 4Perl 6.czeromq 4
IPerl6Perl 6
Jupyter-Perl6JupyterPerl 6.CRakudo Perl 6
IPHPIPython >= 2PHP >= 5.4composerDEPRECATED, use Jupyter-PHP
Jupyter-PHPJupyter 4.0PHP >= 7.0.0composer, php-zmq
IOctaveJupyterOctaveExampleMetaKernel
IScilabJupyterScilabExampleMetaKernel
MATLAB KernelJupyterMatlabpymatbridgeExampleMetaKernel
BashIPython >= 3bashWrapper
PowerShellIPython >= 3WindowsWrapper, Based on Bash Kernel
CloJupyterJupyterClojure >= 1.7
CLJ-JupyterJupyterClojureAbandoned as of 2017-02-12
jupyter-kernel-jsr223Jupyter>=4.0Clojure 1.8clojure-jrs223, Java>=7Java based JSR223 compliant
Hy KernelJupyterHyTutorialtreats Hy as Python pre-processor
Calysto HyJupyterHyTutorialbased on MetaKernel (magics, shell, parallel, etc.)
Redis KernelIPython >= 3redisWrapper
joveio.js
jp-babelJupyterBabel
ICalicoIPython >= 2multipleIndex
IMathicsMathics
IWolframWolfram MathematicaWolfram Mathematica(R), MetakernelMetaKernel
Lua KernelLua
IPurescriptPurescript
IPyLuaLuaFork of Lua Kernel
Calysto SchemeSchemeReference GuideMetaKernel
Calysto ProcessingProcessing.js >= 2MetaKernel
idl_kernelIDLIDL seem to have a built-in kernelstarting with version 8.5
Mochi KernelMochi
Lua (used in Splash)Lua
Apache Toree (formerly Spark Kernel)JupyterScala, Python, RSpark >= 1.5Example
Skulpt Python KernelSkulpt PythonExamplesMetaKernel
MetaKernel BashbashMetaKernel
MetaKernel PythonpythonMetaKernel
IVisualVPythonBall-in-Box
IBrainfuckBrainfuckDemoWrapper
KDB+/Q Kernel (IKdbQ)IPython >= 3.1Qqzmq, qcrypt
KDB+/Q Kernel (KdbQ Kernel)JupyterQ
ICryptolCryptolCVC4
clingJupyter 4C++Example
XonshXonshExampleMetaKernel
PrologPrologMetaKernel
cl-jupyterJupyterCommon LispQuicklispAbout
Maxima-JupyterJupyterMaximaQuicklisp
Calysto LC3Assembly Language for the Little Computer 3
YacasYACAS
IJythonJython 2.7
ROOTJupyterC++/pythonROOT >= 6.05
Gnuplot KernelGnuplotExampleMetaKernel
TclJupyterTcl 8.5Based on Bash Kernel
JJupyterJ 805Examples
JythonJupyter>=4.0Jython>=2.7.0Java>=7Java based JSR223 compliant
CJupyterCgcc
TaQLJupyterTaQLpython-casacoreTaQL tutorial
CoconutJupyterCoconut
SPARQLJupyter 4Python 2.7 or >=3.4rdflibSPARQLWrapperExamplesOptional GraphVizdependency
AIML chatbotJupyter 4Python 2.7pyAIMLExamples
IArmJupyter 4ARMv6 THUMBExamplesBased off of the ARM Cortex M0+ CPU
SoSJupyter 4Python >=3.4ExamplesWorkflow system, Multi-Kernel support
jupyter-nodejsJupyter, iPython 3.xNodeJS, Babel, ClojurescriptExamples
PikeIPython >= 3Pike >= 7.8Wrapper, Based on Bash Kernel
ITypeScriptTypescript >= 2.0Node.js >= 0.10.0
imatlabipykernel >= 4.1MATLAB >= 2016b
jupyter-kotlinJupyterKotlin 1.1-M04 EAPJava >= 8
jupyter_kernel_singularJupyterSingular 4.1.0DemoOptional PySingular for better performance, surf for images, details
spylon-kernelipykernel >=4.5python >= 3.5, scala >= 2.11Apache Spark >=2.0ExampleMetaKernel
mit-scheme-kernelJupyter 4.0MIT Scheme 9.2
elm-kernelJupyterExamples
SciJava Jupyter KernelJupyter 4.3.0Java + 9 scripting languagesJavaExamples
IsbtJupyter 4.3.0sbt >= 1.0.0sbtexample
BeakerXGroovy, Java, Scala, Clojure, Kotlin, SQLexampledocker image
MicroPythonJupyterESP8266/ESP32USB or Webrepldeveloper notebooksrelies on the micro-controller's paste-mode
IJavaJupyterJava 9Java JDK >= 9Binder online demoBased on the new JShell tool
GuileJupyter 5.2Guile 2.0guile-json, openssl
번호 제목 글쓴이 날짜 조회 수
577 oozie 4.1 설치 - maven을 이용한 source compile on hadoop 2.5.2 with postgresql 9.3 총관리자 2015.04.30 860
576 ubuntu에 maven 3.6.1설치 및 환경변수 설정 총관리자 2019.06.02 856
575 Impala의 Queries탭에서 여러조건으로 쿼리 찾기 총관리자 2018.05.09 842
574 [SBT] SBT 사용법 정리(링크) 총관리자 2016.08.04 838
573 dual table만들기 총관리자 2014.05.16 812
572 [shellscript]엑셀파일에서 여러줄에 존재하는 단어를 한줄의 문자열로 합치는 방법(comma로 구분) 총관리자 2019.07.15 811
571 Hive JDBC Connection과 유형별 에러및 필요한 jar파일 총관리자 2021.05.24 810
570 update 샘플 총관리자 2018.03.12 810
569 oozie job 구동시 JA009: User: hadoop is not allowed to impersonate hadoop 오류나는 경우 총관리자 2014.06.02 807
568 hive query에서 mapreduce돌리지 않고 select하는 방법 총관리자 2014.05.23 806
567 [Mybatis]Spring과 연동하지 않고 Java+Mybatis 형태의 프로그램 샘플소스 총관리자 2016.09.01 793
566 mongodb aggregation query를 Java code로 변환한 샘플 총관리자 2016.12.15 777
565 sentry설정 방법및 활성화시 설정이 필요한 파일및 설정값, 계정생성 방법 총관리자 2018.08.16 767
564 oracle to hive data type정리표 총관리자 2018.08.22 763
563 python test.py실행시 "ImportError: No module named pyspark" 혹은 "ImportError: No module named py4j.protocol"등의 오류 발생시 조치사항 총관리자 2017.07.04 761
562 [DBeaver 4.3.0]import/export시 "Client home is not specified for connection" 오류발생시 조치사항 총관리자 2017.12.21 753
561 [Kerberos]Kerberos상태의 클러스터에 JDBC로 접근할때 케이스별 오류내용 총관리자 2020.02.14 747
560 beeline실행시 User: root is not allowed to impersonate오류 발생시 조치사항 총관리자 2016.06.03 744
559 Cloudera Manager web UI의 언어를 한글에서 영문으로 변경하기 총관리자 2018.04.03 741
558 os가 windows7인 host pc에서 ubuntu가 os인 guest pc에 접근하기 위한 네트워크설정 총관리자 2014.04.20 725

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로