Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다.
문의사항은 gooper@gooper.com로 메일을
보내주세요.
Setting up RStudio Server to run with Apache Spark
https://markobigdata.com/2016/03/27/setting-up-rstudio-server-to-run-with-apache-spark/
Zeppelin supports Spark, PySpark, Spark R, Spark SQL with dependency loader.
Python is supported with Matplotlib, Conda, Pandas SQL and PySpark integrations.
http://zeppelin.apache.org/
Jupyter kernels
https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
Here is a list of available kernels. If you are writing your own kernel, feel free to add it to the table!
| Name | Jupyter/IPython Version | Language(s) Version | 3rd party dependencies | Example Notebooks | Notes |
|---|---|---|---|---|---|
| Coarray-Fortran | Jupyter 4.0 | Fortran 2008/2015 | GFortran >= 7.1, OpenCoarrays, MPICH >= 3.2 | Demo, Binder demo | Docker image |
| sparkmagic | Jupyter >=4.0 | Pyspark (Python 2 & 3), Spark (Scala), SparkR (R) | Livy | Notebooks, Docker Images | This kernels are implemented via the magics machinery of the ipython kernel to use Spark via Livy |
| sas_kernel | Jupyter 4.0 | python >= 3.3 | SAS 9.4 or higher | ||
| IPyKernel | Jupyter 4.0 | python 2.7, >= 3.3 | pyzmq | ||
| IJulia | julia >= 0.3 | ||||
| IHaskell | ghc >= 7.6 | ||||
| IRuby | ruby >= 2.1 | ||||
| IJavascript | nodejs >= 0.10 | ||||
| jpCoffeescript | coffeescript >= 1.7 | ||||
| ICSharp | Jupyter 4.0 | C# 4.0+ | scriptcs | ||
| IRKernel | IPython 3.0 | R 3.2 | rzmq | ||
| SageMath | Jupyter 4 | Any | many | ||
| pari_jupyter | Jupyter 4 | PARI/GP >= 2.9 | |||
| IFSharp | IPython 2.0 | F# | Features | ||
| lgo | Jupyter >= 4, JupyterLab | Go >= 1.8 | ZeroMQ (4.x) | Example | Docker image |
| gopherlab | Jupyter 4.1, JupyterLab | Go >= 1.6 | ZeroMQ (4.x) | examples | Deprecated, use gophernotes |
| Gophernotes | Jupyter 4, JupyterLab, nteract | Go >= 1.9 | ZeroMQ 4.x.x | examples | docker image |
| IGo | Go >= 1.4 | ||||
| IScala | Scala | ||||
| Jupyter-scala | IPython>=3.0 | Scala>=2.10 | example | ||
| IErlang | IPython 2.3 | Erlang | rebar | ||
| ITorch | IPython >= 2.2 and <= 5.x | Torch 7 (LuaJIT) | |||
| IElixir | Jupyter < 6.0 | Elixir < 1.5 | Erlang OTP <= 19.3, Rebar | example | IElixir Notebook in Docker |
| ierl | Jupyter >= 4.0 | Erlang 19 or 20, Elixir 1.4 or 1.5, LFE 1.2 | Erlang, (optional) Elixir | ||
| IAldor | IPython >= 1 | Aldor | |||
| IOCaml | IPython >= 1.1 | OCaml >= 4.01 | opam | ||
| OCaml-Jupyter | Jupyter >= 4.0 | OCaml >= 4.02 | opam | Example | Docker image |
| IForth | IPython >= 3 | Forth | |||
| peforth | IPython 6/Jupyter 5 | Forth | Example | python debugger in FORTH syntax | |
| IPerl | Perl 5 | ||||
| Perl6 | Jupyter >= 4 | Perl 6.c | zeromq 4 | ||
| IPerl6 | Perl 6 | ||||
| Jupyter-Perl6 | Jupyter | Perl 6.C | Rakudo Perl 6 | ||
| IPHP | IPython >= 2 | PHP >= 5.4 | composer | DEPRECATED, use Jupyter-PHP | |
| Jupyter-PHP | Jupyter 4.0 | PHP >= 7.0.0 | composer, php-zmq | ||
| IOctave | Jupyter | Octave | Example | MetaKernel | |
| IScilab | Jupyter | Scilab | Example | MetaKernel | |
| MATLAB Kernel | Jupyter | Matlab | pymatbridge | Example | MetaKernel |
| Bash | IPython >= 3 | bash | Wrapper | ||
| PowerShell | IPython >= 3 | Windows | Wrapper, Based on Bash Kernel | ||
| CloJupyter | Jupyter | Clojure >= 1.7 | |||
| CLJ-Jupyter | Jupyter | Clojure | Abandoned as of 2017-02-12 | ||
| jupyter-kernel-jsr223 | Jupyter>=4.0 | Clojure 1.8 | clojure-jrs223, Java>=7 | Java based JSR223 compliant | |
| Hy Kernel | Jupyter | Hy | Tutorial | treats Hy as Python pre-processor | |
| Calysto Hy | Jupyter | Hy | Tutorial | based on MetaKernel (magics, shell, parallel, etc.) | |
| Redis Kernel | IPython >= 3 | redis | Wrapper | ||
| jove | io.js | ||||
| jp-babel | Jupyter | Babel | |||
| ICalico | IPython >= 2 | multiple | Index | ||
| IMathics | Mathics | ||||
| IWolfram | Wolfram Mathematica | Wolfram Mathematica(R), Metakernel | MetaKernel | ||
| Lua Kernel | Lua | ||||
| IPurescript | Purescript | ||||
| IPyLua | Lua | Fork of Lua Kernel | |||
| Calysto Scheme | Scheme | Reference Guide | MetaKernel | ||
| Calysto Processing | Processing.js >= 2 | MetaKernel | |||
| idl_kernel | IDL | IDL seem to have a built-in kernelstarting with version 8.5 | |||
| Mochi Kernel | Mochi | ||||
| Lua (used in Splash) | Lua | ||||
| Apache Toree (formerly Spark Kernel) | Jupyter | Scala, Python, R | Spark >= 1.5 | Example | |
| Skulpt Python Kernel | Skulpt Python | Examples | MetaKernel | ||
| MetaKernel Bash | bash | MetaKernel | |||
| MetaKernel Python | python | MetaKernel | |||
| IVisual | VPython | Ball-in-Box | |||
| IBrainfuck | Brainfuck | Demo | Wrapper | ||
| KDB+/Q Kernel (IKdbQ) | IPython >= 3.1 | Q | qzmq, qcrypt | ||
| KDB+/Q Kernel (KdbQ Kernel) | Jupyter | Q | |||
| ICryptol | Cryptol | CVC4 | |||
| cling | Jupyter 4 | C++ | Example | ||
| Xonsh | Xonsh | Example | MetaKernel | ||
| Prolog | Prolog | MetaKernel | |||
| cl-jupyter | Jupyter | Common Lisp | Quicklisp | About | |
| Maxima-Jupyter | Jupyter | Maxima | Quicklisp | ||
| Calysto LC3 | Assembly Language for the Little Computer 3 | ||||
| Yacas | YACAS | ||||
| IJython | Jython 2.7 | ||||
| ROOT | Jupyter | C++/python | ROOT >= 6.05 | ||
| Gnuplot Kernel | Gnuplot | Example | MetaKernel | ||
| Tcl | Jupyter | Tcl 8.5 | Based on Bash Kernel | ||
| J | Jupyter | J 805 | Examples | ||
| Jython | Jupyter>=4.0 | Jython>=2.7.0 | Java>=7 | Java based JSR223 compliant | |
| C | Jupyter | C | gcc | ||
| TaQL | Jupyter | TaQL | python-casacore | TaQL tutorial | |
| Coconut | Jupyter | Coconut | |||
| SPARQL | Jupyter 4 | Python 2.7 or >=3.4 | rdflib, SPARQLWrapper | Examples | Optional GraphVizdependency |
| AIML chatbot | Jupyter 4 | Python 2.7 | pyAIML | Examples | |
| IArm | Jupyter 4 | ARMv6 THUMB | Examples | Based off of the ARM Cortex M0+ CPU | |
| SoS | Jupyter 4 | Python >=3.4 | Examples | Workflow system, Multi-Kernel support | |
| jupyter-nodejs | Jupyter, iPython 3.x | NodeJS, Babel, Clojurescript | Examples | ||
| Pike | IPython >= 3 | Pike >= 7.8 | Wrapper, Based on Bash Kernel | ||
| ITypeScript | Typescript >= 2.0 | Node.js >= 0.10.0 | |||
| imatlab | ipykernel >= 4.1 | MATLAB >= 2016b | |||
| jupyter-kotlin | Jupyter | Kotlin 1.1-M04 EAP | Java >= 8 | ||
| jupyter_kernel_singular | Jupyter | Singular 4.1.0 | Demo | Optional PySingular for better performance, surf for images, details | |
| spylon-kernel | ipykernel >=4.5 | python >= 3.5, scala >= 2.11 | Apache Spark >=2.0 | Example | MetaKernel |
| mit-scheme-kernel | Jupyter 4.0 | MIT Scheme 9.2 | |||
| elm-kernel | Jupyter | Examples | |||
| SciJava Jupyter Kernel | Jupyter 4.3.0 | Java + 9 scripting languages | Java | Examples | |
| Isbt | Jupyter 4.3.0 | sbt >= 1.0.0 | sbt | example | |
| BeakerX | Groovy, Java, Scala, Clojure, Kotlin, SQL | example | docker image | ||
| MicroPython | Jupyter | ESP8266/ESP32 | USB or Webrepl | developer notebooks | relies on the micro-controller's paste-mode |
| IJava | Jupyter | Java 9 | Java JDK >= 9 | Binder online demo | Based on the new JShell tool |
| Guile | Jupyter 5.2 | Guile 2.0 | guile-json, openssl |