Skip to content

DEVELOPERS.md

Latest commit

 

History

History
183 lines (142 loc) · 7.42 KB

DEVELOPERS.md

File metadata and controls

183 lines (142 loc) · 7.42 KB

Dependencies:

Related repositories for JVM kernel development

Related repositories for autotranslation development

Related repositories for front-end components

Related repositories for general development

Setup your development environment (linux and mac)

mkdir beakerx-devel
cd beakerx-devel
git clone https://github.com/twosigma/beakerx
git clone https://github.com/twosigma/beakerx_kernel_base
git clone https://github.com/twosigma/beakerx_kernel_kotlin
git clone https://github.com/twosigma/beakerx_kernel_clojure
git clone https://github.com/twosigma/beakerx_kernel_scala
git clone https://github.com/twosigma/beakerx_kernel_groovy
git clone https://github.com/twosigma/beakerx_kernel_sql
git clone https://github.com/twosigma/beakerx_kernel_autotranslation
git clone https://github.com/twosigma/beakerx_kernel_java
git clone https://github.com/twosigma/beakerx_tests
git clone https://github.com/twosigma/beakerx_base
git clone https://github.com/twosigma/beakerx_widgets
git clone https://github.com/twosigma/beakerx_tabledisplay
cd beakerx_tests/dev_scripts/
conda env create -n beakerx-devel -f configuration-lab.yml
conda activate beakerx-devel
./install_kernels_lab.sh
./install_lab_local_extensions.sh
cd ..
cd ..

Architecture and Code Overview

TODO update this

BeakerX is a collection of kernels and extensions for Jupyter. The code is organized into subdirectories as follows:

  • beakerx The Python packages. The main beakerx package has:

    • a customized KernelSpec to allow BeakerX to configure the JVMs that run the kernels,

    • a server extension for the javadoc, settings, version, and autotranslation endpoints,

    • the beakerx command line program, which has the bkr2ipynb converter, the py4j server, utilities, install, and uninstall functions.

    • the Python API for the runtime (tables, plots, easyform), including automatically installing a displayer for pandas tables, and autotranslation;

    • the nbextension webpack (compiled JavaScript, TypeScript, CSS, fonts, images); and

    • the compiled Java JARs of each of the kernels, and a directory of shared JARs.

    There is a separate python package (beakerx_magics) for the %%groovy magic so it can be loaded without loading the regular beakerx package (which would turn on display of pandas tables with our table widget).

    BeakerX configures ipython to automatically load the magics in the beakerx_magics package, %load_ext is not required.

    The groovy magic uses the standard Jupyter API, jupyter_client.manager.KernelManager to start the kernel. It then proxies Comm into the inner kernel.

    This package also has the py4j support for the %%python magic. In order for the JVM kernels to be able to start Jupyter kernels they need to be able to call into Python. There is a beakerx py4j_server subcommand for this purpose (for internal use, not for the user). It calls into the groovy magic with its Comm proxy, implemented in Python.

    BeakerX implements autotranslation with a server extension and metaprogramming glue in each language. The glue makes the beakerx object into a proxy for RPC calls to the server extension, using JSON serialization. For JavaScript, the proxy object uses Comm to reach the kernel, which forwards to the server extension.

    The autotranslation server has a separate thread of control from Jupyter, and it manages its own port, which it protects by accepting only local connections and requiring a secure password (which is passed to the kernels via an environment variable). The extra thread is necessary to avoid deadlock in tornado. This might be better done with a queue, as explained in #5039.

    See #7577 for the plan to improve this architecture with shared memory and Apache Arrow.

  • doc Documentation consisting of executable tutorial notebooks. StartHere.ipynb at the top level links to these and is the intended way to navigate them. There is a subdirectory for each language.

  • docker configuration files for creating the Docker image. There is a subdirectory docker/base for an image with BeakerX's dependencies (the Ubuntu and conda packages). The main image is built by compiling BeakerX and installing BeakerX in the base image.

  • js There are two subdirectories of JavaScript and TypeScript, js/lab and js/notebook. New code is being written in TypeScript.

    The lab subdirectory has the extension for Jupyter Lab (distributed by npm). Notebook has two extensions, one for the widgets (which are included in Lab as well, and are also separately distributed with npm for embedded applications such as nbviewer), and one for the notebook application. This adds a tab to the tree view with our options panel.

    And for regular notebook pages the extension handles: running initialization cells, publication, autotranslation, the getCodeCells and runByTag APIs, callbacks for table and plot actions, UI customizations such as changing the fonts, allowing wide code cells, and disabling autosave.

  • kernel The Java implementation of the kernels is here. The main directory is kernel/base which has generic code for all the languages. The base kernel has classes for Jupyter's Comm protocol (a layer over ZMQ), magics, the classpath (including loading from maven), and the kernel parts of the widget APIs.

    There is also a subdirectory for each language which has the evaluator for that language. Scala has wrappers for the widgets so they have native types.

  • test The e2e tests, which are made with webdriver (selenium, chromedriver, jasmine).
⚡