Feature: fork kernel

.gitignore

@@ -1,8 +1,6 @@
 MANIFEST
 build
 dist
-_build
-docs/gh-pages
 *.py[co]
 __pycache__
 *.egg-info
@@ -16,3 +14,15 @@ __pycache__
 .coverage
 .cache
 absolute.json
+
+# Sphinx documentation
+_build
+docs/_build/
+docs/gh-pages
+
+# PyBuilder
+target/
+
+# PyCharm
+.idea/
+*.iml

.travis.yml

@@ -1,19 +1,22 @@
 language: python
 python:
     - "nightly"
-    - '3.6-dev'
+    - "3.7-dev"
+    - 3.6
     - 3.5
-    - 3.4
-    - 3.3
     - 2.7
 sudo: false
 install:
   - pip install --upgrade setuptools pip
-  - pip install --upgrade --pre -e .[test] pytest-cov pytest-warnings codecov
+  - pip install --upgrade --upgrade-strategy eager --pre -e .[test] pytest-cov codecov 'coverage<5'
+  - pip freeze
 script:
   - py.test --cov jupyter_client jupyter_client
 after_success:
   - codecov
 matrix:
     allow_failures:
         - python: nightly
+branches:
+  except:
+    - /^auto-backport-of-pr-[0-9]+$/

MANIFEST.in

@@ -5,6 +5,7 @@ include README.md
 # Documentation
 graft docs
 exclude docs/\#*
+exclude docs/_*
 
 # Examples
 graft examples

conn.json

@@ -0,0 +1,12 @@
+{
+  "shell_port": 56573,
+  "iopub_port": 56574,
+  "stdin_port": 56575,
+  "control_port": 56576,
+  "hb_port": 56577,
+  "ip": "127.0.0.1",
+  "key": "1371111a-7b97c86926519da79e2acbf2",
+  "transport": "tcp",
+  "signature_scheme": "hmac-sha256",
+  "kernel_name": ""
+}

conn_fork.json

@@ -0,0 +1,12 @@
+{
+  "shell_port": 66573,
+  "iopub_port": 66574,
+  "stdin_port": 66575,
+  "control_port": 66576,
+  "hb_port": 66577,
+  "ip": "127.0.0.1",
+  "key": "1371111a-7b97c86926519da79e2acbf3",
+  "transport": "tcp",
+  "signature_scheme": "hmac-sha256",
+  "kernel_name": ""
+}

docs/changelog.rst

@@ -4,6 +4,82 @@
 Changes in Jupyter Client
 =========================
 
+5.2.3
+=====
+
+`5.2.3 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2.3>`__
+
+- Fix hang on close in :class:`.ThreadedKernelClient` (used in QtConsole)
+  when using tornado with asyncio
+  (default behavior of tornado 5, see :ghpull:`352`).
+- Fix errors when using deprecated :attr:`.KernelManager.kernel_cmd`
+  (:ghpull:`343`, :ghpull:`344`).
+
+5.2.2
+=====
+
+`5.2.2 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2.2>`__
+
+- Fix :meth:`.KernelSpecManager.get_all_specs` method in subclasses
+  that only override :meth:`.KernelSpecManager.find_kernel_specs`
+  and :meth:`.KernelSpecManager.get_kernel_spec`.
+  See :ghissue:`338` and :ghpull:`339`.
+- Eliminate occasional error messages during process exit (:ghpull:`336`).
+- Improve error message when attempting to bind on invalid address (:ghpull:`330`).
+- Add missing direct dependency on tornado (:ghpull:`323`).
+
+
+5.2.1
+=====
+
+`5.2.1 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2.1>`__
+
+- Add parenthesis to conditional pytest requirement to work around a bug in the
+  ``wheel`` package, that generate a ``.whl`` which otherwise always depends on
+  ``pytest`` see :ghissue:`324` and :ghpull:`325`.
+
+5.2
+===
+
+`5.2 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2>`__
+
+- Define Jupyter protocol version 5.3:
+
+  - Kernels can now opt to be interrupted by a message sent on the control channel
+    instead of a system signal. See :ref:`kernelspecs` and :ref:`msging_interrupt`
+    (:ghpull:`294`).
+
+- New ``jupyter kernel`` command to launch an installed kernel by name
+  (:ghpull:`240`).
+- Kernelspecs where the command starts with e.g. ``python3`` or
+  ``python3.6``—matching the version ``jupyter_client`` is running on—are now
+  launched with the same Python executable as the launching process (:ghpull:`306`).
+  This extends the special handling of ``python`` added in 5.0.
+- Command line arguments specified by a kernelspec can now include
+  ``{resource_dir}``, which will be substituted with the kernelspec resource
+  directory path when the kernel is launched (:ghpull:`289`).
+- Kernelspecs now have an optional ``metadata`` field to hold arbitrary metadata
+  about kernels—see :ref:`kernelspecs` (:ghpull:`274`).
+- Make the ``KernelRestarter`` class used by a ``KernelManager`` configurable
+  (:ghpull:`290`).
+- When killing a kernel on Unix, kill its process group (:ghpull:`314`).
+- If a kernel dies soon after starting, reassign random ports before restarting
+  it, in case one of the previously chosen ports has been bound by another
+  process (:ghpull:`279`).
+- Avoid unnecessary filesystem operations when finding a kernelspec with
+  :meth:`.KernelSpecManager.get_kernel_spec` (:ghpull:`311`).
+- :meth:`.KernelSpecManager.get_all_specs` will no longer raise an exception on
+  encountering an invalid ``kernel.json`` file. It will raise a warning and
+  continue (:ghpull:`310`).
+- Check for non-contiguous buffers before trying to send them through ZMQ
+  (:ghpull:`258`).
+- Compatibility with upcoming Tornado version 5.0 (:ghpull:`304`).
+- Simplify setup code by always using setuptools (:ghpull:`284`).
+- Soften warnings when setting the sticky bit on runtime files fails
+  (:ghpull:`286`).
+- Various corrections and improvements to documentation.
+
+
 5.1
 ===
 

docs/conf.py

@@ -34,6 +34,7 @@
     'sphinx.ext.autodoc',
     'sphinx.ext.intersphinx',
     'sphinx.ext.napoleon',
+    'sphinxcontrib_github_alt',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -55,6 +56,8 @@
 copyright = '2015, Jupyter Development Team'
 author = 'Jupyter Development Team'
 
+github_project_url = "https://github.com/jupyter/jupyter_client"
+
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.

docs/environment.yml

@@ -3,8 +3,10 @@ channels:
   - conda-forge
 dependencies:
 - pyzmq
-- python==3.5
+- python==3.7
 - traitlets>=4.1
 - jupyter_core
 - sphinx>=1.3.6
 - sphinx_rtd_theme
+- pip:
+  - sphinxcontrib_github_alt

docs/index.rst

@@ -23,6 +23,7 @@ with Jupyter kernels.
 
    kernels
    wrapperkernels
+   kernel_providers
 
 .. toctree::
    :maxdepth: 2

docs/kernel_providers.rst

@@ -0,0 +1,156 @@
+================
+Kernel providers
+================
+
+.. note::
+   This is a new interface under development, and may still change.
+   Not all Jupyter applications use this yet.
+   See :ref:`kernelspecs` for the established way of discovering kernel types.
+
+Creating a kernel provider
+==========================
+
+By writing a kernel provider, you can extend how Jupyter applications discover
+and start kernels. For example, you could find kernels in an environment system
+like conda, or kernels on remote systems which you can access.
+
+To write a kernel provider, subclass
+:class:`jupyter_client.discovery.KernelProviderBase`, giving your provider an ID
+and overriding two methods.
+
+.. class:: MyKernelProvider
+
+   .. attribute:: id
+
+      A short string identifying this provider. Cannot contain forward slash
+      (``/``).
+
+   .. method:: find_kernels()
+
+      Get the available kernel types this provider knows about.
+      Return an iterable of 2-tuples: (name, attributes).
+      *name* is a short string identifying the kernel type.
+      *attributes* is a dictionary with information to allow selecting a kernel.
+
+   .. method:: make_manager(name)
+
+      Prepare and return a :class:`~jupyter_client.KernelManager` instance
+      ready to start a new kernel instance of the type identified by *name*.
+      The input will be one of the names given by :meth:`find_kernels`.
+
+For example, imagine we want to tell Jupyter about kernels for a new language
+called *oblong*::
+
+    # oblong_provider.py
+    from jupyter_client.discovery import KernelProviderBase
+    from jupyter_client import KernelManager
+    from shutil import which
+
+    class OblongKernelProvider(KernelProviderBase):
+        id = 'oblong'
+
+        def find_kernels(self):
+            if not which('oblong-kernel'):
+                return  # Check it's available
+
+            # Two variants - for a real kernel, these could be something like
+            # different conda environments.
+            yield 'standard', {
+                'display_name': 'Oblong (standard)',
+                'language': {'name': 'oblong'},
+                'argv': ['oblong-kernel'],
+            }
+            yield 'rounded', {
+                'display_name': 'Oblong (rounded)',
+                'language': {'name': 'oblong'},
+                'argv': ['oblong-kernel'],
+            }
+
+        def make_manager(self, name):
+            if name == 'standard':
+                return KernelManager(kernel_cmd=['oblong-kernel'],
+                                     extra_env={'ROUNDED': '0'})
+            elif name == 'rounded':
+                return KernelManager(kernel_cmd=['oblong-kernel'],
+                                     extra_env={'ROUNDED': '1'})
+            else:
+                raise ValueError("Unknown kernel %s" % name)
+
+You would then register this with an *entry point*. In your ``setup.py``, put
+something like this::
+
+    setup(...
+        entry_points = {
+        'jupyter_client.kernel_providers' : [
+            # The name before the '=' should match the id attribute
+            'oblong = oblong_provider:OblongKernelProvider',
+        ]
+    })
+
+Finding kernel types
+====================
+
+To find and start kernels in client code, use
+:class:`jupyter_client.discovery.KernelFinder`. This uses multiple kernel
+providers to find available kernels. Like a kernel provider, it has methods
+``find_kernels`` and ``make_manager``. The kernel names it works
+with have the provider ID as a prefix, e.g. ``oblong/rounded`` (from the example
+above).
+
+::
+
+    from jupyter_client.discovery import KernelFinder
+    kf = KernelFinder.from_entrypoints()
+
+    ## Find available kernel types
+    for name, attributes in kf.find_kernels():
+        print(name, ':', attributes['display_name'])
+    # oblong/standard : Oblong (standard)
+    # oblong/rounded : Oblong(rounded)
+    # ...
+
+    ## Start a kernel by name
+    manager = kf.make_manager('oblong/standard')
+    manager.start_kernel()
+
+.. module:: jupyter_client.discovery
+
+.. autoclass:: KernelFinder
+
+   .. automethod:: from_entrypoints
+
+   .. automethod:: find_kernels
+
+   .. automethod:: make_manager
+
+Kernel providers included in ``jupyter_client``
+===============================================
+
+``jupyter_client`` includes two kernel providers:
+
+.. autoclass:: KernelSpecProvider
+
+   .. seealso:: :ref:`kernelspecs`
+
+.. autoclass:: IPykernelProvider
+
+Glossary
+========
+
+Kernel instance
+  A running kernel, a process which can accept ZMQ connections from frontends.
+  Its state includes a namespace and an execution counter.
+
+Kernel type
+  The software to run a kernel instance, along with the context in which a
+  kernel starts. One kernel type allows starting multiple, initially similar
+  kernel instances. For instance, one kernel type may be associated with one
+  conda environment containing ``ipykernel``. The same kernel software in
+  another environment would be a different kernel type. Another software package
+  for a kernel, such as ``IRkernel``, would also be a different kernel type.
+
+Kernel provider
+  A Python class to discover kernel types and allow a client to start instances
+  of those kernel types. For instance, one kernel provider might find conda
+  environments containing ``ipykernel`` and allow starting kernel instances in
+  these environments.