Pre-execute notebooks

[davidbrochart]

In order to hide the start-up time, it would be nice to have the possibility to create a kernel and have it execute the notebook in advance, so that when we enter the URL in the browser, all there is left to do is rendering the HTML and connecting to the kernel.
We could have an option such as --pre-execute=nb (default to 0), that would create a pool of already executed notebooks ready to be served, in case there is sudden batch of connections.

[maartenbreddels]

Yes, I think that's a really good thing to have, especially when voila hosts a single notebook. The only requirement is that the notebook is a 'pure function', i.e. not dependant on parameters, time, etc.

[SylvainCorlay]

Also linking

• Feature: fork kernel jupyter/jupyter_client#441
• WIP: Feature: fork kernel ipython/ipykernel#410

[timkpaine]

Yes, I think that's a really good thing to have, especially when voila hosts a single notebook. The only requirement is that the notebook is a 'pure function', i.e. not dependant on parameters, time, etc.

For those that are not, or when you want to do things like per-user auth inside the notebook once a user has connected, it would be useful to have a few options:

• no preexecution
• fully preexecute notebook
• partial prexecution (start notebooks, maybe accept a list of modules to import to "warm" the kernel, e.g. things like numpy, pandas, tensorflow, anything else that you might want to setup but without any of the side effects of executing the notebook)

As always, fully preexecuting the notebook should never be the default or only behavior since this makes per-user authenticated notebooks difficult or impossible (we use the details of the user's request to inject auth into the notebook).

[davidbrochart]

partial prexecution (start notebooks, maybe accept a list of modules to import to "warm" the kernel, e.g. things like numpy, pandas, tensorflow, anything else that you might want to setup but without any of the side effects of executing the notebook)

partial pre-execution sounds to me like "execute first N cells", and we would let the author of the notebook arrange it so that the first N cells are user-independent.

[timkpaine]

partial pre-execution sounds to me like "execute first N cells", and we would let the author of the notebook arrange it so that the first N cells are user-independent.

this presupposes that the notebook author and the voila deployer are the same person (or closely in sync), which is not always the case. However with my prelaunch hook, I can always inject cells, so I can implement "warm these modules" with my own custom logic by injecting a cell and saying "warm 1 cell". So tl;dr, I can use it if we say partial is "n cells".

[timkpaine]

starting in #726, I have a pretty firm idea of how to do it so should be ready shortly. Interested in any thoughts around how to test it, though.

[maartenbreddels]

Nice work Tim, regarding preheat, we could do that based on a cell tag 'voila-preheat' orso.