[large notebooks] Collapsible headings do not show up anymore

[foice]

Hello, I have used collapsible headings for sometime now, but as of today they not show up any more in my notebooks. I have tried several times to switch off and back on this extension with the http://localhost:8888/nbextensions page, but it does not seem to help. I have also tried on Chrome and Firefox and in both cases the handles to close the heading for the section do not show up.

Any idea what else I could try to get the collapsible headings back?

[jcb91]

Well, it sounds like the extension's attempting, but failing, to load. Could you post any output from the
javascript console? I'd expect to see a complaint about failing to load collapsble_headings.

As to what's causing it, what changes have you recently made to your notebook environment? Any new nbextensions/server extensions? Updated versions? Custom js?

Finally, does this apply equally to existing and new notebooks? I would guess yes, but worth a check...

[foice]

Failed to load extensions: <unavailable> <unavailable>  main.min.js:12156
GET 
http://localhost:8888/nbextensions/widgets/notebook/js/extension.js [HTTP/1.1 404 Not Found 20ms]
Session: kernel_created (57e00ef1-02ff-413e-8959-ede33a3978ca)  main.min.js:23423
Starting WebSockets: ws://localhost:8888/api/kernels/75e48fb9-ba45-4591-8881-9d1bc5a66646  main.min.js:22709
Widgets are not available.  Please install widgetsnbextension or ipywidgets 4.0  main.min.js:33966
Kernel: kernel_connected (75e48fb9-ba45-4591-8881-9d1bc5a66646)  main.min.js:22356
init_cell : running all initialization cells  main.js:26
init_cell : finished running 48 initialization cells  main.js:36
Kernel: kernel_ready (75e48fb9-ba45-4591-8881-9d1bc5a66646)  main.min.js:22356

[foice]

I have widgetsnbextension and ipywidgets (5.1.5).

It works no problem with small notebooks, is very consistent. For large notebooks I get "script is too slow" warning, which I accept to keep going, and almost always I get no collapsible headings.

[jcb91]

It works no problem with small notebooks, is very consistent. For large notebooks I get "script is too slow" warning, which I accept to keep going, and almost always I get no collapsible headings.

Ok, how large is 'large'? The number of cells/headings is potentially relevant in addition to actual file size, I expect. When you say "almost always I get no collapsible headings.", do you mean that for the same notebook, sometimes you do get them, and sometimes you don't? I suspect, then, that this issue is something to do with the browser/javascript engine/notebook deciding to kill a collapsible headings callback which has taken too long, and as a result the nbextension doesn't get correctly loaded. Indeed, I suspect that it'll be a huge number of calls to update_collapsed_headings, so I should probably put a rate-limiter on that, setting a timeout or something...

[foice]

large is maybe 60 to 100 headings of different levels. Coming from Mathematica I am used to fraction a lot the content, duplicate section to keep track of variants of the calculations, etc ...

Sometimes indeed the headings are collapsible, but I cannot say what is the deciding factor. Being fast to reply to the many "script is too slow" warnings seems to help, but I am not sure is a true effect because on Chrome I do not get any of these warning for "script is too slow" and still sometimes loads correctly.

Let me know if I can help in any way with tests or else.

[jcb91]

Right, so I suspect that it may just be the many almost-simultaneous calls to update_collapsed_headings which result from the cells being added to the notebook on loading that are causing everything to grind to a halt. I should really have rate-limited this to begin with, but decided not to in favour of only updating the portions of a notebook which had been altered with each call. This was probably not a great choice on my part, especially since it doesn't help if the mathematica-style brackets are in use anyway.

Let me know if I can help in any way with tests or else.

So to clarify, and just check that I've got the story correct in my head, a few questions/requests for confirmation:

• you're using the mathematica-style brackets, but when this issue occurs neither these nor the icons to the left of headings get displayed.

  • does disabling the brackets change anything?

• large is maybe 60 to 100 headings of different levels.

  Ok, so not ridiculous by any means, but more than most people are using, which is presumably why this hasn't shown up before. Roughly how many cells per section? Do you have an example that shows this behaviour that you could share?

• Being fast to reply to the many "script is too slow" warnings seems to help

  Interesting, could be that the timeout for one call is bloated byt thge time to respond to the slow warning, haha 😆

• on Chrome I do not get any of these warning for "script is too slow" and still sometimes loads correctly

  This may just be variation in how strict different browsers are about deciding that a script is being slow. Which browser(s) are you using other than Chrome?

I'll try to knock together some rate-limiting now, and we can see if it helps

[foice]

At present I do not have the mathematica-style brackets, but I might have activated that option before.

Each section might have 4 to 20 cells, some are initialization cells and might take seconds to evaluate each, some are a lot faster.

I alternate Chrome and Firefox.

[jcb91]

ah, ok. I've had a stab at throttling the update calls in #823 if you fancy trying it out?

[foice]

I can try it, sure. But how do I do that?

[jcb91]

ah, apologies. So, it depends slightly on your python environment, but essentially, to get a copy of the repo incorporating the pr #823, you can use:

git clone https://github.com/ipython-contrib/jupyter_contrib_nbextensions.git
cd jupyter_contrib_nbextensions
git fetch origin pull/823/head:pr823
git checkout pr823
jupyter nbextension install src/jupyter_contrib_nbextensions/nbextensions/collapsible_headings

Alternatively, in one line you can update yoru existing collapsible_headings install javascript to include the PR changes using

jupyter nbextension install --destination=collapsible_headings/main.js https://raw.githubusercontent.com/jcb91/jupyter_contrib_nbextensions/collapsible_headings/src/jupyter_contrib_nbextensions/nbextensions/collapsible_headings/main.js

and adding whichever of the --user, --sys-prefix or --system flags you used to install initially (if you used conda, use --sys-prefix). Note that this won't update the yaml file, so the new parameter won't show up

Does that make sense?

[foice]

I used the first way to install this. I suppose it went through as no error has been thrown.

On the small notebook it still loads fine, but does not work yet on the big one. Here is the JS console of the one that does not work. These two seems to be the most relevant log lines, the full log is below them.

load_extensions Arguments[1]0: "Limit Output"callee: (...)get callee: ThrowTypeError()set callee: ThrowTypeError()caller: (...)get caller: ThrowTypeError()set caller: ThrowTypeError()length: 1Symbol(Symbol.iterator): values()__proto__: Object__defineGetter__: __defineGetter__()__defineSetter__: __defineSetter__()__lookupGetter__: __lookupGetter__()__lookupSetter__: __lookupSetter__()constructor: Object()hasOwnProperty: hasOwnProperty()isPrototypeOf: isPrototypeOf()propertyIsEnumerable: propertyIsEnumerable()toLocaleString: toLocaleString()toString: toString()valueOf: valueOf()get __proto__: __proto__()set __proto__: __proto__()

main.min.js:12156 Failed to load extension: Array[1]0:     "nbextensions/collapsible_headings/main"length: 1__proto__: Array[0] Error: Load timeout for modules: nbextensions/collapsible_headings/main
http://requirejs.org/docs/errors.html#timeout
    at makeError (require.js?v=6da8be3…:165)
    at checkLoaded (require.js?v=6da8be3…:693)
    at require.js?v=6da8be3…:714

Full log from Chrome:

:8888/custom/custom.js?v=20161230120507:2 Binding Shift-Ctrl-Alt-J/K to move cell up/down
require.js?v=6da8be3…:900 TypeError: Cannot read property 'command_shortcuts' of undefined
    at :8888/custom/custom.js?v=20161230120507:3
    at Object.execCb (require.js?v=6da8be3…:1690)
    at Module.check (require.js?v=6da8be3…:865)
    at Module.enable (require.js?v=6da8be3…:1177)
    at Module.init (require.js?v=6da8be3…:783)
    at callGetModule (require.js?v=6da8be3…:1204)
    at Object.completeLoad (require.js?v=6da8be3…:1583)
    at HTMLScriptElement.onScriptLoad (require.js?v=6da8be3…:1711)
check @ require.js?v=6da8be3…:900
main.min.js:24135 Default extension for cell metadata editing loaded.
main.min.js:24221 Raw Cell Format toolbar preset loaded.
main.min.js:24269 Slideshow extension for metadata editing loaded.
main.min.js:29057 actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...
MenuBar.bind_events @ main.min.js:29057
main.min.js:12154 load_extensions Arguments[1]0: "collapsible_headings/main"callee: (...)get callee: ThrowTypeError()set callee: ThrowTypeError()caller: (...)get caller: ThrowTypeError()set caller: ThrowTypeError()length: 1Symbol(Symbol.iterator): values()__proto__: Object
main.min.js:10 accessing `utils` is deprecated. Use `require("base/js/utils")`
get @ main.min.js:10
main.min.js:12154 load_extensions Arguments[1]0: "Limit Output"callee: (...)get callee: ThrowTypeError()set callee: ThrowTypeError()caller: (...)get caller: ThrowTypeError()set caller: ThrowTypeError()length: 1Symbol(Symbol.iterator): values()__proto__: Object__defineGetter__: __defineGetter__()__defineSetter__: __defineSetter__()__lookupGetter__: __lookupGetter__()__lookupSetter__: __lookupSetter__()constructor: Object()hasOwnProperty: hasOwnProperty()isPrototypeOf: isPrototypeOf()propertyIsEnumerable: propertyIsEnumerable()toLocaleString: toLocaleString()toString: toString()valueOf: valueOf()get __proto__: __proto__()set __proto__: __proto__()
main.min.js:12156 Failed to load extension: Array[1]0: "nbextensions/collapsible_headings/main"length: 1__proto__: Array[0] Error: Load timeout for modules: nbextensions/collapsible_headings/main
http://requirejs.org/docs/errors.html#timeout
    at makeError (require.js?v=6da8be3…:165)
    at checkLoaded (require.js?v=6da8be3…:693)
    at require.js?v=6da8be3…:714
(anonymous) @ main.min.js:12156
main.min.js:23423 Session: kernel_created (8aca9db2-66f4-4f64-a84e-704d656a370e)
main.min.js:22709 Starting WebSockets: ws://localhost:8888/api/kernels/e2f3987c-3975-4ee9-bde5-aa99dcb0ebab
main.min.js:22356 Kernel: kernel_connected (e2f3987c-3975-4ee9-bde5-aa99dcb0ebab)
http://localhost:8888/nbextensions/Limit%20Output.js?v=20161230120507 Failed to load resource: the server responded with a status of 404 (Not Found)
main.min.js:12156 Failed to load extension: Array[1]0: "nbextensions/Limit Output"length: 1__proto__: Array[0] Error: Script error for "nbextensions/Limit Output"
http://requirejs.org/docs/errors.html#scripterror
    at makeError (require.js?v=6da8be3…:165)
    at HTMLScriptElement.onScriptError (require.js?v=6da8be3…:1732)
(anonymous) @ main.min.js:12156
main.min.js:22356 Kernel: kernel_ready (e2f3987c-3975-4ee9-bde5-aa99dcb0ebab)
http://localhost:8888/nbextensions/widgets/notebook/js/extension.js?v=20161230120507 Failed to load resource: the server responded with a status of 404 (Not Found)
main.min.js:33966 Widgets are not available.  Please install widgetsnbextension or ipywidgets 4.0
(anonymous) @ main.min.js:33966

[jcb91]

ok, so from the line

Load timeout for modules: nbextensions/collapsible_headings/main

it's at least clear that we're looking at a load timeout which is the problem. Having experimented a little myself using a dummy notebook (see this gist), I find that the notebook takes a very long time to load - something like 20 seconds or so. The collapsible_headings execution itself is of order 10ms, which isn't long enough to cause the timeout, so I think the best option may be to delay binding of heading updates until the notebook is definitely loaded. I'll push another attempt...

[jfbercher]

I use if (Jupyter.notebook._fully_loaded) {...}

[jcb91]

Ok, on further investigation, this isn't actually an issue specific to collapsible_headings. Essentially it's a limitation of the way notebook loading happens. In essence, if your notebook is big enough, then the require calls to load the nbextensions may happen while the notebook is loading, and then all the processing time is hogged by the notebook load, and as a result any and all nbextensions will fail to load. You could increasing the available timeout by adding something like

window.requirejs.config({
    waitseconds: 60,  // default is 30s
});

to your custom.js (which should be loaded before the notebook starts to load), but I can't see much other way round it at the moment, unfortunately