Widget state is not captured when running notebooks

[choldgraf]

I noticed that running a notebook that outputs ipywidgets with jupyter-cache resulted in the widgets not showing up in the output (though other outputs did show up). In the past this has often been because some kind of store_widget_state variable wasn't added.

relevant issue in nbclient: jupyter/nbclient#24

[chrisjsewell]

Note store_widget_state should be True by default: https://github.com/jupyter/nbconvert/blob/c837a22d44d98f6a58d1934bd85af1506df48f21/nbconvert/preprocessors/execute.py#L224

When I get round to it, I'll add a notebook with ipywidgets to the tests

[chrisjsewell]

This looks to be the same issue as nteract/papermill#400:

Minimal example:

ipywidgets.ipynb:

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import interact\n", 
    "def f(x):\n",
    "    return x\n",
    "\n",
    "interact(f, x=10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

from nbformat import read
nb = read("ipywidgets.ipynb", 4)
from nbconvert.preprocessors.execute import executenb
executenb(nb)

output:

{'cells': [{'cell_type': 'code',
   'execution_count': 1,
   'metadata': {},
   'outputs': [{'output_type': 'display_data',
     'metadata': {},
     'data': {'text/plain': '10'}},
    {'output_type': 'execute_result',
     'metadata': {},
     'data': {'text/plain': ''},
     'execution_count': 1}],
   'source': 'from ipywidgets import interact\ndef f(x):\n    return x\n\ninteract(f, x=10)'}],
 'metadata': {'kernelspec': {'display_name': 'Python 3',
   'language': 'python',
   'name': 'python3'},
  'language_info': {'name': 'python',
   'version': '3.7.6',
   'mimetype': 'text/x-python',
   'codemirror_mode': {'name': 'ipython', 'version': 3},
   'pygments_lexer': 'ipython3',
   'nbconvert_exporter': 'python',
   'file_extension': '.py'},
  'widgets': {'application/vnd.jupyter.widget-state+json': {'state': {'681322fe51bb4ffb9cde6343142607ff': {'model_name': 'LayoutModel',
      'model_module': '@jupyter-widgets/base',
      'model_module_version': '1.2.0',
      'state': {'_model_module': '@jupyter-widgets/base',
       '_model_module_version': '1.2.0',
       '_model_name': 'LayoutModel',
       '_view_count': None,
       '_view_module': '@jupyter-widgets/base',
       '_view_module_version': '1.2.0',
       '_view_name': 'LayoutView',
       'align_content': None,
       'align_items': None,
       'align_self': None,
       'border': None,
       'bottom': None,
       'display': None,
       'flex': None,
       'flex_flow': None,
       'grid_area': None,
       'grid_auto_columns': None,
       'grid_auto_flow': None,
       'grid_auto_rows': None,
       'grid_column': None,
       'grid_gap': None,
       'grid_row': None,
       'grid_template_areas': None,
       'grid_template_columns': None,
       'grid_template_rows': None,
       'height': None,
       'justify_content': None,
       'justify_items': None,
       'left': None,
       'margin': None,
       'max_height': None,
       'max_width': None,
       'min_height': None,
       'min_width': None,
       'object_fit': None,
       'object_position': None,
       'order': None,
       'overflow': None,
       'overflow_x': None,
       'overflow_y': None,
       'padding': None,
       'right': None,
       'top': None,
       'visibility': None,
       'width': None}},
     '41708465e7904d2b8ec584ef0755bffd': {'model_name': 'VBoxModel',
      'model_module': '@jupyter-widgets/controls',
      'model_module_version': '1.5.0',
      'state': {'_dom_classes': ['widget-interact'],
       '_model_module': '@jupyter-widgets/controls',
       '_model_module_version': '1.5.0',
       '_model_name': 'VBoxModel',
       '_view_count': None,
       '_view_module': '@jupyter-widgets/controls',
       '_view_module_version': '1.5.0',
       '_view_name': 'VBoxView',
       'box_style': '',
       'children': ['IPY_MODEL_4aacb2793a314676b0e2f7019098c470',
        'IPY_MODEL_1221e24520064dfeb77f760ba87f4793'],
       'layout': 'IPY_MODEL_681322fe51bb4ffb9cde6343142607ff'}},
     '07cc1dbe3c584ec2ab2e2e38293d6f86': {'model_name': 'LayoutModel',
      'model_module': '@jupyter-widgets/base',
      'model_module_version': '1.2.0',
      'state': {'_model_module': '@jupyter-widgets/base',
       '_model_module_version': '1.2.0',
       '_model_name': 'LayoutModel',
       '_view_count': None,
       '_view_module': '@jupyter-widgets/base',
       '_view_module_version': '1.2.0',
       '_view_name': 'LayoutView',
       'align_content': None,
       'align_items': None,
       'align_self': None,
       'border': None,
       'bottom': None,
       'display': None,
       'flex': None,
       'flex_flow': None,
       'grid_area': None,
       'grid_auto_columns': None,
       'grid_auto_flow': None,
       'grid_auto_rows': None,
       'grid_column': None,
       'grid_gap': None,
       'grid_row': None,
       'grid_template_areas': None,
       'grid_template_columns': None,
       'grid_template_rows': None,
       'height': None,
       'justify_content': None,
       'justify_items': None,
       'left': None,
       'margin': None,
       'max_height': None,
       'max_width': None,
       'min_height': None,
       'min_width': None,
       'object_fit': None,
       'object_position': None,
       'order': None,
       'overflow': None,
       'overflow_x': None,
       'overflow_y': None,
       'padding': None,
       'right': None,
       'top': None,
       'visibility': None,
       'width': None}},
     '468c3fb3ac864c41b384c8417b7c1ccd': {'model_name': 'SliderStyleModel',
      'model_module': '@jupyter-widgets/controls',
      'model_module_version': '1.5.0',
      'state': {'_model_module': '@jupyter-widgets/controls',
       '_model_module_version': '1.5.0',
       '_model_name': 'SliderStyleModel',
       '_view_count': None,
       '_view_module': '@jupyter-widgets/base',
       '_view_module_version': '1.2.0',
       '_view_name': 'StyleView',
       'description_width': '',
       'handle_color': None}},
     '4aacb2793a314676b0e2f7019098c470': {'model_name': 'IntSliderModel',
      'model_module': '@jupyter-widgets/controls',
      'model_module_version': '1.5.0',
      'state': {'_dom_classes': [],
       '_model_module': '@jupyter-widgets/controls',
       '_model_module_version': '1.5.0',
       '_model_name': 'IntSliderModel',
       '_view_count': None,
       '_view_module': '@jupyter-widgets/controls',
       '_view_module_version': '1.5.0',
       '_view_name': 'IntSliderView',
       'continuous_update': True,
       'description': 'x',
       'description_tooltip': None,
       'disabled': False,
       'layout': 'IPY_MODEL_07cc1dbe3c584ec2ab2e2e38293d6f86',
       'max': 30,
       'min': -10,
       'orientation': 'horizontal',
       'readout': True,
       'readout_format': 'd',
       'step': 1,
       'style': 'IPY_MODEL_468c3fb3ac864c41b384c8417b7c1ccd',
       'value': 10}},
     '855d7e689a6345f1b7571114e5eca6dd': {'model_name': 'LayoutModel',
      'model_module': '@jupyter-widgets/base',
      'model_module_version': '1.2.0',
      'state': {'_model_module': '@jupyter-widgets/base',
       '_model_module_version': '1.2.0',
       '_model_name': 'LayoutModel',
       '_view_count': None,
       '_view_module': '@jupyter-widgets/base',
       '_view_module_version': '1.2.0',
       '_view_name': 'LayoutView',
       'align_content': None,
       'align_items': None,
       'align_self': None,
       'border': None,
       'bottom': None,
       'display': None,
       'flex': None,
       'flex_flow': None,
       'grid_area': None,
       'grid_auto_columns': None,
       'grid_auto_flow': None,
       'grid_auto_rows': None,
       'grid_column': None,
       'grid_gap': None,
       'grid_row': None,
       'grid_template_areas': None,
       'grid_template_columns': None,
       'grid_template_rows': None,
       'height': None,
       'justify_content': None,
       'justify_items': None,
       'left': None,
       'margin': None,
       'max_height': None,
       'max_width': None,
       'min_height': None,
       'min_width': None,
       'object_fit': None,
       'object_position': None,
       'order': None,
       'overflow': None,
       'overflow_x': None,
       'overflow_y': None,
       'padding': None,
       'right': None,
       'top': None,
       'visibility': None,
       'width': None}},
     '1221e24520064dfeb77f760ba87f4793': {'model_name': 'OutputModel',
      'model_module': '@jupyter-widgets/output',
      'model_module_version': '1.0.0',
      'state': {'_dom_classes': [],
       '_model_module': '@jupyter-widgets/output',
       '_model_module_version': '1.0.0',
       '_model_name': 'OutputModel',
       '_view_count': None,
       '_view_module': '@jupyter-widgets/output',
       '_view_module_version': '1.0.0',
       '_view_name': 'OutputView',
       'layout': 'IPY_MODEL_855d7e689a6345f1b7571114e5eca6dd',
       'msg_id': '',
       'outputs': []}}},
    'version_major': 2,
    'version_minor': 0}}},
 'nbformat': 4,
 'nbformat_minor': 4}

Thinking maybe it may have something to do with trust
(https://jupyter-notebook.readthedocs.io/en/latest/security.html#explicit-trust)

I tried:

from nbformat.sign import TrustNotebookApp 
TrustNotebookApp().sign_notebook(nb)

But no joy 😞

@akhmerov any ideas?

[choldgraf]

ah yes the trust issue does sound familiar to me, I believe I've run into that before with the traditional notebook interface (you'd have to explicitly say you trusted the notebook in order to keep the outputs).

Perhaps this is something that nbclient should tackle - the devs there have been quite active recently in getting it ready to replace the executor in nbconvert

[akhmerov]

Odd: I've never encountered it in jupyter-sphinx, and there the notebooks are generated on the fly and never trusted.

[chrisjsewell]

you'd have to explicitly say you trusted the notebook in order to keep the outputs

Yeh I'm just unclear how to programmatically check/manipulate trust, or where in the execution code it decides to only keep the text/plain mimebundle (I'm assuming that is what's happening, and not anything on the actual ipywidgets end).

[chrisjsewell]

Note TrustNotebookApp is what is called when you run jupyter trust ipywidgets.ipynb, but I'm not sure above if it is having any affect

[chrisjsewell]

@akhmerov @choldgraf can you try running my minimal example at your end, and see if you get the same result

[chrisjsewell]

Yeh I don't know if its anything to do with trust, because I can run this fine in jupyter notebook, with the notebook not trusted

[chrisjsewell]

Also I tried this with nbclient.execute (version 0.1.0) and I get the same result

[choldgraf]

Note that I think this is the issue in nbclient to track for a fix: jupyter/nbclient#24

[choldgraf]

I believe this has now been addressed in nbclient, and is in the alpha release (0.4) of nbclient if we want to test things: jupyter/nbclient#68