implement integrated jupytext support for cocalc-jupyter

[williamstein]

Jupytext adds a menu like this:

So we would add a similar menu and on the backend we would add an extra hook that runs jupytext after saving to ipynb (and when loading as well).

There's an interesting discussion about notebook formats here, and my takeaway is that the best thing for cocalc to do is just make it as easy as possible for people to use jupytext. Another ticket motivated by that discussion would be to use nbdime (and/or jupytext?) to improve the diff view of our TimeTravel viewer for Jupyter notebooks.

[williamstein]

One comment says "Myst-MD notebooks work well with Sphinx and allow cross references and citations,", where Myst-MD is one of the Jupytext options. With CoCalc it would be natural to have a modified frontend that supports and renders extra Myst-MD references and citations (it would just be a different frame type).

[parmentelat]

Hi
Just one word from a user, I thought maybe my own particular workflow could shed some light on some expectations

I used to use .ipynb like everyone else when there was no other choice; howeer when given the chance I went for text-based format only - i.e. not paired with ipynb at all - because it solved so many irritating issued in one stone: easier diffs; easier editions with a text editor for massive replacements or other cleanups; and getting of the outputs that I never want to keep - an opinionated choice, agreed

at that point I've had a rather hesitating period as to which format I would use
in some cases I find Python is the best choice because it is mostly code, and running the notebook directly makes sense; in some others, the contents is vastly text with only a few code snippets, so markdown looks better
then inside the Python family I kind of settled on py:percent because of its native support in vscode; as far as Markdown, over the last year or so I found that Jupyter-book was a nice alternative to older sphinx-based approaches that I'd been using to produce handouts, and so md:myst is my format of choice here

I think it's ok if not every single myst-specific trick is supported, i.e. if some second-order niceties won't render too well; using such features for crucial parts would seem a little premature imho as of now at least

what I do expect though would be the ability to drop my master notebooks into cocalc as-is, without the need for transcoding first; so what I miss most is not so much the menu that you outline in the first post; instead it would be to have cocalc find out that a given .py or .md file is actually a notebook, and open it as such on its own;
it's ok to have to configure the course somewhere to enable that feature if one believes it can be confusing to others, although it feels like this is not quite necessary, imho again

all this of course is just my take on this, hope this helps though

[mforbes]

I have been playing a bit with JupyterBook (actually, just using Sphinx with the MyST extensions) and think that MyST is a very reasonable option. For example, it might be nice to include MyST support for regular Markdown files which could then be quite easily turned into other forms of documentation.

[williamstein]

I've done some myst-related investigation and made a ticket here about steps toward iteratively supporting myst: #5948

[mforbes]

In the meantime, are there a "on save" or "on refresh" hooks we could use to run jupytext whenever CoCalc saves or refreshes a notebook?

[williamstein]

In the meantime, are there a "on save" or "on refresh" hooks we could use to run jupytext whenever CoCalc saves or refreshes a notebook?

No, but if there were can you remind me how you might want to configure them?

When I asked phind.com (so GPT-4) it suggests that the only way to do this in Jupyter now is via writing a custom extension: https://www.phind.com/search?cache=c76fba51-8385-4427-8108-e40c05a70121 so maybe there is no standard way.

Have you considered using the unix watch command in a terminal -- it'll run anything you want whenever a file is changed...