Allow CIDER to load its own middleware via the sideloader

[plexus]

nREPL now has dynamic insertion of middleware, as well a sideloader for requesting resources from the client. CIDER has basic support for sideloading, so it would seem that all the pieces are there to be able to take a vanilla nREPL connection, and upgrade it to a fully CIDER-compliant connection.

After initial testing it seems this isn't quite there yet, there are small tweaks needed in CIDER, cider-nrepl, and nREPL, so this is a meta-issue to keep track of the overall goal and progress.

• Sideloader: no line breaks in base64 content #3038
• Allow starting the sideloader for the tooling session #3039
• Sideloader state in session nrepl/nrepl#243
• Use session classloader when loading deferred middleware cider-nrepl#705
• Sideloader: handle binary files, support multiple directories #3041
• Log errors that happen when loading dynamic middleware nrepl/nrepl#244
• Dynamically upgrade nrepl connection #3044
• Sideloader+dynload dogfooding nrepl/nrepl#246

Related

• ClassLoader streamlining and composability #3049

How to try it out

Start a vanilla nREPL server:

clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version "0.9.0-beta1"}}}' -M: -m nrepl.cmdline --port 1234

Connect to it from Emacs:

(setq
 ;; Make sure we can inspect the nREPL traffic
 nrepl-log-messages t
 ;; Prevent the nrepl pprint middleware from looking for a print function that
 ;; isn't (yet) there. Not necessary but prevents some noise.
 cider-print-fn nil
 ;; Use the cider-nrepl jar that I `make install'ed locally
 ;; we need the classloader-related fixes in the deferred middleware
 cider-dynload-cider-nrepl-version "0.27.0-SNAPSHOT"
 ;; Make sure caching doesn't bite us
 cider-jar-content-cache (make-hash-table :test #'equal))

;; Start the connection
(cider-nrepl-connect
 '( :project-dir "/home/arne/clj-projects/timezoner"
    :host "localhost"
    :port 1234
    :repl-init-function nil
    :session-name nil
    :repl-type clj))

;; Go to a clojure or cider repl buffer and do
;; M-x cider-upgrade-nrepl-connection

[plexus]

With the above changes to cider/cider-nrepl/nrepl I can load the middleware, and invoke a cider-version. Not everything is working yet (still trying things out and investigating) but this is a promising start.

[plexus]

Question regarding the sideloader design for @shen-tian. It seems like there's no way to distinguish between "file not found" and "empty file"? Maybe not the most common edge case but still seems significant...

[bbatsov]

While I don't remember the details as well as Shen, I'm assuming that's just something that came from the early sideloader prototype that was created by @cgrand. Perhaps the reasoning was that an empty file and a missing file are more or less the same as far as the sideloader is concerned. We can probably make the responses more granular if that'd help somehow.

[plexus]

With the last two PRs this actually seems to pretty much work as intended. It's a little slow (so might be worth considering caching responses, since currently every clj file gets served twice), and I didn't try every middleware (the deferred loading means there might still be errors lurking there), but all in all this is quite promising, although you still need a bit of low-level invocations to make it work. Maybe it's time to think about the ergonomics of this. Is the end goal to bundle cider-nrepl (and orchard) with cider? I think it would be preferable to download the jars from clojars, and either adapt our resource loading to deal with the zip files, or just unzip them in a cache location. So we'll need some logic that does that. and then I guess ultimately you would have something like M-x cider-upgrade-nrepl or even (setq cider-auto-upgrade-nrepl t) that detects if the middleware is there or not.

When running directly from source I also had to edit cider.nrepl.version because it references project.clj in a way that doesn't resolve, but I think when we look it up in the jar that shouldn't be an issue.

Looping in @mk at this point

[bbatsov]

Is the end goal to bundle cider-nrepl (and orchard) with cider? I think it would be preferable to download the jars from clojars, and either adapt our resource loading to deal with the zip files, or just unzip them in a cache location. So we'll need some logic that does that. and then I guess ultimately you would have something like M-x cider-upgrade-nrepl or even (setq cider-auto-upgrade-nrepl t) that detects if the middleware is there or not.

Yep, that's the end goal. Or at least one of them - I envision that every client will use sideloading for their deps (e.g. clj-refactor, sayid, etc). The ergonomics definitely have to be improved and that's the main reason I started to work on this and I dropped it back in the day - I couldn't figure out how to properly approach this. E.g. I was thinking of bundling CIDER's deps with the Emacs package and have it sideload them from some lib folder, but this required developing some mapping between the jars and the classes in them (if you wanted to to unzip them on the demand, that is). Also I didn't feel very comfortable committing big binary blobs to the git repo.

I'd be fine with fetching deps from Clojars, although this might slow down the first jack-in/connect. I was also thinking there some be some configurable list of folders, from which you can sideload resources. (useful for user extensions based on eval)

Anyways, I'm totally open to ideas how to make this optimal.

[plexus]

in any case I think it should be opt-in, the current jack-in approach works fine and has benefits over the sideloading (speed, having resources actually on the classpath).

[mk]

@plexus great to see this being this far along already, thank you! 👏👏👏

Regarding fetching deps I'm wondering if there's a way to delegate this to another tool. Could downloading the jars could be handled by shelling out to clojure maybe? Similar to what jack-in does but appending -P to not actually start anything but just download the deps?

[bbatsov]

You have to keep in mind that when someone is doing cider-connect we still want to inject cider-nrepl, but it's not clear whether the user has something like lein or clj available. Unix utilities like curl and wget are unlikely to available on Windows. Generally, it's best to stick to portable Elisp implementation, although I'm not sure if we can handle unzipping in this way. A quick search that's probably not possible. Perhaps it's fine to reach for Unix utils at first and figure out some Windows support down the road...