Display images of various format generated by clojure code in dedicated buffers or inline in the REPL

[blueberry]

Bozidar mentioned that he already has this Idea, but I imagine that it is not high priority now.

I am opening this issue to hear thoughts from other interested users. I would implement that if I wasn't total elisp beginner, but am willing to take on this if in the future (currently I only have tiny time for learning elisp, so that won't be soon! :)

[bbatsov]

Basically the value handlers (the code that deals with return values) should check if the type of some return value is an image and process this differently.

[arrdem]

Kinda poking at this, since I've been doing a bunch of graphs recently. It looks like there's an old sketch at this from Phil here - https://github.com/technomancy/nrepl-discover/blob/master/src/nrepl/discover/samples.clj#L135

[bbatsov]

Yeah, it'd be really nice to finally implement this. Phil was working on middleware discovery and support for rich content types, but sadly this work never got finished. For the images themselves I think it'd be enough to just enhance the eval op to put some content type for special types in the response (and perhaps deliver the images in a format more convenient for external consumption) and on the client side we can look for such results and just visualize them.

[bbatsov]

Actually I noticed that this is pretty much what Phil has done. :-)

[arrdem]

So part one of this is ... probably just picking machinery on the Clojure side for identifying "images" in results which could be displayed nicely.

Part two here as Bozhidar mentioned is extending nrepl-make-response-handler to be more general, in terms of accepting eg. a content-type handler or even just accepting an alist from response types to handlers or some other dispatching structure. Took me a bit of digging to identify that.

Probably the thing to do which is truest to Phil's original vision would be to allow the user to supply their own alist of content-types to content-handlers - see the original idea of defining editor overlays and such with rich data fed from the server.

[arrdem]

@bbatsov yeah the REPL side of this is exactly what Phil's got a prototype of above. Looks like wiring this up on the CIDER side shouldn't be too hard, just need to make nrepl-make-response-handler less dumb and leverage Emacs' existing support for image rendering.

[stardiviner]

Can it support other packages like ob-clojure-literate?

Even though I already implemented a way to display inline image result in
Org-mode buffer. But the implemented method is limited on "Incanter".

As this link:

https://github.com/technomancy/nrepl-discover/blob/master/src/nrepl/discover/samples.clj#L135

code shows, might can improve it.

And hope CIDER can support Org-mode ob-clojure.el inline image result better (currently not supported).

[bbatsov]

Well, if the core functionality is there it can be extended to other packages easily.

[arrdem]

I totally got a cut at this working, between some patches to nrepl-client.el, cider-repl.el and cider/cider-nrepl. The really sticky bits in the changeset are dealing with the various point expectations in cider-repl.el, because the REPL as currently written relies on insert-before-markers to achieve output insertion and it only supports inserting text, not images which means I wound up doing some manual point management which .. isn't ideal.

cider/cider-nrepl patch

diff --git a/project.clj b/project.clj
index 1ebefe0..446620e 100644
--- a/project.clj
+++ b/project.clj
@@ -60,6 +60,7 @@ (defproject cider/cider-nrepl VERSION
                                                      cider.nrepl/wrap-macroexpand
                                                      cider.nrepl/wrap-ns
                                                      cider.nrepl/wrap-out
+                                                     cider.nrepl/wrap-content-type
                                                      cider.nrepl/wrap-pprint
                                                      cider.nrepl/wrap-pprint-fn
                                                      cider.nrepl/wrap-profile
diff --git a/src/cider/nrepl.clj b/src/cider/nrepl.clj
index 1587e7d..bdce0b4 100644
--- a/src/cider/nrepl.clj
+++ b/src/cider/nrepl.clj
@@ -124,6 +124,15 @@ (def-wrapper wrap-pprint cider.nrepl.middleware.pprint/handle-pprint
                :optional (merge wrap-pprint-fn-optional-arguments
                                 {"pprint" "If present and non-nil, pretty-print the result of evaluation."})}}}))
 
+(def-wrapper wrap-content-type cider.nrepl.middleware.content-type/handle-content-type
+  #{"eval"}
+  {:doc "Middleware that adds `content-type` annotoations to the result of the the eval op."
+   :expects #{"eval" "load-file"}
+   :returns {"content-type"
+             "A MIME type for the response, if one can be detected."}
+   :handles {"content-type-middleware"
+             {:doc "Enhances the `eval` op by adding `content-type` to some responses. Not an op in itself."}}})
+
 (def-wrapper wrap-apropos cider.nrepl.middleware.apropos/handle-apropos
   {:doc "Middleware that handles apropos requests"
    :handles {"apropos"
diff --git a/src/cider/nrepl/middleware/content_type.clj b/src/cider/nrepl/middleware/content_type.clj
new file mode 100644
index 0000000..8075b40
--- /dev/null
+++ b/src/cider/nrepl/middleware/content_type.clj
@@ -0,0 +1,43 @@
+(ns cider.nrepl.middleware.content-type
+  (:import clojure.tools.nrepl.transport.Transport
+           java.io.File))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+(defonce recursive?
+  (atom {}))
+
+(defn response+content-type [{:keys [session value] :as response}]
+  (cond (and (instance? File value)
+             (.exists ^File value))
+        (assoc response
+               :value (.getCanonicalPath ^File value)
+               :content-type (.. ^File value getCanonicalFile toURL openConnection getContentType))
+
+        (instance? java.awt.Image value)
+        (with-open [bos (java.io.ByteArrayOutputStream.)]
+          (javax.imageio.ImageIO/write value bos "png") ;; FIXME, use other types?
+          (assoc response
+                 :content-type "image/png;base64"
+                 :value (.encodeToString java.util.Base64$Encoder (.toByteArray bos))))
+
+        :else response))
+
+(defn content-type-transport
+  [^Transport transport]
+  (reify Transport
+    (recv [this]
+      (.recv transport))
+    (recv [this timeout]
+      (.recv transport timeout))
+    (send [this response]
+      #_(.send transport {:out (str (:value response))})
+      #_(.send transport {:out (str (type (:value response)))})
+      (.send transport (#'response+content-type response)))))
+
+(defn handle-content-type
+  [handler msg]
+  (let [{:keys [op transport]} msg]
+    (handler (if (#{"eval" "load-file"} op)
+               (assoc msg :transport (#'content-type-transport transport))
+               msg))))

cider patch

diff --git a/cider-repl.el b/cider-repl.el
index 345a3c2..48fd6d4 100644
--- a/cider-repl.el
+++ b/cider-repl.el
@@ -799,23 +799,56 @@ the symbol."
                t)))
           (t t))))
 
+(defun cider-repl--display-image (type buffer string &optional show-prefix bol)
+  (with-current-buffer buffer
+    (save-excursion
+      (cider-save-marker cider-repl-output-start
+        (cider-save-marker cider-repl-output-end
+          (goto-char cider-repl-input-start-mark)
+          (when (and bol (not (bolp)))
+            (insert-before-markers "\n"))
+          (when show-prefix
+            (insert-before-markers (propertize cider-repl-result-prefix 'font-lock-face 'font-lock-comment-face)))
+          (let ((image (create-image (substring string 1 -1) type)))
+            (insert-image image string))
+          (set-marker cider-repl-input-start-mark (point) buffer)
+          (set-marker cider-repl-prompt-start-mark (point) buffer))))
+    (cider-repl--show-maximum-output)))
+
+(setq cider-repl-content-type-handler-alist
+  '(("image/jpeg" . (lambda (u v s b) (cider-repl--display-image 'jpeg u v s b)))
+    ("image/jpeg;base64" . (lambda (u v s b) (cider-repl--display-image 'jpeg u (base64-decode-string v) s b)))
+    ("image/png". (lambda (u v s b) (cider-repl--display-image 'png u v s b)))
+    ("image/png;base64" . (lambda (u v s b) (cider-repl--display-image 'png u (base64-decode-string v) s v)))))
+
 (defun cider-repl-handler (buffer)
   "Make an nREPL evaluation handler for the REPL BUFFER."
-  (nrepl-make-response-handler buffer
-                               (let (after-first-result-chunk)
+  (let (after-first-result-chunk
+        force-prompt)
+    (nrepl-make-response-handler buffer
                                  (lambda (buffer value)
                                    (cider-repl-emit-result buffer value (not after-first-result-chunk) t)
-                                   (setq after-first-result-chunk t)))
+                                   (setq after-first-result-chunk t))
                                (lambda (buffer out)
                                  (cider-repl-emit-stdout buffer out))
                                (lambda (buffer err)
                                  (cider-repl-emit-stderr buffer err))
                                (lambda (buffer)
-                                 (cider-repl-emit-prompt buffer))
+                                 (cider-repl-emit-prompt buffer)
+                                 (let ((win (get-buffer-window (current-buffer) t)))
+                                   (when (and win force-prompt)
+                                     (with-selected-window win
+                                       (set-window-point win cider-repl-input-start-mark))
+                                     (cider-repl--show-maximum-output))))
                                nrepl-err-handler
-                               (let (after-first-result-chunk)
-                                 (lambda (buffer pprint-out)
-                                   (cider-repl-emit-result buffer pprint-out (not after-first-result-chunk))
+                               (lambda (buffer pprint-out)
+                                 (cider-repl-emit-result buffer pprint-out (not after-first-result-chunk))
+                                 (setq after-first-result-chunk t))
+                               (lambda (buffer value content-type)
+                                   (if-let ((handler (cdr (assoc content-type cider-repl-content-type-handler-alist))))
+                                       (progn (funcall handler buffer value (not after-first-result-chunk) t)
+                                              (setq force-prompt t))
+                                     (cider-repl-emit-result buffer value (not after-first-result-chunk) t))
                                    (setq after-first-result-chunk t)))))
 
 (defun cider-repl--send-input (&optional newline)
diff --git a/nrepl-client.el b/nrepl-client.el
index 058caf9..59c1361 100644
--- a/nrepl-client.el
+++ b/nrepl-client.el
@@ -763,7 +763,8 @@ to the REPL."
 (defun nrepl-make-response-handler (buffer value-handler stdout-handler
                                            stderr-handler done-handler
                                            &optional eval-error-handler
-                                           pprint-out-handler)
+                                           pprint-out-handler
+                                           content-type-handler)
   "Make a response handler for connection BUFFER.
 A handler is a function that takes one argument - response received from
 the server process.  The response is an alist that contains at least 'id'
@@ -780,12 +781,14 @@ EVAL-ERROR-HANDLER is nil, the default `nrepl-err-handler' is used.  If any
 of the other supplied handlers are nil nothing happens for the
 corresponding type of response."
   (lambda (response)
-    (nrepl-dbind-response response (value ns out err status id pprint-out)
+    (nrepl-dbind-response response (content-type value ns out err status id pprint-out)
       (when (buffer-live-p buffer)
         (with-current-buffer buffer
           (when (and ns (not (derived-mode-p 'clojure-mode)))
             (cider-set-buffer-ns ns))))
-      (cond (value
+      (cond ((and value content-type content-type-handler)
+             (funcall content-type-handler buffer value content-type))
+            (value
              (when value-handler
                (funcall value-handler buffer value)))
             (out

[bbatsov]

Nice!

because the REPL as currently written relies on insert-before-markers to achieve output insertion and it only supports inserting text, not images which means I wound up doing some manual point management which .. isn't ideal.

You have to start somewhere. ;-) No need to make it perfect from the start.

[bbatsov]

On the bright side - I'm certain that it's much simpler to get this going for interactive evals and the dedicated result buffer. :-)