[Draft] Proposal: Plugin-based Architecture, Extension API

[ches]

This should eventually supersede #64, the idea is to refine notes in this issue into a proposal/spec over time. I might move this to a wiki page as it becomes more spec than background story, so we can track history of edits by collaborators, but at the moment the GitHub wiki is disabled on this project.

I've mixed terms a bit here—should we adopt nomenclature of ensime-vim "extensions" to avoid confusion when we're talking in the context of Vim "plugins"?

I was thinking about our current architecture as it has been evolving, and it occurred to me that it might be improved by embracing a more plugin-based approach, and almost by construction we would end up with an extensible API that makes it easier for users to do custom scripting against ensime-vim and thus contribute new features more easily (pretty much the essence of #64).

I'm referring primarily to message handlers, functions invoked when particular messages are received from ENSIME. We began with a monolithic client that did just about everything:

• Implemented low-level client/server communication logic.
• Defined the higher-level feature-based function interface for sending calls to the server.
• Implemented all the logic for each specific response message handler.
• Had a sprawling/entangled interface to Vim functionality for UI interaction.
• Even started the server itself (via EnsimeLauncher, but the chain of calls originated from the client).

Piece by piece we're breaking this down, decoupling, making testing easier. It's still in progress.

One tool that we've applied crudely to-date is mixins. Look at debugger.py and typecheck.py and they're just moving food around on the plate: they're still tightly coupled to EnsimeClient, directly meddling in its state as self, they just offered some code organization into more manageable files. protocol.py is the same story but it was additionally solving the problem of alternate versions. It was a reasonable evolutionary solution for the state of the code at the time.

My thinking is that these things can become plugins to the low-level client core, instead of mixins. Rough ideas:

• Plugins are classes that get access to an EnsimeClient by constructor injection.
• Plugins are registered with Ensime, and when it creates instances of EnsimeClient it tells them all the known plugins.
• We provide a function decorator for plugins like @handles('SymbolInfo') to declaratively register handlers for received messages.
• EnsimeClient dispatches the appropriate messages to each registered handler when received.
• "Everything is a plugin"—all feature-level API of EnsimeClient (just about any function that sends or receives a specific ENSIME message) is moved out of it. This will include current "core" functionality but a relatively focused feature area like debugger support might be a good initial use case for hashing out the design.
• Handlers can interact with the UI via the editor object of their injected client instance (is there a better way?).

With this, in theory, users could compose features and add new ones without needing to touch core code—drop a .vim script in ~/.vim/after/plugin/ with a :python block that imports a few things from us like the @handles decorator, and off you go. We reap the benefits of eating our own dogfood by using plugins internally: the API is constantly tested, and if its design is flawed we feel the pain.

This likely converges with the direction that I'm (ever so slowly) trying to go with my pip install-able Python ENSIME client project, so it ought to be straightforward to replace ensime-vim's client impl with it if and when that's ready (hoping ensime-sublime would adopt it too).

These are rough, possibly bad ideas. Feedback welcome, and getting to a point of some working code will help to validate. Some concerns to consider further:

• The above is mainly focused on response handlers—is anything needed to make adding new requests simple? You can just define your own new Vim commands/functions/etc. and wire them to your plugin class that sends via its client, but could any helpers we provide make this easier, avoid name clashes, etc.?
• I haven't thought much about how plugins might deal with protocol versions.
• I haven't thought much about surfacing the functionality through VimL APIs, but it seems like it should be straightforward (?). Might be able to leverage User autocmds as I mentioned on Provide generic API to ensime for other Vim plugins to integrate. #64 for events.
• Making unit testing of plugins easy: requests will probably mainly need to assert on a mock client's send_request, response handlers will assert on various operations of a mock Editor. I'm not sure how cumbersome this will be in practice—need to try some code, and the Editor and EnsimeClient APIs are steadily improving which helps. Any room for improvement on coupling to these dependencies?
• Requires a lot more thought/research, but it would be good to consider how handlers could be async in the future—coroutines in Python 3 maybe; Neovim can surely help a lot but how far can we go without substantially diverging code?

Sorry for the wall of text. 😓

[ktonga]

I really like the general idea, it will make everything less coupled, thus, more modular and easier to test/maintain.

I'm especially interested in this kind of functionality since I've been taking a look at how to implement a deoplete source for Scala based on Ensime, but I'm not too happy re-implementing all the client/server communication, so I think it could be implemented as a plugin plugin. It would be also valuable to be able to declare lazy modules only enabled if certain condition is met, such as is deoplete installed.
But I don't know if it is possible to share code between Python plugins, also if py2 and py3 can talk to each other.

Handlers can interact with the UI via the editor object of their injected client instance (is there a better way?).

Hmm, maybe it's a one by one case, but I think that if the handlers on the plugin will be the ones performing the actions according to the server responses I don't see the need to still pass the editor to the client. Also I think we should implement handlers for user actions on the plugin so it is even easier to decouple the editor from the client, so if the user invokes a command, it's handled by the dedicated plugin and it generates the request which is then sent to the server via the client.

With this, in theory, users could compose features and add new ones without needing to touch core code—drop a .vim script in ~/.vim/after/plugin/ with a :python block that imports a few things from us like the @Handles decorator, and off you go. We reap the benefits of eating our own dogfood by using plugins internally: the API is constantly tested, and if its design is flawed we feel the pain.

This sounds really powerful.

Requires a lot more thought/research, but it would be good to consider how handlers could be async in the future—coroutines in Python 3 maybe; Neovim can surely help a lot but how far can we go without substantially diverging code?

There are a few vim plugins supporting asynchronicity via Neovim or Vim8, but all of them are Python3. We should take a look. Would it be that hard for us to migrate to Python3?

[ches]

I think that if the handlers on the plugin will be the ones performing the actions according to the server responses I don't see the need to still pass the editor to the client.

That's a good point, maybe we'll reach the point that EnsimeClient doesn't need an Editor at all. Already the API that I imagine EnsimeClient retaining with this in mind doesn't really use it, except maybe send_at_position which can be approached another way.

Also I think we should implement handlers for user actions on the plugin so it is even easier to decouple the editor from the client, so if the user invokes a command, it's handled by the dedicated plugin and it generates the request which is then sent to the server via the client.

Yep, this is what I had in mind. That's what the first bullet in the list of further considerations is about, I haven't thought about it much yet but I wonder if there's anything extra we'd need to do to make the wiring (all the way down to Vim commands/functions) obvious to a plugin developer. Probably will have to play with it and see.

Would it be that hard for us to migrate to Python3?

At first I wanted to try to support 2 and 3 in my client project to give ensime-vim an easy migration path, but decided the other day to drop that because all the extra work to do that just drains my motivation with limited time to devote. So I will eventually want to make it happen 😁

I haven't tried yet but I'm guessing we wouldn't really have much to do to run ensime-vim on Python 3, the bigger things in my mind had been actually taking advantage of it, and just informing users—for instance even current OS X doesn't ship with Python 3 so binary distributions like MacVim downloads won't have +python3. That can be pretty easily addressed with a Homebrew installation but it may be a change for some people. Ubuntu Xenial now has Vim packages with +python3 only by default, and probably other distros do or soon will too, so the tide is moving…

About actually taking advantage of it, that's largely the point of my side project aside from separate packaging: I wanted to use asyncio instead of a polling thread since it seems a natural fit. There's this websockets impl that seems good so far. How much a coroutine/futures-based API would pervade ensime-vim if using such a client remains to be seen, this is the learning curve I'm going through now when I find the time to work on it…

I'd say let's get through the big PRs/WIP that's outstanding and then give an earnest shot at getting our tests running under Python 3 (I'm starting to add a bunch of tests for my upcoming PR for #325 since it decouples things a lot). If you want to give that a shot sooner though, be my guest—you can probably just rm -rf .venv && make PYTHON=python3 and see what happens!

[jvican]

I really like the idea! It would be great to have a list of items that need ot be carried out so that any of us or future contributors can help out push this down the line.

I haven't tried yet but I'm guessing we wouldn't really have much to do to run ensime-vim on Python 3, the bigger things in my mind had been actually taking advantage of it, and just informing users—for instance even current OS X doesn't ship with Python 3 so binary distributions like MacVim downloads won't have +python3

I'm in for these changes, however are we sure we want to remove support for Python 2? I'm not sure we can make ensime-vimuse coroutines with support for both versions. Python2 is extremely important because it's the default action and python3 is not usually defined by default.

I'd say let's get through the big PRs/WIP that's outstanding and then give an earnest shot at getting our tests running under Python 3.

Absolutely agree 👍

For the async implementation of ensime-vim, vim-plug introduced asynchronous support for neovim -- maybe we can reuse it. I would focus on neovim (and not vim8), we shouldn't get too distracted trying to get support for all editors/platform and instead rely on future contributors to implement these features. With a predecessor or a similar implementation, they should have more easy to make such contributions. That should lower down the learning barrier.

[ches]

It would be great to have a list of items that need ot be carried out so that any of us or future contributors can help out push this down the line.

Makes sense, I'll think of some checklist items and update this issue body with them after finishing #367, that paves the way for this to be much easier to start.

[sl0thentr0py]

Is supporting Python 3 still on the list? I tried quickly testing if it works out of the box with python3 but lettuce does not work, so I tried porting to aloe (following this lettuce issue). Unfortunately pip fails on aloe and I'm not a python magician, so I gave up for now. If someone can point me on how to port it to python3, I would be happy to help. Otherwise, just bumping for updates.

[ches]

Hi @sl0thentr0py, supporting Python 3 is definitely still on the table. Sorry if you went down a rabbit hole with Lettuce though, Python 3 support is one reason that its days are probably numbered for ensime-vim. If we keep Gherkin specs around at all (e.g. if we like it for an integration test setup), I personally would favor porting to pytest-bdd since we have pytest unit tests now and pytest is pretty great.

If you're interested in exploring Python 3 support we'd happy to have your help, just please open a new issue to let people know it's being worked on. If it's tenable I think we'd want to preserve Python 2 support for awhile longer, as noted above some important platforms we wish to support are still py2 out of the box.

[sl0thentr0py]

Hi @ches, thanks for the quick and helpful reply!
I am going to read the source a bit over the next days and if I commit to porting to Python 3, I will let you guys know by opening an issue.

[drewboardman]

Did anything ever happen with this? It would be nice, since the version of VIM packaged in Ubuntu is built against python3.