Support for plugins

[douglas-reid]

The mixer should support dynamic discovery/loading of adapters via a plugin model. There are a number of possible approaches, including usage of the coming go 1.8 plugin package.

[louiscryan]

@jba I think this is one area where your Go expertise would be most welcome

[jba]

Happy coincidence: I just learned about the 1.8 plugin package today. It's definitely the way to go, if you want to go that way.

However, the motivation for the plugin package was the go command-line tool (AFAICT). You're talking about adding plugins to a service, is that correct? I would think that any sort of dynamic code loading for a high-availability service would be destabilizing.

I think you should be really clear about the use cases, and make sure the benefits outweigh the risks.

[douglas-reid]

We definitely need more clarity here.

At first blush, the thinking was that it might useful to have a way to load plugins at service (re)start that weren't directly compiled into the mixer. We'd obviously want to find a way to protect the mixer from plugin panics, etc.

[jba]

If this a feature to enable third-party users to extend their own mixer instances without touching the code, I can see its benefit.

If it's to enable instances running in Google to be hot-patched without going through a release process, then it's probably unwise. Talk to SRE.

[louiscryan]

Definitely the former. This is a pretty common pattern for performance but extensible platforms in OSS, e.g. nginx

…

On Fri, Dec 2, 2016 at 8:36 AM, Jonathan Amsterdam ***@***.*** > wrote: If this a feature to enable third-party users to extend their own mixer instances without touching the code, I can see its benefit. If it's to enable instances running in Google to be hot-patched without going through a release process, then it's probably unwise. Talk to SRE. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIoKPJZgbL9Ob3vDbNZR_WV6qr4MyOCDks5rEEj1gaJpZM4LAywa> .

[mandarjog]

Other reason is to only load adapters that are needed out of all those are available. Regards, Sent from my Ericsson Blaze Mandar U Jog

…

On Dec 2, 2016, at 8:36 AM, Jonathan Amsterdam ***@***.***> wrote: If this a feature to enable third-party users to extend their own mixer instances without touching the code, I can see its benefit. If it's to enable instances running in Google to be hot-patched without going through a release process, then it's probably unwise. Talk to SRE. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[geeknoid]

The one big problem with the Go plugin model is likely to be resource consumption. Every plugin will end up being huge and share no code with the mixer proper. Having a dozen adapters loaded as plugins compared to being linked directly into the mixer is likely to be >1G additional RAM consumption, and inherently lower efficiency due to increase CPU cache and TLB miss rates.

I was at first hoping that all adapters would be handled as plugins. But now that I realize the cost, I'm not sure this is a good idea at all. We'll have to measure to know for sure I suppose.

[meyerzinn]

Have y'all considered the approach taken by Caddyserver for plugin management? @mholt, the author, knows more about this than I, but basically plugins register themselves in their init function, as shown on the wiki page (https://github.com/mholt/caddy/wiki/Extending-Caddy):

import "github.com/mholt/caddy"

func init() {
	// register a "generic" plugin, like a directive or middleware
	caddy.RegisterPlugin("name", myPlugin)

	// register a plugin that can load the Caddyfile when Caddy starts
	caddy.RegisterCaddyfileLoader("name", myLoader)

	// register a plugin that implements an entire server type
	// for use with Caddy
	caddy.RegisterServerType("name", myServerType)

	// register a function that runs when Caddy emits events
	caddy.RegisterEventHook("eventName", myHookFn)

	// add a function that wraps listeners for the HTTP server
	// (it's more common for a directive to call this rather than a standalone plugin)
	httpserver.AddListenerMiddleware(myListenerMiddleware)

	// ... there are others. See the godoc.
}

Then, the Caddy buildworker (https://github.com/caddyserver/buildworker) builds Caddy like so:

// CaddyPlugin holds information about a Caddy plugin to build.
type CaddyPlugin struct {
	Package string `json:"package"` // fully qualified package import path
	Version string `json:"version"` // commit, tag, or branch to checkout
	Repo    string `json:"repo"`    // git clone URL -- TODO: used?
	Name    string `json:"-"`       // name of plugin: not used here, but used by devportal
	ID      string `json:"-"`       // ID of plugin: not used here, but used by devportal
}

// BuildConfig holds information to conduct a build of some
// version of Caddy and a number of plugins.
type BuildConfig struct {
	CaddyVersion string        `json:"caddy_version"`
	Plugins      []CaddyPlugin `json:"plugins"`
}

Finally, as you can see at https://github.com/caddyserver/buildworker/blob/master/buildenv.go#L730, it plugs in all the plugins configured for this build environment before building caddy itself.

I don't know what the memory consumption of Go's plugin system is, but this seems to work well for Caddy. This would require a re-working of Istio's core to make it more event-focused (or other plugin patterns?) so it's probably not happening in the immediate future, but it's a thought.

[elevran]

Having plugin's (and especially "binary" plugins) would help the operator in configuring an instance of mixer that's best suited for their environment - either by removing unused components or adding different ones that are environment or tool specific.

This is more of a "packaging" concern, where mixer and plugin's can be compiled independently and assembled when needed for deployment. This can take the form of a Dockerfile (e.g., FROM mixer-base COPY plugins ...) or by mounting a volume hosting plugins, allowing easy and independent update to mixer and the plugins.
In either case, the loading of plugin's happens only at mixer start-up time and is not required to change at run-time. Changes to a deployment should be based on immutable infrastructure - if an update is needed a new deployment artifact should be created.