Routes can be fully dynamic and discovered on a per request basis for hyper scale hosting

[Tratcher]

The current proxy routing design assumes all route data is available up front and can be loaded atomically into the route table. The EndpointDataSource routing model assumes this as well.

For more dynamically-configured services, we probably don't want to fetch the entire route table from the db, but instead have a pull and cache for each hostname model.

How can we lazy-load route, backend, and endpoint data on a per request basis?

Assume we're working with a discrete key like Host, not a complicated best-match scenario like Path, and little or no wildcard support (maybe wildcard subdomains but only for specific depths?).

@davidni is this a scenario you'd evaluated?

@rynowak can such a system work within routing, co-exist side by side routing, or would it completely replace routing?

[samsp-msft]

My mental concept is that routing should be one of the steps in the pipeline. But I don't think we have talked enough about what that pipeline looks like. This kind of matches with the conversation today about:
a) The process being matching the host/path to a set of endpoints -> load balancing then picks the right one from that list, accounting for stickiness -> the connection is chosen from a pool, or a new one is created.
b) Passing of client cert information from the front end to workers as part of headers.

For Antares, I would expect to not use a config based route, and instead have a custom routing component. It would manage the connection to the DB and local cache, and return a set of candidate hosts to subsequent steps including the load balancer.

[Tratcher]

Here's the current flow:

• Routes and their backends are loaded at startup and added to the central route table.
• A request arrives, matched against the route table, and the proxy logic begins.
• Routes are reloaded if the config changes.

Here's the flow I think we'd need:

• No routes are loaded at startup
• Per request a custom middleware checks the local route cache.
  • If there's no match then it checks the central database. Should it also do negative caching?
    • If there's a match it loads all of the Backend and Endpoint information.
  • If there's now a match in the cache the proxy logic begins.
• There needs to be change notifications from the database to update items in the cache
• There needs to be idle timeout eviction of routes and backends.

Components that likely need to be redesigned/replaced:

• ReverseProxyConfigManager
• IRouteManager
• IProxyDynamicEndpointDataSource
• DynamicConfigBuilder
• IBackendsRepo
• IRoutesRepo

Need updates:

• IBackendManager - Needs idle timeout cleanup

[rynowak]

Routing's implementation requires all of the path matching logic and the collection Endpoint (routing's definition of Endpoint, not the Proxy's) objects to be available synchronously. For anything that isn't path based you have full control including the ability to run async code.

You can ultimately just do special casing in routing with a catch-all route, and then you have full control.

[Tratcher]

You can ultimately just do special casing in routing with a catch-all route, and then you have full control.

At that point, why use routing at all?

[davidfowl]

Say we don’t use routing, what features would we end up re-implementing and is that worth it for what you would gain?

[Tratcher]

Depends on how strong your db lookup key is. If it's keyed by host and always unique then great. If you need to resolve a set and then find the closest match base on something like path then you'd need to reuse some routing mechanics.

Definitely need customer requirement to constrain this scenario or else it will quickly get out of hand.

[rynowak]

can such a system work within routing, co-exist side by side routing, or would it completely replace routing?

The other thing that I'd mention here is that you don't have to use the routing middleware to use routing's concepts. The contract of the routing middleware is that it assigns an endpoint to the request, so it could be a different implementation and no one would care.

[rynowak]

For more dynamically-configured services, we probably don't want to fetch the entire route table from the db, but instead have a pull and cache for each hostname model.

I guess what I want to know what's the hierarchy of data. Is hostname the first criteria to match, and then only after that we start processing other request-related properties.

Basically if you want to do lazy-loading you have to opinionated about what the boundaries are that are lazy. So whether or not the existing implementation does what you want depends a lot on those boundaries.

[Tratcher]

The contract of the routing middleware is that it assigns an endpoint to the request, so it could be a different implementation and no one would care.

Exactly, I expect a lazy load middleware to do it's own routing and assign an endpoint, but the rest of the pipeline remains the same.

[halter73]

Do we really need to redesign routing to lazily load routes on a per-hostname basis?

Or would we consider this an advanced enough scenario to require some custom code at the beginning of the middleware pipeline that checks the host name, and sees if non-expired routes are already cached for the hostname? If there are no valid routes, it could delay the rest of the pipeline until the routes are loaded.

[samsp-msft]

I go with @Tratcher - in this case its a complete replacement of the routing module. It does its own thing and returns the endpoints for loadbalancing etc to work on.
It would be nice (low pri) if it needs to do sub-routes based on path to be able to have sub-instances of route tables - but that wouldn't be something config needs to be able to express.

[Tratcher]

Or would we consider this an advanced enough scenario to require some custom code at the beginning of the middleware pipeline that checks the host name, and sees if non-expired routes are already cached for the hostname? If there are no valid routes, it could delay the rest of the pipeline until the routes are loaded.

That's pretty much the component I'm talking about. By the time it's done the work to load the route or to check that the route is already loaded, actually running routing is redundant, it can already assign the resulting endpoint.