Rebuilding index doesn't update the indexes of front end servers in a load balanced environment

[jaandrews]

When I rebuild the index via the examine management tab in a load balanced environment on azure, the front end indexes are not updated. I confirmed that they are updated when publishing a node, so I doubt this is a configuration issue.

Umbraco version

I am seeing this issue on Umbraco version: 8.2.2

Reproduction

Bug summary

In a load balanced environment, the indexes on the front end servers are not updated when I rebuild the index via the examine management section in the umbraco backoffice on the admin environment.

Specifics

I have a load environment with 1 admin server and 2 front end server. I've configured the following in the web config.

• Umbraco.Core.LocalTempStorage = EnvironmentTemp
• Umbraco.Examine.LuceneDirectoryFactory = Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine
  Note that when I publish a node, the index updates properly on the front end servers. It only fails to update when doing the bulk rebuild in the examine management section. I am able to work around this issue by deleting the indexes on the front end server to force them to regenerate.

Steps to reproduce

I identified this issue by doing the following.

1. Rebuilt index with examine manager in admin environment.
2. Checked search results page on admin server.
3. Checked search results on front end server.
4. Confirmed that the results are different.

Expected result

The same search results on the front end servers as the admin server.

Actual result

I received different search results.

[Shazwazza]

This is expected behavior currently. There is no 'good' way to rebuild all indexes on your replica/front-end servers without doing it on those servers or clearing them on those servers so they rebuild. This is one of those things that has been discussed for quite some time. I wish I could find one of the original chats about this and what the forward plan is but I'm unsure where that is, probably on the old issue tracker. There's quite a few forum posts about this same thing but I can't find the original discussion on this, so here's a re-cap of what needs to be done and how other people have resolved this. Perhaps @Jeavon can remember?

In order for the rebuild indexes button to work on all servers a signal needs to be sent to all servers. This is done with ICacheRefresher. It is possible to build a custom cache refresher for this single purpose with a custom dashboard like "Rebuild All Indexes on All Servers". This would trigger the cache refresher and all servers would read the instruction to rebuild and process the rebuild. It's probably not too hard to build that but for something we'd want in core or in a package it could probably be much nicer so you could trigger index rebuilds on specific servers. But to do that we'd need to track active servers. There was a note from a few years back about that here https://our.umbraco.com/forum/using-umbraco-and-getting-started/87779-load-balance-on-azure-index-rebuild-issue#comment-277844 and it could be possible but currently we don't track active servers.

[jaandrews]

K. I might be able to hook into an angularjs interceptor to trigger the refresh when the "Rebuild" button is clicked. If that doesn't pan out, I should be able to hook into the IndexOperationComplete examine event to trigger the refresher, which would avoid the need for a custom dashboard. Would be a little slower to propagate, but would work. I would also likely need a check to see whether the server is the master server in this case to prevent the request from looping on the master server.

[Shazwazza]

Yep you could try something like that but beware of the IndexOperationComplete event, this fires anytime a batch operation is completed so that event isn't really going to work for you unless you know the result of that event is from a rebuild in the back office. Also that event may fire up to 3 times for a rebuild in the back office because there are 3x index batches run for each populator. Even when saving one document, that one document is considered a batch so this event will fire. Let us know what you come up with as I'd be interested in that. But also if you want to help try to fix this directly in the core, even as a basic start using cache refreshers we'd be interested in a PR too :)

[jaandrews]

Yeah, I figured there would be some gotchas tying into that event. Will let you know what I come up with and will try to pull together a PR if feasible. Probably won't get to this for a couple weeks at the earliest though.

[umbrabot]

Hi @jaandrews,

We're writing to let you know that we've added the Up For Grabs label to your issue. We feel that this issue is ideal to flag for a community member to work on it. Once flagged here, folk looking for issues to work on will know to look at yours. Of course, please feel free work on this yourself ;-). If there are any changes to this status, we'll be sure to let you know.

For more information about issues and states, have a look at this blog post

Thanks muchly, from your friendly PR team bot :-)

[umbrabot]

Hiya @jaandrews,

Just wanted to let you know that we noticed that this issue got a bit stale and might not be relevant any more.

We will close this issue for now but we're happy to open it up again if you think it's still relevant (for example: it's a feature request that's not yet implemented, or it's a bug that's not yet been fixed).

To open it this issue up again, you can write @umbrabot still relevant in a new comment as the first line. It would be super helpful for us if on the next line you could let us know why you think it's still relevant.

For example:

@umbrabot still relevant
This bug can still be reproduced in version x.y.z

This will reopen the issue in the next few hours.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[rbottema]

@umbrabot still relevant
This is still relevant for set ups where the CMS site is different from the front-facing sites, and the front-facing sites have access to Umbraco blocked.

[umbrabot]

Hiya @jaandrews,

Just wanted to let you know that we noticed that this issue got a bit stale and might not be relevant any more.

We will close this issue for now but we're happy to open it up again if you think it's still relevant (for example: it's a feature request that's not yet implemented, or it's a bug that's not yet been fixed).

To open it this issue up again, you can write @umbrabot still relevant in a new comment as the first line. It would be super helpful for us if on the next line you could let us know why you think it's still relevant.

For example:

@umbrabot still relevant
This bug can still be reproduced in version x.y.z

This will reopen the issue in the next few hours.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂