Add documentation for using health indicators without access to the default database.

[spring-projects-issues]

Sledge Hammer opened DATAGRAPH-1458 and commented

Based on your sample here for multiple databases with different usernames and security:

https://github.com/michael-simons/neo4j-examples-and-tips/tree/master/examples/sdn6-multidb-multi-connections/src/main/java/org/neo4j/tips/sdn/sdn6multidbmulticonnections

All is working in the app :) and I am able to connect to both DBs. However, I recently tested with Spring Boot Admin which calls the actuator health checks in my app and discovered that the health check doesn't seem to be consuming the correct beans / credentials.

I'm marking all the movie beans as primary, but oddly, the fitness driver is showing up, but the movies driver is showing down.

neo4jDOWN
fitnessDriverUP
server Neo4j/4.0.0@localhost:7687
edition enterprise
database fitness
moviesDriverDOWN
error org.neo4j.driver.exceptions.ClientException: Database access is not allowed for user 'movies' with roles [movies_reader].

The full stack trace is:

org.neo4j.driver.exceptions.ClientException: Database access is not allowed for user 'movies' with roles [movies_reader].
at org.neo4j.driver.internal.util.ErrorUtil.newNeo4jError(ErrorUtil.java:80)
at org.neo4j.driver.internal.async.inbound.InboundMessageDispatcher.handleFailureMessage(InboundMessageDispatcher.java:105)
at org.neo4j.driver.internal.messaging.v1.MessageReaderV1.unpackFailureMessage(MessageReaderV1.java:83)
at org.neo4j.driver.internal.messaging.v1.MessageReaderV1.read(MessageReaderV1.java:59)
at org.neo4j.driver.internal.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:83)
at org.neo4j.driver.internal.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:35)
at org.neo4j.driver.internal.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at org.neo4j.driver.internal.async.inbound.MessageDecoder.channelRead(MessageDecoder.java:47)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)

---

Affects: 6.0.1 (2020.0.1)

[spring-projects-issues]

Gerrit Meier commented

Thanks for bringing this to our attention.
The health check itself is only aware of the driver configuration. The database configuration is part of the Spring Data world.
This is also the reason I am wondering how you got the health endpoint saying database fitness? While trying to reproduce your problem to find a possible solution I could only see the default database neo4j for both connections. Of course unless I removed the rights and exceptions were thrown.
Could you provide an example for me?

[spring-projects-issues]

Sledge Hammer commented

Gerrit Meier

I copied the sample you made above with some slight clean up. This is my movies config. The fitness config is exactly the same minus the bean names, etc. and of course no primary. So I would have thought the movies driver would be up and the fitness driver down if anything :).

I'll try step through the health check and see if I see anything. It would difficult to rip this out into a standalone demo with all the entities and stuff.

@Configuration
@EnableNeo4jRepositories(basePackages="org.xxx.xxx.movies.repositories",
transactionManagerRef="moviesTransactionManager",
neo4jMappingContextRef="moviesMappingContext",
neo4jTemplateRef="moviesTemplate")
public class MoviesDataSourceConfig {

private final Environment environment;

@Autowired
public MoviesDataSourceConfig(Environment environment) {
	this.environment = environment;
}

@Bean
@Primary
public Driver moviesDriver() {
	return GraphDatabase.driver(Preconditions.checkNotNull(this.environment.getProperty("org.neo4j.driver.movies.uri")),
								AuthTokens.basic(Preconditions.checkNotNull(this.environment.getProperty("org.neo4j.driver.movies.authentication.username")),
												 Preconditions.checkNotNull(this.environment.getProperty("org.neo4j.driver.movies.authentication.password"))));
}

@Bean
@Primary
public Neo4jClient moviesClient(@Qualifier("moviesDriver") Driver driver) {
	return Neo4jClient.create(driver);
}

@Bean
@Primary
public PlatformTransactionManager moviesTransactionManager(@Qualifier("moviesDriver") Driver driver,
														   @Qualifier("moviesDatabaseSelectionProvider") DatabaseSelectionProvider databaseSelectionProvider) {
	return new Neo4jTransactionManager(driver, databaseSelectionProvider);
}

@Bean
@Primary
public Neo4jMappingContext moviesMappingContext(ApplicationContext applicationContext,
											    Neo4jConversions neo4jConversions) throws ClassNotFoundException {
	Set<Class<?>> initialEntityClasses = new EntityScanner(applicationContext).scan(Node.class);
	Neo4jMappingContext context = new Neo4jMappingContext(neo4jConversions);
	context.setInitialEntitySet(initialEntityClasses);
	return context;
}

@Bean
@Primary
public Neo4jOperations moviesTemplate(@Qualifier("moviesClient") Neo4jClient neo4jClient,
									  @Qualifier("moviesMappingContext") Neo4jMappingContext mappingContext,
									  @Qualifier("moviesDatabaseSelectionProvider") DatabaseSelectionProvider databaseSelectionProvider) {
	return new Neo4jTemplate(neo4jClient, mappingContext, databaseSelectionProvider);
}

@Bean
public DatabaseSelectionProvider moviesDatabaseSelectionProvider() {
	return DatabaseSelectionProvider.createStaticDatabaseSelectionProvider("movies");
}

}

[spring-projects-issues]

Sledge Hammer commented

Some preliminary findings:

The Neo4jReactiveHealthIndicator constructor does in fact get called twice as expected.

2020-12-07 15:26:32.545 INFO 55000 --- [ main] Driver : Direct driver instance 1419816353 created for server address localhost:7687
2020-12-07 15:26:32.703 INFO 55000 --- [ main] Driver : Direct driver instance 1370631942 created for server address localhost:7687

First call, I get passed driver 54a0ada1 which is the 14198 one.

Second call, I get passed driver 51b22f06 which is the 13706 one.

Then I get called repeatedly where only 54a0ada1 gets passed in. Kind of hard to follow what's what... but it seems like you should account for all driver beans rather then just the one. I don't seem to be getting any generic driver or auto-configured driver since I'm not using the standard properties, but I don't seem to be able to see anything like database, etc. in the debugger

[spring-projects-issues]

Sledge Hammer commented

Aside from it appearing that only one driver been seems to be updating, isn't this an issue too?

Mono<Tuple2<String, ResultSummary>> runHealthCheckQuery() {
	// We use WRITE here to make sure UP is returned for a server that supports
	// all possible workloads
	return Mono.using(() -> this.driver.rxSession(Neo4jHealthIndicator.DEFAULT_SESSION_CONFIG), (session) -> {
		RxResult result = session.run(Neo4jHealthIndicator.CYPHER);
		return Mono.from(result.records()).map((record) -> record.get("edition").asString())
				.zipWhen((edition) -> Mono.from(result.consume()));
	}, RxSession::close);
}

In my code to get the path stuff working, I did something like this:

	QueryRunner queryRunner = Neo4jTransactionManager.retrieveTransaction(this.driver, targetDatabase);

	if (queryRunner == null)
		queryRunner = this.driver.session(Neo4jTransactionUtils.defaultSessionConfig(targetDatabase));

Copied from AutoCloseableQueryRunner. I had to get the session for the database rather then just the default one

[michael-simons]

Hello Sledge Hammer. I cannot find your handle on GitHub, so I just hope you might see this issue.

This is not something we can fix in Spring Data Neo4j. I updated my example you linked in the issue with one possible solution.
The solution is also the one I am gonna document in SDN itself.

Feel free to open a ticket at Boot to about that topic. I'm super happy to help the colleagues over there to create something similar built in.

The driver itself does not know anything about a database or not, just about the server or cluster.