Skip to content

Datasource initialisation with JPA schema configured in deferred mode happens asynchronously #22852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wikisamuel opened this issue Aug 10, 2020 · 11 comments
Assignees
Labels
type: bug A general bug
Milestone

Comments

@wikisamuel
Copy link

When upgrading from Spring Boot 2.2.9 to 2.3.0, unit tests cannot handle large datasets anymore.

When logging with a debug level, we see extShutdownHook drops tables before the end of the insertions, hence the test fails.

Here is a minimal reproducible example : it works with 2.2.9, but when upgrading to 2.3.0 we can see 10k rows seems too much.

The problem still exist in the last version (2.3.2).

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Aug 10, 2020
@snicoll
Copy link
Member

snicoll commented Aug 10, 2020

Thanks for the report and the sample. Ignoring that the project does not compile because it uses JUnit4 and the vintage engine has been excluded, I can run the test and it passes when I removed the exclusion.

Given that your example works for me, you'll have to share more details about your environment (OS, JDK). Attaching the logs of the execution (either as a file or on gist.github.com) would be helpful as well.

@snicoll snicoll added the status: waiting-for-feedback We need additional information before we can continue label Aug 10, 2020
@wikisamuel
Copy link
Author

Thanks for the reply. Maybe your computer is just faster than mine and you should try with a larger dataset. I've uploaded a new version with 20k rows so you can test.

Here are the logs I've just had.

I'm using Windows 10 Enterprise 1909 18363.900 and JDK 11.0.7+10.

@spring-projects-issues spring-projects-issues added status: feedback-provided Feedback has been provided and removed status: waiting-for-feedback We need additional information before we can continue labels Aug 10, 2020
@snicoll
Copy link
Member

snicoll commented Aug 10, 2020

That is interesting, thanks. Spring Boot 2.3 has switched the JPA bootstrap mode to deferred and it might be causing an issue for you. To confirm this and provide you a temporary workaround if it does, can you please add spring.data.jpa.repositories.bootstrap-mode=default to your configuration to see if that problem goes away?

@snicoll snicoll added status: waiting-for-feedback We need additional information before we can continue and removed status: feedback-provided Feedback has been provided labels Aug 10, 2020
@rohitrajt
Copy link

rohitrajt commented Aug 11, 2020

I have the same problem. In my case, the unit tests works fine in 2.3.0. We have been running with 2.3.0 for quite some time now without any issues. Today, when i upgrade to 2.3.2 i see the same error as mentioned in this issue. Also I tried the solution provided by @snicoll and it still has the same issue.

I also confirmed that my unit test context starts with DEFAULT bootstrap mode, [ Test worker] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.

@spring-projects-issues spring-projects-issues added status: feedback-provided Feedback has been provided and removed status: waiting-for-feedback We need additional information before we can continue labels Aug 11, 2020
@philwebb philwebb changed the title unit tests cannot handle large datasets anymore Unit tests cannot handle large datasets anymore Aug 11, 2020
@philwebb philwebb added type: bug A general bug and removed status: feedback-provided Feedback has been provided status: waiting-for-triage An issue we've not yet triaged labels Aug 11, 2020
@philwebb philwebb added this to the 2.3.x milestone Aug 11, 2020
@wikisamuel
Copy link
Author

The workaround provided by @snicoll has worked for me! Thanks!

@snicoll
Copy link
Member

snicoll commented Aug 11, 2020

@rohitrajt I am not sure that you have the same problem, especially given that @wikisamuel has confirmed the workaround work for them. To figure it out and help you, please open a separate issue with a small sample that works with 2.3.0 and breaks with 2.3.2 (a zip or a link to a GitHub repo). Thanks!

@snicoll snicoll changed the title Unit tests cannot handle large datasets anymore Datasource initialisation with JPA schema creating in deferred mode happens asynchronously Aug 11, 2020
@snicoll
Copy link
Member

snicoll commented Aug 11, 2020

So that's our basic datasource init friend again.

@Override
public void postProcessEntityManagerFactory(EntityManagerFactory entityManagerFactory) {
this.delegate.postProcessEntityManagerFactory(entityManagerFactory);
AsyncTaskExecutor bootstrapExecutor = this.factoryBean.getBootstrapExecutor();
if (bootstrapExecutor != null) {
bootstrapExecutor.execute(() -> DataSourceInitializedPublisher.this
.publishEventIfRequired(this.factoryBean, entityManagerFactory));
}
) when JPA is in place is done using the right callback but that doesn't seem to work.

I am not entirely sure if that's a framework limitation or our (rather convoluted) arrangement.

@snicoll snicoll changed the title Datasource initialisation with JPA schema creating in deferred mode happens asynchronously Datasource initialisation with JPA schema configured in deferred mode happens asynchronously Aug 11, 2020
@snicoll
Copy link
Member

snicoll commented Aug 11, 2020

According to @jhoeller we shouldn't invoke the bootstrapExecutor ourselves. If we fire the event directly, the execution will be part of the lock so that might fix that issue. I need to figure out why the bootstrapExecutor was used manually first.

@snicoll
Copy link
Member

snicoll commented Aug 12, 2020

Relates to #14061

@snicoll snicoll added the for: team-attention An issue we'd like other members of the team to review label Aug 31, 2020
@philwebb philwebb removed the for: team-attention An issue we'd like other members of the team to review label Sep 16, 2020
@snicoll snicoll self-assigned this Sep 22, 2020
@snicoll snicoll modified the milestones: 2.3.x, 2.3.5 Sep 22, 2020
@snicoll snicoll added this to the 2.3.x milestone Sep 22, 2020
@snicoll snicoll added the status: blocked An issue that's blocked on an external project change label Sep 22, 2020
@snicoll
Copy link
Member

snicoll commented Sep 22, 2020

Unfortunately, we can't switch to this as I thought as we send an event in that EMF lock and that leads to a deadlock in framework .

@snicoll
Copy link
Member

snicoll commented Sep 22, 2020

I've created spring-projects/spring-framework#25799

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A general bug
Projects
None yet
Development

No branches or pull requests

6 participants