All calls to save() or fetch() on ParseUser and ParseInstallation will freeze if previous call to saveEventually() has failed

[cjosepha]

After ParseUser.saveEventually() or ParseInstallation.saveEventually() has failed, the sdk seems to be in a corrupted state as all subsequent calls to save() or fetch() on ParseUser and ParseInstallation will freeze. Restarting the app or the device will not solve the issue and only a re-installation of the app allows to save/fetch ParseUser/ParseInstallation again.
This issue occurs with any version of parse-server and any version of Parse-SDK-Android.

Steps to reproduce the issue on ParseUser (same for ParseInstallation)

1. Log in
2. Delete user session on the server
3. Put new data to user
4. Call user.saveEventually() -> Returns invalid session token error (expected)
5. Close app and relaunch it
6. Put new data to user
7. Call user.save() -> Freeze forever

Note that terminating the app then relaunching it will not change anything : any call to user.save() will freeze (as user.saveInBackground() never completes).

Parse initialization code used

Parse.setLogLevel(Parse.LOG_LEVEL_VERBOSE);
Parse.initialize(new Parse.Configuration.Builder(application)
.applicationId(BuildConfig.PARSE_APP_ID)
.clientKey(BuildConfig.PARSE_CLIENT_KEY)
.enableLocalDataStore()
.server(BuildConfig.SERVER_URL + ":" + BuildConfig.SERVER_PORT + "/parse/")
.build());

Standalone app showing the issue

https://github.com/DanGTZ/Parse-Android-SDK-Test

The tests testSaveEventuallyInstallation and testSaveEventuallyUser passe the first time they are launched, then subsequent launches will freeze on the first save()

https://github.com/DanGTZ/Parse-Android-SDK-Test/blob/master/app/src/androidTest/java/com/perfexpert/parsetest/ParseSaveEventuallyTest.java

Workaround

This issue occurs after loading invalid data from the cache at step 5. To prevent loading invalid data at startup, clear the ParseUser or ParseInstallation disk cache just after the saveEventually() has failed (step 4). To do so, you can use this gist : https://gist.github.com/DanGTZ/7c95d6fc0bb95d3d7433ab7dc309eeac

[cjosepha]

Currently I have users of my app that can't use it because their installation is in this state : any save or fetch in background never returns.
I'm looking for a solution to solve this with an app update, without the need to reinstall the app, in order to prevent data loss.

[flovilmart]

@dangtz this looks quite serious. Are you able to pinpoint the exact reason why this behavior is exhibited?

[cjosepha]

@flovilmart it seems that there is a deadlock in the code that manage operations queue for persisted singletons like ParseInstallation and ParseUser, because I'm not able to reproduce this issue with a basic ParseObject, even if I pin the object in LDS.
May be some operations queue resources are not released properly when the saveEventually() fails, but my poor knowledge of the sdk code doesn't allows me to tell you more.

[flovilmart]

@dangtz because you're able ton consistently reproduce it, can you perhaps open a failing PR, that exhibit the deadlock? Perhaps plucking a debugger and checking the stack / threads once dead locked can help pinpoint the issue.

[cjosepha]

@flovilmart by "failing PR" you mean creating new unit tests in the sdk?
Also to reproduce the issue, one step is to close the current session (call to a custom cloud function "closeAllSession" that I created for my test) : do I need to mock the server to do this in the sdk?

[flovilmart]

by "failing PR" you mean creating new unit tests in the sdk?

Yep, I mean a new test that is currently failing without a fix.

do I need to mock the server to do this in the sdk?

Most probably, adding a mock would be the way to go as we never run edge to edge tests with a live server for this SDK IIUC.

[cjosepha]

@flovilmart I'm not able to reproduce this issue with unit tests #831
After more integration testing in my own app, it seems that the freeze occurs only after relaunching the app after the saveEventually() fails.

[flovilmart]

Ok, so that means that the full app is in an invalid state which is quite bad as we have a persistent issue.

[cjosepha]

Yes, and now I have to take my time to prepare a publication for my users to explain them the only solution to the problem is to reinstall the app and then lost all their local data (pinned in localdatastore) :-(

[flovilmart]

Nah, we can do better I believe no? If we find the soUrce of the deadlock, we can ensure that at next restart everything is back on track

[cjosepha]

That's what I was trying to do.

The exact async method that doesn't return is this one :
https://github.com/parse-community/Parse-SDK-Android/blob/master/Parse/src/main/java/com/parse/ParseObject.java#L1604

[cjosepha]

@flovilmart here is a standalone app showing the issue :

https://github.com/DanGTZ/Parse-Android-SDK-Test

The tests testSaveEventuallyInstallation and testSaveEventuallyUser passe the first time they are launched, then subsequent launches will freeze on the first save()

https://github.com/DanGTZ/Parse-Android-SDK-Test/blob/master/app/src/androidTest/java/com/perfexpert/parsetest/ParseSaveEventuallyTest.java

Expects a local parse-server with application id "parse_app_id" and client key "parse_client_key".

[flovilmart]

@Jawnnypoo and @rogerhu , is that something you guys you could have a look at?

[cjosepha]

Here is the cloud code that is needed for this test :

Parse.Cloud.define("closeAllSessions", function(request, response) {
  if (!request.user) {
    response.error("A user must be logged in");
  } else {
    var sessionQuery = new Parse.Query(Session);
    sessionQuery.equalTo('user', request.user);
    sessionQuery.find({
      useMasterKey: true
    }).then(function(sessions) {
      console.log(sessions.length + " sessions will be closed");
      return Parse.Object.destroyAll(sessions, {
        useMasterKey: true
      });
    }).then(function() {
      response.success();
    }, function(error) {
      response.error(error.message);
    });
  }
});

[rogerhu]

Thanks for the repro -- I see the same thing. Note I had to change your code to be new Parse.Query(Parse.Session); to get things to work.

[cjosepha]

@rogerhu sorry forgot to add this line before the function : var Session = Parse.Object.extend('_Session');

[cjosepha]

More info, once the saveEventually has failed :

• The problem occurs only after relaunching the app

• The problem doesn't occur if there is a successful saving of the object (User or Installation) before relaunching the app.

• The problem doesn't occur if LDS is not enabled

I guess something wrong is persisted in LDS when saveEventually fails. Then after relaunch, the wrong data is loaded from LDS and make any subsequent save deadlocking. If the app is not relaunched, user logs in and save the object, the wrong data is removed from LDS, due to successful saving, and the problem never occurs. I think this is the most likely hypothesis.

[cjosepha]

This issue occurs after loading invalid data from the cache at step 5. To prevent loading invalid data at startup, clear the ParseUser or ParseInstallation disk cache just after the saveEventually() has failed (step 4). To do so, you can use this gist : https://gist.github.com/DanGTZ/7c95d6fc0bb95d3d7433ab7dc309eeac

I've also updated my test app so you can see the impact of clearing cache : https://github.com/DanGTZ/Parse-Android-SDK-Test

[Jawnnypoo]

Fixed with #831