SaveEventually and sudden termination - silent failures?

[geraldhinson]

My app uses the following model to save objects.

saveAnObject (PFObject *obj)
{
    // save locally first
    //
    [obj pinInBackgroundWithBlock:^(BOOL success, NSError *error)
     {
         if (success == true)
         {
             // local save succeeded - now save to cloud
             [obj saveEventually];
         }
         else
         {
             // local save failed
             //
             .... do error processing ....
         }
     }];
}

However, after reading this http://stackoverflow.com/questions/32617917/parse-saveeventually-failing-when-app-is-terminated-quickly and then reading the parse ios/osx code on github it appears that saveEventually returns after allocating a task and (important) prior to writing anything to disk. In other words it seems quite possible that a saveEventually could fail silently if an app is killed prior to the task allocated having a chance to persist the "save to cloud" request.

True?

If so, this probably explains why I occasionally see objects in my local store that are never persisted in the cloud. From the documentation I concluded that saveEventually was persisting / logging the save request to disk prior to returning control to the application.... ie. classic write-ahead logging to ensure the data was not lost in the event of untimely app termination.

Assuming the above is true, is there any way for me to query the local datastore to determine if there is data that has not yet been persisted to the cloud (ie. to detect saveEventually failures?).

[richardjrossiii]

Yes, that is true. All IO in the SDK is non-blocking of the calling thread, which is very critical for any user-facing application, as the last thing you want is your UI stuttering while you are attempting to write to disk.

(In fact, even our synchronous APIs don't do IO on the calling thread - the calling thread just does a semaphore wait until the task is completed, while letting the work still be done in the background).

Most applications shouldn't ever run into 'sudden termination' (e.g. crashes), and if you're having these consistently I'd strongly recommend solving the source of these crashes rather than try and patch them up at the data model level...

In terms of having some way of knowing when the objects are persisted locally, this may be a use-case for issue #319 to be implemented, though I'm still not a huge fan of emitting NSNotifications...

If you really must detect these edge cases, I suppose it would be possible to query all local objects with a nil objectId on launch, and emit additional saveEventually calls for them...

That would, of course, require having a fixed set of class names to search for, but I'd wager that most applications would have such a fixed list.

[nlutsenko]

In the case of sudden termination - saveEventually here is going to behave the same way as pinning to Local Datastore, meaning if app was killed before the object was pinned (before you got to the callback) - it's not going to be persisted locally.

What saveEventually does is basically wraps the logic around pinning the object, saving it to the server, unpinning the object. As well as it includes code for executing these saves across app executions.

You can achieve extra guarantee here by wrapping saveEventually calls into pinning the object once again to a pin with specific name (e.g. unsavedObjects) and before calling into saveEventually - pin objects to that pin, after saveEventually succeeds - unpin objects from that pin.

[geraldhinson]

Interesting comments above (and thanks for reformatting that code block for me).

To be clear, the app is not suffering from sudden crashes that need fixing. That said, no app (my code) nor library (your code) is perfect and there needs to be a model for knowing with certainty that a new or updated object is truly saved at the app level and a model for recovering from errors when this is not the case.

I suspect (but have not proven) that what is occurring is an app termination on IOS because either:

1. the app has been moved to the background and IOS nukes it to reclaim the memory for some other app. Apple documents very clearly that all apps / libraries MUST handle this scenario.
   OR
2. the user is killing the app explicitly using the IOS interface (the good 'ol double punch, swipe up)

If I understand your codebase correctly, either of these occurrences prior to the parse library actually writing to disk the data required for saveEventually semantics will cause a loss of data. (And this is what the stackoverflow link (above) documented occurring as well).

I'll check the device in question to see if I have objects that are missing your server-assigned object id as you suggest. If this is the case, then I can recover from the issue quite easily, but I don't believe that I should be dealing with this issue at the app level.

Also, to further clarify the semantics a bit:

• does the callback on pinInBackgroundWithBlock not get called until after the data has been persisted in the local datastore? This is my understanding.
• does the callback on saveEventually (the callback version of this api vs the one I'm currently using) not get called until after the data has been persisted to disk for sending to the cloud later? Or, does the callback happen after the task is allocated?

Given the current design (api returns after a task is allocated) it seems that an app can never assume that data is actually saved when using saveEventually (non-callback version at least)? The semantics are "kick off a queue-data-for-the-cloud-task and pray it completes", right?

I'll update this thread after I've had a chance to examine the local datastore objects that did not make it to the cloud to see if indeed they are missing their object ids still. It seems the next step here depends upon that answer.

[geraldhinson]

OK, just retrieved the device that has 3 object in its local datastore that were not successfully queued to the cloud. As suspected, all 3 objects have no objectId assigned and as stated earlier there are no crash logs for the app. The person using the app on the device in question also reported that the app seemed to working fine with no visible crashes during use.

Thoughts?

[geraldhinson]

Richard

Per your recommendation above:

"If you really must detect these edge cases, I suppose it would be possible to query all local objects with a nil objectId on launch, and emit additional saveEventually calls for them..."

I wrote the following:

void
_ParseCloud::Sync_RedoSaveEventually (char* pClass)
{
    NSString* nsClassName = [NSString stringWithUTF8String:(const char *) pClass];

    printf ("\nSync_RedoSaveEventually: Syncing <%s> db\n", pClass);

    PFQuery *query = [PFQuery queryWithClassName: nsClassName];
    [query fromLocalDatastore];
    [query whereKeyDoesNotExist:@"objectId" ];
    query.limit = 1000;

    [query findObjectsInBackgroundWithBlock:^(NSArray *objects, NSError *error)
     {
         if (!error)
         {
             if (objects.count == 0)
             {
                 return;
             }

             NSLog(@"Successfully retrieved %lu rows without objectId assigned.", (unsigned long)objects.count);

             // saveEventually failed to do its thing... try again
             for (PFObject *object in objects)
             {
                 Sync_ForceSyncByOtherDevices (object);  <--- this just changes one of the values
                 saveAnObject (object); <-- this is shown in original question
             }
         }
         else
         {
             // Log details of the failure
             NSLog(@"Error: %@ %@", error, [error userInfo]);
             sprintf (m_LoggedError, "%s", error.localizedDescription.UTF8String);
         }

     }];
}

This successfully finds the rows that are in the local datastore but not in the cloud, BUT calling saveEventually on them again results in a row in the cloud table / class with most values undefined. (ie. the 2nd saveEventually did not propagate the values from the local datastore...)

The results can be seen here: https://dl.dropboxusercontent.com/u/9116971/ParseCloudWithUndefinedFields.png

So, the "find all objects missing an objectId and issue saveEventually() on them again" doesn't seem to work as expected. What would you recommend?

[geraldhinson]

Oh, forgot to mention, I also tried simply issuing a saveEventually call in the for *object in objects loop above. This resulted in no change locally and no objects saved to the cloud either.

So, I modified it to change one of the fields and then issued my usual pin followed by saveEventually. This caused the empty objects to appear in the cloud but did NOT result in an objectId being assigned to the objects in the local datastore... they show up with objectId missing even after the logic above is run and the empty objects are saved to the cloud.

[geraldhinson]

Richard, any further advice on how to get my objects saved to the cloud since the additional saveEventually call on the ones in the local datastore without objectIds assigned didn't work?

[geraldhinson]

Update on this after upgrading from Parse v1.8.3 to v1.8.5 today.

After re-building with 1.8.5 I re-ran the logic that searched for objects in the local datastore missing their objectIds then reissued saveEventually calls on them as such:

'''
void
_ParseCloud::Sync_RedoSaveEventually (char* pClass)
{
NSString* nsClassName = [NSString stringWithUTF8String:(const char *) pClass];

printf ("\nSync_RedoSaveEventually: Syncing <%s> db\n", pClass);

PFQuery *query = [PFQuery queryWithClassName: nsClassName];
[query fromLocalDatastore];
[query whereKeyDoesNotExist:@"objectId" ];
query.limit = 1000;

[query findObjectsInBackgroundWithBlock:^(NSArray *objects, NSError *error)
 {
     if (!error)
     {
         if (objects.count == 0)
         {
             return;
         }

         NSLog(@"Successfully retrieved %lu rows without objectId assigned.", (unsigned long)objects.count);

         // saveEventually failed to do its thing... try again
         for (PFObject *object in objects)
         {
             [object saveEventually];
         }
     }
     else
     {
         // Log details of the failure
         NSLog(@"Error: %@ %@", error, [error userInfo]);
         sprintf (m_LoggedError, "%s", error.localizedDescription.UTF8String);
     }

 }];

}
'''

This behaved a bit differently from 1.8.3 in that the local datastore objects returned WERE assigned an object id, so subsequence runs of this logic no longer found them. However, the objects stored in the cloud were still as before with almost all of their fields undefined.

So, there are 2 issues: 1) the original call to saveEventually did not do the save. 2) subsequent calls to saveEventually result in incorrect objects being stored in the cloud.

If I can deal with #1 as it is pretty easy to detect via the missing objectId, but I can't go live with this app unless I can get past #2 above.

Please advise.

[geraldhinson]

Richard, et al:

As far as the objects getting saved with undefined fields part of my question here, could it be related to this issue: #131

It looks pretty similar to me.

[richardjrossiii]

@geraldhinson It's most likely not related, but if your object graph is cyclic (e.g. A -> B -> A), pinning it can sometimes lose information, especially if the entire object graph has yet to be fetched into memory yet.

Due to the LDS architecture, for performance reasons we cannot fetch the entire object graph & subgraph before re-pinning it, as we could very quickly run out of memory for any complex object graph, and so we limit our search recursion to a depth of 1.

If you can simplify your object model it may solve this problem for you.

[geraldhinson]

I don't use pointers at all - so no cycles. Simple "singleton" objects across the board in my app.

[geraldhinson]

I've uploaded a bug repro project here: https://dl.dropboxusercontent.com/u/9116971/ParseReproSaveEventuallyBug.zip

This illustrates two problems:

1. objects can be lost after saveEventually returns and either IOS terminates an app in the background to reclaim its memory (not an app bug) OR the user terminates an app using the IOS task manager (also not a bug)

2. reissuing a saveEventually on the objects that were successfully pinned in the local datastore, but were never saved to the cloud (hence missing an objectId) does not result in them being saved to the cloud

My app has objects in the local datastore that were not saved to the cloud because of this issue (which I consider a "bug" not a "discussion" as this issue is currently labeled for some reason). I need to hear from you parse-ians how to recover this data because other devices that are sync'ing to the cloud need it as well for app correctness.

Thanks!