LDS Performance Improvements

[richardjrossiii]

This is a master issue for all things related to local datastore performance improvements.

• Make LDS stop checkpointing and making copies of objects for mutable containers (Removed 'mutable containers' functionality. #102)
• Make LDS query matching more efficient by not loading every object into memory for a specified pin.
• Make LDS use PFObjectState and PFUserState instead of PFObject and PFUser, to reduce the amount of locking required for every LDS operation.
• Improve LDS PFRelation tracking.
• Update LDS database schema to allow for query matching in the database layer itself.

For anyone reaching this issue as a result of currently bad LDS performance here's our current recommendations:

• Keep your object schemas as small as possible - the faster it is to parse them from JSON, the better.
• Relations are currently inefficient. If you can avoid them, do so.
• Minimize usage of any values of the following classes:
  • NSArray
  • NSDictionary
  • PFACL
  • PFGeoPoint
• Whenever possible, use smaller pins. Because LDS loads all objects in a single pin before evaluating a query, the smaller the pin is, the better.

[rawrmaan]

+1, LDS performance needs lots of work. I'm glad you're finally publicly acknowledging this. See my angry bug report from April 2015: https://developers.facebook.com/bugs/1575754175998187

[FreudGit]

+1: 2.3 sec to get one row on a table of 800 objects... Ouch.

[RickDT]

👍

[FreudGit]

Hi

Sorry to came back. Do you think one of these point can be associate to a milestone?

Maybe not clean, but a PFQuery with a whereKey pointing to a String can be filtered direct from the DB (json), without unwrap.

Just with this, Query time will be 99% smaller.

Just my 2 cents

[nlutsenko]

Hey @FreudGit, absolutely!
I've just attached a PR to this that implements the first bullet point in that list.
From our testing - it should improve performance significantly.

But you are right - we should attach more to milestones. Making it happen...

[FreudGit]

WOW, thanks for the fast reply.

PR? This was a bit to technical for me. But i'm happy if we can target and see better performance in the next release\weeks. 👍

[nlutsenko]

PR is a shorthand for Pull Request

Since removing mutable containers functionality can break few flows for people, we are sitting on it for a little bit longer than next release. I think we are going to ship it with 1.8.4

[FreudGit]

Thanks again @nlutsenko.
Wish 1.8.4 will come soon or that you will be ok to move to 1.8.3 milestone. Let me know if\when there is a way to try.

(completely off-topic)= how do you manage the multiple version(android, etc). Do you sync revision with other team(Android, .net, etc? Just curious!

[nlutsenko]

@FreudGit, 1.8.4 is coming very very soon after 1.8.4. A small tip is to look at the release schedule of 3 previous releases, it might get you an idea of how we try to release at the current moment.

So far we don't sync versions between Android/.NET/iOS(OSX), since we have separate dependencies.
If we are releasing a new feature that is cross-platform, say new Sessions APIs that we released at F8 - we do the release at the same time. Most of the time though - we have separate release schedules.

[FreudGit]

Thanks(still curious). Are the dev of each language share code\need to have the more similar structure as possible?

[nlutsenko]

Not sure about the last question...
We don't share any code currently, but the APIs/class structure and architecture are unified and share a lot of the same principles and patterns, though adapted to best platform practices.

[FreudGit]

Sorry for my bad english. And have a nice day.

Just curious if, for example, Android LDS have the same performance problem because very similair coding approach.

Dont want to bug anymore, no need to reply and no more question!

[nlutsenko]

Got you. I think the answer is yes - we share the same architecture and approaches for LDS, so say removing mutable containers on Android will also improve performance quite a lot!
No problem, please bug us, and I am always happy to help! 😁

[FreudGit]

@nlutsenko Are 1.8.4 still a target for this ? :)

[nlutsenko]

I wish my answer was yes.
We are aiming for 1.8.4 for early/mid next week, so looks like 1.8.5 is the target for this.

[Mohd-Aamir-M]

1.8.5 does contain good performance on LDS pinning and fetching?

And, LDS performance issue persists on other parse SDKs like Android, OSX etc.

[oskarl]

Any update? Does anyone have tips for workarounds?

[Mohd-Aamir-M]

I am glad to know that iOS sdk 1.9.0 is finally released so will it solve LDS issues?

[nlutsenko]

There are improvements for LDS, yes.
Solve - possibly, but #102 is marked for 1.10.0 - which will have a much bigger impact on the performance compared to 1.9.0.

[flovilmart]

Yeah, perhaps with and architectural proof of concept. Optimizing the storage for queries is where I believe we should go ie: creating additional tables with optimal indexes and then querying the table that holds the data by objectId. What do you think?

[eduardbosch]

This will speed up querying in LDS a lot. Could be a good start.

---

Anyway, I'm planning to move all my LDS to realm, so I can make queries directly to any data properties. Also, this will speed up a lot saving the data to LDS.

I was thinking about 3 options:

1. Reuse classes that extend from ParseObject as RealmObject, so data could be serialized directly to realm and users don't have to convert their classes to realm DB. But I don't think that this is possible.
2. Do it the opposite way, reuse the classes that extend from RealmObject so ParseQuery can serialize to this kind of objects.
3. Create classes that extend directly from RealmObject, then query ParseObjects and do the conversion. When the conversion is done, save any extra fields existing in the ParseObjectnot defined in the RealmObject, as 3 arrays. An array of string keys, an array of string values and a third array that defines the types to allow deserializing. This arrays will allow saving any new fields.

Then, I was planning to use realm DB directly to make my queries.

To avoid dealing with realm DB, another improvement could be to translate the Parse queries to realm queries.

Those are just some thoughts about this topic.

What do you think?

Is this a good LDS option? Could be a viable solution?

I think the best solution is to have an automatic SQLite / CoreData like you are planning, so users will need only the Parse SDK. But to improve the LDS a lot more, I think that we can write an LDS abstraction that can be saved to some different kinds of DBs like realm or other.

[ajonescodes]

Are there any good ideas / success stories from others going from LDS to Realm? Core Data? Also didn't know LDS had been deprecated.

[eduardbosch]

@flovilmart @dplewis Any news on start a LDS alternative?

Thanks

[flovilmart]

Not yet

…

On Sep 5, 2017, 05:49 -0400, Eduard Bosch Bertran ***@***.***>, wrote: @flovilmart @dplewis Any news on start a LDS alternative? Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[eduardbosch]

Hi to all again. I'm starting using the iOS lib and I'll have to implement an LDS.

Has anyone used any alternative to Parse LDS to combine the Parse SDK with Realm or another local database? I think I'll go with Realm as it allows working with lots of entries very efficiently.

I think that this is the biggest issue has Parse right now. With a good LDS, it'll be great.

[TomWFox]

@eduardbosch what solution did you come to? I wonder whether offering a bridge between Parse and say Realm could be useful. Although we haven't had many recent reports I think that LDS is still not the best especially under high loads.

[eduardbosch]

@Tom-Fox1 I haven't implemented offline data on iOS and I don't know if I'll add it to my app in the near future. I think that LDS is not a good solution for high data DBs with large pins. It'll be really good to have a bridge between Parse and SQLite or Realm. That will make offline data access really shine.

Have you thought about any possible workaround?

[TomWFox]

@eduardbosch I’ve found that LDS has worked well for me, probably because I only need to pins a very small number of objects. However, I appreciate that it has problems and I see it as one of the most important issues to address with this SDK so I’m just looking into it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If you believe it should stay open, please let us know! As always, we encourage contributions, check out the Contributing Guide