Question: Should I use Redux-persist if I have large objects in state?

[thorbenandresen]

(This is amazing, it relieved me of a lot of pain I experienced with persisting my state manually. Thanks so much!)

My Situation
I am building a react-native app which is a client for pocket.co. So currently I query their REST API, convert the response into an Immutable.JS object and then load the object with the articles into my reducer and persist the entire article object via AsyncStorage.set(). When I then query the API for updates, I load my entire article object via AsyncStorage.get(), merge it with the updates and then load the merged article object into the reducer. I first render the articles when these operations have completed.

My article object:

Reading/Writing the article object from/to AsyncStorage (including JSON.stringify/parse) can be quite expensive when you have have lot of articles. I tested persisting the reducer the article object with redux-persist, but once I mutate the state (e.g. starring an article, deleting an article) my app gets very unresponsive and eventually crashes.

My Question
What are the best practices if you use redux as your database? Persisting the object manually with an object store like Realm or is there a way to make this work with redux-persist?

[rt2zz]

What you are trying to do I suspect will always be somewhat expensive since you want to put all articles into a single Map. This is not a case that redux-persist handles particularly well out of the box.

Off the top of my head, you might consider is storing each article under a separate key, possibly outside of redux entirely. Then in redux you can store a List with your article ids. Then whenever you need to fetch an article(s) you have an async get method that loads from storage.

Hopefully that helps. Let me know what you ultimately end up figuring out, and if there is a clean way for redux-persist to help support this use case I am all ears.

[JulianKingman]

I'm also interested in performance considerations. I am retrieving tasks from the server (up to 10,000 at once), storing them in redux, and then persisting. I'm assuming that the persistence is the slowest part, it takes about 2 minutes to fully load and persist the data. Would changing the storage driver to Realm fix this?

[nhayfield]

i found that redux-persist and async storage works fine for large amounts of data on ios but chokes in android. i have begun working on a solution but haven't fully integrated it yet: sriraman/react-native-shared-preferences#5
and
https://github.com/nhayfield/react-native-shared-preferences
i may continue but i'm going to attempt to up the storage limit for asyncstorage in android first based on this:
https://github.com/facebook/react-native/pull/11656/files

[JulianKingman]

That's cool, good work :)
I still have the issue with iOS, though.

Perhaps someone can clarify something for me, which I think may be the root of the problem. It looks here as though only the immediate substate or a redux store is diffed and updated or not, is that correct? If yes, then I believe that means that this:

// Scenario A
// Store1 === [...manyThings], Store2 === [...manyThings], Store3 === [...manyThings],
persistStore(Store1);
persistStore(Store2);
persistStore(Store3);

Would perform magnitudes better than this:

// Scenario B
// Store === {Collection1: [...manyThings],  Collection2: [...manyThings], Collection3: 
// Collection1 === Store1, etc...
[...manyThings]}
persistStore(Store)

Is that correct?

To explain, in Scenario A, my collections are separated into a store for each. As documents come flooding in to Store1, it's checking each individual document (...manyThings) in the collection to see if it already exists and was downloaded. Documents are relatively small, so no problem.
In Scenario B, as documents are flooding into Store, Collection1 is diff'd against the new incoming Collection1, which won't match because new documents are still coming in (even though documents within do match). As a result, the persistor persists the collection as much as possible until the document collections finally match. In my current use-case, this is around 10,000 records.

[rt2zz]

@JulianKingman sceario A is workable in some scenarios, but I think there are ways to get scenario B to work for you as well. e.g.

1. if you set debounce: 1000 in config it will write to disk at most once every second.
2. Another option if you are batch loading these documents at specific times, you can call persistor.pause() until you document sync is complete.

[cbfranca]

@thorbenandresen , how did you solved this?

[Dante-101]

I am also interested in potential solutions.

My state has few large objects at 1st level (4-7 MB in total) which don't update much. Since the state is persisted under one key in v5, I am worried that it will be a big performance impact and battery drain for the device even when I change a boolean in some other key of the state.

I tried using nested reducers for the large objects but unfortunately, I could not get them to persist at all! It will be great if someone can help out with an example.

[nhhockeyplayer]

guys
i am using redux... ngrx
deleting 1000+ objects is way slow
i put async on the methods

any ideas on how to speed this up or make it spin off in its own thread... async isnt cutting it

[cristianocca]

I'm curious about this as well. How well can the library handle large objects? Does it persist constantly / after any change?
I'm wondering what's faster: A simple JSON.stringify + write to disk (is this what the library would do when using react-native-fs?), vs using "Realm".

[jamesisaac]

Does it persist constantly / after any change?

Yes, that's the idea. You can use the throttle config option to partially mitigate this though.

I'm wondering what's faster: A simple JSON.stringify + write to disk (is this what the library would do when using react-native-fs?), vs using "Realm".

Of course Realm would be faster, it's designed from the ground up with performance on large datasets in mind. This library is great for being something that can seamlessly slot into a Redux architecture, bringing with it all of Redux's benefits, but not the most performant choice if you're expecting lots of writes.

[jovylle]

i found that redux-persist and async storage works fine for large amounts of data on ios but chokes in android. i have begun working on a solution but haven't fully integrated it yet: sriraman/react-native-shared-preferences#5
and
https://github.com/nhayfield/react-native-shared-preferences
i may continue but i'm going to attempt to up the storage limit for asyncstorage in android first based on this:
https://github.com/facebook/react-native/pull/11656/files

Why or when did it got Choked?
is is the problem with the Asyncstorage it self? and should I try replacing it with, SQLlite

[bendelonlee]

It's been asked by @cbfranca, but I'll ask again: @thorbenandresen, how did you solve this?

Thanks in advance should you find the time to answer

[afleissner2019]

Could anyone familiar with this error message.. I checked my Redux dev Tools it seems that there is a huge file that causes the problem. It happens every time I upload the 6.1 Mb jpeg file, the browser let me save the large file ,however I keep seeing this error message. Do you have any idea on how to fix it. Thank you in advance