Support for "upgradeneeded"

[tluckett77]

Working on a fairly complex project has presented the challenge of managing database data versions. Has there been given any thought into supporting database version control through the "upgradeneeded" event. It seems to be in the library, however, used in self management rather than surfacing an interface for application usage. Right now the only option I can see is to manually store and test for versions, which introduces additional challenged around validation and upgrading, or is there a recommendation on how best to deal with data version discrepancies. Thanks.

[cyrilletuzi]

Hi @tluckett77, thanks for your feedback.

The initial purpose of this lib wasn't to propose a full API equivalent to the indexedDb one, but to provide an API as simple as the localStorage one, but stored via indexedDb for performance.

But recently, new methods have been provided to extend the features of the lib, so your request could be considered.

Have you seen the new Map-like methods introduced in 7.2? Isn't the multiple store scenario suitable for your need (storing keys like v1_, v2_...)?

If not, this feature should be designed. As a first thought, the version could be managed in the providers configuration. The most difficult part is that the feature should work when going in localStorage fallback mode too.

[tluckett77]

This seems to be a viable way of handling version control for data. There may be a considerable stumbling block with one of the main valuable features of the library. The JSON Schema Validation. It would require a growing history of JSON validation Schemas to access all possible versions.

For example: Say person 'A' uses the web application at version 1 and comes back when Version 6 is in the wild. First Version 1 has to be validated as per the required validation, before applying any migration effects. This means every version of the database for the life of the application will grow the the footprint of the application. 1 schema, per field per version.

Thinking it through on the surface, in the spirit of trying to keep things simple would it be helpful to give moderated control back to the developer when it comes to validation in certain circumstances. My Thought is by adding a currentVersion in either

• a public setter to give the option of version control
• a provider property also optional to trigger version control

Say when version control is being used (vis-à-vis Version Controlled Mode) , the validation for getItem only happens when the the existing version and the currentVersion are the same. This would mean only 1 Schema ever needed, the most recent one. Rather than returning a validation error, as currently happens.

Version Controlled Mode would return a payload rather than the raw data as it does today. For example:

interface LocalStoreVersionPayload<T> {
  dataVersion: number;
  data: T;
}

Now it's possible to retrieve the data, getting around the required validation (the data is stale at this point and shouldn't be used) and can then go through a migration effect. By doing it this way, the need for versioned keys and iterations are no longer needed.

As for the localStorage fallback, I am not exactly sure what to suggest there. I am not fully aware of all the cases in which this can happen outside of old browsers and privacy/incognito modes. Seems like using a version mode should only apply to indexedDB and throw an error on fallback, or console log warning that versioning is being ignored.

[cyrilletuzi]

Version Controlled Mode would return a payload rather than the raw data as it does today

Doesn't seem a good option to me, as the signature of getItem is already quite complex/variable because of validation requirements.

Do we agree that migration can only happen when the app starts (and not on punctual calls of getItem)? In this case, the lib would just need to provide a method (or else) indicating the version.

An exact explanation of the steps of your migration scenario could help to design this feature correctly.

the validation for getItem only happens when the the existing version and the currentVersion are the same. This would mean only 1 Schema ever needed, the most recent one.

Whatever the case, data could have been forged at any time. So validation should happen all the time. But, even if not recommended, bypassing validation is already possible (see documentation).

[tluckett77]

The crux of the challenge is knowing when the client data has become obsolete (and for the sake of an application, more likely than not, unsafe for use). If it's obsolete, running any schema validation on it through the library still leaves the problem of having a schema per version per key/field.

So validation should happen all the time. But, even if not recommended, bypassing validation is already possible

Is it feasible to add a getter to request the current version of the DB? ie getDBVersion
If it is then it becomes possible to decide when to use getItem vs getUnsafeItem

Then the only piece needed is to surface the ability to trigger versioning as suggested as a "Version Controlled Mode".

Do we agree that migration can only happen when the app starts (and not on punctual calls of getItem)?

I think this comes down to the architecture of the application. For simple applications I would say most likely yes, at startup check and process migration. For modular or lazy-loaded applications this becomes problematic especially if each module stores data for itself independent of other modules. When you add NGRX/Redux style state management and Effects the complexity grows even more.

I do think this could be easily remedied though. By having the ability to know the current version of the DB via a getDBVersion, the "startup version" could be added at app startup to store state. This can then be easily referenced at anytime in any module and effect. Which provides the ability to do migrations at module load.

Thus, a simple mechanism for getting and setting a version could provided additional flexibility with minimal impact to the library.

[cyrilletuzi]

Ok, thanks for your feedback.

I'll add the investigation for this feature on the roadmap. But don't expect it too soon, it's a busy period for me at work, I'll go into that after holidays.

[tluckett77]

Thanks for taking the time to think about this possible enhancement!

[cyrilletuzi]

Can you give an example of migration? There's one thing I don't understand: even if you skip validation during migration, you need to know the structure of previous versions to be able to migrate the data, right? So you still need to keep memory of all previous versions, or am I missing something?

[cyrilletuzi]

Hi @tluckett77, any news on this?

Current work for v8 may interfere with the feature you asked (as the lib may need to use the upgradeneeded event for itself), so I'd be more confident if we think the issue before it's too late.