Migrate storage account from Azure to AWS

[agafgfafa]

We will be moving the Open VSX storage account from Azure to AWS. We will need a technical plan and a communications plan, both of which will be driven from within this issue.

The move will be substantial in that we'll likely need to put the central service into a read-only mode while the ~1.7TB of extension data is moved across the Internet. The application deployment will also likely need to be altered to point to the new storage account.

As a first draft, here's how I see the sequence of events:
0. EF communicates the migration to the Open VSX community ahead of time, as well as a proposed timeline. Tentatively, I would like to suggest this migration occur Thursday, Nov 27 at 9h00 Eastern Time

1. EF (Software Dev? + Releng) to add a notice of the migration to the Open VSX home page
2. EF Infra to create a staging and a production storage account on AWS
3. EF Infra + Releng migrate Azure staging storage to AWS staging storage and point Open VSX staging to the new storage. Test and document the process. Undo it, then redo it again to make sure process is sane.
4. EF Infra + Releng On migration day (Nov 27), set Open VSX in read-only mode and follow the process for production data.

[agafgfafa]

/cc @tfroment @chrisguindon @pstankie @fredg02

[chrisguindon]

EF (Software Dev? + Releng) to add a notice of the migration to the Open VSX home page

We can replace the current banner that mentions the updated Open VSX Publisher Agreement. The change is in this file:

open-vsx.org/website/src/page-settings.tsx

Line 196 in f04cef8

banner: {

@gnugomez can help with this update.

I suggest we plan to publish the notice on 19 or 20 November. This gives about seven days before the planned migration date. @agafgfafa can confirm if this timing works for him!

Thanks

[agafgfafa]

@chrisguindon that works. Our SLO states a minimum of 3 business days of notice for non-blocking maintenance.

Thanks for the rapid response!

[netomi]

apart from the change of storage location, open-vsx.org has some additional functionality that currently is only supported for the Azure storage provider:

• download counts

which are visible for each extension and are also used in the relevance search:

This functionality needs to be replicated for Aws before doing the migration imho.

[netomi]

Azure seems to have something that is called insights logs, which seems to be available as blobs with a duration of 1h.

We download these logs on a regular basis, increase download counts and delete the relevant blob afterwards in Azure.

[netomi]

The app already has support for something called Migrations.

We could implement a job to migrate files stored in one storage location to another one, such that that would happen while the service is running normally. New publications will be stored in the primary location (AWS), while existing files will still be served from the current location (Azure) until they are migrated.

[agafgfafa]

https://status.open-vsx.org/maintenance is the correct page to link to, and the maintenance event is now there.