slow array/metadata/sub_group creation with GCS #2545
Unanswered
AndrewAnnex
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I am using zarr 2.18.3 on google cloud to create empty zarr groups with seeded metadata and arrays in the following manner:
Unfortunately the performance isn't very good, at about 5 seconds per iteration.
Without simply writing to the local disk and manually syncing to google cloud storage, is there a more performant method to be attempted here? I think I've seen references to async methods but I haven't seen a worked out example in python in the docs, or I could use the parallel methodology but I am not sure if it's possible to pickle the top level group object, but maybe I could get that top level group made and then re-open the group with zarr.open?
tips and ideas would be appreciated
Beta Was this translation helpful? Give feedback.
All reactions