How to decrease traffic when writing into region of zarr archive #3089
Unanswered
meteoDaniel
asked this question in
Q&A
Replies: 1 comment 2 replies
-
No one ? Am I the only facing issues with region write? |
Beta Was this translation helpful? Give feedback.
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
we are initialize empty zarrs with shape:
(31, 96, 1000, 1000) and chunks : (31, 96, 50, 50) because we need all data from dim 1 and 2 in each query.
Now we write into region again:
During writing along the first dimension day by day, the whole chunk (aka dataset) needs to be loaded to write into region. This causes a lot of traffic.
We would like to go into s3 object store with the process but we need to decrease the traffic. Do you know how to achive this?
The previous solution used append mode, which causes chunk sizes (1, 96, 50, 50) and very long queries against the dataset.
Best regards
Daniel
Beta Was this translation helpful? Give feedback.
All reactions