Key encoding for dbm-style databases

[alimanfoo]

There are a number of embedded key-value stores available, including the built-in dbm-style databases provided by the Python standard library, as well as BerkeleyDB and Kyoto Cabinet. These provide a MutableMapping interface but generally expect keys to be bytes objects not strings, and will return bytes objects when asked for an iterator over keys. Zarr uses text strings for keys currently, and so these key-value databases cannot be used directly with Zarr, however it would be nice and should be straighforward to provide some support.

There are two options. (1) Provide a MutableMapping wrapper class that performs key encoding/decoding (e.g., 'zarr.storage.KeyValueStore' or 'zarr.storage.DbmStore'). (2) Add a key_encoding option to the Array and Group classes, which would be None by default (preserving current behaviour to use text strings as keys) but if not None would be used to encode and decode keys outside of the store.

[alimanfoo]

I'm leaning towards (2) as it seems reasonable and avoids yet another layer of indirection for the user.

[jakirkham]

One thing to consider is whether this will open the door to handling a variety of different encodings.

Also related, will the encoding information be stored somehow?

[alimanfoo]

On Mon, Feb 27, 2017 at 4:35 PM, jakirkham ***@***.***> wrote: One thing to consider is whether this will open the door to handling a variety of different encodings. Also related, will the encoding information be stored somehow?

That's a good question. In both of the options I put above it would be up to the user to select the right key encoding when interacting with an existing array or group, i.e., the key encoding would not get stored in the group or array metadata.

…

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/133#issuecomment-282773421>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QqU1PnesLCV5noTpVxzi2bEBiu-Fks5rgvtqgaJpZM4MM4IK> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: [email protected] Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721