HDFStore still corrupted reads with utf8

[wabu]

hdf5 read still fails to read back some utf8 encoded strings back correctly.

I written a test script that:

• puts random valid utf8 strings into a hdf5 file
• reads them back with pytables and decodes them
• uses HDFStore.select to read them
  The hdfstore often fails, but the data is stored correctly inside the hdf5 file.

Here's an example:

                        orig                                         h5ed
0  '7PFKۑw\x1d\x1bxyv\x03;S'   '\x002\x003\x004\x005\x006\x007\x008\x009'
1              '\x12漣\x03_L'  ':\x00;\x00<\x00=\x00>\x00?\x00@\x00A\x00B'
2            'ӉwxIy⣂&QC\x14'   '\x00C\x00D\x00E\x00F\x00G\x00H\x00I\x00J'
3          'u\x01\x10aۋZ9<s'   '\x00T\x00U\x00V\x00W\x00X\x00Y\x00Z\x00['
4       '!\x0b\x06зt\x13QMR'                                      '\x00c'

see #6505 for a previous issue on the topic

[wabu]

version info:

python: 3.3.2.final.0
python-bits: 64
OS: Linux
OS-release: 3.2.0-61-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_NZ.UTF-8

pandas: 0.14.0rc1-73-g8793356
Cython: 0.19.1
numpy: 1.8.1
scipy: 0.12.0
IPython: 2.0.0-dev
patsy: 0.2.1
scikits.timeseries: None
dateutil: 2.2
pytz: 2013b
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
...

[jreback]

ok...feel free to have a look, AFAICT it correct decides it can't decode then uses the vectorized decode. Not sure if the error is that data.astype('U16').astype(object) does the wrong thing if it is encoded and only has ascii (so the 'regular' numpy coercion works), but I am not that familiar with what it is actually doing (numpy)

[jreback]

maybe make an example that has astype decodables in one section and then non-decodables.

plus the example needs to always pass/fail (on these cases). of course that means have to figure WHY it is failing!

[jreback]

This might be the issue here: numpy/numpy#3939

If this is the case, then may have to either detect when can use this or only do the vectorize (which is much slower, unfortunately); but better to be correct and slow then wrong and fast

This might be related as well:
numpy/numpy#3258

PyTables 'fixes' this for a VLUnicodeAtom (which is not used in tables), so we may need to do something similar
https://github.com/PyTables/PyTables/blob/develop/tables/atom.py#L1108

[jreback]

@wabu fix for this?

[wabu]

still did not dig deep enough into this ...

[OmerJog]

Is there a fix for this bug?

[TomAugspurger]

Seems to still be open. Are you interested in working on it?

…

On Mon, Jun 17, 2019 at 1:47 PM OmerJog ***@***.***> wrote: Is there a fix for this bug? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7244?email_source=notifications&email_token=AAKAOIW4LTLOCD6VEIS73DLP27L3LA5CNFSM4AP2FZW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX4DEZI#issuecomment-502805093>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIUH3K3D7YKC7C3TIITP27L3LANCNFSM4AP2FZWQ> .