Consider using msgpack.encode instead tostring in hash function

[olegrok]

What's wrong with tostring:

tarantool> tostring(922337203685477580LL)
---
- 922337203685477580LL
...

tarantool> tostring(922337203685477580ULL)
---
- 922337203685477580ULL
...

Possible consequences:

tarantool> box.space.test:insert({922337203685477580LL})
---
- [922337203685477580]
...

tarantool> tostring(box.space.test:get({922337203685477580LL})[1])
---
- 922337203685477580ULL
...

Hashes calculated for "922337203685477580ULL" and "922337203685477580LL" are different. This could lead to some unexpected results for users. (I understand that tarantool internally uses msgpuck that stores any positive integer as unsigned)

Without breaking of backward compatibility I suggest to introduce new hash function that will be stable for such cases. Or at least document such behaviour.

[Gerold103]

I totally agree, good catch, thanks for reporting it. Unfortunately, I can't change the existing vshard.router.bucket_id() function, because it will corrupt existing users, but I can deprecate it, and introduce a new function. Then the old one will be deleted after some time. I am going to call a new one vshard.router.bucket_id_mpcrc32() - this will explicitly say, that it uses MessagePack and crc32. In case more functions will be added, I will just use different suffixes for them.
Sounds ok?

[olegrok]

Yes, agree with you, let's introduce such function that uses msgpack instead of tostring

[R-omk]

However, it is also worth considering that numbers in msgpack can be encoded differently.

[olegrok]

However, it is also worth considering that numbers in msgpack can be encoded differently.

Could you please give an example?

Without explicit casts I assume it's safe

tarantool> msgpack.encode(1)
---
- "\x01"
...

tarantool> msgpack.encode(1ULL)
---
- "\x01"
...

tarantool> msgpack.encode(1LL)
---
- "\x01"
...

tarantool> msgpack.encode(ffi.cast('double', 1))
---
- !!binary yz/wAAAAAAAA
...
tarantool> tostring(ffi.cast('double', 1))
---
- 'cdata<double>: 0x0102d885c8'
...

And without losing the precision of the number, that's obvious.

tarantool> msgpack.encode(1844674407370955161LL)
---
- !!binary zxmZmZmZmZmZ
...
tarantool> msgpack.encode(1844674407370955161ULL)
---
- !!binary zxmZmZmZmZmZ
...
tarantool> msgpack.encode(1844674407370955161)
---
- !!binary zxmZmZmZmZoA
...
tarantool> 1844674407370955161
---
- 1844674407370955264
...

[Gerold103]

tarantool> msgpack.encode(ffi.cast('float', 1))
---
- !!binary yj+AAAA=
...

tarantool> msgpack.encode(ffi.cast('double', 1))
---
- !!binary yz/wAAAAAAAA
...

tarantool> msgpack.encode(1)
---
- "\x01"
...

My plan was to call tonumber() on cdata numbers before passing them to encode.
Now I already think, that I can fix the existing bucket_id() in the same way - just call tonumber(), if see any cdata number. It won't make worse, because currently bucket_id() is just broken for such values. Assuming, that it was always used for normal numbers only, tonumber() won't break existing users, but will make it safe.

[Gerold103]

Whoever is interested in participation of the new API and behaviour development: join https://lists.tarantool.org/pipermail/tarantool-patches/2020-February/014451.html thread.