Skip to content

MessagePack extensions for the datetime and interval types #3077

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Aug 22, 2022
Merged

MessagePack extensions for the datetime and interval types #3077

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a2e2694
Initial commit
xuniq Aug 9, 2022
0994328
Add draft
xuniq Aug 11, 2022
c772900
Update extensions
xuniq Aug 11, 2022
3e4f5cb
Add datetime type
xuniq Aug 11, 2022
a17e1f3
Add interval type and links
xuniq Aug 11, 2022
5bd3629
Minor additions
xuniq Aug 12, 2022
890257d
Update translation
xuniq Aug 12, 2022
8c11308
fix typo
xuniq Aug 15, 2022
e6aacf2
Apply suggestions from text review
xuniq Aug 17, 2022
1753c95
Fix typo in .po file
xuniq Aug 19, 2022
eb329d4
Update .po file
xuniq Aug 22, 2022
add3a5a
Update .po file
xuniq Aug 22, 2022
cb0ef9c
Update .po file
xuniq Aug 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions doc/book/box/data_model.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@ In this case, the number will be stored as `cdata`_.

.. _index-box_decimal:

**decimal**. The Tarantool decimal type is stored as a MsgPack ext (Extension).
**decimal**. The Tarantool decimal type is stored as a :ref:`MsgPack ext <msgpack_ext-decimal>` (Extension).
Values with the decimal type are not floating-point values although
they may contain decimal points.
They are exact with up to 38 digits of precision.
Expand All @@ -315,7 +315,7 @@ Example: a value returned by a function in the :ref:`decimal <decimal>` module.
**datetime**. Introduced in :tarantool-release:`2.10.0`.
The Tarantool ``datetime`` type facilitates operations with date and time,
accounting for leap years or the varying number of days in a month.
It is stored as a MsgPack ext (Extension).
It is stored as a :ref:`MsgPack ext <msgpack_ext-datetime>` (Extension).
Operations with this data type use code from `c-dt <https://github.com/tarantool/c-dt>`_, a third-party library.

For more information, see :doc:`Module datetime </reference/reference_lua/datetime>`.
Expand All @@ -326,7 +326,7 @@ For more information, see :doc:`Module datetime </reference/reference_lua/dateti
The Tarantool ``interval`` type represents periods of time.
They can be added to or subtracted from ``datetime`` values or each other.
Operations with this data type use code from `c-dt <https://github.com/tarantool/c-dt>`_, a third-party library.

The type is stored as a :ref:`MsgPack ext <msgpack_ext-interval>` (Extension).
For more information, see :doc:`Module datetime </reference/reference_lua/datetime>`.

.. _index-box_string:
Expand Down Expand Up @@ -356,7 +356,7 @@ Example: ``"\65 \66 \67"``.
**uuid**. The Tarantool uuid type is used for
:ref:`Universally Unique Identifiers <uuid-module>`.
Since version :doc:`2.4.1 </release/2.4.1>` Tarantool stores
``uuid`` values as a MsgPack ext (Extension).
``uuid`` values as a :ref:`MsgPack ext <msgpack_ext-uuid>` (Extension).

Example: ``64d22e4d-ac92-4a23-899a-e5934af5479``.

Expand Down
221 changes: 183 additions & 38 deletions doc/dev_guide/internals/msgpack_extensions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
.. _internals-msgpack_ext:

.. _msgpack_ext-decimal:
.. _internals-msgpack_ext:

----------------------------
MessagePack extensions
----------------------------

Tarantool uses predefined MessagePack extension types to represent some
of the special values. Extension types include ``MP_DECIMAL``, ``MP_UUID``
and ``MP_ERROR``. These types require special attention from the connector
developers, as they must be treated separately from the default MessagePack
types, and correctly mapped to programming language types.
of the special values. Extension types include ``MP_DECIMAL``, ``MP_UUID``,
``MP_ERROR``, ``MP_DATETIME``, and ``MP_INTERVAL``.
These types require special attention from the connector developers,
as they must be treated separately from the default MessagePack types,
and correctly mapped to programming language types.

.. _msgpack_ext-decimal:

*******************************
The DECIMAL type
Expand All @@ -19,19 +20,19 @@ The DECIMAL type
The MessagePack EXT type ``MP_EXT`` together with the extension type
``MP_DECIMAL`` is a header for values of the DECIMAL type.

MP_DECIMAL is 1.
``MP_DECIMAL`` type is 1.

`MessagePack spec <https://github.com/msgpack/msgpack/blob/master/spec.md>`_
`MessagePack specification <https://github.com/msgpack/msgpack/blob/master/spec.md>`_
defines two kinds of types:

* ``fixext 1/2/4/8/16`` types have fixed length so the length is not encoded explicitly;
* ``fixext 1/2/4/8/16`` types have fixed length so the length is not encoded explicitly.
* ``ext 8/16/32`` types require the data length to be encoded.

``MP_EXP`` + optional ``length`` imply using one of these types.

The decimal MessagePack representation looks like this:

.. code-block:: none
.. code-block:: none

+--------+-------------------+------------+===============+
| MP_EXT | length (optional) | MP_DECIMAL | PackedDecimal |
Expand All @@ -42,7 +43,7 @@ Here ``length`` is the length of ``PackedDecimal`` field, and it is of type

``PackedDecimal`` has the following structure:

.. code-block:: none
.. code-block:: none

<--- length bytes -->
+-------+=============+
Expand All @@ -61,7 +62,7 @@ The rightmost digit in the array is the least significant.
The first byte of the ``BCD`` array contains the first digit of the number,
represented as follows:

.. code-block:: none
.. code-block:: none

| 4 bits | 4 bits |
= 0x = the 1st digit
Expand All @@ -70,7 +71,7 @@ represented as follows:
The last byte of the ``BCD`` array contains the last digit of the number and the
final ``nibble``, represented as follows:

.. code-block:: none
.. code-block:: none

| 4 bits | 4 bits |
= the last digit = nibble
Expand All @@ -84,21 +85,21 @@ The final ``nibble`` represents the number's sign:

The decimal ``-12.34`` will be encoded as ``0xd6,0x01,0x02,0x01,0x23,0x4d``:

.. code-block:: none
.. code-block:: none

|MP_EXT (fixext 4) | MP_DECIMAL | scale | 1 | 2,3 | 4 (minus) |
| 0xd6 | 0x01 | 0x02 | 0x01 | 0x23 | 0x4d |

The decimal 0.000000000000000000000000000000000010
will be encoded as ``0xc7,0x03,0x01,0x24,0x01,0x0c``:

.. code-block:: none
.. code-block:: none

| MP_EXT (ext 8) | length | MP_DECIMAL | scale | 1 | 0 (plus) |
| 0xc7 | 0x03 | 0x01 | 0x24 | 0x01 | 0x0c |


.. _msgpack_ext-uuid:
.. _msgpack_ext-uuid:

**********************************
The UUID type
Expand All @@ -107,31 +108,35 @@ The UUID type
The MessagePack EXT type ``MP_EXT`` together with the extension type
``MP_UUID`` for values of the UUID type. Since version :doc:`2.4.1 </release/2.4.1>`.

MP_UUID is 2.
``MP_UUID`` type is 2.

The `MessagePack spec <https://github.com/msgpack/msgpack/blob/master/spec.md>`_
defines ``d8`` to mean fixext with size 16, and a uuid's size is always 16.
So the uuid MessagePack representation looks like this:
The `MessagePack specification <https://github.com/msgpack/msgpack/blob/master/spec.md>`_
defines ``d8`` to mean ``fixext`` with size 16, and a UUID's size is always 16.
So the UUID MessagePack representation looks like this:

.. code-block:: none
.. code-block:: none

+--------+------------+-----------------+
| MP_EXT | MP_UUID | UuidValue |
| = d8 | = 2 | = 16-byte value |
+--------+------------+-----------------+


The 16-byte value has 2 digits per byte.
Typically it consists of 11 fields, which are encoded as big endian
unsigned integers in the following order: time_low (4 bytes), time_mid
(2 bytes), time_hi_and_version (2 bytes), clock_seq_hi_and_reserved (1
byte), clock_seq_low (1 byte), node[0], ..., node[5] (1 byte each).
Typically, it consists of 11 fields, which are encoded as big-endian
unsigned integers in the following order:

* ``time_low`` (4 bytes)
* ``time_mid`` (2 bytes)
* ``time_hi_and_version`` (2 bytes)
* ``clock_seq_hi_and_reserved`` (1 byte)
* ``clock_seq_low`` (1 byte)
* ``node[0]``, ..., ``node[5]`` (1 byte each)

Some of the functions in :ref:`Module uuid <uuid-module>` can produce values
which are compatible with the UUID data type.
For example, after

.. code-block:: none
.. code-block:: none

uuid = require('uuid')
box.schema.space.create('t')
Expand All @@ -141,26 +146,28 @@ For example, after

a peek at the server response packet will show that it contains

.. code-block:: none
.. code-block:: none

d8 02 f6 42 3b df b4 9e 49 13 b3 61 07 40 c9 70 2e 4b

.. _msgpack_ext-error:
.. _msgpack_ext-error:

****************************************************
The ERROR type
****************************************************

Since version :doc:`2.4.1 </release/2.4.1>`, responses for errors have extra information
following what was described in :ref:`Box protocol -- responses for errors
<box_protocol-responses_error>`.
following what was described in
:ref:`Box protocol -- responses for errors <box_protocol-responses_error>`.
This is a "compatible" enhancement, because clients that expect old-style
server responses should ignore map components that they do not recognize.
Notice, however, that there has been a renaming of a constant:
formerly IPROTO_ERROR in ./box/iproto_constants.h was 0x31,
now IPROTO_ERROR is 0x52 and IPROTO_ERROR_24 is 0x31.
formerly ``IPROTO_ERROR`` in :file:`./box/iproto_constants.h` was ``0x31``,
now ``IPROTO_ERROR`` is ``0x52`` and ``IPROTO_ERROR_24`` is ``0x31``.

.. code-block:: none
``MP_ERROR`` type is 3.

.. code-block:: none

++=========================+============================+
|| | |
Expand All @@ -170,8 +177,8 @@ now IPROTO_ERROR is 0x52 and IPROTO_ERROR_24 is 0x31.
++=========================+============================+
MP_MAP

The extra information, most of which is also in :doc:`error object
</reference/reference_lua/box_error/new>` fields, is:
The extra information, most of which is also in
:doc:`error object </reference/reference_lua/box_error/new>` fields, is:

``MP_ERROR_TYPE`` (0x00) (MP_STR) Type that implies source, as in :samp:`{error_object}.base_type`, for example "ClientError".

Expand Down Expand Up @@ -203,7 +210,7 @@ For example, in version 2.4.1 or later, if we try to create a duplicate space wi
``conn:eval([[box.schema.space.create('_space');]])`` |br|
the server response will look like this:

.. code-block:: none
.. code-block:: none

ce 00 00 00 88 MP_UINT = HEADER + BODY SIZE
83 MP_MAP, size 3 (i.e. 3 items in header)
Expand Down Expand Up @@ -233,3 +240,141 @@ the server response will look like this:
00 MP_UINT = error number
05 MP_ERROR_ERRCODE
0a MP_UINT = eror code ER_SPACE_EXISTS


.. _msgpack_ext-datetime:

**********************************
The DATETIME type
**********************************

Since version :doc:`2.10.0 </release/2.10.0>`.
The MessagePack EXT type ``MP_EXT`` together with the extension type
``MP_DATETIME`` is a header for values of the DATETIME type.
It creates a container with a payload of 8 or 16 bytes.

``MP_DATETIME`` type is 4.

The `MessagePack specification <https://github.com/msgpack/msgpack/blob/master/spec.md>`_
defines ``d7`` to mean ``fixext`` with size 8 or ``d8`` to mean ``fixext`` with size 16.

So the datetime MessagePack representation looks like this:

.. code-block:: none

+---------+----------------+==========+-----------------+
| MP_EXT | MP_DATETIME | seconds | nsec; tzoffset; |
| = d7/d8 | = 4 | | tzindex; |
+---------+----------------+==========+-----------------+

MessagePack data contains:

* Seconds (8 bytes) as an unencoded 64-bit signed integer stored in the little-endian order.

* The optional fields (8 bytes), if any of them have a non-zero value.
The fields include ``nsec``, ``tzoffset``, and ``tzindex`` packed in the little-endian order.

For more information about the datetime type, see :ref:`datetime field type details <index-box_datetime>`
and :doc:`reference for the datetime module </reference/reference_lua/datetime>`.

.. _msgpack_ext-interval:

**********************************
The INTERVAL type
**********************************

Since version :doc:`2.10.0 </release/2.10.0>`.
The MessagePack EXT type ``MP_EXT`` together with the extension type
``MP_INTERVAL`` is a header for values of the INTERVAL type.

``MP_INTERVAL`` type is 6.

The interval is saved as a variant of a map with a predefined number of known attribute names.
If some attributes are undefined, they are omitted from the generated payload.

The interval MessagePack representation looks like this:

.. code-block:: none

+--------+-------------------------+-------------+----------------+
| MP_EXT | Size of packed interval | MP_INTERVAL | PackedInterval |
+--------+-------------------------+-------------+----------------+

Packed interval consists of:

* Packed number of non-zero fields.
* Packed non-null fields.

Each packed field has the following structure:

.. code-block:: none

+----------+=====================+
| field ID | field value |
+----------+=====================+

The number of defined (non-null) fields can be zero.
In this case, the packed interval will be encoded as integer 0.

List of the field IDs:

* 0 -- year
* 1 -- month
* 2 -- week
* 3 -- day
* 4 -- hour
* 5 -- minute
* 6 -- second
* 7 -- nanosecond
* 8 -- adjust

**Example**

Interval value ``1 years, 200 months, -77 days`` is encoded in the following way:

.. code-block:: tarantoolsession

tarantool> I = datetime.interval.new{year = 1, month = 200, day = -77}
---
...

tarantool> I
---
- +1 years, 200 months, -77 days
...

tarantool> M = msgpack.encode(I)
---
...

tarantool> M
---
- !!binary xwsGBAABAczIA9CzCAE=
...

tarantool> tohex = function(s) return (s:gsub('.', function(c) return string.format('%02X ', string.byte(c)) end)) end
---
...

tarantool> tohex(M)
---
- 'C7 0B 06 04 00 01 01 CC C8 03 D0 B3 08 01 '
...

Where:

* C7 -- MP_EXT
* 0B -- size of a packed interval value (11 bytes)
* 06 -- MP_INTERVAL type
* 04 -- number of defined fields
* 00 -- field ID (year)
* 01 -- packed value ``1``
* 01 -- field ID (month)
* CCC8 -- packed value ``200``
* 03 -- field ID (day)
* D0B3 -- packed value ``-77``
* 08 -- field ID (adjust)
* 01 -- packed value ``1`` (DT_LIMIT)

For more information about the interval type, see :ref:`interval field type details <index-box_interval>`
and :doc:`description of the datetime module </reference/reference_lua/datetime>`.
6 changes: 3 additions & 3 deletions locale/ru/LC_MESSAGES/book/box/data_model.po
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,12 +531,12 @@ msgstr ""
msgid ""
"**uuid**. The Tarantool uuid type is used for :ref:`Universally Unique "
"Identifiers <uuid-module>`. Since version :doc:`2.4.1 </release/2.4.1>` "
"Tarantool stores ``uuid`` values as a MsgPack ext (Extension)."
"Tarantool stores ``uuid`` values as a :ref:`MsgPack ext <msgpack_ext-uuid>` (Extension)."
msgstr ""
"**uuid**. Тип uuid в Tarantool используется для :ref:`универсальных "
"уникальных идентификаторов (UUID) <uuid-module>`. Начиная с версии "
":doc:`2.4.1 </release/2.4.1>` Tarantool хранит значения ``uuid`` в формате "
"MsgPack ext (Extension)."
":doc:`2.4.1 </release/2.4.1>`, Tarantool хранит значения ``uuid`` в формате "
":ref:`MsgPack ext <msgpack_ext-uuid>` (Extension)."

msgid "Example: ``64d22e4d-ac92-4a23-899a-e5934af5479``."
msgstr "Пример: ``64d22e4d-ac92-4a23-899a-e5934af5479``."
Expand Down
Loading