New msgpack api (2.11 and 2.10) #3605

andreyaksenov · 2023-08-02T12:03:20Z

Documented the following API members:

msgpack_object[key] and the corresponding msgpack_object:get(key) function:
- https://docs.d.tarantool.io/en/doc/2.11-msgpack-api/reference/reference_lua/msgpack/#msgpack-object-item
- https://docs.d.tarantool.io/en/doc/2.11-msgpack-api/reference/reference_lua/msgpack/#lua-function.msgpack_object.get
The iterator_object:take_array(count) function: https://docs.d.tarantool.io/en/doc/2.11-msgpack-api/reference/reference_lua/msgpack/#lua-function.iterator_object.take_array
The cfg.encode_error_as_ext configuration property: https://docs.d.tarantool.io/en/doc/2.11-msgpack-api/reference/reference_lua/msgpack/#msgpack-cfg

Note that updated docs reference testable code snippets for making sure the code shown in docs is correct.

doc/reference/reference_lua/msgpack.rst

locker · 2023-08-07T10:03:47Z

doc/reference/reference_lua/msgpack.rst

@@ -241,7 +236,7 @@ Below is a list of all ``msgpack`` functions and members.

 .. function:: decode_array_header(byte-array, size)

-    Call the mp_decode_array function in the `MsgPuck <http://rtsisyk.github.io/msgpuck/>`_ library
+    Call the `MsgPuck <https://rtsisyk.github.io/msgpuck/>`_'s ``mp_decode_array`` function


May be worth mentioning here that this function expects the data to be a MsgPack array. If it isn't, an exception is raised. Same comment for decode_map_header.

locker · 2023-08-07T10:21:48Z

doc/reference/reference_lua/msgpack.rst

@@ -528,7 +538,7 @@ userdata/cdata, you can use this code:
 .. NOTE::

    To achieve the same effect for only one call to ``msgpack.encode()``
-    (i.e. without changing the configuration permanently), you can use
+    (that is without changing the configuration permanently), you can use
    ``msgpack.encode({1, x, y, 2}, {encode_invalid_numbers = true})``.


This isn't true:

tarantool> msgpack.encode({1, 2}, {encode_invalid_numbers = true}) --- - error: 'msgpack.encode: argument 2 must be of type ''struct ibuf''' ...

It's possible to do this though:

msgpack.new({encode_invalid_numbers = true}).encode({1, 2})

Thanks. Also found out the the example above with httpc.curl doesn't work:

tarantool> msgpack.encode(httpc.curl) --- - error: unsupported Lua type 'userdata' ... tarantool> msgpack.encode(httpc.curl, {encode_use_tostring=true}) --- - error: 'msgpack.encode: argument 2 must be of type ''struct ibuf''' ...

Changed to msgpack.cfg{encode_use_tostring = true}.

locker · 2023-08-07T10:27:03Z

doc/reference/reference_lua/msgpack.rst

+                    *   if a MsgPack object is an associative array, ``key`` is the string value that specifies the element key.
+
+        :return: an element of the MsgPack array.
+                 If the specified key is missing in the array, `get` returns ``nil``.


May be worth mentioning that this function raises an error in case the value stored in the object isn't a MsgPack array or map.

locker · 2023-08-07T10:38:01Z

doc/code_snippets/test/msgpack/msgpack_object_iterator_map_test.lua

+local first = mp_map_iterator:decode()           -- returns 'foo'
+local second = mp_map_iterator:decode()          -- returns 'a'
+mp_map_iterator:skip()                           -- returns none, skips 'bar'
+local fourth = mp_map_iterator:decode()          -- returns ['b1', 'b2', 'b3']


Strictly speaking, map elements will not necessarily be returned in this order. A Lua map is unordered so the order in which it's encoded in MsgPack depends on Lua internals. It may be better to rework the example to put decoded elements back into a map/set.

locker · 2023-08-07T10:44:29Z

doc/reference/reference_lua/msgpack.rst

-    ..  code-block:: lua
+    ..  literalinclude:: /code_snippets/test/msgpack/msgpack_object_test.lua
+        :language: lua
+        :lines: 1,29-32


Looks like you forgot line 5: line 31 refers to mp_from_string defined at line 5.

It's difficult to review these examples. IMO better inline them in the text.

Thanks, fixed. We decided to use testable examples to make sure we don't miss any breaking changes that might break the code snippets used in docs. Looks like more 'atomic' examples that show only the minimal API usage would be more convenient for review - msgpack_object_test.lua actually mixes examples that can be split up.

locker · 2023-08-07T10:47:07Z

doc/code_snippets/test/msgpack/msgpack_object_iterator_take_test.lua

+local size = mp_array_iterator:decode_array_header()  -- returns 3
+local first = mp_array_iterator:decode()              -- returns 10
+mp_array_iterator:skip()                              -- returns none, skips 20
+local mp_array_new = mp_array_iterator:take()


Why is it called mp_array_new? It's a MsgPack object that contains 30, i.e. it doesn't have anything to with MsgPack arrays.

mixed up with take_array, thanks for pointing out

xuniq · 2023-08-08T10:01:04Z

doc/reference/reference_lua/msgpack.rst


        *   -   :ref:`msgpack.decode_map_header(byte-array, size) <msgpack-decode_map_header>`
-            -   Skip a map header in a raw MsgPack string
+            -   Call the `MsgPuck <https://rtsisyk.github.io/msgpuck/>`_'s ``mp_decode_map`` function and return the map size and a pointer to the first map component.


Suggested change

- Call the `MsgPuck <https://rtsisyk.github.io/msgpuck/>`_'s ``mp_decode_map`` function and return the map size and a pointer to the first map component.

- Call the `MsgPuck <https://rtsisyk.github.io/msgpuck/>`_'s ``mp_decode_map`` function and return the map size and a pointer to the first map component

xuniq · 2023-08-08T10:04:50Z

doc/reference/reference_lua/msgpack.rst

@@ -301,8 +305,8 @@ The MsgPack output structure can be specified with the ``__serialize`` parameter

 * 'seq', 'sequence', 'array' - table encoded as an array
 * 'map', 'mappping' - table encoded as a map
-* function - the meta-method called to unpack serializable representation
-  of table, cdata or userdata objects
+* function - the meta-method called to unpack the serializable representation


Suggested change

* function - the meta-method called to unpack the serializable representation

* function -- the meta-method called to unpack the serializable representation

Same for lines 306-307

xuniq · 2023-08-08T10:07:52Z

doc/reference/reference_lua/msgpack.rst

 excessively sparse arrays will be handled as maps.

 **msgpack.cfg() example 1:**

 If ``msgpack.cfg.encode_invalid_numbers = true`` (the default),
 then NaN and Inf are legal values. If that is not desirable, then
-ensure that ``msgpack.encode()`` will not accept them, by saying
+ensure that ``msgpack.encode()`` do not accept them, by saying


Suggested change

ensure that ``msgpack.encode()`` do not accept them, by saying

ensure that ``msgpack.encode()`` does not accept them, by saying

xuniq · 2023-08-08T10:39:26Z

doc/reference/reference_lua/msgpack.rst

+
+        The index key used to get the array element might be one of the following:
+
+        *   if a MsgPack object is an array, ``key`` is the integer value (start with 1) that specifies the element index.


Suggested change

* if a MsgPack object is an array, ``key`` is the integer value (start with 1) that specifies the element index.

* if a MsgPack object is an array, the ``key`` is an integer value (starting with 1) that specifies the element index.

xuniq · 2023-08-08T10:39:39Z

doc/reference/reference_lua/msgpack.rst

+
+        :param number/string key: the index key used to get the array element, which might be one of the following:
+
+                    *   if a MsgPack object is an array, ``key`` is the integer value (start with 1) that specifies the element index.


Suggested change

* if a MsgPack object is an array, ``key`` is the integer value (start with 1) that specifies the element index.

* if a MsgPack object is an array, the ``key`` is an integer value (starting with 1) that specifies the element index.

andreyaksenov linked an issue Aug 2, 2023 that may be closed by this pull request

Document msgpack.object:index #3257

Closed

andreyaksenov force-pushed the 2.11-msgpack-api branch 12 times, most recently from 3194898 to 5ee66d1 Compare August 4, 2023 14:57

This was linked to issues Aug 7, 2023

Document msgpack.cfg.encode_error_as_ext #2417

Closed

Document msgpack object iterator take_array method #2749

Closed

andreyaksenov force-pushed the 2.11-msgpack-api branch 3 times, most recently from a16772e to 313e50d Compare August 7, 2023 08:49

andreyaksenov changed the title ~~New msgpack api~~ New msgpack api (2.11 and 2.10) Aug 7, 2023

andreyaksenov force-pushed the 2.11-msgpack-api branch from 313e50d to df2858c Compare August 7, 2023 09:21

andreyaksenov requested review from CuriousGeorgiy and locker August 7, 2023 09:42

CuriousGeorgiy requested changes Aug 7, 2023

View reviewed changes

doc/reference/reference_lua/msgpack.rst Outdated Show resolved Hide resolved

locker reviewed Aug 7, 2023

View reviewed changes

locker assigned andreyaksenov Aug 7, 2023

andreyaksenov force-pushed the 2.11-msgpack-api branch 5 times, most recently from fb14e6f to 0519abf Compare August 7, 2023 13:02

andreyaksenov force-pushed the 2.11-msgpack-api branch 3 times, most recently from 7918deb to 4f48bae Compare August 7, 2023 14:21

andreyaksenov requested a review from CuriousGeorgiy August 8, 2023 07:04

andreyaksenov force-pushed the 2.11-msgpack-api branch 2 times, most recently from 3f1c765 to 755768e Compare August 8, 2023 07:32

andreyaksenov requested a review from xuniq August 8, 2023 07:57

CuriousGeorgiy approved these changes Aug 8, 2023

View reviewed changes

andreyaksenov force-pushed the 2.11-msgpack-api branch from 755768e to a0b61f6 Compare August 8, 2023 08:55

xuniq approved these changes Aug 8, 2023

View reviewed changes

New msgpack api

a43fb40

andreyaksenov force-pushed the 2.11-msgpack-api branch from a0b61f6 to a43fb40 Compare August 8, 2023 11:27

andreyaksenov merged commit d21a9a7 into latest Aug 8, 2023

andreyaksenov deleted the 2.11-msgpack-api branch August 8, 2023 11:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

New msgpack api (2.11 and 2.10) #3605

New msgpack api (2.11 and 2.10) #3605

Uh oh!

andreyaksenov commented Aug 2, 2023 •

edited

Loading

Uh oh!

Uh oh!

locker Aug 7, 2023

Uh oh!

locker Aug 7, 2023

Uh oh!

andreyaksenov Aug 7, 2023

Uh oh!

locker Aug 7, 2023

Uh oh!

locker Aug 7, 2023

Uh oh!

locker Aug 7, 2023

Uh oh!

andreyaksenov Aug 7, 2023

Uh oh!

locker Aug 7, 2023

Uh oh!

andreyaksenov Aug 7, 2023

Uh oh!

xuniq Aug 8, 2023

Uh oh!

xuniq Aug 8, 2023

Uh oh!

xuniq Aug 8, 2023

Uh oh!

xuniq Aug 8, 2023

Uh oh!

xuniq Aug 8, 2023

Uh oh!

Uh oh!

	- Call the `MsgPuck <https://rtsisyk.github.io/msgpuck/>`_'s ``mp_decode_map`` function and return the map size and a pointer to the first map component.
	- Call the `MsgPuck <https://rtsisyk.github.io/msgpuck/>`_'s ``mp_decode_map`` function and return the map size and a pointer to the first map component

	* function - the meta-method called to unpack the serializable representation
	* function -- the meta-method called to unpack the serializable representation

	ensure that ``msgpack.encode()`` do not accept them, by saying
	ensure that ``msgpack.encode()`` does not accept them, by saying


		The index key used to get the array element might be one of the following:

		* if a MsgPack object is an array, ``key`` is the integer value (start with 1) that specifies the element index.

	* if a MsgPack object is an array, ``key`` is the integer value (start with 1) that specifies the element index.
	* if a MsgPack object is an array, the ``key`` is an integer value (starting with 1) that specifies the element index.


		:param number/string key: the index key used to get the array element, which might be one of the following:

		* if a MsgPack object is an array, ``key`` is the integer value (start with 1) that specifies the element index.

New msgpack api (2.11 and 2.10) #3605

New msgpack api (2.11 and 2.10) #3605

Uh oh!

Conversation

andreyaksenov commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andreyaksenov commented Aug 2, 2023 •

edited

Loading