Keyword arguments and generalized unpacking for C++ API #372

dean0x7d · 2016-08-29T22:34:31Z

General idea

A Python function can be called with the syntax:

foo(a1, a2, *args, ka=1, kb=2, **kwargs)

This PR adds support for the equivalent syntax in C++:

foo(a1, a2, *args, "ka"_a=1, "kb"_a=2, **kwargs)

In addition, generalized unpacking is implemented, as per PEP 448, which allows calls with multiple * and ** unpackings in Python:

bar(*args1, 99, *args2, 101, **kwargs1, kz=200, **kwargs2)

and in C++:

bar(*args1, 99, *args2, 101, **kwargs1, "kz"_a=200, **kwargs2)

Impact on compile time

The new functionality is SFINAEd-off into a separate instantiation path from the existing positional-only arguments. Thus, the compile time will not increase for existing code. When (and only when) keywords and/or generalized unpacking are used, a little more template metaprogramming is in play, but it's not really any more complicated than the existing def/arg machinery for naming arguments and assigning default values. There are no compile time slowdowns as far as I've seen.

Python version compatibility

PEP 448 is implemented only in Python >= 3.5, but the C++ API allows these kinds of calls for any Python version. Limiting it by version is also possible, but I don't think it's necessary. The function calls are still perfectly valid, it's just that:

f(**kwargs1, **kwargs2)  # Python 3.5

would look like:

kwargs1.update(kwargs2)  # Python < 3.5
f(**kwargs1)

Applications

This PR also implements a few functions using this new functionality.

Python's print function is replicated in the C++ API including the keyword arguments sep, end, file, flush. E.g.:

py::print(v1, v2, "sep"_a=" -- ");

The str.format method is added. Together with the _s UDL for str, this allows the following syntax in C++:

auto str1 = "Hello, {}! Your number is {}"_s.format("World", 42);
// or
auto str2 = "Hello, {name}! Your number is {number}"_s.format("number"_a=42, "name"_a="World");

The py::dict class also gets a keyword constructor which replicates its Python counterpart:

auto d = dict("number"_a=42, "name"_a="World");

Feedback?

I hope this is not considered too much of an abuse of C++. Let me know if this is OK so far.

wjakob · 2016-08-30T12:41:24Z

include/pybind11/cast.h

@@ -308,6 +309,7 @@ template <typename type> class type_caster_base : public type_caster_generic {
 };

 template <typename type, typename SFINAE = void> class type_caster : public type_caster_base<type> { };
+template <typename type> using make_caster = type_caster<intrinsic_t<type>>;


this type alias is a good idea -- should have added that much earlier :)

For consistency, it would be nice to this make_caster alias in all places where type_caster<typename intrinsic_type<type>::type> is currently used (quite a few I think)

wjakob · 2016-08-30T12:59:46Z

After a brief look, this seems like very a nice idea (and I like the print function). I'll have to set a aside a bit more time to look at all the metatemplate code but for now it seems very reasonable.

Is there a symmetrical change that would also have to be done to function call handlers, or are these feature-complete as far as PEP 448 goes?

dean0x7d · 2016-08-30T13:45:24Z

Is there a symmetrical change that would also have to be done to function call handlers, or are these feature-complete as far as PEP 448 goes?

As far as I can tell, everything is already fine in the other direction. Generalized unpacking is just syntax sugar for the interpreter. The multiple tuples/dicts are merged and the C API only sees the usual pair of args/kwargs.

aldanor · 2016-08-30T22:14:23Z

This is pretty cool. A few lines in the changelog would be nice

dean0x7d · 2016-09-04T23:02:51Z

Updated docs with the new call syntax. Updated changelog with all the new features.
The py::dict class also got a keyword constructor which replicates its Python counterpart:

auto d = dict("number"_a=42, "name"_a="World");

MSVC had an issue which required a slightly ugly workaround. It's not too bad, but perhaps somebody has a better solution.

wjakob · 2016-09-05T00:36:54Z

include/pybind11/attr.h


    const char *name;
 };

 /// Annotation for keyword arguments with default values
 template <typename T> struct arg_t : public arg {
    constexpr arg_t(const char *name, const T &value, const char *descr = nullptr)
-        : arg(name), value(value), descr(descr) { }
-    T value;
+        : arg(name), value(&value), descr(descr) { }


Hmm, this change (pointer instead of an explicit copy of value) seems problematic to me. It means that arg_t cannot receive something that is constructed on the fly, e.g.

m.def("fun", py::arg("arg") = MyArg(1, 2, 3));

Temporary objects (even ones which are not bound to a const&) are only destroyed after the full expression is evaluated. A full expression is one which is not contained within any other expression (so essentially, temp object lifetime is extended to a semicolon).

This makes the following line perfectly safe:

f("arg"_a=MyArg(1, 2, 3));

On the other hand,

auto a = "arg"_a=MyArg(1, 2, 3); f(a);

would not be safe, but this is currently a compile time error because unpacking_collector only accepts rvalues: arg<T>&&. (Although, I do need to add this safeguard to process_attribute as well.)

That said, I think the ideal implementation of arg_t would be to hold a py::object instead of T:

struct arg_t : arg { template <typename T> arg_t(const char *name, const T &value, const char *descr = nullptr) : arg(name), value(make_caster<T>::cast(value, return_value_policy::automatic)), descr(descr) { } object value; const char *descr; }

The to-python cast would then be done right away instead of being delayed. I didn't make this change originally because default arguments are collected with the automatic policy while function calls use automatic_reference by default. Would there be any downside to always using automatic for call argument collection? (Users should be able to override by doing a manual py::cast(value, policy) when assigning an argument.)

Right -- that makes sense. Is the restriction to rvalues unavoidable? If it's easy to also support your example

auto a = "arg"_a=MyArg(1, 2, 3); f(a);

with minor changes then that would be nice. Switching to py::object for the underlying argument storage in this case is fine to me. The only reason for the current state of affairs is because of the way in which def() arguments were processed previously.

Regarding value policies: this unlikely to be a serious concern for default arguments constructed at module load time, and it's always safe to make a copy (which is what will happen for const T& instances that have not been previously observed by pybind11).

Switching to py::object would be the best case. There would be no restrictions on py::arg_t then. As a bonus, the template code would also be simplified since py::arg_t would no longer be a template class (it would only have a template constructor). py::arg_t should create it's py::object using the automatic policy which looks like the only reasonable choice.

There is just one small inconsistency that this would introduce: Keyword arguments would then be passed to function calls with the automatic policy, while positional arguments would use automatic_reference (which is the function call default). I don't think this a big issue, but maybe you have some thoughts.

I don't think it is a big issue. In any case, the cast to a py::object could also be done manually with any kind of policy, which is then assigned to py:arg_t.

wjakob · 2016-09-05T00:57:26Z

I think this one is almost ready to go. I added a few comments. The only major gotcha is the pointer-vs-value storage in arg_t, can you take a look at that? Maybe I am just missing something though.

jagerman · 2016-09-05T19:23:12Z

include/pybind11/common.h

+/// Utility types for metaprogramming
+template <bool...> struct bools { };
+template <typename...> struct always_true : std::true_type { };
+template <typename...> struct always_false : std::false_type { };


Could these be written as: template <typename...> using always_true = std::true_type; and likewise for always_false?

Yes, that was the initial implementation, however GCC 4.8 had an issue with it in one spot (which is weird since even MSVC compiled it correctly). I'll need to investigate a bit more to see what's up.

wjakob · 2016-09-06T03:01:39Z

Let me know when you think this is ready to merge.

A Python function can be called with the syntax: ```python foo(a1, a2, *args, ka=1, kb=2, **kwargs) ``` This commit adds support for the equivalent syntax in C++: ```c++ foo(a1, a2, *args, "ka"_a=1, "kb"_a=2, **kwargs) ``` In addition, generalized unpacking is implemented, as per PEP 448, which allows calls with multiple * and ** unpacking: ```python bar(*args1, 99, *args2, 101, **kwargs1, kz=200, **kwargs2) ``` and ```c++ bar(*args1, 99, *args2, 101, **kwargs1, "kz"_a=200, **kwargs2) ```

Replicates Python API including keyword arguments.

The variadic handle::operator() offers the same functionality as well as mixed positional, keyword, * and ** arguments. The tests are also superseded by the ones in `test_callbacks`.

MSVC fails to compile if the constructor is defined out-of-line. The error states that it cannot deduce the type of the default template parameter which is used for SFINAE.

With this change arg_t is no longer a template, but it must remain so for backward compatibility. Thus, a non-template arg_v is introduced, while a dummy template alias arg_t is there to keep old code from breaking. This can be remove in the next major release. The implementation of arg_v also needed to be placed a little earlier in the headers because it's not a template any more and unpacking_collector needs more than a forward declaration.

dean0x7d · 2016-09-06T15:02:29Z

OK, if everything looks good with the changes to py::arg this should be ready to go. See the last commit message for py::arg details.

For the docs, I added a quick section for py::print, but everything is still in Advanced. I think the reorganization is better left for a separate PR. Better docs for the type wrappers can also be added at that point.

I addressed all the comments and squashed the smaller changes (remove dict.get, improved comments) with their respective commits to keep the git history relatively tidy.

wjakob · 2016-09-06T15:12:56Z

include/pybind11/cast.h

+template <typename T>
+arg_v arg::operator=(T &&value) const { return {name, std::forward<T>(value)}; }
+
+/// Alias for backward compatibility -- to be remove in version 2.0


very compact 👍

wjakob · 2016-09-06T15:31:01Z

Awesome, merged! 🎉

wjakob reviewed Aug 30, 2016
View reviewed changes

dean0x7d force-pushed the keywords branch from 7a8596a to 1f13b0c Compare September 4, 2016 22:55

dean0x7d changed the title ~~WIP: Keyword arguments and generalized unpacking for C++ API~~ Keyword arguments and generalized unpacking for C++ API Sep 4, 2016

wjakob reviewed Sep 5, 2016
View reviewed changes

dean0x7d mentioned this pull request Sep 5, 2016

Allow arbitrary class_ template option ordering #385

Merged

jagerman reviewed Sep 5, 2016
View reviewed changes

jagerman mentioned this pull request Sep 5, 2016

Iterators efficiency and instance tracking #376

Open

dean0x7d added 8 commits September 6, 2016 14:39

Make arg_t hold a pointer instead of a copy of the value

317524f

Add py::print() function

67990d9

Replicates Python API including keyword arguments.

Add py::str::format() method

66aa272

Add py::dict() keyword constructor

15a112f

Document calling function with keyword arguments from C++

625bd48

Remove superseded handle::operator() overloads

16db1bf

The variadic handle::operator() offers the same functionality as well as mixed positional, keyword, * and ** arguments. The tests are also superseded by the ones in `test_callbacks`.

Workaround for py::dict() constructor on MSVC

56e86ed

MSVC fails to compile if the constructor is defined out-of-line. The error states that it cannot deduce the type of the default template parameter which is used for SFINAE.

dean0x7d added 2 commits September 6, 2016 16:41

Apply make_caster and intrinsic_t aliases everywhere

8fe13b8

dean0x7d force-pushed the keywords branch from 1f13b0c to 60b2680 Compare September 6, 2016 14:50

wjakob reviewed Sep 6, 2016
View reviewed changes

wjakob merged commit a3dbdc6 into pybind:master Sep 6, 2016

dean0x7d deleted the keywords branch September 6, 2016 23:01

jagerman mentioned this pull request Sep 8, 2016

Small template simplifications #401

Merged

dean0x7d mentioned this pull request Oct 15, 2016

Make a v2.0 release? #442

Closed

7 tasks

rwgk mentioned this pull request Feb 9, 2023

FWD pybind11 google/pybind11clif#372

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keyword arguments and generalized unpacking for C++ API #372

Keyword arguments and generalized unpacking for C++ API #372

dean0x7d commented Aug 29, 2016 •

edited

Loading

wjakob Aug 30, 2016

wjakob Sep 5, 2016

wjakob commented Aug 30, 2016

dean0x7d commented Aug 30, 2016

aldanor commented Aug 30, 2016

dean0x7d commented Sep 4, 2016 •

edited

Loading

wjakob Sep 5, 2016

dean0x7d Sep 5, 2016 •

edited

Loading

wjakob Sep 5, 2016

dean0x7d Sep 5, 2016

wjakob Sep 6, 2016

wjakob commented Sep 5, 2016

jagerman Sep 5, 2016

dean0x7d Sep 5, 2016

wjakob commented Sep 6, 2016

dean0x7d commented Sep 6, 2016

wjakob Sep 6, 2016

wjakob commented Sep 6, 2016

Keyword arguments and generalized unpacking for C++ API #372

Keyword arguments and generalized unpacking for C++ API #372

Conversation

dean0x7d commented Aug 29, 2016 • edited Loading

General idea

Impact on compile time

Python version compatibility

Applications

Feedback?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjakob commented Aug 30, 2016

dean0x7d commented Aug 30, 2016

aldanor commented Aug 30, 2016

dean0x7d commented Sep 4, 2016 • edited Loading

Choose a reason for hiding this comment

dean0x7d Sep 5, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjakob commented Sep 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjakob commented Sep 6, 2016

dean0x7d commented Sep 6, 2016

Choose a reason for hiding this comment

wjakob commented Sep 6, 2016

dean0x7d commented Aug 29, 2016 •

edited

Loading

dean0x7d commented Sep 4, 2016 •

edited

Loading

dean0x7d Sep 5, 2016 •

edited

Loading