Merge remote-tracking branch 'upstream/master' into relax-class-arguments

Author: jagerman

docs/advanced.rst

@@ -552,15 +552,150 @@ and the Python ``list``, ``set`` and ``dict`` data structures are automatically
 enabled. The types ``std::pair<>`` and ``std::tuple<>`` are already supported
 out of the box with just the core :file:`pybind11/pybind11.h` header.
 
+The major downside of these implicit conversions is that containers must be
+converted (i.e. copied) on every Python->C++ and C++->Python transition, which
+can have implications on the program semantics and performance. Please read the
+next sections for more details and alternative approaches that avoid this.
+
 .. note::
 
-    Arbitrary nesting of any of these types is supported.
+    Arbitrary nesting of any of these types is possible.
 
 .. seealso::
 
     The file :file:`tests/test_python_types.cpp` contains a complete
     example that demonstrates how to pass STL data types in more detail.
 
+.. _opaque:
+
+Treating STL data structures as opaque objects
+==============================================
+
+pybind11 heavily relies on a template matching mechanism to convert parameters
+and return values that are constructed from STL data types such as vectors,
+linked lists, hash tables, etc. This even works in a recursive manner, for
+instance to deal with lists of hash maps of pairs of elementary and custom
+types, etc.
+
+However, a fundamental limitation of this approach is that internal conversions
+between Python and C++ types involve a copy operation that prevents
+pass-by-reference semantics. What does this mean?
+
+Suppose we bind the following function
+
+.. code-block:: cpp
+
+    void append_1(std::vector<int> &v) {
+       v.push_back(1);
+    }
+
+and call it from Python, the following happens:
+
+.. code-block:: pycon
+
+   >>> v = [5, 6]
+   >>> append_1(v)
+   >>> print(v)
+   [5, 6]
+
+As you can see, when passing STL data structures by reference, modifications
+are not propagated back the Python side. A similar situation arises when
+exposing STL data structures using the ``def_readwrite`` or ``def_readonly``
+functions:
+
+.. code-block:: cpp
+
+    /* ... definition ... */
+
+    class MyClass {
+        std::vector<int> contents;
+    };
+
+    /* ... binding code ... */
+
+    py::class_<MyClass>(m, "MyClass")
+        .def(py::init<>)
+        .def_readwrite("contents", &MyClass::contents);
+
+In this case, properties can be read and written in their entirety. However, an
+``append`` operaton involving such a list type has no effect:
+
+.. code-block:: pycon
+
+   >>> m = MyClass()
+   >>> m.contents = [5, 6]
+   >>> print(m.contents)
+   [5, 6]
+   >>> m.contents.append(7)
+   >>> print(m.contents)
+   [5, 6]
+
+Finally, the involved copy operations can be costly when dealing with very
+large lists. To deal with all of the above situations, pybind11 provides a
+macro named ``PYBIND11_MAKE_OPAQUE(T)`` that disables the template-based
+conversion machinery of types, thus rendering them *opaque*. The contents of
+opaque objects are never inspected or extracted, hence they *can* be passed by
+reference. For instance, to turn ``std::vector<int>`` into an opaque type, add
+the declaration
+
+.. code-block:: cpp
+
+    PYBIND11_MAKE_OPAQUE(std::vector<int>);
+
+before any binding code (e.g. invocations to ``class_::def()``, etc.). This
+macro must be specified at the top level (and outside of any namespaces), since
+it instantiates a partial template overload. If your binding code consists of
+multiple compilation units, it must be present in every file preceding any
+usage of ``std::vector<int>``. Opaque types must also have a corresponding
+``class_`` declaration to associate them with a name in Python, and to define a
+set of available operations, e.g.:
+
+.. code-block:: cpp
+
+    py::class_<std::vector<int>>(m, "IntVector")
+        .def(py::init<>())
+        .def("clear", &std::vector<int>::clear)
+        .def("pop_back", &std::vector<int>::pop_back)
+        .def("__len__", [](const std::vector<int> &v) { return v.size(); })
+        .def("__iter__", [](std::vector<int> &v) {
+           return py::make_iterator(v.begin(), v.end());
+        }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */
+        // ....
+
+The ability to expose STL containers as native Python objects is a fairly
+common request, hence pybind11 also provides an optional header file named
+:file:`pybind11/stl_bind.h` that does exactly this. The mapped containers try
+to match the behavior of their native Python counterparts as much as possible.
+
+The following example showcases usage of :file:`pybind11/stl_bind.h`:
+
+.. code-block:: cpp
+
+    // Don't forget this
+    #include <pybind11/stl_bind.h>
+
+    PYBIND11_MAKE_OPAQUE(std::vector<int>);
+    PYBIND11_MAKE_OPAQUE(std::map<std::string, double>);
+
+    // ...
+
+    // later in binding code:
+    py::bind_vector<std::vector<int>>(m, "VectorInt");
+    py::bind_map<std::map<std::string, double>>(m, "MapStringDouble");
+
+Please take a look at the :ref:`macro_notes` before using the
+``PYBIND11_MAKE_OPAQUE`` macro.
+
+.. seealso::
+
+    The file :file:`tests/test_opaque_types.cpp` contains a complete
+    example that demonstrates how to create and expose opaque types using
+    pybind11 in more detail.
+
+    The file :file:`tests/test_stl_binders.cpp` shows how to use the
+    convenience STL container wrappers.
+
+
 Binding sequence data types, iterators, the slicing protocol, etc.
 ==================================================================
 
@@ -1094,108 +1229,6 @@ section.
     The ``py::exception`` wrapper for creating custom exceptions cannot (yet)
     be used as a base type.
 
-.. _opaque:
-
-Treating STL data structures as opaque objects
-==============================================
-
-pybind11 heavily relies on a template matching mechanism to convert parameters
-and return values that are constructed from STL data types such as vectors,
-linked lists, hash tables, etc. This even works in a recursive manner, for
-instance to deal with lists of hash maps of pairs of elementary and custom
-types, etc.
-
-However, a fundamental limitation of this approach is that internal conversions
-between Python and C++ types involve a copy operation that prevents
-pass-by-reference semantics. What does this mean?
-
-Suppose we bind the following function
-
-.. code-block:: cpp
-
-    void append_1(std::vector<int> &v) {
-       v.push_back(1);
-    }
-
-and call it from Python, the following happens:
-
-.. code-block:: pycon
-
-   >>> v = [5, 6]
-   >>> append_1(v)
-   >>> print(v)
-   [5, 6]
-
-As you can see, when passing STL data structures by reference, modifications
-are not propagated back the Python side. A similar situation arises when
-exposing STL data structures using the ``def_readwrite`` or ``def_readonly``
-functions:
-
-.. code-block:: cpp
-
-    /* ... definition ... */
-
-    class MyClass {
-        std::vector<int> contents;
-    };
-
-    /* ... binding code ... */
-
-    py::class_<MyClass>(m, "MyClass")
-        .def(py::init<>)
-        .def_readwrite("contents", &MyClass::contents);
-
-In this case, properties can be read and written in their entirety. However, an
-``append`` operaton involving such a list type has no effect:
-
-.. code-block:: pycon
-
-   >>> m = MyClass()
-   >>> m.contents = [5, 6]
-   >>> print(m.contents)
-   [5, 6]
-   >>> m.contents.append(7)
-   >>> print(m.contents)
-   [5, 6]
-
-To deal with both of the above situations, pybind11 provides a macro named
-``PYBIND11_MAKE_OPAQUE(T)`` that disables the template-based conversion
-machinery of types, thus rendering them *opaque*. The contents of opaque
-objects are never inspected or extracted, hence they can be passed by
-reference. For instance, to turn ``std::vector<int>`` into an opaque type, add
-the declaration
-
-.. code-block:: cpp
-
-    PYBIND11_MAKE_OPAQUE(std::vector<int>);
-
-before any binding code (e.g. invocations to ``class_::def()``, etc.). This
-macro must be specified at the top level, since instantiates a partial template
-overload. If your binding code consists of multiple compilation units, it must
-be present in every file preceding any usage of ``std::vector<int>``. Opaque
-types must also have a corresponding ``class_`` declaration to associate them
-with a name in Python, and to define a set of available operations:
-
-.. code-block:: cpp
-
-    py::class_<std::vector<int>>(m, "IntVector")
-        .def(py::init<>())
-        .def("clear", &std::vector<int>::clear)
-        .def("pop_back", &std::vector<int>::pop_back)
-        .def("__len__", [](const std::vector<int> &v) { return v.size(); })
-        .def("__iter__", [](std::vector<int> &v) {
-           return py::make_iterator(v.begin(), v.end());
-        }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */
-        // ....
-
-Please take a look at the :ref:`macro_notes` before using this feature.
-
-.. seealso::
-
-    The file :file:`tests/test_opaque_types.cpp` contains a complete
-    example that demonstrates how to create and expose opaque types using
-    pybind11 in more detail.
-
 .. _eigen:
 
 Transparent conversion of dense and sparse Eigen data types

docs/basics.rst

@@ -178,16 +178,16 @@ The keyword names also appear in the function signatures within the documentatio
 A shorter notation for named arguments is also available:
 
 .. code-block:: cpp
-    
+
     // regular notation
     m.def("add1", &add, py::arg("i"), py::arg("j"));
     // shorthand
     using namespace pybind11::literals;
     m.def("add2", &add, "i"_a, "j"_a);
 
-The :var:`_a` suffix forms a C++11 literal which is equivalent to :class:`arg`. 
-Note that the literal operator must first be made visible with the directive 
-``using namespace pybind11::literals``. This does not bring in anything else 
+The :var:`_a` suffix forms a C++11 literal which is equivalent to :class:`arg`.
+Note that the literal operator must first be made visible with the directive
+``using namespace pybind11::literals``. This does not bring in anything else
 from the ``pybind11`` namespace except for literals.
 
 .. _default_args:
@@ -229,7 +229,7 @@ The default values also appear within the documentation.
 The shorthand notation is also available for default arguments:
 
 .. code-block:: cpp
-    
+
     // regular notation
     m.def("add1", &add, py::arg("i") = 1, py::arg("j") = 2);
     // shorthand

docs/compiling.rst

@@ -26,7 +26,7 @@ Building with cppimport
 Building with CMake
 ===================
 
-For C++ codebases that have an existing CMake-based build system, a Python 
+For C++ codebases that have an existing CMake-based build system, a Python
 extension module can be created with just a few lines of code:
 
 .. code-block:: cmake
@@ -37,14 +37,14 @@ extension module can be created with just a few lines of code:
     add_subdirectory(pybind11)
     pybind11_add_module(example example.cpp)
 
-This assumes that the pybind11 repository is located in a subdirectory named 
+This assumes that the pybind11 repository is located in a subdirectory named
 :file:`pybind11` and that the code is located in a file named :file:`example.cpp`.
 The CMake command ``add_subdirectory`` will import a function with the signature
 ``pybind11_add_module(<name> source1 [source2 ...])``. It will take care of all
 the details needed to build a Python extension module on any platform.
 
-The target Python version can be selected by setting the ``PYBIND11_PYTHON_VERSION`` 
-variable before adding the pybind11 subdirectory. Alternatively, an exact Python 
+The target Python version can be selected by setting the ``PYBIND11_PYTHON_VERSION``
+variable before adding the pybind11 subdirectory. Alternatively, an exact Python
 installation can be specified by setting ``PYTHON_EXECUTABLE``.
 
 A working sample project, including a way to invoke CMake from :file:`setup.py` for

docs/faq.rst

@@ -140,7 +140,7 @@ the included test suite contains the following symbol:
 
     .. code-block:: none
 
-        _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_​
+        _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_
 
 .. only:: not html
 

include/pybind11/cast.h

@@ -683,7 +683,7 @@ template <typename... Tuple> class type_caster<std::tuple<Tuple...>> {
     static PYBIND11_DESCR element_names() {
         return detail::concat(type_caster<typename intrinsic_type<Tuple>::type>::name()...);
     }
-    
+
     static PYBIND11_DESCR name() {
         return type_descr(_("Tuple[") + element_names() + _("]"));
     }

include/pybind11/pybind11.h

@@ -1085,7 +1085,7 @@ template <typename Type> class enum_ : public class_<Type> {
             def("__ne__", [](const Type &value, UnderlyingType value2) { return (UnderlyingType) value != value2; });
         }
         def("__hash__", [](const Type &value) { return (UnderlyingType) value; });
-        // Pickling and unpickling -- needed for use with the 'multiprocessing' module 
+        // Pickling and unpickling -- needed for use with the 'multiprocessing' module
         def("__getstate__", [](const Type &value) { return pybind11::make_tuple((UnderlyingType) value); });
         def("__setstate__", [](Type &p, tuple t) { new (&p) Type((Type) t[0].cast<UnderlyingType>()); });
         m_entries = entries;
@@ -1169,7 +1169,7 @@ PYBIND11_NOINLINE inline void keep_alive_impl(int Nurse, int Patient, handle arg
     keep_alive_impl(nurse, patient);
 }
 
-template <typename Iterator, typename Sentinel, bool KeyIterator = false>
+template <typename Iterator, typename Sentinel, bool KeyIterator, typename... Extra>
 struct iterator_state {
     Iterator it;
     Sentinel end;
@@ -1185,10 +1185,10 @@ template <typename Iterator,
           typename ValueType = decltype(*std::declval<Iterator>()),
           typename... Extra>
 iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) {
-    typedef detail::iterator_state<Iterator, Sentinel> state;
+    typedef detail::iterator_state<Iterator, Sentinel, false, Extra...> state;
 
     if (!detail::get_type_info(typeid(state))) {
-        class_<state>(handle(), "")
+        class_<state>(handle(), "iterator")
             .def("__iter__", [](state &s) -> state& { return s; })
             .def("__next__", [](state &s) -> ValueType {
                 if (!s.first)
@@ -1203,15 +1203,16 @@ iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) {
 
     return (iterator) cast(state { first, last, true });
 }
+
 template <typename Iterator,
           typename Sentinel,
           typename KeyType = decltype((*std::declval<Iterator>()).first),
           typename... Extra>
 iterator make_key_iterator(Iterator first, Sentinel last, Extra &&... extra) {
-    typedef detail::iterator_state<Iterator, Sentinel, true> state;
+    typedef detail::iterator_state<Iterator, Sentinel, true, Extra...> state;
 
     if (!detail::get_type_info(typeid(state))) {
-        class_<state>(handle(), "")
+        class_<state>(handle(), "iterator")
             .def("__iter__", [](state &s) -> state& { return s; })
             .def("__next__", [](state &s) -> KeyType {
                 if (!s.first)