Add basic support for tag-based static polymorphism (pybind#1326)

* Add basic support for tag-based static polymorphism

Sometimes it is possible to look at a C++ object and know what its dynamic type is,
even if it doesn't use C++ polymorphism, because instances of the object and its
subclasses conform to some other mechanism for being self-describing; for example,
perhaps there's an enumerated "tag" or "kind" member in the base class that's always
set to an indication of the correct type. This might be done for performance reasons,
or to permit most-derived types to be trivially copyable. One of the most widely-known
examples is in LLVM: https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html

This PR permits pybind11 to be informed of such conventions via a new specializable
detail::polymorphic_type_hook<> template, which generalizes the previous logic for
determining the runtime type of an object based on C++ RTTI. Implementors provide
a way to map from a base class object to a const std::type_info* for the dynamic
type; pybind11 then uses this to ensure that casting a Base* to Python creates a
Python object that knows it's wrapping the appropriate sort of Derived.

There are a number of restrictions with this tag-based static polymorphism support
compared to pybind11's existing support for built-in C++ polymorphism:

- there is no support for this-pointer adjustment, so only single inheritance is permitted
- there is no way to make C++ code call new Python-provided subclasses
- when binding C++ classes that redefine a method in a subclass, the .def() must be
  repeated in the binding for Python to know about the update

But these are not much of an issue in practice in many cases, the impact on the
complexity of pybind11's innards is minimal and localized, and the support for
automatic downcasting improves usability a great deal.

Author: oremanj

docs/advanced/classes.rst

@@ -999,3 +999,86 @@ described trampoline:
     requires a more explicit function binding in the form of
     ``.def("foo", static_cast<int (A::*)() const>(&Publicist::foo));``
     where ``int (A::*)() const`` is the type of ``A::foo``.
+
+Custom automatic downcasters
+============================
+
+As explained in :ref:`inheritance`, pybind11 comes with built-in
+understanding of the dynamic type of polymorphic objects in C++; that
+is, returning a Pet to Python produces a Python object that knows it's
+wrapping a Dog, if Pet has virtual methods and pybind11 knows about
+Dog and this Pet is in fact a Dog. Sometimes, you might want to
+provide this automatic downcasting behavior when creating bindings for
+a class hierarchy that does not use standard C++ polymorphism, such as
+LLVM [#f4]_. As long as there's some way to determine at runtime
+whether a downcast is safe, you can proceed by specializing the
+``pybind11::polymorphic_type_hook`` template:
+
+.. code-block:: cpp
+
+    enum class PetKind { Cat, Dog, Zebra };
+    struct Pet {   // Not polymorphic: has no virtual methods
+        const PetKind kind;
+        int age = 0;
+      protected:
+        Pet(PetKind _kind) : kind(_kind) {}
+    };
+    struct Dog : Pet {
+        Dog() : Pet(PetKind::Dog) {}
+        std::string sound = "woof!";
+        std::string bark() const { return sound; }
+    };
+
+    namespace pybind11 {
+        template<> struct polymorphic_type_hook<Pet> {
+            static const void *get(const Pet *src, const std::type_info*& type) {
+                // note that src may be nullptr
+                if (src && src->kind == PetKind::Dog) {
+                    type = &typeid(Dog);
+                    return static_cast<const Dog*>(src);
+                }
+                return src;
+            }
+        };
+    } // namespace pybind11
+
+When pybind11 wants to convert a C++ pointer of type ``Base*`` to a
+Python object, it calls ``polymorphic_type_hook<Base>::get()`` to
+determine if a downcast is possible. The ``get()`` function should use
+whatever runtime information is available to determine if its ``src``
+parameter is in fact an instance of some class ``Derived`` that
+inherits from ``Base``. If it finds such a ``Derived``, it sets ``type
+= &typeid(Derived)`` and returns a pointer to the ``Derived`` object
+that contains ``src``. Otherwise, it just returns ``src``, leaving
+``type`` at its default value of nullptr. If you set ``type`` to a
+type that pybind11 doesn't know about, no downcasting will occur, and
+the original ``src`` pointer will be used with its static type
+``Base*``.
+
+It is critical that the returned pointer and ``type`` argument of
+``get()`` agree with each other: if ``type`` is set to something
+non-null, the returned pointer must point to the start of an object
+whose type is ``type``. If the hierarchy being exposed uses only
+single inheritance, a simple ``return src;`` will achieve this just
+fine, but in the general case, you must cast ``src`` to the
+appropriate derived-class pointer (e.g. using
+``static_cast<Derived>(src)``) before allowing it to be returned as a
+``void*``.
+
+.. [#f4] https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html
+
+.. note::
+
+    pybind11's standard support for downcasting objects whose types
+    have virtual methods is implemented using
+    ``polymorphic_type_hook`` too, using the standard C++ ability to
+    determine the most-derived type of a polymorphic object using
+    ``typeid()`` and to cast a base pointer to that most-derived type
+    (even if you don't know what it is) using ``dynamic_cast<void*>``.
+
+.. seealso::
+
+    The file :file:`tests/test_tagbased_polymorphic.cpp` contains a
+    more complete example, including a demonstration of how to provide
+    automatic downcasting for an entire class hierarchy without
+    writing one get() function for each class.

docs/classes.rst

@@ -228,8 +228,8 @@ just brings them on par.
 
 .. _inheritance:
 
-Inheritance and automatic upcasting
-===================================
+Inheritance and automatic downcasting
+=====================================
 
 Suppose now that the example consists of two data structures with an
 inheritance relationship:
@@ -298,7 +298,7 @@ inheritance relationship. This is reflected in Python:
 
     >>> p = example.pet_store()
     >>> type(p)  # `Dog` instance behind `Pet` pointer
-    Pet          # no pointer upcasting for regular non-polymorphic types
+    Pet          # no pointer downcasting for regular non-polymorphic types
     >>> p.bark()
     AttributeError: 'Pet' object has no attribute 'bark'
 
@@ -330,11 +330,11 @@ will automatically recognize this:
 
     >>> p = example.pet_store2()
     >>> type(p)
-    PolymorphicDog  # automatically upcast
+    PolymorphicDog  # automatically downcast
     >>> p.bark()
     u'woof!'
 
-Given a pointer to a polymorphic base, pybind11 performs automatic upcasting
+Given a pointer to a polymorphic base, pybind11 performs automatic downcasting
 to the actual derived type. Note that this goes beyond the usual situation in
 C++: we don't just get access to the virtual functions of the base, we get the
 concrete derived type including functions and attributes that the base type may

include/pybind11/cast.h

@@ -774,9 +774,45 @@ template <typename T1, typename T2> struct is_copy_constructible<std::pair<T1, T
     : all_of<is_copy_constructible<T1>, is_copy_constructible<T2>> {};
 #endif
 
+NAMESPACE_END(detail)
+
+// polymorphic_type_hook<itype>::get(src, tinfo) determines whether the object pointed
+// to by `src` actually is an instance of some class derived from `itype`.
+// If so, it sets `tinfo` to point to the std::type_info representing that derived
+// type, and returns a pointer to the start of the most-derived object of that type
+// (in which `src` is a subobject; this will be the same address as `src` in most
+// single inheritance cases). If not, or if `src` is nullptr, it simply returns `src`
+// and leaves `tinfo` at its default value of nullptr.
+//
+// The default polymorphic_type_hook just returns src. A specialization for polymorphic
+// types determines the runtime type of the passed object and adjusts the this-pointer
+// appropriately via dynamic_cast<void*>. This is what enables a C++ Animal* to appear
+// to Python as a Dog (if Dog inherits from Animal, Animal is polymorphic, Dog is
+// registered with pybind11, and this Animal is in fact a Dog).
+//
+// You may specialize polymorphic_type_hook yourself for types that want to appear
+// polymorphic to Python but do not use C++ RTTI. (This is a not uncommon pattern
+// in performance-sensitive applications, used most notably in LLVM.)
+template <typename itype, typename SFINAE = void>
+struct polymorphic_type_hook
+{
+    static const void *get(const itype *src, const std::type_info*&) { return src; }
+};
+template <typename itype>
+struct polymorphic_type_hook<itype, detail::enable_if_t<std::is_polymorphic<itype>::value>>
+{
+    static const void *get(const itype *src, const std::type_info*& type) {
+        type = src ? &typeid(*src) : nullptr;
+        return dynamic_cast<const void*>(src);
+    }
+};
+
+NAMESPACE_BEGIN(detail)
+
 /// Generic type caster for objects stored on the heap
 template <typename type> class type_caster_base : public type_caster_generic {
     using itype = intrinsic_t<type>;
+
 public:
     static constexpr auto name = _<type>();
 
@@ -793,32 +829,28 @@ template <typename type> class type_caster_base : public type_caster_generic {
         return cast(&src, return_value_policy::move, parent);
     }
 
-    // Returns a (pointer, type_info) pair taking care of necessary RTTI type lookup for a
-    // polymorphic type.  If the instance isn't derived, returns the non-RTTI base version.
-    template <typename T = itype, enable_if_t<std::is_polymorphic<T>::value, int> = 0>
+    // Returns a (pointer, type_info) pair taking care of necessary type lookup for a
+    // polymorphic type (using RTTI by default, but can be overridden by specializing
+    // polymorphic_type_hook). If the instance isn't derived, returns the base version.
     static std::pair<const void *, const type_info *> src_and_type(const itype *src) {
-        const void *vsrc = src;
         auto &cast_type = typeid(itype);
         const std::type_info *instance_type = nullptr;
-        if (vsrc) {
-            instance_type = &typeid(*src);
-            if (!same_type(cast_type, *instance_type)) {
-                // This is a base pointer to a derived type; if it is a pybind11-registered type, we
-                // can get the correct derived pointer (which may be != base pointer) by a
-                // dynamic_cast to most derived type:
-                if (auto *tpi = get_type_info(*instance_type))
-                    return {dynamic_cast<const void *>(src), const_cast<const type_info *>(tpi)};
-            }
+        const void *vsrc = polymorphic_type_hook<itype>::get(src, instance_type);
+        if (instance_type && !same_type(cast_type, *instance_type)) {
+            // This is a base pointer to a derived type. If the derived type is registered
+            // with pybind11, we want to make the full derived object available.
+            // In the typical case where itype is polymorphic, we get the correct
+            // derived pointer (which may be != base pointer) by a dynamic_cast to
+            // most derived type. If itype is not polymorphic, we won't get here
+            // except via a user-provided specialization of polymorphic_type_hook,
+            // and the user has promised that no this-pointer adjustment is
+            // required in that case, so it's OK to use static_cast.
+            if (const auto *tpi = get_type_info(*instance_type))
+                return {vsrc, tpi};
         }
         // Otherwise we have either a nullptr, an `itype` pointer, or an unknown derived pointer, so
         // don't do a cast
-        return type_caster_generic::src_and_type(vsrc, cast_type, instance_type);
-    }
-
-    // Non-polymorphic type, so no dynamic casting; just call the generic version directly
-    template <typename T = itype, enable_if_t<!std::is_polymorphic<T>::value, int> = 0>
-    static std::pair<const void *, const type_info *> src_and_type(const itype *src) {
-        return type_caster_generic::src_and_type(src, typeid(itype));
+        return type_caster_generic::src_and_type(src, cast_type, instance_type);
     }
 
     static handle cast(const itype *src, return_value_policy policy, handle parent) {

tests/CMakeLists.txt

@@ -57,6 +57,7 @@ set(PYBIND11_TEST_FILES
   test_smart_ptr.cpp
   test_stl.cpp
   test_stl_binders.cpp
+  test_tagbased_polymorphic.cpp
   test_virtual_functions.cpp
 )
 

tests/test_tagbased_polymorphic.cpp

@@ -0,0 +1,136 @@
+/*
+    tests/test_tagbased_polymorphic.cpp -- test of polymorphic_type_hook
+
+    Copyright (c) 2018 Hudson River Trading LLC <[email protected]>
+
+    All rights reserved. Use of this source code is governed by a
+    BSD-style license that can be found in the LICENSE file.
+*/
+
+#include "pybind11_tests.h"
+#include <pybind11/stl.h>
+
+struct Animal
+{
+    enum class Kind {
+        Unknown = 0,
+        Dog = 100, Labrador, Chihuahua, LastDog = 199,
+        Cat = 200, Panther, LastCat = 299
+    };
+    static const std::type_info* type_of_kind(Kind kind);
+    static std::string name_of_kind(Kind kind);
+
+    const Kind kind;
+    const std::string name;
+
+  protected:
+    Animal(const std::string& _name, Kind _kind)
+        : kind(_kind), name(_name)
+    {}
+};
+
+struct Dog : Animal
+{
+    Dog(const std::string& _name, Kind _kind = Kind::Dog) : Animal(_name, _kind) {}
+    std::string bark() const { return name_of_kind(kind) + " " + name + " goes " + sound; }
+    std::string sound = "WOOF!";
+};
+
+struct Labrador : Dog
+{
+    Labrador(const std::string& _name, int _excitement = 9001)
+        : Dog(_name, Kind::Labrador), excitement(_excitement) {}
+    int excitement;
+};
+
+struct Chihuahua : Dog
+{
+    Chihuahua(const std::string& _name) : Dog(_name, Kind::Chihuahua) { sound = "iyiyiyiyiyi"; }
+    std::string bark() const { return Dog::bark() + " and runs in circles"; }
+};
+
+struct Cat : Animal
+{
+    Cat(const std::string& _name, Kind _kind = Kind::Cat) : Animal(_name, _kind) {}
+    std::string purr() const { return "mrowr"; }
+};
+
+struct Panther : Cat
+{
+    Panther(const std::string& _name) : Cat(_name, Kind::Panther) {}
+    std::string purr() const { return "mrrrRRRRRR"; }
+};
+
+std::vector<std::unique_ptr<Animal>> create_zoo()
+{
+    std::vector<std::unique_ptr<Animal>> ret;
+    ret.emplace_back(new Labrador("Fido", 15000));
+
+    // simulate some new type of Dog that the Python bindings
+    // haven't been updated for; it should still be considered
+    // a Dog, not just an Animal.
+    ret.emplace_back(new Dog("Ginger", Dog::Kind(150)));
+
+    ret.emplace_back(new Chihuahua("Hertzl"));
+    ret.emplace_back(new Cat("Tiger", Cat::Kind::Cat));
+    ret.emplace_back(new Panther("Leo"));
+    return ret;
+}
+
+const std::type_info* Animal::type_of_kind(Kind kind)
+{
+    switch (kind) {
+        case Kind::Unknown: break;
+
+        case Kind::Dog: break;
+        case Kind::Labrador: return &typeid(Labrador);
+        case Kind::Chihuahua: return &typeid(Chihuahua);
+        case Kind::LastDog: break;
+
+        case Kind::Cat: break;
+        case Kind::Panther: return &typeid(Panther);
+        case Kind::LastCat: break;
+    }
+
+    if (kind >= Kind::Dog && kind <= Kind::LastDog) return &typeid(Dog);
+    if (kind >= Kind::Cat && kind <= Kind::LastCat) return &typeid(Cat);
+    return nullptr;
+}
+
+std::string Animal::name_of_kind(Kind kind)
+{
+    std::string raw_name = type_of_kind(kind)->name();
+    py::detail::clean_type_id(raw_name);
+    return raw_name;
+}
+
+namespace pybind11 {
+    template <typename itype>
+    struct polymorphic_type_hook<itype, detail::enable_if_t<std::is_base_of<Animal, itype>::value>>
+    {
+        static const void *get(const itype *src, const std::type_info*& type)
+        { type = src ? Animal::type_of_kind(src->kind) : nullptr; return src; }
+    };
+}
+
+TEST_SUBMODULE(tagbased_polymorphic, m) {
+    py::class_<Animal>(m, "Animal")
+        .def_readonly("name", &Animal::name);
+    py::class_<Dog, Animal>(m, "Dog")
+        .def(py::init<std::string>())
+        .def_readwrite("sound", &Dog::sound)
+        .def("bark", &Dog::bark);
+    py::class_<Labrador, Dog>(m, "Labrador")
+        .def(py::init<std::string, int>(), "name"_a, "excitement"_a = 9001)
+        .def_readwrite("excitement", &Labrador::excitement);
+    py::class_<Chihuahua, Dog>(m, "Chihuahua")
+        .def(py::init<std::string>())
+        .def("bark", &Chihuahua::bark);
+    py::class_<Cat, Animal>(m, "Cat")
+        .def(py::init<std::string>())
+        .def("purr", &Cat::purr);
+    py::class_<Panther, Cat>(m, "Panther")
+        .def(py::init<std::string>())
+        .def("purr", &Panther::purr);
+    m.def("create_zoo", &create_zoo);
+};

tests/test_tagbased_polymorphic.py

@@ -0,0 +1,20 @@
+from pybind11_tests import tagbased_polymorphic as m
+
+
+def test_downcast():
+    zoo = m.create_zoo()
+    assert [type(animal) for animal in zoo] == [
+        m.Labrador, m.Dog, m.Chihuahua, m.Cat, m.Panther
+    ]
+    assert [animal.name for animal in zoo] == [
+        "Fido", "Ginger", "Hertzl", "Tiger", "Leo"
+    ]
+    zoo[1].sound = "woooooo"
+    assert [dog.bark() for dog in zoo[:3]] == [
+        "Labrador Fido goes WOOF!",
+        "Dog Ginger goes woooooo",
+        "Chihuahua Hertzl goes iyiyiyiyiyi and runs in circles"
+    ]
+    assert [cat.purr() for cat in zoo[3:]] == ["mrowr", "mrrrRRRRRR"]
+    zoo[0].excitement -= 1000
+    assert zoo[0].excitement == 14000