diff options
Diffstat (limited to 'ext/pybind11/docs/advanced')
| -rw-r--r-- | ext/pybind11/docs/advanced/cast/chrono.rst | 81 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/cast/custom.rst | 85 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/cast/eigen.rst | 50 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/cast/functional.rst | 113 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/cast/index.rst | 41 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/cast/overview.rst | 146 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/cast/stl.rst | 154 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/classes.rst | 634 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/exceptions.rst | 142 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/functions.rst | 311 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/misc.rst | 229 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/pycpp/index.rst | 13 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/pycpp/numpy.rst | 299 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/pycpp/object.rst | 96 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/pycpp/utilities.rst | 57 | ||||
| -rw-r--r-- | ext/pybind11/docs/advanced/smart_ptrs.rst | 143 |
16 files changed, 2594 insertions, 0 deletions
diff --git a/ext/pybind11/docs/advanced/cast/chrono.rst b/ext/pybind11/docs/advanced/cast/chrono.rst new file mode 100644 index 000000000..6d4a5ee55 --- /dev/null +++ b/ext/pybind11/docs/advanced/cast/chrono.rst @@ -0,0 +1,81 @@ +Chrono +====== + +When including the additional header file :file:`pybind11/chrono.h` conversions +from C++11 chrono datatypes to python datetime objects are automatically enabled. +This header also enables conversions of python floats (often from sources such +as `time.monotonic()`, `time.perf_counter()` and `time.process_time()`) into +durations. + +An overview of clocks in C++11 +------------------------------ + +A point of confusion when using these conversions is the differences between +clocks provided in C++11. There are three clock types defined by the C++11 +standard and users can define their own if needed. Each of these clocks have +different properties and when converting to and from python will give different +results. + +The first clock defined by the standard is ``std::chrono::system_clock``. This +clock measures the current date and time. However, this clock changes with to +updates to the operating system time. For example, if your time is synchronised +with a time server this clock will change. This makes this clock a poor choice +for timing purposes but good for measuring the wall time. + +The second clock defined in the standard is ``std::chrono::steady_clock``. +This clock ticks at a steady rate and is never adjusted. This makes it excellent +for timing purposes, however the value in this clock does not correspond to the +current date and time. Often this clock will be the amount of time your system +has been on, although it does not have to be. This clock will never be the same +clock as the system clock as the system clock can change but steady clocks +cannot. + +The third clock defined in the standard is ``std::chrono::high_resolution_clock``. +This clock is the clock that has the highest resolution out of the clocks in the +system. It is normally a typedef to either the system clock or the steady clock +but can be its own independent clock. This is important as when using these +conversions as the types you get in python for this clock might be different +depending on the system. +If it is a typedef of the system clock, python will get datetime objects, but if +it is a different clock they will be timedelta objects. + +Provided conversions +-------------------- + +.. rubric:: C++ to Python + +- ``std::chrono::system_clock::time_point`` → ``datetime.datetime`` + System clock times are converted to python datetime instances. They are + in the local timezone, but do not have any timezone information attached + to them (they are naive datetime objects). + +- ``std::chrono::duration`` → ``datetime.timedelta`` + Durations are converted to timedeltas, any precision in the duration + greater than microseconds is lost by rounding towards zero. + +- ``std::chrono::[other_clocks]::time_point`` → ``datetime.timedelta`` + Any clock time that is not the system clock is converted to a time delta. + This timedelta measures the time from the clocks epoch to now. + +.. rubric:: Python to C++ + +- ``datetime.datetime`` → ``std::chrono::system_clock::time_point`` + Date/time objects are converted into system clock timepoints. Any + timezone information is ignored and the type is treated as a naive + object. + +- ``datetime.timedelta`` → ``std::chrono::duration`` + Time delta are converted into durations with microsecond precision. + +- ``datetime.timedelta`` → ``std::chrono::[other_clocks]::time_point`` + Time deltas that are converted into clock timepoints are treated as + the amount of time from the start of the clocks epoch. + +- ``float`` → ``std::chrono::duration`` + Floats that are passed to C++ as durations be interpreted as a number of + seconds. These will be converted to the duration using ``duration_cast`` + from the float. + +- ``float`` → ``std::chrono::[other_clocks]::time_point`` + Floats that are passed to C++ as time points will be interpreted as the + number of seconds from the start of the clocks epoch. diff --git a/ext/pybind11/docs/advanced/cast/custom.rst b/ext/pybind11/docs/advanced/cast/custom.rst new file mode 100644 index 000000000..c854e7fcd --- /dev/null +++ b/ext/pybind11/docs/advanced/cast/custom.rst @@ -0,0 +1,85 @@ +Custom type casters +=================== + +In very rare cases, applications may require custom type casters that cannot be +expressed using the abstractions provided by pybind11, thus requiring raw +Python C API calls. This is fairly advanced usage and should only be pursued by +experts who are familiar with the intricacies of Python reference counting. + +The following snippets demonstrate how this works for a very simple ``inty`` +type that that should be convertible from Python types that provide a +``__int__(self)`` method. + +.. code-block:: cpp + + struct inty { long long_value; }; + + void print(inty s) { + std::cout << s.long_value << std::endl; + } + +The following Python snippet demonstrates the intended usage from the Python side: + +.. code-block:: python + + class A: + def __int__(self): + return 123 + + from example import print + print(A()) + +To register the necessary conversion routines, it is necessary to add +a partial overload to the ``pybind11::detail::type_caster<T>`` template. +Although this is an implementation detail, adding partial overloads to this +type is explicitly allowed. + +.. code-block:: cpp + + namespace pybind11 { namespace detail { + template <> struct type_caster<inty> { + public: + /** + * This macro establishes the name 'inty' in + * function signatures and declares a local variable + * 'value' of type inty + */ + PYBIND11_TYPE_CASTER(inty, _("inty")); + + /** + * Conversion part 1 (Python->C++): convert a PyObject into a inty + * instance or return false upon failure. The second argument + * indicates whether implicit conversions should be applied. + */ + bool load(handle src, bool) { + /* Extract PyObject from handle */ + PyObject *source = src.ptr(); + /* Try converting into a Python integer value */ + PyObject *tmp = PyNumber_Long(source); + if (!tmp) + return false; + /* Now try to convert into a C++ int */ + value.long_value = PyLong_AsLong(tmp); + Py_DECREF(tmp); + /* Ensure return code was OK (to avoid out-of-range errors etc) */ + return !(value.long_value == -1 && !PyErr_Occurred()); + } + + /** + * Conversion part 2 (C++ -> Python): convert an inty instance into + * a Python object. The second and third arguments are used to + * indicate the return value policy and parent object (for + * ``return_value_policy::reference_internal``) and are generally + * ignored by implicit casters. + */ + static handle cast(inty src, return_value_policy /* policy */, handle /* parent */) { + return PyLong_FromLong(src.long_value); + } + }; + }} // namespace pybind11::detail + +.. warning:: + + When using custom type casters, it's important to declare them consistently + in every compilation unit of the Python extension module. Otherwise, + undefined behavior can ensue. diff --git a/ext/pybind11/docs/advanced/cast/eigen.rst b/ext/pybind11/docs/advanced/cast/eigen.rst new file mode 100644 index 000000000..b83ca9af9 --- /dev/null +++ b/ext/pybind11/docs/advanced/cast/eigen.rst @@ -0,0 +1,50 @@ +Eigen +===== + +`Eigen <http://eigen.tuxfamily.org>`_ is C++ header-based library for dense and +sparse linear algebra. Due to its popularity and widespread adoption, pybind11 +provides transparent conversion support between Eigen and Scientific Python linear +algebra data types. + +Specifically, when including the optional header file :file:`pybind11/eigen.h`, +pybind11 will automatically and transparently convert + +1. Static and dynamic Eigen dense vectors and matrices to instances of + ``numpy.ndarray`` (and vice versa). + +2. Returned matrix expressions such as blocks (including columns or rows) and + diagonals will be converted to ``numpy.ndarray`` of the expression + values. + +3. Returned matrix-like objects such as Eigen::DiagonalMatrix or + Eigen::SelfAdjointView will be converted to ``numpy.ndarray`` containing the + expressed value. + +4. Eigen sparse vectors and matrices to instances of + ``scipy.sparse.csr_matrix``/``scipy.sparse.csc_matrix`` (and vice versa). + +This makes it possible to bind most kinds of functions that rely on these types. +One major caveat are functions that take Eigen matrices *by reference* and modify +them somehow, in which case the information won't be propagated to the caller. + +.. code-block:: cpp + + /* The Python bindings of these functions won't replicate + the intended effect of modifying the function arguments */ + void scale_by_2(Eigen::Vector3f &v) { + v *= 2; + } + void scale_by_2(Eigen::Ref<Eigen::MatrixXd> &v) { + v *= 2; + } + +To see why this is, refer to the section on :ref:`opaque` (although that +section specifically covers STL data types, the underlying issue is the same). +The :ref:`numpy` sections discuss an efficient alternative for exposing the +underlying native Eigen types as opaque objects in a way that still integrates +with NumPy and SciPy. + +.. seealso:: + + The file :file:`tests/test_eigen.cpp` contains a complete example that + shows how to pass Eigen sparse and dense data types in more detail. diff --git a/ext/pybind11/docs/advanced/cast/functional.rst b/ext/pybind11/docs/advanced/cast/functional.rst new file mode 100644 index 000000000..5d0a01d13 --- /dev/null +++ b/ext/pybind11/docs/advanced/cast/functional.rst @@ -0,0 +1,113 @@ +Functional +########## + +The following features must be enabled by including :file:`pybind11/functional.h`. + + +Callbacks and passing anonymous functions +========================================= + +The C++11 standard brought lambda functions and the generic polymorphic +function wrapper ``std::function<>`` to the C++ programming language, which +enable powerful new ways of working with functions. Lambda functions come in +two flavors: stateless lambda function resemble classic function pointers that +link to an anonymous piece of code, while stateful lambda functions +additionally depend on captured variables that are stored in an anonymous +*lambda closure object*. + +Here is a simple example of a C++ function that takes an arbitrary function +(stateful or stateless) with signature ``int -> int`` as an argument and runs +it with the value 10. + +.. code-block:: cpp + + int func_arg(const std::function<int(int)> &f) { + return f(10); + } + +The example below is more involved: it takes a function of signature ``int -> int`` +and returns another function of the same kind. The return value is a stateful +lambda function, which stores the value ``f`` in the capture object and adds 1 to +its return value upon execution. + +.. code-block:: cpp + + std::function<int(int)> func_ret(const std::function<int(int)> &f) { + return [f](int i) { + return f(i) + 1; + }; + } + +This example demonstrates using python named parameters in C++ callbacks which +requires using ``py::cpp_function`` as a wrapper. Usage is similar to defining +methods of classes: + +.. code-block:: cpp + + py::cpp_function func_cpp() { + return py::cpp_function([](int i) { return i+1; }, + py::arg("number")); + } + +After including the extra header file :file:`pybind11/functional.h`, it is almost +trivial to generate binding code for all of these functions. + +.. code-block:: cpp + + #include <pybind11/functional.h> + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + m.def("func_arg", &func_arg); + m.def("func_ret", &func_ret); + m.def("func_cpp", &func_cpp); + + return m.ptr(); + } + +The following interactive session shows how to call them from Python. + +.. code-block:: pycon + + $ python + >>> import example + >>> def square(i): + ... return i * i + ... + >>> example.func_arg(square) + 100L + >>> square_plus_1 = example.func_ret(square) + >>> square_plus_1(4) + 17L + >>> plus_1 = func_cpp() + >>> plus_1(number=43) + 44L + +.. warning:: + + Keep in mind that passing a function from C++ to Python (or vice versa) + will instantiate a piece of wrapper code that translates function + invocations between the two languages. Naturally, this translation + increases the computational cost of each function call somewhat. A + problematic situation can arise when a function is copied back and forth + between Python and C++ many times in a row, in which case the underlying + wrappers will accumulate correspondingly. The resulting long sequence of + C++ -> Python -> C++ -> ... roundtrips can significantly decrease + performance. + + There is one exception: pybind11 detects case where a stateless function + (i.e. a function pointer or a lambda function without captured variables) + is passed as an argument to another C++ function exposed in Python. In this + case, there is no overhead. Pybind11 will extract the underlying C++ + function pointer from the wrapped function to sidestep a potential C++ -> + Python -> C++ roundtrip. This is demonstrated in :file:`tests/test_callbacks.cpp`. + +.. note:: + + This functionality is very useful when generating bindings for callbacks in + C++ libraries (e.g. GUI libraries, asynchronous networking libraries, etc.). + + The file :file:`tests/test_callbacks.cpp` contains a complete example + that demonstrates how to work with callbacks and anonymous functions in + more detail. diff --git a/ext/pybind11/docs/advanced/cast/index.rst b/ext/pybind11/docs/advanced/cast/index.rst new file mode 100644 index 000000000..36586af5c --- /dev/null +++ b/ext/pybind11/docs/advanced/cast/index.rst @@ -0,0 +1,41 @@ +Type conversions +################ + +Apart from enabling cross-language function calls, a fundamental problem +that a binding tool like pybind11 must address is to provide access to +native Python types in C++ and vice versa. There are three fundamentally +different ways to do this—which approach is preferable for a particular type +depends on the situation at hand. + +1. Use a native C++ type everywhere. In this case, the type must be wrapped + using pybind11-generated bindings so that Python can interact with it. + +2. Use a native Python type everywhere. It will need to be wrapped so that + C++ functions can interact with it. + +3. Use a native C++ type on the C++ side and a native Python type on the + Python side. pybind11 refers to this as a *type conversion*. + + Type conversions are the most "natural" option in the sense that native + (non-wrapped) types are used everywhere. The main downside is that a copy + of the data must be made on every Python ↔ C++ transition: this is + needed since the C++ and Python versions of the same type generally won't + have the same memory layout. + + pybind11 can perform many kinds of conversions automatically. An overview + is provided in the table ":ref:`conversion_table`". + +The following subsections discuss the differences between these options in more +detail. The main focus in this section is on type conversions, which represent +the last case of the above list. + +.. toctree:: + :maxdepth: 1 + + overview + stl + functional + chrono + eigen + custom + diff --git a/ext/pybind11/docs/advanced/cast/overview.rst b/ext/pybind11/docs/advanced/cast/overview.rst new file mode 100644 index 000000000..ab37b90be --- /dev/null +++ b/ext/pybind11/docs/advanced/cast/overview.rst @@ -0,0 +1,146 @@ +Overview +######## + +.. rubric:: 1. Native type in C++, wrapper in Python + +Exposing a custom C++ type using :class:`py::class_` was covered in detail +in the :doc:`/classes` section. There, the underlying data structure is +always the original C++ class while the :class:`py::class_` wrapper provides +a Python interface. Internally, when an object like this is sent from C++ to +Python, pybind11 will just add the outer wrapper layer over the native C++ +object. Getting it back from Python is just a matter of peeling off the +wrapper. + +.. rubric:: 2. Wrapper in C++, native type in Python + +This is the exact opposite situation. Now, we have a type which is native to +Python, like a ``tuple`` or a ``list``. One way to get this data into C++ is +with the :class:`py::object` family of wrappers. These are explained in more +detail in the :doc:`/advanced/pycpp/object` section. We'll just give a quick +example here: + +.. code-block:: cpp + + void print_list(py::list my_list) { + for (auto item : my_list) + std::cout << item << " "; + } + +.. code-block:: pycon + + >>> print_list([1, 2, 3]) + 1 2 3 + +The Python ``list`` is not converted in any way -- it's just wrapped in a C++ +:class:`py::list` class. At its core it's still a Python object. Copying a +:class:`py::list` will do the usual reference-counting like in Python. +Returning the object to Python will just remove the thin wrapper. + +.. rubric:: 3. Converting between native C++ and Python types + +In the previous two cases we had a native type in one language and a wrapper in +the other. Now, we have native types on both sides and we convert between them. + +.. code-block:: cpp + + void print_vector(const std::vector<int> &v) { + for (auto item : v) + std::cout << item << "\n"; + } + +.. code-block:: pycon + + >>> print_vector([1, 2, 3]) + 1 2 3 + +In this case, pybind11 will construct a new ``std::vector<int>`` and copy each +element from the Python ``list``. The newly constructed object will be passed +to ``print_vector``. The same thing happens in the other direction: a new +``list`` is made to match the value returned from C++. + +Lots of these conversions are supported out of the box, as shown in the table +below. They are very convenient, but keep in mind that these conversions are +fundamentally based on copying data. This is perfectly fine for small immutable +types but it may become quite expensive for large data structures. This can be +avoided by overriding the automatic conversion with a custom wrapper (i.e. the +above-mentioned approach 1). This requires some manual effort and more details +are available in the :ref:`opaque` section. + +.. _conversion_table: + +List of all builtin conversions +------------------------------- + +The following basic data types are supported out of the box (some may require +an additional extension header to be included). To pass other data structures +as arguments and return values, refer to the section on binding :ref:`classes`. + ++------------------------------------+---------------------------+-------------------------------+ +| Data type | Description | Header file | ++====================================+===========================+===============================+ +| ``int8_t``, ``uint8_t`` | 8-bit integers | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``int16_t``, ``uint16_t`` | 16-bit integers | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``int32_t``, ``uint32_t`` | 32-bit integers | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``int64_t``, ``uint64_t`` | 64-bit integers | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``ssize_t``, ``size_t`` | Platform-dependent size | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``float``, ``double`` | Floating point types | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``bool`` | Two-state Boolean type | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``char`` | Character literal | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``wchar_t`` | Wide character literal | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``const char *`` | UTF-8 string literal | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``const wchar_t *`` | Wide string literal | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::string`` | STL dynamic UTF-8 string | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::wstring`` | STL dynamic wide string | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::pair<T1, T2>`` | Pair of two custom types | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::tuple<...>`` | Arbitrary tuple of types | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::reference_wrapper<...>`` | Reference type wrapper | :file:`pybind11/pybind11.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::complex<T>`` | Complex numbers | :file:`pybind11/complex.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::array<T, Size>`` | STL static array | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::vector<T>`` | STL dynamic array | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::valarray<T>`` | STL value array | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::list<T>`` | STL linked list | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::map<T1, T2>`` | STL ordered map | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::unordered_map<T1, T2>`` | STL unordered map | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::set<T>`` | STL ordered set | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::unordered_set<T>`` | STL unordered set | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::optional<T>`` | STL optional type (C++17) | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::experimental::optional<T>`` | STL optional type (exp.) | :file:`pybind11/stl.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::function<...>`` | STL polymorphic function | :file:`pybind11/functional.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::chrono::duration<...>`` | STL time duration | :file:`pybind11/chrono.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``std::chrono::time_point<...>`` | STL date/time | :file:`pybind11/chrono.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``Eigen::Matrix<...>`` | Eigen: dense matrix | :file:`pybind11/eigen.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``Eigen::Map<...>`` | Eigen: mapped memory | :file:`pybind11/eigen.h` | ++------------------------------------+---------------------------+-------------------------------+ +| ``Eigen::SparseMatrix<...>`` | Eigen: sparse matrix | :file:`pybind11/eigen.h` | ++------------------------------------+---------------------------+-------------------------------+ diff --git a/ext/pybind11/docs/advanced/cast/stl.rst b/ext/pybind11/docs/advanced/cast/stl.rst new file mode 100644 index 000000000..bbd23732b --- /dev/null +++ b/ext/pybind11/docs/advanced/cast/stl.rst @@ -0,0 +1,154 @@ +STL containers +############## + +Automatic conversion +==================== + +When including the additional header file :file:`pybind11/stl.h`, conversions +between ``std::vector<>``, ``std::list<>``, ``std::set<>``, and ``std::map<>`` +and the Python ``list``, ``set`` and ``dict`` data structures are automatically +enabled. The types ``std::pair<>`` and ``std::tuple<>`` are already supported +out of the box with just the core :file:`pybind11/pybind11.h` header. + +The major downside of these implicit conversions is that containers must be +converted (i.e. copied) on every Python->C++ and C++->Python transition, which +can have implications on the program semantics and performance. Please read the +next sections for more details and alternative approaches that avoid this. + +.. note:: + + Arbitrary nesting of any of these types is possible. + +.. seealso:: + + The file :file:`tests/test_python_types.cpp` contains a complete + example that demonstrates how to pass STL data types in more detail. + +.. _opaque: + +Making opaque types +=================== + +pybind11 heavily relies on a template matching mechanism to convert parameters +and return values that are constructed from STL data types such as vectors, +linked lists, hash tables, etc. This even works in a recursive manner, for +instance to deal with lists of hash maps of pairs of elementary and custom +types, etc. + +However, a fundamental limitation of this approach is that internal conversions +between Python and C++ types involve a copy operation that prevents +pass-by-reference semantics. What does this mean? + +Suppose we bind the following function + +.. code-block:: cpp + + void append_1(std::vector<int> &v) { + v.push_back(1); + } + +and call it from Python, the following happens: + +.. code-block:: pycon + + >>> v = [5, 6] + >>> append_1(v) + >>> print(v) + [5, 6] + +As you can see, when passing STL data structures by reference, modifications +are not propagated back the Python side. A similar situation arises when +exposing STL data structures using the ``def_readwrite`` or ``def_readonly`` +functions: + +.. code-block:: cpp + + /* ... definition ... */ + + class MyClass { + std::vector<int> contents; + }; + + /* ... binding code ... */ + + py::class_<MyClass>(m, "MyClass") + .def(py::init<>) + .def_readwrite("contents", &MyClass::contents); + +In this case, properties can be read and written in their entirety. However, an +``append`` operation involving such a list type has no effect: + +.. code-block:: pycon + + >>> m = MyClass() + >>> m.contents = [5, 6] + >>> print(m.contents) + [5, 6] + >>> m.contents.append(7) + >>> print(m.contents) + [5, 6] + +Finally, the involved copy operations can be costly when dealing with very +large lists. To deal with all of the above situations, pybind11 provides a +macro named ``PYBIND11_MAKE_OPAQUE(T)`` that disables the template-based +conversion machinery of types, thus rendering them *opaque*. The contents of +opaque objects are never inspected or extracted, hence they *can* be passed by +reference. For instance, to turn ``std::vector<int>`` into an opaque type, add +the declaration + +.. code-block:: cpp + + PYBIND11_MAKE_OPAQUE(std::vector<int>); + +before any binding code (e.g. invocations to ``class_::def()``, etc.). This +macro must be specified at the top level (and outside of any namespaces), since +it instantiates a partial template overload. If your binding code consists of +multiple compilation units, it must be present in every file preceding any +usage of ``std::vector<int>``. Opaque types must also have a corresponding +``class_`` declaration to associate them with a name in Python, and to define a +set of available operations, e.g.: + +.. code-block:: cpp + + py::class_<std::vector<int>>(m, "IntVector") + .def(py::init<>()) + .def("clear", &std::vector<int>::clear) + .def("pop_back", &std::vector<int>::pop_back) + .def("__len__", [](const std::vector<int> &v) { return v.size(); }) + .def("__iter__", [](std::vector<int> &v) { + return py::make_iterator(v.begin(), v.end()); + }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */ + // .... + +The ability to expose STL containers as native Python objects is a fairly +common request, hence pybind11 also provides an optional header file named +:file:`pybind11/stl_bind.h` that does exactly this. The mapped containers try +to match the behavior of their native Python counterparts as much as possible. + +The following example showcases usage of :file:`pybind11/stl_bind.h`: + +.. code-block:: cpp + + // Don't forget this + #include <pybind11/stl_bind.h> + + PYBIND11_MAKE_OPAQUE(std::vector<int>); + PYBIND11_MAKE_OPAQUE(std::map<std::string, double>); + + // ... + + // later in binding code: + py::bind_vector<std::vector<int>>(m, "VectorInt"); + py::bind_map<std::map<std::string, double>>(m, "MapStringDouble"); + +Please take a look at the :ref:`macro_notes` before using the +``PYBIND11_MAKE_OPAQUE`` macro. + +.. seealso:: + + The file :file:`tests/test_opaque_types.cpp` contains a complete + example that demonstrates how to create and expose opaque types using + pybind11 in more detail. + + The file :file:`tests/test_stl_binders.cpp` shows how to use the + convenience STL container wrappers. diff --git a/ext/pybind11/docs/advanced/classes.rst b/ext/pybind11/docs/advanced/classes.rst new file mode 100644 index 000000000..4a423b578 --- /dev/null +++ b/ext/pybind11/docs/advanced/classes.rst @@ -0,0 +1,634 @@ +Classes +####### + +This section presents advanced binding code for classes and it is assumed +that you are already familiar with the basics from :doc:`/classes`. + +.. _overriding_virtuals: + +Overriding virtual functions in Python +====================================== + +Suppose that a C++ class or interface has a virtual function that we'd like to +to override from within Python (we'll focus on the class ``Animal``; ``Dog`` is +given as a specific example of how one would do this with traditional C++ +code). + +.. code-block:: cpp + + class Animal { + public: + virtual ~Animal() { } + virtual std::string go(int n_times) = 0; + }; + + class Dog : public Animal { + public: + std::string go(int n_times) override { + std::string result; + for (int i=0; i<n_times; ++i) + result += "woof! "; + return result; + } + }; + +Let's also suppose that we are given a plain function which calls the +function ``go()`` on an arbitrary ``Animal`` instance. + +.. code-block:: cpp + + std::string call_go(Animal *animal) { + return animal->go(3); + } + +Normally, the binding code for these classes would look as follows: + +.. code-block:: cpp + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::class_<Animal> animal(m, "Animal"); + animal + .def("go", &Animal::go); + + py::class_<Dog>(m, "Dog", animal) + .def(py::init<>()); + + m.def("call_go", &call_go); + + return m.ptr(); + } + +However, these bindings are impossible to extend: ``Animal`` is not +constructible, and we clearly require some kind of "trampoline" that +redirects virtual calls back to Python. + +Defining a new type of ``Animal`` from within Python is possible but requires a +helper class that is defined as follows: + +.. code-block:: cpp + + class PyAnimal : public Animal { + public: + /* Inherit the constructors */ + using Animal::Animal; + + /* Trampoline (need one for each virtual function) */ + std::string go(int n_times) override { + PYBIND11_OVERLOAD_PURE( + std::string, /* Return type */ + Animal, /* Parent class */ + go, /* Name of function */ + n_times /* Argument(s) */ + ); + } + }; + +The macro :func:`PYBIND11_OVERLOAD_PURE` should be used for pure virtual +functions, and :func:`PYBIND11_OVERLOAD` should be used for functions which have +a default implementation. There are also two alternate macros +:func:`PYBIND11_OVERLOAD_PURE_NAME` and :func:`PYBIND11_OVERLOAD_NAME` which +take a string-valued name argument between the *Parent class* and *Name of the +function* slots. This is useful when the C++ and Python versions of the +function have different names, e.g. ``operator()`` vs ``__call__``. + +The binding code also needs a few minor adaptations (highlighted): + +.. code-block:: cpp + :emphasize-lines: 4,6,7 + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::class_<Animal, PyAnimal /* <--- trampoline*/> animal(m, "Animal"); + animal + .def(py::init<>()) + .def("go", &Animal::go); + + py::class_<Dog>(m, "Dog", animal) + .def(py::init<>()); + + m.def("call_go", &call_go); + + return m.ptr(); + } + +Importantly, pybind11 is made aware of the trampoline helper class by +specifying it as an extra template argument to :class:`class_`. (This can also +be combined with other template arguments such as a custom holder type; the +order of template types does not matter). Following this, we are able to +define a constructor as usual. + +Note, however, that the above is sufficient for allowing python classes to +extend ``Animal``, but not ``Dog``: see ref:`virtual_and_inheritance` for the +necessary steps required to providing proper overload support for inherited +classes. + +The Python session below shows how to override ``Animal::go`` and invoke it via +a virtual method call. + +.. code-block:: pycon + + >>> from example import * + >>> d = Dog() + >>> call_go(d) + u'woof! woof! woof! ' + >>> class Cat(Animal): + ... def go(self, n_times): + ... return "meow! " * n_times + ... + >>> c = Cat() + >>> call_go(c) + u'meow! meow! meow! ' + +Please take a look at the :ref:`macro_notes` before using this feature. + +.. note:: + + When the overridden type returns a reference or pointer to a type that + pybind11 converts from Python (for example, numeric values, std::string, + and other built-in value-converting types), there are some limitations to + be aware of: + + - because in these cases there is no C++ variable to reference (the value + is stored in the referenced Python variable), pybind11 provides one in + the PYBIND11_OVERLOAD macros (when needed) with static storage duration. + Note that this means that invoking the overloaded method on *any* + instance will change the referenced value stored in *all* instances of + that type. + + - Attempts to modify a non-const reference will not have the desired + effect: it will change only the static cache variable, but this change + will not propagate to underlying Python instance, and the change will be + replaced the next time the overload is invoked. + +.. seealso:: + + The file :file:`tests/test_virtual_functions.cpp` contains a complete + example that demonstrates how to override virtual functions using pybind11 + in more detail. + +.. _virtual_and_inheritance: + +Combining virtual functions and inheritance +=========================================== + +When combining virtual methods with inheritance, you need to be sure to provide +an override for each method for which you want to allow overrides from derived +python classes. For example, suppose we extend the above ``Animal``/``Dog`` +example as follows: + +.. code-block:: cpp + + class Animal { + public: + virtual std::string go(int n_times) = 0; + virtual std::string name() { return "unknown"; } + }; + class Dog : public class Animal { + public: + std::string go(int n_times) override { + std::string result; + for (int i=0; i<n_times; ++i) + result += bark() + " "; + return result; + } + virtual std::string bark() { return "woof!"; } + }; + +then the trampoline class for ``Animal`` must, as described in the previous +section, override ``go()`` and ``name()``, but in order to allow python code to +inherit properly from ``Dog``, we also need a trampoline class for ``Dog`` that +overrides both the added ``bark()`` method *and* the ``go()`` and ``name()`` +methods inherited from ``Animal`` (even though ``Dog`` doesn't directly +override the ``name()`` method): + +.. code-block:: cpp + + class PyAnimal : public Animal { + public: + using Animal::Animal; // Inherit constructors + std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, Animal, go, n_times); } + std::string name() override { PYBIND11_OVERLOAD(std::string, Animal, name, ); } + }; + class PyDog : public Dog { + public: + using Dog::Dog; // Inherit constructors + std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, Dog, go, n_times); } + std::string name() override { PYBIND11_OVERLOAD(std::string, Dog, name, ); } + std::string bark() override { PYBIND11_OVERLOAD(std::string, Dog, bark, ); } + }; + +A registered class derived from a pybind11-registered class with virtual +methods requires a similar trampoline class, *even if* it doesn't explicitly +declare or override any virtual methods itself: + +.. code-block:: cpp + + class Husky : public Dog {}; + class PyHusky : public Husky { + using Dog::Dog; // Inherit constructors + std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, Husky, go, n_times); } + std::string name() override { PYBIND11_OVERLOAD(std::string, Husky, name, ); } + std::string bark() override { PYBIND11_OVERLOAD(std::string, Husky, bark, ); } + }; + +There is, however, a technique that can be used to avoid this duplication +(which can be especially helpful for a base class with several virtual +methods). The technique involves using template trampoline classes, as +follows: + +.. code-block:: cpp + + template <class AnimalBase = Animal> class PyAnimal : public AnimalBase { + using AnimalBase::AnimalBase; // Inherit constructors + std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, AnimalBase, go, n_times); } + std::string name() override { PYBIND11_OVERLOAD(std::string, AnimalBase, name, ); } + }; + template <class DogBase = Dog> class PyDog : public PyAnimal<DogBase> { + using PyAnimal<DogBase>::PyAnimal; // Inherit constructors + // Override PyAnimal's pure virtual go() with a non-pure one: + std::string go(int n_times) override { PYBIND11_OVERLOAD(std::string, DogBase, go, n_times); } + std::string bark() override { PYBIND11_OVERLOAD(std::string, DogBase, bark, ); } + }; + +This technique has the advantage of requiring just one trampoline method to be +declared per virtual method and pure virtual method override. It does, +however, require the compiler to generate at least as many methods (and +possibly more, if both pure virtual and overridden pure virtual methods are +exposed, as above). + +The classes are then registered with pybind11 using: + +.. code-block:: cpp + + py::class_<Animal, PyAnimal<>> animal(m, "Animal"); + py::class_<Dog, PyDog<>> dog(m, "Dog"); + py::class_<Husky, PyDog<Husky>> husky(m, "Husky"); + // ... add animal, dog, husky definitions + +Note that ``Husky`` did not require a dedicated trampoline template class at +all, since it neither declares any new virtual methods nor provides any pure +virtual method implementations. + +With either the repeated-virtuals or templated trampoline methods in place, you +can now create a python class that inherits from ``Dog``: + +.. code-block:: python + + class ShihTzu(Dog): + def bark(self): + return "yip!" + +.. seealso:: + + See the file :file:`tests/test_virtual_functions.cpp` for complete examples + using both the duplication and templated trampoline approaches. + +Extended trampoline class functionality +======================================= + +The trampoline classes described in the previous sections are, by default, only +initialized when needed. More specifically, they are initialized when a python +class actually inherits from a registered type (instead of merely creating an +instance of the registered type), or when a registered constructor is only +valid for the trampoline class but not the registered class. This is primarily +for performance reasons: when the trampoline class is not needed for anything +except virtual method dispatching, not initializing the trampoline class +improves performance by avoiding needing to do a run-time check to see if the +inheriting python instance has an overloaded method. + +Sometimes, however, it is useful to always initialize a trampoline class as an +intermediate class that does more than just handle virtual method dispatching. +For example, such a class might perform extra class initialization, extra +destruction operations, and might define new members and methods to enable a +more python-like interface to a class. + +In order to tell pybind11 that it should *always* initialize the trampoline +class when creating new instances of a type, the class constructors should be +declared using ``py::init_alias<Args, ...>()`` instead of the usual +``py::init<Args, ...>()``. This forces construction via the trampoline class, +ensuring member initialization and (eventual) destruction. + +.. seealso:: + + See the file :file:`tests/test_alias_initialization.cpp` for complete examples + showing both normal and forced trampoline instantiation. + +.. _custom_constructors: + +Custom constructors +=================== + +The syntax for binding constructors was previously introduced, but it only +works when a constructor with the given parameters actually exists on the C++ +side. To extend this to more general cases, let's take a look at what actually +happens under the hood: the following statement + +.. code-block:: cpp + + py::class_<Example>(m, "Example") + .def(py::init<int>()); + +is short hand notation for + +.. code-block:: cpp + + py::class_<Example>(m, "Example") + .def("__init__", + [](Example &instance, int arg) { + new (&instance) Example(arg); + } + ); + +In other words, :func:`init` creates an anonymous function that invokes an +in-place constructor. Memory allocation etc. is already take care of beforehand +within pybind11. + +.. _classes_with_non_public_destructors: + +Non-public destructors +====================== + +If a class has a private or protected destructor (as might e.g. be the case in +a singleton pattern), a compile error will occur when creating bindings via +pybind11. The underlying issue is that the ``std::unique_ptr`` holder type that +is responsible for managing the lifetime of instances will reference the +destructor even if no deallocations ever take place. In order to expose classes +with private or protected destructors, it is possible to override the holder +type via a holder type argument to ``class_``. Pybind11 provides a helper class +``py::nodelete`` that disables any destructor invocations. In this case, it is +crucial that instances are deallocated on the C++ side to avoid memory leaks. + +.. code-block:: cpp + + /* ... definition ... */ + + class MyClass { + private: + ~MyClass() { } + }; + + /* ... binding code ... */ + + py::class_<MyClass, std::unique_ptr<MyClass, py::nodelete>>(m, "MyClass") + .def(py::init<>) + +Implicit conversions +==================== + +Suppose that instances of two types ``A`` and ``B`` are used in a project, and +that an ``A`` can easily be converted into an instance of type ``B`` (examples of this +could be a fixed and an arbitrary precision number type). + +.. code-block:: cpp + + py::class_<A>(m, "A") + /// ... members ... + + py::class_<B>(m, "B") + .def(py::init<A>()) + /// ... members ... + + m.def("func", + [](const B &) { /* .... */ } + ); + +To invoke the function ``func`` using a variable ``a`` containing an ``A`` +instance, we'd have to write ``func(B(a))`` in Python. On the other hand, C++ +will automatically apply an implicit type conversion, which makes it possible +to directly write ``func(a)``. + +In this situation (i.e. where ``B`` has a constructor that converts from +``A``), the following statement enables similar implicit conversions on the +Python side: + +.. code-block:: cpp + + py::implicitly_convertible<A, B>(); + +.. note:: + + Implicit conversions from ``A`` to ``B`` only work when ``B`` is a custom + data type that is exposed to Python via pybind11. + +.. _static_properties: + +Static properties +================= + +The section on :ref:`properties` discussed the creation of instance properties +that are implemented in terms of C++ getters and setters. + +Static properties can also be created in a similar way to expose getters and +setters of static class attributes. It is important to note that the implicit +``self`` argument also exists in this case and is used to pass the Python +``type`` subclass instance. This parameter will often not be needed by the C++ +side, and the following example illustrates how to instantiate a lambda getter +function that ignores it: + +.. code-block:: cpp + + py::class_<Foo>(m, "Foo") + .def_property_readonly_static("foo", [](py::object /* self */) { return Foo(); }); + +Operator overloading +==================== + +Suppose that we're given the following ``Vector2`` class with a vector addition +and scalar multiplication operation, all implemented using overloaded operators +in C++. + +.. code-block:: cpp + + class Vector2 { + public: + Vector2(float x, float y) : x(x), y(y) { } + + Vector2 operator+(const Vector2 &v) const { return Vector2(x + v.x, y + v.y); } + Vector2 operator*(float value) const { return Vector2(x * value, y * value); } + Vector2& operator+=(const Vector2 &v) { x += v.x; y += v.y; return *this; } + Vector2& operator*=(float v) { x *= v; y *= v; return *this; } + + friend Vector2 operator*(float f, const Vector2 &v) { + return Vector2(f * v.x, f * v.y); + } + + std::string toString() const { + return "[" + std::to_string(x) + ", " + std::to_string(y) + "]"; + } + private: + float x, y; + }; + +The following snippet shows how the above operators can be conveniently exposed +to Python. + +.. code-block:: cpp + + #include <pybind11/operators.h> + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::class_<Vector2>(m, "Vector2") + .def(py::init<float, float>()) + .def(py::self + py::self) + .def(py::self += py::self) + .def(py::self *= float()) + .def(float() * py::self) + .def("__repr__", &Vector2::toString); + + return m.ptr(); + } + +Note that a line like + +.. code-block:: cpp + + .def(py::self * float()) + +is really just short hand notation for + +.. code-block:: cpp + + .def("__mul__", [](const Vector2 &a, float b) { + return a * b; + }, py::is_operator()) + +This can be useful for exposing additional operators that don't exist on the +C++ side, or to perform other types of customization. The ``py::is_operator`` +flag marker is needed to inform pybind11 that this is an operator, which +returns ``NotImplemented`` when invoked with incompatible arguments rather than +throwing a type error. + +.. note:: + + To use the more convenient ``py::self`` notation, the additional + header file :file:`pybind11/operators.h` must be included. + +.. seealso:: + + The file :file:`tests/test_operator_overloading.cpp` contains a + complete example that demonstrates how to work with overloaded operators in + more detail. + +Pickling support +================ + +Python's ``pickle`` module provides a powerful facility to serialize and +de-serialize a Python object graph into a binary data stream. To pickle and +unpickle C++ classes using pybind11, two additional functions must be provided. +Suppose the class in question has the following signature: + +.. code-block:: cpp + + class Pickleable { + public: + Pickleable(const std::string &value) : m_value(value) { } + const std::string &value() const { return m_value; } + + void setExtra(int extra) { m_extra = extra; } + int extra() const { return m_extra; } + private: + std::string m_value; + int m_extra = 0; + }; + +The binding code including the requisite ``__setstate__`` and ``__getstate__`` methods [#f3]_ +looks as follows: + +.. code-block:: cpp + + py::class_<Pickleable>(m, "Pickleable") + .def(py::init<std::string>()) + .def("value", &Pickleable::value) + .def("extra", &Pickleable::extra) + .def("setExtra", &Pickleable::setExtra) + .def("__getstate__", [](const Pickleable &p) { + /* Return a tuple that fully encodes the state of the object */ + return py::make_tuple(p.value(), p.extra()); + }) + .def("__setstate__", [](Pickleable &p, py::tuple t) { + if (t.size() != 2) + throw std::runtime_error("Invalid state!"); + + /* Invoke the in-place constructor. Note that this is needed even + when the object just has a trivial default constructor */ + new (&p) Pickleable(t[0].cast<std::string>()); + + /* Assign any additional state */ + p.setExtra(t[1].cast<int>()); + }); + +An instance can now be pickled as follows: + +.. code-block:: python + + try: + import cPickle as pickle # Use cPickle on Python 2.7 + except ImportError: + import pickle + + p = Pickleable("test_value") + p.setExtra(15) + data = pickle.dumps(p, 2) + +Note that only the cPickle module is supported on Python 2.7. The second +argument to ``dumps`` is also crucial: it selects the pickle protocol version +2, since the older version 1 is not supported. Newer versions are also fine—for +instance, specify ``-1`` to always use the latest available version. Beware: +failure to follow these instructions will cause important pybind11 memory +allocation routines to be skipped during unpickling, which will likely lead to +memory corruption and/or segmentation faults. + +.. seealso:: + + The file :file:`tests/test_pickling.cpp` contains a complete example + that demonstrates how to pickle and unpickle types using pybind11 in more + detail. + +.. [#f3] http://docs.python.org/3/library/pickle.html#pickling-class-instances + +Multiple Inheritance +==================== + +pybind11 can create bindings for types that derive from multiple base types +(aka. *multiple inheritance*). To do so, specify all bases in the template +arguments of the ``class_`` declaration: + +.. code-block:: cpp + + py::class_<MyType, BaseType1, BaseType2, BaseType3>(m, "MyType") + ... + +The base types can be specified in arbitrary order, and they can even be +interspersed with alias types and holder types (discussed earlier in this +document)---pybind11 will automatically find out which is which. The only +requirement is that the first template argument is the type to be declared. + +There are two caveats regarding the implementation of this feature: + +1. When only one base type is specified for a C++ type that actually has + multiple bases, pybind11 will assume that it does not participate in + multiple inheritance, which can lead to undefined behavior. In such cases, + add the tag ``multiple_inheritance``: + + .. code-block:: cpp + + py::class_<MyType, BaseType2>(m, "MyType", py::multiple_inheritance()); + + The tag is redundant and does not need to be specified when multiple base + types are listed. + +2. As was previously discussed in the section on :ref:`overriding_virtuals`, it + is easy to create Python types that derive from C++ classes. It is even + possible to make use of multiple inheritance to declare a Python class which + has e.g. a C++ and a Python class as bases. However, any attempt to create a + type that has *two or more* C++ classes in its hierarchy of base types will + fail with a fatal error message: ``TypeError: multiple bases have instance + lay-out conflict``. Core Python types that are implemented in C (e.g. + ``dict``, ``list``, ``Exception``, etc.) also fall under this combination + and cannot be combined with C++ types bound using pybind11 via multiple + inheritance. diff --git a/ext/pybind11/docs/advanced/exceptions.rst b/ext/pybind11/docs/advanced/exceptions.rst new file mode 100644 index 000000000..348337916 --- /dev/null +++ b/ext/pybind11/docs/advanced/exceptions.rst @@ -0,0 +1,142 @@ +Exceptions +########## + +Built-in exception translation +============================== + +When C++ code invoked from Python throws an ``std::exception``, it is +automatically converted into a Python ``Exception``. pybind11 defines multiple +special exception classes that will map to different types of Python +exceptions: + +.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| + ++--------------------------------------+------------------------------+ +| C++ exception type | Python exception type | ++======================================+==============================+ +| :class:`std::exception` | ``RuntimeError`` | ++--------------------------------------+------------------------------+ +| :class:`std::bad_alloc` | ``MemoryError`` | ++--------------------------------------+------------------------------+ +| :class:`std::domain_error` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`std::invalid_argument` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`std::length_error` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`std::out_of_range` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`std::range_error` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`pybind11::stop_iteration` | ``StopIteration`` (used to | +| | implement custom iterators) | ++--------------------------------------+------------------------------+ +| :class:`pybind11::index_error` | ``IndexError`` (used to | +| | indicate out of bounds | +| | accesses in ``__getitem__``, | +| | ``__setitem__``, etc.) | ++--------------------------------------+------------------------------+ +| :class:`pybind11::value_error` | ``ValueError`` (used to | +| | indicate wrong value passed | +| | in ``container.remove(...)`` | ++--------------------------------------+------------------------------+ +| :class:`pybind11::key_error` | ``KeyError`` (used to | +| | indicate out of bounds | +| | accesses in ``__getitem__``, | +| | ``__setitem__`` in dict-like | +| | objects, etc.) | ++--------------------------------------+------------------------------+ +| :class:`pybind11::error_already_set` | Indicates that the Python | +| | exception flag has already | +| | been initialized | ++--------------------------------------+------------------------------+ + +When a Python function invoked from C++ throws an exception, it is converted +into a C++ exception of type :class:`error_already_set` whose string payload +contains a textual summary. + +There is also a special exception :class:`cast_error` that is thrown by +:func:`handle::call` when the input arguments cannot be converted to Python +objects. + +Registering custom translators +============================== + +If the default exception conversion policy described above is insufficient, +pybind11 also provides support for registering custom exception translators. +To register a simple exception conversion that translates a C++ exception into +a new Python exception using the C++ exception's ``what()`` method, a helper +function is available: + +.. code-block:: cpp + + py::register_exception<CppExp>(module, "PyExp"); + +This call creates a Python exception class with the name ``PyExp`` in the given +module and automatically converts any encountered exceptions of type ``CppExp`` +into Python exceptions of type ``PyExp``. + +When more advanced exception translation is needed, the function +``py::register_exception_translator(translator)`` can be used to register +functions that can translate arbitrary exception types (and which may include +additional logic to do so). The function takes a stateless callable (e.g. a +function pointer or a lambda function without captured variables) with the call +signature ``void(std::exception_ptr)``. + +When a C++ exception is thrown, the registered exception translators are tried +in reverse order of registration (i.e. the last registered translator gets the +first shot at handling the exception). + +Inside the translator, ``std::rethrow_exception`` should be used within +a try block to re-throw the exception. One or more catch clauses to catch +the appropriate exceptions should then be used with each clause using +``PyErr_SetString`` to set a Python exception or ``ex(string)`` to set +the python exception to a custom exception type (see below). + +To declare a custom Python exception type, declare a ``py::exception`` variable +and use this in the associated exception translator (note: it is often useful +to make this a static declaration when using it inside a lambda expression +without requiring capturing). + + +The following example demonstrates this for a hypothetical exception classes +``MyCustomException`` and ``OtherException``: the first is translated to a +custom python exception ``MyCustomError``, while the second is translated to a +standard python RuntimeError: + +.. code-block:: cpp + + static py::exception<MyCustomException> exc(m, "MyCustomError"); + py::register_exception_translator([](std::exception_ptr p) { + try { + if (p) std::rethrow_exception(p); + } catch (const MyCustomException &e) { + exc(e.what()); + } catch (const OtherException &e) { + PyErr_SetString(PyExc_RuntimeError, e.what()); + } + }); + +Multiple exceptions can be handled by a single translator, as shown in the +example above. If the exception is not caught by the current translator, the +previously registered one gets a chance. + +If none of the registered exception translators is able to handle the +exception, it is handled by the default converter as described in the previous +section. + +.. seealso:: + + The file :file:`tests/test_exceptions.cpp` contains examples + of various custom exception translators and custom exception types. + +.. note:: + + You must call either ``PyErr_SetString`` or a custom exception's call + operator (``exc(string)``) for every exception caught in a custom exception + translator. Failure to do so will cause Python to crash with ``SystemError: + error return without exception set``. + + Exceptions that you do not plan to handle should simply not be caught, or + may be explicity (re-)thrown to delegate it to the other, + previously-declared existing exception translators. diff --git a/ext/pybind11/docs/advanced/functions.rst b/ext/pybind11/docs/advanced/functions.rst new file mode 100644 index 000000000..f291e8222 --- /dev/null +++ b/ext/pybind11/docs/advanced/functions.rst @@ -0,0 +1,311 @@ +Functions +######### + +Before proceeding with this section, make sure that you are already familiar +with the basics of binding functions and classes, as explained in :doc:`/basics` +and :doc:`/classes`. The following guide is applicable to both free and member +functions, i.e. *methods* in Python. + +Return value policies +===================== + +Python and C++ use fundamentally different ways of managing the memory and +lifetime of objects managed by them. This can lead to issues when creating +bindings for functions that return a non-trivial type. Just by looking at the +type information, it is not clear whether Python should take charge of the +returned value and eventually free its resources, or if this is handled on the +C++ side. For this reason, pybind11 provides a several `return value policy` +annotations that can be passed to the :func:`module::def` and +:func:`class_::def` functions. The default policy is +:enum:`return_value_policy::automatic`. + +Return value policies are tricky, and it's very important to get them right. +Just to illustrate what can go wrong, consider the following simple example: + +.. code-block:: cpp + + /* Function declaration */ + Data *get_data() { return _data; /* (pointer to a static data structure) */ } + ... + + /* Binding code */ + m.def("get_data", &get_data); // <-- KABOOM, will cause crash when called from Python + +What's going on here? When ``get_data()`` is called from Python, the return +value (a native C++ type) must be wrapped to turn it into a usable Python type. +In this case, the default return value policy (:enum:`return_value_policy::automatic`) +causes pybind11 to assume ownership of the static ``_data`` instance. + +When Python's garbage collector eventually deletes the Python +wrapper, pybind11 will also attempt to delete the C++ instance (via ``operator +delete()``) due to the implied ownership. At this point, the entire application +will come crashing down, though errors could also be more subtle and involve +silent data corruption. + +In the above example, the policy :enum:`return_value_policy::reference` should have +been specified so that the global data instance is only *referenced* without any +implied transfer of ownership, i.e.: + +.. code-block:: cpp + + m.def("get_data", &get_data, return_value_policy::reference); + +On the other hand, this is not the right policy for many other situations, +where ignoring ownership could lead to resource leaks. +As a developer using pybind11, it's important to be familiar with the different +return value policies, including which situation calls for which one of them. +The following table provides an overview of available policies: + +.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| + ++--------------------------------------------------+----------------------------------------------------------------------------+ +| Return value policy | Description | ++==================================================+============================================================================+ +| :enum:`return_value_policy::take_ownership` | Reference an existing object (i.e. do not create a new copy) and take | +| | ownership. Python will call the destructor and delete operator when the | +| | object's reference count reaches zero. Undefined behavior ensues when the | +| | C++ side does the same, or when the data was not dynamically allocated. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python. | +| | This policy is comparably safe because the lifetimes of the two instances | +| | are decoupled. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::move` | Use ``std::move`` to move the return value contents into a new instance | +| | that will be owned by Python. This policy is comparably safe because the | +| | lifetimes of the two instances (move source and destination) are decoupled.| ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::reference` | Reference an existing object, but do not take ownership. The C++ side is | +| | responsible for managing the object's lifetime and deallocating it when | +| | it is no longer used. Warning: undefined behavior will ensue when the C++ | +| | side deletes an object that is still referenced and used by Python. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::reference_internal` | Indicates that the lifetime of the return value is tied to the lifetime | +| | of a parent object, namely the implicit ``this``, or ``self`` argument of | +| | the called method or property. Internally, this policy works just like | +| | :enum:`return_value_policy::reference` but additionally applies a | +| | ``keep_alive<0, 1>`` *call policy* (described in the next section) that | +| | prevents the parent object from being garbage collected as long as the | +| | return value is referenced by Python. This is the default policy for | +| | property getters created via ``def_property``, ``def_readwrite``, etc. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::automatic` | This is the default return value policy, which falls back to the policy | +| | :enum:`return_value_policy::take_ownership` when the return value is a | +| | pointer. Otherwise, it uses :enum:`return_value::move` or | +| | :enum:`return_value::copy` for rvalue and lvalue references, respectively. | +| | See above for a description of what all of these different policies do. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the | +| | return value is a pointer. This is the default conversion policy for | +| | function arguments when calling Python functions manually from C++ code | +| | (i.e. via handle::operator()). You probably won't need to use this. | ++--------------------------------------------------+----------------------------------------------------------------------------+ + +Return value policies can also be applied to properties: + +.. code-block:: cpp + + class_<MyClass>(m, "MyClass") + .def_property("data", &MyClass::getData, &MyClass::setData, + py::return_value_policy::copy); + +Technically, the code above applies the policy to both the getter and the +setter function, however, the setter doesn't really care about *return* +value policies which makes this a convenient terse syntax. Alternatively, +targeted arguments can be passed through the :class:`cpp_function` constructor: + +.. code-block:: cpp + + class_<MyClass>(m, "MyClass") + .def_property("data" + py::cpp_function(&MyClass::getData, py::return_value_policy::copy), + py::cpp_function(&MyClass::setData) + ); + +.. warning:: + + Code with invalid return value policies might access unitialized memory or + free data structures multiple times, which can lead to hard-to-debug + non-determinism and segmentation faults, hence it is worth spending the + time to understand all the different options in the table above. + +.. note:: + + One important aspect of the above policies is that they only apply to + instances which pybind11 has *not* seen before, in which case the policy + clarifies essential questions about the return value's lifetime and + ownership. When pybind11 knows the instance already (as identified by its + type and address in memory), it will return the existing Python object + wrapper rather than creating a new copy. + +.. note:: + + The next section on :ref:`call_policies` discusses *call policies* that can be + specified *in addition* to a return value policy from the list above. Call + policies indicate reference relationships that can involve both return values + and parameters of functions. + +.. note:: + + As an alternative to elaborate call policies and lifetime management logic, + consider using smart pointers (see the section on :ref:`smart_pointers` for + details). Smart pointers can tell whether an object is still referenced from + C++ or Python, which generally eliminates the kinds of inconsistencies that + can lead to crashes or undefined behavior. For functions returning smart + pointers, it is not necessary to specify a return value policy. + +.. _call_policies: + +Additional call policies +======================== + +In addition to the above return value policies, further `call policies` can be +specified to indicate dependencies between parameters. There is currently just +one policy named ``keep_alive<Nurse, Patient>``, which indicates that the +argument with index ``Patient`` should be kept alive at least until the +argument with index ``Nurse`` is freed by the garbage collector. Argument +indices start at one, while zero refers to the return value. For methods, index +``1`` refers to the implicit ``this`` pointer, while regular arguments begin at +index ``2``. Arbitrarily many call policies can be specified. When a ``Nurse`` +with value ``None`` is detected at runtime, the call policy does nothing. + +This feature internally relies on the ability to create a *weak reference* to +the nurse object, which is permitted by all classes exposed via pybind11. When +the nurse object does not support weak references, an exception will be thrown. + +Consider the following example: here, the binding code for a list append +operation ties the lifetime of the newly added element to the underlying +container: + +.. code-block:: cpp + + py::class_<List>(m, "List") + .def("append", &List::append, py::keep_alive<1, 2>()); + +.. note:: + + ``keep_alive`` is analogous to the ``with_custodian_and_ward`` (if Nurse, + Patient != 0) and ``with_custodian_and_ward_postcall`` (if Nurse/Patient == + 0) policies from Boost.Python. + +.. seealso:: + + The file :file:`tests/test_keep_alive.cpp` contains a complete example + that demonstrates using :class:`keep_alive` in more detail. + +.. _python_objects_as_args: + +Python objects as arguments +=========================== + +pybind11 exposes all major Python types using thin C++ wrapper classes. These +wrapper classes can also be used as parameters of functions in bindings, which +makes it possible to directly work with native Python types on the C++ side. +For instance, the following statement iterates over a Python ``dict``: + +.. code-block:: cpp + + void print_dict(py::dict dict) { + /* Easily interact with Python types */ + for (auto item : dict) + std::cout << "key=" << item.first << ", " + << "value=" << item.second << std::endl; + } + +It can be exported: + +.. code-block:: cpp + + m.def("print_dict", &print_dict); + +And used in Python as usual: + +.. code-block:: pycon + + >>> print_dict({'foo': 123, 'bar': 'hello'}) + key=foo, value=123 + key=bar, value=hello + +For more information on using Python objects in C++, see :doc:`/advanced/pycpp/index`. + +Accepting \*args and \*\*kwargs +=============================== + +Python provides a useful mechanism to define functions that accept arbitrary +numbers of arguments and keyword arguments: + +.. code-block:: python + + def generic(*args, **kwargs): + ... # do something with args and kwargs + +Such functions can also be created using pybind11: + +.. code-block:: cpp + + void generic(py::args args, py::kwargs kwargs) { + /// .. do something with args + if (kwargs) + /// .. do something with kwargs + } + + /// Binding code + m.def("generic", &generic); + +The class ``py::args`` derives from ``py::tuple`` and ``py::kwargs`` derives +from ``py::dict``. Note that the ``kwargs`` argument is invalid if no keyword +arguments were actually provided. Please refer to the other examples for +details on how to iterate over these, and on how to cast their entries into +C++ objects. A demonstration is also available in +``tests/test_kwargs_and_defaults.cpp``. + +.. warning:: + + Unlike Python, pybind11 does not allow combining normal parameters with the + ``args`` / ``kwargs`` special parameters. + +Default arguments revisited +=========================== + +The section on :ref:`default_args` previously discussed basic usage of default +arguments using pybind11. One noteworthy aspect of their implementation is that +default arguments are converted to Python objects right at declaration time. +Consider the following example: + +.. code-block:: cpp + + py::class_<MyClass>("MyClass") + .def("myFunction", py::arg("arg") = SomeType(123)); + +In this case, pybind11 must already be set up to deal with values of the type +``SomeType`` (via a prior instantiation of ``py::class_<SomeType>``), or an +exception will be thrown. + +Another aspect worth highlighting is that the "preview" of the default argument +in the function signature is generated using the object's ``__repr__`` method. +If not available, the signature may not be very helpful, e.g.: + +.. code-block:: pycon + + FUNCTIONS + ... + | myFunction(...) + | Signature : (MyClass, arg : SomeType = <SomeType object at 0x101b7b080>) -> NoneType + ... + +The first way of addressing this is by defining ``SomeType.__repr__``. +Alternatively, it is possible to specify the human-readable preview of the +default argument manually using the ``arg_v`` notation: + +.. code-block:: cpp + + py::class_<MyClass>("MyClass") + .def("myFunction", py::arg_v("arg", SomeType(123), "SomeType(123)")); + +Sometimes it may be necessary to pass a null pointer value as a default +argument. In this case, remember to cast it to the underlying type in question, +like so: + +.. code-block:: cpp + + py::class_<MyClass>("MyClass") + .def("myFunction", py::arg("arg") = (SomeType *) nullptr); diff --git a/ext/pybind11/docs/advanced/misc.rst b/ext/pybind11/docs/advanced/misc.rst new file mode 100644 index 000000000..c13df7bf8 --- /dev/null +++ b/ext/pybind11/docs/advanced/misc.rst @@ -0,0 +1,229 @@ +Miscellaneous +############# + +.. _macro_notes: + +General notes regarding convenience macros +========================================== + +pybind11 provides a few convenience macros such as +:func:`PYBIND11_MAKE_OPAQUE` and :func:`PYBIND11_DECLARE_HOLDER_TYPE`, and +``PYBIND11_OVERLOAD_*``. Since these are "just" macros that are evaluated +in the preprocessor (which has no concept of types), they *will* get confused +by commas in a template argument such as ``PYBIND11_OVERLOAD(MyReturnValue<T1, +T2>, myFunc)``. In this case, the preprocessor assumes that the comma indicates +the beginning of the next parameter. Use a ``typedef`` to bind the template to +another name and use it in the macro to avoid this problem. + + +Global Interpreter Lock (GIL) +============================= + +The classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be +used to acquire and release the global interpreter lock in the body of a C++ +function call. In this way, long-running C++ code can be parallelized using +multiple Python threads. Taking :ref:`overriding_virtuals` as an example, this +could be realized as follows (important changes highlighted): + +.. code-block:: cpp + :emphasize-lines: 8,9,33,34 + + class PyAnimal : public Animal { + public: + /* Inherit the constructors */ + using Animal::Animal; + + /* Trampoline (need one for each virtual function) */ + std::string go(int n_times) { + /* Acquire GIL before calling Python code */ + py::gil_scoped_acquire acquire; + + PYBIND11_OVERLOAD_PURE( + std::string, /* Return type */ + Animal, /* Parent class */ + go, /* Name of function */ + n_times /* Argument(s) */ + ); + } + }; + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::class_<Animal, PyAnimal> animal(m, "Animal"); + animal + .def(py::init<>()) + .def("go", &Animal::go); + + py::class_<Dog>(m, "Dog", animal) + .def(py::init<>()); + + m.def("call_go", [](Animal *animal) -> std::string { + /* Release GIL before calling into (potentially long-running) C++ code */ + py::gil_scoped_release release; + return call_go(animal); + }); + + return m.ptr(); + } + + +Binding sequence data types, iterators, the slicing protocol, etc. +================================================================== + +Please refer to the supplemental example for details. + +.. seealso:: + + The file :file:`tests/test_sequences_and_iterators.cpp` contains a + complete example that shows how to bind a sequence data type, including + length queries (``__len__``), iterators (``__iter__``), the slicing + protocol and other kinds of useful operations. + + +Partitioning code over multiple extension modules +================================================= + +It's straightforward to split binding code over multiple extension modules, +while referencing types that are declared elsewhere. Everything "just" works +without any special precautions. One exception to this rule occurs when +extending a type declared in another extension module. Recall the basic example +from Section :ref:`inheritance`. + +.. code-block:: cpp + + py::class_<Pet> pet(m, "Pet"); + pet.def(py::init<const std::string &>()) + .def_readwrite("name", &Pet::name); + + py::class_<Dog>(m, "Dog", pet /* <- specify parent */) + .def(py::init<const std::string &>()) + .def("bark", &Dog::bark); + +Suppose now that ``Pet`` bindings are defined in a module named ``basic``, +whereas the ``Dog`` bindings are defined somewhere else. The challenge is of +course that the variable ``pet`` is not available anymore though it is needed +to indicate the inheritance relationship to the constructor of ``class_<Dog>``. +However, it can be acquired as follows: + +.. code-block:: cpp + + py::object pet = (py::object) py::module::import("basic").attr("Pet"); + + py::class_<Dog>(m, "Dog", pet) + .def(py::init<const std::string &>()) + .def("bark", &Dog::bark); + +Alternatively, you can specify the base class as a template parameter option to +``class_``, which performs an automated lookup of the corresponding Python +type. Like the above code, however, this also requires invoking the ``import`` +function once to ensure that the pybind11 binding code of the module ``basic`` +has been executed: + +.. code-block:: cpp + + py::module::import("basic"); + + py::class_<Dog, Pet>(m, "Dog") + .def(py::init<const std::string &>()) + .def("bark", &Dog::bark); + +Naturally, both methods will fail when there are cyclic dependencies. + +Note that compiling code which has its default symbol visibility set to +*hidden* (e.g. via the command line flag ``-fvisibility=hidden`` on GCC/Clang) can interfere with the +ability to access types defined in another extension module. Workarounds +include changing the global symbol visibility (not recommended, because it will +lead unnecessarily large binaries) or manually exporting types that are +accessed by multiple extension modules: + +.. code-block:: cpp + + #ifdef _WIN32 + # define EXPORT_TYPE __declspec(dllexport) + #else + # define EXPORT_TYPE __attribute__ ((visibility("default"))) + #endif + + class EXPORT_TYPE Dog : public Animal { + ... + }; + +Note also that it is possible (although would rarely be required) to share arbitrary +C++ objects between extension modules at runtime. Internal library data is shared +between modules using capsule machinery [#f6]_ which can be also utilized for +storing, modifying and accessing user-defined data. Note that an extension module +will "see" other extensions' data if and only if they were built with the same +pybind11 version. Consider the following example: + +.. code-block:: cpp + + auto data = (MyData *) py::get_shared_data("mydata"); + if (!data) + data = (MyData *) py::set_shared_data("mydata", new MyData(42)); + +If the above snippet was used in several separately compiled extension modules, +the first one to be imported would create a ``MyData`` instance and associate +a ``"mydata"`` key with a pointer to it. Extensions that are imported later +would be then able to access the data behind the same pointer. + +.. [#f6] https://docs.python.org/3/extending/extending.html#using-capsules + + +Generating documentation using Sphinx +===================================== + +Sphinx [#f4]_ has the ability to inspect the signatures and documentation +strings in pybind11-based extension modules to automatically generate beautiful +documentation in a variety formats. The python_example repository [#f5]_ contains a +simple example repository which uses this approach. + +There are two potential gotchas when using this approach: first, make sure that +the resulting strings do not contain any :kbd:`TAB` characters, which break the +docstring parsing routines. You may want to use C++11 raw string literals, +which are convenient for multi-line comments. Conveniently, any excess +indentation will be automatically be removed by Sphinx. However, for this to +work, it is important that all lines are indented consistently, i.e.: + +.. code-block:: cpp + + // ok + m.def("foo", &foo, R"mydelimiter( + The foo function + + Parameters + ---------- + )mydelimiter"); + + // *not ok* + m.def("foo", &foo, R"mydelimiter(The foo function + + Parameters + ---------- + )mydelimiter"); + +By default, pybind11 automatically generates and prepends a signature to the docstring of a function +registered with ``module::def()`` and ``class_::def()``. Sometimes this +behavior is not desirable, because you want to provide your own signature or remove +the docstring completely to exclude the function from the Sphinx documentation. +The class ``options`` allows you to selectively suppress auto-generated signatures: + +.. code-block:: cpp + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::options options; + options.disable_function_signatures(); + + m.def("add", [](int a, int b) { return a + b; }, "A function which adds two numbers"); + + return m.ptr(); + } + +Note that changes to the settings affect only function bindings created during the +lifetime of the ``options`` instance. When it goes out of scope at the end of the module's init function, +the default settings are restored to prevent unwanted side effects. + +.. [#f4] http://www.sphinx-doc.org +.. [#f5] http://github.com/pybind/python_example diff --git a/ext/pybind11/docs/advanced/pycpp/index.rst b/ext/pybind11/docs/advanced/pycpp/index.rst new file mode 100644 index 000000000..6885bdcff --- /dev/null +++ b/ext/pybind11/docs/advanced/pycpp/index.rst @@ -0,0 +1,13 @@ +Python C++ interface +#################### + +pybind11 exposes Python types and functions using thin C++ wrappers, which +makes it possible to conveniently call Python code from C++ without resorting +to Python's C API. + +.. toctree:: + :maxdepth: 2 + + object + numpy + utilities diff --git a/ext/pybind11/docs/advanced/pycpp/numpy.rst b/ext/pybind11/docs/advanced/pycpp/numpy.rst new file mode 100644 index 000000000..8b46b7c83 --- /dev/null +++ b/ext/pybind11/docs/advanced/pycpp/numpy.rst @@ -0,0 +1,299 @@ +.. _numpy: + +NumPy +##### + +Buffer protocol +=============== + +Python supports an extremely general and convenient approach for exchanging +data between plugin libraries. Types can expose a buffer view [#f2]_, which +provides fast direct access to the raw internal data representation. Suppose we +want to bind the following simplistic Matrix class: + +.. code-block:: cpp + + class Matrix { + public: + Matrix(size_t rows, size_t cols) : m_rows(rows), m_cols(cols) { + m_data = new float[rows*cols]; + } + float *data() { return m_data; } + size_t rows() const { return m_rows; } + size_t cols() const { return m_cols; } + private: + size_t m_rows, m_cols; + float *m_data; + }; + +The following binding code exposes the ``Matrix`` contents as a buffer object, +making it possible to cast Matrices into NumPy arrays. It is even possible to +completely avoid copy operations with Python expressions like +``np.array(matrix_instance, copy = False)``. + +.. code-block:: cpp + + py::class_<Matrix>(m, "Matrix") + .def_buffer([](Matrix &m) -> py::buffer_info { + return py::buffer_info( + m.data(), /* Pointer to buffer */ + sizeof(float), /* Size of one scalar */ + py::format_descriptor<float>::format(), /* Python struct-style format descriptor */ + 2, /* Number of dimensions */ + { m.rows(), m.cols() }, /* Buffer dimensions */ + { sizeof(float) * m.rows(), /* Strides (in bytes) for each index */ + sizeof(float) } + ); + }); + +The snippet above binds a lambda function, which can create ``py::buffer_info`` +description records on demand describing a given matrix. The contents of +``py::buffer_info`` mirror the Python buffer protocol specification. + +.. code-block:: cpp + + struct buffer_info { + void *ptr; + size_t itemsize; + std::string format; + int ndim; + std::vector<size_t> shape; + std::vector<size_t> strides; + }; + +To create a C++ function that can take a Python buffer object as an argument, +simply use the type ``py::buffer`` as one of its arguments. Buffers can exist +in a great variety of configurations, hence some safety checks are usually +necessary in the function body. Below, you can see an basic example on how to +define a custom constructor for the Eigen double precision matrix +(``Eigen::MatrixXd``) type, which supports initialization from compatible +buffer objects (e.g. a NumPy matrix). + +.. code-block:: cpp + + /* Bind MatrixXd (or some other Eigen type) to Python */ + typedef Eigen::MatrixXd Matrix; + + typedef Matrix::Scalar Scalar; + constexpr bool rowMajor = Matrix::Flags & Eigen::RowMajorBit; + + py::class_<Matrix>(m, "Matrix") + .def("__init__", [](Matrix &m, py::buffer b) { + typedef Eigen::Stride<Eigen::Dynamic, Eigen::Dynamic> Strides; + + /* Request a buffer descriptor from Python */ + py::buffer_info info = b.request(); + + /* Some sanity checks ... */ + if (info.format != py::format_descriptor<Scalar>::format()) + throw std::runtime_error("Incompatible format: expected a double array!"); + + if (info.ndim != 2) + throw std::runtime_error("Incompatible buffer dimension!"); + + auto strides = Strides( + info.strides[rowMajor ? 0 : 1] / sizeof(Scalar), + info.strides[rowMajor ? 1 : 0] / sizeof(Scalar)); + + auto map = Eigen::Map<Matrix, 0, Strides>( + static_cat<Scalar *>(info.ptr), info.shape[0], info.shape[1], strides); + + new (&m) Matrix(map); + }); + +For reference, the ``def_buffer()`` call for this Eigen data type should look +as follows: + +.. code-block:: cpp + + .def_buffer([](Matrix &m) -> py::buffer_info { + return py::buffer_info( + m.data(), /* Pointer to buffer */ + sizeof(Scalar), /* Size of one scalar */ + /* Python struct-style format descriptor */ + py::format_descriptor<Scalar>::format(), + /* Number of dimensions */ + 2, + /* Buffer dimensions */ + { (size_t) m.rows(), + (size_t) m.cols() }, + /* Strides (in bytes) for each index */ + { sizeof(Scalar) * (rowMajor ? m.cols() : 1), + sizeof(Scalar) * (rowMajor ? 1 : m.rows()) } + ); + }) + +For a much easier approach of binding Eigen types (although with some +limitations), refer to the section on :doc:`/advanced/cast/eigen`. + +.. seealso:: + + The file :file:`tests/test_buffers.cpp` contains a complete example + that demonstrates using the buffer protocol with pybind11 in more detail. + +.. [#f2] http://docs.python.org/3/c-api/buffer.html + +Arrays +====== + +By exchanging ``py::buffer`` with ``py::array`` in the above snippet, we can +restrict the function so that it only accepts NumPy arrays (rather than any +type of Python object satisfying the buffer protocol). + +In many situations, we want to define a function which only accepts a NumPy +array of a certain data type. This is possible via the ``py::array_t<T>`` +template. For instance, the following function requires the argument to be a +NumPy array containing double precision values. + +.. code-block:: cpp + + void f(py::array_t<double> array); + +When it is invoked with a different type (e.g. an integer or a list of +integers), the binding code will attempt to cast the input into a NumPy array +of the requested type. Note that this feature requires the +:file:``pybind11/numpy.h`` header to be included. + +Data in NumPy arrays is not guaranteed to packed in a dense manner; +furthermore, entries can be separated by arbitrary column and row strides. +Sometimes, it can be useful to require a function to only accept dense arrays +using either the C (row-major) or Fortran (column-major) ordering. This can be +accomplished via a second template argument with values ``py::array::c_style`` +or ``py::array::f_style``. + +.. code-block:: cpp + + void f(py::array_t<double, py::array::c_style | py::array::forcecast> array); + +The ``py::array::forcecast`` argument is the default value of the second +template parameter, and it ensures that non-conforming arguments are converted +into an array satisfying the specified requirements instead of trying the next +function overload. + +Structured types +================ + +In order for ``py::array_t`` to work with structured (record) types, we first need +to register the memory layout of the type. This can be done via ``PYBIND11_NUMPY_DTYPE`` +macro which expects the type followed by field names: + +.. code-block:: cpp + + struct A { + int x; + double y; + }; + + struct B { + int z; + A a; + }; + + PYBIND11_NUMPY_DTYPE(A, x, y); + PYBIND11_NUMPY_DTYPE(B, z, a); + + /* now both A and B can be used as template arguments to py::array_t */ + +Vectorizing functions +===================== + +Suppose we want to bind a function with the following signature to Python so +that it can process arbitrary NumPy array arguments (vectors, matrices, general +N-D arrays) in addition to its normal arguments: + +.. code-block:: cpp + + double my_func(int x, float y, double z); + +After including the ``pybind11/numpy.h`` header, this is extremely simple: + +.. code-block:: cpp + + m.def("vectorized_func", py::vectorize(my_func)); + +Invoking the function like below causes 4 calls to be made to ``my_func`` with +each of the array elements. The significant advantage of this compared to +solutions like ``numpy.vectorize()`` is that the loop over the elements runs +entirely on the C++ side and can be crunched down into a tight, optimized loop +by the compiler. The result is returned as a NumPy array of type +``numpy.dtype.float64``. + +.. code-block:: pycon + + >>> x = np.array([[1, 3],[5, 7]]) + >>> y = np.array([[2, 4],[6, 8]]) + >>> z = 3 + >>> result = vectorized_func(x, y, z) + +The scalar argument ``z`` is transparently replicated 4 times. The input +arrays ``x`` and ``y`` are automatically converted into the right types (they +are of type ``numpy.dtype.int64`` but need to be ``numpy.dtype.int32`` and +``numpy.dtype.float32``, respectively) + +Sometimes we might want to explicitly exclude an argument from the vectorization +because it makes little sense to wrap it in a NumPy array. For instance, +suppose the function signature was + +.. code-block:: cpp + + double my_func(int x, float y, my_custom_type *z); + +This can be done with a stateful Lambda closure: + +.. code-block:: cpp + + // Vectorize a lambda function with a capture object (e.g. to exclude some arguments from the vectorization) + m.def("vectorized_func", + [](py::array_t<int> x, py::array_t<float> y, my_custom_type *z) { + auto stateful_closure = [z](int x, float y) { return my_func(x, y, z); }; + return py::vectorize(stateful_closure)(x, y); + } + ); + +In cases where the computation is too complicated to be reduced to +``vectorize``, it will be necessary to create and access the buffer contents +manually. The following snippet contains a complete example that shows how this +works (the code is somewhat contrived, since it could have been done more +simply using ``vectorize``). + +.. code-block:: cpp + + #include <pybind11/pybind11.h> + #include <pybind11/numpy.h> + + namespace py = pybind11; + + py::array_t<double> add_arrays(py::array_t<double> input1, py::array_t<double> input2) { + auto buf1 = input1.request(), buf2 = input2.request(); + + if (buf1.ndim != 1 || buf2.ndim != 1) + throw std::runtime_error("Number of dimensions must be one"); + + if (buf1.size != buf2.size) + throw std::runtime_error("Input shapes must match"); + + /* No pointer is passed, so NumPy will allocate the buffer */ + auto result = py::array_t<double>(buf1.size); + + auto buf3 = result.request(); + + double *ptr1 = (double *) buf1.ptr, + *ptr2 = (double *) buf2.ptr, + *ptr3 = (double *) buf3.ptr; + + for (size_t idx = 0; idx < buf1.shape[0]; idx++) + ptr3[idx] = ptr1[idx] + ptr2[idx]; + + return result; + } + + PYBIND11_PLUGIN(test) { + py::module m("test"); + m.def("add_arrays", &add_arrays, "Add two NumPy arrays"); + return m.ptr(); + } + +.. seealso:: + + The file :file:`tests/test_numpy_vectorize.cpp` contains a complete + example that demonstrates using :func:`vectorize` in more detail. diff --git a/ext/pybind11/docs/advanced/pycpp/object.rst b/ext/pybind11/docs/advanced/pycpp/object.rst new file mode 100644 index 000000000..8fc165d16 --- /dev/null +++ b/ext/pybind11/docs/advanced/pycpp/object.rst @@ -0,0 +1,96 @@ +Python types +############ + +Available wrappers +================== + +All major Python types are available as thin C++ wrapper classes. These +can also be used as function parameters -- see :ref:`python_objects_as_args`. + +Available types include :class:`handle`, :class:`object`, :class:`bool_`, +:class:`int_`, :class:`float_`, :class:`str`, :class:`bytes`, :class:`tuple`, +:class:`list`, :class:`dict`, :class:`slice`, :class:`none`, :class:`capsule`, +:class:`iterable`, :class:`iterator`, :class:`function`, :class:`buffer`, +:class:`array`, and :class:`array_t`. + +Casting back and forth +====================== + +In this kind of mixed code, it is often necessary to convert arbitrary C++ +types to Python, which can be done using :func:`py::cast`: + +.. code-block:: cpp + + MyClass *cls = ..; + py::object obj = py::cast(cls); + +The reverse direction uses the following syntax: + +.. code-block:: cpp + + py::object obj = ...; + MyClass *cls = obj.cast<MyClass *>(); + +When conversion fails, both directions throw the exception :class:`cast_error`. + +Calling Python functions +======================== + +It is also possible to call python functions via ``operator()``. + +.. code-block:: cpp + + py::function f = <...>; + py::object result_py = f(1234, "hello", some_instance); + MyClass &result = result_py.cast<MyClass>(); + +Keyword arguments are also supported. In Python, there is the usual call syntax: + +.. code-block:: python + + def f(number, say, to): + ... # function code + + f(1234, say="hello", to=some_instance) # keyword call in Python + +In C++, the same call can be made using: + +.. code-block:: cpp + + using pybind11::literals; // to bring in the `_a` literal + f(1234, "say"_a="hello", "to"_a=some_instance); // keyword call in C++ + +Unpacking of ``*args`` and ``**kwargs`` is also possible and can be mixed with +other arguments: + +.. code-block:: cpp + + // * unpacking + py::tuple args = py::make_tuple(1234, "hello", some_instance); + f(*args); + + // ** unpacking + py::dict kwargs = py::dict("number"_a=1234, "say"_a="hello", "to"_a=some_instance); + f(**kwargs); + + // mixed keywords, * and ** unpacking + py::tuple args = py::make_tuple(1234); + py::dict kwargs = py::dict("to"_a=some_instance); + f(*args, "say"_a="hello", **kwargs); + +Generalized unpacking according to PEP448_ is also supported: + +.. code-block:: cpp + + py::dict kwargs1 = py::dict("number"_a=1234); + py::dict kwargs2 = py::dict("to"_a=some_instance); + f(**kwargs1, "say"_a="hello", **kwargs2); + +.. seealso:: + + The file :file:`tests/test_python_types.cpp` contains a complete + example that demonstrates passing native Python types in more detail. The + file :file:`tests/test_callbacks.cpp` presents a few examples of calling + Python functions from C++, including keywords arguments and unpacking. + +.. _PEP448: https://www.python.org/dev/peps/pep-0448/ diff --git a/ext/pybind11/docs/advanced/pycpp/utilities.rst b/ext/pybind11/docs/advanced/pycpp/utilities.rst new file mode 100644 index 000000000..ba0dbef88 --- /dev/null +++ b/ext/pybind11/docs/advanced/pycpp/utilities.rst @@ -0,0 +1,57 @@ +Utilities +######### + +Using Python's print function in C++ +==================================== + +The usual way to write output in C++ is using ``std::cout`` while in Python one +would use ``print``. Since these methods use different buffers, mixing them can +lead to output order issues. To resolve this, pybind11 modules can use the +:func:`py::print` function which writes to Python's ``sys.stdout`` for consistency. + +Python's ``print`` function is replicated in the C++ API including optional +keyword arguments ``sep``, ``end``, ``file``, ``flush``. Everything works as +expected in Python: + +.. code-block:: cpp + + py::print(1, 2.0, "three"); // 1 2.0 three + py::print(1, 2.0, "three", "sep"_a="-"); // 1-2.0-three + + auto args = py::make_tuple("unpacked", true); + py::print("->", *args, "end"_a="<-"); // -> unpacked True <- + +Evaluating Python expressions from strings and files +==================================================== + +pybind11 provides the :func:`eval` and :func:`eval_file` functions to evaluate +Python expressions and statements. The following example illustrates how they +can be used. + +Both functions accept a template parameter that describes how the argument +should be interpreted. Possible choices include ``eval_expr`` (isolated +expression), ``eval_single_statement`` (a single statement, return value is +always ``none``), and ``eval_statements`` (sequence of statements, return value +is always ``none``). + +.. code-block:: cpp + + // At beginning of file + #include <pybind11/eval.h> + + ... + + // Evaluate in scope of main module + py::object scope = py::module::import("__main__").attr("__dict__"); + + // Evaluate an isolated expression + int result = py::eval("my_variable + 10", scope).cast<int>(); + + // Evaluate a sequence of statements + py::eval<py::eval_statements>( + "print('Hello')\n" + "print('world!');", + scope); + + // Evaluate the statements in an separate Python file on disk + py::eval_file("script.py", scope); diff --git a/ext/pybind11/docs/advanced/smart_ptrs.rst b/ext/pybind11/docs/advanced/smart_ptrs.rst new file mode 100644 index 000000000..23072b6bf --- /dev/null +++ b/ext/pybind11/docs/advanced/smart_ptrs.rst @@ -0,0 +1,143 @@ +Smart pointers +############## + +std::unique_ptr +=============== + +Given a class ``Example`` with Python bindings, it's possible to return +instances wrapped in C++11 unique pointers, like so + +.. code-block:: cpp + + std::unique_ptr<Example> create_example() { return std::unique_ptr<Example>(new Example()); } + +.. code-block:: cpp + + m.def("create_example", &create_example); + +In other words, there is nothing special that needs to be done. While returning +unique pointers in this way is allowed, it is *illegal* to use them as function +arguments. For instance, the following function signature cannot be processed +by pybind11. + +.. code-block:: cpp + + void do_something_with_example(std::unique_ptr<Example> ex) { ... } + +The above signature would imply that Python needs to give up ownership of an +object that is passed to this function, which is generally not possible (for +instance, the object might be referenced elsewhere). + +std::shared_ptr +=============== + +The binding generator for classes, :class:`class_`, can be passed a template +type that denotes a special *holder* type that is used to manage references to +the object. If no such holder type template argument is given, the default for +a type named ``Type`` is ``std::unique_ptr<Type>``, which means that the object +is deallocated when Python's reference count goes to zero. + +It is possible to switch to other types of reference counting wrappers or smart +pointers, which is useful in codebases that rely on them. For instance, the +following snippet causes ``std::shared_ptr`` to be used instead. + +.. code-block:: cpp + + py::class_<Example, std::shared_ptr<Example> /* <- holder type */> obj(m, "Example"); + +Note that any particular class can only be associated with a single holder type. + +One potential stumbling block when using holder types is that they need to be +applied consistently. Can you guess what's broken about the following binding +code? + +.. code-block:: cpp + + class Child { }; + + class Parent { + public: + Parent() : child(std::make_shared<Child>()) { } + Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */ + private: + std::shared_ptr<Child> child; + }; + + PYBIND11_PLUGIN(example) { + py::module m("example"); + + py::class_<Child, std::shared_ptr<Child>>(m, "Child"); + + py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent") + .def(py::init<>()) + .def("get_child", &Parent::get_child); + + return m.ptr(); + } + +The following Python code will cause undefined behavior (and likely a +segmentation fault). + +.. code-block:: python + + from example import Parent + print(Parent().get_child()) + +The problem is that ``Parent::get_child()`` returns a pointer to an instance of +``Child``, but the fact that this instance is already managed by +``std::shared_ptr<...>`` is lost when passing raw pointers. In this case, +pybind11 will create a second independent ``std::shared_ptr<...>`` that also +claims ownership of the pointer. In the end, the object will be freed **twice** +since these shared pointers have no way of knowing about each other. + +There are two ways to resolve this issue: + +1. For types that are managed by a smart pointer class, never use raw pointers + in function arguments or return values. In other words: always consistently + wrap pointers into their designated holder types (such as + ``std::shared_ptr<...>``). In this case, the signature of ``get_child()`` + should be modified as follows: + +.. code-block:: cpp + + std::shared_ptr<Child> get_child() { return child; } + +2. Adjust the definition of ``Child`` by specifying + ``std::enable_shared_from_this<T>`` (see cppreference_ for details) as a + base class. This adds a small bit of information to ``Child`` that allows + pybind11 to realize that there is already an existing + ``std::shared_ptr<...>`` and communicate with it. In this case, the + declaration of ``Child`` should look as follows: + +.. _cppreference: http://en.cppreference.com/w/cpp/memory/enable_shared_from_this + +.. code-block:: cpp + + class Child : public std::enable_shared_from_this<Child> { }; + +.. _smart_pointers: + +Custom smart pointers +===================== + +pybind11 supports ``std::unique_ptr`` and ``std::shared_ptr`` right out of the +box. For any other custom smart pointer, transparent conversions can be enabled +using a macro invocation similar to the following. It must be declared at the +level before any binding code: + +.. code-block:: cpp + + PYBIND11_DECLARE_HOLDER_TYPE(T, SmartPtr<T>); + +The first argument of :func:`PYBIND11_DECLARE_HOLDER_TYPE` should be a +placeholder name that is used as a template parameter of the second argument. +Thus, feel free to use any identifier, but use it consistently on both sides; +also, don't use the name of a type that already exists in your codebase. + +Please take a look at the :ref:`macro_notes` before using this feature. + +.. seealso:: + + The file :file:`tests/test_smart_ptr.cpp` contains a complete example + that demonstrates how to work with custom reference-counting holder types + in more detail. |