Cross-extension-module dependencies

It is good programming practice to organize large projects as modules that interact with each other via well defined interfaces. With Boost.Python it is possible to reflect this organization at the C++ level at the Python level. This is, each logical C++ module can be organized as a separate Python extension module.

At first sight this might seem natural and straightforward. However, it is a fairly complex problem to establish cross-extension-module dependencies while maintaining the same ease of use Boost.Python provides for classes that are wrapped in the same extension module. To a large extent this complexity can be hidden from the author of a Boost.Python extension module, but not entirely.

The recipe

Suppose there is an extension module that exposes certain instances of the C++ std::vector template library such that it can be used from Python in the following manner:

import std_vector
v = std_vector.double([1, 2, 3, 4])
v.push_back(5)
v.size()

Suppose the std_vector module is done well and reflects all C++ functions that are useful at the Python level, for all C++ built-in data types (std_vector.int, std_vector.long, etc.).

Suppose further that there is statistic module with a C++ class that has constructors or member functions that use or return a std::vector. For example:

class xy {
  public:
    xy(const std::vector<double>& x, const std::vector<double>& y) : m_x(x), m_y(y) {}
    const std::vector<double>& x() const { return m_x; }
    const std::vector<double>& y() const { return m_y; }
    double correlation();
  private:
    std::vector<double> m_x;
    std::vector<double> m_y;
}

What is more natural than reusing the std_vector extension module to expose these constructors or functions to Python?

Unfortunately, what seems natural needs a little work in both the std_vector and the statistics module.

In the std_vector extension module, std::vector<double> is exposed to Python in the usual way with the class_builder<> template. To also enable the automatic conversion of std::vector<double> function arguments or return values in other Boost.Python C++ modules, the converters that convert a std::vector<double> C++ object to a Python object and vice versa (i.e. the to_python() and from_python() template functions) have to be exported. For example:

  #include <boost/python/cross_module.hpp>
  //...
  class_builder<std::vector<double> > v_double(std_vector_module, "double");
  export_converters(v_double);

In the extension module that wraps class xy we can now import these converters with the import_converters<> template. For example:

  #include <boost/python/cross_module.hpp>
  //...
  import_converters<std::vector<double> > v_double_converters("std_vector", "double");

That is all. All the attributes that are defined for std_vector.double in the std_vector Boost.Python module will be available for the returned objects of xy.x() and xy.y(). Similarly, the constructor for xy will accept objects that were created by the std_vectormodule.

Placement of `import_converters<>` template instantiations

import_converts<> can be viewed as a drop-in replacement for class_wrapper<>, and the recommendations for the placement of class_wrapper<> template instantiations also apply to to import_converts<>. In particular, it is important that an instantiation of class_wrapper<> is visible to any code which wraps a C++ function with a T, T*, const T&, etc. parameter or return value. Therefore you may want to group all class_wrapper<> and import_converts<> instantiations at the top of your module's init function, then def() the member functions later to avoid problems with inter-class dependencies.

Non-copyable types

export_converters() instantiates C++ template functions that invoke the copy constructor of the wrapped type. For a type that is non-copyable this will result in compile-time error messages. In such a case, export_converters_noncopyable() can be used to export the converters that do not involve the copy constructor of the wrapped type. For example:

class_builder<store> py_store(your_module, "store");
export_converters_noncopyable(py_store);

The corresponding import_converters<> statement does not need any special attention:

import_converters<store> py_store("noncopyable_export", "store");

Python module search path

The std_vector and statistics modules can now be used in the following way:

import std_vector
import statistics
x = std_vector.double([1, 2, 3, 4])
y = std_vector.double([2, 4, 6, 8])
xy = statistics.xy(x, y)
xy.correlation()

In this example it is clear that Python has to be able to find both the std_vector and the statistics extension module. In other words, both extension modules need to be in the Python module search path (sys.path).

The situation is not always this obvious. Suppose the statistics module has a random() function that returns a vector of random numbers with a given length:

import statistics
x = statistics.random(5)
y = statistics.random(5)
xy = statistics.xy(x, y)
xy.correlation()

A naive user will not easily anticipate that the std_vector module is used to pass the x and y vectors around. If the std_vector module is in the Python module search path, this form of ignorance is of no harm. On the contrary, we are glad that we do not have to bother the user with details like this.

If the std_vector module is not in the Python module search path, a Python exception will be raised:

Traceback (innermost last):
  File "foo.py", line 2, in ?
    x = statistics.random(5)
ImportError: No module named std_vector

As is the case with any system of a non-trivial complexity, it is important that the setup is consistent and complete.

Two-way module dependencies

Boost.Python supports two-way module dependencies. This is best illustrated by a simple example.

Suppose there is a module ivect that implements vectors of integers, and a similar module dvect that implements vectors of doubles. We want to be able do convert an integer vector to a double vector and vice versa. For example:

import ivect
iv = ivect.ivect((1,2,3,4,5))
dv = iv.as_dvect()

The last expression will implicitly import the dvect module in order to enable the conversion of the C++ representation of dvect to a Python object. The analogous is possible for a dvect:

import dvect
dv = dvect.dvect((1,2,3,4,5))
iv = dv.as_ivect()

Now the ivect module is imported implicitly.

Note that the two-way dependencies are possible because the dependencies are resolved only when needed. This is, the initialization of the ivect module does not rely on the dvect module, and vice versa. Only if as_dvect() or as_ivect() is actually invoked will the corresponding module be implicitly imported. This also means that, for example, the dvect module does not have to be available at all if as_dvect() is never used.

Clarification of compile-time and link-time dependencies

Boost.Python's support for resolving cross-module dependencies at runtime does not imply that compile-time dependencies are eliminated. For example, the statistics extension module in the example above will need to #include <vector>. This is immediately obvious from the definition of class xy.

If a library is wrapped that consists of both header files and compiled components (e.g. libdvect.a, dvect.lib, etc.), both the Boost.Python extension module with the export_converters() statement and the module with the import_converters<> statement need to be linked against the object library. Ideally one would build a shared library (e.g. libdvect.so, dvect.dll, etc.). However, this introduces the issue of having to configure the search path for the dynamic loading correctly. For small libraries it is therefore often more convenient to ignore the fact that the object files are loaded into memory more than once.

Summary of motivation for cross-module support

The main purpose of Boost.Python's cross-module support is to allow for a modular system layout. With this support it is straightforward to reflect C++ code organization at the Python level. Without the cross-module support, a multi-purpose module like std_vector would be impractical because the entire wrapper code would somehow have to be duplicated in all extension modules that use it, making them harder to maintain and harder to build.

Another motivation for the cross-module support is that two extension modules that wrap the same class cannot both be imported into Python. For example, if there are two modules A and B that both wrap a given class X, this will work:

import A
x = A.X()

This will also work:

import B
x = B.X()

However, this will fail:

import A
import B
python: /net/cci/rwgk/boost/boost/python/detail/extension_class.hpp:866:
static void boost::python::detail::class_registry<X>::register_class(boost::python::detail::extension_class_base *):
Assertion `static_class_object == 0' failed.
Abort

A good solution is to wrap class X only once. Depending on the situation, this could be done by module A or B, or an additional small extension module that only wraps and exports class X.

Finally, there can be important psychological or political reasons for using the cross-module support. If a group of classes is lumped together with many others in a huge module, the authors will have difficulties in being identified with their work. The situation is much more transparent if the work is represented by a module with a recognizable name. This is not just a question of strong egos, but also of getting credit and funding.

Why not use `export_converters()` universally?

There is some overhead associated with the Boost.Python cross-module support. Depending on the platform, the size of the code generated by export_converters() is roughly 10%-20% of that generated by class_builder<>. For a large extension module with many wrapped classes, this could mean a significant difference. Therefore the general recommendation is to use export_converters() only for classes that are likely to be used as function arguments or return values in other modules.

© Copyright Ralf W. Grosse-Kunstleve 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.

Updated: April 2001