123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736 |
- [chapter Frequently Asked Questions (FAQs)
- [quickbook 1.7]
- [id faq]
- ]
- [section How can I wrap a function which takes a function pointer as an argument?]
- If what you're trying to do is something like this:
- ``
- typedef boost::function<void (string s) > funcptr;
- void foo(funcptr fp)
- {
- fp("hello,world!");
- }
- BOOST_PYTHON_MODULE(test)
- {
- def("foo",foo);
- }
- ``
- And then:
- ``
- >>> def hello(s):
- ... print s
- ...
- >>> foo(hello)
- hello, world!
- ``
- The short answer is: "you can't". This is not a
- Boost.Python limitation so much as a limitation of C++. The
- problem is that a Python function is actually data, and the only
- way of associating data with a C++ function pointer is to store it
- in a static variable of the function. The problem with that is
- that you can only associate one piece of data with every C++
- function, and we have no way of compiling a new C++ function
- on-the-fly for every Python function you decide to pass
- to `foo`. In other words, this could work if the C++
- function is always going to invoke the /same/ Python
- function, but you probably don't want that.
- If you have the luxury of changing the C++ code you're
- wrapping, pass it an `object` instead and call that;
- the overloaded function call operator will invoke the Python
- function you pass it behind the `object`.
- [endsect]
- [section I'm getting the "attempt to return dangling reference" error.
- What am I doing wrong?]
- That exception is protecting you from causing a nasty crash. It usually
- happens in response to some code like this:
- ``
- period const &get_floating_frequency() const
- {
- return boost::python::call_method<period const &>(
- m_self,"get_floating_frequency");
- }
- ``
- And you get:
- ``
- ReferenceError: Attempt to return dangling reference to object of type:
- class period
- ``
- In this case, the Python method invoked by `call_method`
- constructs a new Python object. You're trying to return a reference to a
- C++ object (an instance of `class period`) contained within
- and owned by that Python object. Because the called method handed back a
- brand new object, the only reference to it is held for the duration of
- `get_floating_frequency()` above. When the function returns,
- the Python object will be destroyed, destroying the instance of
- `class period`, and leaving the returned reference dangling.
- That's already undefined behavior, and if you try to do anything with
- that reference you're likely to cause a crash. Boost.Python detects this
- situation at runtime and helpfully throws an exception instead of letting
- you do that.
- [endsect]
- [section Is `return_internal_reference` efficient?]
- [*Q:] /I have an object composed of 12 doubles. A `const&` to
- this object is returned by a member function of another class. From the
- viewpoint of using the returned object in Python I do not care if I get
- a copy or a reference to the returned object. In Boost.Python I have the
- choice of using `copy_const_reference` or `return_internal_reference`.
- Are there considerations that would lead me to prefer one over the other,
- such as size of generated code or memory overhead?/
- [*A:] `copy_const_reference` will make an instance with storage
- for one of your objects, `size = base_size + 12 * sizeof(double)`.
- `return_internal_reference` will make an instance with storage for a
- pointer to one of your objects, `size = base_size + sizeof(void*)`.
- However, it will also create a weak reference object which goes in the
- source object's weakreflist and a special callback object to manage the
- lifetime of the internally-referenced object. My guess?
- `copy_const_reference` is your friend here, resulting in less overall
- memory use and less fragmentation, also probably fewer total
- cycles.
- [endsect]
- [section How can I wrap functions which take C++ containers as arguments?]
- Ralf W. Grosse-Kunstleve provides these notes:
- # Using the regular `class_<>` wrapper:
- ``
- class_<std::vector<double> >("std_vector_double")
- .def(...)
- ...
- ;
- ``
- This can be moved to a template so that several types (`double`, `int`,
- `long`, etc.) can be wrapped with the same code. This technique is used
- in the file `scitbx/include/scitbx/array_family/boost_python/flex_wrapper.h`
- in the "scitbx" package. The file could easily be modified for
- wrapping `std::vector<>` instantiations.
- This type of C++/Python binding is most suitable for containers
- that may contain a large number of elements (>10000).
- # Using custom rvalue converters. Boost.Python "rvalue converters"
- match function signatures such as:
- ``
- void foo(std::vector<double> const &array); // pass by const-reference
- void foo(std::vector<double> array); // pass by value
- ``
- Some custom rvalue converters are implemented in the file
- `scitbx/include/scitbx/boost_python/container_conversions.h`
- This code can be used to convert from C++ container types such as
- `std::vector<>` or `std::list<>` to Python tuples and vice
- versa. A few simple examples can be found in the file
- `scitbx/array_family/boost_python/regression_test_module.cpp`
- Automatic C++ container <-> Python tuple conversions are most
- suitable for containers of moderate size. These converters generate
- significantly less object code compared to alternative 1 above.
- A disadvantage of using alternative 2 is that operators such as
- arithmetic +,-,*,/,% are not available. It would be useful to have custom
- rvalue converters that convert to a "math_array" type instead of tuples.
- This is currently not implemented but is possible within the framework of
- Boost.Python V2 as it will be released in the next couple of weeks. [ed.:
- this was posted on 2002/03/10]
- It would also be useful to also have "custom lvalue converters" such
- as `std::vector<>` <-> Python list. These converters would
- support the modification of the Python list from C++. For example:
- C++:
- ``
- void foo(std::vector<double> &array)
- {
- for(std::size_t i=0;i<array.size();i++) {
- array[i] *= 2;
- }
- }
- ``
- Python: [python]
- ``
- >>> l = [1, 2, 3]
- >>> foo(l)
- >>> print l
- [2, 4, 6]
- ``
- Custom lvalue converters require changes to the Boost.Python core library
- and are currently not available.
- P.S.:
- The "scitbx" files referenced above are available via anonymous
- CVS:
- ``
- cvs -d:pserver:anonymous@cvs.cctbx.sourceforge.net:/cvsroot/cctbx login
- cvs -d:pserver:anonymous@cvs.cctbx.sourceforge.net:/cvsroot/cctbx co scitbx
- ``
- [endsect]
- [section fatal error C1204:Compiler limit:internal structure overflow]
- [*Q:] /I get this error message when compiling a large source file. What can I do?/
- [*A:] You have two choices:
- # Upgrade your compiler (preferred)
- # Break your source file up into multiple translation units.
- `my_module.cpp`: [c++]
- ``
- ...
- void more_of_my_module();
- BOOST_PYTHON_MODULE(my_module)
- {
- def("foo", foo);
- def("bar", bar);
- ...
- more_of_my_module();
- }
- ``
- `more_of_my_module.cpp`:
- ``
- void more_of_my_module()
- {
- def("baz", baz);
- ...
- }
- ``
- If you find that a `class_<...>` declaration
- can't fit in a single source file without triggering the error, you
- can always pass a reference to the `class_` object to a
- function in another source file, and call some of its member
- functions (e.g. `.def(...)`) in the auxilliary source
- file:
- `more_of_my_class.cpp`:
- ``
- void more_of_my_class(class<my_class>& x)
- {
- x
- .def("baz", baz)
- .add_property("xx", &my_class::get_xx, &my_class::set_xx)
- ;
- ...
- }
- ``
- [endsect]
- [section How do I debug my Python extensions?]
- Greg Burley gives the following answer for Unix GCC users:
- [:Once you have created a boost python extension for your c++ library or
- class, you may need to debug the code. Afterall this is one of the
- reasons for wrapping the library in python. An expected side-effect or
- benefit of using BPL is that debugging should be isolated to the c++
- library that is under test, given that python code is minimal and
- boost::python either works or it doesn't. (ie. While errors can occur
- when the wrapping method is invalid, most errors are caught by the
- compiler ;-).
- The basic steps required to initiate a gdb session to debug a c++
- library via python are shown here. Note, however that you should start
- the gdb session in the directory that contains your BPL my_ext.so
- module.
- ``
- (gdb) target exec python
- (gdb) run
- >>> from my_ext import *
- >>> [C-c]
- (gdb) break MyClass::MyBuggyFunction
- (gdb) cont
- >>> pyobj = MyClass()
- >>> pyobj.MyBuggyFunction()
- Breakpoint 1, MyClass::MyBuggyFunction ...
- Current language: auto; currently c++
- (gdb) do debugging stuff
- ``
- ]
- Greg's approach works even better using Emacs' "gdb"
- command, since it will show you each line of source as you step through it.
- On *Windows*, my favorite debugging solution is the debugger that
- comes with Microsoft Visual C++ 7. This debugger seems to work with code
- generated by all versions of Microsoft and Metrowerks toolsets; it's rock
- solid and "just works" without requiring any special tricks from the
- user.
- Raoul Gough has provided the following for gdb on Windows:
- [:gdb support for Windows DLLs has improved lately, so it is
- now possible to debug Python extensions using a few
- tricks. Firstly, you will need an up-to-date gdb with support
- for minimal symbol extraction from a DLL. Any gdb from version 6
- onwards, or Cygwin gdb-20030214-1 and onwards should do. A
- suitable release will have a section in the gdb.info file under
- Configuration - Native - Cygwin Native -
- Non-debug DLL symbols. Refer to that info section for more
- details of the procedures outlined here.
- Secondly, it seems necessary to set a breakpoint in the
- Python interpreter, rather than using ^C to break execution. A
- good place to set this breakpoint is PyOS_Readline, which will
- stop execution immediately before reading each interactive
- Python command. You have to let Python start once under the
- debugger, so that it loads its own DLL, before you can set the
- breakpoint:
- ``
- $ gdb python
- GNU gdb 2003-09-02-cvs (cygwin-special)
- [...]
- (gdb) run
- Starting program: /cygdrive/c/Python22/python.exe
- Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32
- Type "help", "copyright", "credits" or "license" for more information.
- >>> ^Z
- Program exited normally.
- (gdb) break *&PyOS_Readline
- Breakpoint 1 at 0x1e04eff0
- (gdb) run
- Starting program: /cygdrive/c/Python22/python.exe
- Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32
- Type "help", "copyright", "credits" or "license" for more information.
- Breakpoint 1, 0x1e04eff0 in python22!PyOS_Readline ()
- from /cygdrive/c/WINNT/system32/python22.dll
- (gdb) cont
- Continuing.
- >>> from my_ext import *
- Breakpoint 1, 0x1e04eff0 in python22!PyOS_Readline ()
- from /cygdrive/c/WINNT/system32/python22.dll
- (gdb) # my_ext now loaded (with any debugging symbols it contains)
- ``
- ]
- [h2 Debugging extensions through Boost.Build]
- If you are launching your extension module tests with _bb_ using the
- `boost-python-runtest` rule, you can ask it to launch your
- debugger for you by adding "--debugger=/debugger/" to your bjam
- command-line:
- ``
- bjam -sTOOLS=vc7.1 "--debugger=devenv /debugexe" test
- bjam -sTOOLS=gcc -sPYTHON_LAUNCH=gdb test
- ``
- It can also be extremely useful to add the `-d+2` option when
- you run your test, because Boost.Build will then show you the exact
- commands it uses to invoke it. This will invariably involve setting up
- PYTHONPATH and other important environment variables such as
- LD_LIBRARY_PATH which may be needed by your debugger in order to get
- things to work right.
- [endsect]
- [section Why doesn't my `*=` operator work?]
- [*Q:] ['I have exported my class to python, with many overloaded
- operators. it works fine for me except the `*=`
- operator. It always tells me "can't multiply sequence with non int
- type". If I use `p1.__imul__(p2)` instead of
- `p1 *= p2`, it successfully executes my code. What's
- wrong with me?]
- [*A:] There's nothing wrong with you. This is a bug in Python
- 2.2. You can see the same effect in Pure Python (you can learn a lot
- about what's happening in Boost.Python by playing with new-style
- classes in Pure Python).
- ``
- >>> class X(object):
- ... def __imul__(self, x):
- ... print 'imul'
- ...
- >>> x = X()
- >>> x *= 1
- ``
- To cure this problem, all you need to do is upgrade your Python to
- version 2.2.1 or later.
- [endsect]
- [section Does Boost.Python work with Mac OS X?]
- It is known to work under 10.2.8 and 10.3 using
- Apple's gcc 3.3 compiler:
- ``gcc (GCC) 3.3 20030304 (Apple Computer, Inc. build 1493)``
- Under 10.2.8 get the August 2003 gcc update (free at [@http://connect.apple.com]).
- Under 10.3 get the Xcode Tools v1.0 (also free).
- Python 2.3 is required. The Python that ships with 10.3 is
- fine. Under 10.2.8 use these commands to install Python
- as a framework:
- ``./configure --enable-framework
- make
- make frameworkinstall``
- The last command requires root privileges because the target
- directory is `/Library/Frameworks/Python.framework/Versions/2.3`.
- However, the installation does not interfere with the Python
- version that ships with 10.2.8.
- It is also crucial to increase the `stacksize` before
- starting compilations, e.g.:
- ``limit stacksize 8192k``
- If the `stacksize` is too small the build might crash with
- internal compiler errors.
- Sometimes Apple's compiler exhibits a bug by printing an error
- like the following while compiling a
- `boost::python::class_<your_type>`
- template instantiation:
- ``
- .../inheritance.hpp:44: error: cannot
- dynamic_cast `p' (of type `struct cctbx::boost_python::<unnamed>::add_pair*
- ') to type `void*' (source type is not polymorphic)
- ``
- We do not know a general workaround, but if the definition of
- `your_type` can be modified the following was found
- to work in all cases encountered so far:
- ``
- struct your_type
- {
- // before defining any member data
- #if defined(__MACH__) && defined(__APPLE_CC__) && __APPLE_CC__ == 1493
- bool dummy_;
- #endif
- // now your member data, e.g.
- double x;
- int j;
- // etc.
- };
- ``
- [endsect]
- [section How can I find the existing PyObject that holds a C++ object?]
- [: "I am wrapping a function that always returns a pointer to an
- already-held C++ object."]
- One way to do that is to hijack the mechanisms used for wrapping a class
- with virtual functions. If you make a wrapper class with an initial
- PyObject* constructor argument and store that PyObject* as "self", you
- can get back to it by casting down to that wrapper type in a thin wrapper
- function. For example:
- ``
- class X { X(int); virtual ~X(); ... };
- X* f(); // known to return Xs that are managed by Python objects
- // wrapping code
- struct X_wrap : X
- {
- X_wrap(PyObject* self, int v) : self(self), X(v) {}
- PyObject* self;
- };
- handle<> f_wrap()
- {
- X_wrap* xw = dynamic_cast<X_wrap*>(f());
- assert(xw != 0);
- return handle<>(borrowed(xw->self));
- }
- ...
- def("f", f_wrap());
- class_<X,X_wrap,boost::noncopyable>("X", init<int>())
- ...
- ;
- ``
- Of course, if X has no virtual functions you'll have to use
- `static_cast` instead of `dynamic_cast` with no
- runtime check that it's valid. This approach also only works if the
- `X` object was constructed from Python, because
- `X`\ s constructed from C++ are of course never
- `X_wrap` objects.
- Another approach to this requires you to change your C++ code a bit;
- if that's an option for you it might be a better way to go. work we've
- been meaning to get to anyway. When a `shared_ptr<X>` is
- converted from Python, the shared_ptr actually manages a reference to the
- containing Python object. When a shared_ptr<X> is converted back to
- Python, the library checks to see if it's one of those "Python object
- managers" and if so just returns the original Python object. So you could
- just write `object(p)` to get the Python object back. To
- exploit this you'd have to be able to change the C++ code you're wrapping
- so that it deals with shared_ptr instead of raw pointers.
- There are other approaches too. The functions that receive the Python
- object that you eventually want to return could be wrapped with a thin
- wrapper that records the correspondence between the object address and
- its containing Python object, and you could have your f_wrap function
- look in that mapping to get the Python object out.
- [endsect]
- [section How can I wrap a function which needs to take ownership of a raw pointer?]
- [*Q:] Part of an API that I'm wrapping goes something like this:
- ``
- struct A {}; struct B { void add( A* ); }
- where B::add() takes ownership of the pointer passed to it.
- ``
- However:
- ``
- a = mod.A()
- b = mod.B()
- b.add( a )
- del a
- del b
- # python interpreter crashes
- # later due to memory corruption.
- ``
- Even binding the lifetime of a to b via `with_custodian_and_ward` doesn't prevent
- the python object a from ultimately trying to delete the object it's pointing to.
- Is there a way to accomplish a 'transfer-of-ownership' of a wrapped C++ object?
- --Bruce Lowery
- Yes: Make sure the C++ object is held by auto_ptr:
- ``
- class_<A, std::auto_ptr<A> >("A")
- ...
- ;
- ``
- Then make a thin wrapper function which takes an auto_ptr parameter:
- ``
- void b_insert(B &b, std::auto_ptr<A> a)
- {
- b.insert(a.get());
- a.release();
- }
- ``
- Wrap that as B.add. Note that pointers returned via `manage_new_object`
- will also be held by `auto_ptr`, so this transfer-of-ownership
- will also work correctly.
- [endsect]
- [section Compilation takes too much time and eats too much memory!
- What can I do to make it faster?]
- Please refer to the `Reducing Compiling Time` section in the _tutorial_.
- [endsect]
- [section How do I create sub-packages using Boost.Python?]
- Please refer to the `Creating Packages` section in the _tutorial_.
- [endsect]
- [section error C2064: term does not evaluate to a function taking 2 arguments]
- /Niall Douglas provides these notes:/
- If you see Microsoft Visual C++ 7.1 (MS Visual Studio .NET 2003) issue
- an error message like the following it is most likely due to a bug
- in the compiler:
- ``
- boost\boost\python\detail\invoke.hpp(76):
- error C2064: term does not evaluate to a function taking 2 arguments"
- ``
- This message is triggered by code like the following:
- ``
- #include <boost/python.hpp>
- using namespace boost::python;
- class FXThread
- {
- public:
- bool setAutoDelete(bool doso) throw();
- };
- void Export_FXThread()
- {
- class_< FXThread >("FXThread")
- .def("setAutoDelete", &FXThread::setAutoDelete)
- ;
- }
- ``
- The bug is related to the `throw()` modifier.
- As a workaround cast off the modifier. E.g.:
- ``
- .def("setAutoDelete", (bool (FXThread::*)(bool)) &FXThread::setAutoDelete)
- ``
- (The bug has been reported to Microsoft.)
- [endsect]
- [section How can I automatically convert my custom string type to and from a Python string?]
- /Ralf W. Grosse-Kunstleve provides these notes:/
- Below is a small, self-contained demo extension module that shows
- how to do this. Here is the corresponding trivial test:
- ``
- import custom_string
- assert custom_string.hello() == "Hello world."
- assert custom_string.size("california") == 10
- ``
- If you look at the code you will find:
- * A custom `to_python` converter (easy):
- `custom_string_to_python_str`
- *A custom lvalue converter (needs more code):
- `custom_string_from_python_str`
- The custom converters are registered in the global Boost.Python
- registry near the top of the module initialization function. Once
- flow control has passed through the registration code the automatic
- conversions from and to Python strings will work in any module
- imported in the same process.
- ``
- #include <boost/python/module.hpp>
- #include <boost/python/def.hpp>
- #include <boost/python/to_python_converter.hpp>
- namespace sandbox { namespace {
- class custom_string
- {
- public:
- custom_string() {}
- custom_string(std::string const &value) : value_(value) {}
- std::string const &value() const { return value_; }
- private:
- std::string value_;
- };
- struct custom_string_to_python_str
- {
- static PyObject* convert(custom_string const &s)
- {
- return boost::python::incref(boost::python::object(s.value()).ptr());
- }
- };
- struct custom_string_from_python_str
- {
- custom_string_from_python_str()
- {
- boost::python::converter::registry::push_back(
- &convertible,
- &construct,
- boost::python::type_id<custom_string>());
- }
- static void* convertible(PyObject* obj_ptr)
- {
- if (!PyString_Check(obj_ptr)) return 0;
- return obj_ptr;
- }
- static void construct(
- PyObject* obj_ptr,
- boost::python::converter::rvalue_from_python_stage1_data* data)
- {
- const char* value = PyString_AsString(obj_ptr);
- if (value == 0) boost::python::throw_error_already_set();
- void* storage = (
- (boost::python::converter::rvalue_from_python_storage<custom_string>*)
- data)->storage.bytes;
- new (storage) custom_string(value);
- data->convertible = storage;
- }
- };
- custom_string hello() { return custom_string("Hello world."); }
- std::size_t size(custom_string const &s) { return s.value().size(); }
- void init_module()
- {
- using namespace boost::python;
- boost::python::to_python_converter<
- custom_string,
- custom_string_to_python_str>();
- custom_string_from_python_str();
- def("hello", hello);
- def("size", size);
- }
- }} // namespace sandbox::<anonymous>
- BOOST_PYTHON_MODULE(custom_string)
- {
- sandbox::init_module();
- }
- ``
- [endsect]
- [section Why is my automatic to-python conversion not being found?]
- /Niall Douglas provides these notes:/
- If you define custom converters similar to the ones
- shown above the `def_readonly()` and `def_readwrite()`
- member functions provided by `boost::python::class_` for
- direct access to your member data will not work as expected.
- This is because `def_readonly("bar",&foo::bar)` is
- equivalent to:
- ``
- .add_property("bar", make_getter(&foo::bar, return_internal_reference()))
- ``
- Similarly, `def_readwrite("bar",&foo::bar)` is
- equivalent to:
- ``
- .add_property("bar", make_getter(&foo::bar, return_internal_reference()),
- make_setter(&foo::bar, return_internal_reference())
- ``
- In order to define return value policies compatible with the
- custom conversions replace `def_readonly()` and
- `def_readwrite()` by `add_property()`. E.g.:
- ``
- .add_property("bar", make_getter(&foo::bar, return_value_policy<return_by_value>()),
- make_setter(&foo::bar, return_value_policy<return_by_value>()))
- ``
- [endsect]
- [section Is Boost.Python thread-aware/compatible with multiple interpreters?]
- /Niall Douglas provides these notes:/
- The quick answer to this is: no.
- The longer answer is that it can be patched to be so, but it's
- complex. You will need to add custom lock/unlock wrapping of every
- time your code enters Boost.Python (particularly every virtual
- function override) plus heavily modify
- `boost/python/detail/invoke.hpp` with custom unlock/lock
- wrapping of every time Boost.Python enters your code. You must
- furthermore take care to /not/ unlock/lock when Boost.Python
- is invoking iterator changes via `invoke.hpp`.
- There is a patched `invoke.hpp` posted on the C++-SIG
- mailing list archives and you can find a real implementation of all
- the machinery necessary to fully implement this in the TnFOX
- project at [@http://sourceforge.net/projects/tnfox/ this]
- SourceForge project location.
- [endsect]
|