123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222 |
- [section:python Python Bindings]
- [python]
- Boost.MPI provides an alternative MPI interface from the _Python_
- programming language via the `boost.mpi` module. The
- Boost.MPI Python bindings, built on top of the C++ Boost.MPI using the
- _BoostPython_ library, provide nearly all of the functionality of
- Boost.MPI within a dynamic, object-oriented language.
- The Boost.MPI Python module can be built and installed from the
- `libs/mpi/build` directory. Just follow the [link
- mpi.getting_started.config configuration] and [link mpi.getting_started.config.installation
- installation] instructions for the C++ Boost.MPI. Once you have
- installed the Python module, be sure that the installation location is
- in your `PYTHONPATH`.
- [section:quickstart Quickstart]
- [python]
- Getting started with the Boost.MPI Python module is as easy as
- importing `boost.mpi`. Our first "Hello, World!" program is
- just two lines long:
- import boost.mpi as mpi
- print "I am process %d of %d." % (mpi.rank, mpi.size)
- Go ahead and run this program with several processes. Be sure to
- invoke the `python` interpreter from `mpirun`, e.g.,
-
- [pre
- mpirun -np 5 python hello_world.py
- ]
- This will return output such as:
- [pre
- I am process 1 of 5.
- I am process 3 of 5.
- I am process 2 of 5.
- I am process 4 of 5.
- I am process 0 of 5.
- ]
- Point-to-point operations in Boost.MPI have nearly the same syntax in
- Python as in C++. We can write a simple two-process Python program
- that prints "Hello, world!" by transmitting Python strings:
- import boost.mpi as mpi
- if mpi.world.rank == 0:
- mpi.world.send(1, 0, 'Hello')
- msg = mpi.world.recv(1, 1)
- print msg,'!'
- else:
- msg = mpi.world.recv(0, 0)
- print (msg + ', '),
- mpi.world.send(0, 1, 'world')
- There are only a few notable differences between this Python code and
- the example [link mpi.tutorial.point_to_point in the C++
- tutorial]. First of all, we don't need to write any initialization
- code in Python: just loading the `boost.mpi` module makes the
- appropriate `MPI_Init` and `MPI_Finalize` calls. Second, we're passing
- Python objects from one process to another through MPI. Any Python
- object that can be pickled can be transmitted; the next section will
- describe in more detail how the Boost.MPI Python layer transmits
- objects. Finally, when we receive objects with `recv`, we don't need
- to specify the type because transmission of Python objects is
- polymorphic.
- When experimenting with Boost.MPI in Python, don't forget that help is
- always available via `pydoc`: just pass the name of the module or
- module entity on the command line (e.g., `pydoc
- boost.mpi.communicator`) to receive complete reference
- documentation. When in doubt, try it!
- [endsect:quickstart]
- [section:user_data Transmitting User-Defined Data]
- Boost.MPI can transmit user-defined data in several different ways.
- Most importantly, it can transmit arbitrary _Python_ objects by pickling
- them at the sender and unpickling them at the receiver, allowing
- arbitrarily complex Python data structures to interoperate with MPI.
- Boost.MPI also supports efficient serialization and transmission of
- C++ objects (that have been exposed to Python) through its C++
- interface. Any C++ type that provides (de-)serialization routines that
- meet the requirements of the Boost.Serialization library is eligible
- for this optimization, but the type must be registered in advance. To
- register a C++ type, invoke the C++ function [funcref
- boost::mpi::python::register_serialized
- register_serialized]. If your C++ types come from other Python modules
- (they probably will!), those modules will need to link against the
- `boost_mpi` and `boost_mpi_python` libraries as described in the [link
- mpi.getting_started.config.installation installation section]. Note that you do
- *not* need to link against the Boost.MPI Python extension module.
- Finally, Boost.MPI supports separation of the structure of an object
- from the data it stores, allowing the two pieces to be transmitted
- separately. This "skeleton/content" mechanism, described in more
- detail in a later section, is a communication optimization suitable
- for problems with fixed data structures whose internal data changes
- frequently.
- [endsect:user_data]
- [section:collectives Collectives]
- Boost.MPI supports all of the MPI collectives (`scatter`, `reduce`,
- `scan`, `broadcast`, etc.) for any type of data that can be
- transmitted with the point-to-point communication operations. For the
- MPI collectives that require a user-specified operation (e.g., `reduce`
- and `scan`), the operation can be an arbitrary Python function. For
- instance, one could concatenate strings with `all_reduce`:
- mpi.all_reduce(my_string, lambda x,y: x + y)
- The following module-level functions implement MPI collectives:
- all_gather Gather the values from all processes.
- all_reduce Combine the results from all processes.
- all_to_all Every process sends data to every other process.
- broadcast Broadcast data from one process to all other processes.
- gather Gather the values from all processes to the root.
- reduce Combine the results from all processes to the root.
- scan Prefix reduction of the values from all processes.
- scatter Scatter the values stored at the root to all processes.
- [endsect:collectives]
- [section:skeleton_content Skeleton/Content Mechanism]
- Boost.MPI provides a skeleton/content mechanism that allows the
- transfer of large data structures to be split into two separate stages,
- with the skeleton (or, "shape") of the data structure sent first and
- the content (or, "data") of the data structure sent later, potentially
- several times, so long as the structure has not changed since the
- skeleton was transferred. The skeleton/content mechanism can improve
- performance when the data structure is large and its shape is fixed,
- because while the skeleton requires serialization (it has an unknown
- size), the content transfer is fixed-size and can be done without
- extra copies.
- To use the skeleton/content mechanism from Python, you must first
- register the type of your data structure with the skeleton/content
- mechanism *from C++*. The registration function is [funcref
- boost::mpi::python::register_skeleton_and_content
- register_skeleton_and_content] and resides in the [headerref
- boost/mpi/python.hpp <boost/mpi/python.hpp>] header.
- Once you have registered your C++ data structures, you can extract
- the skeleton for an instance of that data structure with `skeleton()`.
- The resulting `skeleton_proxy` can be transmitted via the normal send
- routine, e.g.,
- mpi.world.send(1, 0, skeleton(my_data_structure))
- `skeleton_proxy` objects can be received on the other end via `recv()`,
- which stores a newly-created instance of your data structure with the
- same "shape" as the sender in its `"object"` attribute:
- shape = mpi.world.recv(0, 0)
- my_data_structure = shape.object
- Once the skeleton has been transmitted, the content (accessed via
- `get_content`) can be transmitted in much the same way. Note, however,
- that the receiver also specifies `get_content(my_data_structure)` in its
- call to receive:
- if mpi.rank == 0:
- mpi.world.send(1, 0, get_content(my_data_structure))
- else:
- mpi.world.recv(0, 0, get_content(my_data_structure))
- Of course, this transmission of content can occur repeatedly, if the
- values in the data structure--but not its shape--changes.
- The skeleton/content mechanism is a structured way to exploit the
- interaction between custom-built MPI datatypes and `MPI_BOTTOM`, to
- eliminate extra buffer copies.
- [endsect:skeleton_content]
- [section:compatibility C++/Python MPI Compatibility]
- Boost.MPI is a C++ library whose facilities have been exposed to Python
- via the Boost.Python library. Since the Boost.MPI Python bindings are
- build directly on top of the C++ library, and nearly every feature of
- C++ library is available in Python, hybrid C++/Python programs using
- Boost.MPI can interact, e.g., sending a value from Python but receiving
- that value in C++ (or vice versa). However, doing so requires some
- care. Because Python objects are dynamically typed, Boost.MPI transfers
- type information along with the serialized form of the object, so that
- the object can be received even when its type is not known. This
- mechanism differs from its C++ counterpart, where the static types of
- transmitted values are always known.
- The only way to communicate between the C++ and Python views on
- Boost.MPI is to traffic entirely in Python objects. For Python, this
- is the normal state of affairs, so nothing will change. For C++, this
- means sending and receiving values of type `boost::python::object`,
- from the _BoostPython_ library. For instance, say we want to transmit
- an integer value from Python:
- comm.send(1, 0, 17)
- In C++, we would receive that value into a Python object and then
- `extract` an integer value:
- [c++]
- boost::python::object value;
- comm.recv(0, 0, value);
- int int_value = boost::python::extract<int>(value);
- In the future, Boost.MPI will be extended to allow improved
- interoperability with the C++ Boost.MPI and the C MPI bindings.
- [endsect:compatibility]
- [section:reference Reference]
- The Boost.MPI Python module, `boost.mpi`, has its own
- [@boost.mpi.html reference documentation], which is also
- available using `pydoc` (from the command line) or
- `help(boost.mpi)` (from the Python interpreter).
- [endsect:reference]
- [endsect:python]
|