B1. General run-time tuning
https://tinyurl.com/yabwy5en
MCA: The Modular Component Architecture (MCA) is the backbone for much of Open MPI's functionality. It is a series of frameworks, components, and modules that are assembled at run-time to create an MPI implementation.
MCA parameters are the basic unit of run-time tuning for Open MPI. They are simple "key = value" pairs that are used extensively throughout the code base.
btl option:
export OMPI_MCA_btl_openib_allow_ib=1
export OMPI_MCA_btl=self,vader,openib
export OMPI_MCA_btl=self,vader,uct
export OMPI_MCA_btl=^tcp
B2. Set value of MCA parameters?
https://docs.oracle.com/cd/E19708-01/821-1319-10/mca-params.html
B6 Mapping, Ranking, and Binding: Oh My!
Open MPI employs a three-phase procedure for assigning process locations and ranks:
export OMPI_MCA_pml=ucx
There are 3 ways to set MCA parameters:
1. Command line: The highest-precedence method is setting MCA parameters, the format used on the command line is "
--mca <param_name> <value>"$ mpirun --mca mpi_show_handle_leaks 1 -np 4 a.out
$ mpirun --mca param "value with multiple words" ...2. Environment variable: OMPI_MCA_<param_name>$ export OMPI_MCA_mpi_show_handle_leaks=1 $ mpirun -np 4 a.out3. Aggregate MCA parameter files:Q11: https://www.open-mpi.org/faq/?category=tuningB3. Setting MCA ParametersQ13: https://www.open-mpi.org/faq/?category=tuningEach MCA framework has a top-level MCA parameter that can be used to include or excludecomponents from a given run.# Tell Open MPI to exclude the tcp and openib BTL components and implicitly include allthe rest $ mpirun --mca btl ^tcp,openib ... # Tell Open MPI to include *only* thecomponents listed here and implicitly ignore all the# rest (i.e., the loopback, shared memory, and OpenFabrics (a.k.a., "OpenIB") MPIpoint-to-point components): $ mpirun --mca btl self,sm,openib ...Note that ^ can only be the prefix of the entire value because the inclusive and exclusivebehavior are mutually exclusive. Specifically, since the exclusive behavior means "use allcomponents except these", it does not make sense to mix it with the inclusive behavior ofnot specifying it (i.e., "use all of these components"). https://www.open-mpi.org/doc/v4.0/man1/mpirun.1.php#sect20MPI shared memory communicationsThevaderBTL is a low-latency, high-bandwidth mechanism for transferring data between two processes via shared memory. This BTL can only be used between processes executing on the same node.Beginning with the v1.8 series, thevaderBTL replaces thesmBTLMCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.0.3) MCA btl: openib (MCA v2.1.0, API v3.1.0, Component v4.0.3) MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.0.3) MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.0.3)Shows all the MCA parameters for all BTL components thatompi_infofinds.# to show all MCA parameters: ompi_info --param btl all$ ompi_info --param btl all --level 9NOTE: the UCX PML is now the preferred method of InfiniBand support in openMPI 4.0.x (btl is built-in methods)Exported Environment Variables
All environment variables that are named in the form OMPI_* will automatically be exported to new processes on the local and remote nodes. Environmental parameters can also be set/forwarded to the new processes using the MCA parameter mca_base_env_list. The -x option to mpirun has been deprecated, but the syntax of the MCA param follows that prior example. While the syntax of the -x option and MCA param allows the definition of new variables, note that the parser for these options are currently not very sophisticated - it does not even understand quoted values. Users are advised to set variables in the environment and use the option to export them; not to define them.Setting MCA Parameters
The -mca switch allows the passing of parameters to various MCA (Modular Component Architecture) modules. MCA modules have direct impact on MPI programs because they allow tunable parameters to be set at run time (such as which BTL communication device driver to use, what parameters to pass to that BTL, etc.).The -mca switch takes two arguments: <key> and <value>. The <key> argument generally specifies which MCA module will receive the value. For example, the <key> "btl" is used to select which BTL to be used for transporting MPI messages. The <value> argument is the value that is passed. For example:
- mpirun -mca btl tcp,self -np 1 foo
- Tells Open MPI to use the "tcp" and "self" BTLs, and to run a single copy of "foo" an allocated node.
- mpirun -mca btl self -np 1 foo
- Tells Open MPI to use the "self" BTL, and to run a single copy of "foo" an allocated node.
The -mca switch can be used multiple times to specify different <key> and/or <value> arguments. If the same <key> is specified more than once, the <value>s are concatenated with a comma (",") separating them.Note that the -mca switch is simply a shortcut for setting environment variables. The same effect may be accomplished by setting corresponding environment variables before running mpirun. The form of the environment variables that Open MPI sets is:OMPI_MCA_<key>=<value>Thus, the -mca switch overrides any previously set environment variables. The -mca settings similarly override MCA parameters set in the $OPAL_PREFIX/etc/openmpi-mca-params.conf or $HOME/.openmpi/mca-params.conf file.Unknown <key> arguments are still set as environment variable -- they are not checked (by mpirun) for correctness. Illegal or incorrect <value> arguments may or may not be reported -- it depends on the specific MCA module.To find the available component types under the MCA architecture, or to find the available parameters for a specific component, use the ompi_info command. See the ompi_info(1) man page for detailed information on the command.B4. What is processor affinity?Open MPI supports processor affinity on a variety of systems through process binding, in which each MPI process, along with its threads, is "bound" to a specific subset of processing resources (cores, sockets, etc.).Affinity can improve performance by inhibiting excessive process movement — for example, away from "hot" caches or NUMA memory. Judicious bindings can improve performance by reducing resource contention (by spreading processes apart from one another) or improving interprocess communications (by placing processes close to one another).Note that processor affinity probably should not be used when a node is over-subscribed (i.e., more processes are launched than there are processors).memory affinity? Simply: some memory will be faster to access (for a given process) than others.see if your system is supported processor affinity?/memory affinity?$ ompi_info | grep hwlocMCA hwloc: hwloc191 (MCA v2.0, API v2.0, Component v1.8.4)B5 tell Open MPI to use processor and/or memory affinityQ19 https://www.open-mpi.org/faq/?category=tuning
--byslot: Alias for--bycore.--bycore: When laying out processes, put sequential MPI processes on adjacent processor cores. *(Default)*--bysocket: When laying out processes, put sequential MPI processes on adjacent processor sockets.--bynode: When laying out processes, put sequential MPI processes on adjacent nodes.
The use of processor and memory affinity evolved rapidly, starting with Open MPI version:
- mpirun(1) man page for current Open MPI release
- mpirun(1) man page for Open MPI v4.0.x
- mpirun(1) man page for Open MPI v3.1.x
- mpirun(1) man page for Open MPI v3.0.x
- mpirun(1) man page for Open MPI v2.0.x
- mpirun(1) man page for Open MPI v1.10.x
- mpirun(1) man page for Open MPI v1.8.x
- mpirun(1) man page for Open MPI v1.7.x
- mpirun(1) man page for Open MPI v1.6.x
- mapping
- Assigns a default location to each process
- ranking
- Assigns an MPI_COMM_WORLD rank value to each process
- binding
- Constrains each process to run on specific processors
The mapping step is used to assign a default location to each process based on the mapper being employed. Mapping by slot, node, and sequentially results in the assignment of the processes to the node level. In contrast, mapping by object, allows the mapper to assign the process to an actual object on each node.
Note: the location assigned to the process is independent of where it will be bound - the assignment is used solely as input to the binding algorithm.
The mapping of process processes to nodes can be defined not just with general policies but also, if necessary, using arbitrary mappings that cannot be described by a simple policy. One can use the "sequential mapper," which reads the hostfile line by line, assigning processes to nodes in whatever order the hostfile specifies. Use the -mca rmaps seq option. For example, using the same hostfile as before:
mpirun -hostfile myhostfile -mca rmaps seq ./a.out
will launch three processes, one on each of nodes aa, bb, and cc, respectively. The slot counts don’t matter; one process is launched per line on whatever node is listed on the line.
Another way to specify arbitrary mappings is with a rankfile, which gives you detailed control over process binding as well. Rankfiles are discussed below.
The second phase focuses on the ranking of the process within the job’s MPI_COMM_WORLD. Open MPI separates this from the mapping procedure to allow more flexibility in the relative placement of MPI processes. This is best illustrated by considering the following two cases where we used the —map-by ppr:2:socket option:
node aa node bb
rank-by core 0 1 ! 2 3 4 5 ! 6 7
rank-by socket 0 2 ! 1 3 4 6 ! 5 7
rank-by socket:span 0 4 ! 1 5 2 6 ! 3 7
Ranking by core and by slot provide the identical result - a simple progression of MPI_COMM_WORLD ranks across each node. Ranking by socket does a round-robin ranking within each node until all processes have been assigned an MCW rank, and then progresses to the next node. Adding the span modifier to the ranking directive causes the ranking algorithm to treat the entire allocation as a single entity - thus, the MCW ranks are assigned across all sockets before circling back around to the beginning.
The binding phase actually binds each process to a given set of processors. This can improve performance if the operating system is placing processes suboptimally. For example, it might oversubscribe some multi-core processor sockets, leaving other sockets idle; this can lead processes to contend unnecessarily for common resources. Or, it might spread processes out too widely; this can be suboptimal if application performance is sensitive to interprocess communication costs. Binding can also keep the operating system from migrating processes excessively, regardless of how optimally those processes were placed to begin with.
The processors to be used for binding can be identified in terms of topological groupings - e.g., binding to an l3cache will bind each process to all processors within the scope of a single L3 cache within their assigned location. Thus, if a process is assigned by the mapper to a certain socket, then a —bind-to l3cache directive will cause the process to be bound to the processors that share a single L3 cache within that socket.
Alternatively, processes can be assigned to processors based on their local rank on a node using the --bind-to cpu-list:ordered option with an associated --cpu-list "0,2,5". In this example, the first process on a node will be bound to cpu 0, the second process on the node will be bound to cpu 2, and the third process on the node will be bound to cpu 5. --bind-to will also accept cpulist:ortered as a synonym to cpu-list:ordered. Note that an error will result if more processes are assigned to a node than cpus are provided.
To help balance loads, the binding directive uses a round-robin method when binding to levels lower than used in the mapper. For example, consider the case where a job is mapped to the socket level, and then bound to core. Each socket will have multiple cores, so if multiple processes are mapped to a given socket, the binding algorithm will assign each process located to a socket to a unique core in a round-robin manner.
Alternatively, processes mapped by l2cache and then bound to socket will simply be bound to all the processors in the socket where they are located. In this manner, users can exert detailed control over relative MCW rank location and binding.
Finally, --report-bindings can be used to report bindings.
As an example, consider a node with two processor sockets, each comprising four cores. We run mpirun with -np 4 --report-bindings and the following additional options:
% mpirun ... --map-by core --bind-to core
[...] ... binding child [...,0] to cpus 0001
[...] ... binding child [...,1] to cpus 0002
[...] ... binding child [...,2] to cpus 0004
[...] ... binding child [...,3] to cpus 0008
% mpirun ... --map-by socket --bind-to socket
[...] ... binding child [...,0] to socket 0 cpus 000f
[...] ... binding child [...,1] to socket 1 cpus 00f0
[...] ... binding child [...,2] to socket 0 cpus 000f
[...] ... binding child [...,3] to socket 1 cpus 00f0
% mpirun ... --map-by core:PE=2 --bind-to core
[...] ... binding child [...,0] to cpus 0003
[...] ... binding child [...,1] to cpus 000c
[...] ... binding child [...,2] to cpus 0030
[...] ... binding child [...,3] to cpus 00c0
% mpirun ... --bind-to none
Here, --report-bindings shows the binding of each process as a mask. In the first case, the processes bind to successive cores as indicated by the masks 0001, 0002, 0004, and 0008. In the second case, processes bind to all cores on successive sockets as indicated by the masks 000f and 00f0. The processes cycle through the processor sockets in a round-robin fashion as many times as are needed. In the third case, the masks show us that 2 cores have been bound per process. In the fourth case, binding is turned off and no bindings are reported.
Open MPI’s support for process binding depends on the underlying operating system. Therefore, certain process binding options may not be available on every system.
Process binding can also be set with MCA parameters. Their usage is less convenient than that of mpirun options. On the other hand, MCA parameters can be set not only on the mpirun command line, but alternatively in a system or user mca-params.conf file or as environment variables, as described in the MCA section below. Some examples include:
mpirun option MCA parameter key value
--map-by core rmaps_base_mapping_policy core
--map-by socket rmaps_base_mapping_policy socket
--rank-by core rmaps_base_ranking_policy core
--bind-to core hwloc_base_binding_policy core
--bind-to socket hwloc_base_binding_policy socket
--bind-to none hwloc_base_binding_policy none
Portable Hardware Locality(HWLOC) (included in OpenMPI)
The Portable Hardware Locality (hwloc) software package provides a portable abstraction (across OS, versions, architectures, ...) of the hierarchical topology of modern architectures, including NUMA memory nodes, sockets, shared caches, cores and simultaneous multithreading. It also gathers various system attributes such as cache and memory information as well as the locality of I/O devices such as network interfaces, InfiniBand HCAs or GPUs.
The Portable Hardware Locality (hwloc) software package provides a portable abstraction (across OS, versions, architectures, ...) of the hierarchical topology of modern architectures, including NUMA memory nodes, sockets, shared caches, cores and simultaneous multithreading. It also gathers various system attributes such as cache and memory information as well as the locality of I/O devices such as network interfaces, InfiniBand HCAs or GPUs.
MPI InfiniBand, RoCE, and iWARP communications
https://www.open-mpi.org/faq/?category=openfabrics#ib-components
support for high-speed interconnect networks
https://www.open-mpi.org/faq/?category=openfabrics#run-ucx
https://www.open-mpi.org/faq/?category=building#build-p2p
Compare transport mechanism: https://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy
II. OMPI_MCA_btl
ompi_info --param btl all
MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.1.0)
MCA btl: openib (MCA v2.1.0, API v3.1.0, Component v4.1.0)
MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.1.0)
MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.0)
| What is the vader BTL? |
The vader BTL is a low-latency, high-bandwidth mechanism for transferring data between two processes via shared memory. This BTL can only be used between processes executing on the same node.
Beginning with the v1.8 series, the vader BTL replaces the sm BTL
III. Using UCX with OpenMPI
http://openucx.github.io/ucx/faq.html
#1.a. See all available transports of OMPI:
module load openmpi
ompi_info |grep btl
MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.0.3)MCA btl: openib (MCA v2.1.0, API v3.1.0, Component v4.0.3)MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.0.3)MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.0.3)#1.b. See all available transports/Devices of UCX:module load openmpiucx_info -d# Transport: tcp
# Device: eth0 # capabilities: # bandwidth: 113.16 MB/sec
# Device: eth1 # capabilities: # bandwidth: 113.16 MB/sec# Device: posix # capabilities: # bandwidth: 12179.00 MB/sec# Device: ib0# capabilities: # bandwidth: 4457.00 MB/sec# Transport: self # Device: self # capabilities: # bandwidth: 6911.00 MB/sec # Transport: mm # Device: sysv # capabilities: # bandwidth: 12179.00 MB/sec
# Transport: ud (or ud_verbs)
# Device: mlx4_0:1
# capabilities:
# bandwidth: 3774.15 MB/sec
# Transport: rc (or rc_verbs)
# Device: mlx4_0:1
# capabilities:
# bandwidth: 3774.15 MB/sec
# Transport: cm
# Device: mlx4_0:1
# capabilities:
# bandwidth: 2985.42 MB/sec
# Transport: knem
# Device: knem
# capabilities:
# bandwidth: 13862.00 MB/sec
2.a. Force to use UCX
export OMPI_MCA_btl=^vader,tcp,openib,uctexport OMPI_MCA_pml=ucx
2.b. Choose a specific transport/device
export UCX_TLS=self,mm,knem,sm,ud,rc,tcp
export UCX_NET_DEVICES=mlx4_0:1,ib0
