Selecting Fabrics

By default, the Intel® MPI Library selects a network fabric based on the list of fabrics specified in I_MPI_FABRICS_LIST. To select a specific network fabric combination, use the -genv option to assign a value to the I_MPI_FABRICS variable. You can also assign a value using the export command.

If the specified fabric is not available, Intel® MPI Library will go down the list specified in I_MPI_FABRICS_LIST and select the next available fabric.

You can disable this fallback behavior by using the I_MPI_FALLBACK variable:

　　$ export I_MPI_FALLBACK=0

By default, the fallback is enabled. If I_MPI_FABRICS is set, the fallback will be disabled.

TCP Socket Connection

Use the following command to run an MPI program over TCP sockets using the available Ethernet connection on the cluster. The program does not use shared memory within a node:

　　$ mpirun -genv I_MPI_FABRICS tcp -n <# of processes> ./myprog

Shared Memory

Use the following command to run an MPI program over the shared-memory fabric (shm) only:

　　$ mpirun -genv I_MPI_FABRICS shm -n <# of processes> ./myprog

Shared Memory and DAPL* Connection

To use shared memory for intra-node communication and the Direct Access Programming Library* (DAPL*) layer for inter-node communication, use the following command:

$ mpirun -genv I_MPI_FABRICS shm:dapl -n <# of processes> ./myprog

Use the I_MPI_DAPL_UD environment variable to enable connectionless DAPL User Datagrams* (DAPL UD*):

　　$ export I_MPI_DAPL_UD=enable

　　$ mpirun -genv I_MPI_FABRICS shm:dapl -n <# of processes> ./myprog

This is the default method if no fabric options are selected.

Shared memory and TMI*

To use shared memory for intra-node communication and the Tag Matching Interface* (TMI*) for inter-node communication, use the following command (make sure that you have libtmi.so library in the search path of the ldd command):

　　$ mpirun -genv I_MPI_FABRICS shm:tmi -n <# of processes> ./myprog

This is the recommended method if using Intel® Omni Scale Fabric (formerly Intel® True Scale) or the Myricom* MX interface.

Shared memory and OFA*

To select shared memory for intra-node communication and OpenFabrics* Enterprise Distribution (OFED) verbs for inter-node communication, use the following command:

　　$ mpirun -genv I_MPI_FABRICS shm:ofa -n <# of processes> ./myprog

This is the recommended method if using the Open Fabrics* Enterprise Distribution (OFED*) software stack.

Shared memory and OFI*

To select shared memory for intra-node communication and OpenFabrics Interface* (OFI*)  for inter-node communication, use the following command:

　　$ mpirun -genv I_MPI_FABRICS shm:ofi -n <# of processes> ./myprog

This is the recommended method if using the OpenFabrics Interface* (OFI*) software stack.

Multi-rail capability

If your cluster is equipped with several connection cards or multi-port cards, you can improve bandwidth of communications using the following settings:

　　$ export I_MPI_FABRICS=shm:ofa

　　$ export I_MPI_OFA_NUM_ADAPTERS=<num>

Where <num> is the number of connection adapters (1 by default).

If connection cards have several ports, you can specify the number of ports using the following setting:

　　$ export I_MPI_OFA_NUM_PORTS=<num>

For more details see the Fabrics Control topic in Intel® MPI Library Reference Manual for Linux* OS.

After you successfully run your application using the Intel MPI Library over any of the fabrics described, you can move your application from one cluster to another and use different fabrics between the nodes without re-linking. If you encounter problems, see Debugging and Testing for possible solutions.

Additionally, using mpirun is the recommended practice when using a resource manager, such as PBS Pro* or LSF*.

For example, to run the application in the PBS environment, follow these steps:

1. Create a PBS launch script that specifies number of nodes requested and sets your Intel MPI Library environment. For example, create a pbs_run.sh file with the following content:

   #PBS -l nodes=2:ppn=1

   #PBS -l walltime=1:30:00

   #PBS -q workq

   #PBS -V

   # Set Intel MPI environment

   mpi_dir=<installdir>/<arch>/bin

   cd $PBS_O_WORKDIR

   source $mpi_dir/mpivars.sh

   # Launch application

   mpirun -n <# of processes> ./myprog

2. Submit the job using the PBS qsub command:

   $ qsub pbs_run.sh

When using mpirun under a job scheduler, you do not need to determine the number of available nodes. Intel MPI Library automatically detects the available nodes through the Hydra process manager.