Computing at SLAC
Search SLAC

MPI Tutorial

MPI's "Hello World"

Name this little "Hello World" program hello.c:
#include <stdio.h>
#include <mpi.h>

int main(int argc, char *argv[]) {
  int numprocs, rank, namelen;
  char processor_name[MPI_MAX_PROCESSOR_NAME];

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Get_processor_name(processor_name, &namelen);

  printf("Process %d on %s out of %d\n", rank, processor_name, numprocs);

  MPI_Finalize();
}

Compiling and Linking MPI Programs

Once you have your MPI example program, you can compile and link it with the MPI version that supports the network type you need.
  • Linux with Ethernet or Infiniband (OpenMPI):
    mpicc hello.c -o hello
    Open MPI is using the following environment variables for alternative compilers:
    C: OMPI_CC
    C++: OMPI_CXX
    Fortran 77: OMPI_F77
    Fortran 90: OMPI_FC
    The comple list of environment variables for compilers, compiler flags, linkters, and linkger flags can be found in the OpenMPI FAQ

Running MPI Programs

Logon to the pinto cluster: ssh pinto
Make sure your MPI program is compiled for these machines (RHEL5).

Add the machines you want your parallel program to run on to a file:

alfw@pinto> cat machinefile.pinto
pinto01
pinto01
pinto02
pinto02
This will first use pinto01 and then use pinto02.
If you need more processes than CPUs are in your machinefile, MPI will start again at the top of the file.

To run your program interactively, execute one of these commands

  • OpenMPI on Linux:

    Make sure the path to all OpenMPI commands gets added automatically at login time! You may use mpi-selector to establish your preference.

    • Linux with Ethernet:
      alfw@pinto> mpirun -np 4 -machinefile machinefile.pinto hello
      Process 0 on pinto01 out of 4
      Process 1 on pinto01 out of 4
      Process 3 on pinto02 out of 4
      Process 2 on pinto02 out of 4
      
    • Linux without Infiniband or LSF:
      If you want to run your OpenMPI code on, lets say, a cluster of desktop machines where you don't have Infiniband or LSF, you should use these parameters on your mpirun command:
      mpirun --mca btl "^ib" --mca pls_rsh_agent ssh -np 4 \
      -machinefile machinefile.desktops hello
      
      This will exclude the Infiniband driver (ib) as a choice for the communication network and will use ssh instead of the LSF wrapper to start your programs on the remote machines.

LSF Integration and Usage

Currently there is one LSF queue defined for running MPI jobs via LSF. The mpi-ibq. The mpi-ibq has been designated as the default production MPI queue and is configured to use the 32 node pinto Infiniband cluster. Presently the mpi-ibq is setup to use a maximum of 8 processors per machine. Which gives this queue a total of 256 available processors.

Before you submit an MPI job to LSF, make sure you have the correct mpirun binary in your $PATH environment variable. On the pinto cluster this is accomplished by using mpi-selector. The system default is set for the cluster but you may choose to use a different version of OpenMPI and prefer to set a user level default.

To submit your MPI programs via LSF to either cluster, you have to compile your MPI jobs just as you would for interactive job launch.
Instead of using mpirun for program start, you have to use the following LSF command from either a morab or rhel6-64 or pinto machine:

To submit jobs to the default mpi-ibq (pinto cluster):

bsub -a mympi -n <number of processors> <OpenMPI_program>
Where -a mympi is the esub required to run OpenMPI jobs.

Example:

bsub -a mympi -n 10 ~/Prog/OpenMPI/bin/hello++

Each MPI queue by default is configured with the span resource empty (span[]) to allow jobs to be scheduled on machines that have an empty processor based on normal LSF load considerations. If you wish to specify a specific spanning resource option to suit your parallel batch job you may use -R "span[ptile=<value>]" on the bsub command.

Where <value> is the number of processors you wish to use on each host.

Example:

bsub -a mympi -n 10 -R "span[ptile=2]" ~/Prog/OpenMPI/bin/hello++
In this example, "span[ptile=2]" will require the job to run on two processors per host.

Some jobs may perform better when run on a single processor per host. In this case you would want to specify -R "span[ptile=1]". Consider that we currently have 64 hosts allocated to the mpiq. If you use "span[ptile=1]" the -n <number of processors> option on bsub could be no greater than 32 for your job to run.

Example:

bsub -a mympi -n 32 -R "span[ptile=1]" ~/Prog/OpenMPI/bin/hello++

Submitting an OpenMPI job to use the old package verson of OpenMPI job presently works by using the -a openmpi parameter.

Example:

bsub -q mpi_ibq -a openmpi -n 10 ~/Prog/OpenMPI/bin/hello++

References






Neal Adams and Alf Wachsmann. Last Modified: Oct 22, 2010.