Parallel Computing at SLAC

Getting Help

General Help

If you encounter any problems with the hardware or software mentioned here, please send an email to unix-admin@slac.stanford.edu.

MPI Users Mailing List

Hardware

Public Hardware

For some pictures, see HPC Linux Farm.

• Software Development and Testing:
  morab cluster of 16 Dell PowerEdge 1650 machines with Myrinet interconnect.
  They have 2 1.4 GHz Pentium III CPUs and 2 GB memory.
  Login to this cluster via ssh and the generic node name of morab. These nodes can also be used via LSF.
• Production Runs
  64 node orlov cluster with Myrinet interconnect.
  They are Rackable systems with 2 hyperthreading enabled 2.6 GHz Xeon CPUs and 2 GB of memory.
  They are accessible only via LSF.

Please note that code compiled on the morab with Pentium 4 optimization for the orlovs will not run on the morabs because they are Pentium 3!

Private Hardware

• Prof. Tom Abel (KIPAC)
  72 1.5 GHz Itanium 2 CPU SGI Altix system with 224 GB of main memory.
• Klystron Department
  Cluster with 22 Sun V20z. Each has two 2.3 GHz Opteron CPUs, 1 GigE and 2 GB of memory.
• Prof. Steve Allan (KIPAC)
  Cluster with 16 Sun V20z. Each has two dual core 2 GHz Opteron CPUs, 1 GigE and 4 GB of memory.

Software

MPI

• MPICH with Myrinet support
  This is currently the only SCCS supported version of MPICH. It will not work on clusters without Myrinet hardware. This version is 32 bit only.
  The default compilers invoked by mpicc and friends are:
  C/C++: The gcc in your search path.
  Fortran: The Intel ifort compiler in your search path.
  You can adjust your search path to include this compiler by adding
  /afs/slac.stanford.edu/package/intel_tools/compiler9.0/@sys/fs/bin
  to your PATH environment variable.
• MPICH with Ethernet support
  Currently, only a user supplied version is in use. This version is not publicly accessible.
  If you need an MPI version over Ethernet, see OpenMPI below.
• LAM/MPI with Myrinet support
  There is no officially supported version available.
• OpenMPI with Ethernet, Infiniband, Myrinet, and Shared memory support
  The goal is to move away from all previously mentioned MPI versions and switch to this one. It will have support for all the different interconnect used at SLAC. It will be available as 32 bit and 64 bit version.
  The default compilers will be the Intel Compilers.
  Current state:
  Currently, two versions of OpenMPI are available:
  • a 64 bit version with Ethernet and Infiniband support and
  • a 32 bit version with Ethernet, Myrinet, and Shared memory support
  Both versions are available via:
  /afs/slac.stanford.edu/package/OpenMPI/{bin,include,lib}/
  The proper path resolution is done via AFS' @sys variable depending on the system you are on.
  The default compilers are the Intel Compilers version 9 for C/C++ and Fortran77/90.
  You should source the necessary Intel environments before using it:
  source /afs/slac/package/intel_tools/compiler9.0/amd64_linux26/cc/bin/iccvars.csh
  for the C/C++ environment and
  source /afs/slac/package/intel_tools/compiler9.0/amd64_linux26/fc/bin/ifortvars.csh
  for the Fortran environment (or both).
• MPI on SGI Altix
  SGI's ProPack comes with a special version of MPI optimized for the Altix system.
  Please consult the man pages (man mpi) for details.
• MPI for Mac OS X
  We currently do not support any MPI versions for Mac OS X.

OpenMP

• For Linux: Intel Compilers 9
  The Intel compilers for Linux support OpenMP on 32 bit and 64 bit Linux. Version 8 of the compilers support 32 bit only! The compilers are available via
  Fortran: /afs/slac.stanford.edu/package/intel_tools/compiler9.0/@sys/fs/bin
  C/C++: /afs/slac.stanford.edu/package/intel_tools/compiler9.0/@sys/cc/bin
• For Solaris: Sun Studio 10 -- at the time of this writing, this does not work at SLAC!
  The Sun Studio tools support OpenMP on Solaris. This software is available on all Sun Solaris machines (SPARC as well as x86) under
  /opt/SUNWspro/bin/
  or
  /opt/SUNWspro.Studio10/bin/
  or
  /afs/slac.stanford.edu/package/sunworkshop/studio10/@sys/SUNWspro/bin/

Mixed MPI and OpenMP Programs

It is possible to use OpenMP inside an MPI program.
Please refer to this little SLAC specific tutorial for how to do this.

Debugger

• The GNU debugger gdb is available in your search path on all SLAC machines.
• The Sun Solaris debugger dbx is available under the same path as the Sun Studio compilers, typically /opt/SUNWspro/bin/dbx
• Totalview
  SLAC has 1 (one) license for up to 4 CPUs. The debugger is avialable for Linux only.
  It is accessable via /usr/local/bin/totalview or /afs/slac.stanford.edu/package/totalview/toolworks/totalview.7.0.0-1/bin/totalview
  It is available for 32 bit or 64 bit under these paths.

Performance Analyzer, Profiler

• Sun Studio tool analyzer
  This tool is available under the same path as the Sun Studio compilers, typically /opt/SUNWspro/bin/analyzer
• MPE
  The MPICH profiler extension is currently not available at SLAC.
• Paradyn and other open source tools are currently not available at SLAC.

Libraries and Tools

• HDF5
  A parallel version of this file format is available for 32 bit Linux under
  /afs/slac/package/hdf/5/1.6.5-parallel/i386_linux24/{bin,lib,include}/
  To use it, you need to also use the production version of MPICH for Myrinet.
• Linear Algebra Packages
  There is no parallel version of these packages available at SLAC.
• Parallel Filesystems:
  There are no parallel filesystems available at SLAC.

Further Reading

• Sun Studio Compilers and Tools