Computing at SLAC
Search SLAC

Parallel Computing at SLAC

Getting Help

General Help

If you encounter any problems with the hardware or software mentioned here, please send an email to unix-admin@slac.stanford.edu.

See also the Lecture Series: "Introduction to Parallel Programming"

MPI Users Mailing List

There is a a mailinglist for all parallel computing related issues: mpi-users
To subscribe, send mail to majordomo@slac.stanford.edu with the following in the body of the message:
subscribe mpi-users
end

Hardware

Public Hardware

For some pictures, see HPC Linux Farm.

  • Software Development and Testing:
    morab cluster of 16 Dell PowerEdge 1650 machines with Myrinet interconnect.
    They have 2 1.4 GHz Pentium III CPUs and 2 GB memory.
    Login to this cluster via ssh and the generic node name of morab. These nodes can also be used via LSF.
  • Production Runs
    64 node orlov cluster with Myrinet interconnect.
    They are Rackable systems with 2 hyperthreading enabled 2.6 GHz Xeon CPUs and 2 GB of memory.
    They are accessible only via LSF.

Please note that code compiled on the morab with Pentium 4 optimization for the orlovs will not run on the morabs because they are Pentium 3!

Private Hardware

  • Prof. Tom Abel (KIPAC)
    72 1.5 GHz Itanium 2 CPU SGI Altix system with 224 GB of main memory.
  • Klystron Department
    Cluster with 22 Sun V20z. Each has two 2.3 GHz Opteron CPUs, 1 GigE and 2 GB of memory.
  • Prof. Steve Allan (KIPAC)
    Cluster with 16 Sun V20z. Each has two dual core 2 GHz Opteron CPUs, 1 GigE and 4 GB of memory.

Software

MPI

Please note that 64 bit Linux means Red Hat Enterprise Linux 4 only!
  • MPICH with Myrinet support
    This is currently the only SCCS supported version of MPICH. It will not work on clusters without Myrinet hardware. This version is 32 bit only.
    The default compilers invoked by mpicc and friends are:
    C/C++: The gcc in your search path.
    Fortran: The Intel ifort compiler in your search path.
    You can adjust your search path to include this compiler by adding
    /afs/slac.stanford.edu/package/intel_tools/compiler9.0/@sys/fs/bin
    to your PATH environment variable.
  • MPICH with Ethernet support
    Currently, only a user supplied version is in use. This version is not publicly accessible.
    If you need an MPI version over Ethernet, see OpenMPI below.
  • LAM/MPI with Myrinet support
    There is no officially supported version available.
  • OpenMPI with Ethernet, Infiniband, Myrinet, and Shared memory support
    The goal is to move away from all previously mentioned MPI versions and switch to this one. It will have support for all the different interconnect used at SLAC. It will be available as 32 bit and 64 bit version.
    The default compilers will be the Intel Compilers.
    Current state:
    Currently, two versions of OpenMPI are available:
    • a 64 bit version with Ethernet and Infiniband support and
    • a 32 bit version with Ethernet, Myrinet, and Shared memory support
    Both versions are available via:
    /afs/slac.stanford.edu/package/OpenMPI/{bin,include,lib}/
    The proper path resolution is done via AFS' @sys variable depending on the system you are on.
    The default compilers are the Intel Compilers version 9 for C/C++ and Fortran77/90.
    You should source the necessary Intel environments before using it:
    source /afs/slac/package/intel_tools/compiler9.0/amd64_linux26/cc/bin/iccvars.csh
    for the C/C++ environment and
    source /afs/slac/package/intel_tools/compiler9.0/amd64_linux26/fc/bin/ifortvars.csh
    for the Fortran environment (or both).
  • MPI on SGI Altix
    SGI's ProPack comes with a special version of MPI optimized for the Altix system.
    Please consult the man pages (man mpi) for details.
  • MPI for Mac OS X
    We currently do not support any MPI versions for Mac OS X.
Click here for a little SLAC specific MPI tutorial.

OpenMP

  • For Linux: Intel Compilers 9
    The Intel compilers for Linux support OpenMP on 32 bit and 64 bit Linux. Version 8 of the compilers support 32 bit only! The compilers are available via
    Fortran: /afs/slac.stanford.edu/package/intel_tools/compiler9.0/@sys/fs/bin
    C/C++: /afs/slac.stanford.edu/package/intel_tools/compiler9.0/@sys/cc/bin
  • For Solaris: Sun Studio 10 -- at the time of this writing, this does not work at SLAC!
    The Sun Studio tools support OpenMP on Solaris. This software is available on all Sun Solaris machines (SPARC as well as x86) under
    /opt/SUNWspro/bin/
    or
    /opt/SUNWspro.Studio10/bin/
    or
    /afs/slac.stanford.edu/package/sunworkshop/studio10/@sys/SUNWspro/bin/
Click here for a little SLAC specific OpenMP tutorial.

Mixed MPI and OpenMP Programs

It is possible to use OpenMP inside an MPI program.
Please refer to this little SLAC specific tutorial for how to do this.

Debugger

  • The GNU debugger gdb is available in your search path on all SLAC machines.
  • The Sun Solaris debugger dbx is available under the same path as the Sun Studio compilers, typically /opt/SUNWspro/bin/dbx
  • Totalview
    SLAC has 1 (one) license for up to 4 CPUs. The debugger is avialable for Linux only.
    It is accessable via /usr/local/bin/totalview or /afs/slac.stanford.edu/package/totalview/toolworks/totalview.7.0.0-1/bin/totalview
    It is available for 32 bit or 64 bit under these paths.

Performance Analyzer, Profiler

  • Sun Studio tool analyzer
    This tool is available under the same path as the Sun Studio compilers, typically /opt/SUNWspro/bin/analyzer
  • MPE
    The MPICH profiler extension is currently not available at SLAC.
  • Paradyn and other open source tools are currently not available at SLAC.

Libraries and Tools

  • HDF5
    A parallel version of this file format is available for 32 bit Linux under
    /afs/slac/package/hdf/5/1.6.5-parallel/i386_linux24/{bin,lib,include}/
    To use it, you need to also use the production version of MPICH for Myrinet.
  • Linear Algebra Packages
    There is no parallel version of these packages available at SLAC.
  • Parallel Filesystems:
    There are no parallel filesystems available at SLAC.

Further Reading






Alf Wachsmann. Last Modified: Jan 31, 2008.