-- Stanford Linear Accelerator Center

UNIX Disk Space

UNIX at SLAC
Backup
Updated: 20 Apr 2005
--

SCS maintains several varieties of permanent and temporary disk storage on the SLAC UNIX systems, each appropriate for different levels of ownership, data access, lifetime, and backup. The purpose of this memo is to aid you in selecting the right kind of storage for your needs.

User Home Directories

User home directories are allocated in the AFS file system. Each user's home directory is created in its own AFS volume with its own individual disk space quota. You may request an increase in quota by filling out the AFS Quota Request Form. This form includes current guidelines on volume sizes.

You may also use this form to request an additional volume, indicating where, within your home directory tree, you want the new volume mounted, and let us know your plans for the disk space. Planning information helps us to allocate the right kind of disk space to you, and helps us predict disk space growth (see Real Costs of Disk Space) and sometimes enables us to suggest better ways to satisfy your storage needs. For example, space for a project involving more than one person should usually be allocated as group space rather than personal space; and large data files used by I/O-intensive programs should probably not be stored in AFS at all. As a rule of thumb, we usually do not allocate more than a few gigabytes in total to an individual's AFS home directory; such large requests are nearly always an indication that there is a better way of meeting your storage requirements.

With the exception of certain special-purpose group volumes, all AFS volumes are backed up on a regular basis. AFS automatically maintains an on-line copy of the previous day's version of every volume. Your primary backup volume is always available as .backup in your home directory; you can mount other backup volumes yourself when needed. Older files can only be restored by SCS staff. You can request restoration of an older version of one of your AFS volumes via the AFS Volume Restore Request Form Please include a range of dates from which you want the volume restored. We will restore the volume and mount it at a temporary location so that you can copy whatever files you need.

Group Directories

Each group can request space for the storage of files that are of interest to the group as a whole or to subsets of the group. This space may be in either NFS or AFS (or both), depending on the characteristics of the data and its typical access patterns.

A special variant of group space is for holding and organizing web pages. This space is always in AFS, and follows structure guidelines developed by the WWW technical committee.

FTP Space

Each user and each group can request both outgoing and incoming FTP space. This space is always in AFS, and is both world-accessible through FTP and through AFS. Files should be placed in group FTP space if they will be needed after the person leaves SLAC. Only if the user wants them to disappear when they leave SLAC, should the files be stored in personal FTP space. Use the AFS Quota Request Form to request an FTP directory. Anonymous FTP at SLAC  describes how to read from and write into this space.

Local Temporary Storage

Each machine has a local directory named /tmp. In this directory you can put temporary work files. Files are automatically cleaned up from /tmp three days after their last modified date. In some operating systems, the contents of /tmp may also be lost if the machine is rebooted, so any file placed here should be easily replaceable. /tmp is never backed up, and if it is filled to the point where system operation is affected, the system administrator may remove files without warning. It is the perfect place to download a file and untar it for a cursory look, since it is not backed up.

Each machine also has a local directory named /usr/tmp. It is similar to /tmp, except that it is cleaned up after seven days.

Some machines may also have a local directory named /usr/work, in which files may be stored for up to 30 days.

Sitewide Temporary Storage

A shared work area is available to all users as /nfs/slac/work. Large projects with medium retention times can be stored there. Files are automatically deleted from this directory after 30 days from their last modification date. This filesystem is not backed up. This is a good place to build and port software systems, or to park data files temporarily.

Compute Farm Temporary Storage

Another shared work area is available to all users running on the compute farm as /nfs/farm/work. Files are automatically deleted from this directory after 7 days from their last modification date. This filesystem is not backed up. This is a good place to put large files needed by batch jobs.

Compute Farm Staging Space

For data on 3480 or 3490 cartridges in the STK silo, or on 8mm or 4mm tape, there are large temporary disks on dedicated file servers, called staging disks, intended to hold data on its way to or from tape. This space is managed by the staging system, and access is through special staging commands. The least recently used files are automatically removed from disk when the space is needed by newer files. The disk space is not backed up, since it is in principle already stored on tape somewhere.

Private Workstation Space

As the raw cost of disk space falls, and the size of low-end disks increases, you may be tempted to use the "left-over" space on the system disk of your desktop workstation. Allocating this space as a large scratch area is a reasonable way to do so. For example, you could increase the default size of the /tmp partition or (noting that /tmp is routinely erased when the system is rebooted) add a large /work filesystem.

However, SCS strongly recommends against storing any non-temporary data on your workstation and specifically does not support the storage of such data on your workstation's system disk. Here are some of the reasons:

  • Our standard procedure for updating the operating system or recovering from any non-trivial corruption of system files is to do a fresh install, destroying any data on the system disk.
  • SCS policy is to only backup our centrally-maintained file servers; it is your responsibility to backup any data you store locally.
  • SCS does not support sharing data (e.g., via NFS) between workstations, and strongly recommends against this practice because of its effect on the stability of both the server and the clients.

If you choose to put non-temporary data on your system disk it will be entirely your own responsibility to back it up before, and restore it after, operating system updates. Moreover, if the need to preserve such data prevents the timely installation of a necessary system update, you may be asked to remove your workstation from the SLAC network until the update can proceed.

If you feel your application requires the use of local storage (e.g., for performance reasons) we recommend that you: buy and install a second hard disk so that the system can be re-installed without overwriting your data; and develop and implement a procedure for backing up your data.

A note about the /scswork Filesystem:    Our standard installation guidelines include a 1 GB partition mounted at /scswork, provided there's sufficient room on the system disk. This space, if it exists, is reserved for use by SCS only. If you do not wish to give up this space to SCS, please make other arrangements before your disk is partitioned and the operating system installed.


Owner: Boeheim