Tapes and Tape Archiving at SLAC

Last Updated: 20 Jun 2008

At SLAC most archiving is done automatically through HPSS and users may use mstore to save their data in this system. Users no longer need to maintain their own tape and tape drive.

Individual User Space

Individual users who wish to archive their private data collection can create space in mstore and write files to this space. By default, this space is readable by everyone and writeable by the individual (a.k.a the owner). Users create the space using the mstore commands. See documentation to change the default privileges.

Group Space

Users working together often want a common area where data is archived . In mstore this is referred to as group space. A grpname must first be defined and this is done by submitting the Group Space Request Form. Once established, users who are members of the Unix group whose name is identical to grpname, can use the mstore commands to manage data in the directory sub-tree under grpname. It may be desirable to designate a person to do the archiving and pass that task on to other members when the designated person moves on (i.e. graduates, joins another experiment, etc). The Group Space Request Form can be used to request both mstore and astore space even though the form only mentions mstore.

Recommendations

Access to mstore is only available from one of the farm machines like noric or tersk. The commands will not work from machines like flora.

The recommended minimum filesize for mstore is between 100 MB and 10 GB. If you have many small files tar them up into one large file before storing them. Writing out many small files puts an undesired load on the system. You may also want to use the -p option on the mstore put command so that the files do not take up all the online disk space ( -p means purge and files are transferred to tape ).

If you are storing large files (i.e. >10GB) or intend to archive a large amount of data (i.e. 100's GB), use astore.

There is a limited amount of disk space that is shared by all users.

Retrieving and reading your files will be easy if you archive them in manageable sizes. There are no exact formulas for the perfect filesize because it depends on what data is being stored. But, keep in mind how difficult it will be to search through one large file versus a set of files divided in a way meaningful_to_you.

MSTORE Examples

Create a file in user space:

Create a file in a subdirectory (subdirectory will be created automatically) :

Retrieve an archived data set:

mstore get /nfs/mstore/u/chee/saveset

See if the file has been stored in offline storage

mstore ls -l -L offline

ASTORE

astore commands work just like mstore commands, substituting the name astore for mstore.

For astore group space, request astore group space using the Group Space Request Form.

Recommendations

The recommended minimum filesize for astore is between 10 GB and 200 GB. If you have many small files tar them up into one large file before storing them. Writing out many small files puts an undesired load on the system.

This page maintained by the Tools and Utilities group. Please email suggestions, problems, feedback to Chee