Commands used to "stage" data to and from Silo

1. Getting data back from Silo to disk

   The E143 DST data are all on Silo tapes. There is a catalog of tape number, file number and run number. You need to move the data to a disk file before using it. The command to do this is:

   stagein -VTAPENAME -qFILENUM -FF -L32768 -sMMM FN

   where:
   TAPENAME = Silo tapename, eg qa0072
   FILENUM = file number on tape, eg 1
   MMM = maximum disk file size in MB, eg 300
   FN = filename which will be used for this data

   Data will be copied from tape to the staging disk, which is dedicated to this useage. For normal operation, you do not need to know the actual name of this disk. There must be sufficient room on the staging disk for a file up to MMM MB in size; otherwise, stagein will fail. Hence, do not specify MMM much larger than it needs to be. The catalog has information to help make this estimate. I am not sure what happens if the actual file is longer than MMM MB. Specifying 300 MB is sufficient for every run; however, most runs are significantly smaller. Using a more realistic number gives you a better chance of finding that much free space.

   The filename FN may point to any disk or partition, even one with practically no free space (eg your home directory), because stagein does not actually put the data there. It will simply contruct a symbolic link from FN to wherever the file really is on the staging disk. When stagein returns, you can direct the analysis program to use FN as input.

   This command may take some time to execute. If the file is already on disk (eg from an earlier stagein command by you or somebody else), stagein should know about it and immediately return with FN set up to link to the file.

   The disk file will be purged by the staging system when it needs room for other datasets; however, the link FN is your responsibility. Use the rm command to get rid of it. Note that the real disk file could be gone even if you have a link pointing to it, i.e. always execute the stagein command, and do not rely on the presence of a lin.

   There is also a command which integrates stagein with the analysis program. See silo analysis documentation for details.

2. Other useful staging commands

   If you want to get rid of a staged disk file, say
   stageclr -VTAPENAME -qFILENUM
You can only stageclr your own files. Please do not use the standard Unix command rm for this purpose. The space freed up may not be known to the staging system.

   Use stageqry -s to see how much space is available in the staging system.

   If you want to see which tapes are active, issue the command tpstat. Unlike the other commands, you must be logged onto the Unix machine known as silosrv1.

3. Moving data from 8-mm tape to Silo

   This only needs to be done once, so you may never have to do it. Make sure you are the rightful owner before writing to the tape!

   Copy data from 8-mm tape to disk one file at a time using the usual E143 program and anal-setup.dat file. This disk data file is then in the right format for any E143 job to read. Repeat for as many files as you want to put on a Silo tape.

   We will then move these files to Silo tape:

   stagewrt -vTAPENAME -q1-NFILE -lsl -FF -L32768 -gCART FN1 FN2 ...

   where:
   TAPENAME = Silo tape name, eg qa0072
   NFILE = # files to copy, eg 4
   CART = a fixed keyword, not to be substituted
   FN1 = filename of first disk file
   FN2 = filename of second disk file
   etc
   If this command is not issued from the directory where the disk data files are residing, filenames need to be fully qualified, eg /nfs/morgan01/u2/e143/FILENAME. The disk files are automatically deleted upon successful completion of stagewrt. You can override this feature and keep them by adding the -K option.

   This command takes a bit of time, so it might be a good idea to run it in batch or in background. Be sure to keep an error file, and check that it runs OK.

4. Reserving staging space for writing

   This is useful when copying data from 8-mm tape to disk, as discussed above. It can, of course, be used for any output file.

   stageout -VTAPENAME -qFILENUM -lnl -FF -L32768 -sMMM -poutput FN ...

   where:
   TAPENAME = (arbitrary) tapename, e.g. FOO
   FILENUM = (arbitrary) file number, eg 1
   MMM = space to be reserved in MB, eg 300
   FN = filename associated with it, e.g. BAR

   When this command returns successfully, it will have created a link from FN to an actual disk location. You can then simply write to FN.

   Note that you have reserved MMM MB of space, but your actual useage may be more or less. If you try to write more than the reserved space and if there is indeed more space available, it will work.