Data Transfer from 4mm/8mm to Silo Tape
This is a guide by example on how to transfer 4mm and 8mm data to silo tapes using the compute farm resources and staging system. For details on how the staging system works, explanations of options, and farm resources click here. The staging system has its own scratch and local disks that it keeps track of. It does not recognize disks outside its domain and will complain with messages like 'permission denied' if a
user tries to stage data outside the staging domain.
Drives and stackers
4mm
There are 18 slots and 2 drives in the 4mm stacker. The slots are numbered from top to bottom, 1 through 18. The device names for the 4mm stackers are:
dev/rmt4
dev/rmt5
These drives may be used simultaneously. The drive in use scans the slots and makes a note which slots are empty. The stacker door is under lock and key and also has a digital readout status window that shows when the robot is in motion. For example, if
you unlock the door, the status window tells you to wait while the robot arm parks itself out of the way. stacker218 is the program that controls the stacker and drive's actions, lists what's in the slots, and gives the drive's
status.
8mm There are 10 slots and 1 drive per 8mm stacker and there are four 8mm stackers at SLAC. The slots are numbered from top to bottom, 10 through 1. The device names for the 8mm stackers are:
dev/rmt0
dev/rmt1
dev/rmt2
dev/rmt3
A set of lights on the side of the stacker door indicates the various states the drives can be in. stacker is the program that controls the stacker and drive's actions, lists what's in the slots, and gives the drive's status.
How Usage is Scheduled
The drives/stackers are shared with several other users. Babar users may not always have access to the drives. The scheduling of who uses which stacker is handled in an informal way. A chalk board on the wall of the compute farm
contains a list of userids and phone numbers of the people currently using the drives. If a drive is empty, the person fills in their userid, device name, and phone number on the board and it is theirs to use. If a desired drive is in use, then a phone
call to the current user is usually enough to obtain use of the drive. Update the chalk board with your userid, device name, and phone number before using it. At the moment, the 4mm stacker, /dev/rmt4 and the 8mm stacker, /dev/rmt0 are reserved for
transferring the simulation data from stacker to silo. A message will be posted when this changes. An example of the chalk board may look like this:
Drive Used By ext
---------------------------
/dev/rmt0 CHEE x3353
/dev/rmt4 CHEE x3353
/dev/rmt2 WCW x2909
Reading data from 4mm or 8mm tapes
Once the tapes are loaded into the stacker, logical names must be assigned
so the system knows how to refer to them. The data can then be staged to
disk where it can be shared by one or more users. Tapes mailed to SLAC are
mapped with logical names when they are loaded into the stacker so users
should not reset these names. The 'stacker relate' information is here
for completeness.
Assign a unique, uppercase, six character name to each tape.
For 4mm:
stacker218 relate name slot device
stacker218 relate COLO01 SLOT1 /dev/rmt4
For 8mm:
stacker relate name slot device
stacker relate COLO01 SLOT1 /dev/rmt0
Read a file from the tape using the assigned logical name. In the
following examples assume the files are 60MB in size, fixed format,
lrecl of 32400 and a softlink name of dat1. For the range of files, the
softlink names are dat4, dat5, dat6.
For 4mm:
stagein -VCOLO01 -lnl -gEXB218 -FF -L32400 -q1 -s60 dat1
For 8mm:
stagein -VCOLO01 -lnl -gSTKR0 -FF -L32400 -q1 -s60 mylink
To read a range of files (4 through 6) from the same tape:
For 4mm:
stagein -VCOLO01 -lnl -gEXB218 -FF -L32400 -q4-6 -s60 dat4 dat5 dat6
For 8mm:
stagein -VCOLO01 -lnl -gSTKR0 -FF -L32400 -q4-6 -s60 dat4 dat5 dat6
Where:
- -V = volume serial or tape id
- -lnl = non-label or standard label ( lsl )
- -g = group device name (directs system to read from 4mm, 8mm, or silo)
- -FF = data is formatted fixed or unfixed (FF or FU)
- -L = logical record size
- -q = file sequence number. This can be a single file or a range
- -s = estimated size of file. it is not to your advantage to declare a number much larger than the real file size. If your file is only 30 mbytes and you set -s200 then the staging system will allocate 200 mbytes for your request, leaving less disk
space for others or worse, yourself.
Reading Data That is a Tar of a Directory
- saveset is a file that is a tar of a directory as opposed to file which refers to the individual file within a saveset.
- gtar can extract files from a saveset, tar cannot.
- 'device name' followed by a '.1' tells the drive NOT to rewind after a read. i.e. tar -vxf /dev/rmt4.1
-
Assume the following tape characteristics for examples 1 and 1A.
Logical record size 32400, fixed-format, non-labelled, 4 savesets, each
saveset having 3 files, 200 MB each, and softlink names like saveset1,
saveset2,...savesetN.
Example 1
Purpose: Stage in savesets from tape to staging disk.
Input: 4mm tape with logical name of BHABH1
Device: STACKER218 drive 4
stagein -VBHABH1 -FF -L32400 -lnl -s600 -q1-4 -gEXB218 saveset1 saveset2 saveset3 saveset4
For 8mm:
Input: 8mm tape with logical name of BHABH1
Device: STKR0
stagein -VBHABH1 -FF -L32400 -lnl -s600 -q1-4 -gSTKR0 saveset1 saveset2 saveset3 saveset4
-
Example 1A
Purpose: Extract one file from each saveset. Only gtar can do this.
(TMP is only a token needed by staging)
Reserve Disk Space on the Staging Disk for Each File
(TMP is required as a token but not used).
stageout -VTMP -s200 OUT1
stageout -VTMP -s200 OUT2
stageout -VTMP -s200 OUT3
stageout -VTMP -s200 OUT4
Make an index of files in each saveset, including the file sizes
(Only the first saveset index is shown here).
tar -tvf saveset1
Results:
drwxr-sr-x 1500/2011 0 Aug 17 08:54 1996 ./
-rwxr-xr-x 1500/1011 33808836 Aug 17 17:33 1996 ./run020001.xdr
-rwxr-xr-x 1500/1011 44427600 Aug 17 17:34 1996 ./run020001.fz
-rw-r--r-- 1500/2011 2631604 Aug 17 17:34 1996 ./run020001.log
Extract the desired files into the previously allocated output area
gtar -O -xvf saveset1 ./run020001.xdr > OUT1
gtar -O -xvf saveset2 ./run020002.xdr > OUT2
gtar -O -xvf saveset3 ./run020003.xdr > OUT3
gtar -O -xvf saveset4 ./run020004.xdr > OUT4
Writing data from the staging disks to 4mm/8mm tape
Syntax: stagewrt -Vvolume stage_opts diskfilename
After the desired data files have been read (stagein) to staging disk,
they can be written out to 4mm, 8mm, or silo tape.
Example 2
Purpose: Write a file named beam1 to an 8mm tape (assume the output tape
has been assigned the logical name of BEAM and the input data has been
staged).
stagewrt -VBEAM -lnl -FF -gSTKR0 -L32400 -q1 -s60 beam1
Write two files from disk to 8mm tape.
stagewrt -VBEAM -lnl -FF -gSTKR0 -L32400 -q1-2 -s60 beam1 beam2
Example 3
Purpose: Write two disk files, dat1 and dat2, to 4mm tape (assume the
output tape has been assigned the logical name of COLO and input data
has been staged).
stagewrt -VCOLO -lnl -gEXB218 -FF -L32400 -q1-2 -s60 dat1 dat2
Reading data from STK silo tape
Select the volume to be read. This example uses volume QJ0999.
(All silo tapes are standard label and silo is the default so '-g' is not
needed).
To stagein a single file:
stagein -VQJ0999 -L32400 -FF -lsl -q1 -s70 beam1
To stagein a range of files (Files 1 thru 3)
stagein -VQJ0999 -L32400 -FF -lsl -q1-3 -s70 beam1 beam2 beam3
(Files 5 thru 7):
stagein -VQJ0999 -L32400 -FF -lsl -q5-7 -s70 beam5 beam6 beam7
To see what was staged, use stageqry:
stageqry -VQJ0999
Writing data to STK silo tape
Three ways to write to silo tape.
- disk data to silo tape
- stacker data to silo tape
- silo data from one silo tape to another
To write disk data to silo tape either cd to the data directory or
specify the full pathname to the stagewrt command. Assume the following:
Data files are 'beam' followed by an integer (i.e. beam1, beam2), the files
are 200MB in size, and resides in the directory /u2/chee.
From directory /u2/chee:
stagewrt -Vtapename stage_opts outputname
stagewrt -VQJ0999 -FU -L32400 -lsl -s200 -q1 -fbeam1 beam1
The name beam1 is written as part of the header information
when the -f option is used.
From directory /u2:
stagewrt -VQJ0999 -FU -L32400 -lsl -s200 -q1 -fbeam1 /u2/chee/beam1
To read data from the stacker (8mm) and write to a silo tape:
stagein -VCOLO01 -FF -L32400 -lnl -gSTKR0 -q1-3 -s100 beam1 beam2 beam3
stagewrt -VQJ0999 -FF -L32400 -lsl -q1-3 -s100 -fbeam1:beam2:beam3 beam1 beam2 beam3
To write data from silo tape to silo tape.
Given input QJ0998 and output QJ0999, write the 4th through 6th files from
QJ0998 to QJ0999.
stagein -VQJ0998 -lsl -FF -L32400 -q4-6 -s200 mu4 mu5 mu6
stagewrt -VQJ0999 -lsl -FF -L32400 -q1-3 -s200 -fmu4:mu5:mu6 mu4 mu5 mu6
This will take the 4th through 6th files from QJ0998 and write them as the
first three files on QJ0999. The names of the files will be mu4, mu5, and mu6
on the output tape.
Cleaning Up
Use the stageclr command to clean up the staging disk when done.
From the examples above, the logical volume names would be specified to
relinquish disk space back to the staging system.
For Example:
stageclr -VCOLO01
stageclr -VBHABH1
stageclr -VQJ0999
For disk space that was not associated with any volume, use the token name.
For Example:
stageclr -VTMP
where TMP was the token name used with the stageout command to reserve disk
space. Users may write to this disk area but not want to save the data so
stagewrt would not be called.
Possible Problems Using the Stacker
The tar command may not always recognize a tar tape's data format.
If you get the error:
"tar: 0511-193 An error occurred while reading from the media
There is not enough memory available now".
This means tar couldn't decipher the data format while extracting
the data. Try staging in the data first, then tar -vxf
The stacker hardware is fairly robust but not perfect. If you see:
"Device not ready"
This could mean:
1) The device is being used by someone else.
2) The door to the stacker is not completely closed.
3) The program is confused and after a several minutes will reset itself.
Output silo tapes
For simulation work, contact Torre Wenaus, wenaus@mailbox.SLAC.Stanford.EDU. For other types of data contact chee@slac.stanford.edu. Include the following information in your request:
- fullname
- number of tapes required
- type of data. e.g. raw, monte carlo, backup, tar files
- your email address
- contact person, in case tapes need to be recycled
- how long will tape be needed. e.g 1 week, 6months, 1 year
Other Useful Tape Tools
- tapemap - Utility that produces a dump of a tape volume. Type 'man tapemap' for help. 8mm I/O is about 500k/second so a full 8mm tape may run several hours.
- tosilo - Script that copies disk data to silo tape. Located in BFROOT/prod/bin. For help, enter the command 'tosilo help'
- fromsilo - Script that stages files from silo tape. Located in BFROOT/prod/bin. For help, enter the command 'fromsilo help'
- tapestat Located in BFROOT/prod/bin/. Gives status of the 4mm/8mm stackers.
Maintained by Charlotte Hee. Send suggestions, comments, or additions to
chee@slac.stanford.edu, 415-926-3353
|
