******************************************************************
* *
* Stanford Data Center *
* Stanford University *
* Stanford, Ca. 94305 *
* *
* (c)Copyright 1994 by the Board of Trustees of the *
* Leland Stanford Junior University *
* All rights reserved *
* Printed in the United States of America *
* *
******************************************************************
SPIRES (TM) is a trademark of Stanford University.
This manual was written for people who own SPIRES files or who are responsible for the maintenance of SPIRES files. If you are one of these people, you should already be familiar with many aspects of SPIRES. We consider the material presented in the SPIRES primer "A Guide to Data Base Development" as the pre-requisite for this manual. For instance, if you do not know the terms "subfile", "deferred queue", "Global FOR", "File Definer" and "file definition", you should begin with that manual, which introduces some very basic concepts of File Management that we will expound in this manual.
Though the Table of Contents may not show it, this document is divided into three major portions:
In SPIRES manuals, examples of sessions are shown with prompts and messages from SPIRES as they actually appear on the terminal screen; SPIRES prompts usually end with a question mark or greater-than sign (">"), while SPIRES messages usually begin with a hyphen. The commands and responses you type are shown in lowercase, following a prompt. (You may generally use upper or lower case interchangeably when you actually use SPIRES.) In some editions of this manual, what you type will be shown in bold, while the computer's responses are shown in normal type. For example:
-OK to clear? ok
Here SPIRES types "-OK to clear?" and you type "ok".
In formal command syntax descriptions, uppercase letters denote command verbs or other command elements to be entered exactly as shown; a value for lowercase terms and characters must be supplied by you. For example:
SELECT subfile-name
To use this particular command, you type the command verb SELECT with the name of the desired subfile, for instance, "Restaurant":
-> select restaurant
where "->" is the prompt from SPIRES.
Brackets ([ ]) denote optional elements. Braces ({ }) indicate that you must specify one and only one of the alternatives within the braces. Within the braces or brackets, a vertical line (|) separates possible choices. Neither brackets nor braces are to be typed as part of the command. For example:
SHOW SPIRES MAIL [CLEAR|NOCLEAR]
could be entered as
SHOW SPIRES MAIL or SHOW SPIRES MAIL CLEAR or SHOW SPIRES MAIL NOCLEAR
depending on whether you want the option. Here is an example from the report writer command language using braces:
AT {START|TOP} OF PAGE ...
You must choose one of the words in the braces when using this command. For example, this command could be entered as
AT START OF PAGE ...
or as
AT TOP OF PAGE ...
Sections and subsections of this manual whose titles are preceded by an asterisk in parentheses -- (*) -- may be considered optional reading at that point. Usually the material provides details that most users will not need about how to handle uncommon situations, or technical information about how SPIRES internally handles some particular command.
If you are familiar with other SPIRES manuals, you will notice many substantial differences in the areas of structure and style between this manual and those. These changes result from a study to evaluate SPIRES documentation in which different contingents of SPIRES users were interviewed. Though we'd like to reprint here all the complimentary comments we received, it's perhaps more useful to discuss some of the criticisms and suggestions, and mention how this manual addresses them.
Most of the users we interviewed suggested that SPIRES documentation would be more useful if it were more "task-oriented". Most SPIRES users come to documentation trying to accomplish a specific task, and may not know or recall the names of the tools used to do it. To address this concern, we have organized most of this manual, the "User Guide" portion, by tasks. In other words, to find out how to perform a certain file management task, you would probably look up the task in the table of contents or index. All the basic information about solving the task will appear together in one chapter or often in one section of a chapter.
Another request, related to the previous one, was to provide a standardized presentation of each command with full reference material. You will find that in Part 6, with all the commands presented in alphabetic order. Hence, if you already know the name of the command you're interested in, you can quickly find the reference material on it in Part 6. Online, you will discover, if you EXPLAIN any of the commands, that the section of documentation you see comes from that part of the manual.
In addition, depending on the source of your copy of this manual, you may notice that this manual has been printed with proportional spacing, guide words in the margins, and other special typographical improvements.
Besides the new style and structure, there is plenty of new material. Though much of the information in this manual has been seen before in the reference manuals "File Definition", "Formats" and "Technical Notes", much of it is new too. We think you will appreciate having the material together in a single manual, despite any duplications between manuals.
If you think of any aspects of SPIRES file management that weren't covered in this manual and which you'd like to see addressed in a future edition, please send them to your SPIRES consultant, or send them directly to SPIRES Documentation, Stanford Data Center, Stanford University, Stanford, California 94305.
This manual is the creation of many contributors, including the earliest designers of SPIRES back in the late 1960's and recent users who suggested improvements to the explanation of this or that feature in SPIBILD. In general tribute to all, we here thank many specific individuals for their significant help.
SPIRES is developed and maintained by the SPIRALS team of Stanford University's Stanford Data Center. Programming responsibility for SPIRES is shared by Dick Guertin and Bill Kiefer, both of whom contributed enormously to the development of this manual. Chuck Boeheim, responsible for the VM/CMS implementation of SPIRES, was very helpful in that area. Sandy Laws, Peter Tuttle, Lynn McRae, Norman Roth and Becky Morton also provided important technical support and review.
Others outside of SPIRALS provided concrete suggestions about the manual's contents, including Ed Begley of I.T.S.'s Community Information Resources and Hannah Kaufmann and Michael Sperberg-McQueen of Princeton University's Center for Information Technology.
The style of this manual is significantly different from SPIRES manuals written in the past, as is its physical design. Major contributors in this area include DRG staff members Lynne Sinclair, Jeff Rensch and Tony Navarrete, as well as Terry Oldano, Director's Office, and Anne Janzer, Systems. A new GML format for this document was patiently and painstakingly created by Greg Bruce.
For printing support, we acknowledge the assistance of staff members Lloyd Warble and Mark Lawrence of Electronic Printing and Lincoln Ong of Operations. Charley Hoyt of Systems deserves special commendation for his customization work on Waterloo Script's Photo processor especially for this document.
The manual was written by John Klemm, who thanks those named above and everyone else who provided ideas, patience, and non-addictive stimuli when they were most needed.
Managing a SPIRES file consists of tasks in three broad categories:
- record-transaction tasks, consisting of adding, updating or removing "batches" of records from the file; [You might consider that adding, updating and removing records individually in SPIRES are file management tasks, albeit on a smaller scale. But in general, the term "file management tasks" refers to work involving groups of records rather than individual ones.]
- routine maintenance tasks, ensuring that the records are being stored efficiently, that the indexes are being updated, that the file has a backup copy in storage, etc.
- repair tasks, perhaps involving the destruction or reconstruction of an index, an entire subfile or an entire file, perhaps done in response to information from a maintenance task.
The same file management capabilities are available to all SPIRES files, regardless of size. But many of them are not usually needed by small files; the test to determine whether SPIRES is storing some data inefficiently could cost more than the inefficiency itself does. But for large files (say, over 100,000 records), such tests could be very cost-effective in the long run, which is how long such files generally last.
The other parts of this manual cover the file management tasks in detail, principally for the benefit of owners of large files. Owners of small files typically do not have the need, interest or incentive to learn so much about SPIRES, and certainly shouldn't have to do so.
In view of that, the rest of this part of the manual introduces the most important aspects of file management for all SPIRES files. If you own a small file, you will probably find enough information here to meet your needs; if not, suggestions for further reading in the rest of the document are made along the way. But even if you own or manage larger files, you may find this introduction useful too.
Many of the terms that show up in discussions of SPIRES file management are informally defined in a glossary at the end of this manual. The index in the back includes a complete list of the file management tasks addressed by this manual, under the heading "file management task", with page number references to the appropriate sections, of course. Online, you can see the list by issuing the command EXPLAIN FILE MANAGEMENT TASK, which also gives you online access to all those sections of the manual too.
For small SPIRES files, file management may be quite simple:
- records are updated one at a time;
- the JOBGEN program causes the file to be processed every night so that the tree is updated using the information in the deferred queue, and the indexes are updated;
- the file is destroyed with the ZAP FILE command when it is no longer needed.
In a sense, such files have little use for "file-management procedures"; SPIRES takes care of the most important task (automatic processing of the file) without any intervention from you.
JOBGEN is a program, running after midnight each night, that examines the deferred queue of each SPIRES file requesting this service. If the deferred queue is empty, JOBGEN does nothing more to that file. But if it is not empty, JOBGEN creates a job to process the file in the SPIBILD program ("SPI" stands for SPIRES; "BILD" stands for "file-building". SPIBILD's tasks center around file-building.); the job usually runs a few minutes later. The file owner's account is charged for the cost of the job. [See 2.2.1.]
JOBGEN runs during the night because computing rates are cheaper then, and because the files are less likely to be in use at that time. (In general, SPIBILD jobs need exclusive access to a file during processing.)
The AUTOGEN flag, which is most often controlled by the file owner, determines whether JOBGEN is responsible for processing the file. The flag is set in one of three ways:
- 1) When the file is defined, the file owner determines a default setting for the AUTOGEN flag. Specifically, if the file definition contains the NOAUTOGEN statement, then NOAUTOGEN is the basic (default) setting for the file, i.e., JOBGEN is not responsible for the file. (File Definer's NOAUTOGEN attribute of the FILE statement has the same effect.) Otherwise, the basic setting is AUTOGEN.
- 2) The file owner, or a user with Master access to the file [The file definition may give other users certain file-owner powers -- this is called "Master access". This, and other forms of file access, such as Process access, are discussed in the glossary at the end of the manual.] may issue the SET AUTOGEN or SET NOAUTOGEN command to change to the desired setting.
- 3) Every time the file is processed, by either JOBGEN or you, the file returns to its basic setting, according to the file definition (see step 1).
To turn off automatic processing for awhile, you can just issue the SET NOAUTOGEN command when you have the subfile selected in SPIRES:
-> select staff addresses -> show autogen -AUTOGEN in effect -> set noautogen ->
On the other hand, if you have NOAUTOGEN in effect but decide that it's time for the file to be processed again, you can issue the SET AUTOGEN command, and the next time JOBGEN finds data in the deferred queue, the file will be processed:
-> show autogen -NOAUTOGEN in effect -> set autogen ->
Remember that the file will revert to its default setting of AUTOGEN or NOAUTOGEN, according to the file definition, after the file is processed.
Occasionally you may not want to wait for overnight processing of a file by JOBGEN. You may need your indexes updated immediately, for one reason or another, so you need the file processed now. (This scenario assumes the indexes aren't "immediate indexes", which are updated each time a record transaction occurs.)
Processing a file is quite a simple task to do, but unless you have compelling reasons to process the file yourself, you should leave it to JOBGEN. [See 2.2.1.]
If you are the file owner, or if you have PROCESS access to the file, you can process a SPIRES file. Processing the file is most commonly done in the SPIBILD program. To process a SPIRES file, you first call SPIBILD and then issue the PROCESS command:
-> spibild
-Welcome to SPIBILD
-? process orv.gq.doc.addresses
-Begin passing of 4 records of record-type REC01
-Completed passing of 4 records total
-Processed: 4 DEFQ records of record-type REC01
-Detaching File: GQ.DOC.ADDRESSES
08/11/90 File Read/Write Counts Information
File: GQ.DOC.ADDRESSES Reads Writes
CKPT data set 4 1
DEFQ data set 7 4
MSTR data set 3 0
Residual data set 3 6
Total for file: 17 11
-Compute time: 0.246 seconds
-Elapsed time: 2.407 seconds
-Core usage: 0267/0939
-? exit
-Exiting SPIBILD
Command>
The commands SPIBILD and EXIT take you in and out of SPIBILD. The SPIBILD program is similar to SPIRES in that you can issue WYLBUR and MILTEN commands within its environment, though you cannot issue SPIRES commands.
The PROCESS command causes the named file to be processed, and the output from the command as the processing occurs usually looks like the example above. If you try it but have problems, try using the EXPLAIN command to find out more about any error messages you received -- the EXPLAIN command is available in SPIBILD.
You can also process the file in SPIRES itself, without having to go into the SPIBILD program. Processing the file in SPIRES has some limitations that may affect you, but in most cases, they will not be significant. The advantages of processing in SPIRES include the ability to put the PROCESS command into a protocol, and being able to direct the error messages into the "trace log" (SET TLOG). [See 2.2.2 for a comparison of file-processing in SPIBILD and SPIRES.]
Most of the time, SPIBILD processes your files without problems. However, sometimes power outages or system problems occur during a SPIBILD session, causing the file processing to stop in mid-stream.
You generally find out that your file has been affected by a SPIBILD problem in one of three ways:
- the computer tells you that a problem occurred as you are processing the file in SPIBILD;
- you get output from JOBGEN that tells you that the file-processing job it created did not run properly;
- users discover that they cannot accomplish any subfile transactions (adds, updates, removes) for the file because the transactions are blocked when a SPIBILD processing error has occurred.
This doesn't mean the file is damaged; it merely means that the file is unstable, and normal subfile transactions cannot resume till SPIBILD cleans up whatever problem it had.
In general, the best way to "recover the file" is to issue the RECOVER command in SPIBILD, naming the file to recover:
Command> spibild
-Welcome to SPIBILD
-? recover gq.doc.addresses
-Begin passing of 12 records of record-type REC01
-Completed passing of 12 records total
-Processed: 12 DEFQ records of record-type REC01
-Detaching File: GQ.DOC.ADDRESSES
08/11/90 File Read/Write Counts Information
File: GQ.DOC.ADDRESSES Reads Writes
CKPT data set 4 1
DEFQ data set 4 4
MSTR data set 2 0
Residual data set 3 12
Rec-type REC01 2 2
Rec-type ZIN02 2 5
Total for file: 17 24
-Compute time: 1.833 seconds
-Elapsed time: 5.900 seconds
-Core usage: 0210/0939
-?
The RECOVER command begins to process the file again, though if there is a large amount of deferred queue data, the RECOVER command may successfully complete without actually finishing the file processing. When the RECOVER command completes its work, the file should be back to normal.
On the other hand, if you receive error messages rather than the messages shown in the example above, your file still has problems that need to be resolved. [See 2.1.4 for a detailed discussion of SPIBILD problems you might encounter.]
In rare cases when a file's problems are so serious that its integrity is questionable, or when a disk-storage problem arises, your computer center may need to use backup copies of data sets to restore the file to its state at the point when the backup was made. If you believe you need to take advantage of this service, contact your SPIRES consultant for help.
One of the most common duties of the file manager is to "build the file" -- that is, add many records to the file at once to get it started. Another common task that arises occasionally is to add, update and/or remove many records of a subfile at one time.
There are quite a few ways to apply multiple record-transactions to a subfile at once. In general, the easiest and least expensive methods (to the user doing the work) use SPIRES, putting the record transactions into the deferred queue. Then, when the file is processed in SPIBILD, perhaps automatically overnight, the deferred queue data will be applied to the tree of the subfile.
On the other hand, if getting the data into the tree and having the indexes reflect the new data immediately is important to you, you can use SPIBILD techniques instead. Remember that they may cost more than the SPIRES techniques, particularly if the processing is not occurring overnight, when rates are likely to be cheaper. On the other hand, if all the charges are being borne by the file owner, then doing all the work at once in SPIBILD is likely to be cheaper than putting the data first into the deferred queue in SPIRES and later having SPIBILD move the data into the tree.
Descriptions and comparisons of all the different procedures you can use for multiple record-transactions appear later, in several chapters of Part 2. [See 2.3, 2.4, 2.5.] Here we will discuss the three simplest and most general procedures of SPIBILD, probably the most useful ones to owners of small files:
- the BATCH procedure in SPIBILD, which allows you to add new records and update, merge and remove records whose keys you know; and
- the MERGE procedure in Global FOR in SPIBILD, which lets you update multiple records at once using the same merge input for each one, choosing the records to be updated with non-key criteria.
- the REMOVE command in Global FOR in SPIBILD, which lets you remove multiple records at once using non-key criteria to determine the records to remove.
The INPUT BATCH command in SPIBILD provides a fast and reliable method of getting data into a SPIRES subfile and getting the indexes updated. It's available to any user who can process the file in SPIBILD.
SPIBILD examines the data for each added and updated record, rejecting it if it breaks any of the processing rules defined for the subfile. Each record that succeeds is placed in the tree of the file. For removes, SPIBILD removes that record from the tree. SPIBILD then updates any indexes defined for the subfile.
For this procedure, your data must be in the standard SPIRES format, with each record preceded by a command followed by a semicolon ("ADD;" or "UPDATE 226-82-3846;") and (except for REMOVE) followed by an additional semicolon. [See 2.3.5.]
Here's a brief example of some data in the standard SPIRES format:
ADD; ITEM = 24-ounce Klein bottles; UNIT = carton of 12; UNIT.PRICE = $10.50; STOCK = 12; ; MERGE 134; ITEM = Mobius film strips; UNIT = "a ""grab bag"" of 10 assorted subjects"; ; REMOVE 19;
There is one caveat about using this procedure: the ESTABLISH command used at the start processes the file, which processes all the transactions currently in the deferred queue. If you don't want that to happen for some reason, consider using the INPUT BATCH command in SPIRES.
Here are the steps of the INPUT BATCH procedure in SPIBILD:
If you aren't already in SPIBILD, issue the SPIBILD command:
Command> spibild -Welcome to SPIBILD -?
You must have the input data in your active file. If you don't already have it there, use the appropriate WYLBUR command to put it there. For example,
-? use wyl.gq.jnk.supply.input -?
You will get one line of informational data about each successful record from the INPUT BATCH command by default. If you are working with dozens of records, those messages can be tedious, and they add a small degree of inefficiency too. You can turn them off by issuing the command SET INFORMATION MESSAGES = 0; it does not suppress any error messages. [See 2.1.5.3.]
It is a good idea to tell SPIRES to put the data for transactions that fail into a special data set called an "exception file". By extracting the bad data from the good data, SPIRES makes it easier for you to find the problems in the data so that you can correct it and submit it again with a subsequent INPUT BATCH command.
The exception file is an ORVYL data set that you name yourself, in the SET EXCEPTION command. For example:
-? set exception bad.supplies.records -?
SPIRES will put any errant records into the ORVYL data set called BAD.SUPPLIES.RECORDS on your account. [See 2.1.5.2.]
Issue the ESTABLISH command to tell SPIBILD which subfile you want to use and to have SPIBILD process the file containing it:
ESTABLISH subfile-name
For example:
-? establish supplies -Processing file: GQ.JNK.SUPPLIES -Begin passing of 2 records of record-type REC01 -Completed passing of 2 records total -Processed: 2 DEFQ records of record-type REC01 -Compute time: 0.196 seconds -Elapsed time: 1.227 seconds -Core usage: 0210/0939 -?
Next issue the INPUT BATCH command to tell SPIBILD to read the input data from your active file and apply it to the subfile:
INPUT BATCH
Continuing the above example:
-? input batch
- ADD 1, 1 @Line 6. Key = 356
- UPD 1, 1 @Line 10. Key = 134
- REM 1, 1 @Line 11. Key = 19
... (etc.) ...
-Begin passing of 94 records of record-type REC01
-Completed passing of 94 records total
- Requests/Success:
ADD 72 72
UPD 16 16
REM 6 6
SUM 94 94
-End of BATCH/BUILD
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 9 7
MSTR data set 6 0
Residual data set 53 51
Rec-type REC01 2 14
Rec-type ZIN02 1 13
Total for file: 75 86
-Compute time: 4.688 seconds
-Elapsed time: 14.448 seconds
-Core usage: 0279/0939
-?
In this example, the information messages were not turned off (step 3); had they been, the first three lines shown would not have appeared, since they indicate successful transactions.
The most common problem is that one or more transactions fail due to input errors, such as an invalid element name, or an improper element value. SPIBILD will display error messages telling you what the problem is. Assuming that you have set an exception file, it is easy to retrieve the unsuccessful data, using the ORVYL command GET:
-? get bad.supplies.records OK to clear? ok -?
The data is now in your active file. After correcting it, you can return to step 3 and continue the procedure.
If the system fails during any step of the procedure except step 6, just restart from the beginning when it is convenient to. If it fails during the INPUT BATCH command, you need to determine the point where it failed, i.e., figure out which was the last record to be fully processed. A command called SHOW RECORD RECOVERY may be useful in this regard. For full details, see the complete discussion on the INPUT BATCH procedure in SPIBILD. [See 2.3.1.]
Frequently the file manager may find that many records sharing some element value need to have that element value changed or that they need to be removed from the subfile. Removing records will be discussed in the next section; here we will discuss updating them.
In the updating situation, SPIBILD provides a MERGE ALL command using Global FOR, which merges the data in your active file into all the records that meet the Global FOR criteria. Like SPIBILD's BATCH procedure, this procedure causes the file to be processed prior to the MERGE command and SPIBILD processes the file again at the end of it, so it can be relatively expensive. [See 2.4.2.]
The procedure for using it is described below. Note: You must be able to process the file to use this procedure.
If you aren't in SPIBILD already, get in:
Command> spibild -Welcome to SPIBILD ->
Tell SPIBILD which subfile you want with the ESTABLISH command, which also processes the file:
-> establish supplies -Processing file: GQ.JNK.SUPPLIES -Compute time: 0.104 seconds -Elapsed time: 1.073 seconds -Core usage: 0130/0939 ->
In this example, the intent is to find records with the value RCA for the singly-occurring element SUPPLIER and replace it with the value "General Electric". See the SPIRES manual "Searching and Updating" for more information on merging in general (for instance, how to add new occurrences, replace multiple existing ones, remove occurrences, etc.).
You must have the input data in your active file. It can be in the standard SPIRES format or in a custom format; if the latter, you should issue the SET FORMAT command as part of this step. [The format must be a merge format. See the manual "SPIRES Formats" for further information.] For example,
-> collect
1. > supplier = General Electric;
2. > *** <- user presses ATTN/BREAK
->
In SPIBILD, the only Global FOR classes available are the SUBFILE and STORED classes. You can, however, use a WHERE clause and SET SCAN commands to further refine the group of records to be updated.
For example,
-? for subfile where supplier = rca -? set scan start 10000 -? set scan stop 19999 -?
The SET SCAN START and STOP commands tell SPIBILD to look only at records with key values from 10000 to 19999. This is considerably more efficient than "FOR SUBFILE WHERE SUPPLIER = RCA" alone, if all the records you want are within the limited range.
The STORED class would be preferable to use if, for example, you could easily create a stored result or stack in SPIRES of the records you wanted to update. Then SPIBILD could retrieve the records for updating directly, rather than have to use WHERE-clause examination techniques to find the records to update. Suppose, for instance, that in SPIRES you could store the search result from the command FIND SUPPLIER = RCA. Using that stored search would be much more efficient than having SPIBILD examine each record in the subfile.
The MERGE ALL command merges the data in your active file into each record fitting the Global FOR class, after which SPIBILD processes the file so that the indexes are all updated.
-? merge all from active
- Requests/Success:
MER 38 38
SUM 38 38
-End of BATCH/BUILD
-Begin passing of 38 records of record-type REC01
-Completed passing of 38 records total
-Processing: 38 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 6 82
MSTR data set 5 0
Residual data set 4 46
Rec-type REC01 1 0
Rec-type ZIN02 1 25
Total for file: 21 154
-Compute time: 7.894 seconds
-Elapsed time: 19.521 seconds
-Core usage: 0256/0933
-?
If an error occurs during step 5, SPIRES will tell you about it and stop. The most likely error comes from bad input data, i.e., an invalid element name or value for that element. That will most likely happen on the very first record -- if it won't go into one record, it probably won't go into any.
To recover from this problem, simply correct the data and start again from the ESTABLISH command in step 2.
If you discover that you updated the wrong records, you will need to determine another way to find those records and perhaps "re-merge" the old data into them.
If the system crashes during the first 4 steps of the procedure, simply restart it at your convenience. If the system crashes during the MERGE processing, see the explanation for handling this problem in the full discussion of the MERGE procedure in Global FOR in SPIBILD later in this manual. [See 2.4.2.] A simple possibility worth mentioning here is to simply restart the procedure and merge the same data again into the records that succeeded. Do not do this, though, if the merge requests are creating new element occurrences or if the data is being used to compute other element values.
Sometime you may have to remove many records from the subfile based on some criteria they share, such as all records added to the subfile more than two years ago, or records that don't have occurrences of some element.
This situation is quite easily handled in SPIRES, using the REMOVE command in Global FOR mode. REMOVE is also available under Global FOR in SPIBILD and has the same effect; in fact, it works more efficiently there than in SPIRES. However, you are limited to the Global FOR classes SUBFILE (with WHERE clauses, SET SCAN and SET FILTER commands to limit the records examined and removed) and STORED (with WHERE clauses, the SET SCAN LIMIT and SET FILTER commands) in SPIBILD.
Suppose, for example, that you want to remove 50 records of a 1000-record subfile, and you can find those 50 records using a simple search:
-> find supplier = woolco -Result: 50 SUPPLIES ->
In that case, you may want to simply remove the records in the search result using SPIRES:
-> for result +> remove all -Removed: 50 record(s) +>
The removal requests are in the deferred queue, so the records won't be removed from the tree till the file is next processed.
Another possibility would be to store the result (or stack) in SPIRES for use under the FOR STORED command in SPIBILD:
-> show result -Result: 50 SUPPLIES -> store result.woolco -'RESULT.WOOLCO' put in ORVYL file system ->
In SPIBILD, you can use the command FOR STORED RESULT.WOOLCO to remove only the records in the result (see step 3 below); under SPIBILD processing the file would be processed, and the records would be removed from the tree and "un-indexed" immediately.
Especially if there are several hundred records to remove (say, more than 15 percent of the subfile), the SPIBILD methods discussed below are preferable.
Warning: If the subfile is fairly large (say, more than 1000 records) and you want to remove more than half of them, you should consider zapping the goal record-type and rebuilding it with only the records you want to keep. [See 2.8.3.]
One final aspect to consider between the SPIRES and SPIBILD methods of removing records in Global FOR -- in SPIRES, the removal requests go into the deferred queue and can be unqueued if you later determine that you removed the wrong records. In SPIBILD, once they are removed, they are gone. Often when file managers remove many records from a subfile using SPIBILD, they make sure they have a backup copy of the file around, just in case. [See 2.1.4.3.]
The procedure for removing multiple records in SPIBILD is described below. If you plan to use FOR STORED processing, as described above, create your stored result or stack in SPIRES before beginning. Note: You must be able to process the file to use this procedure.
If you aren't in SPIBILD already, issue the SPIBILD command:
-> spibild -Welcome to SPIBILD -?
Tell SPIBILD which subfile you want with the ESTABLISH command, which also processes the file:
-? establish supplies -Processing file: GQ.JNK.SUPPLIES -Compute time: 0.208 seconds -Elapsed time: 1.037 seconds -Core usage: 0130/0939 -?
Remember that you are limited here to the FOR SUBFILE or FOR STORED command, with an optional WHERE clause. You should also use SET SCAN commands to limit the records examined, if you can, since that will save unnecessary work.
If you're not familiar with the SET SCAN commands, you might think of them this way: suppose you are asked to look for one book in the library, and you have no card catalog (index) to use. You must look on every shelf in the library till you find the book you want. But if you are told that the book you want must be on the north end of the third floor, the work you must do has been narrowed considerably. The SET SCAN commands do that type of limiting, telling SPIRES what record keys to examine.
For example, here we are removing old records from a subfile with slot keys:
-? for subfile where date-added < 1985 -? set scan stop 500 -?
It happens that we know that no records added before 1985 have keys greater than 500. The SET SCAN STOP command shown ensures that only records whose keys are 500 or less will be examined for removal.
To use FOR STORED instead, you would replace FOR SUBFILE with FOR STORED, along with the name of the stored stack or result.
-? for stored result.woolco
You may also add a WHERE clause to further limit the records that are removed; SET SCAN LIMIT is the only SET SCAN command allowed with FOR STORED, however.
Issue the REMOVE ALL command to tell SPIBILD to remove all the records that match the criteria established by the Global FOR class:
-? remove all
- REM 1, 1 @Line 1. Key = 131
- REM 2, 2 @Line 1. Key = 132
- REM 3, 3 @Line 1. Key = 134
- REM 4, 4 @Line 1. Key = 135
... (etc.) ...
- Requests/Success:
REM 58 58
SUM 58 58
-End of BATCH/BUILD
-Begin passing of 58 records of record-type REC01
-Completed passing of 58 records total
-Processed: 58 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 6 120
MSTR data set 5 0
Residual data set 7 65
Rec-type REC01 2 58
Rec-type ZIN02 1 20
Total for file: 25 264
-Compute time: 0.516 seconds
-Elapsed time: 13.240 seconds
-Core usage: 0204/0933
-?
If you discover that you removed the wrong records, the records are gone and cannot be recovered except via any backup copies of the file or data you may be able to use.
If the system crashes during the procedure, you may simply restart the procedure at your convenience.
As its manager, you have access to many interesting statistics about your SPIRES file, such as the amount of storage space the file is using, or the kinds of commands users are issuing when they select your subfile.
Here are some of the statistical questions you might ask about your file, along with possible answers.
This is a useful question to ask because storage charges are based on the number of blocks the file uses. To find out the answer, issue the ORVYL command SHOW FILES, including the name of the file and the BLOCKS option:
-> show files like almanac, blocks ORVYL File System ALMANAC.CKPT -- 7 BLKS (F: 0, L: 6) ALMANAC.DEFQ -- 2 BLKS (F: 0, L: 1) ALMANAC.MSTR -- 6 BLKS (F: 0, L: 5) ALMANAC.REC1 -- 14 BLKS (F: 0, L: 13) ALMANAC.REC2 -- 108 BLKS (F: 0, L: 107) ALMANAC.RES -- 123 BLKS (F: 0, L: 122) ->
The total of the number of blocks (the "BLKS") is the size of your file (260 in this example). [See 3.2.1.]
One way to get these statistics is to issue the SHOW FILE ACTIVITY command when you have a subfile of the file selected. For example,
-> select almanac
-> show file activity
01/26/87 Activity Record of File GQ.JNK.ALMANAC
Last Compiled: 01/07/87 at 23:55:02
Record Base Blocks Adds Deletes Updates Max Rec
Type Block Used -DS Length
ENTRY -Slot 0 14 -1 3272 0 22 339
REC2 1 105 -2 373 0 1841 72
REC3 8 Comb -2 3180 6 89 18
REC4 25 Comb -2 2564 2 407 417
REC5 103 Comb -2 7 0 21 2004
REC6 104 Comb -2 2 0 10 11063
Residual 2 123 3488 7 414 11067
Def Queue 1 3
->
By subtracting the "Deletes" from the "Adds", you get the number of records in each record-type. The ENTRY record-type, for instance, has 3272 entries (3272 minus 0).
The SHOW FILE ACTIVITY command displays some other useful information too, such as the date and time when the file was last compiled. [See 4.2.2.]
Another way is to use the ATTACH command to "attach" each record-type of the file, followed by the SHOW SUBFILE SIZE command. The SHOW SUBFILE SIZE command does exactly the work you do -- subtracting the "Deletes" from the "Adds":
-> attach 1 of gq.jnk.almanac -> show select -Record-type ENTRY of GQ.JNK.ALMANAC attached -> show subfile size -The subfile has 3272 records -> attach 2 of gq.jnk.almanac (etc.)
The ATTACH command lets you use any record-type of a file as if it were the goal record-type of a selected subfile. [See 5.4.]
If you need more information about a particular transaction in the deferred queue, such as when it happened or who did it, issue the SHOW SUBFILE TRANSACTIONS command, optionally followed by the key of the record you are interested in. For example,
-> show subfile transactions 12933
01/26/87 Transaction Log of File GQ.DOC.ERRATA
for Record-type G1SECT
Date Time Account Id Type Command Grp Key Value
01/26/87 15:45:28 GQ.JNK ADD Update 12933
01/26/87 13:32:35 GQ.JNK ADD Update 12933
01/26/87 10:54:10 GQ.JNK ADD Add 12933
->
Information about the transactions for the record that have been made since the file was last processed are shown, listed in reverse chronological order. This example shows that user GQ.JNK added the record at 10:54 a.m., and updated it twice later in the afternoon.
If you omit a specific key from the command, SPIRES will list all the transactions for the selected subfile that are in the deferred queue, in chronological order. More information about how to read the transaction log appears later in this manual. [See 4.2.3.1.]
This is representative of many questions you may have about how users are taking advantage (or failing to) of your subfile. You may want to know how many users are selecting it, or what commands they are issuing to use it.
You can ask SPIRES to "log" the use of your subfile by adding some statements to your file definition (directly or via your File Definer input). Later you can analyze the log's contents to help you evaluate the usefulness of your file, and to help you look for ways it could be improved, e.g., new indexes it needs, current indexes it doesn't need, etc.
Directions for establishing a log appear later in this manual. [See 4.3.]
Other statistical information can tell the file manager how efficiently the file's blocks are being used for storage, and how many levels deep are the trees of the various data sets of the file. This information, and much more, is available through the STATUS and SHOW FILE STATUS commands. [See 3.2.3.] But in general, those commands are of interest (and of need) to managers of large files.
When you no longer need a SPIRES file, you should destroy it, so that it will stop accruing storage charges. If the data contained in it is important to you, you might want to make a printed listing of the data or perhaps save the data in another data set on disk or tape before you "zap" the file.
The SPIRES command that erases the file is ZAP FILE, which names the file to be erased. It is quite easy to use (almost too easy, considering its drastic effects):
-> zap file gq.jnk.supplies -Do you really want to destroy file GQ.JNK.SUPPLIES ? ok ->
Once you confirm your desire, the ORVYL data sets of the file are quickly erased and the file is gone. Only the file owner can issue the ZAP FILE command for a file. [See 2.11.]
After a ZAP FILE command, the file definition still exists in the FILEDEF subfile; if you no longer need it, you should remove it from there, or consider moving it to the BACKFILE subfile. If you don't do one or the other, the subfiles of the file will continue to show up in the list from the SHOW SUBFILES command.
Also get rid of code for the file from other system subfiles, such as FORMATS, RECDEF or EXTDEF, as part of your cleanup activities. [See 2.11 for the complete description of the procedure, which describes how to find and get rid of any other code.]
The next four parts of this manual are a "User Guide" to file management. In these sections you'll find descriptions of procedures you can use to accomplish various file management tasks.
Here in Part 2, the subject is the building and destroying of record-types, subfiles and files, the most important responsibilities of every SPIRES file manager.
Many of the procedures described in this part of the manual require you to use a program called SPIBILD. The rest of this first chapter describes the SPIBILD program in general, introducing various commands and "sub-procedures" that may be useful to you as you follow the procedures in the remainder of Part 2.
Those remaining chapters of Part 2 are:
SPIBILD is a part of SPIRES that builds the data sets containing a file's records. Most procedures that involve SPIBILD cause the file to be "processed". When you process a file, the records in the file's deferred queue -- that is, those that have been added or updated since the file was last processed -- are moved from the deferred queue to the tree. Also, any records with removal requests in the deferred queue are removed from the tree. Along the way, changes are made to the indexes as appropriate, by adding, updating and removing data from them to reflect the changes to the tree.
Processing a file can be done in SPIRES or SPIBILD, depending sometimes on specific needs but generally on personal preference. [See 2.2.2, 2.2.3 for information about processing a file in SPIRES.]
You can process your own files anytime you want, or you can allow other people to process them. Or, if you want, SPIBILD will process your files automatically each night, saving you the trouble.
You can use SPIBILD as part of many other tasks too, which are discussed later in Part 2:
- to add, update or remove many records at once
- to build new indexes for a subfile that already exists
- to remove indexes you no longer need
- to remove one subfile in a file when you no longer need it but need to keep the file
- to rebuild indexes, or entire subfiles, or entire files
The rest of Chapter 2.1 discusses aspects of SPIBILD in general, which relate to most of the tasks discussed in the rest of Part 2. The remaining sections of this chapter are:
When you want to process one of your files yourself, you must use SPIBILD, the program that builds the data sets containing SPIRES records.
The online version of SPIBILD is an ORVYL program invoked by the SPIBILD command:
SPIBILD [(parameters)]
where the most popular optional parameters are:
- QUIET, to suppress the welcome message, and
- WYLBUR, to request that the command SET WYLBUR be issued automatically
Separate them with a comma or blank to request both. See Part 6 for the full syntax, or online, [EXPLAIN SPIBILD COMMAND.]
For example:
-> spibild -Welcome to SPIBILD 87.06 -?
In this example, 87.06 is the release number, indicating that this version of SPIBILD was released in June of 1987.
Unlike SPIRES, SPIBILD has no "uplow" setting, because none of its commands are case-sensitive. [SPIRES has an uplow setting because the data the user types in some commands, such as DISPLAY or FIND, may be case-sensitive.] You can type commands in uppercase, lowercase or any mixture you like.
Once you have invoked SPIBILD, you can issue any SPIBILD command, or any WYLBUR, ORVYL or MILTEN commands. To use SPIRES commands, or those of any other SPIRES processor, you must invoke that processor, leaving SPIBILD.
To leave SPIBILD, issue the EXIT command:
EXIT [QUIET]
where the QUIET option suppresses the exit message from SPIBILD.
The EXIT command takes you back to WYLBUR:
-? exit -Exiting SPIBILD Command>
You can also leave SPIBILD by issuing a command to invoke another SPIRES processor, e.g., "SPIRES".
Between the SPIBILD and EXIT commands, you will probably issue several SPIBILD commands, as part of procedures discussed in other chapters of Part 2. Other SPIBILD commands useful in many different SPIBILD procedures are discussed later in this chapter. [See 2.1.5.]
The remainder of this section discusses features and commands that the SPIBILD environment has in common with SPIRES. Most are fully documented in chapter A.2 of the SPIRES reference manual "Searching and Updating", with specific sections referenced below. In summary, those that affect SPIBILD are:
- The EXPLAIN command is available in SPIBILD, for explanations of SPIRES terms and error messages. Subfile explanations, available in SPIRES, are not available in SPIBILD, however. Nor are the SHOW EXAMPLE and SHOW SYNTAX commands. You can put an explanation in your active file by preceding the command with the IN ACTIVE prefix (see below).
- The SHOW EVAL command is available in SPIBILD as well as in SPIRES, although many variables (those involving search results or protocol execution, for example) are not available to be evaluated. The command (documented in the "SPIRES Protocols" manual) could also be used for on-the-spot computations in SPIBILD or for evaluation of static variables before and after SPIBILD processing. One likely use of SHOW EVAL in SPIBILD might be to determine which version of the program (i.e., Test or Public) you are currently using:
-? show eval $version
87.01.27 PUBLIC System
-?
- Unless otherwise mentioned in their documentation, command verbs and their options can be abbreviated to their first three characters. [See A.2.7 of "SPIRES Searching and Updating".]
- The SET WYLBUR and SET NOWYLBUR commands can be issued to specify whether WYLBUR or SPIBILD examines commands first, passing unrecognized commands to the other program. WYLBUR Exec files will not execute in SPIBILD unless SET WYLBUR is in effect. Extended Exec files will execute in SPIBILD, however, regardless of SET WYLBUR or SET NOWYLBUR. [See A.2.5.1 of "SPIRES Searching and Updating".]
- You may use the WYLBUR active file during various SPIBILD activities. For example, it may contain data that you want to merge into a SPIRES file. The IN ACTIVE prefix, with the CLEAR and CONTINUE options, may be used on some commands to direct data to your active file. [See A.2.9 of "SPIRES Searching and Updating".] The following commands relating to the active file are applicable in SPIBILD:
- SET LIST and SET NOLIST -- The default is LIST, which means that SPIBILD will automatically list your active file at the end of any command that puts data into it. [See A.2.9.2 of "SPIRES Searching and Updating".]
- SET CLEAR and SET NOCLEAR -- The default is NOCLEAR, which means that SPIBILD will not automatically clear your active file if you use the IN ACTIVE prefix without either the CLEAR or CONTINUE option. [See A.2.9.3 of "SPIRES Searching and Updating".]
- SET LENGTH -- This command can be used to change the default line length of 72, the value of the $LENGTH variable. [See A.2.9.4 of "SPIRES Searching and Updating".]
- The ATTN/BREAK key has several uses in SPIBILD:
- If you press the key in response to a command prompt, WYLBUR will either reprompt or place you in Collect mode.
- If you press the key after you have begun typing a command but before you press Return, the command you were typing is ignored, and you are prompted for a new command.
- If you press the key while SPIBILD is executing a command, you will interrupt the processing. SPIBILD may ask you if it can continue the processing, or it may simply stop altogether. Often in this case, if you press the key twice in succession, a session break will result. Type GO if you want the SPIBILD session to continue.
- If SPIBILD asks a question (such as "-Continue processing?"), pressing the ATTN/BREAK key is equivalent to answering NO.
- You can see how many read and write operations have taken place involving blocks of the files involved in a SPIBILD task by issuing the SET FILE COUNTS command prior to the main commands of the task in SPIBILD. SET FILE COUNTS is quite similar in effect to the SHOW FILE COUNTS command in SPIRES, though it does have some significant differences. See Part 6 for more information, or online, [EXPLAIN SET FILE COUNTS COMMAND.]
You can use SPIBILD through the batch system as well as interactively. That allows you to process SPIRES files, merge records into a subfile, or execute any other SPIBILD task by submitting jobs to the batch system.
There are several reasons why you might want to use SPIBILD in batch:
- 1) The input data you have for a SPIRES file is on tape or is too large to fit in your active file. [If the data is in standard SPIRES format, you can also use the BATCH command in SPIRES. [See the SPIRES manual "Searching and Updating", section D.8.]]
- 2) You want the SPIBILD work to occur at a time when you won't be logged on (in the middle of the night, for instance), so you submit a batch job to run for you at that time.
- 3) You don't want to tie up your terminal while your SPIBILD work is being done.
You can use either of two different programs when you want to use SPIBILD in batch:
The next two sections will provide detail about using these two programs for SPIBILD work and will give you a clearer picture of which to use in particular situations. [See 2.1.3.1, 2.1.3.2.] In general, more people choose to use BATWYL because of its flexibility, but here are some guidelines for choosing between them:
- If you are using SPIBILD to merge some data into a SPIRES file from a tape, you must use batch SPIBILD.
- If the input file is too large to fit into your WYLBUR active file or is in a format that cannot be imported into your active file, you must use batch SPIBILD.
- If you need return codes from the job in the event of SPIBILD problems, you should use batch SPIBILD.
- If you need to use WYLBUR Exec files, or to leave SPIBILD during the job, you should use BATWYL.
A BATWYL job is basically an interactive session controlled from the batch processor. Unlike some batch programs where writing the appropriate JCL can be very tedious and difficult, BATWYL uses very simple JCL. The challenge in constructing BATWYL jobs is to anticipate every prompt that BATWYL will send to the "virtual terminal" and include an appropriate response in the command stream.
Complete details on using BATWYL can be found in the document "Using BATWYL", available through the PUBLISH command.
Use the following JCL to run BATWYL jobs:
// JOB // EXEC BATWYL[,PARM='ATTN=*'] //SYSIN DD * ... interactive commands ...
The statements will be formally described below.
First though, here's an example of a simple BATWYL job that does some SPIBILD work (the numbers on the left are not part of the input but are used for reference in the explanation that follows):
1. // JOB
2. // EXEC BATWYL,PARM='ATTN=*'
3. //SYSIN DD *
4. call spires
5. select manual
6. for adds
7. in active display all
8. print forms=3hol duplex
9. in active show subfile transactions
10. ok
11. print forms=3hol duplex
12. exit
13. call spibild
14. process gq.doc.manual
15. logoff
In the above example, following the JCL (lines 1-3), the job calls SPIRES (4), places all the newly added records in the active file (6,7), and then prints them (8). Next, it puts the transactions display into the active file for printing (9-11). The "ok" (10) is not a command, but is the response to the SPIRES prompt "OK to clear?" that would result from the preceding command. (More likely, you would add the CLEAR option to the IN ACTIVE prefix on the SHOW SUBFILE TRANSACTIONS command, to avoid the prompt altogether.)
Next the job calls SPIBILD, processes the file, and logs off, ending the job (13-15).
Here are descriptions of the JCL statements you use to run BATWYL:
// JOB
You must begin all jobs with the JOB statement. It may be as simple as shown, or may include a job name, account number, bin number, etc. [See the document "Batch Processing" for more information.] Be sure to type blanks as shown in JCL, such as at least one before the word JOB.
// EXEC BATWYL,PARM='ATTN=*'
The EXEC statement calls the BATWYL program. The PARM option shown here allows you to create an ATTN/BREAK response by typing three asterisks at the end of a command line. The character doesn't have to be an asterisk; whatever character you choose should be specified in the ATTN clause instead. If you will not need to simulate an ATTN/BREAK, you do not need the PARM option (nor the comma preceding it). Other options are also available.
//SYSIN DD *[,DCB=LRECL=240]
The SYSIN DD statement shown above indicates that the commands to be executed by WYLBUR follow this statement. The document "Using BATWYL" describes how you can tell the program that the commands are stored elsewhere. If you add the DCB option shown, you raise the allowed length of commands from 80 to 240 characters. Note that SPIBILD commands may not exceed 160 characters, but WYLBUR and Milten commands can be as long as 235 characters.
... Interactive commands ...
Here you list the commands you want executed. If commands exceed 80 characters in length, be sure to add the DCB option to the SYSIN DD statement shown above. Note that SPIBILD commands absolutely may not exceed 160 characters.
All interactive commands are available, with the exception of those that invoke full-screen processes, such as Prism or full-screen SPIRES applications.
To use SPIBILD, you must include a SPIBILD command. After that, all SPIBILD commands are available to you. Note, however, that command options limited to batch SPIBILD, such as the "ON ddname" option on the BATCH command, are unavailable, because you are using interactive SPIBILD through the batch system, not batch SPIBILD.
Finally, be aware that output lines longer than 133 characters will be truncated.
Batch SPIBILD gives you direct access to the SPIBILD program through the batch system. You can, for instance, run jobs that process other user's files if you can do so yourself online. In addition, you may use your WYLBUR active file for input and output, and may issue WYLBUR commands such as DELETE or PRINT. However, there are some differences from online SPIBILD, and those restrictions are described in appropriate places throughout this manual and are summarized in this section.
Use the following JCL to run batch SPIBILD jobs. (Statements marked with an asterisk are optional, and are explained below.)
1. // JOB 2. // EXEC SPIBILD *3. //DDNAME DD DSN='WYL.gg.uuu.dsname',DISP=SHR, *4. // VOL=SER=volume,UNIT=unit *5. //SYSIN DD * 6. ... spibild commands ... 7. LOGOFF
For general information about batch jobs and JCL, see the "Batch Processing" manual, available through the PUBLISH command. Detail information about each of these specific statements appears below.
First, though, here is a sample job that uses batch SPIBILD:
// JOB // EXEC SPIBILD //DDNAME DD DSN='WYL.GQ.DOC.MANUAL.INPUT',DISP=SHR //SYSIN DD * batch 'manual' on ddname logoff clear
This job batches records in the data set named in the DDNAME DD statement into the subfile MANUAL.
Here are descriptions of the JCL statements you use to run batch SPIBILD:
1. // JOB
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number, output bin, region size, maximum number of output lines allowed (5000, by default), etc. See "Batch Processing" for more information.
Note that you never need to change the default region size, no matter how much SPIBILD work you are doing -- the region size of the job controls a small program that invokes interactive SPIBILD, which always has more than enough region for any SPIBILD task.
2. // EXEC SPIBILD[,PARM='parm']
The EXEC statement calls the batch SPIBILD program. The PARM option is occasionally used to set special escape characters, or to request versions of SPIBILD other than the public one, such as internal test versions, if they are available to you. The parameters most commonly specified are:
ESC=x OUT=y ACCOUNT=gg.uuu LIBRARY=[ORV.gg.uuu.]libname
If used, ESC and OUT need to appear in the PARM option before the other two.
ESC and OUT have the same effect in Batch SPIBILD as they do in the BATWYL program. Please see the document "Using BATWYL", Data Center document number 409, for more information.
*3. //DDNAME DD DSN='WYL.gg.uuu.dsname',DISP=SHR, *4. // VOL=SER=volume,UNIT=unit
Use the optional DDNAME statement when you want SPIBILD to use input data from a data set stored on tape or on disk. Generally speaking, it is easier for you to use the USE and GET commands in batch SPIBILD to bring data into your active file from stored data sets, but you must use the DDNAME statement if either of the following is true:
- the input data is on tape;
- the input data is not in a format that can be brought into your WYLBUR active file.
The data set may have a RECFM value of F or FB or U, with a fixed LRECL of up to 4090 bytes.
If the data set is cataloged, you can eliminate the second line of the DD statement shown above, as well as the comma at the end of the first line. Otherwise, specify the tape or disk volume number, and the unit TAPE or DISK. Be sure to include a /*SETUP card if you're using tapes.
*5. //SYSIN DD *[,DCB=LRECL=240]
The SYSIN statement shown above indicates that the commands to be executed by SPIBILD follow this statement. Thus, it should be the last JCL statement before the list of commands.
If you want, you can store the commands separately and then tell batch SPIBILD where to find them for execution. You do this by replacing the * in the SYSIN statement with the same type of values that follow the DD in the DDNAME statement above: the name of the data set (DSN) containing the commands, the disposition (DISP), and the VOL and UNIT information if the data set is not cataloged. The data set must be saved either in EDIT format, or at any fixed length up to LRECL=240.
If you need to issue commands longer than 80 characters (but no longer than 240) in your batch SPIBILD job, add the DCB option shown. Be aware that SPIBILD commands are limited to 160 characters, and WYLBUR and Milten commands are limited to 235 characters. (Remember too that you can abbreviate many command verbs and options, which might shrink long commands to fewer than 80 characters.)
6. SPIBILD commands
Here you list the commands you want executed, each command consisting of one and only one line of input. (See the previous paragraph about commands longer than 80 characters.)
All SPIBILD commands are available and work as you would expect, with the following exceptions:
If a SPIBILD command would prompt you for input if you were in interactive SPIBILD, then batch SPIBILD will assume a response of ATTN (the ATTN/BREAK key).
7. LOGOFF [CLEAR]
If you omit the LOGOFF statement, batch SPIBILD will supply it for you, and give you a return code of 4.
Though command lines may be as long as 150 characters, output from batch SPIBILD cannot exceed 133 characters per line. Lines longer than 133 characters will be truncated on output.
There are other lines of JCL you can add if you want, such as JOBPARM and SYSPRINT. See the "Batch Processing" manual for more information about them.
The following condition codes are supplied by SPIBILD if a job terminates abnormally:
Condition code Meaning
4 - LOGOFF CLEAR was supplied by the
program. The input stream ran out
without the proper LOGOFF command.
20 - Virtual terminal communications
problem.
24 - SUZAN path to Milten disconnected,
probably because ORVYL died.
28 - SUZAN path to SPIBILD disconnected.
32 - No SYSIN or COMMAND input data.
Normally when you use SPIBILD or SPIRES for a file-processing task, the task is completed without problems or complications. However, problems that can cause a file-processing command to go awry do occasionally arise. For example, while you are waiting for a file-processing command to execute:
- the system might fail;
- you might accidentally disconnect your terminal, causing the system to log you off;
- SPIBILD or SPIRES could encounter a serious problem with your file, causing it to stop executing the command immediately.
The results of these problems vary extensively, depending on the command being executed, on the precise point when the processing stopped, etc. In most cases, the file has suffered no ill effects at all; in other cases, there may be data integrity problems that SPIBILD or SPIRES needs to repair.
For example, suppose the system fails while SPIBILD is updating a subfile's indexes with some data from newly added records. At the time of the failure, half the indexes have been updated, but not the other half. As a result, the indexes are not reliable, and they won't be until the problem is repaired.
Occasionally your own programming error can cause problems too. For instance, changing a file definition carelessly can sometimes cause serious data integrity problems the next time the file is processed, requiring you to "back up" to a previous version of the file, stored prior to the change, to restore your file to a usable condition.
The next section describes the most common problems encountered during SPIBILD or SPIRES file-processing, with suggested methods for handling them. [See 2.1.4.1.]
Invariably, a result of interrupted file-processing commands is that the NO JOBGEN flag is set, blocking all file transactions (such as adds and removes) till it is reset.
In almost all situations where file-processing is interrupted, the file is fine; if you return to SPIBILD or SPIRES and issue the PROCESS command, the problem will be cleared up, with the JOBGEN flag reset so that file updates can continue. If you do not want to spend the time or money on a complete processing of the file, you can issue the RECOVER command instead, which does a "mini-process" of the file: just enough work to clear up any minor problems and reset the JOBGEN flag. The complete RECOVER/PROCESS command procedure for file recovery is described in the next section. [See 2.1.4.2.]
Another technique, used very infrequently, is available to files whose file-processing is automatically handled by JOBGEN. [See 2.2.1.] The SET JOBGEN command is at the heart of that procedure. It is used less frequently because all it does is reset the JOBGEN flag, so that file updates are allowed again. That allows the file to be processed again via JOBGEN, starting that night.
Unfortunately, the SET JOBGEN command can create serious data integrity problems, and should not be used unless there will be no file updates between the time you set the JOBGEN flag and the time the JOBGEN-created batch job processes the file. See Part 6 for more information; online, [EXPLAIN SET JOBGEN COMMAND.]
In extremely rare cases, such severe problems can occur that the file is considered destroyed. When a problem like that occurs, it is more likely due to system problems (such as a disk-drive failure) than to a SPIBILD problem. But in any case, backup copies of the file or parts of the file are a vital part of SPIRES applications.
Some basic backup services are provided by your computer center, which are always adequate if JOBGEN handles your file-processing needs. However, you may choose to handle a file's backup yourself. A discussion of the available services appears later in this section. [See 2.1.4.3.]
Though most file-processing tasks in SPIBILD and SPIRES run quite smoothly, this section discusses the most frequently encountered problems (some described by their system error numbers), with suggested solutions. Remember that the EXPLAIN command can be used in SPIBILD as well as SPIRES to get information about system messages and error codes.
This problem could be caused by a system crash or by your terminal becoming disconnected. Note that although the file will be in a somewhat unstable state, users can still use the file for searching, though probably not for updating, until you can execute one of the two recovery procedures to stabilize it completely. In any case, the file is not damaged, and the recovery procedures will return it to normal.
To recover the file, follow the RECOVER/PROCESS procedure. [See 2.1.4.2.]
You may have misspelled the name of the file. If that's not the case, return to SPIRES, select a subfile of the file, and check the value of the $FILENAME system variable (SHOW EVAL $FILENAME) to be sure you were trying to process the correct file.
Another possibility is that you are trying to use a file that hasn't been compiled yet. That sometimes happens when you are rebuilding a file: you issue the ZAP FILE command to destroy the old file and then go to SPIBILD to rebuild it, without first compiling it again. The file doesn't exist till it is compiled.
This error indicates that other people have subfiles of the file selected. You may want to use the SET SEARCH/SET SHARE procedure to circumvent this problem. [See 2.1.5.1.] Note: SPIRES automatically "sets SEARCH" when you issue the PROCESS command in SPIRES, so S56 errors are not a problem there.
To process someone else's file, you must have WRITE access to all the ORVYL data sets of the file except the MSTR, as well as WRITE access to the SYSTEM.CHKPT data set on the file owner's account. If you don't have the proper access, the command will fail with an S116 error. The file owner can correct the problem by issuing SET PERMIT commands for each of the data sets.
This list is not comprehensive -- other problems can stop file-processing too. In general, the error messages, combined with information available through the EXPLAIN command, should lead you to the problem and suggest a solution. Of course, file-processing problems should be taken seriously, so if you need any help, be sure to see your SPIRES consultant as soon as possible.
Procedure: RECOVER/PROCESS
Purpose: General recovery from SPIBILD problems
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Issue the PROCESS or RECOVER command
3. Fix the problem, if step 2 fails
When a SPIBILD command finishes abnormally, it may leave the file in an unstable state. It will definitely leave the NO JOBGEN flag set, meaning that automatic processing of the file by JOBGEN will not occur, and no file updates are allowed.
The RECOVER/PROCESS procedure is a general one for solving such problems; special factors for particular SPIBILD tasks that don't get completed are discussed under individual procedures.
If you aren't already in SPIBILD, issue the SPIBILD command:
-> spibild -Welcome to SPIBILD -?
Note: If you use the PROCESS command (see step 2), you can do this procedure in SPIRES; however, RECOVER and SPIBILD commands that process the file other than PROCESS are available only in SPIBILD.
in SPIBILD, depending on your preference (see below), issue either the RECOVER or PROCESS command:
RECOVER [ORV.gg.uuu.]filename PROCESS [ORV.gg.uuu.]filename
Note that the PROCESS command in this context means any SPIBILD command that processes the file, such as the BATCH, MERGE and ESTABLISH commands -- they will all have the same effect in terms of recovery.
Also note that if you are processing the file in SPIRES, you follow a slightly different procedure here at step 2: first, SELECT a subfile of the file and then issue the PROCESS command in this form:
PROCESS [NOWARN]
NOWARN tells SPIRES not to prompt you for confirmation that you want to process the file.
It may seem odd to try recovering from the problem before determining just what the problem is, but in fact, you can easily waste a lot of time trying to understand the problem, only to discover that you can solve it simply by issuing either the RECOVER or PROCESS command. Since issuing either of these commands cannot make the situation worse, but stands an excellent chance of fixing it, giving that a try is usually worthwhile.
Here are some notes to help you choose between PROCESS and RECOVER:
The PROCESS command:
- tries to completely process the file. If it cleans up the problem, then the deferred queue will be empty when the command is completed.
- may execute for a long time if the deferred queue is large.
- can be done in SPIRES, e.g., under protocol control.
The RECOVER command:
- does only a limited amount of work. (Technically speaking, it processes two pass stacks' worth of data.) For most files, where the deferred queue is not large, that will empty the deferred queue, making RECOVER equivalent to PROCESS. But for files with large deferred queues, the RECOVER command will leave records in the deferred queue, waiting for the next processing of the file.
- will not execute for very long. Thus, it may be preferable to PROCESS in situations when you need to restore the JOBGEN flag as quickly as possible so that users can resume updating records in the file.
- is available only in SPIBILD.
The sample session below shows the successful execution of the RECOVER command:
-> spibild
-Welcome to SPIBILD
-? recover bionotes
-Begin passing of 32 records of record-type REC01
-Completed passing of 32 records total
-Begin passing of 30 records of record-type REC01
-Completed passing of 62 records total
-Processed: 62 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.BIONOTES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.BIONOTES Reads Writes
CKPT data set 4 1
DEFQ data set 6 120
MSTR data set 5 0
Residual data set 7 65
Rec-type REC01 2 58
Rec-type ZIN02 1 20
Total for file: 25 264
-Compute time: 0.258 seconds
-Elapsed time: 2.310 seconds
-Core usage: 0210/0939
-?
If no errors are indicated, then the recovery was successful, users can begin using your file again, and the procedure is complete.
Note: If SPIBILD would create more than two pass stacks' worth of data, then it quits the RECOVER command without processing the file, and exits SPIBILD. Hence, in the example above, there was only enough data in the DEFQ for two pass stacks; had there been more, the message "Processed: ..." and those that follow it would have been replaced by "Exiting SPIBILD". Though the file wouldn't have been completely processed, it would be back to a useable state again.
However, if errors are indicated, as in the example below, you will need to continue to step 3:
-? recover orv.lo.ony.tunes -Action E170 error -While processing record, key = 2 -Failure when goal record-type = REC01 -Exiting SPIBILD Command>
Notice that failure invariably invokes an automatic exit from SPIBILD.
You can find out what went wrong by examining the display from the SPIBILD session, in particular, by examining the error messages and using the EXPLAIN command to find out details about specific errors. The error explanations should help you determine whether you can fix the problems or whether you should contact your SPIRES consultant. In any event, more work will be required before the file can be processed properly.
Everyone knows that having backup copies of computer files is a good idea, but the more reliable computer systems become, the less important it may seem. In fact, few of the thousands of SPIRES files created at Stanford have ever needed restoring from the system-created backup tapes due to system problems. But that fact begs the question -- history shows a number of occasions when file restoration has been necessary. In many cases, though, the problem was caused by a file owner accidentally issuing a destructive command (such as ZAP FILE) or making an unfortunate programming decision that destroys the data integrity of the file.
File backups are thus a crucial ingredient of SPIRES applications. File owners should be aware of what file backup is automatically done, what they themselves can do in addition, and what to do if they need to use the backup data sets to restore a SPIRES file.
Backup of SPIRES files is available in four forms:
If you think you need to restore a SPIRES file using the backup files created by either of those two methods, contact your SPIRES consultant for assistance.
Procedure: ORVCOPY backup Purpose: Make backup copies of data sets of a SPIRES file Who can execute it: file owner Basic steps: 1. Create the ORVCOPY JCL. 2. Run the job.
The batch program ORVCOPY, the ORVYL file dump/restore utility program, lets you back up the data sets of a SPIRES file to tape or disk and later restore them to the ORVYL file system. Besides the data itself, attributes of the file such as account-access permits are copied too.
This procedure describes the technique a file owner would use to make a backup copy of a SPIRES file stored on his or her account, storing it on tape. The reverse, restoring the copy back to the ORVYL file system, is covered briefly in the next section. [The document "ORVYL User's Guide" contains complete information about ORVCOPY, including information about storing the data on disk and dealing with other accounts.]
Note though that an ORVCOPY job can copy only data sets belonging to the account under which the job is run. Similarly, it can restore data sets only to the account under which it is run. But the dump job and the restore job can be run on separate accounts, providing a way in which data sets can be moved from one account to another. If you are moving data between accounts with ORVCOPY, be sure that all accounts have access to the tape or disk data sets to which the data is dumped.
Warning: if you use this procedure to move a SPIRES file from one account to another, the MSTR data set of the file, which contains the name of the file (including the owner's account number), must be changed as well. This change must be done by hand, as described at the end of the next section. [See 2.1.4.5.]
Use the following JCL for an ORVCOPY job to copy data sets:
1. // JOB [TIME=n] 2. /*SETUP T OUTPUT=tapeid 3. // EXEC ORVCOPY 4. //OUTPUT DD DSN=WYL.gg.uuu.name,DISP=(NEW,CATLG), 5. // UNIT=TAPE[,VOL=SER=volume] 6. dump commands
For general information about batch jobs and JCL, see the "Batch Processing" manual, available from Document Sales. Specific information about each statement in the ORVCOPY JCL appears below, and a sample job follow that.
Below are descriptions of the JCL statements you'd use to run an ORVCOPY job to copy ORVYL data sets to a new rental tape or empty tape belonging to you:
1. // JOB [TIME=n]
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number or output bin. See "Batch Processing" for more information.
Depending on how many blocks you are copying, you may need to add the TIME parameter to request more than the default 30 CPU seconds. As a rule of thumb, figure about 450 blocks can be copied per CPU second.
2. /*SETUP T OUTPUT=tapeid
The SETUP statement tells the system that the job requires tapes to be mounted. There are many different values for "tapeid", but the most common are:
3. // EXEC ORVCOPY
The EXEC statement calls the ORVCOPY program.
4. //OUTPUT DD DSN=WYL.gg.uuu.name,DISP=(NEW,CATLG), 5. // UNIT=TAPE[,VOL=SER=volume]
The OUTPUT DD statement tells the program the destination for the copy of the ORVYL data sets. Be sure the name you type for the DSN parameter is unique to your account; it must follow the naming conventions for OS data sets. (If you can't give it a unique name, replace CATLG with KEEP in the DISP parameter.) If you use your own tape rather than a scratch tape, include the VOL=SER parameter with the volser number, as given in the SETUP statement earlier.
6. dump commands
The DUMP command tells ORVCOPY what data sets to copy. You can include as many DUMP commands as you need. A single command can copy from one to all of your ORVYL data sets, depending on the option specified:
DUMP option
where "option" is one of the following:
NAME=filename[,RENAME=name] - copy the single named ORVYL data set,
giving the copy a new name if the
RENAME option is specified
LIKE=string - copy all data sets with names beginning
with the given string
RANGE=name1,name2 - copy data sets "name1" and "name2", and
all those whose names fall between them
in alphanumeric order; "name1" must come
before "name2" in alphanumeric order.
ALL - copy all data sets belonging to your account
Do not include the "ORV.gg.uuu." file prefix in any of these options. It is not necessary since you can only copy files on the account under which the job is run.
Be sure to put a blank between the DUMP command and the option you use; however, no other blanks should appear in the DUMP command.
Here is a sample job run by account GQ.DOC that uses ORVCOPY to copy the data sets of the file GQ.DOC.ERRATA:
// JOB /* SETUP T OUTPUT=SCRTCH // EXEC ORVCOPY //OUTPUT DD DSN=WYL.GQ.DOC.ERRATA.BACKUP,DISP=(NEW,KEEP), // UNIT=TAPE DUMP LIKE=ERRATA
When you want, issue the RUN command to submit the job. After it runs, you should examine the output to get the assigned tape number if you asked for a rental tape. You will need it later if you need to restore the files. [See 2.1.4.5.]
Again, see the document "Batch Processing" for information on running jobs, fetching the output, releasing tapes, etc.
Procedure: ORVCOPY restore Purpose: Restore a SPIRES file using backup data sets Who can execute it: file owner Basic steps: 1. Create the ORVCOPY JCL. 2. Run the job.
In the previous section, the procedure for creating a copy of the ORVYL data sets comprising a SPIRES file using ORVCOPY was discussed. In this section, its less frequently used complement will be covered.
If you need to restore a SPIRES file to a version you created using ORVCOPY, you also use ORVCOPY to get it back. Besides the data itself, attributes of each data set such as account-access permits are reset when ORVCOPY restores a file.
This procedure describes the technique a file owner would use to restore a SPIRES file stored on his or her account, using a copy stored on tape. The reverse, creating the copy, was discussed in the previous section. See the document "ORVYL User's Guide" for full information about ORVCOPY.
Use the following JCL for an ORVCOPY job to restore data sets:
1. // JOB [TIME=n] 2. /*SETUP T INPUT=CAT 3. // EXEC ORVCOPY 4. //INPUT DD DSN=WYL.gg.uuu.name,DISP=OLD 5. restore commands
For general information about batch jobs and JCL, see the "Batch Processing" manual, available from Document Sales. Specific information about each of the statements in the ORVCOPY JCL appears below, and some sample jobs follow that.
Below are descriptions of the JCL statements you'd use to run an ORVCOPY job to restore ORVYL data sets from a tape.
1. // JOB [TIME=n]
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number or output bin. See "Batch Processing" for more information.
Depending on how many blocks you are restoring, you may need to add the TIME parameter to request more than the default 30 CPU seconds. As a rule of thumb, figure about 450 blocks can be restored per CPU second.
2. /*SETUP T INPUT=CAT
The SETUP statement tells the system that the job requires tapes to be mounted. Specify the CAT value if the data set named in line 4 below is cataloged -- the system will find the tape volume serial number in the catalog. If the data set is not cataloged (e.g., it wasn't uniquely named), replace CAT with the volser number of the tape, which you will also need to add to line 4 (see below).
3. // EXEC ORVCOPY
The EXEC statement calls the ORVCOPY program.
4. //INPUT DD DSN=WYL.gg.uuu.name,DISP=OLD
The INPUT DD statement tells the program the source for the copy of the ORVYL data sets. If the data set is not cataloged, include the UNIT=TAPE parameter and the VOL=SER parameter with the volser number.
5. restore commands
The RESTORE command tells ORVCOPY what data sets to restore. You can include as many RESTORE commands as you need. A single command can restore from one to all of the data sets in the INPUT DD data set, depending on the option specified:
RESTORE option [REPLACE]
where "option" is one of the following:
NAME=filename[,RENAME=name] - restore the single named ORVYL data
set, giving the copy a new name if
the RENAME option is included
LIKE=string - restore all data sets with names
beginning with the given string
RANGE=name1,name2 - restore data sets "name1" and "name2", and
all those whose names fall between them
in alphanumeric order; "name1" must come
before "name2" in alphanumeric order.
ALL - restore all the data sets on the tape
Don't include the "ORV.gg.uuu." file prefix in any of these options; also, don't put blanks within the option (for instance, between LIKE and the equals sign).
Include the REPLACE option if you want ORVCOPY to replace existing data sets with the same names as those being restored. If you leave it off, ORVCOPY will only restore files that don't currently exist in ORVYL.
Below is a sample job run by account GQ.DOC that uses ORVCOPY to restore the data sets of the file GQ.DOC.ERRATA. It is the counterpart to the example in the previous section.
// JOB /* SETUP T INPUT=CAT // EXEC ORVCOPY //INPUT DD DSN=WYL.GQ.DOC.ERRATA.BACKUP,DISP=OLD RESTORE LIKE=ERRATA
When you want, issue the RUN command to submit the job. After it runs, check the file-creation date of each data set by adding the DATE option to the SHOW FILES command. See Part 6, or online, [EXPLAIN SHOW FILES COMMAND.] Those that were restored will have the date they were restored listed for "CR", the creation date.
See the document "Batch Processing" for information on running jobs, fetching the output, releasing tapes, etc.
As mentioned before, this procedure can be used to move data sets from one account to another, which means you could use it to move an entire SPIRES file from one account to another. Generally s