******************************************************************
* *
* Stanford Data Center *
* Stanford University *
* Stanford, Ca. 94305 *
* *
* (c)Copyright 1994 by the Board of Trustees of the *
* Leland Stanford Junior University *
* All rights reserved *
* Printed in the United States of America *
* *
******************************************************************
SPIRES (TM) is a trademark of Stanford University.
This manual was written for people who own SPIRES files or who are responsible for the maintenance of SPIRES files. If you are one of these people, you should already be familiar with many aspects of SPIRES. We consider the material presented in the SPIRES primer "A Guide to Data Base Development" as the pre-requisite for this manual. For instance, if you do not know the terms "subfile", "deferred queue", "Global FOR", "File Definer" and "file definition", you should begin with that manual, which introduces some very basic concepts of File Management that we will expound in this manual.
Though the Table of Contents may not show it, this document is divided into three major portions:
In SPIRES manuals, examples of sessions are shown with prompts and messages from SPIRES as they actually appear on the terminal screen; SPIRES prompts usually end with a question mark or greater-than sign (">"), while SPIRES messages usually begin with a hyphen. The commands and responses you type are shown in lowercase, following a prompt. (You may generally use upper or lower case interchangeably when you actually use SPIRES.) In some editions of this manual, what you type will be shown in bold, while the computer's responses are shown in normal type. For example:
-OK to clear? ok
Here SPIRES types "-OK to clear?" and you type "ok".
In formal command syntax descriptions, uppercase letters denote command verbs or other command elements to be entered exactly as shown; a value for lowercase terms and characters must be supplied by you. For example:
SELECT subfile-name
To use this particular command, you type the command verb SELECT with the name of the desired subfile, for instance, "Restaurant":
-> select restaurant
where "->" is the prompt from SPIRES.
Brackets ([ ]) denote optional elements. Braces ({ }) indicate that you must specify one and only one of the alternatives within the braces. Within the braces or brackets, a vertical line (|) separates possible choices. Neither brackets nor braces are to be typed as part of the command. For example:
SHOW SPIRES MAIL [CLEAR|NOCLEAR]
could be entered as
SHOW SPIRES MAIL or SHOW SPIRES MAIL CLEAR or SHOW SPIRES MAIL NOCLEAR
depending on whether you want the option. Here is an example from the report writer command language using braces:
AT {START|TOP} OF PAGE ...
You must choose one of the words in the braces when using this command. For example, this command could be entered as
AT START OF PAGE ...
or as
AT TOP OF PAGE ...
Sections and subsections of this manual whose titles are preceded by an asterisk in parentheses -- (*) -- may be considered optional reading at that point. Usually the material provides details that most users will not need about how to handle uncommon situations, or technical information about how SPIRES internally handles some particular command.
If you are familiar with other SPIRES manuals, you will notice many substantial differences in the areas of structure and style between this manual and those. These changes result from a study to evaluate SPIRES documentation in which different contingents of SPIRES users were interviewed. Though we'd like to reprint here all the complimentary comments we received, it's perhaps more useful to discuss some of the criticisms and suggestions, and mention how this manual addresses them.
Most of the users we interviewed suggested that SPIRES documentation would be more useful if it were more "task-oriented". Most SPIRES users come to documentation trying to accomplish a specific task, and may not know or recall the names of the tools used to do it. To address this concern, we have organized most of this manual, the "User Guide" portion, by tasks. In other words, to find out how to perform a certain file management task, you would probably look up the task in the table of contents or index. All the basic information about solving the task will appear together in one chapter or often in one section of a chapter.
Another request, related to the previous one, was to provide a standardized presentation of each command with full reference material. You will find that in Part 6, with all the commands presented in alphabetic order. Hence, if you already know the name of the command you're interested in, you can quickly find the reference material on it in Part 6. Online, you will discover, if you EXPLAIN any of the commands, that the section of documentation you see comes from that part of the manual.
In addition, depending on the source of your copy of this manual, you may notice that this manual has been printed with proportional spacing, guide words in the margins, and other special typographical improvements.
Besides the new style and structure, there is plenty of new material. Though much of the information in this manual has been seen before in the reference manuals "File Definition", "Formats" and "Technical Notes", much of it is new too. We think you will appreciate having the material together in a single manual, despite any duplications between manuals.
If you think of any aspects of SPIRES file management that weren't covered in this manual and which you'd like to see addressed in a future edition, please send them to your SPIRES consultant, or send them directly to SPIRES Documentation, Stanford Data Center, Stanford University, Stanford, California 94305.
This manual is the creation of many contributors, including the earliest designers of SPIRES back in the late 1960's and recent users who suggested improvements to the explanation of this or that feature in SPIBILD. In general tribute to all, we here thank many specific individuals for their significant help.
SPIRES is developed and maintained by the SPIRALS team of Stanford University's Stanford Data Center. Programming responsibility for SPIRES is shared by Dick Guertin and Bill Kiefer, both of whom contributed enormously to the development of this manual. Chuck Boeheim, responsible for the VM/CMS implementation of SPIRES, was very helpful in that area. Sandy Laws, Peter Tuttle, Lynn McRae, Norman Roth and Becky Morton also provided important technical support and review.
Others outside of SPIRALS provided concrete suggestions about the manual's contents, including Ed Begley of I.T.S.'s Community Information Resources and Hannah Kaufmann and Michael Sperberg-McQueen of Princeton University's Center for Information Technology.
The style of this manual is significantly different from SPIRES manuals written in the past, as is its physical design. Major contributors in this area include DRG staff members Lynne Sinclair, Jeff Rensch and Tony Navarrete, as well as Terry Oldano, Director's Office, and Anne Janzer, Systems. A new GML format for this document was patiently and painstakingly created by Greg Bruce.
For printing support, we acknowledge the assistance of staff members Lloyd Warble and Mark Lawrence of Electronic Printing and Lincoln Ong of Operations. Charley Hoyt of Systems deserves special commendation for his customization work on Waterloo Script's Photo processor especially for this document.
The manual was written by John Klemm, who thanks those named above and everyone else who provided ideas, patience, and non-addictive stimuli when they were most needed.
Managing a SPIRES file consists of tasks in three broad categories:
- record-transaction tasks, consisting of adding, updating or removing "batches" of records from the file; [You might consider that adding, updating and removing records individually in SPIRES are file management tasks, albeit on a smaller scale. But in general, the term "file management tasks" refers to work involving groups of records rather than individual ones.]
- routine maintenance tasks, ensuring that the records are being stored efficiently, that the indexes are being updated, that the file has a backup copy in storage, etc.
- repair tasks, perhaps involving the destruction or reconstruction of an index, an entire subfile or an entire file, perhaps done in response to information from a maintenance task.
The same file management capabilities are available to all SPIRES files, regardless of size. But many of them are not usually needed by small files; the test to determine whether SPIRES is storing some data inefficiently could cost more than the inefficiency itself does. But for large files (say, over 100,000 records), such tests could be very cost-effective in the long run, which is how long such files generally last.
The other parts of this manual cover the file management tasks in detail, principally for the benefit of owners of large files. Owners of small files typically do not have the need, interest or incentive to learn so much about SPIRES, and certainly shouldn't have to do so.
In view of that, the rest of this part of the manual introduces the most important aspects of file management for all SPIRES files. If you own a small file, you will probably find enough information here to meet your needs; if not, suggestions for further reading in the rest of the document are made along the way. But even if you own or manage larger files, you may find this introduction useful too.
Many of the terms that show up in discussions of SPIRES file management are informally defined in a glossary at the end of this manual. The index in the back includes a complete list of the file management tasks addressed by this manual, under the heading "file management task", with page number references to the appropriate sections, of course. Online, you can see the list by issuing the command EXPLAIN FILE MANAGEMENT TASK, which also gives you online access to all those sections of the manual too.
For small SPIRES files, file management may be quite simple:
- records are updated one at a time;
- the JOBGEN program causes the file to be processed every night so that the tree is updated using the information in the deferred queue, and the indexes are updated;
- the file is destroyed with the ZAP FILE command when it is no longer needed.
In a sense, such files have little use for "file-management procedures"; SPIRES takes care of the most important task (automatic processing of the file) without any intervention from you.
JOBGEN is a program, running after midnight each night, that examines the deferred queue of each SPIRES file requesting this service. If the deferred queue is empty, JOBGEN does nothing more to that file. But if it is not empty, JOBGEN creates a job to process the file in the SPIBILD program ("SPI" stands for SPIRES; "BILD" stands for "file-building". SPIBILD's tasks center around file-building.); the job usually runs a few minutes later. The file owner's account is charged for the cost of the job. [See 2.2.1.]
JOBGEN runs during the night because computing rates are cheaper then, and because the files are less likely to be in use at that time. (In general, SPIBILD jobs need exclusive access to a file during processing.)
The AUTOGEN flag, which is most often controlled by the file owner, determines whether JOBGEN is responsible for processing the file. The flag is set in one of three ways:
- 1) When the file is defined, the file owner determines a default setting for the AUTOGEN flag. Specifically, if the file definition contains the NOAUTOGEN statement, then NOAUTOGEN is the basic (default) setting for the file, i.e., JOBGEN is not responsible for the file. (File Definer's NOAUTOGEN attribute of the FILE statement has the same effect.) Otherwise, the basic setting is AUTOGEN.
- 2) The file owner, or a user with Master access to the file [The file definition may give other users certain file-owner powers -- this is called "Master access". This, and other forms of file access, such as Process access, are discussed in the glossary at the end of the manual.] may issue the SET AUTOGEN or SET NOAUTOGEN command to change to the desired setting.
- 3) Every time the file is processed, by either JOBGEN or you, the file returns to its basic setting, according to the file definition (see step 1).
To turn off automatic processing for awhile, you can just issue the SET NOAUTOGEN command when you have the subfile selected in SPIRES:
-> select staff addresses -> show autogen -AUTOGEN in effect -> set noautogen ->
On the other hand, if you have NOAUTOGEN in effect but decide that it's time for the file to be processed again, you can issue the SET AUTOGEN command, and the next time JOBGEN finds data in the deferred queue, the file will be processed:
-> show autogen -NOAUTOGEN in effect -> set autogen ->
Remember that the file will revert to its default setting of AUTOGEN or NOAUTOGEN, according to the file definition, after the file is processed.
Occasionally you may not want to wait for overnight processing of a file by JOBGEN. You may need your indexes updated immediately, for one reason or another, so you need the file processed now. (This scenario assumes the indexes aren't "immediate indexes", which are updated each time a record transaction occurs.)
Processing a file is quite a simple task to do, but unless you have compelling reasons to process the file yourself, you should leave it to JOBGEN. [See 2.2.1.]
If you are the file owner, or if you have PROCESS access to the file, you can process a SPIRES file. Processing the file is most commonly done in the SPIBILD program. To process a SPIRES file, you first call SPIBILD and then issue the PROCESS command:
-> spibild
-Welcome to SPIBILD
-? process orv.gq.doc.addresses
-Begin passing of 4 records of record-type REC01
-Completed passing of 4 records total
-Processed: 4 DEFQ records of record-type REC01
-Detaching File: GQ.DOC.ADDRESSES
08/11/90 File Read/Write Counts Information
File: GQ.DOC.ADDRESSES Reads Writes
CKPT data set 4 1
DEFQ data set 7 4
MSTR data set 3 0
Residual data set 3 6
Total for file: 17 11
-Compute time: 0.246 seconds
-Elapsed time: 2.407 seconds
-Core usage: 0267/0939
-? exit
-Exiting SPIBILD
Command>
The commands SPIBILD and EXIT take you in and out of SPIBILD. The SPIBILD program is similar to SPIRES in that you can issue WYLBUR and MILTEN commands within its environment, though you cannot issue SPIRES commands.
The PROCESS command causes the named file to be processed, and the output from the command as the processing occurs usually looks like the example above. If you try it but have problems, try using the EXPLAIN command to find out more about any error messages you received -- the EXPLAIN command is available in SPIBILD.
You can also process the file in SPIRES itself, without having to go into the SPIBILD program. Processing the file in SPIRES has some limitations that may affect you, but in most cases, they will not be significant. The advantages of processing in SPIRES include the ability to put the PROCESS command into a protocol, and being able to direct the error messages into the "trace log" (SET TLOG). [See 2.2.2 for a comparison of file-processing in SPIBILD and SPIRES.]
Most of the time, SPIBILD processes your files without problems. However, sometimes power outages or system problems occur during a SPIBILD session, causing the file processing to stop in mid-stream.
You generally find out that your file has been affected by a SPIBILD problem in one of three ways:
- the computer tells you that a problem occurred as you are processing the file in SPIBILD;
- you get output from JOBGEN that tells you that the file-processing job it created did not run properly;
- users discover that they cannot accomplish any subfile transactions (adds, updates, removes) for the file because the transactions are blocked when a SPIBILD processing error has occurred.
This doesn't mean the file is damaged; it merely means that the file is unstable, and normal subfile transactions cannot resume till SPIBILD cleans up whatever problem it had.
In general, the best way to "recover the file" is to issue the RECOVER command in SPIBILD, naming the file to recover:
Command> spibild
-Welcome to SPIBILD
-? recover gq.doc.addresses
-Begin passing of 12 records of record-type REC01
-Completed passing of 12 records total
-Processed: 12 DEFQ records of record-type REC01
-Detaching File: GQ.DOC.ADDRESSES
08/11/90 File Read/Write Counts Information
File: GQ.DOC.ADDRESSES Reads Writes
CKPT data set 4 1
DEFQ data set 4 4
MSTR data set 2 0
Residual data set 3 12
Rec-type REC01 2 2
Rec-type ZIN02 2 5
Total for file: 17 24
-Compute time: 1.833 seconds
-Elapsed time: 5.900 seconds
-Core usage: 0210/0939
-?
The RECOVER command begins to process the file again, though if there is a large amount of deferred queue data, the RECOVER command may successfully complete without actually finishing the file processing. When the RECOVER command completes its work, the file should be back to normal.
On the other hand, if you receive error messages rather than the messages shown in the example above, your file still has problems that need to be resolved. [See 2.1.4 for a detailed discussion of SPIBILD problems you might encounter.]
In rare cases when a file's problems are so serious that its integrity is questionable, or when a disk-storage problem arises, your computer center may need to use backup copies of data sets to restore the file to its state at the point when the backup was made. If you believe you need to take advantage of this service, contact your SPIRES consultant for help.
One of the most common duties of the file manager is to "build the file" -- that is, add many records to the file at once to get it started. Another common task that arises occasionally is to add, update and/or remove many records of a subfile at one time.
There are quite a few ways to apply multiple record-transactions to a subfile at once. In general, the easiest and least expensive methods (to the user doing the work) use SPIRES, putting the record transactions into the deferred queue. Then, when the file is processed in SPIBILD, perhaps automatically overnight, the deferred queue data will be applied to the tree of the subfile.
On the other hand, if getting the data into the tree and having the indexes reflect the new data immediately is important to you, you can use SPIBILD techniques instead. Remember that they may cost more than the SPIRES techniques, particularly if the processing is not occurring overnight, when rates are likely to be cheaper. On the other hand, if all the charges are being borne by the file owner, then doing all the work at once in SPIBILD is likely to be cheaper than putting the data first into the deferred queue in SPIRES and later having SPIBILD move the data into the tree.
Descriptions and comparisons of all the different procedures you can use for multiple record-transactions appear later, in several chapters of Part 2. [See 2.3, 2.4, 2.5.] Here we will discuss the three simplest and most general procedures of SPIBILD, probably the most useful ones to owners of small files:
- the BATCH procedure in SPIBILD, which allows you to add new records and update, merge and remove records whose keys you know; and
- the MERGE procedure in Global FOR in SPIBILD, which lets you update multiple records at once using the same merge input for each one, choosing the records to be updated with non-key criteria.
- the REMOVE command in Global FOR in SPIBILD, which lets you remove multiple records at once using non-key criteria to determine the records to remove.
The INPUT BATCH command in SPIBILD provides a fast and reliable method of getting data into a SPIRES subfile and getting the indexes updated. It's available to any user who can process the file in SPIBILD.
SPIBILD examines the data for each added and updated record, rejecting it if it breaks any of the processing rules defined for the subfile. Each record that succeeds is placed in the tree of the file. For removes, SPIBILD removes that record from the tree. SPIBILD then updates any indexes defined for the subfile.
For this procedure, your data must be in the standard SPIRES format, with each record preceded by a command followed by a semicolon ("ADD;" or "UPDATE 226-82-3846;") and (except for REMOVE) followed by an additional semicolon. [See 2.3.5.]
Here's a brief example of some data in the standard SPIRES format:
ADD; ITEM = 24-ounce Klein bottles; UNIT = carton of 12; UNIT.PRICE = $10.50; STOCK = 12; ; MERGE 134; ITEM = Mobius film strips; UNIT = "a ""grab bag"" of 10 assorted subjects"; ; REMOVE 19;
There is one caveat about using this procedure: the ESTABLISH command used at the start processes the file, which processes all the transactions currently in the deferred queue. If you don't want that to happen for some reason, consider using the INPUT BATCH command in SPIRES.
Here are the steps of the INPUT BATCH procedure in SPIBILD:
If you aren't already in SPIBILD, issue the SPIBILD command:
Command> spibild -Welcome to SPIBILD -?
You must have the input data in your active file. If you don't already have it there, use the appropriate WYLBUR command to put it there. For example,
-? use wyl.gq.jnk.supply.input -?
You will get one line of informational data about each successful record from the INPUT BATCH command by default. If you are working with dozens of records, those messages can be tedious, and they add a small degree of inefficiency too. You can turn them off by issuing the command SET INFORMATION MESSAGES = 0; it does not suppress any error messages. [See 2.1.5.3.]
It is a good idea to tell SPIRES to put the data for transactions that fail into a special data set called an "exception file". By extracting the bad data from the good data, SPIRES makes it easier for you to find the problems in the data so that you can correct it and submit it again with a subsequent INPUT BATCH command.
The exception file is an ORVYL data set that you name yourself, in the SET EXCEPTION command. For example:
-? set exception bad.supplies.records -?
SPIRES will put any errant records into the ORVYL data set called BAD.SUPPLIES.RECORDS on your account. [See 2.1.5.2.]
Issue the ESTABLISH command to tell SPIBILD which subfile you want to use and to have SPIBILD process the file containing it:
ESTABLISH subfile-name
For example:
-? establish supplies -Processing file: GQ.JNK.SUPPLIES -Begin passing of 2 records of record-type REC01 -Completed passing of 2 records total -Processed: 2 DEFQ records of record-type REC01 -Compute time: 0.196 seconds -Elapsed time: 1.227 seconds -Core usage: 0210/0939 -?
Next issue the INPUT BATCH command to tell SPIBILD to read the input data from your active file and apply it to the subfile:
INPUT BATCH
Continuing the above example:
-? input batch
- ADD 1, 1 @Line 6. Key = 356
- UPD 1, 1 @Line 10. Key = 134
- REM 1, 1 @Line 11. Key = 19
... (etc.) ...
-Begin passing of 94 records of record-type REC01
-Completed passing of 94 records total
- Requests/Success:
ADD 72 72
UPD 16 16
REM 6 6
SUM 94 94
-End of BATCH/BUILD
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 9 7
MSTR data set 6 0
Residual data set 53 51
Rec-type REC01 2 14
Rec-type ZIN02 1 13
Total for file: 75 86
-Compute time: 4.688 seconds
-Elapsed time: 14.448 seconds
-Core usage: 0279/0939
-?
In this example, the information messages were not turned off (step 3); had they been, the first three lines shown would not have appeared, since they indicate successful transactions.
The most common problem is that one or more transactions fail due to input errors, such as an invalid element name, or an improper element value. SPIBILD will display error messages telling you what the problem is. Assuming that you have set an exception file, it is easy to retrieve the unsuccessful data, using the ORVYL command GET:
-? get bad.supplies.records OK to clear? ok -?
The data is now in your active file. After correcting it, you can return to step 3 and continue the procedure.
If the system fails during any step of the procedure except step 6, just restart from the beginning when it is convenient to. If it fails during the INPUT BATCH command, you need to determine the point where it failed, i.e., figure out which was the last record to be fully processed. A command called SHOW RECORD RECOVERY may be useful in this regard. For full details, see the complete discussion on the INPUT BATCH procedure in SPIBILD. [See 2.3.1.]
Frequently the file manager may find that many records sharing some element value need to have that element value changed or that they need to be removed from the subfile. Removing records will be discussed in the next section; here we will discuss updating them.
In the updating situation, SPIBILD provides a MERGE ALL command using Global FOR, which merges the data in your active file into all the records that meet the Global FOR criteria. Like SPIBILD's BATCH procedure, this procedure causes the file to be processed prior to the MERGE command and SPIBILD processes the file again at the end of it, so it can be relatively expensive. [See 2.4.2.]
The procedure for using it is described below. Note: You must be able to process the file to use this procedure.
If you aren't in SPIBILD already, get in:
Command> spibild -Welcome to SPIBILD ->
Tell SPIBILD which subfile you want with the ESTABLISH command, which also processes the file:
-> establish supplies -Processing file: GQ.JNK.SUPPLIES -Compute time: 0.104 seconds -Elapsed time: 1.073 seconds -Core usage: 0130/0939 ->
In this example, the intent is to find records with the value RCA for the singly-occurring element SUPPLIER and replace it with the value "General Electric". See the SPIRES manual "Searching and Updating" for more information on merging in general (for instance, how to add new occurrences, replace multiple existing ones, remove occurrences, etc.).
You must have the input data in your active file. It can be in the standard SPIRES format or in a custom format; if the latter, you should issue the SET FORMAT command as part of this step. [The format must be a merge format. See the manual "SPIRES Formats" for further information.] For example,
-> collect
1. > supplier = General Electric;
2. > *** <- user presses ATTN/BREAK
->
In SPIBILD, the only Global FOR classes available are the SUBFILE and STORED classes. You can, however, use a WHERE clause and SET SCAN commands to further refine the group of records to be updated.
For example,
-? for subfile where supplier = rca -? set scan start 10000 -? set scan stop 19999 -?
The SET SCAN START and STOP commands tell SPIBILD to look only at records with key values from 10000 to 19999. This is considerably more efficient than "FOR SUBFILE WHERE SUPPLIER = RCA" alone, if all the records you want are within the limited range.
The STORED class would be preferable to use if, for example, you could easily create a stored result or stack in SPIRES of the records you wanted to update. Then SPIBILD could retrieve the records for updating directly, rather than have to use WHERE-clause examination techniques to find the records to update. Suppose, for instance, that in SPIRES you could store the search result from the command FIND SUPPLIER = RCA. Using that stored search would be much more efficient than having SPIBILD examine each record in the subfile.
The MERGE ALL command merges the data in your active file into each record fitting the Global FOR class, after which SPIBILD processes the file so that the indexes are all updated.
-? merge all from active
- Requests/Success:
MER 38 38
SUM 38 38
-End of BATCH/BUILD
-Begin passing of 38 records of record-type REC01
-Completed passing of 38 records total
-Processing: 38 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 6 82
MSTR data set 5 0
Residual data set 4 46
Rec-type REC01 1 0
Rec-type ZIN02 1 25
Total for file: 21 154
-Compute time: 7.894 seconds
-Elapsed time: 19.521 seconds
-Core usage: 0256/0933
-?
If an error occurs during step 5, SPIRES will tell you about it and stop. The most likely error comes from bad input data, i.e., an invalid element name or value for that element. That will most likely happen on the very first record -- if it won't go into one record, it probably won't go into any.
To recover from this problem, simply correct the data and start again from the ESTABLISH command in step 2.
If you discover that you updated the wrong records, you will need to determine another way to find those records and perhaps "re-merge" the old data into them.
If the system crashes during the first 4 steps of the procedure, simply restart it at your convenience. If the system crashes during the MERGE processing, see the explanation for handling this problem in the full discussion of the MERGE procedure in Global FOR in SPIBILD later in this manual. [See 2.4.2.] A simple possibility worth mentioning here is to simply restart the procedure and merge the same data again into the records that succeeded. Do not do this, though, if the merge requests are creating new element occurrences or if the data is being used to compute other element values.
Sometime you may have to remove many records from the subfile based on some criteria they share, such as all records added to the subfile more than two years ago, or records that don't have occurrences of some element.
This situation is quite easily handled in SPIRES, using the REMOVE command in Global FOR mode. REMOVE is also available under Global FOR in SPIBILD and has the same effect; in fact, it works more efficiently there than in SPIRES. However, you are limited to the Global FOR classes SUBFILE (with WHERE clauses, SET SCAN and SET FILTER commands to limit the records examined and removed) and STORED (with WHERE clauses, the SET SCAN LIMIT and SET FILTER commands) in SPIBILD.
Suppose, for example, that you want to remove 50 records of a 1000-record subfile, and you can find those 50 records using a simple search:
-> find supplier = woolco -Result: 50 SUPPLIES ->
In that case, you may want to simply remove the records in the search result using SPIRES:
-> for result +> remove all -Removed: 50 record(s) +>
The removal requests are in the deferred queue, so the records won't be removed from the tree till the file is next processed.
Another possibility would be to store the result (or stack) in SPIRES for use under the FOR STORED command in SPIBILD:
-> show result -Result: 50 SUPPLIES -> store result.woolco -'RESULT.WOOLCO' put in ORVYL file system ->
In SPIBILD, you can use the command FOR STORED RESULT.WOOLCO to remove only the records in the result (see step 3 below); under SPIBILD processing the file would be processed, and the records would be removed from the tree and "un-indexed" immediately.
Especially if there are several hundred records to remove (say, more than 15 percent of the subfile), the SPIBILD methods discussed below are preferable.
Warning: If the subfile is fairly large (say, more than 1000 records) and you want to remove more than half of them, you should consider zapping the goal record-type and rebuilding it with only the records you want to keep. [See 2.8.3.]
One final aspect to consider between the SPIRES and SPIBILD methods of removing records in Global FOR -- in SPIRES, the removal requests go into the deferred queue and can be unqueued if you later determine that you removed the wrong records. In SPIBILD, once they are removed, they are gone. Often when file managers remove many records from a subfile using SPIBILD, they make sure they have a backup copy of the file around, just in case. [See 2.1.4.3.]
The procedure for removing multiple records in SPIBILD is described below. If you plan to use FOR STORED processing, as described above, create your stored result or stack in SPIRES before beginning. Note: You must be able to process the file to use this procedure.
If you aren't in SPIBILD already, issue the SPIBILD command:
-> spibild -Welcome to SPIBILD -?
Tell SPIBILD which subfile you want with the ESTABLISH command, which also processes the file:
-? establish supplies -Processing file: GQ.JNK.SUPPLIES -Compute time: 0.208 seconds -Elapsed time: 1.037 seconds -Core usage: 0130/0939 -?
Remember that you are limited here to the FOR SUBFILE or FOR STORED command, with an optional WHERE clause. You should also use SET SCAN commands to limit the records examined, if you can, since that will save unnecessary work.
If you're not familiar with the SET SCAN commands, you might think of them this way: suppose you are asked to look for one book in the library, and you have no card catalog (index) to use. You must look on every shelf in the library till you find the book you want. But if you are told that the book you want must be on the north end of the third floor, the work you must do has been narrowed considerably. The SET SCAN commands do that type of limiting, telling SPIRES what record keys to examine.
For example, here we are removing old records from a subfile with slot keys:
-? for subfile where date-added < 1985 -? set scan stop 500 -?
It happens that we know that no records added before 1985 have keys greater than 500. The SET SCAN STOP command shown ensures that only records whose keys are 500 or less will be examined for removal.
To use FOR STORED instead, you would replace FOR SUBFILE with FOR STORED, along with the name of the stored stack or result.
-? for stored result.woolco
You may also add a WHERE clause to further limit the records that are removed; SET SCAN LIMIT is the only SET SCAN command allowed with FOR STORED, however.
Issue the REMOVE ALL command to tell SPIBILD to remove all the records that match the criteria established by the Global FOR class:
-? remove all
- REM 1, 1 @Line 1. Key = 131
- REM 2, 2 @Line 1. Key = 132
- REM 3, 3 @Line 1. Key = 134
- REM 4, 4 @Line 1. Key = 135
... (etc.) ...
- Requests/Success:
REM 58 58
SUM 58 58
-End of BATCH/BUILD
-Begin passing of 58 records of record-type REC01
-Completed passing of 58 records total
-Processed: 58 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 6 120
MSTR data set 5 0
Residual data set 7 65
Rec-type REC01 2 58
Rec-type ZIN02 1 20
Total for file: 25 264
-Compute time: 0.516 seconds
-Elapsed time: 13.240 seconds
-Core usage: 0204/0933
-?
If you discover that you removed the wrong records, the records are gone and cannot be recovered except via any backup copies of the file or data you may be able to use.
If the system crashes during the procedure, you may simply restart the procedure at your convenience.
As its manager, you have access to many interesting statistics about your SPIRES file, such as the amount of storage space the file is using, or the kinds of commands users are issuing when they select your subfile.
Here are some of the statistical questions you might ask about your file, along with possible answers.
This is a useful question to ask because storage charges are based on the number of blocks the file uses. To find out the answer, issue the ORVYL command SHOW FILES, including the name of the file and the BLOCKS option:
-> show files like almanac, blocks ORVYL File System ALMANAC.CKPT -- 7 BLKS (F: 0, L: 6) ALMANAC.DEFQ -- 2 BLKS (F: 0, L: 1) ALMANAC.MSTR -- 6 BLKS (F: 0, L: 5) ALMANAC.REC1 -- 14 BLKS (F: 0, L: 13) ALMANAC.REC2 -- 108 BLKS (F: 0, L: 107) ALMANAC.RES -- 123 BLKS (F: 0, L: 122) ->
The total of the number of blocks (the "BLKS") is the size of your file (260 in this example). [See 3.2.1.]
One way to get these statistics is to issue the SHOW FILE ACTIVITY command when you have a subfile of the file selected. For example,
-> select almanac
-> show file activity
01/26/87 Activity Record of File GQ.JNK.ALMANAC
Last Compiled: 01/07/87 at 23:55:02
Record Base Blocks Adds Deletes Updates Max Rec
Type Block Used -DS Length
ENTRY -Slot 0 14 -1 3272 0 22 339
REC2 1 105 -2 373 0 1841 72
REC3 8 Comb -2 3180 6 89 18
REC4 25 Comb -2 2564 2 407 417
REC5 103 Comb -2 7 0 21 2004
REC6 104 Comb -2 2 0 10 11063
Residual 2 123 3488 7 414 11067
Def Queue 1 3
->
By subtracting the "Deletes" from the "Adds", you get the number of records in each record-type. The ENTRY record-type, for instance, has 3272 entries (3272 minus 0).
The SHOW FILE ACTIVITY command displays some other useful information too, such as the date and time when the file was last compiled. [See 4.2.2.]
Another way is to use the ATTACH command to "attach" each record-type of the file, followed by the SHOW SUBFILE SIZE command. The SHOW SUBFILE SIZE command does exactly the work you do -- subtracting the "Deletes" from the "Adds":
-> attach 1 of gq.jnk.almanac -> show select -Record-type ENTRY of GQ.JNK.ALMANAC attached -> show subfile size -The subfile has 3272 records -> attach 2 of gq.jnk.almanac (etc.)
The ATTACH command lets you use any record-type of a file as if it were the goal record-type of a selected subfile. [See 5.4.]
If you need more information about a particular transaction in the deferred queue, such as when it happened or who did it, issue the SHOW SUBFILE TRANSACTIONS command, optionally followed by the key of the record you are interested in. For example,
-> show subfile transactions 12933
01/26/87 Transaction Log of File GQ.DOC.ERRATA
for Record-type G1SECT
Date Time Account Id Type Command Grp Key Value
01/26/87 15:45:28 GQ.JNK ADD Update 12933
01/26/87 13:32:35 GQ.JNK ADD Update 12933
01/26/87 10:54:10 GQ.JNK ADD Add 12933
->
Information about the transactions for the record that have been made since the file was last processed are shown, listed in reverse chronological order. This example shows that user GQ.JNK added the record at 10:54 a.m., and updated it twice later in the afternoon.
If you omit a specific key from the command, SPIRES will list all the transactions for the selected subfile that are in the deferred queue, in chronological order. More information about how to read the transaction log appears later in this manual. [See 4.2.3.1.]
This is representative of many questions you may have about how users are taking advantage (or failing to) of your subfile. You may want to know how many users are selecting it, or what commands they are issuing to use it.
You can ask SPIRES to "log" the use of your subfile by adding some statements to your file definition (directly or via your File Definer input). Later you can analyze the log's contents to help you evaluate the usefulness of your file, and to help you look for ways it could be improved, e.g., new indexes it needs, current indexes it doesn't need, etc.
Directions for establishing a log appear later in this manual. [See 4.3.]
Other statistical information can tell the file manager how efficiently the file's blocks are being used for storage, and how many levels deep are the trees of the various data sets of the file. This information, and much more, is available through the STATUS and SHOW FILE STATUS commands. [See 3.2.3.] But in general, those commands are of interest (and of need) to managers of large files.
When you no longer need a SPIRES file, you should destroy it, so that it will stop accruing storage charges. If the data contained in it is important to you, you might want to make a printed listing of the data or perhaps save the data in another data set on disk or tape before you "zap" the file.
The SPIRES command that erases the file is ZAP FILE, which names the file to be erased. It is quite easy to use (almost too easy, considering its drastic effects):
-> zap file gq.jnk.supplies -Do you really want to destroy file GQ.JNK.SUPPLIES ? ok ->
Once you confirm your desire, the ORVYL data sets of the file are quickly erased and the file is gone. Only the file owner can issue the ZAP FILE command for a file. [See 2.11.]
After a ZAP FILE command, the file definition still exists in the FILEDEF subfile; if you no longer need it, you should remove it from there, or consider moving it to the BACKFILE subfile. If you don't do one or the other, the subfiles of the file will continue to show up in the list from the SHOW SUBFILES command.
Also get rid of code for the file from other system subfiles, such as FORMATS, RECDEF or EXTDEF, as part of your cleanup activities. [See 2.11 for the complete description of the procedure, which describes how to find and get rid of any other code.]
The next four parts of this manual are a "User Guide" to file management. In these sections you'll find descriptions of procedures you can use to accomplish various file management tasks.
Here in Part 2, the subject is the building and destroying of record-types, subfiles and files, the most important responsibilities of every SPIRES file manager.
Many of the procedures described in this part of the manual require you to use a program called SPIBILD. The rest of this first chapter describes the SPIBILD program in general, introducing various commands and "sub-procedures" that may be useful to you as you follow the procedures in the remainder of Part 2.
Those remaining chapters of Part 2 are:
SPIBILD is a part of SPIRES that builds the data sets containing a file's records. Most procedures that involve SPIBILD cause the file to be "processed". When you process a file, the records in the file's deferred queue -- that is, those that have been added or updated since the file was last processed -- are moved from the deferred queue to the tree. Also, any records with removal requests in the deferred queue are removed from the tree. Along the way, changes are made to the indexes as appropriate, by adding, updating and removing data from them to reflect the changes to the tree.
Processing a file can be done in SPIRES or SPIBILD, depending sometimes on specific needs but generally on personal preference. [See 2.2.2, 2.2.3 for information about processing a file in SPIRES.]
You can process your own files anytime you want, or you can allow other people to process them. Or, if you want, SPIBILD will process your files automatically each night, saving you the trouble.
You can use SPIBILD as part of many other tasks too, which are discussed later in Part 2:
- to add, update or remove many records at once
- to build new indexes for a subfile that already exists
- to remove indexes you no longer need
- to remove one subfile in a file when you no longer need it but need to keep the file
- to rebuild indexes, or entire subfiles, or entire files
The rest of Chapter 2.1 discusses aspects of SPIBILD in general, which relate to most of the tasks discussed in the rest of Part 2. The remaining sections of this chapter are:
When you want to process one of your files yourself, you must use SPIBILD, the program that builds the data sets containing SPIRES records.
The online version of SPIBILD is an ORVYL program invoked by the SPIBILD command:
SPIBILD [(parameters)]
where the most popular optional parameters are:
- QUIET, to suppress the welcome message, and
- WYLBUR, to request that the command SET WYLBUR be issued automatically
Separate them with a comma or blank to request both. See Part 6 for the full syntax, or online, [EXPLAIN SPIBILD COMMAND.]
For example:
-> spibild -Welcome to SPIBILD 87.06 -?
In this example, 87.06 is the release number, indicating that this version of SPIBILD was released in June of 1987.
Unlike SPIRES, SPIBILD has no "uplow" setting, because none of its commands are case-sensitive. [SPIRES has an uplow setting because the data the user types in some commands, such as DISPLAY or FIND, may be case-sensitive.] You can type commands in uppercase, lowercase or any mixture you like.
Once you have invoked SPIBILD, you can issue any SPIBILD command, or any WYLBUR, ORVYL or MILTEN commands. To use SPIRES commands, or those of any other SPIRES processor, you must invoke that processor, leaving SPIBILD.
To leave SPIBILD, issue the EXIT command:
EXIT [QUIET]
where the QUIET option suppresses the exit message from SPIBILD.
The EXIT command takes you back to WYLBUR:
-? exit -Exiting SPIBILD Command>
You can also leave SPIBILD by issuing a command to invoke another SPIRES processor, e.g., "SPIRES".
Between the SPIBILD and EXIT commands, you will probably issue several SPIBILD commands, as part of procedures discussed in other chapters of Part 2. Other SPIBILD commands useful in many different SPIBILD procedures are discussed later in this chapter. [See 2.1.5.]
The remainder of this section discusses features and commands that the SPIBILD environment has in common with SPIRES. Most are fully documented in chapter A.2 of the SPIRES reference manual "Searching and Updating", with specific sections referenced below. In summary, those that affect SPIBILD are:
- The EXPLAIN command is available in SPIBILD, for explanations of SPIRES terms and error messages. Subfile explanations, available in SPIRES, are not available in SPIBILD, however. Nor are the SHOW EXAMPLE and SHOW SYNTAX commands. You can put an explanation in your active file by preceding the command with the IN ACTIVE prefix (see below).
- The SHOW EVAL command is available in SPIBILD as well as in SPIRES, although many variables (those involving search results or protocol execution, for example) are not available to be evaluated. The command (documented in the "SPIRES Protocols" manual) could also be used for on-the-spot computations in SPIBILD or for evaluation of static variables before and after SPIBILD processing. One likely use of SHOW EVAL in SPIBILD might be to determine which version of the program (i.e., Test or Public) you are currently using:
-? show eval $version
87.01.27 PUBLIC System
-?
- Unless otherwise mentioned in their documentation, command verbs and their options can be abbreviated to their first three characters. [See A.2.7 of "SPIRES Searching and Updating".]
- The SET WYLBUR and SET NOWYLBUR commands can be issued to specify whether WYLBUR or SPIBILD examines commands first, passing unrecognized commands to the other program. WYLBUR Exec files will not execute in SPIBILD unless SET WYLBUR is in effect. Extended Exec files will execute in SPIBILD, however, regardless of SET WYLBUR or SET NOWYLBUR. [See A.2.5.1 of "SPIRES Searching and Updating".]
- You may use the WYLBUR active file during various SPIBILD activities. For example, it may contain data that you want to merge into a SPIRES file. The IN ACTIVE prefix, with the CLEAR and CONTINUE options, may be used on some commands to direct data to your active file. [See A.2.9 of "SPIRES Searching and Updating".] The following commands relating to the active file are applicable in SPIBILD:
- SET LIST and SET NOLIST -- The default is LIST, which means that SPIBILD will automatically list your active file at the end of any command that puts data into it. [See A.2.9.2 of "SPIRES Searching and Updating".]
- SET CLEAR and SET NOCLEAR -- The default is NOCLEAR, which means that SPIBILD will not automatically clear your active file if you use the IN ACTIVE prefix without either the CLEAR or CONTINUE option. [See A.2.9.3 of "SPIRES Searching and Updating".]
- SET LENGTH -- This command can be used to change the default line length of 72, the value of the $LENGTH variable. [See A.2.9.4 of "SPIRES Searching and Updating".]
- The ATTN/BREAK key has several uses in SPIBILD:
- If you press the key in response to a command prompt, WYLBUR will either reprompt or place you in Collect mode.
- If you press the key after you have begun typing a command but before you press Return, the command you were typing is ignored, and you are prompted for a new command.
- If you press the key while SPIBILD is executing a command, you will interrupt the processing. SPIBILD may ask you if it can continue the processing, or it may simply stop altogether. Often in this case, if you press the key twice in succession, a session break will result. Type GO if you want the SPIBILD session to continue.
- If SPIBILD asks a question (such as "-Continue processing?"), pressing the ATTN/BREAK key is equivalent to answering NO.
- You can see how many read and write operations have taken place involving blocks of the files involved in a SPIBILD task by issuing the SET FILE COUNTS command prior to the main commands of the task in SPIBILD. SET FILE COUNTS is quite similar in effect to the SHOW FILE COUNTS command in SPIRES, though it does have some significant differences. See Part 6 for more information, or online, [EXPLAIN SET FILE COUNTS COMMAND.]
You can use SPIBILD through the batch system as well as interactively. That allows you to process SPIRES files, merge records into a subfile, or execute any other SPIBILD task by submitting jobs to the batch system.
There are several reasons why you might want to use SPIBILD in batch:
- 1) The input data you have for a SPIRES file is on tape or is too large to fit in your active file. [If the data is in standard SPIRES format, you can also use the BATCH command in SPIRES. [See the SPIRES manual "Searching and Updating", section D.8.]]
- 2) You want the SPIBILD work to occur at a time when you won't be logged on (in the middle of the night, for instance), so you submit a batch job to run for you at that time.
- 3) You don't want to tie up your terminal while your SPIBILD work is being done.
You can use either of two different programs when you want to use SPIBILD in batch:
The next two sections will provide detail about using these two programs for SPIBILD work and will give you a clearer picture of which to use in particular situations. [See 2.1.3.1, 2.1.3.2.] In general, more people choose to use BATWYL because of its flexibility, but here are some guidelines for choosing between them:
- If you are using SPIBILD to merge some data into a SPIRES file from a tape, you must use batch SPIBILD.
- If the input file is too large to fit into your WYLBUR active file or is in a format that cannot be imported into your active file, you must use batch SPIBILD.
- If you need return codes from the job in the event of SPIBILD problems, you should use batch SPIBILD.
- If you need to use WYLBUR Exec files, or to leave SPIBILD during the job, you should use BATWYL.
A BATWYL job is basically an interactive session controlled from the batch processor. Unlike some batch programs where writing the appropriate JCL can be very tedious and difficult, BATWYL uses very simple JCL. The challenge in constructing BATWYL jobs is to anticipate every prompt that BATWYL will send to the "virtual terminal" and include an appropriate response in the command stream.
Complete details on using BATWYL can be found in the document "Using BATWYL", available through the PUBLISH command.
Use the following JCL to run BATWYL jobs:
// JOB // EXEC BATWYL[,PARM='ATTN=*'] //SYSIN DD * ... interactive commands ...
The statements will be formally described below.
First though, here's an example of a simple BATWYL job that does some SPIBILD work (the numbers on the left are not part of the input but are used for reference in the explanation that follows):
1. // JOB
2. // EXEC BATWYL,PARM='ATTN=*'
3. //SYSIN DD *
4. call spires
5. select manual
6. for adds
7. in active display all
8. print forms=3hol duplex
9. in active show subfile transactions
10. ok
11. print forms=3hol duplex
12. exit
13. call spibild
14. process gq.doc.manual
15. logoff
In the above example, following the JCL (lines 1-3), the job calls SPIRES (4), places all the newly added records in the active file (6,7), and then prints them (8). Next, it puts the transactions display into the active file for printing (9-11). The "ok" (10) is not a command, but is the response to the SPIRES prompt "OK to clear?" that would result from the preceding command. (More likely, you would add the CLEAR option to the IN ACTIVE prefix on the SHOW SUBFILE TRANSACTIONS command, to avoid the prompt altogether.)
Next the job calls SPIBILD, processes the file, and logs off, ending the job (13-15).
Here are descriptions of the JCL statements you use to run BATWYL:
// JOB
You must begin all jobs with the JOB statement. It may be as simple as shown, or may include a job name, account number, bin number, etc. [See the document "Batch Processing" for more information.] Be sure to type blanks as shown in JCL, such as at least one before the word JOB.
// EXEC BATWYL,PARM='ATTN=*'
The EXEC statement calls the BATWYL program. The PARM option shown here allows you to create an ATTN/BREAK response by typing three asterisks at the end of a command line. The character doesn't have to be an asterisk; whatever character you choose should be specified in the ATTN clause instead. If you will not need to simulate an ATTN/BREAK, you do not need the PARM option (nor the comma preceding it). Other options are also available.
//SYSIN DD *[,DCB=LRECL=240]
The SYSIN DD statement shown above indicates that the commands to be executed by WYLBUR follow this statement. The document "Using BATWYL" describes how you can tell the program that the commands are stored elsewhere. If you add the DCB option shown, you raise the allowed length of commands from 80 to 240 characters. Note that SPIBILD commands may not exceed 160 characters, but WYLBUR and Milten commands can be as long as 235 characters.
... Interactive commands ...
Here you list the commands you want executed. If commands exceed 80 characters in length, be sure to add the DCB option to the SYSIN DD statement shown above. Note that SPIBILD commands absolutely may not exceed 160 characters.
All interactive commands are available, with the exception of those that invoke full-screen processes, such as Prism or full-screen SPIRES applications.
To use SPIBILD, you must include a SPIBILD command. After that, all SPIBILD commands are available to you. Note, however, that command options limited to batch SPIBILD, such as the "ON ddname" option on the BATCH command, are unavailable, because you are using interactive SPIBILD through the batch system, not batch SPIBILD.
Finally, be aware that output lines longer than 133 characters will be truncated.
Batch SPIBILD gives you direct access to the SPIBILD program through the batch system. You can, for instance, run jobs that process other user's files if you can do so yourself online. In addition, you may use your WYLBUR active file for input and output, and may issue WYLBUR commands such as DELETE or PRINT. However, there are some differences from online SPIBILD, and those restrictions are described in appropriate places throughout this manual and are summarized in this section.
Use the following JCL to run batch SPIBILD jobs. (Statements marked with an asterisk are optional, and are explained below.)
1. // JOB 2. // EXEC SPIBILD *3. //DDNAME DD DSN='WYL.gg.uuu.dsname',DISP=SHR, *4. // VOL=SER=volume,UNIT=unit *5. //SYSIN DD * 6. ... spibild commands ... 7. LOGOFF
For general information about batch jobs and JCL, see the "Batch Processing" manual, available through the PUBLISH command. Detail information about each of these specific statements appears below.
First, though, here is a sample job that uses batch SPIBILD:
// JOB // EXEC SPIBILD //DDNAME DD DSN='WYL.GQ.DOC.MANUAL.INPUT',DISP=SHR //SYSIN DD * batch 'manual' on ddname logoff clear
This job batches records in the data set named in the DDNAME DD statement into the subfile MANUAL.
Here are descriptions of the JCL statements you use to run batch SPIBILD:
1. // JOB
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number, output bin, region size, maximum number of output lines allowed (5000, by default), etc. See "Batch Processing" for more information.
Note that you never need to change the default region size, no matter how much SPIBILD work you are doing -- the region size of the job controls a small program that invokes interactive SPIBILD, which always has more than enough region for any SPIBILD task.
2. // EXEC SPIBILD[,PARM='parm']
The EXEC statement calls the batch SPIBILD program. The PARM option is occasionally used to set special escape characters, or to request versions of SPIBILD other than the public one, such as internal test versions, if they are available to you. The parameters most commonly specified are:
ESC=x OUT=y ACCOUNT=gg.uuu LIBRARY=[ORV.gg.uuu.]libname
If used, ESC and OUT need to appear in the PARM option before the other two.
ESC and OUT have the same effect in Batch SPIBILD as they do in the BATWYL program. Please see the document "Using BATWYL", Data Center document number 409, for more information.
*3. //DDNAME DD DSN='WYL.gg.uuu.dsname',DISP=SHR, *4. // VOL=SER=volume,UNIT=unit
Use the optional DDNAME statement when you want SPIBILD to use input data from a data set stored on tape or on disk. Generally speaking, it is easier for you to use the USE and GET commands in batch SPIBILD to bring data into your active file from stored data sets, but you must use the DDNAME statement if either of the following is true:
- the input data is on tape;
- the input data is not in a format that can be brought into your WYLBUR active file.
The data set may have a RECFM value of F or FB or U, with a fixed LRECL of up to 4090 bytes.
If the data set is cataloged, you can eliminate the second line of the DD statement shown above, as well as the comma at the end of the first line. Otherwise, specify the tape or disk volume number, and the unit TAPE or DISK. Be sure to include a /*SETUP card if you're using tapes.
*5. //SYSIN DD *[,DCB=LRECL=240]
The SYSIN statement shown above indicates that the commands to be executed by SPIBILD follow this statement. Thus, it should be the last JCL statement before the list of commands.
If you want, you can store the commands separately and then tell batch SPIBILD where to find them for execution. You do this by replacing the * in the SYSIN statement with the same type of values that follow the DD in the DDNAME statement above: the name of the data set (DSN) containing the commands, the disposition (DISP), and the VOL and UNIT information if the data set is not cataloged. The data set must be saved either in EDIT format, or at any fixed length up to LRECL=240.
If you need to issue commands longer than 80 characters (but no longer than 240) in your batch SPIBILD job, add the DCB option shown. Be aware that SPIBILD commands are limited to 160 characters, and WYLBUR and Milten commands are limited to 235 characters. (Remember too that you can abbreviate many command verbs and options, which might shrink long commands to fewer than 80 characters.)
6. SPIBILD commands
Here you list the commands you want executed, each command consisting of one and only one line of input. (See the previous paragraph about commands longer than 80 characters.)
All SPIBILD commands are available and work as you would expect, with the following exceptions:
If a SPIBILD command would prompt you for input if you were in interactive SPIBILD, then batch SPIBILD will assume a response of ATTN (the ATTN/BREAK key).
7. LOGOFF [CLEAR]
If you omit the LOGOFF statement, batch SPIBILD will supply it for you, and give you a return code of 4.
Though command lines may be as long as 150 characters, output from batch SPIBILD cannot exceed 133 characters per line. Lines longer than 133 characters will be truncated on output.
There are other lines of JCL you can add if you want, such as JOBPARM and SYSPRINT. See the "Batch Processing" manual for more information about them.
The following condition codes are supplied by SPIBILD if a job terminates abnormally:
Condition code Meaning
4 - LOGOFF CLEAR was supplied by the
program. The input stream ran out
without the proper LOGOFF command.
20 - Virtual terminal communications
problem.
24 - SUZAN path to Milten disconnected,
probably because ORVYL died.
28 - SUZAN path to SPIBILD disconnected.
32 - No SYSIN or COMMAND input data.
Normally when you use SPIBILD or SPIRES for a file-processing task, the task is completed without problems or complications. However, problems that can cause a file-processing command to go awry do occasionally arise. For example, while you are waiting for a file-processing command to execute:
- the system might fail;
- you might accidentally disconnect your terminal, causing the system to log you off;
- SPIBILD or SPIRES could encounter a serious problem with your file, causing it to stop executing the command immediately.
The results of these problems vary extensively, depending on the command being executed, on the precise point when the processing stopped, etc. In most cases, the file has suffered no ill effects at all; in other cases, there may be data integrity problems that SPIBILD or SPIRES needs to repair.
For example, suppose the system fails while SPIBILD is updating a subfile's indexes with some data from newly added records. At the time of the failure, half the indexes have been updated, but not the other half. As a result, the indexes are not reliable, and they won't be until the problem is repaired.
Occasionally your own programming error can cause problems too. For instance, changing a file definition carelessly can sometimes cause serious data integrity problems the next time the file is processed, requiring you to "back up" to a previous version of the file, stored prior to the change, to restore your file to a usable condition.
The next section describes the most common problems encountered during SPIBILD or SPIRES file-processing, with suggested methods for handling them. [See 2.1.4.1.]
Invariably, a result of interrupted file-processing commands is that the NO JOBGEN flag is set, blocking all file transactions (such as adds and removes) till it is reset.
In almost all situations where file-processing is interrupted, the file is fine; if you return to SPIBILD or SPIRES and issue the PROCESS command, the problem will be cleared up, with the JOBGEN flag reset so that file updates can continue. If you do not want to spend the time or money on a complete processing of the file, you can issue the RECOVER command instead, which does a "mini-process" of the file: just enough work to clear up any minor problems and reset the JOBGEN flag. The complete RECOVER/PROCESS command procedure for file recovery is described in the next section. [See 2.1.4.2.]
Another technique, used very infrequently, is available to files whose file-processing is automatically handled by JOBGEN. [See 2.2.1.] The SET JOBGEN command is at the heart of that procedure. It is used less frequently because all it does is reset the JOBGEN flag, so that file updates are allowed again. That allows the file to be processed again via JOBGEN, starting that night.
Unfortunately, the SET JOBGEN command can create serious data integrity problems, and should not be used unless there will be no file updates between the time you set the JOBGEN flag and the time the JOBGEN-created batch job processes the file. See Part 6 for more information; online, [EXPLAIN SET JOBGEN COMMAND.]
In extremely rare cases, such severe problems can occur that the file is considered destroyed. When a problem like that occurs, it is more likely due to system problems (such as a disk-drive failure) than to a SPIBILD problem. But in any case, backup copies of the file or parts of the file are a vital part of SPIRES applications.
Some basic backup services are provided by your computer center, which are always adequate if JOBGEN handles your file-processing needs. However, you may choose to handle a file's backup yourself. A discussion of the available services appears later in this section. [See 2.1.4.3.]
Though most file-processing tasks in SPIBILD and SPIRES run quite smoothly, this section discusses the most frequently encountered problems (some described by their system error numbers), with suggested solutions. Remember that the EXPLAIN command can be used in SPIBILD as well as SPIRES to get information about system messages and error codes.
This problem could be caused by a system crash or by your terminal becoming disconnected. Note that although the file will be in a somewhat unstable state, users can still use the file for searching, though probably not for updating, until you can execute one of the two recovery procedures to stabilize it completely. In any case, the file is not damaged, and the recovery procedures will return it to normal.
To recover the file, follow the RECOVER/PROCESS procedure. [See 2.1.4.2.]
You may have misspelled the name of the file. If that's not the case, return to SPIRES, select a subfile of the file, and check the value of the $FILENAME system variable (SHOW EVAL $FILENAME) to be sure you were trying to process the correct file.
Another possibility is that you are trying to use a file that hasn't been compiled yet. That sometimes happens when you are rebuilding a file: you issue the ZAP FILE command to destroy the old file and then go to SPIBILD to rebuild it, without first compiling it again. The file doesn't exist till it is compiled.
This error indicates that other people have subfiles of the file selected. You may want to use the SET SEARCH/SET SHARE procedure to circumvent this problem. [See 2.1.5.1.] Note: SPIRES automatically "sets SEARCH" when you issue the PROCESS command in SPIRES, so S56 errors are not a problem there.
To process someone else's file, you must have WRITE access to all the ORVYL data sets of the file except the MSTR, as well as WRITE access to the SYSTEM.CHKPT data set on the file owner's account. If you don't have the proper access, the command will fail with an S116 error. The file owner can correct the problem by issuing SET PERMIT commands for each of the data sets.
This list is not comprehensive -- other problems can stop file-processing too. In general, the error messages, combined with information available through the EXPLAIN command, should lead you to the problem and suggest a solution. Of course, file-processing problems should be taken seriously, so if you need any help, be sure to see your SPIRES consultant as soon as possible.
Procedure: RECOVER/PROCESS
Purpose: General recovery from SPIBILD problems
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Issue the PROCESS or RECOVER command
3. Fix the problem, if step 2 fails
When a SPIBILD command finishes abnormally, it may leave the file in an unstable state. It will definitely leave the NO JOBGEN flag set, meaning that automatic processing of the file by JOBGEN will not occur, and no file updates are allowed.
The RECOVER/PROCESS procedure is a general one for solving such problems; special factors for particular SPIBILD tasks that don't get completed are discussed under individual procedures.
If you aren't already in SPIBILD, issue the SPIBILD command:
-> spibild -Welcome to SPIBILD -?
Note: If you use the PROCESS command (see step 2), you can do this procedure in SPIRES; however, RECOVER and SPIBILD commands that process the file other than PROCESS are available only in SPIBILD.
in SPIBILD, depending on your preference (see below), issue either the RECOVER or PROCESS command:
RECOVER [ORV.gg.uuu.]filename PROCESS [ORV.gg.uuu.]filename
Note that the PROCESS command in this context means any SPIBILD command that processes the file, such as the BATCH, MERGE and ESTABLISH commands -- they will all have the same effect in terms of recovery.
Also note that if you are processing the file in SPIRES, you follow a slightly different procedure here at step 2: first, SELECT a subfile of the file and then issue the PROCESS command in this form:
PROCESS [NOWARN]
NOWARN tells SPIRES not to prompt you for confirmation that you want to process the file.
It may seem odd to try recovering from the problem before determining just what the problem is, but in fact, you can easily waste a lot of time trying to understand the problem, only to discover that you can solve it simply by issuing either the RECOVER or PROCESS command. Since issuing either of these commands cannot make the situation worse, but stands an excellent chance of fixing it, giving that a try is usually worthwhile.
Here are some notes to help you choose between PROCESS and RECOVER:
The PROCESS command:
- tries to completely process the file. If it cleans up the problem, then the deferred queue will be empty when the command is completed.
- may execute for a long time if the deferred queue is large.
- can be done in SPIRES, e.g., under protocol control.
The RECOVER command:
- does only a limited amount of work. (Technically speaking, it processes two pass stacks' worth of data.) For most files, where the deferred queue is not large, that will empty the deferred queue, making RECOVER equivalent to PROCESS. But for files with large deferred queues, the RECOVER command will leave records in the deferred queue, waiting for the next processing of the file.
- will not execute for very long. Thus, it may be preferable to PROCESS in situations when you need to restore the JOBGEN flag as quickly as possible so that users can resume updating records in the file.
- is available only in SPIBILD.
The sample session below shows the successful execution of the RECOVER command:
-> spibild
-Welcome to SPIBILD
-? recover bionotes
-Begin passing of 32 records of record-type REC01
-Completed passing of 32 records total
-Begin passing of 30 records of record-type REC01
-Completed passing of 62 records total
-Processed: 62 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.BIONOTES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.BIONOTES Reads Writes
CKPT data set 4 1
DEFQ data set 6 120
MSTR data set 5 0
Residual data set 7 65
Rec-type REC01 2 58
Rec-type ZIN02 1 20
Total for file: 25 264
-Compute time: 0.258 seconds
-Elapsed time: 2.310 seconds
-Core usage: 0210/0939
-?
If no errors are indicated, then the recovery was successful, users can begin using your file again, and the procedure is complete.
Note: If SPIBILD would create more than two pass stacks' worth of data, then it quits the RECOVER command without processing the file, and exits SPIBILD. Hence, in the example above, there was only enough data in the DEFQ for two pass stacks; had there been more, the message "Processed: ..." and those that follow it would have been replaced by "Exiting SPIBILD". Though the file wouldn't have been completely processed, it would be back to a useable state again.
However, if errors are indicated, as in the example below, you will need to continue to step 3:
-? recover orv.lo.ony.tunes -Action E170 error -While processing record, key = 2 -Failure when goal record-type = REC01 -Exiting SPIBILD Command>
Notice that failure invariably invokes an automatic exit from SPIBILD.
You can find out what went wrong by examining the display from the SPIBILD session, in particular, by examining the error messages and using the EXPLAIN command to find out details about specific errors. The error explanations should help you determine whether you can fix the problems or whether you should contact your SPIRES consultant. In any event, more work will be required before the file can be processed properly.
Everyone knows that having backup copies of computer files is a good idea, but the more reliable computer systems become, the less important it may seem. In fact, few of the thousands of SPIRES files created at Stanford have ever needed restoring from the system-created backup tapes due to system problems. But that fact begs the question -- history shows a number of occasions when file restoration has been necessary. In many cases, though, the problem was caused by a file owner accidentally issuing a destructive command (such as ZAP FILE) or making an unfortunate programming decision that destroys the data integrity of the file.
File backups are thus a crucial ingredient of SPIRES applications. File owners should be aware of what file backup is automatically done, what they themselves can do in addition, and what to do if they need to use the backup data sets to restore a SPIRES file.
Backup of SPIRES files is available in four forms:
If you think you need to restore a SPIRES file using the backup files created by either of those two methods, contact your SPIRES consultant for assistance.
Procedure: ORVCOPY backup Purpose: Make backup copies of data sets of a SPIRES file Who can execute it: file owner Basic steps: 1. Create the ORVCOPY JCL. 2. Run the job.
The batch program ORVCOPY, the ORVYL file dump/restore utility program, lets you back up the data sets of a SPIRES file to tape or disk and later restore them to the ORVYL file system. Besides the data itself, attributes of the file such as account-access permits are copied too.
This procedure describes the technique a file owner would use to make a backup copy of a SPIRES file stored on his or her account, storing it on tape. The reverse, restoring the copy back to the ORVYL file system, is covered briefly in the next section. [The document "ORVYL User's Guide" contains complete information about ORVCOPY, including information about storing the data on disk and dealing with other accounts.]
Note though that an ORVCOPY job can copy only data sets belonging to the account under which the job is run. Similarly, it can restore data sets only to the account under which it is run. But the dump job and the restore job can be run on separate accounts, providing a way in which data sets can be moved from one account to another. If you are moving data between accounts with ORVCOPY, be sure that all accounts have access to the tape or disk data sets to which the data is dumped.
Warning: if you use this procedure to move a SPIRES file from one account to another, the MSTR data set of the file, which contains the name of the file (including the owner's account number), must be changed as well. This change must be done by hand, as described at the end of the next section. [See 2.1.4.5.]
Use the following JCL for an ORVCOPY job to copy data sets:
1. // JOB [TIME=n] 2. /*SETUP T OUTPUT=tapeid 3. // EXEC ORVCOPY 4. //OUTPUT DD DSN=WYL.gg.uuu.name,DISP=(NEW,CATLG), 5. // UNIT=TAPE[,VOL=SER=volume] 6. dump commands
For general information about batch jobs and JCL, see the "Batch Processing" manual, available from Document Sales. Specific information about each statement in the ORVCOPY JCL appears below, and a sample job follow that.
Below are descriptions of the JCL statements you'd use to run an ORVCOPY job to copy ORVYL data sets to a new rental tape or empty tape belonging to you:
1. // JOB [TIME=n]
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number or output bin. See "Batch Processing" for more information.
Depending on how many blocks you are copying, you may need to add the TIME parameter to request more than the default 30 CPU seconds. As a rule of thumb, figure about 450 blocks can be copied per CPU second.
2. /*SETUP T OUTPUT=tapeid
The SETUP statement tells the system that the job requires tapes to be mounted. There are many different values for "tapeid", but the most common are:
3. // EXEC ORVCOPY
The EXEC statement calls the ORVCOPY program.
4. //OUTPUT DD DSN=WYL.gg.uuu.name,DISP=(NEW,CATLG), 5. // UNIT=TAPE[,VOL=SER=volume]
The OUTPUT DD statement tells the program the destination for the copy of the ORVYL data sets. Be sure the name you type for the DSN parameter is unique to your account; it must follow the naming conventions for OS data sets. (If you can't give it a unique name, replace CATLG with KEEP in the DISP parameter.) If you use your own tape rather than a scratch tape, include the VOL=SER parameter with the volser number, as given in the SETUP statement earlier.
6. dump commands
The DUMP command tells ORVCOPY what data sets to copy. You can include as many DUMP commands as you need. A single command can copy from one to all of your ORVYL data sets, depending on the option specified:
DUMP option
where "option" is one of the following:
NAME=filename[,RENAME=name] - copy the single named ORVYL data set,
giving the copy a new name if the
RENAME option is specified
LIKE=string - copy all data sets with names beginning
with the given string
RANGE=name1,name2 - copy data sets "name1" and "name2", and
all those whose names fall between them
in alphanumeric order; "name1" must come
before "name2" in alphanumeric order.
ALL - copy all data sets belonging to your account
Do not include the "ORV.gg.uuu." file prefix in any of these options. It is not necessary since you can only copy files on the account under which the job is run.
Be sure to put a blank between the DUMP command and the option you use; however, no other blanks should appear in the DUMP command.
Here is a sample job run by account GQ.DOC that uses ORVCOPY to copy the data sets of the file GQ.DOC.ERRATA:
// JOB /* SETUP T OUTPUT=SCRTCH // EXEC ORVCOPY //OUTPUT DD DSN=WYL.GQ.DOC.ERRATA.BACKUP,DISP=(NEW,KEEP), // UNIT=TAPE DUMP LIKE=ERRATA
When you want, issue the RUN command to submit the job. After it runs, you should examine the output to get the assigned tape number if you asked for a rental tape. You will need it later if you need to restore the files. [See 2.1.4.5.]
Again, see the document "Batch Processing" for information on running jobs, fetching the output, releasing tapes, etc.
Procedure: ORVCOPY restore Purpose: Restore a SPIRES file using backup data sets Who can execute it: file owner Basic steps: 1. Create the ORVCOPY JCL. 2. Run the job.
In the previous section, the procedure for creating a copy of the ORVYL data sets comprising a SPIRES file using ORVCOPY was discussed. In this section, its less frequently used complement will be covered.
If you need to restore a SPIRES file to a version you created using ORVCOPY, you also use ORVCOPY to get it back. Besides the data itself, attributes of each data set such as account-access permits are reset when ORVCOPY restores a file.
This procedure describes the technique a file owner would use to restore a SPIRES file stored on his or her account, using a copy stored on tape. The reverse, creating the copy, was discussed in the previous section. See the document "ORVYL User's Guide" for full information about ORVCOPY.
Use the following JCL for an ORVCOPY job to restore data sets:
1. // JOB [TIME=n] 2. /*SETUP T INPUT=CAT 3. // EXEC ORVCOPY 4. //INPUT DD DSN=WYL.gg.uuu.name,DISP=OLD 5. restore commands
For general information about batch jobs and JCL, see the "Batch Processing" manual, available from Document Sales. Specific information about each of the statements in the ORVCOPY JCL appears below, and some sample jobs follow that.
Below are descriptions of the JCL statements you'd use to run an ORVCOPY job to restore ORVYL data sets from a tape.
1. // JOB [TIME=n]
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number or output bin. See "Batch Processing" for more information.
Depending on how many blocks you are restoring, you may need to add the TIME parameter to request more than the default 30 CPU seconds. As a rule of thumb, figure about 450 blocks can be restored per CPU second.
2. /*SETUP T INPUT=CAT
The SETUP statement tells the system that the job requires tapes to be mounted. Specify the CAT value if the data set named in line 4 below is cataloged -- the system will find the tape volume serial number in the catalog. If the data set is not cataloged (e.g., it wasn't uniquely named), replace CAT with the volser number of the tape, which you will also need to add to line 4 (see below).
3. // EXEC ORVCOPY
The EXEC statement calls the ORVCOPY program.
4. //INPUT DD DSN=WYL.gg.uuu.name,DISP=OLD
The INPUT DD statement tells the program the source for the copy of the ORVYL data sets. If the data set is not cataloged, include the UNIT=TAPE parameter and the VOL=SER parameter with the volser number.
5. restore commands
The RESTORE command tells ORVCOPY what data sets to restore. You can include as many RESTORE commands as you need. A single command can restore from one to all of the data sets in the INPUT DD data set, depending on the option specified:
RESTORE option [REPLACE]
where "option" is one of the following:
NAME=filename[,RENAME=name] - restore the single named ORVYL data
set, giving the copy a new name if
the RENAME option is included
LIKE=string - restore all data sets with names
beginning with the given string
RANGE=name1,name2 - restore data sets "name1" and "name2", and
all those whose names fall between them
in alphanumeric order; "name1" must come
before "name2" in alphanumeric order.
ALL - restore all the data sets on the tape
Don't include the "ORV.gg.uuu." file prefix in any of these options; also, don't put blanks within the option (for instance, between LIKE and the equals sign).
Include the REPLACE option if you want ORVCOPY to replace existing data sets with the same names as those being restored. If you leave it off, ORVCOPY will only restore files that don't currently exist in ORVYL.
Below is a sample job run by account GQ.DOC that uses ORVCOPY to restore the data sets of the file GQ.DOC.ERRATA. It is the counterpart to the example in the previous section.
// JOB /* SETUP T INPUT=CAT // EXEC ORVCOPY //INPUT DD DSN=WYL.GQ.DOC.ERRATA.BACKUP,DISP=OLD RESTORE LIKE=ERRATA
When you want, issue the RUN command to submit the job. After it runs, check the file-creation date of each data set by adding the DATE option to the SHOW FILES command. See Part 6, or online, [EXPLAIN SHOW FILES COMMAND.] Those that were restored will have the date they were restored listed for "CR", the creation date.
See the document "Batch Processing" for information on running jobs, fetching the output, releasing tapes, etc.
As mentioned before, this procedure can be used to move data sets from one account to another, which means you could use it to move an entire SPIRES file from one account to another. Generally speaking, it is preferable (and cheaper) to do that by following the procedure described later in this manual. [See 2.10.]
But if you have used this procedure to move a SPIRES file from one account to another, the MSTR data set of the file, which contains the name of the file (including the owner's account number), must be changed as well. This change is best done with a special program called "fixmstr", because "recompiling" the file will not change the account to the correct one, and, in fact, SPIRES will not even allow you to recompile the file since the owning account number listed in the MSTR does not match the account the file is stored on. To change the account number and/or the file's name in the MSTR:
Command> call fixmstr acc gq.rlg (filename) or Command> call fixmstr acc gq.rlg Specify filename: <you supply the name at this prompt>
Note that you must call fixmstr from the account of the new file owner, and filename is just the file's name without the .MSTR suffix.
Procedure: COPY DEFQ
Purpose: Copy the deferred queue of a file belonging to
another account
Who can execute it: users with Copy access
Basic steps:
1. Enter SPIRES.
2. Use the COPY DEFQ command to copy the deferred queue.
Application designers constructing complex applications usually want to make sure there are adequate ways to back out any changes the application has made to files when problems occur. Frequently, doing the application's work using transaction groups or getting rid of changes with the ZAP DEFQ command is an adequate solution.
On the other hand, another method is to make backup copies of as much as possible before starting to make a complicated series of changes. In most cases, copying the entire file would be overkill, though that is certainly possible, thanks to the COPY FILE command. [See 2.10.] Another solution is to copy only the data sets of the file that would be changed by the impending transactions, in particular, the DEFQ data set. [If the file has immediate indexes that would be changed, you would either want to copy them first as well, somehow, or use some other method.]
The COPY DEFQ command lets you copy the deferred queue data set of a file that doesn't belong to you, if you have been granted Copy access to that file by the file owner. [See 2.10, step 2 to see how to do that.]
Restoring the deferred queue, if necessary, must be done by the file owner, of course. This can be accomplished either indirectly, by dumping the data set to tape and restoring it on the file owner's account with the ORVCOPY program [See 2.1.4.4, 2.1.4.5.] or directly, by using ORVYL's SET CLP, ERASE and CREATE commands. See your SPIRES consultant if you need help restoring the deferred queue.
Note that you cannot use those ORVYL commands to make the original copy from the file owner's account because you can't issue the SET CLP command to get copy access to someone else's SPIRES files; only the file owner can do that. (Remember, not being able to run the application on the file owner's account was one of our original assumptions.) The COPY DEFQ command circumvents that problem.
The COPY DEFQ command is available only in SPIRES. Issue one of those commands online, or in a BATWYL job:
Command> spires -Welcome to SPIRES ... if in trouble, try HELP ->
In a batch SPIRES job, you are already in SPIRES, so you omit this command from the command stream.
The COPY DEFQ command makes a copy of the deferred queue data set of the named file, saving it as an ORVYL data set on your account:
COPY DEFQ ORV.gg.uuu.filename [SHARE]
It must be issued from the copying account, NOT from the file owner's account. As it begins copying the data set, SPIRES displays a message showing how many blocks must be copied:
-> show eval $account GQ.JNK -> copy defq orv.gq.doc.restaurant Start copy of 50 blocks from data set DEFQ ->
See Part 6, or online EXPLAIN COPY DEFQ COMMAND, if you have any problems or need more information about the command.
What you do with the copied DEFQ is up to you. Some of the possibilities were discussed above. But chances are good that your application will work successfully, and you will not need the copy. Assuming that's true, at some point, you will want to get rid of it. (WARNING: Be sure your previous copy doesn't still exist the next time you use the COPY DEFQ command for the file -- if it does, the command will fail with an S40 error.)
To discard the copied DEFQ, issue these two commands:
SET CLP filename.DEFQ ERASE filename.DEFQ
where "filename" is the name of the copied file, without any account prefix, unless you'd like to include your own, preceded by "ORV.".
Earlier in this chapter, in section 2.1.2, we discussed some details about the SPIBILD environment, such as the command prompt, the EXPLAIN command, the ATTN/BREAK key, etc. That discussion centered around many SPIRES commands and concepts that carry over into SPIBILD. Here in section 2.1.5 and its subsections, we will consider some other aspects of using SPIBILD, aspects that either differ considerably from SPIRES or have no SPIRES equivalent.
For example, a problem arises when you want to process a file in SPIBILD when it is selected by other users. SPIBILD will not process the file but will instead give you an error message (S56).
SPIBILD has several commands that themselves do little or no work but which affect the way the primary SPIBILD commands work. For instance, the SET SEARCH command does nothing at the time it is issued, but when you issue a subsequent PROCESS command, it lets SPIBILD process the file even though users have the subfile selected.
Problems such as that, and the commands that solve them, are discussed in this section. Specific details about each command, including its syntax, appear at the end of the manual.
Note: All these commands must be issued before the command they are meant to affect -- they establish conditions under which the processing command (e.g., BATCH) is to operate. Also, some of the conditions are cleared after the processing command executes and must be reset if further SPIBILD processing under those conditions is desired. Be sure to see the detail section for each command if you need that information.
Specifically, here are the topics of the subsections of section 2.1.5:
Procedure: SET SEARCH/SET SHARE
Purpose: Get access to a file being used by others so
that you can do SPIBILD tasks; let users
use file for searching during SPIBILD work
Available in: SPIBILD, FASTBILD
Who can use it: file owner
users with Process access to the file
Basic steps:
1. Issue the SET SHARE or SET SEARCH command.
2. Issue the file-processing commands.
You cannot do any SPIBILD task of consequence to a file when it is being used by other accounts, unless you issue the SET SEARCH or SET SHARE command first:
-? process gq.jnk.almanac -System ATCHFILE error, code=S56 -Compute time: 0.019 seconds -Elapsed time: 0.256 seconds -Core usage: 0051/0939 -? explain s56 S56: The file is busy, not currently available. -?
How can SPIBILD tell you, the file owner, that your own file is not available to you? The reason is that the processing activity can interfere with the current users' activities, causing their terminal sessions to be locked up and invalidating their search results. Hence, SPIBILD protects them by blocking the processing commands till the file is free. [If your file has subfile logging in effect, you may be able to determine who is currently using your file. [See Part 6 for PERFORM USER LIST; online, use the EXPLAIN command.] Additionally, ORVYL can provide the information to you through the command "SHOW VFILES LIKE ORV.gg.uuu.filename". Type HELP SHOW VFILES for information about that command.]
However, times arise when it is imperative for you to process a file even though it is in use by others. The SET SEARCH and SET SHARE commands tell SPIBILD (and FASTBILD) to let your processing commands through. (Note: This problem does not occur when you process a file in SPIRES; SET SEARCH, explained below, is automatically in effect.)
In addition, the SET SEARCH command lets users continue to use the file for searching while the file is being processed. This is important for large files that might take a long time (say, over 10 minutes) for processing, especially files that must be available for searching 24 hours a day. [Note: no subfile logging can occur during the processing, so if you are logging subfile use, you will not receive any log entries for that period of time. [See 4.3.]]
The SET SHARE command, on the other hand, simply locks users out of the file till the SPIBILD or FASTBILD processing is completed, which means that users' terminals may get locked till your processing is over. For users trying to use the file for searching, this can be quite irritating.
But oddly enough, for users entering data, particularly with a prompting format or in a full-screen environment such as Prism, it may actually be preferable to SET SHARE. Under SET SEARCH, a record-transaction command such as ADD will cause an error (usually S494) at the end of the processing, discarding the input; under SET SHARE, the terminal would lock up, but eventually the transaction would occur.
Hence, SET SEARCH is most often used for files whose processing takes a considerable amount of time (10 minutes or more). The processing is probably done overnight when users are not trying to enter data; but the file is still needed for searching by nighttime users. SET SEARCH is usually added to the command stream that handles the regular processing.
SET SHARE, on the other hand, tends to be used on an ad hoc basis, when a file's deferred queue is small so the processing won't take long. Any user whose terminal is locked by the processing would only be inconvenienced for a moment or two.
You can issue the SET SEARCH command in SPIBILD or FASTBILD if the file you want to work with has a private checkpoint data set. Any file having any immediate indexes has a private checkpoint data set; many others do too. [The SHOW FILES command can tell you whether a SPIRES file has a private checkpoint data set. Look for the data sets for the specific file (i.e., those whose names begin with the file name, such as "filename.RES"). If there is one called "filename.CKPT", that is the private checkpoint data set.]
-? set search -?
If the file does not have a private checkpoint data set, issuing the SET SEARCH command will have precisely the same effect as SET SHARE:
-? set share -?
But neither command has any effect on anyone till the file-processing command is issued.
If you change your mind after issuing either command, you can undo your request with the SET NOSEARCH or SET NOSHARE command. (In SPIRES, SET SEARCH is automatic and unsettable; SET NOSEARCH is not available to change it.)
For example,
-? process gq.jnk.almanac -Begin passing of 11 records of record-type REC01 ... etc. ... -Compute time: 0.117 seconds -Elapsed time: 1.736 seconds -Core usage: 0069/0939 -?
More details about these two commands can be found in Part 6. Online, EXPLAIN SET SEARCH COMMAND or SET SHARE COMMAND.
Procedure: Exception File
Purpose: Place a copy of bad input data into a separate
data set to make corrections easier
Available in: SPIRES, SPIBILD, FASTBILD
Who can use it: file owner
users with Process access to the file
Basic steps:
1. Issue the SET EXCEPTION command.
2. Issue the command that adds the data into the subfile.
3. GET the exception file, correct the data that failed.
4. Resubmit the corrected data.
5. Cleanup: ERASE the exception file.
When input data causes errors, SPIRES programs generally report them to you so that you can correct them before trying to put the data into the subfile again. When input for multiple records is involved, locating the few problem records in the input file can be tedious, however. An "exception file", into which all the data for failed transactions is placed, makes the correction procedure much easier.
Exception files work with input in the standard SPIRES format or in a custom input format, as long as the format does not execute the HOLD Uproc. See the manual "SPIRES Formats" for information about input formats.
The SET EXCEPTION command creates an empty ORVYL data set that the SPIRES program will use to hold any bad data found by the processing command.
SET EXCEPTION orvyl-filename [REPLACE]
Here is the start of a SPIBILD session in which data will be added to a subfile using the INPUT BATCH command:
Command> spibild -Welcome to SPIBILD -? set exception unruly.records -?
If you change your mind and decide not to use an exception file, you can issue the CLEAR EXCEPTION command at this point. Remember, however, that an empty ORVYL data set has been created; step 5 talks about how to erase it.
Issue the commands that create the subfile transactions, such as:
SPIRES: SELECT subfile-name, and one of the following:
- INPUT BATCH - INPUT ADDUPDATE - INPUT MERGE
- INPUT ADD - INPUT ADDMERGE
SPIBILD: ESTABLISH subfile-name, and one of the following:
- INPUT BATCH - INPUT ADDUPDATE - INPUT MERGE
- INPUT ADD - INPUT ADDMERGE
FAST BATCH subfile-name
FASTBILD: FAST BUILD 'subfile-name'
During the processing of the input command, where data is coming from the active file or some other external file (e.g., from a tape), any data that causes a serious-level error, which prevents that record transaction from succeeding, is placed in the exception file.
The exception file will not work properly if lines of the input file exceed 235 characters, the maximum WYLBUR line length. That problem could arise only in batch SPIBILD and FASTBILD, in which case the lines are truncated.
Another significant limit is that no single record in the input file can exceed 80 blocks (about 160K bytes); that is larger than a record can be in SPIRES, but the limit could be significant if the input data for the record contains data that is discarded during the input process, perhaps by the input format.
The example below, continuing the SPIBILD session started above, begins with a SET MESSAGES command that suppresses messages about individual records that succeed in getting into the subfile; only the failures will get individual messages. [See 2.1.5.3.] In this simple example, we will batch two records into the subfile, the second of which contains an error:
-? set information messages = 0 -? use input.records <-- put the records in our active file -? establish biblio -Processing file: GQ.JNK.BIBLIOGRAPHY -Compute time: 0.123 seconds -Elapsed time: 1.966 seconds -Core usage: 0065/0933 -? input batch -Element=HAVE.READ: -Serious data error, code=E49 -Error at or before line 5. -Update terminated, code=S261 -Error at or before line 6. -*ADD 2, 1 @Line 6 Key=341 Err = S261 -Begin passing of 1 record of record-type REC01 -Completed passing of 1 record total - Requests/Success: ADD 2 1 SUM 2 1 -End of BATCH/BUILD ... etc. ... -Core usage: 0257/0933 -?
It is useful to have a listing of the errors, especially if there are many of them; the line numbers are particularly useful. You might want to consider running a batch SPIBILD job so that the job's output will have the error messages in it, which you can print and keep by your side as you make corrections to the data.
On the other hand, you might want a summary of the number and kinds of errors that occurred during the processing. To display this information, which is accumulated by SPIRES and SPIBILD, you can use the SHOW ERROR COUNTS command. See Part 6, or online, [EXPLAIN SHOW ERROR COUNTS COMMAND.]
Be aware that when the processing command ends, an automatic CLEAR EXCEPTION is executed. Thus, if you issue another processing command where you want an exception file, you should issue another SET EXCEPTION command. You cannot use the same exception file again, however, unless it is empty or you are willing to replace it with the new data.
Issue the GET command to place the exception file data into your active file:
GET exception.filename [CLEAR] [RENUMBER]
(GET is an ORVYL command, similar to WYLBUR's USE command.) You can then correct the data as necessary.
All the input data for each rejected record is placed in the exception file, including any record separators. For example, if the input data is in the standard SPIRES format, rejected records would be put in the exception file with the "command" that preceded them (e.g., "ADD;" or "UPDATE 492;") and the extra semicolon at the end.
In addition, the original line numbering of the input data set is maintained. So, if the first rejected record is on lines 12 through 21 of the input data set, it will have the same line numbers in the exception file. [If several data sets at once are being input, the line numbers could get entangled. The fact that the line numbers are retained could, in fact, cause a problem, with the result that the GET command would not "get" the entire exception file. You'll know the problem occurred if ORVYL responds to the GET command with the warning message "Command does not replace". In that case, issue the GET command again, adding the RENUMBER option.]
The example continues:
-? get unruly.records clear
-? list 5 <-- the line number from the error message
5. have.read = mo;
-? change "m" to "n" in 5
5. have.read = no;
-?
If the exception file is so large that the GET command cannot place the entire file in your active file, contact your SPIRES consultant for help.
Once you have corrected the data, you need to resubmit it for processing. How you do that depends on various factors. For instance, if you used FASTBILD to enter the records, you will not be able to use it for the corrections since FASTBILD requires that the record-type be empty when you start. (Of course, if every single record failed, the record-type is still empty, but in that case, the exception file did you no good.)
But if you used SPIBILD, you will likely use the same command to resubmit the corrections. If so, remember that you can set another exception file in case you have any more problems.
So, next in the example, we batch the corrected records into the subfile:
-? establish biblio -Attached file: GQ.JNK.BIBLIOGRAPHY -? input batch <-- now batch in the corrected records -Begin passing of 1 record of record-type REC01 -Completed passing of 1 record total - Requests/Success: ADD 1 1 SUM 1 1 -End of BATCH/BUILD ... etc. ... -?
When you are finished with an exception file, issue the ORVYL command ERASE to erase it.
-? erase unruly.records UNRULY.RECORDS erased in ORVYL File System -?
You can control the type of messages that SPIRES, SPIBILD and FASTBILD display during a session with the SET MESSAGES command. In most file-management situations, the default setting for each program works fine. In fact, probably the most common use of the command is in protocols and formats, where SET MESSAGES is used to suppress system messages from SPIRES, allowing the application to create and display its own.
Its most common use in SPIBILD is to suppress information messages when you are updating a subfile (e.g., using the BATCH or MERGE commands in SPIBILD). By default, SPIBILD prints a line of information for each record processed:
-> spibild
-Welcome to SPIBILD
-? establish trade stats
-Processing file: GQ.JPR.TRADE.STATS
-Begin passing of 6 records of record-type REC01
-Completed passing of 6 records total
-Processed: 6 DEFQ records of record-type REC01
-Compute time: 0.168 seconds
-Elapsed time: 2.938 seconds
-Core usage: 0210/0939
-? input batch
- ADD 1, 1 @Line 1. Key = USA1936
- ADD 2, 2 @Line 13. Key = USA1937
- ADD 3, 3 @Line 22. Key = USA1938
...
- ADD 47, 47 @Line 800. Key = USA1982
-Element=AG.IMPORTS: Value must be numeric -Serious data
error, code=E49
-Error at or before line 818.
-Update terminated, code=S261
-Error at or before line 821.
-*ADD 48, 47 @Line 821. Key = USA1983 Err = S261
- ADD 49, 48 @Line 835. Key = USA1984
-Begin passing of 48 records of record-type REC01
-Completed passing of 48 records total
- Requests/Success:
ADD 49 48
SUM 49 48
-End of BATCH/BUILD
-Detaching File: GQ.JPR.TRADE.STATS
08/11/90 File Read/Write Counts Information
File: GQ.JPR.TRADE.STATS Reads Writes
CKPT data set 4 1
DEFQ data set 4 3
MSTR data set 4 0
Residual data set 5 16
Rec-type REC01 1 3
Rec-type ZIN02 2 0
Total for file: 20 23
-Compute time: 3.182 seconds
-Elapsed time: 20.839 seconds
-Core usage: 0279/0939
-?
The information about successful transactions can be interesting, but the information about failed transactions, which is usually more important, can easily get overlooked. Moreover, the information messages slow down the processing: each information line for a record requires SPIBILD to stop its current processing, create a message and request a terminal I/O to send you the information.
If you set the information messages to the "0" level from the default level of "2", SPIBILD still gives you the most important information as well as information about any failed transactions. The session above would look like this if information messages were turned off:
-> spibild -Welcome to SPIBILD -? set information messages = 0 -? establish trade stats -Processing file: GQ.JPR.TRADE.STATS -Passing: 6 records of record-type REC01 -Processed: 6 DEFQ records of record-type REC01 -Compute time: 0.168 seconds -Elapsed time: 2.938 seconds -Core usage: 0210/0939 -? input batch -Element=AG.IMPORTS: Value must be numeric -Serious data error, code=E49 -Error at or before line 818. -Update terminated, code=S261 -Error at or before line 821. -*ADD 48, 47 @Line 821. Key = USA1983 Err = S261 -Passing: 48 records of record-type REC01 - Requests/Success: ADD 49 48 SUM 49 48 -End of BATCH/BUILD -Compute time: 3.182 seconds -Elapsed time: 20.839 seconds -Core usage: 0279/0939 -?
The only change is that SPIBILD does not display the information lines for the successful records.
Turning off the other types of messages (Serious, Warning) is not recommended for file-processing in SPIRES nor for SPIBILD and FASTBILD tasks, since serious problems could be masked. And, generally speaking, there is not much reason to set messages to any other level than 0 or 2 in SPIBILD and FASTBILD. See Part 6 for more information, or online, [EXPLAIN SET MESSAGES.]
Occasionally, emergency situations may arise where you don't want to pass data to one or more indexes, or where you only want to pass data from one of the goal record-types in the file. [Production applications may have similar needs that are routine, but such applications are uncommon.] The commands described below, available only in SPIBILD, can be used as temporary measures to control the record-types that are passing data or those that are receiving data:
- SET INDEX record-names. This command names the record-types that can receive passed information, chosen from the file's index record-types. No passing will be done to any other record-types. It and SET NOINDEX below affect only the primary work of any SPIBILD command, not any preliminary file-processing the command may do.
- SET NOINDEX record-names. This command names the record-types that are not to receive passed information as they normally would. All other index record-types will receive passed data as usual.
- SET PASS [n]. This command names the number (1 to 64) of the logical record-type that is to pass data, chosen from the file's goal record-types. No other goal record-types in the file will pass any data. If no number is specified, then all goal record-types will pass data as usual. SET PASS by itself is usually issued to countermand a SET NOPASS command. This command and SET NOPASS below affect all processing work of any SPIBILD command, including any preliminary file-processing the command may do.
- SET NOPASS [n]. This command names the number (1 to 64) of the logical record-type that is not to pass data. All other goal record-types will pass data as usual. If no number is specified, no index passing will occur. This command is necessary when using the FAST BATCH command in SPIBILD, and also has uses when you know that updating activity hasn't caused changes that will affect the indexes. [See 2.4.3.2.]
Intended for use in solving data integrity problems, all four of these commands can make such problems worse if not used very carefully. You should probably discuss the problem with your SPIRES consultant before using any of these commands. See Part 6 for full details about each of these commands, or, online, use the EXPLAIN command.
The SET REINDEX command may also be useful.
[See 6.2.10.]
Note: Another way to prevent any passing at all during a processing operation is to code the NOPASS statement in the file definition. The "nopass" condition would stay in effect for all SPIBILD tasks until the NOPASS statement was removed from the definition. See the SPIRES manual "File Definition" for details.
Procedure: Reporting in SPIBILD
Purpose: Write report data during SPIBILD input
processing
Available in: SPIBILD
Who can use it: file owner
users with Process access to the file
Basic steps:
1. DEFINE AREA areaname(rows,cols) ON FILE
2. ASSIGN AREA areaname TO orvyl.dataset.name [REPLACE]
3. Continue the session, including BATCH or MERGE task
4. CLOSE AREA areaname
When you use the SPIBILD commands BATCH or MERGE for subfile transactions and you use an input format to handle the data, the format can simultaneously produce an output report, containing information about the processed records.
Technical coding information on how to create these versatile formats appears in the manual "SPIRES Formats", section B.16.3.
To use them in SPIBILD, you need one or more "device services areas", areas of memory in the computer used to hold data for particular physical or logical devices. Most useful in this situation is the FILE area, which allows you to send the report to an ORVYL data set.
To use the FILE area in SPIBILD for report output, you issue the following commands before the BATCH or MERGE command in SPIBILD:
This command, documented in full in the SPIRES manual "Device Services", defines an area on the FILE area. The name should be the same name given in the IN-AREA statement in the format definition. If you specified "IN-AREA = FILE", then this command is not necessary.
In this command, you assign the area to an ORVYL data set -- that is where the report will end up. If the data set exists already, you can add the REPLACE option to the command.
Issue any other SPIBILD commands you want, including the SET FORMAT command if you have not set the format already. Then do your BATCH or MERGE task. Afterwards, before leaving SPIBILD or starting another task, continue to step 4.
This command detaches the ORVYL data set, allowing you to issue a GET command ("GET orvyl.dataset.name") to examine the report.
Other device services commands are available in SPIBILD, too. They are fully documented in the "Device Services" manual:
BLANK areaname READ areaname SHOW AREAS SHOW AREA areaname SET AREA areaname functions SET WRITE|NOWRITE SET READ|NOREAD
Processing a SPIRES file has three major effects:
- 1) It moves records from the deferred queue to the tree.
- 2) It updates indexes using the information in the deferred queue (except for immediate indexes, which change at the time of the record transaction).
- 3) It empties the deferred queue, discarding overhead data kept for each transaction (e.g., date, time, account doing transaction) and all but the latest copy of records in the deferred queue.
The first effect is important because records in the deferred queue are handled a bit less efficiently than those in the tree. That's because many commands must scan both the deferred queue and the tree when there are records in the deferred queue; if the deferred queue is empty, they work faster. For example, if you were creating a report using all the records in a large file, it would probably be worth your while to create the report when the deferred queue is empty, right after it is processed.
The specific contents of this chapter are:
Many file owners don't work directly with SPIBILD at all. They let a program called JOBGEN handle the processing of their files in SPIBILD automatically. By default, JOBGEN has that responsibility for a file. The authority is taken away from JOBGEN when NOAUTOGEN is in effect.
Each night after midnight, JOBGEN examines the deferred queues of all SPIRES files that have AUTOGEN set. For any file with data in the deferred queue, JOBGEN creates and submits a simple Batch SPIBILD job that processes the file and handles any batch requests submitted through the BATCH command in SPIRES.
The result is that files with data in the deferred queue are processed overnight, when computer rates are likely to be cheaper and the files are less likely to be used by end-users. Thus, the file is "fresh" the next morning, with updated indexes and no records in the deferred queue.
For files with no data in the deferred queue, no processing is needed, so no SPIBILD job is submitted and no charges are assessed.
Whether a file is to be processed by JOBGEN is indicated by the "AUTOGEN flag". The basic AUTOGEN setting for a file is determined in the file definition. By default, files are established with AUTOGEN, meaning that JOBGEN will be responsible for processing the file overnight. However, if the file owner adds the NOAUTOGEN statement to the file definition and compiles it, then the file's basic setting will be NOAUTOGEN, meaning that the JOBGEN program will not process the file. [WARNING: Account GG.PRD, under which JOBGEN is run, must have Read access to the DEFQ data set of the file; otherwise, JOBGEN can't see if any data exists in the deferred queue. By default, when you compile a new file definition, all accounts have Read access to the DEFQ data set. The ORVYL command "SHOW PERMITS filename.DEFQ" will tell you what accounts have access. To allow account GG.PRD access if it has none, issue the commands "SET CLP filename.DEFQ", "SET PERMIT filename.DEFQ FOR GG.PRD READ" and "SET NOCLP filename.DEFQ".]
Important: The file is always reset to its basic setting whenever the file is processed, whether by you or by JOBGEN.
If you are the file owner, you may change the basic AUTOGEN setting at any time by changing the file definition appropriately. You may make the change directly to the file definition itself or use File Definer to create a new definition with the change. Use File Definer only if you still have File Definer input that created the latest version of the definition.
Find the FILE keyword in your File Definer input and add or remove the NOAUTOGEN attribute, depending on whether you want NOAUTOGEN to be the basic setting. Note: if you don't have a FILE statement, then AUTOGEN is the file's basic setting. You can add a FILE input line like this, if you want NOAUTOGEN:
FILE / NOAUTOGEN
If you need help with this change, see the SPIRES manual "File Definer".
Next use File Definer to generate a revised file definition with these commands:
SPIRES ENTER FILE DEFINER INPUT ACTIVE (save your revised input file for later) GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key (recompile the file definition; see below)
If you prefer to change the file definition directly, get the file definition from the FILEDEF subfile (or wherever you keep your working copy). If you want to change the basic setting from NOAUTOGEN to AUTOGEN, you'll find the NOAUTOGEN statement several lines after the key (FILE). Simply remove it and update the record in FILEDEF.
If you want to change the basic setting from AUTOGEN to NOAUTOGEN, you need to add the NOAUTOGEN statement; you can put it right after the key and FILEDEF will move it to its appropriate location (just after the BIN statement), e.g.:
FILE = GQ.JNK.ALMANAC; NOAUTOGEN; (etc.)
Then update the record in the FILEDEF subfile.
Once you've changed the file definition in FILEDEF, recompile it:
RECOMPILE [gg.uuu].filename
If the file definition compiles properly, the basic AUTOGEN setting has changed. If it does not compile, SPIRES will tell you so. You will need to correct the file definition as SPIRES directs.
The file owner and any users with Master access to the file can still turn JOBGEN processing on or off, on a one-time basis, by issuing one of these commands:
SET AUTOGEN SET NOAUTOGEN
when a subfile of the file is selected in SPIRES.
You can see the current setting of the AUTOGEN flag by issuing the command SHOW AUTOGEN:
-> show autogen -AUTOGEN in effect -> set noautogen ->
If you issue the SET AUTOGEN command, then JOBGEN will begin checking the file's deferred queue that evening. The first evening that JOBGEN finds data to process in the deferred queue, it will submit a job to process the file. The file will then be reset to its basic AUTOGEN setting, as determined in the file definition.
On the other hand, if you issue the SET NOAUTOGEN command, then JOBGEN will not do any work for your file until the flag is reset to AUTOGEN. That happens when the SET AUTOGEN command is issued, or, if the basic setting is AUTOGEN, when you next process the file yourself.
When any SPIBILD task terminates abnormally, SPIBILD may set the "NO JOBGEN flag", which specifies that JOBGEN should not cause the file to be processed until the file owner gives the ok. Perhaps more significantly, the NO JOBGEN flag prohibits any data base transactions -- users who try receive S495 errors.
Thus, the NO JOBGEN flag affects more than simply JOBGEN. It stops all subfile transactions and any subfile logging in effect. [See 2.1.4.]
Most of the time, users discover that a problem occurred with the overnight processing of the file sometime later, when they try to make subfile modifications but get S495 errors. But there are other ways to find out too:
- The "log" from the SPIBILD session (either the online messages displayed or the output from the batch SPIBILD job) will indicate that a problem occurred and ended the SPIBILD session. When that happens, SPIBILD sets the NO JOBGEN flag. [Even if the output from JOBGEN-created jobs is normally purged (requested by the file definition statement "BIN = PURGE"), it will be printed if a SPIBILD problem occurs. In fact, if JOBGEN output unexpectedly appears in your bin, be sure to examine it carefully to see if there was a problem.]
Note also that JOBGEN processing jobs include the SET SEARCH command, which allows searching to continue during the file's processing. This can cause problems of its own if users try to update the file during processing. See Part 6 of this manual for more information about SET SEARCH. Online, EXPLAIN SET SEARCH COMMAND.
- The SHOW AUTOGEN command indicates when the NO JOBGEN flag is set:
-> show autogen
-AUTOGEN in effect (NO JOBGEN)
->
- The $DEFQTEST function can be used within applications to determine whether the NO JOBGEN flag is set. [See the manual "SPIRES Protocols", or EXPLAIN $DEFQTEST FUNCTION online.]
Generally, users who discover that the NO JOBGEN flag is set should notify the file owner, particularly if it is interfering with their work (e.g., they cannot add any records).
What should the file owner do? In most situations, he or she should follow the RECOVER procedure, described earlier. [See 2.1.4.2.]
Procedure: PROCESS/ESTABLISH
Purpose: Move records from the DEFQ to the tree; Clear DEFQ
Who can use it: file owner
users with Process access to the file
Basic steps:
1. SPIBILD
2. Issue any SPIBILD special-processing commands desired.
3. Issue the PROCESS or ESTABLISH command.
or
1. SPIRES
2. SELECT subfile-name
3. Issue the PROCESS command.
When you explicitly want to process a SPIRES file, you may be in either SPIBILD or SPIRES. SPIBILD has more special-processing features than SPIRES does, so there be reasons you will need to use SPIBILD rather than SPIRES. On the other hand, SPIRES has some advantages SPIBILD does not, such as the ability to put the processing under protocol control, and better control over messages.
Specifically, here are the significant differences between processing a file in SPIBILD and in SPIRES:
- SPIBILD exclusively has several commands that handle special file-processing circumstances, including SET INDEX, SET NOINDEX, SET PASS, SET NOPASS and SET STACK SIZE.
- If problems arise, recovery operations involving the RECOVER command must be done in SPIBILD. [See 2.1.4.2.]
- File-processing in SPIRES can be controlled from a protocol, with $NO testing for failure; that means a file can be compiled, built and processed within a SPIRES protocol.
- Under file-processing in SPIRES, messages from the procedure can be handled by SET TLOG, so that they will be logged rather than displayed at the terminal. For more information, [EXPLAIN SET TLOG.]
- File-processing in SPIRES simplifies testing when you are building files and testing indexes in SPIRES; you don't need to enter a different program and then return to SPIRES and re-select the subfile.
- For a small or temporary file that you are building in SPIRES, processing the file in SPIRES is more efficient than having immediate indexes do that work, if you were avoiding SPIBILD processing for some other reason.
- "File count" statistics are shown at the end of the SPIBILD process by default; they are not shown in SPIRES (see discussion below).
In SPIBILD, here is the procedure to follow:
Issue the SPIBILD command to enter that program. [See 2.1.2.]
Command> spibild -Welcome to SPIBILD -?
In batch SPIBILD, you are already in SPIBILD, so you omit this command from the JCL.
Several optional commands or procedures may be useful at this point, such as:
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
The chief file-processing command is PROCESS:
PROCESS filename
where "filename" must be in the form "ORV.gg.uuu.filename" if the file does not belong to you.
An alternative is the ESTABLISH command:
ESTABLISH subfile-name
where "subfile-name" is the name of a subfile in the file. The ESTABLISH command is meant for Global FOR tasks in SPIBILD, where more work with the file will be done after the file-processing is finished. Hence, it does not release the file for use by other accounts when it is done, as does the PROCESS command.
In general, use the PROCESS command, unless either:
- another procedure you're following requires the ESTABLISH command (e.g., using Global FOR in SPIBILD); or
- you have forgotten the name of the file but remember one of its subfile names.
In SPIRES, follow this procedure:
Issue the SPIRES command to enter that program.
Command> spires -Welcome to SPIRES ->
In batch SPIRES, you are already in SPIRES, so you omit this command from the JCL.
Select a subfile of the file you want to process.
SELECT [&gg.uuu.filename] subfile.name
For example, the subfile ALBUMS is part of the file called GQ.JNK.RECORDINGS:
-> select albums ->
At this point, if desired, you may issue SET PASSTRACE commands to enable tracing of Passprocs. See the manual "SPIRES File Definition", or online, [EXPLAIN SET PASSTRACE COMMAND.]
In SPIRES, the syntax for the PROCESS command is:
PROCESS [NOWARN]
The NOWARN option tells SPIRES not to ask you to confirm your request, as it does by default.
For example:
-> process -Do you want to PROCESS file GQ.JNK.RECORDINGS? ok -Processed: 39 DEFQ records of record type REC01 Compute time: 0.044 seconds Elapsed time: 1.583 seconds ->
Depending on the size of the deferred queue and the number of indexes to update, file-processing in either SPIRES or SPIBILD could take only a few seconds or up to several hours. But the "average file" with, say, several dozen records to process requires a minute or two of elapsed time. Messages reporting the progress of the processing will appear occasionally. A description of all the messages you will see appears later in this chapter. [See 2.2.4.] or online, [EXPLAIN FILE PROCESSING, MESSAGES.]
Here is how a typical SPIBILD session involving the PROCESS command might look:
Command> spibild
-Welcome to SPIBILD
-? process orv.gq.jnk.recordings
-Begin passing of 14 records of record-type ALBUM
-Completed passing of 14 records total; Key: 15
-Begin passing of 4 records of record-type ALBUM
-Completed passing of 18 records total
-Processed: 18 DEFQ records of record-type ALBUM
-Begin passing of 118 records of record-type XREC19
-Completed passing of 118 records total
-Processed: 118 DEFQ records of record-type XREC19
-Detaching File: GQ.JNK.TRECORDINGS
08/11/90 File Read/Write Counts Information
File: GQ.JNK.TRECORDINGS Reads Writes
CKPT data set 4 114
DEFQ data set 42 244
MSTR data set 26 0
Residual data set 506 559
Rec-type REC02 8 10
Rec-type REC03 156 157
... etc. ...
Total for file: 1038 1537
-Compute time: 4.989 seconds
-Elapsed time: 45.425 seconds
-Core usage: 0267/0939
-?
The messages tell us that this file has two goal record-types, ALBUM and XREC19, each with its own set of indexes to update. SPIBILD processed the 136 DEFQ records in about 45 seconds, using 4.989 real CPU seconds.
Though most file-processing tasks run quite smoothly, problems do occasionally arise. The most common problems encountered during SPIBILD processing were discussed in chapter 2.1. [See 2.1.4.1.] or [EXPLAIN PROBLEMS IN SPIBILD.]
Procedure: PROCESS SUBFILE
Purpose: Move one subfile's records from the DEFQ to the tree
Who can use it: file owner
users with Process access to the file
Basic steps:
1. SPIRES
2. SELECT subfile-name
3. Issue the PROCESS SUBFILE command.
In some situations, you may not want to process the entire file. For example, suppose you have added some records to a subfile that acts as a lookup table for the primary record-type of the file. You need the lookups to begin using the new records right away, but for efficiency reasons, the lookup actions won't look in the deferred queue for the table records. You could process the file, but that could cause problems for the people currently using the primary subfile.
So the PROCESS SUBFILE procedure lets you process only the selected subfile's worth of deferred-queue data. As soon as the record-type is processed, its deferred-queue data is discarded from the defq, leaving any data from other record-types of the file. If there is no other defq data, including logs or batch requests, then the defq is re-initialized, just as if a full-fledged file process had been done.
Issue the SPIRES command to enter that program.
Command> spires -Welcome to SPIRES ->
In batch SPIRES, you are already in SPIRES, so you omit this command from the JCL.
Select the subfile you want to process.
SELECT [&gg.uuu.filename] subfile.name
For example, the subfile ARTIST TABLE is part of the file called GQ.JNK.RECORDINGS:
-> select artist table ->
In SPIRES, the syntax for the PROCESS SUBFILE command is:
PROCESS SUBFILE [NOWARN]
The NOWARN option tells SPIRES not to ask you to confirm your request, as it does by default. It also tells SPIRES not to warn you if you have a result and/or stack in effect for the selected subfile; the PROCESS SUBFILE command must clear any stack or result in order to work successfully.
For example:
-> process subfile -Do you want to PROCESS file GQ.JNK.RECORDINGS? ok -Processed: 23 DEFQ records of record type REC12 Compute time: 0.031 seconds Elapsed time: 1.285 seconds ->
Depending on the size of the deferred queue and the number of indexes to update, subfile-processing could take only a few seconds or up to several hours. But the "average file" with, say, several dozen records to process requires no more than a minute or two of elapsed time. Messages reporting the progress of the processing will appear occasionally. A description of all the messages you will see appears later in this chapter. [See 2.2.4 or online, EXPLAIN FILE PROCESSING, MESSAGES.]
Though most file-processing tasks run quite smoothly, problems do occasionally arise. The most common problems encountered during file-processing were discussed in chapter 2.1. [See 2.1.4.1 or online, EXPLAIN PROBLEMS IN SPIBILD.]
If the process fails, note that the NOJOBGEN flag will be set, blocking subsequent updates for any record-type of the file until the file is successfully processed or recovery takes place. [See 2.1.4.2 or online, EXPLAIN RECOVERING A FILE.]
In practice then, there are three techniques used by file owners to process a file:
- 1) give JOBGEN the responsibility of overnight processing [See 2.2.1.]
- 2) submit a batch job (batch SPIBILD or BATWYL) to process the file on a regular basis, probably through the SUBMIT program [See 2.1.3, 2.2.2.]
- 3) process the file manually whenever desired, using the PROCESS/ESTABLISH procedure [See 2.1.3.]
As file owner, you can change between these methods whenever you like. For regular processing of a file, most file owners choose either the first or second method, and then manually process the file when special needs arise. Alternatively, you could decide to manually process the file most of the time, but on any given day, choose to have JOBGEN take the responsibility for that night only.
Here are some factors to consider when you are deciding what method to use:
The overhead costs of file processing decrease per record as the number of records processed increases. For example, five jobs that each process one new record will cost more than one job that processes five new records at once. If a file has only one or two transactions per day, it might not be worthwhile to have the file processed each night by JOBGEN; a job that processed the file once a week might be preferable.
Generally, the most important reason to process a file is to update the indexes to reflect the record transactions. [See 2.2 for other reasons.] You should consider how important it is that the indexes accurately reflect the current contents of the subfiles. (If constant accuracy is crucial, you probably want to consider making the indexes "immediate indexes", in which case file processing is not so important.)
JOBGEN makes a backup copy of the deferred queue of each file before it submits the job to process it. In conjunction with "dump tapes" made regularly to copy all ORVYL data sets, the backed-up deferred queues can be used to restore a SPIRES file to its state as of the previous night's processing. However, if the file owner has manually processed the file after the dump tape was last made, the data handled by that process would not be available in JOBGEN's deferred queue dump tapes. Many applications that cannot use JOBGEN processing have jobs to process the file that begin by making a backup copy of the file. [See 2.1.4.3.]
Many applications require other work to be done immediately before or after a file is processed. For example, you might want to print a copy of all records being removed, or save a copy of the transactions log of one of the file's subfiles, or make a backup copy of the file. In such situations, you would probably write your own job to process the file, surrounded by JCL and commands to handle the other tasks too.
The log data for subfiles with command logging is data in the deferred queue. Any data in the deferred queue, even if it's only log data, will cause JOBGEN to process a file.
Users can submit "batch requests" for subfile updates in SPIRES (using the BATCH command) that are processed by JOBGEN when it creates the SPIBILD job during overnight processing. Such batch requests, admittedly quite rare, reside in the deferred queue; they are completely discarded if the file is processed by anyone other than JOBGEN, without warning. If there is a chance that users are entering data via the BATCH command in SPIRES, you should either rely on JOBGEN processing the file, or be sure to check for batch requests before processing it yourself. [See "SPIRES Searching and Updating", section D.8, for more information.]
The above list of points to consider when choosing how your file should be processed are only guidelines. You will have to decide what to do based on your own conditions and preferences. Testing the possibilities is highly recommended, particularly if your application is complex.
Remember, it is easy to switch from one method to the other, either for a single processing of the file, or for a longer period of time.
When you process a file, either by an explicit PROCESS command or as part of a SPIBILD command, SPIRES or SPIBILD displays information about the processing as it proceeds. For example, here is a simple SPIBILD session showing the messages SPIBILD provides in most cases:
-? process workshop
-Begin passing of 84 records of record-type REC01
-Completed passing of 84 records total
-Begin passing of 19 records of record-type REC01
-Completed passing of 103 records total
-Processed: 103 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.WORKSHOP
08/11/90 File Read/Write Counts Information
File: GQ.JNK.WORKSHOP Reads Writes
CKPT data set 4 1
DEFQ data set 13 6
MSTR data set 4 0
Residual data set 48 166
Rec-type REC01 1 128
Rec-type ZIND2 2 208
Total for file: 72 509
-Compute time: 8.221 seconds
-Elapsed time: 56.840 seconds
-Core usage: 0244/0939
-?
This section will describe what those messages mean. In addition, it will discuss the file's command log, which may be displayed when you process a file.
The passing messages appear only if the deferred queue contains data that must be passed to indexes. The first message gives you several pieces of information:
- 1) SPIBILD is now working with the deferred-queue records of the named record-type.
- 2) The record-type is a goal record-type with indexes.
- 3) SPIBILD has moved the specified number of records from the deferred queue into the tree for that record-type.
- 4) SPIBILD is beginning to apply data from the "pass stack", an area in memory in which the index data is collected, into the indexes for the record-type. The pass stack contains the index data for that number of records.
This message appears when SPIBILD has finished applying the pass stack to the index record-types, has discarded the pass stack and is ready to start over with the next goal records, if any. The number "n" is a running total of the records of the goal record-type being worked on that have been successfully processed. The key of the record with which SPIBILD is resuming will also appear at the end if there are more goal records of the same record-type to process, as in the example above.
The pass stack has a limited size. If it fills up with data before processing all the deferred-queue records for that record-type, then SPIBILD will display the first passing message, "pass" that data, empty the pass stack, display the "Completed" passing message, and begin filling it up again. That's why the example above shows two pairs of passing messages for record-type REC01 -- the pass stack filled up from the first 84 records. After the index records were updated using that data, SPIBILD began working with the remaining REC01 records in the deferred queue.
The last section of this chapter describes the pass stack in more detail. [See 2.2.6.] or [EXPLAIN FILE PROCESSING, TECHNICAL DETAILS.]
This message appears when all processing is completed for a given record-type. That is, SPIBILD has applied the data in the deferred queue for that record-type to the tree, and updated all the indexes. This message displays the total number of records processed for the record-type.
The order in which the record-types are processed is the order in which they are defined in the file definition: in alphanumeric order by name. For example, REC01 is processed before REC02, which is processed before REC10, which precedes ZIN01.
If the deferred queue contains records for several different record-types, there will be several sets of passing and processing messages.
Many SPIRES files have subfiles for which log information is collected, invoked by the LOG and STATISTICS statements in the file definition. [See 4.3.] If the STATISTICS statement has the value "1", the log information is displayed at this point in the file processing.
The log data may be thousands of lines long. If you want to stop the display, press the ATTN/BREAK key. The log display should stop, and the file processing will resume. (Pressing ATTN/BREAK too many times may cause a session break; if so, simply type GO as directed to resume the processing.)
SPIBILD is finished working with the file and detaches it, as confirmed by this message. When you process a file is SPIRES, you will not see this message; SPIRES does not "de-select" the subfile, so the file remains attached.
SPIBILD (but not SPIRES) summarizes the I/O counts for each data set of the file that was involved in the processing. These messages can be suppressed with the CLEAR FILE COUNTS command. For further information about them, see the SET FILE COUNTS command in Part 6 of this manual; online, [EXPLAIN SET FILE COUNTS COMMAND.]
For SPIRES, to see this information, use the SHOW FILE COUNTS and CLEAR FILE COUNTS commands, which are described in the manual "SPIRES Technical Notes". Online, [EXPLAIN SHOW FILE COUNTS.]
This message displays the number of real CPU seconds used to process the file. At Stanford, this number must be multiplied by a constant to get the CPU value for which you are charged; see your consultant for more information.
This message displays the actual amount of time passed since the processing began.
This SPIBILD (not SPIRES) message displays the number of kilobytes (1024 bytes) in main memory used to process the file (nnnn) compared to the number available (mmmm). The amount available is often three or four times the amount needed. If the amount used gets very close to the amount available (say, within 10K), contact your SPIRES consultant.
When problems arise during file processing, SPIBILD or SPIRES displays appropriate error messages. You can get more information about them by issuing the EXPLAIN command. More information about specific SPIBILD problems appears elsewhere in this manual. [See 2.1.4.1.] or online, [EXPLAIN SPIBILD PROBLEMS.]
Before another user can process or perform any other SPIBILD task on one of your files, there are two actions you must take as the file owner:
The two steps are described in detail below.
You must change and recompile the file definition for this step to take effect. You can either use File Definer to change the file definition or change the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file.
If you are using File Definer to change the definition, add (or change) the PROCESS attribute onto the FILE statement of your File Definer input:
FILE/ PROCESS gg.uuu
where "gg.uuu" is the user's account number. If you don't already have a FILE statement in your input, add it as the first line of the input. You can include multiple accounts, by separating them with commas, or you can specify them in other standard forms, such as "GG...." or PUBLIC to give access to an entire group or the public at large.
Then use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key recompile the file, as described below...
If you are changing the file definition directly, get it from the FILEDEF subfile in SPIRES and add at the end (or change, if it already exists):
FILE-ACCESS = PROCESS; ACCOUNTS = gg.uuu;
You can include multiple accounts, by separating them with commas, or specify them in other standard forms, such as "GG...." or PUBLIC to give access to an entire group or the public at large.
When you've made the change, replace the old version with the new in the FILEDEF subfile.
Once you've changed the file definition, you must recompile it to effect the changes:
RECOMPILE [gg.uuu].filename
The next step is to tell ORVYL to allow the user to write data to each data set of the file except the MSTR. You must issue three separate ORVYL commands for each data set:
SET CLP filename.dsname SET PERMIT filename.dsname FOR gg.uuu WRITE SET NOCLP filename.dsname
where "filename" is the name of your file and "dsname" is the type of data set: CKPT, DEFQ, REC1, REC2, etc. and RES. (To see which RECn data sets exist for your file, type the command "SHOW FILES LIKE filename.REC".)
The other user may now process your file.
Note: some file-management procedures may cause ORVYL to lose the permits established here. In particular, procedures in which a data set is completely replaced or temporarily erased will require you to reset the ORVYL permits you set here. The most notable examples are the Rebalancing procedure and the Rebuilding a SPIRES File procedure.
[Note: The following section provides technical details on the passing procedure used by SPIBILD. It is unnecessary reading for most SPIRES file managers.]
The SPIRES/SPIBILD/FASTBILD processors make use of the Linkage Sections of a File Definition (GOALREC-NAME sections) in the following ways:
Each record type of a file is processed in ascending order by record-type number (all REC1's, then all REC2's, etc.). Goal records associated with each record-type are processed either in the order in which they are input under BATCH or MERGE command processing, or in ascending key order if through the DEFQ. Note that REMOVE, UPDATE and MERGE commands under BATCH processing normally cause DEFQ entries to be created and later processed in key order. ADD commands under BATCH processing are handled in the order they are received.
Limiting the discussion to just the PROCESS command in SPIBILD will simplify what happens, and make it possible for you to determine what happens on a "recovery" situation. For simplicity, assume all records come from the DEFQ.
If the record-type does NOT have an associated Linkage Section (or all INDEX-NAME sections are marked NOPASS), then skip to Phase 2.
For ADD-type records, a copy of the record is first added to the TREE. If the record fails to add because a copy already exists in the TREE, then it is assumed that "recovery" is taking place, and the TREE copy is replaced. The DEFQ record is now sent to the passing process marked for pointer INSERTION.
For UPDATE-type or REMOVE-type records, the original TREE copy of the record is read up and sent to the passing process marked for pointer DELETION. Then, for the UPDATE-type, the record from the DEFQ is sent to the passing process marked for pointer INSERTION.
Note that the values passed should come from data within the goal records themselves. Values created from $variables, like $UDATE, or retrieved from other records ($LOOKUP) may cause index entries that won't be deleted when the goal record is removed or updated. [See 2.2.7, 2.2.8.]
The passing process is controlled by the linkage section. Each linkage section consists of:
- A Common portion before the first INDEX-NAME that defines the pointer (PTR-ELEM) as well as any global qualifiers (QUAL-ELEM).
- Individual INDEX-NAME portions, in which index record-types are named, and index values are defined. There can also be sub-index and/or local qualifiers defined.
For each record received (marked for pointer INSERTION or DELETION), the PTR-ELEM and global QUAL-ELEM portion of the Linkage is processed first. One and only one item is placed in the "pass stack" for each item in this Common portion. Passproc rules $PASS.OCC or A165 may be used to cause no entry, or something like $PASS.ELEM or A167:1 (multiple passer) can be used to cause the first occurrence of the first existing element in the list to be placed in the pass stack. If no value can be found and $PASS.OCC (A165) or $PASS.DEF (A171) was not used, then a null value is placed in the pass stack.
Once the Common portion of the Linkage Section is finished, processing begins on each INDEX-NAME portion. The INDEX-NAME, SUB-INDEX, and local QUAL-ELEM terms constitute one INDEX-NAME portion. These terms are "traversed" in both a forward and backward manner beginning with a forward scan starting with INDEX-NAME and running through the last term. Each term causes one value to be retrieved from the goal record.
If no value can be found during a forward scan, and $PASS.OCC (A165) or $PASS.EDF (A171) was not used, then NO INDEX ENTRY is defined, and backward scan commenses. If the forward scan can be completed, then an INDEX ENTRY is defined, and backward scan commences.
During backward scan, each term attempts to retrieve a value from the goal record from where it left off during the previous forward scan. If another value can be retrieved, then forward scan resumes from this point, and if it continues forward through the last term, then another INDEX ENTRY is defined. However, if during any forward scan, new terms are within the structural bounds of previous terms, then those structural bounds limit the values retrieved to occurrences within those bounds. Terms may be bound by a "common root", which is some structural level from which separate branches to the terms can be traced.
Here are some examples using a single INDEX-NAME section with different layouts of elements in goal records. Note: only significant elements are shown.
INDEX-NAME = index; GOALREC-ELEM = WORD; PTR-GROUP = ptstr; QUAL-ELEM = qual1; GOALREC-ELEM = DATE; QUAL-ELEM = qual2; GOALREC-ELEM = TIME;
The layout of WORD, DATE, and TIME in the goal records can vary in many ways. Structural boundaries can make a big difference in what happens. Consider the following record:
Record-level WORD(1) WORD(2) DATE(1) DATE(2) TIME(1) TIME(2) ------------------|--------|--------|--------|--------|--------|
All elements are doubly occurring, and all are at the record level. The following INDEX ENTRIES will be constructed:
WORD(1) DATE(1) TIME(1) WORD(1) DATE(1) TIME(2) WORD(1) DATE(2) TIME(1) WORD(1) DATE(2) TIME(2) WORD(2) DATE(1) TIME(1) WORD(2) DATE(1) TIME(2) WORD(2) DATE(2) TIME(1) WORD(2) DATE(2) TIME(2)
Now consider the following layout:
TIME(1) TIME(2)
DATE(1) | TS(1) DATE(2) | TS(2)
|--------| |--------|
WORD(1) WORD(2) | DS(1) | DS(2)
-|--------|--------|-------------------|
Here, TIME is in a structure (TS) which has been defined as an element of another structure (DS) along with DATE, and that structure is defined at the record level along with WORD.
The following INDEX ENTRIES will be created:
WORD(1) DATE(1) TIME(1) WORD(1) DATE(2) TIME(2) WORD(2) DATE(1) TIME(1) WORD(2) DATE(2) TIME(2)
After completing the first forward pass, an attempt to pick up another value of TIME on a backward scan would have required leaving the structural bounds defined by the first DATE. On a backward scan, structural bounds defined by a previous element can't be crossed, so the backward scan continues by trying to get another DATE value. The boundary now is the record-level defined by WORD. Since the DATE structure has its root at that level, we can proceed forward again beginning with DATE(2), and then TIME(2). Scanning backward again takes us all the way back to WORD(1) from which we now pick up WORD(2) and scan forward again.
If there had been other occurrences of the TIME structure associated with DATE(1), then we would have picked all of them up along with DATE(1), but none of them with DATE(2). The same thing would have happened if TIME had been simply a multiply occurring element inside the DATE structure instead of being nested within a multiply occurring structure defined inside the DATE structure.
That is,
DATE(1) TIME(1) DATE(2) TIME(2)
|--------| |--------|
WORD(1) WORD(2) | DS(1) | DS(2)
-|--------|--------|-------------------|
Now, consider the following layout:
WORD(1) WORD(2)
|--------|
DATE(1) DATE(2) | WS(1)
|--------|----------|
TIME(1) TIME(2) | DS(1)
-|--------|--------|
Here, WORD is a multiply occurring element of a structure (WS) defined as an element of another structure (DS) containing a multiply occurring DATE element, and that structure is defined at the record-level along with a multiply occurring TIME element.
The following INDEX ENTRIES will be created:
WORD(1) DATE(1) TIME(1) WORD(1) DATE(1) TIME(2) WORD(1) DATE(2) TIME(1) WORD(1) DATE(2) TIME(2) WORD(2) DATE(1) TIME(1) WORD(2) DATE(1) TIME(2) WORD(2) DATE(2) TIME(1)
That's all possible combinations because the entries have a "common root" at the record level. Although the access path is upside-down, there still are structural bounds, beginning at WORD and working down to TIME at the record level. However, the next example shows how bounds can apply, even in upside-down accessing.
WORD(1) WORD(2)
DATE(1) | WS(1) DATE(2) | WS(2)
) |--------| |--------|
TIME(1) TIME(2) | DS(1) | DS(2)
-|--------|--------|-------------------|
Here, WORD is within a structure (WS) which has been defined as an element of another structure (DS) along with DATE, and that structure is defined at the record level along with TIME.
The following INDEX ENTRIES will be created:
WORD(1) DATE(1) TIME(1) WORD(1) DATE(1) TIME(2) WORD(2) DATE(2) TIME(1) WORD(2) DATE(2) TIME(2)
Once all INDEX ENTRIES for an INDEX-NAME section have been placed in the pass stack, the next INDEX-NAME section is processed. An INDEX-NAME section is consider completed when a backward scan for the INDEX-NAME term fails to retrieve a value. When all INDEX-NAME sections have been completed, passing for the record is finished, but the information concerning index updates only exists in the pass stack.
When the pass stack fills up or no more records are retrieved from the DEFQ, then the pass stack is processed by sorting it down to unique INDEX ENTRIES cancelling duplicate entries and any matching INSERTION/DELETION pairs. The result is a list of INSERTION and/or DELETION entries to be applied to the indexes. The appropriate index records are added, removed, or updated using the the pass stack information. Because of the sort, an index record need only be read once in order to apply all the pass stack information associated with that record.
When a Pass Stack's worth of goal records has been passed, those same goal records are reread from the DEFQ. Each REMOVE-type record causes the TREE copy to be removed; each UPDATE-type record is used to replace the TREE copy. Then the DEFQ record is marked as DEQueued. When the group of records that contributed to the the pass stack are all DEQueued, Phase 1 repeats with another group of goal records until all DEFQ record for this record-type have been processed. When done with all records, continue with Phase 3.
All the records in the DEFQ associated with one record-type are read sequentially. For each ADD-type record, the record is added to the TREE. For each REMOVE-type record, the TREE copy of the record is removed. For each UPDATE-type record, the TREE copy is replaced by the DEFQ copy. When done with all records, continue with Phase 3.
When all the records for a particular record-type have been processed, the next logical record-type is processed starting back at Phase 1. When all record-types have been processed, the DEFQ is cleared and the PROCESS command is finished.
Note that Immediate Indexing in SPIRES does Phase 1 for just the single record being immediately indexed, and only those INDEX-NAME terms marked as IMMEDIATE are sent to the pass stack. These same records are usually processed again later by SPIBILD, but then only the non-IMMEDIATE terms are sent to the pass stack because the IMMEDIATE terms are already done. The TREE copy of the record is not REMOVED or UPDATED during immediate indexing. The DEFQ information becomes the latest copy of the record, and may participate in another Immediate Indexing transaction.
Passing to indexes requires you be able to undo that indexing. Here's a simple example of something that works just fine.
RECORD-NAME = GOAL; ....etc; ELEM = MODDATE; LEN = 4; OCC = 1; INPROC = $DATE(UPD); OUTPROC = $DATE.OUT;
An index record might be defined something like this:
RECORD-NAME = INDX; KEY = MODDATE; LEN = 4; INPROC = $DATE; OUTPROC = $DATE.OUT; ELEM = pointer; ....etc;
The Linkage section would probably look like this:
GOALREC-NAME = GOAL;
EXTERNAL-NAME = GOAL;
PTR-ELEM = pointer; ....etc;
INDEX-NAME = INDX;
SEARCHTERMS = MODDATE;
SEARCHPROC = $DATE(TRUNC);
PASSPROC = $PASS(BINARY);
GOALREC-ELEM = MODDATE;
PTR-GROUP = pointer;
If you ADD a new record to the database, MODDATE gets today's date as its value. The Linkage section for GOAL passes MODDATE to the INDX index. Then, the new record is placed in the GOAL record's TREE. [See 2.2.6.]
Now, several days pass and you decide to update this GOAL record. The updated record get's today's date as the value of MODDATE, and the record is placed in the DEFQ. Of course, "today's date" is different than when the record was originally added. When passing occurs, the TREE copy of the record is read and the value of MODDATE stored in that record is sent to the pass stack marked for DELETION.
The record in the DEFQ then passes its MODDATE value to the pass stack marked for INSERTION. The DELETION and INSERTION values are different, so the old value will be un-indexed, and the new value will be indexed.
But here's a simple example of something that doesn't work. To save space, you decide to make MODDATE into a VIRTUAL element, and have it pass the equivalent of the $UDATE variable. Your MODDATE element in the GOAL record is defined as follows:
RECORD-NAME = GOAL; ....etc; VIRTUAL; ELEM = MODDATE; LEN = 4; OCC = 1; INPROC = $DATE; OUTPROC = $CALL(GENDATE); USERDEFS; USERPROC = GENDATE; UPROC = SET VALUE = $DATE;
Everything else remains the same.
If you ADD a new record to this database, MODDATE has no value because it is VIRTUAL. But the Linkage section obtains a value from MODDATE to pass to the INDX, and that value is today's date. Then, the new record is placed in the GOAL record's TREE.
So far, it seems like everything is fine. But now, several days pass and you decide to update this GOAL record. The record is placed in the DEFQ. Again, MODDATE has no value because it is VIRTUAL.
When passing occurs, the TREE copy of the record is read and the value of MODDATE created is today's date, which is sent to the pass stack marked for DELETION. However, that wasn't the date that was originally indexed.
The record in the DEFQ then passes its MODDATE created value to the pass stack marked for INSERTION. The DELETION and INSERTION values are identical! This is a matching INSERTION/DELETION pair, and it is eliminated from the pass stack. The result is that NO INDX entries are changed.
Even if you tried to REMOVE the GOAL record at some date after it was originally added and processed, you would not be able to un-index the original date from the INDX. This is an example of why you should NOT pass values dependent upon variables.
Indexes are usually built from values contained within the GOAL record itself. There may be some transformation of the value, but it is a consistent transformation which will always produce the same index value(s) for a given GOAL value.
For example, consider GOAL and ZINDX records defined as:
RECORD-NAME = GOAL; KEY = something; ....etc; ELEM = ZIPCODE; RECORD-NAME = ZINDX; KEY = ZIPCODE; ELEM = pointer; ....etc;
The Linkage section might contain something like this:
GOALREC-NAME = GOAL;
PTR-ELEM = pointer; ....etc;
INDEX-NAME = ZINDX;
SEARCHTERMS = ZIPCODE;
PASSPROC = $PASS;
GOALREC-ELEM = ZIPCODE;
PTR-GROUP = pointer;
Occurrences of ZIPCODE in the GOAL records are passed to the ZINDX so you can search for GOAL records by ZIPCODE. There's nothing unusual here.
But now you discover there is a separate subfile (call ZIPTABLE) with records that act like a table pairing ZIPCODE with CITY names. Records in the ZIPTABLE subfile might look like this:
ZIPCODE = 94040; CITY = MTN VIEW; ZIPCODE = 94041; CITY = MTN VIEW; ZIPCODE = 94022; CITY = LOS ALTOS; ZIPCODE = 94023; CITY = LOS ALTOS HILLS;
You decide you want to build a CITY index in your database, using your ZIPCODE to lookup CITY in this separate subfile, and passing that value to a word index. You add the following record-type to your file definition:
RECORD-NAME = ZNAME; KEY = WORD; ELEM = pointer; ....etc;
You also add a VIRTUAL element to your GOAL record definition:
VIRTUAL;
ELEM = CITY;
REDEFINES = ZIPCODE;
OUTPROC = $CALL(GETCITY);
USERDEFS;
USERPROC = GETCITY;
UPROC = SET VALUE = $LOOKSUBF($VAL,REPLACE,ZIPTABLE,CITY);
And finally, you add the following to the Linkage section:
INDEX-NAME = ZNAME;
SEARCHTERMS = CITY;
SEARCHPROC = $TRUNC;
PASSPROC = $PASS(PHRASE) / $BREAK;
GOALREC-ELEM = CITY;
PTR-GROUP = pointer;
The first time you add a GOAL record, ZIPCODE is used to lookup CITY in the ZIPTABLE subfile, and that values is broken into words and sent to the ZNAME index.
But what happens if the table changes after you've built ZNAME entries? For example, assume you used 94040 as a ZIPCODE which built ZNAME index entries for MTN and VIEW. Then the 94040 record in the ZIPTABLE subfile is changed like this:
ZIPCODE = 94040; CITY = MOUNTAIN VIEW;
The next time a GOAL record accesses the table for 94040 it will obtain MOUNTAIN VIEW as the CITY, and that is NOT what was originally sent to the ZNAME index. If you tried to REMOVE a GOAL record which had 94040 as a ZIPCODE, the word VIEW would be unindexed, but not the word MTN. Likewise, if you UPDATE a GOAL record which has 94040 as a ZIPCODE, and changed it to 94022, you would unindex the word VIEW, but NOT the word MTN. And then you would index the words LOS and ALTOS. Your record would still be found by: FIND CITY MTN; even though the ZIPCODE is no longer associated with MTN VIEW.
Even if you updated all GOAL records that contained 94040 as the ZIPCODE, and didn't change the ZIPCODE, you wouldn't affect the ZNAME index. That's because both the TREE and DEFQ copies of the GOAL record would retrieve the same 94040 ZIPTABLE record, and would get MOUNTAIN VIEW as the CITY. These terms cancel in the pass stack, and ZNAME is left unchanged. [See 2.2.6.]
This is the danger in passing values obtained by $LOOKSUBF or $LOOKSUBG. The accessed value from such records may not change once they are stored. You can add new table entries with new codes, and you can change the accessing codes in your GOAL record, but you can't allow the accessed values in the table to change.
A better solution, to get what you want without fear of table changes, is to have the ZIPTABLE subfile build its own CITY word index. Then, instead of building a ZNAME index, you code "Indirect Search". [EXPLAIN INDIRECT SEARCHING.]
SPIRES, SPIBILD and FASTBILD provide several ways for you to add multiple records to a subfile at once. The different methods work under different circumstances -- which one you choose will depend on the particular situation you have.
Below is a comparative summary of the methods. The procedures involving SPIBILD are described in detail in the rest of this chapter; those involving FASTBILD are described later in Part 2. [See 2.12.] Information about the SPIRES procedures appears in the SPIRES manual "Searching and Updating".
This chapter ends with a short section about the format of the data in input data sets used with these procedures [See 2.3.5.] and a section on how to resume data input if a problem terminates your input processing in SPIBILD using one of the methods described in this chapter. [See 2.3.6.]
There are 14 techniques for adding multiple records to a subfile in a single shot:
- 1. (SPIRES) the INPUT BATCH procedure
- 2. (SPIRES) the INPUT ADD procedure
- 3. (SPIRES) the INPUT ADDUPDATE procedure
- 4. (SPIRES) the INPUT ADDMERGE procedure
- 5. (SPIRES) the BATCH procedure
- 6. (SPIBILD) the INPUT BATCH procedure
- 7. (SPIBILD) the INPUT ADD procedure
- 8. (SPIBILD) the INPUT ADDUPDATE procedure
- 9. (SPIBILD) the INPUT ADDMERGE procedure
- 10. (SPIBILD) the LOAD procedure
- 11. (SPIRES or SPIBILD) the INPUT LOAD procedure
- 12. (SPIBILD) the FAST BATCH / FAST LOAD procedure
- 13. (FASTBILD) the FAST BUILD procedure
- 14. (FASTBILD) the LOAD procedure
Which you should use in a given situation depends primarily on these factors:
- If it's in the internal SPIRES format (i.e., coming from another SPIRES file via the GENERATE LOAD command), you must use the LOAD procedure in SPIBILD or FASTBILD, or the INPUT LOAD procedure, or the FAST LOAD procedure in SPIBILD.
- If it's in the standard SPIRES format, you can use the INPUT BATCH procedures in SPIRES or SPIBILD, the BATCH procedure in SPIRES, the FAST BATCH procedure in SPIBILD or the FAST BUILD procedure in FASTBILD.
- If it's in a customized format, you can use the INPUT ADD, INPUT ADDUPDATE or INPUT ADDMERGE procedures in SPIRES or SPIBILD, the FAST BATCH procedure in SPIBILD, or the FAST BUILD procedure in FASTBILD.
Below are descriptions of each of the techniques listed above.
The INPUT BATCH command builds records from the input data set and adds them to the file's deferred queue. The data may be in the standard SPIRES format and may include, besides records to be added, records to be updated, merged or removed. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.
The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. It can be used with data stored on tape through a BATSPI batch job.
Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner, not the user doing the input, who pays for only the input costs.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.
See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES in general. For online help, EXPLAIN INPUT BATCH.
The INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands build records from the input data set and add them to the file's deferred queue. The data must be in a form that can be read by a customized input format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.
INPUT ADD can handle only records being added. INPUT ADDUPDATE and INPUT ADDMERGE check the subfile to see whether a record with the same key as given in the input data already exists, in which case that record is replaced with the new input (INPUT ADDUPDATE) or the input is merged into that record (INPUT ADDMERGE).
These INPUT commands in SPIRES are available to any user with update privileges to the subfile. Each can be used with data stored on tape through a BATSPI batch job.
Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner, not the user doing the input, who pays for only the input costs.
See the SPIRES manual "Searching and Updating" for more information about the INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands in SPIRES. For online help, EXPLAIN INPUT ADD COMMAND, etc.
The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.
This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.
The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user does not know till the next day whether the records were accepted, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.
The input data must be in the standard SPIRES format. It is not limited to added records, but can include updates, merges or removes too.
A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched, of course; but the request which tells JOBGEN where the data is gets thrown away. This problem makes the BATCH command a somewhat unreliable method, particularly if you do not know the file owner.
For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".
The INPUT BATCH command in SPIBILD builds records from the input data set and adds them directly to the tree. As it does that, it builds the pass stack (the changes to be applied to the subfile's indexes) for the added records, which is processed and emptied when it fills up or when the end of the input data is reached.
The data must be in the standard SPIRES format and may include records to be updated, merged or removed. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the records involved.
The INPUT BATCH procedure in SPIBILD is unlike the INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.
The input data may be on disk or tape, though if on tape, the procedure must be done through a batch SPIBILD job.
The steps of this procedure are described in detail in the next section. [EXPLAIN BATCH PROCEDURE IN SPIBILD online.]
The INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands in SPIBILD build records from the input data set and add them directly to the tree. As they do that, they build the pass stack (the changes to be applied to the subfile's indexes) for the added records, which is processed and emptied when it fills up or when the end of the input data is reached.
The data must be in a form that can be read by a customized input format. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the records involved.
INPUT ADD can handle only records being added. INPUT ADDUPDATE and INPUT ADDMERGE check the subfile to see whether a record with the same key as given in the input data already exists, in which case that record is replaced with the new input (INPUT ADDUPDATE) or the input is merged into that record (INPUT ADDMERGE).
These INPUT procedures in SPIBILD are unlike their counterparts in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use these SPIBILD procedures. Again, the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.
The input data may be on disk or tape, though if on tape, the procedure must be done through a batch SPIBILD job.
The steps of this procedure are described in detail later in this chapter. [EXPLAIN INPUT ADD PROCEDURE IN SPIBILD online.]
You can use the LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file).
The input data is the internal SPIRES form of the records, exactly as stored in the ORVYL data sets. To create the data, you use the Global FOR command GENERATE LOAD in SPIRES. Then in SPIBILD, you issue the LOAD command to add the data to the appropriate file.
The major advantage of this method is that it avoids the substantial overhead of executing Outproc rules when the input data set is being created, and of executing Inproc rules when the data set is being loaded into the new file.
The input data for the LOAD procedure must come from a disk data set created by the GENERATE LOAD command; it cannot come directly from a tape.
Again, because the LOAD command causes the file to be processed, you cannot use it unless you have Process access to the file.
The LOAD procedure is covered in detail later in this chapter. [See 2.3.3.]
You can use the INPUT LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file). It is similar to the LOAD procedure and has the same advantages; it is different in that it is available in SPIRES as well as SPIBILD, which allows you to take advantage of deferred-queue processing, to use it within a protocol, or to use it with temporary files, which are available only in SPIRES.
In SPIRES, INPUT LOAD also has the WITH FAST and WITH SEARCHONLY options, which speed up the processing, at the cost of some restrictions.
The input data is the internal SPIRES form of the records, exactly as stored in the ORVYL data sets. To create the data, you use the Global FOR command GENERATE LOAD in SPIRES. Then in SPIRES or SPIBILD, you issue the INPUT LOAD command to add the data to the appropriate file.
The major advantage of this method is that it avoids the substantial overhead of executing Outproc rules when the input data set is being created, and of executing Inproc rules when the data set is being loaded into the new file.
You cannot use it unless you have Master access to the file.
The INPUT LOAD procedure is covered in detail later in this chapter. [See 2.3.3a.]
The FAST BATCH and FAST LOAD procedures for adding records are specialized techniques that can be used only in certain circumstances:
- The record-type into which the records will be added must be completely empty, unless the record-type has a slot key.
- The records in the input data must either be in ascending key order or else have their keys assigned by SPIRES in ascending order (e.g., a slot key).
The records will go into the record-type very quickly and efficiently, many times faster than they probably would with the standard INPUT BATCH or INPUT ADD procedure in SPIBILD (see above). The gain in efficiency comes at the loss of two features of the INPUT procedures:
- The FAST procedures do not pass data to indexes; if the record-type has indexes, you must build them later, using the BUILD procedure. [See 2.7.1.]
- The FAST procedures do not checkpoint the data. That means that the procedure must be started over again if a major problem such as system failure occurs.
FAST BATCH and FAST LOAD are very useful when you have hundreds of thousands of records to add to a new file. They break the file-building process into several pieces (building the goal record-type with the FAST BATCH or FAST LOAD command and building the indexes later with the BUILD command), which can be scheduled independently. For example, you might need many hours to build a large file using the BATCH command in SPIBILD; you would need much less total time to do the same work using FAST BATCH and BUILD, plus you could do the two pieces whenever convenient.
To use the FAST BATCH procedure, you must have Process access to the file. The input data for the FAST BATCH procedure may be on tape, in which case you must use batch SPIBILD, and it may be formatted, as long as you have an input format that can read it.
You can use the FAST LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file). The data must be stored in a load file online; it cannot be on tape. The FAST LOAD procedure is similar to the LOAD procedure and has the same advantages; but it also has the same advantages as the FAST BATCH procedure.
FAST BATCH and FAST LOAD, though similar to FASTBILD in some respects (see below), are much easier to use. The complete procedure is described later in this chapter. [See 2.3.4.]
Of all the procedures, FAST BUILD and LOAD in FASTBILD are the most difficult to use. Like FAST BATCH, they can be used only if the goal record-type is empty to begin with. They also separate building the goal record-type from building the indexes, though you can have FASTBILD take care of the index-building automatically, if you like.
Also like FAST BATCH, FAST BUILD and LOAD do not checkpoint the data; if problems arise, the procedure must be restarted. Problems tend to arise more often in FASTBILD because the JCL and commands you must put together in order to use it can be quite complex. Several limits that are not always easy to determine must be specified in the job stream; errors in these estimates can cause serious problems, requiring you to start the job over again.
But on the other hand, the FASTBILD procedures are the most efficient of all when you are building a record-type of thousands of records. The index-building process is considerably more efficient in FASTBILD than SPIBILD, particularly in I/O usage.
You can use the FASTBILD procedures if you have Process access to the file. The input data may be formatted (though you will need an input format to read it) and may be on tape. If it is formatted in the internal SPIRES format, i.e., if it is coming directly from another SPIRES file, you use LOAD instead of FAST BUILD.
More information about FASTBILD and a complete description of the FAST BUILD and LOAD procedures appear in a later chapter. [See 2.12.]
Procedure: INPUT BATCH procedure in SPIBILD
Purpose: Add, update or remove multiple records in a subfile
using standard SPIRES format
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Prepare the input data set.
3. Issue any SPIBILD special-processing commands desired.
4. ESTABLISH subfile-name
5. Issue the INPUT BATCH command.
This procedure allows you to add multiple records to a subfile at once using SPIBILD. The input data is in the standard SPIRES format, and may also contain requests for record updates and removes as well. [See 2.3.5.]
You may use this procedure either online or in batch. [See 2.1.3.] If using it in batch, you will need to modify the steps of the procedure to fit batch, rather than interactive, work.
To invoke SPIBILD, issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET EXCEPTION command, to name an ORVYL data set into which unacceptable records from the input data set are placed. [See 2.1.5.2.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of lines in the input file. It is used for recovery purposes when a previous SPIBILD task has failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. [See the description for that command in Part 6, or online, EXPLAIN SET STACK SIZE COMMAND.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT BATCH command will work properly.
The INPUT BATCH command in SPIBILD batches the input data into the named subfile:
For online SPIBILD, and the BATWYL program: INPUT BATCH For batch SPIBILD: [FROM ddname] INPUT BATCH
Here is a typical online SPIBILD session using the INPUT BATCH procedure:
Command> spibild
-Welcome to SPIBILD
-? use data.from.bakers
-? set exception crummy.cookies
-? set information messages = 0
-? establish cookies
-Compute time: 0.238 seconds
-Elapsed time: 1.544 seconds
-Core usage: 0065/0933
-? input batch
-Element=OVEN.TEMP: -Must be integer -Serious data error, code=E21
-Error at or before line 396.
-Update terminated, code=S261
-Error at or before line 399.
-*ADD 27, 26 @Line 399 Key = CHOCCHIP Err = S261
-Begin passing of 52 records of record-type REC01
-Completed passing of 52 records total
- Requests/Success:
ADD 53 52
SUM 53 52
-End of BATCH/BUILD
-Detaching File: AM.OSS.COOKIES
08/11/90 File Read/Write Counts Information
File: AM.OSS.COOKIES Reads Writes
CKPT data set 4 1
DEFQ data set 9 7
MSTR data set 6 0
Residual data set 23 29
Rec-type REC01 2 14
Rec-type ZIN02 22 22
Total for file: 66 73
-Compute time: 9.772 seconds
-Elapsed time: 48.202 seconds
-Core usage: 0257/0933
-?
The record that failed to be added (because the value of the OVEN.TEMP element was not an integer) was placed in the exception file, an ORVYL data set called CRUMMY.COOKIES. The other 52 records in the input data set were added successfully.
The most common problem that occurs was shown above: one or more records fail to go into the subfile because an Inproc processing rule rejects them. The solution is to find the rejected records, correct them, and try the procedure again with only those records in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected records there. [See 2.1.5.2.]
System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.
If the failure happens prior to the INPUT BATCH command, you should just start the entire procedure over again when you can.
If it happens during the INPUT BATCH command, you will want to determine the last record that made it through the entire process (i.e., that was added and indexed). All records added before the last "-Begin passing:" message that's not followed by a "Completed passing..." message are in the subfile; all those added after it will be discarded during file recovery. You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. You may either delete the successful records from the front of the input data set or skip them using the SET SKIP command, but in either case, you should restart the procedure, omitting the records that previously went in successfully. [See 2.3.6.]
Procedures: INPUT ADD procedure in SPIBILD
INPUT ADDUPDATE procedure in SPIBILD
INPUT ADDMERGE procedure in SPIBILD
Purpose: Add multiple records to a subfile using an input
format; also, for INPUT ADDUPDATE, update records if
they already exist; also, for INPUT ADDMERGE, merge
data into records if they already exist
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Prepare the input data set.
3. Issue any SPIBILD special-processing commands desired.
4. ESTABLISH subfile-name
5. Set the input format.
6. Issue the appropriate INPUT command.
This procedure allows you to add multiple records to a subfile at once using SPIBILD. The input data will be read by a customized input format. See the manual "SPIRES Formats" for information about input formats for multiple records; online, [EXPLAIN FORMATS, FOR MULTIPLE-RECORD INPUT.]
In addition, by using either the INPUT ADDUPDATE or INPUT ADDMERGE command, you can tell SPIBILD not to reject a record in the input data if its key matches the key of a record already in the subfile, but instead to use the input data to update that record. In the case of INPUT ADDUPDATE, the input data for the record completely replaces the record already in the subfile; for INPUT ADDMERGE, the input data is merged into the record already there. Particularly for INPUT ADDMERGE, you may need to include some special processing techniques within the input format, which are described in the "SPIRES Formats" manual; online, [EXPLAIN FORMATS, FOR ADDUPDATE COMMAND.]
You may use these procedures either online or in batch. [See 2.1.3.] If using one in batch, you will need to modify the steps of the procedure to fit batch, rather than interactive, work.
To invoke SPIBILD, issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET EXCEPTION command, to name an ORVYL data set into which rejected records from the input data set are placed. [See 2.1.5.2.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of lines in the input file. It is used for recovery purposes when a previous SPIBILD task has failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
- the SET PTRACE command, in one or more of its forms, to display tracing information for input or output processing rules (Inproc, Inclose, Outproc) executed during the record input processing.
You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT command will work properly.
Depending on what the format does, any of the commands listed below may be involved. All except SET FORMAT are optional; those you use should be issued in the order shown:
- ALLOCATE. This command allocates a vgroup, usually so that stored values can be restored to it (see RESTORE STATIC).
- RESTORE STATIC. This command restores previously stored values to a vgroup. See "SPIRES Protocols" for more information about these two commands; online, [EXPLAIN ALLOCATE.] or [EXPLAIN RESTORE STATIC.]
- DEFINE AREA and ASSIGN AREA. These commands (as well as CLOSE AREA to be issued after the INPUT command) are needed when the input format is also creating a report. [See 2.1.5.5.]
- SET FORMAT. The SET FORMAT command sets the named format, or, if the "*" form of the command is used, executes the startup frame of the format named in the previous SET FORMAT command. You must start with a "SET FORMAT format-name" command, with one or more SET FORMAT * commands optionally to follow.
SET FORMAT {format-name|*} [,parm|'parm'|"parm"]
- SET FTRACE. SET FTRACE enables format-tracing messages, most often used for debugging. [See "SPIRES Formats", section B.7.2.]
Issue the INPUT ADD, INPUT ADDUPDATE or INPUT ADDMERGE command to batch the input data into the named subfile:
For online SPIBILD, and the BATWYL program:
[USING frame] INPUT {ADD|ADDUPDATE|ADDMERGE}
For batch SPIBILD:
[FROM ddname] [USING frame] INPUT {ADD|ADDUPDATE|ADDMERGE}
Here is a typical online SPIBILD session using the INPUT ADD procedure:
Command> spibild -Welcome to SPIBILD -? use new.students.data -? set information messages = 0 -? clear file counts -? establish students -Compute time: 0.238 seconds -Elapsed time: 1.544 seconds -Core usage: 0065/0933 -? set format new -? input add -Begin passing of 36 records of record-type REC01 -Completed passing of 36 records total - Requests/Success: ADD 36 36 SUM 36 36 -End of BATCH/BUILD -Compute time: 7.229 seconds -Elapsed time: 24.820 seconds -Core usage: 0257/0933 -?
The most common problem that occurs is that one or more records fail to go into the subfile because an Inproc processing rule rejects them. The solution is to find the rejected records, correct them, and try the procedure again with only those records in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected records there. [See 2.1.5.2.]
System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.
If the failure happens prior to the INPUT command, you should just start the entire procedure over again when you can.
If it happens during the INPUT command, you will want to determine the last record that made it through the entire process (i.e., that was added and indexed). All records added before the last "-Begin passing:" message that's not followed by a "-Completed passing..." message are in the subfile; all those added after it will be discarded during file recovery. You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. You may either delete the successful records from the front of the input data set or skip them using the SET SKIP command, but in either case, you should restart the procedure, omitting the records that previously went in successfully. [See 2.3.6.]
Procedure: LOAD procedure in SPIBILD
Purpose: Add multiple records in internal SPIRES format to a
subfile
Who can use it: file owner
users with See access to the file
containing the data, and Process access
to the file receiving the data
Basic Steps:
1. Create the input data set with GENERATE LOAD.
2. SPIBILD
3. Issue any SPIBILD special-processing commands desired.
4. Issue the LOAD command.
The LOAD procedure is useful when the records you want to add to a subfile are already in a SPIRES subfile with identical record characteristics. You may want to move records from the test version of a file to the production version. Or you might want to rebuild a file, using the records that are already in it.
You may want to compare this technique with the INPUT LOAD procedure, described in the next section, which works in SPIRES as well as SPIBILD [See 2.3.3a.] and with the FAST LOAD procedure in SPIBILD. [See 2.3.4.]
With this technique, you create an ORVYL data set containing the internal, stored form of the desired records. From there, the records can be loaded into another subfile in SPIBILD. This technique avoids the overhead of running the element values in the records through Outproc rules (as the data set is created) and Inproc and Inclose rules (as the data set is read back in), which can lead to substantial savings in processing time and costs.
The two record-types must share the same internal structure. That means that the way any given record would be stored in one must be identical to the way it would be stored in the other. One easy way to test the relationship between the two is to select each subfile and compare the displays from the SHOW ELEMENT CHARACTERISTICS command. The element names, displayed in the last column, do not have to match, but everything else shown should match, row for row. [Technically, there are some allowable differences besides element names. In theory, the element types do not have to match (except for the "Struc" (Structure) type). But such theories don't usually make good practice.] If they do not match, then the load will probably fail.
You must have See access or better to the file containing the data in order to use the GENERATE LOAD command, and you must have Process access to the receiving file in order to use the LOAD command.
Here are the steps of the procedure:
To create the load of records, you need to select the subfile containing the data in SPIRES.
Next, establish a Global FOR class containing the records you want. You must begin with a "FOR class" command, such as FOR SUBFILE, that may contain VIA and WHERE clauses. You may also issue SET SCAN commands and SET FILTER commands with "TYPE = SCAN" to further limit the records to be retrieved. [See the manuals "Sequential Record Processing in SPIRES: Global FOR" and "SPIRES Technical Notes" for more information about these commands.]
Next, issue the GENERATE LOAD command:
GENERATE LOAD loadname [REPLACE] [FIRST|*|NEXT|n|REST|LAST|ALL]
where "loadname" is the name of an ORVYL data set. The options at the end are the standard ones for a Global FOR command, specifying which records are to be processed. The default is ALL, for all the records in the established class. See Part 6, or online, [EXPLAIN GENERATE LOAD.]
Here, for instance, is a sample session that creates a load of all Stanford records in a subfile:
-> spires -Welcome to SPIRES-3 ... If in trouble, try 'HELP' -> select consortium members -> for subfile where university = stanford +> generate load stanford.members +>
The most likely problems you could have with GENERATE LOAD are these:
At this point, you could move the load data set to a tape using the ORVCOPY program. Later, when you are ready to load it into the receiving file, you restore it to an online ORVYL data set, again using ORVCOPY. [See 2.1.4.3.]
Note too that if you need to examine the records in the load at a later time, for example, to see the keys of the records there, you can use the FOR LOAD command in Global FOR to process them.
By the way, in step 4, you will need to know the name of the file and the name of the record-type into which the data is being loaded. If you do not know that information and are about to proceed to step 2, you should select the subfile you want to load the data into, and issue the SHOW SUBFILE INFORMATION command, which will tell you the name of the file, and the name of the goal record-type (listed under "Record") that you will specify in the LOAD command.
Issue the SPIBILD command online, or if you want to load the data via a BATWYL job, place it in the command stream.
+> spibild -Welcome to SPIBILD -?
If you are loading the data via a batch SPIBILD job, the program puts you in SPIBILD automatically, so you can omit this command.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add" message displayed for each new record. [See 2.1.5.3.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of records in the load of records. It is used for recovery purposes when a previous LOAD command failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
Note that you can use neither an input format nor an exception file with the LOAD procedure.
The LOAD command loads the data into the desired file:
LOAD loadname [RENUMBER] INTO record-name OF filename
where "loadname" is the name specified in the GENERATE LOAD command (see step 1) and "filename" and "record-name" are the names of the file and the particular record-type into which the data is to be loaded.
The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).
The LOAD command begins by processing the named file. Then it loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN LOAD COMMAND.]
Here is a continuation of the example started above:
-? set information messages = 0 -? clear file counts -? load stanford.members into rec01 of gq.jnk.su.chapter -Compute time: 0.199 seconds -Elapsed time: 6.767 seconds -Core usage: 0065/0933 -Begin passing of 49 records of record-type REC01 -Completed passing of 49 records total - Requests/Success: ADD 49 49 SUM 49 49 -End of BATCH/BUILD -Compute time: 3.012 seconds -Elapsed time: 18.798 seconds -Core usage: 0132/0933 -?
Note that the messages displayed from the LOAD command are identical to those from the BATCH command. The individual "add" messages for the records were suppressed in this example thanks to the SET MESSAGES command just prior to the LOAD command. [See 2.2.4 for information about the other messages shown.]
Below are described some problems that occasionally arise during LOAD processing.
An S311 error message indicates that the characteristics of the record-type into which you are trying to load the data don't match the characteristics of the record-type containing the original data. Though some minor differences may exist, such as differences in element names, the two record-types should have practically identical internal structures, as described at the start of this section. If they do not, the LOAD command will fail.
An S413 error message indicates that an attempt was made to load a record with a key that matches a record already in the record-type. LOAD works only with new records, not updates of existing ones. If this problem occurs with slot records, you can tell SPIBILD to ignore the provided keys of the incoming records and replace them with new slot numbers by adding the RENUMBER option to the LOAD command.
If the system crashes during the initial processing of the file, i.e., after you have issued the LOAD command but before the loading has begun, simply restart the procedure from step 2. Alternatively, you might want to use the PROCESS/RECOVER procedure to recover the file first, and then continue this procedure from step 2. [See 2.1.4.2.]
- you can start over again from step 2, ignoring the S413 errors you would get for the records previously added, [But if you used the RENUMBER option to tell SPIBILD to assign new slot keys to the records, you will not get any S413 errors when you restart the procedure; the same records will go in again.]
- you can determine how many records of the load made it into the subfile and then use the SET SKIP command in SPIBILD to tell SPIBILD how many records in the load to skip before starting the load again. [See 2.2.5.]
- you can determine which records made it into the subfile and then create a new load data set without those records (i.e., start again at step 1). You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. Alternatively, you could return to SPIRES, select the subfile and use various searching techniques to determine which was the last record to get into the new subfile.
Procedure: INPUT LOAD procedure
Purpose: Add multiple records in internal SPIRES format to a
subfile
Who can use it: file owner
users with See access to the file
containing the data, and Master access
to the file receiving the data
Basic Steps:
In SPIRES:
1. Create the input data set with GENERATE LOAD.
2. SELECT subfile.name
3. Issue any special-processing commands desired.
4. Issue the INPUT LOAD command.
5. Cleanup: Other considerations
In SPIBILD:
1. Create the input data set with GENERATE LOAD.
2. SPIBILD
3. ESTABLISH subfile.name
4. Issue any special-processing commands desired.
5. Issue the INPUT LOAD command.
The INPUT LOAD procedure is useful when the records you want to add to a subfile are already in a SPIRES subfile with identical record characteristics. You may want to move records from the test version of a file to the production version, or vice versa. Or you might want to rebuild a file, using the records that are already in it.
This technique is quite similar to the LOAD procedure in SPIBILD, but has these differences:
- It is available in SPIRES as well as SPIBILD, meaning it could be incorporated into a protocol, or used with temporary files.
- In SPIRES, it has the WITH FAST and WITH SEARCHONLY options, which speed up the processing considerably; they are not available to the LOAD procedure.
- It has an UPDATE option, whose effect is to make SPIRES treat the incoming records as "addupdates" rather than just "adds"; the LOAD procedure can handle only adds.
With this technique, you create an ORVYL data set containing the internal, stored form of the desired records. From there, the records can be loaded into another subfile in SPIRES or SPIBILD. This technique avoids the overhead of running the element values in the records through Outproc rules (as the data set is created) and Inproc and Inclose rules (as the data set is read back in), which can lead to substantial savings in processing time and costs.
The two record-types must share the same internal structure. That means that the way any given record would be stored in one must be identical to the way it would be stored in the other. One easy way to test the relationship between the two is to select each subfile and compare the displays from the SHOW ELEMENT CHARACTERISTICS command. The element names, displayed in the last column, do not have to match, but everything else shown should match, row for row. [Technically, there are some allowable differences besides element names. In theory, the element types do not have to match (except for the "Struc" (Structure) type). But such theories don't usually make good practice.] If they do not match, then the load will probably fail.
You must have See access or better to the file containing the data in order to use the GENERATE LOAD command, and you must have Process access to the receiving file in order to use the LOAD command.
This procedure is probably most commonly used in SPIRES because of the first two factors listed above that distinguish it from the LOAD procedure in SPIBILD. In SPIBILD, it is very similar to the LOAD procedure; the only significant difference is that INPUT LOAD has the ability to handle record updates as well as adds. A relatively insignificant difference is that with INPUT LOAD, you only need to know the name of the subfile into which you want to load the records; with LOAD, you must know the name of the file and the goal record-type.
In choosing between SPIRES and SPIBILD, remember that SPIRES will leave the records in the deferred queue for subsequent processing. SPIBILD will process the file, put the new records into the tree and leave the deferred queue empty.
If you choose to follow the procedure in SPIRES, here are the steps. (The steps for the SPIBILD procedure appear afterwards.)
To create the load of records, you need to select the subfile containing the data in SPIRES.
Next, establish a Global FOR class containing the records you want. You must begin with a "FOR class" command, such as FOR SUBFILE, that may contain VIA and WHERE clauses. You may also issue SET SCAN commands and SET FILTER commands with "TYPE = SCAN" to further limit the records to be retrieved. [See the manuals "Sequential Record Processing in SPIRES: Global FOR" and "SPIRES Technical Notes" for more information about these commands.]
Next, issue the GENERATE LOAD command:
GENERATE LOAD loadname [REPLACE] [FIRST|*|NEXT|n|REST|LAST|ALL]
where "loadname" is the name of an ORVYL data set. The options at the end are the standard ones for a Global FOR command, specifying which records are to be processed. The default is ALL, for all the records in the established class. See Part 6, or online [EXPLAIN GENERATE LOAD.]
Here, for instance, is a sample session that creates a load of all Stanford records in a subfile:
-> spires -Welcome to SPIRES-3 ... If in trouble, try 'HELP' -> select consortium members -> for subfile where university = stanford +> generate load stanford.members +>
The most likely problems you could have with GENERATE LOAD are these:
At this point, you could move the load data set to a tape using the ORVCOPY program. Later, when you are ready to load it into the receiving file, you restore it to an online ORVYL data set, again using ORVCOPY. [See 2.1.4.3.]
Note too that if you need to examine the records in the load at a later time, for example, to see the keys of the records there, you can use the FOR LOAD command in Global FOR to process them.
If you intend to use the WITH FAST option on the INPUT LOAD command, you may want to use the PROCESS/ESTABLISH procedure at this point to make sure the deferred queue is empty. [See 2.2.2.] You may also want to take whatever steps are needed to prevent any other users from trying to add records to the subfile at the same time as you are doing so -- WITH FAST requires and empty deferred queue and exclusive update access to the file, and users attempting to add records will be blocked. (This entire paragraph is relevant only under WITH FAST processing. See the description of the INPUT LOAD command in Part 6 for information about its value; online, [EXPLAIN INPUT LOAD COMMAND.] )
If you are continuing immediately from step 1, then you are already in SPIRES; but if not, issue the SPIRES command. Then select the target subfile, that is, the subfile into which you are loading the records.
+> select su chapter ->
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIRES displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add" or "update" message displayed for each new record. [See 2.1.5.3.]
- the SET SKIP command, which skips the specified number of records in the load of records. It is used for recovery purposes when a previous INPUT LOAD command failed during a batch input process. [See 2.3.6.]
Note that you can use neither an input format nor an exception file with the INPUT LOAD procedure.
The INPUT LOAD command loads the data into the desired file:
[WITH FAST|WITH SEARCHONLY] INPUT LOAD loadname [RENUMBER|UPDATE]
where "loadname" is the name specified in the GENERATE LOAD command.
The WITH FAST and WITH SEARCHONLY options request faster, more efficient processing of the data at the cost of you accepting several restrictions. Details are discussed in the reference section on INPUT LOAD in the back of this manual. [See Part 6, or EXPLAIN LOAD COMMAND for more information.]
The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).
By default, only new records are allowed to be loaded with the INPUT LOAD command -- if a record being loaded has the same key as a record already in the subfile, then the new one is rejected. The UPDATE option tells SPIRES to treat the new one as an update of the record already there, in essence turning the "add" into an "addupdate". That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN LOAD COMMAND.]
Here is a continuation of the example started above:
-> set information messages = 0 -> input load stanford.members renumber - Requests/Success: ADD 49 49 SUM 49 49 -End of load ->
Note that the messages displayed from the INPUT LOAD command are basically identical to those from the BATCH command. The individual "add" and "update" messages for the records were suppressed in this example, thanks to the SET MESSAGES command just prior to the INPUT LOAD command. [See 2.2.4 for information about the other messages shown.]
If you have problems with the INPUT LOAD command, see the discussion at the end of this section.
Once you are certain that all went well, you will probably want to discard the load file you generated in step 1.
If you used the WITH FAST option on INPUT LOAD in step 4, then if your file has immediate indexing, no further updates to the subfile will be allowed until the file is processed. You may want to process the file at this point to restore it to its normal state. [See 2.2.2.]
If you choose to follow the procedure in SPIBILD, here are the steps.
Follow the instructions under step 1 above of the SPIRES version of the procedure.
Issue the SPIBILD command online, or to load the data via a BATWYL job, place it in the command stream.
+> spibild -Welcome to SPIBILD -?
If you are loading the data via a batch SPIBILD job, the program puts you in SPIBILD automatically, so you can omit this command.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add" or "update" message displayed for each new record. [See 2.1.5.3.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of records in the load of records. It is used for recovery purposes when a previous INPUT LOAD command failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
Note that you can use neither an input format nor an exception file with the INPUT LOAD procedure.
You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT LOAD command will work properly.
The INPUT LOAD command loads the data into the desired file:
INPUT LOAD loadname [RENUMBER|UPDATE]
where "loadname" is the name specified in the GENERATE LOAD command.
The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).
By default, only new records are allowed to be loaded with the INPUT LOAD command -- if a record being loaded has the same key as a record already in the subfile, then the new one is rejected. The UPDATE option tells SPIBILD to treat the new one as an update of the record already there, in essence turning the "add" into an "addupdate".
The INPUT LOAD command loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN INPUT LOAD COMMAND.]
Here is a continuation of the example started above:
-? set information messages = 0 -? clear file counts -? establish su chapter -Compute time: 0.199 seconds -Elapsed time: 6.767 seconds -Core usage: 0065/0933 -? input load stanford.members -Begin passing of 49 records of record-type REC01 -Completed passing of 49 records total - Requests/Success: ADD 49 49 SUM 49 49 -End of BATCH/BUILD -Compute time: 3.012 seconds -Elapsed time: 18.798 seconds -Core usage: 0132/0933 -?
Note that the messages displayed from the INPUT LOAD command are identical to those from the BATCH command. The individual "add" or "update" messages for the records were suppressed in this example thanks to the SET MESSAGES command prior to the INPUT LOAD command. [See 2.2.4 for information about the other messages shown.]
Below are described some problems that occasionally arise during INPUT LOAD processing.
An S311 error message indicates that the characteristics of the record-type into which you are trying to load the data don't match the characteristics of the record-type containing the original data. Though some minor differences may exist, such as differences in element names, the two record-types should have practically identical internal structures, as described at the start of this section. If they do not, the INPUT LOAD command will fail.
An S413 error message indicates that an attempt was made to load a record with a key that matches a record already in the record-type. By default, INPUT LOAD works only with new records, not updates of existing ones. If this problem occurs with slot records, you can tell SPIRES to ignore the provided keys of the incoming records and replace them with new slot numbers by adding the RENUMBER option to the INPUT LOAD command. You may alternatively want to use the UPDATE option to tell SPIRES to use the load copy of the record as an update to the copy already in the file.
In SPIBILD, if the system crashes during the initial processing of the file, i.e., after you have issued the LOAD command but before the loading has begun, simply restart the procedure from step 2. Alternatively, you might want to use the PROCESS/RECOVER procedure to recover the file first, and then continue this procedure from step 2. [See 2.1.4.2.]
In SPIBILD, if the system crashes during the actual loading, records added before the last "-Begin passing" message that's not followed by a "-Completed passing..." message are in the subfile; those added after it are in the deferred queue but will be discarded during file recovery. You have three choices:
- you can start over again from step 2 of the procedure you were using, ignoring the S413 errors you would get for the records previously added. [But if you used the RENUMBER or UPDATE option to tell SPIBILD to assign new slot keys to the records, you will not get any S413 errors when you restart the procedure; the same records will go in again.]
- you can determine how many records of the load made it into the subfile and then use the SET SKIP command in SPIBILD to tell SPIBILD how many records in the load to skip before starting the load again. [See 2.2.5.]
- you can determine which records made it into the subfile and then create a new load data set without those records (i.e., start again at step 1). You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. Alternatively, you could return to SPIRES, select the subfile and use various searching techniques to determine which was the last record to get into the new subfile.
In SPIRES, if you included the WITH FAST option on the INPUT LOAD command, you may want to simply ZAP DEFQ and start again from step 2; the only records in the deferred queue are the ones you put there. In fact, even if you didn't use the WITH FAST option, it may be worth checking with SHOW SUBFILE TRANSACTIONS to see if yours are the only records in the deferred queue. (Remember though that other record-types in the file may have defq data too, which would be lost with a ZAP DEFQ command.)
Procedure: FAST BATCH or FAST LOAD
Purpose: Add multiple records to an empty subfile very
quickly, under special conditions
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Process the file with the PROCESS/ESTABLISH procedure.
3. Prepare the input data set.
4. SET NOPASS
5. Set the input format, if necessary.
6. Issue any SPIBILD special-processing commands desired.
7. Issue the FAST BATCH or FAST LOAD command.
8. Build the indexes, if any.
The FAST BATCH and FAST LOAD procedures provide the fastest "easy" method of getting new records into a subfile, at the loss of two features common to most other methods:
- FAST BATCH / FAST LOAD does not checkpoint the input data, meaning that the process must be completely redone in the event of a system failure; and
- FAST BATCH / FAST LOAD does not build indexes from the input data. That must be done separately, using the BUILD command.
These procedures are especially useful when you want to build a new goal record-type with thousands of records. They let you split the work into two separate pieces: building the goal record-type and building its indexes. This is critical in situations where the amount of time available in a single computer session is less than the amount needed to build and index the record-type at one time with the INPUT BATCH procedure in SPIBILD.
To use FAST BATCH or FAST LOAD, you must meet several other restrictions too:
- The record-type into which the data is being added must be completely empty, unless it is a slot record-type.
- The records in the input data must either be in ascending key order, or have no keys at all, letting SPIBILD assign them (e.g., slot keys). [If the record-type is a slot record-type and the input data contains slot key values, then the records do not have to be in ascending key order, but the input processing will be less efficient than if they were.]
- All input must be ADD requests; this process cannot handle updates or removes.
- For FAST LOAD, the input data must already be in the internal form of SPIRES records, in a file created by the GENERATE LOAD command; the record characteristics of the record-type the records came from must match those of the record-type into which you want to FAST LOAD them. [See 2.3.3 for further information about the "load" type of batch input in general.]
If you are using the FAST LOAD procedure, you need to prepare the data set containing the input. Follow step 1 in "LOAD Procedure in SPIBILD" to generate the load file. [See 2.3.3.]
Issue the SPIBILD command online or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job you are already in SPIBILD, so you omit this command from the command stream.
This step is only necessary if the file has other record-types in which data already exists. It is important to ensure that no passing between other record-types of the file would be attempted when the FAST BATCH command is executing.
Follow the PROCESS/ESTABLISH procedure described earlier in this manual. To execute its most simple form, simply issue the PROCESS command:
PROCESS [ORV.gg.uuu.]filename
where "gg.uuu" is the file owner's account, which must be specified if you're processing someone else's file. [See 2.2.2.]
For FAST LOAD, you can skip this step; your input is ready to use.
For FAST BATCH, if you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file.
The data may be in the standard SPIRES format, or it may be formatted in some other way, as long as you have an input format that can be used to read it. Always make certain the format works properly before trying to use it to batch hundreds of records into a subfile; a minor failure can be expensive when multiplied by hundreds of records. [See the manual "SPIRES Formats" for information about creating input formats.]
For FAST BATCH in batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set. If the data set is stored online, you may issue a USE command in the JCL at this point; otherwise, be sure to include a DD statement for the data set, to which you will refer in the FAST BATCH command later. [See 2.1.3.2.]
Unless you are planning to use the "ON ddname" option in batch SPIBILD, be sure that the input data set is in your active file at this point.
The records in the input data set should either be in order by key, or have their keys assigned by SPIRES as they enter the data base, e.g., slot keys. Records that are out of order will be rejected (with the exception of records with keys that are going into a slot record-type, which can be handled in any order).
You must specifically tell SPIBILD not to try to pass index data during the FAST BATCH / FAST LOAD process. The best way is to issue the SET NOPASS command in SPIBILD.
-? set nopass -?
Another method is to add the NOPASS statement to the linkage section for the record-type in the file definition and recompile it. When you are done with the FAST BATCH / FAST LOAD procedure, remove the statement from the file definition and recompile it again.
For FAST LOAD, skip this step.
For FAST BATCH, if the data will be read through a SPIRES input format, issue the SET FORMAT command at this point; otherwise, skip to step 6. Depending on what the format does, any of the commands listed below may be involved. All are optional; those you use should be issued in the order shown:
- ALLOCATE. This command allocates a vgroup, usually so that stored values can be restored to it (see RESTORE STATIC).
- RESTORE STATIC. This command restores previously stored values to a vgroup. [See "SPIRES Protocols" for more information about these two commands.]
- DEFINE AREA and ASSIGN AREA. These commands (as well as CLOSE AREA to be issued after the FAST BATCH command) are needed when the input format is also creating a report. [See 2.1.5.5.]
- SET FORMAT. The SET FORMAT command sets the named format, or, if the "*" form of the command is used, executes the startup frame of the format named in the previous SET FORMAT command. You must start with a "SET FORMAT format-name" command, with one or more SET FORMAT * commands optionally to follow.
SET FORMAT {format-name|*} [,parm|'parm'|"parm"]
- SET FTRACE. SET FTRACE enables format-tracing messages, most often used for debugging. [See "SPIRES Formats", section B.7.2.]
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET EXCEPTION command, to name an ORVYL data set into which unacceptable records from the input data set are placed (FAST BATCH only). [See 2.1.5.2.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the SET SKIP command, which skips the specified number of lines (FAST BATCH) or records (FAST LOAD) in the input file. Though normally used for recovery purposes, it can be used here if you simply want to skip a given number of lines or records of the input data set. For FAST BATCH, it is only useful when the input is in the standard SPIRES format. [See 2.3.6.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
- the SET PTRACE command, in one or more of its forms, to display tracing information for input or output processing rules (Inproc, Inclose, Outproc) executed during the record input processing.
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
The FAST BATCH command processes the file and batches the input data into the named subfile:
For online SPIBILD, and the BATWYL program: FAST BATCH subfile.name For batch SPIBILD: FAST BATCH 'subfile.name' ON ddname
The FAST LOAD command processes the file and loads the records from the named ORVYL data set into the named record-type of the named file:
FAST LOAD loadname [RENUMBER] INTO record-name OF filename
where "loadname" is the name specified in the GENERATE LOAD command (see Preliminary Work above) and "filename" and "record-name" are the names of the file and the particular record-type into which the data is to be loaded.
The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).
The FAST LOAD command begins by processing the named file. Then it loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN FAST LOAD COMMAND.]
Here is a sample session showing typical use of the FAST BATCH procedure:
Command> spibild -Welcome to SPIBILD -? process fish ... the file is processed ... -? use shad.to.add -? set nopass -? set exception bad.shad -? clear file counts -? set information messages = 0 -? fast batch shad specimens -Processing file: GQ.JNK.FISH -Compute time: 0.117 seconds -Elapsed time: 1.423 seconds -Core usage: 0065/0939 - Requests/Success: ADD 500 500 SUM 500 500 -End of BATCH/BUILD -Compute time: 0.850 seconds -Elapsed time: 3.108 seconds -Core usage: 0197/0939 -?
Notice how very little processing time (0.85 seconds) was involved to add 500 records; while the records that generated this example are admittedly small, the "compute time" needed was low because none of the backup work SPIBILD normally does to allow recovery was done here.
The FAST LOAD procedure would look exactly the same except for the USE command, which would not be there, and the FAST BATCH command, which would be replaced by a FAST LOAD command, such as:
-? load shad.to.add into rec01 of gq.jnk.fish
If the record-type has any indexes, you will need to "build" them at some point. The index-building procedure is described in detail later. [See 2.7.1, steps 4 through 7.] Remember if you build the indexes in the same SPIBILD session to "turn on" passing again, e.g., using the SET PASS command.
The example above continues, as we build the indexes.
-? set pass -? establish shad specimens -? for subfile -? build zin02, zin03, zin04 -Begin passing of 500 records of record-type REC01 -Complete passing of 500 records total -End of BATCH/BUILD -Compute time: 5.361 seconds -Elapsed time: 35.639 seconds -Core usage: 0214/0939 -?
The most likely problems you could have with FAST BATCH are these:
Most of the problems described above for FAST BATCH could happen for FAST LOAD as well, and are handled the same. Also, problems related to the load input data could arise; they are the same as the problems described in the previous section on the LOAD procedure. [See 2.3.3.] Remember, however, that restarting FAST LOAD will require you to zap the record-type, so that it will be empty when you start (unless you are working with records having slot keys). Alternatively, you may want to retain the records that did get in, build the indexes with what you have (step 8) and resume the load with the LOAD procedure, which does not require an empty record-type to start with.
Most of the procedures for adding records to a SPIRES subfile allow the input data to be formatted so that it can be read by either the standard SPIRES format or by a custom input format. Custom formats are discussed in detail in the manual "SPIRES Formats", in section C.9 in particular.
This section presents a summary of the standard SPIRES format, discussed in detail in the SPIRES manual "Searching and Updating", sections D.2.2, D.2.3, D.2.6 and D.8.1.
Keep in mind that some of the procedures discussed in this chapter (2.3) limit their scope to added records, to which some of this information may not apply.
The basic syntax of individual element values is:
element-name = value;
where "element-name" is the name of the element, and "value" is the data value to be assigned to that element. The equals sign is optional; the semicolon at the end is required.
Here are other important rules to remember:
COMMENT = SPIRES will put a blank after the comma,
so that the comma and 'so' don't get run together.;
NAME = Merrick Butte; COMMENTS = Breeder;
COMMENTS = "This semicolon; is not really needed.";
COMMENTS = "See ""SPIRES Formats"" for details.";
COMMENTS;
MEMBER;
NAME = Jenny Lake;
STATE = Wyoming;
MEMBER;
NAME = Mitchell Butte;
STATE = Utah;
ADD;
ADDUPDATE;
ADDMERGE;
UPDATE key;
MERGE key;
REMOVE key;
REMOVE 94025;
MERGE 85-194;
NAME = Mrs. Osgood Azamile;
;
REMOVE 83-019;
ADDUPDATE;
ID = 86-028;
NAME = Miss Bea Hayve;
;
When a SPIBILD command such as INPUT BATCH, LOAD or FAST BATCH terminates in the middle of adding new records to a subfile, you must decide how to recover from the problem and finish doing your input. The procedures described earlier in this chapter for those commands describe how to determine the cause of the problem and make some suggestions for how to solve it.
Often, the solution is to somehow determine which record was the last record in the input data to be successfully added to the subfile. Once that is determined, you can delete the input up through that record, or use the SET SKIP command to tell SPIBILD to skip that much input before starting the input procedure again. [Keep in mind for FAST BATCH or FAST LOAD that if you decide to use this method for recovery, you cannot use the FAST procedure when you resume adding records, since the goal record-type must be completely empty in order to use FAST BATCH or FAST LOAD. You can, however, switch to the INPUT BATCH or INPUT ADD or LOAD procedure, or, if not many records were successfully added anyway, consider zapping the record-type and starting over again with FAST BATCH or FAST LOAD. If you do switch to another procedure, build the indexes using the data already in the subfile before you batch more records in. [See 2.7.1, steps 4 through 7.]]
How you determine the last record to be successfully added depends on whether the records were being indexed or not. For instance, they would not be indexed if:
- you were using the FAST BATCH / FAST LOAD procedure in SPIBILD; or
- you had SET NOPASS; or
- the goal record-type had no indexes.
But if they are indexed, you can take advantage of a pair of handy commands, SHOW RECORD RECOVERY and SET SKIP. We'll start with the discussion for the indexed records, followed by the discussion for the non-indexed records.
SPIBILD keeps track of the number of lines of input read (for INPUT BATCH, INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE) or the number of records read (for LOAD), storing that number away each time a group of records have been completely processed, i.e., added to the tree and fully indexed. If the processing completely finishes, then SPIBILD discards the information. But if the processing is interrupted, you can issue the SHOW RECORD RECOVERY command to show you that number:
-> select albums -> show record recovery -SET SKIP 1593 -Goal record-type = 1, Key count = 2 894 895 ->
The significant number here is the SET SKIP value, the number of lines (or records) of input successfully processed. [The other information is useful in some ways. This information is about the records that are only half-processed, i.e., whose index data was not completely processed. During recovery, SPIBILD will discard those records and "unindex" them if any indexing was done. The data for those records appears in the input data after the number of lines given in the "skip" indicator so it will be input again when you use the SET SKIP command. See Part 6, or online, [EXPLAIN SHOW RECORD RECOVERY COMMAND.]]
When you are ready to resume the SPIBILD work, start with the RECOVER/PROCESS procedure. [See 2.1.4.2.] Then start again the procedure you were using when the problem occurred, incorporating the SET SKIP command at the point where it is mentioned. See Part 6 for more information, or online, [EXPLAIN SET SKIP COMMAND.]
This procedure is somewhat more complicated when the input for the INPUT BATCH command is a mixture of transaction types, rather than only new records to be added. SET SKIP only registers added records, so the number will not get updated when non-add transactions are being processed, even though they may indeed get completely indexed. The best way to handle this situation is to get the skip number using the SHOW RECORD RECOVERY command and then use the RECOVER/PROCESS procedure to clean up the file.
Next, examine the input data past the number of lines you can skip and, in SPIRES, use the indexes (or simply display records by key) to look for those records. When you find input data that has no matching record, you have found the point where the processing stopped. Then determine how many more lines past the skip number you should tell SPIBILD to skip to begin at that point.
Note too, in both cases, that you may also discard the input that succeeded rather than use SET SKIP to skip over it. SET SKIP is most handy when the input data is not yours to change or is on a tape.
If the data is not indexed, then SPIBILD does not keep track of the SET SKIP figure. But in fact, recovery is much simpler because there will be no half-processed records that SPIBILD must discard in order to clean up properly.
The solution here is to use the RECOVER/PROCESS procedure to do any file cleanup that might be necessary, and then return to SPIRES to examine the file to determine where the processing stopped. If you had information messages turned on during the processing that failed, you will know from them how far the processing got. If not, you will need to examine records from the input data set, looking for their counterparts in the file. When you can't find one, that is the point where the processing stopped.
You can either delete the input to that point, or determine how many lines of the input should be skipped to begin processing at that point, including that number in a SET SKIP command when you restart the procedure.
SPIRES and SPIBILD provide several ways for you to update multiple records in a subfile in a single operation. The different methods work under different circumstances -- which one you choose will depend on the particular situation you have.
Below is a comparative summary of the methods. The procedures involving SPIRES are described in the "SPIRES Searching and Updating" manual. SPIBILD's INPUT BATCH, INPUT ADDUPDATE and INPUT ADDMERGE procedures were described in the previous chapter, but the other SPIBILD procedures are discussed later in this chapter. This chapter also includes a section about how SPIBILD determines when an updated record requires re-indexing. [See 2.4.3.]
A discussion about the format of the data in input data sets used with these procedures also appeared in the previous chapter. [See 2.3.5.]
There are twelve techniques for updating multiple records in a subfile in one fell swoop:
- 1. (SPIRES) the INPUT BATCH procedure
- 2. (SPIRES) the INPUT MERGE procedure
- 3. (SPIRES) the INPUT ADDMERGE procedure
- 4. (SPIRES) the INPUT ADDUPDATE procedure
- 5. (SPIRES) the BATCH procedure
- 6. (SPIRES) the MERGE procedure in Global FOR
- 7. (SPIRES) the WITH UPDATE procedure
- 8. (SPIBILD) the INPUT BATCH procedure
- 9. (SPIBILD) the INPUT MERGE procedure
- 10. (SPIBILD) the INPUT ADDMERGE procedure
- 11. (SPIBILD) the INPUT ADDUPDATE procedure
- 12. (SPIBILD) the MERGE procedure in Global FOR
With so many choices, it can be challenging to decide which procedure to use. The following list of questions, along with the descriptions that follow, may help you make up your mind:
Below are descriptions of each of the techniques.
You can use the INPUT BATCH command to add new records, modify existing records and remove existing records. SPIRES places the data into the file's deferred queue. When updates, merges or removes are involved, the data must be in the standard SPIRES format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.
The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.
Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner (or the account that next processes the file), not the user doing the input, who pays for only the input costs.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.
See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES. For online help, EXPLAIN INPUT BATCH.
You use the INPUT MERGE procedure to merge data into existing records in the subfile. An input format that handles merge processing does most of the work, usually getting the data, including the key of each record to be updated, from an input data set. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.
SPIRES places each updated record in the deferred queue, so unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. Hence, the cost of file processing is not borne by users of the INPUT MERGE procedure.
See the SPIRES manual "Searching and Updating" for more information about the INPUT MERGE command in SPIRES. For online help, EXPLAIN INPUT MERGE.
The INPUT ADDMERGE and INPUT ADDUPDATE procedures are very similar to the INPUT ADD precedure, using the input to add new records to the subfile. A custom input format does most of the work, usually getting the data, including the key of each record to be added, from an input data set. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set. If SPIRES finds that the input key matches the key of a record already in the subfile, then the input is either used to replace that record (ADDUPDATE) or to be merged into that record (ADDMERGE).
SPIRES places each updated record in the deferred queue, so unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. Hence, the cost of file processing is not borne by users of these INPUT procedures.
See the SPIRES manual "Searching and Updating" for more information about the INPUT ADDMERGE and INPUT ADDUPDATE commands in SPIRES. For online help, EXPLAIN INPUT ADDMERGE or INPUT ADDUPDATE.
The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.
This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.
The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user doesn't know till the next day whether the records were accepted, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.
The input data must be in the standard SPIRES format. It can include adds, updates, merges and removes.
A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched; but the request telling JOBGEN where the data is gets thrown away. This problem makes the BATCH command in SPIRES quite unreliable.
For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".
In SPIRES, the MERGE command can update multiple records, determined by the "FOR class" command in effect. The input data may be provided in several ways:
- it may be in the standard SPIRES format in the active file, with the data read only once and applied to each record being merged;
- it may be formatted input in the active file, with the data read and applied to each record being merged;
- it may be requested from the user by the merge format, or it may be provided by the format itself.
As the MERGE command executes, it merges the data into each record, placing the updated version in the deferred queue. Unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. The cost of file processing is not borne by users of this procedure.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic MERGE command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN WITH FAST PREFIX.
See the manual "Sequential Processing in SPIRES: Global FOR" for more information about the MERGE command in Global FOR.
WITH UPDATE is a command prefix for the DISPLAY and TYPE commands. Using the WITH UPDATE procedure, you can, to a limited degree:
- display records, simultaneously updating them; or
- display records, simultaneously updating other records in other subfiles or record-types; or
- update a record, simultaneously updating other records in other subfiles or record-types.
The most noteworthy limits of the WITH UPDATE procedure are:
- You must use a custom format containing special code to handle the extraordinary updating.
- The user must have secure-switch 14 set.
- You cannot add new occurrences or remove old occurrences of elements in the records being updated; you can only replace existing occurrences, with values that are the same lengths as the values being replaced.
- Inproc rules are executed for the replacement values, but not INCLOSE rules.
For more information about this specialized technique, see the manual "SPIRES Formats", chapter C.13, or "EXPLAIN WITH UPDATE PREFIX, RESTRICTIONS" online.
You can use the INPUT BATCH command in SPIBILD to add new records, modify existing records and remove existing records. The INPUT BATCH command applies the data in the input data set (in the standard SPIRES format) directly to the tree. For merges and updates, the INPUT BATCH command usually puts the data from the input data set (in the standard SPIRES format) into the deferred queue, which is then processed after all the input has been read.
The data may include adds, updates, merges and removes. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the data involved.
The INPUT BATCH procedure in SPIBILD is more expensive than the similar INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.
The input data may be on disk or tape, though if on tape, the procedure must be done through a Batch SPIBILD job.
The steps of this procedure were described in detail in the previous chapter. [See 2.3.1 or EXPLAIN INPUT BATCH PROCEDURE IN SPIBILD online.]
You can use SPIBILD's INPUT MERGE procedure to merge data into existing records in the subfile. [See 2.4.1.] You must use an input format that handles merge processing. It does most of the work, usually getting the data, including the key of each record to be updated, from an input data set. The data to be read may come from your active file or a data set stored on disk or tape.
Processing the file is part of the INPUT MERGE procedure -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. You cannot use this procedure unless your account can process the file.
Like their counterparts in SPIRES, the INPUT ADDMERGE and INPUT ADDUPDATE procedures in SPIBILD are very similar to the INPUT ADD precedure, using the input to add new records to the subfile. A custom input format does most of the work, usually getting the data, including the key of each record to be added, from an input data set. The data to be read must be in your active file or, if you use batch SPIBILD, a data set stored on disk or tape. If SPIRES finds that the input key matches the key of a record already in the subfile, then the input is either used to replace that record (ADDUPDATE) or to be merged into that record (ADDMERGE).
Processing the file is part of these INPUT procedures -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. You can use these procedures only if your account can process the file.
The INPUT ADDMERGE and INPUT ADDUPDATE procedures are discussed along with the INPUT ADD procedure in the previous chapter. [See 2.3.2 or EXPLAIN INPUT ADD PROCEDURE IN SPIBILD online.]
This procedure is similar to the INPUT MERGE procedure described above, except that the records to be updated are determined by the Global FOR criteria specified, not by the input data. The input data usually consists of one or more values that are all to be merged into each record. The input data may be in the standard SPIRES format, or a custom input format may be used. [See 2.4.2.]
The data can come from your active file, or, if you are using batch SPIBILD, from a tape or another WYLBUR data set. There may even be no input data set at all -- a custom format can provide the data itself, or it may reference the existing record to make changes based on its current values (such as reducing all occurrences of a PRICE element by half).
Processing the file is part of this procedure too -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. To use this procedure, your account must be able to process the file.
Procedure: INPUT MERGE in SPIBILD
Purpose: Merge data into multiple records (determined by the
input) using a custom input format
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Prepare the input data set.
3. Issue any SPIBILD special-processing commands desired.
4. ESTABLISH subfile-name
5. Set the input format.
6. INPUT MERGE
The INPUT MERGE procedure merges data into records using a custom input format. The input data for each record must contain its key, so that SPIRES can retrieve the record for updating. [To merge data into records identified by criteria other than the key, use the MERGE procedure in Global FOR in SPIBILD. [See 2.4.2.]]
Typically, an input data set might contain a line for each record to be updated, containing the key and a new data value for one or more elements. For example, this could be an input data set for a mythical doctor's billing system, where the data in the first column are keys to records already in the subfile:
N1785 9/17/86 Payment $5.00 B7922 9/17/86 Office Visit $4.00 C0018 9/17/86 Payment $8.00
What happens to the input data is determined by the input format. It could be added as new data, or as replacement data, or as data that causes existing data to be removed or recomputed (the data above could cause a BALANCE element in each record to be refigured, for instance). Information on creating merge formats appears in chapters C.5 and C.9 of the manual "SPIRES Formats".
Be aware that you can use this procedure only if you have a custom merge format that reads each key and its associated data. If the data to be merged is in the standard SPIRES format, you should simply use SPIBILD's INPUT BATCH procedure. [See 2.3.1, 2.3.5.]
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL command stream.
If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set on disk.
Be sure the input data set can read the input data and apply it properly to individual SPIRES records. Always make certain the format works properly before trying to use it to merge hundreds of records in a subfile -- a small problem can be quite costly when multiplied by hundreds of records.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET EXCEPTION command, to name an ORVYL data set into which unacceptable records from the input data set are placed. [See 2.1.5.2.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of lines in the input file. It is used for recovery purposes when a previous SPIBILD task has failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. [See the description for that command in Part 6, or online, EXPLAIN SET STACK SIZE COMMAND.]
- the SET PTRACE command, in one or more of its forms, to display tracing information for input or output processing rules (Inproc, Inclose, Outproc) executed during the record input processing.
The ESTABLISH command in SPIBILD "selects" the named subfile and processes the file containing it.
For online SPIBILD, and the BATWYL program: ESTABLISH [&gg.uuu.filename] subfile-name For batch SPIBILD: ESTABLISH [&gg.uuu.filename] 'subfile-name'
You must set the desired format. Depending on what the format does, any of the commands listed below may be involved. All are optional except for SET FORMAT; those you use should be issued in the order shown:
- ALLOCATE. This command allocates a vgroup, usually so that stored values can be restored to it (see RESTORE STATIC).
- RESTORE STATIC. This command restores previously stored values to a vgroup. [See "SPIRES Protocols" for more information about these two commands.]
- DEFINE AREA and ASSIGN AREA. These commands (as well as CLOSE AREA to be issued after the FAST BATCH command) are needed when the input format is also creating a report. [See 2.1.5.5.]
- SET FORMAT. The SET FORMAT command sets the named format, or, if the "*" form of the command is used, executes the startup frame of the format named in the previous SET FORMAT command. You must start with a "SET FORMAT format-name" command, with one or more SET FORMAT * commands optionally to follow.
SET FORMAT {format-name|*} [,parm|'parm'|"parm"]
- SET FTRACE. SET FTRACE enables format-tracing messages, most often used for debugging. [See "SPIRES Formats", section B.7.2.]
The INPUT MERGE command in SPIBILD merges the input data into the subfile according to the specifications of the set format.
For online SPIBILD, and the BATWYL program: [USING frame] INPUT MERGE For batch SPIBILD: [FROM ddname] [USING frame] INPUT MERGE
Here is a sample online SPIBILD session showing typical usage of the INPUT MERGE procedure:
Command> spibild -Welcome to SPIBILD -? use grades.on.test6 -? set exception bad.grades replace -? set information messages = 0 -? establish student grades -Compute time: 0.146 seconds -Elapsed time: 1.502 seconds -Core usage: 0065/0933 -? input merge - Requests/Success: MER 462 462 SUM 462 462 -End of BATCH/BUILD -Passing: 462 records of record-type REC01 -Processed: 462 DEFQ records of record-type REC01 -Compute time: 5.762 seconds -Elapsed time: 28.122 seconds -Core usage: 0256/0933 -?
This example shows a case where the merge data caused indexes to be updated; in some cases, no index data is involved, so the records don't need to be re-indexed, which is somewhat cheaper. [See 2.4.3.1.]
The most common problem that occurs is that one or more records fail to get updated because an Inproc processing rule rejects the data. The solution is to find the rejected data, correct it, and try the procedure again with only that data in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected data there. [See 2.1.5.2.]
System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.
- simply restart the procedure using the same data. This is the simplest method, but don't use it if the merge requests are creating new element or structure occurrences, or, say, causing addition into amount elements. Completely resubmitting such requests would cause erroneous data in those records that were successfully processed the first time.
- determine the last record that was updated. SPIBILD's information messages -- if you didn't turn them off -- can be useful here, but you should check the records actually in the file in SPIRES. This might be tedious, but it is the most efficient method (it means you won't update the same records again). You can delete the data for the successful records from the front of the input data set and then restart the procedure. Alternatively, restart the procedure and include the SET SKIP command to tell SPIBILD how many lines of the input to skip in step 3.
Procedure: MERGE in SPIBILD with Global FOR
Purpose: Merge data into multiple records determined by
Global FOR
Who can use it: file owner
users with PROCESS access to the file
Basic Steps:
1) Create a stored stack or result, if appropriate.
2) SPIBILD
3) Prepare the input data set, if any.
4) Issue any SPIBILD special-processing commands desired.
5) ESTABLISH subfile.name
6) Set the input format, if any.
7) Establish the Global FOR environment.
8) Issue the MERGE command.
This MERGE procedure merges data into records determined by the established Global FOR environment. This contrasts with the "normal" MERGE procedure in SPIBILD, where the keys of the records to be updated are contained in the input data set. Under Global FOR, however, an input data set may or may not exist. If it does, it almost always contains pieces of data that are all to be merged into each record, with or without the aid of a custom format. If the input data set doesn't exist, then the data is supplied or computed by a custom format directly.
Information on writing merge formats to be used with Global FOR in SPIBILD appears in section C.9.4 of the manual "SPIRES Formats".
If you are merging into all the records in the subfile, skip ahead to step 2. In SPIBILD, you can use the FOR SUBFILE command with a WHERE clause to determine which records to update. However, it is much more efficient, if you can do this, to search for the records using indexes in SPIRES, store the result, and then use the FOR STORED command in SPIBILD to identify the records in the stored result. This technique also works for stored stacks that were created in SPIRES.
To do this, create your search result or stack in SPIRES, using commands like FIND, SEQUENCE and STACK. Then, depending on whether you have a result or a stack, issue the STORE RESULT.name or STORE STACK.name command to save the result or stack as an ORVYL data set. Don't use the TEMPORARY option on those commands; the result or stack won't be around long enough for SPIBILD to use. [If you have a result, there is a slim possibility that FOR STORED cannot work with it. To avoid this problem, you may want to turn the result into a stack (e.g., by issuing the commands FOR RESULT and STACK ALL) before storing it. See the technical information on the FOR STORED command in Chapter 6 of this manual; online, [EXPLAIN FOR STORED COMMAND.]]
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command.
If the data to be merged will be read from an input data set, prepare it for use. If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set on disk.
If you are using an input format to read the data, be sure it works properly before trying to use it to update hundreds of records in a subfile -- a small problem can be quite costly when multiplied by hundreds of records.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET PTRACE command, in one or more of its forms, to display tracing information for input or output processing rules (Inproc, Inclose, Outproc) executed during the record input processing.
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
Issue the ESTABLISH command, which processes the file and prepares the subfile for direct update operations.
For online SPIBILD, and the BATWYL program: ESTABLISH subfile.name For batch SPIBILD: ESTABLISH 'subfile.name'
If you are using an input format, you set it now; otherwise, skip to step 7. Depending on what the format does, any of the commands listed below may be involved. All are optional except for SET FORMAT; those you use should be issued in the order shown:
- ALLOCATE. This command allocates a vgroup, usually so that stored values can be restored to it (see RESTORE STATIC).
- RESTORE STATIC. This command restores previously stored values to a vgroup. [See "SPIRES Protocols" for more information about these two commands.]
- DEFINE AREA and ASSIGN AREA. These commands (as well as CLOSE AREA to be issued after the FAST BATCH command) are needed when the input format is also creating a report. [See 2.1.5.5.]
- SET FORMAT. The SET FORMAT command sets the named format, or, if the "*" form of the command is used, executes the startup frame of the format named in the previous SET FORMAT command. You must start with a "SET FORMAT format-name" command, with one or more SET FORMAT * commands optionally to follow.
SET FORMAT {format-name|*} [,parm|'parm'|"parm"]
- SET FTRACE. SET FTRACE enables format-tracing messages, most often used for debugging. [See "SPIRES Formats", section B.7.2.]
Establish the Global FOR environment by issuing the FOR SUBFILE or FOR STORED command, optionally followed by other Global FOR commands mentioned below. The purpose of these commands is to identify the specific records to be updated.
No VIA clause is allowed on the FOR SUBFILE command, but you may add a WHERE clause as needed, to limit the records to be updated using element criteria:
FOR SUBFILE [WHERE clause]
If you have stored a stack or result containing the records you want to update (see step 1), you would use the FOR STORED command:
FOR STORED {RESULT.name|STACK.name} [WHERE clause]
The SET SCAN commands can give you further flexibility (and savings) in determining which records are to be examined for processing. The SET SCAN commands allowed in SPIBILD under FOR SUBFILE are:
SET SCAN BACKWARD SET SCAN LIMIT n SET SCAN PREFIX string SET SCAN START string SET SCAN STOP string
Only SET SCAN LIMIT is allowed under FOR STORED. Note that SET SCAN PREFIX has no meaning with slot subfiles.
Details about these commands, including WHERE clauses, appear in the manual "Sequential Processing in SPIRES: Global FOR".
The SET FILTER command, with the TYPE = SCAN option, works similarly to a WHERE clause but is more versatile. You can, for example, use it to locate records when you have complicated criteria involving elements within structures, a situation that WHERE clauses often cannot handle. See section 4.4.2 of the Global FOR manual referenced above; online [EXPLAIN FILTERS, ON WHERE CLAUSES.]
The SKIP command tells SPIBILD to skip the specified number of records meeting the Global FOR criteria:
SKIP [n|FIRST]
where "n" is the number of records to be skipped. If you specify neither "n" nor FIRST (for the FIRST record), the default is FIRST. Except for the limited options, SKIP works the same in SPIBILD as it does in SPIRES. See section 5.10 of the Global FOR manual; online [EXPLAIN SKIP COMMAND.]
This example shows how a WHERE clause and SET SCAN processing may be used to identify the records to be updated in the least expensive manner. Suppose you want to update all the records with ADD.DATE element values of 1986 whose keys begin with the code KL:
-? for subfile where add.date 1986 -? set scan prefix kl -?
These two commands work more efficiently together than would the single command FOR SUBFILE WHERE ADD.DATE 1986 AND ID PREFIX KL. The SET SCAN command limits the records that are examined to those whose "id" begins with KL; but with the single command, SPIBILD would examine all records, looking for those whose "id" begins with KL, which might be hundreds or thousands more records. If you will be using MERGE processing under Global FOR in SPIBILD, you should be familiar with the SET SCAN commands, which can save you a great deal in processing costs.
Note in the example that SPIBILD does not use the Global FOR prompt that SPIRES uses (+?) after you issue the FOR SUBFILE command.
The MERGE command in SPIBILD merges the input data into the named subfile according to the specifications of the set format.
MERGE [ALL|NEXT|n|REST] [FROM {ACTIVE|ddname}]
Again, "n" represents the number of records you want processed; the default is REST. You need to specify the FROM option if you want SPIBILD to read data from the active file (or from a DD data set in batch SPIBILD). But if the input data is being provided by an input format that does not need to read the data from the active file, you should omit the FROM option.
Here is a sample online SPIBILD session showing typical usage of the MERGE procedure in SPIBILD:
Command> list
1. manager = Sandy Claws;
Command> spibild
-Welcome to SPIBILD
-? set information messages = 0
-? establish personnel
-Compute time: 0.138 seconds
-Elapsed time: 1.638 seconds
-Core usage: 0065/0933
-? for subfile where supervisor = ebenezer scrootch
-? merge all from active
- Requests/Success:
MER 24 24
SUM 24 24
-End of BATCH/BUILD
-Passing: 24 records of record-type EMPL01
-Processed: 24 DEFQ records of record-type EMPL01
-Compute time: 2.119 seconds
-Elapsed time: 8.152 seconds
-Core usage: 0256/0933
-?
This example shows a case where the merge data caused indexes to be updated; in some cases, no index data is involved, so the records don't need to be re-indexed, which is somewhat cheaper. [See 2.4.3.1.]
Because this procedure usually applies the same data to all the records it updates, if the data fails to go into one record, it usually fails to go into all of them. In most cases where input failure occurs, you just correct the input data and restart the procedure.
System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.
- simply restart the procedure using the same data. This is the simplest method, but don't use it if the merge requests are creating new element or structure occurrences, or, say, causing addition into amount elements. Completely resubmitting such requests would cause erroneous data in those records that were successfully processed the first time.
- determine the last record that was updated. SPIBILD's information messages -- if you didn't turn them off -- can be useful here, but you should check the records actually in the file in SPIRES. This might be tedious, but it is the most efficient method (it means you won't update the same records again). You should issue the SKIP command (see above) to skip the successful records when you restart the procedure.
- Since SKIP works only with FOR SUBFILE and not FOR STORED, with FOR STORED processing, you might want to recreate your result or stack in SPIRES, eliminating the successfully-updated records, store the new result or stack, and restart the procedure using that one.
In most situations where you are updating multiple records in a single pass, the distinctions between the procedures described at the beginning of this chapter are adequate for choosing the right one. And in terms of cost, for smaller files the differences between the procedures are seldom very significant.
For large files or files with somewhat unusual characteristics, however, the differences can be quite significant. Moreover, some extra thought and work on your part may lead to considerable savings that SPIBILD would not provide you on its own.
The situations covered in the subsections of this section require fairly technical discussions, and this section should be considered optional reading unless:
- you are concerned about cutting costs any way you can; and
- at least one of the following is true:
- you are working with a file containing thousands of records;
- you will be updating hundreds of records at a time;
- you are working with extremely large records, containing hundreds of element values, many of which are indexed.
Techniques are available that alter the standard way SPIBILD processes data, to increase efficiency when the input data or the file is different from the norm. The techniques, described in separate sections that follow, are:
- 1) the SET NOPASS command, useful when merging data in SPIBILD that has no effect on any index records; [See 2.4.3.2.]
- 2) the PASS-ELEM statement in file definitions, which has a similar use but which is built into the file's characteristics; [See 2.4.3.3.]
The first subsection provides a brief overview of how updated records affect indexes, which is useful information to know if you must decide whether to use any of these efficiency techniques.
In SPIRES, whether records are updated or merged, they are placed in the deferred queue as "updates" -- that is, SPIRES places the complete updated record in the deferred queue, so that it can replace the existing tree copy of the record. Merge data by itself is not stored in the deferred queue; SPIRES applies it to the record immediately, and places the updated record in the deferred queue as an "update".
SPIBILD has different ways to handle merge and update requests, i.e., when the data is being merged by BATCH or MERGE commands in SPIBILD. For updates, which are handled by the BATCH command only, SPIBILD places each updated record in the deferred queue. As soon as SPIBILD finishes with the input data, it processes the deferred queue again, moving the updated records into the tree (removing the old copies) and determining what changes need to be made to the indexes. [The technical details of how SPIBILD efficiently re-indexes updated records were presented earlier in this manual. In brief, the index data is generated for the old record, each item marked as a "delete", and the index data is generated for the new copy, with each item marked as an "add". SPIBILD sorts the index data, discarding any pairs that are identical except for opposite "add" and "delete" markers. What isn't discarded then gets applied to the indexes. [See 2.2.6.]]
For updates, SPIBILD needs to check the data to see if the indexes need to be changed because it doesn't know what elements, if any, have been changed from one copy of the record to the next; all it knows is that it has a new copy of the record to replace the old one. [When SPIBILD gets an updated record, it is a new copy of the existing record, which may or may not have been changed. SPIBILD assumes the record has been changed, and replaces the old copy with the new one in the tree. During indexing, however, SPIBILD determines which indexed values have not changed from the old record to the new, and discards that data from the pass stack, as described in the previous note.]
For merges, the circumstances are different: SPIBILD knows exactly what elements are being updated, since it is doing the work. Hence, it may determine that the only elements being updated by a merge request are non-indexed ones, so the merged record doesn't need to go to the deferred queue for re-indexing; instead, it can go directly to the tree. SPIBILD can do this for goal record-types that have a "pass-elements list", created when the file is compiled, which identifies the elements that affect indexes -- if the merge request doesn't change any elements on the list, there's no need to re-index the record.
On the other hand, if the merge request does change an element in the pass-elements list, the record goes to the deferred queue for re-indexing. Naturally, the trip to the deferred queue means more work for SPIBILD (and more cost to you) since all the re-indexing work must be done.
These facts lead to several conclusions:
- In SPIBILD, it's more efficient to merge data that doesn't affect indexes than data that does.
- If you are updating records with data that doesn't affect indexes, the total processing cost (i.e., to get the data into the tree) will be less if you:
- use the SET NOPASS command to turn off index passing; or
- merge the data in SPIBILD, taking advantage of the pass-elements list, rather than update the records in SPIRES or SPIBILD.
The next two sections will discuss the two methods mentioned above:
When updated records in the deferred queue are processed, SPIBILD must generate two sets of index data for each record: the data for the tree copy, to be deleted, and the data for the new copy, to be added. To avoid unnecessary work, SPIBILD sorts them together, so that pairs that are identical except for the "add" or "delete" designation are discarded -- those pairs indicate element values that were not changed, which thus don't need to be re-indexed.
If a record is updated but no changes are made to the elements that affect indexes, then SPIBILD will create all the index data and subsequently discard it. This superfluous work may be worth avoiding in applications where non-indexed data is updated regularly.
As described in the next section, on pass-element lists, SPIBILD tries to avoid this unnecessary work in the case of merges done in SPIBILD (though without some intervention on the file owner's part, SPIBILD may not be able to avoid it).
The SET NOPASS command is a fairly simple command you can use for updates and merges no matter whether you do them in SPIRES or SPIBILD -- it guarantees the extra work won't be done. If you are merging records in SPIBILD, using SET NOPASS can save you the trouble of determining whether a pass-elements list is in effect.
The technique is to add the SET NOPASS command to the SPIBILD procedure you are using, at the time prescribed in the procedure's description. For instance, if you are processing a file in SPIBILD after updating records in SPIRES, you issue the SET NOPASS command before the PROCESS command in SPIBILD. [See 2.2.2.]
WARNING: Do not use this technique unless you are absolutely certain that no data in the deferred queue affects any indexes. If it does, and you use the SET NOPASS command to turn off passing, then data integrity problems (discrepancies between the goal records and the indexes) will occur. If you aren't certain, or if you can't control the update procedure (including file processing) from start to finish, do not use the SET NOPASS command.
You might want to process the file first, before you apply any of the updates, in order to clear any other data from the deferred queue. In that case, don't issue the SET NOPASS command until you are ready to process the new data -- you want the data already in the deferred queue to be processed normally.
Merged record processing in SPIBILD can be more efficient if SPIBILD finds a "pass-elements list", a list of the elements that affect indexing. When only non-indexed elements are merged, SPIBILD can put the record directly into the tree, rather than putting it in the deferred queue for indexing later.
Unfortunately for many subfiles, SPIRES can't always create a pass-elements list on its own, and if you don't help it along, it won't create one. When a subfile doesn't have one, SPIBILD has no idea whether any given element will affect an index, and so must place all merged records back into the deferred queue. To understand why this is true, we need to know more about pass-elements lists.
When it compiles a file definition, SPIRES creates the pass-elements list from the elements it knows are being passed to indexes. Specifically, SPIRES places the following elements into the pass-elements list:
- elements named in the GOALREC-NAME statement; and
- elements named in any of the following Passproc rules:
- $PASS.ELEM proc (A167)
- $PASS.UPPER proc (A162)
- $PASS.BY proc (A163)
- $PASS.COND proc (A168)
SPIRES only builds this list if it is sure that no other elements in the record-type can affect the index values. If such an element existed and if it were changed in a SPIBILD merge request but no elements in the pass-elements list were changed, then the related index values would not be updated, since SPIBILD would only re-index the record if an element on the pass-elements list had been merged too.
SPIRES doesn't build the list if any of the above-listed elements:
- is a virtual element; or
- has the $CALL proc (A62) in its Passproc string; or
- is passed in its output form, i.e., is processed through its Outproc string before being passed.
If SPIRES doesn't build the list, then all merged records handled by SPIBILD are placed in the deferred queue for index processing. This certainly isn't a tragedy, but obviously, if no indexed elements are affected by the merge data, then SPIBILD's having to regenerate index information is extra and unnecessary work. In a large application, such inefficiency could be quite expensive.
You can tell SPIRES to create the pass-elements list anyway, even if the circumstances that would prevent its creation occur, by adding the PASS-ELEM statement to the file definition, at the end of the appropriate linkage section:
PASS-ELEM; or PASS-ELEM = element.list;
You would want to specify in the element list any elements not included in the list as described above that SPIRES should add to the list -- that is, elements whose values affect the passed values of other elements.
The PASS-ELEM statement cannot be added with File Definer; you must change the file definition directly. Select the FILEDEF subfile, retrieve the current copy of your file definition, and add the PASS-ELEM statement as described above to the appropriate linkage section. Then update the record and recompile the file definition in SPIRES.
Here are some examples describing how you might use pass-element lists. Suppose that the only reason SPIRES won't create a pass-element list for your subfile is because one of the elements you are passing is a virtual element but whose value has nothing to do with any other element. You could tell SPIRES to create the list anyway, just by adding "PASS-ELEM;" to the file definition.
Or suppose, for another example, that the indexed virtual element SHIPPING.DATE is created by adding 10 days to the value of the non-indexed element ORDER.DATE. If you code in the file definition:
PASS-ELEM = ORDER.DATE;
then SPIRES will create a pass-elements list, including ORDER.DATE.
WARNING: Don't code the PASS-ELEM statement unless you're sure that either:
- all elements that have any effect on indexes are listed in the PASS-ELEM statement, or will be placed in the list automatically, as described above; or,
- you will never use SPIBILD to merge any elements not on the list that are supposed to affect indexed values. If you do, the indexes won't get updated properly, and data integrity problems will result.
Removing records from a subfile in a single operation is a simple task. The SPIRES and SPIBILD procedures that remove records are easy to use, and much less complex than those for adding or updating records. There are several different methods, each of which shines under separate circumstances.
Below is a set of questions and answers to help you decide which technique to use, followed by a comparative summary of the methods. Two of the techniques are described in detail in this chapter. The others have been described either in Chapter 2.3, or in other SPIRES manuals.
There are seven ways to remove multiple records in a single operation:
- 1. (SPIRES) the REMOVE procedure in Global FOR
- 2. (SPIRES) the INPUT BATCH procedure
- 3. (SPIRES) the BATCH procedure
- 4. (SPIBILD) the REMOVE procedure in Global FOR
- 5. (SPIBILD) the INPUT BATCH procedure
- 6. (SPIBILD) the ZAP RECORD-TYPE procedure
- 7. (SPIRES) the ZAP FILE procedure
Here are some questions to help you determine which method to use, followed by descriptions of each.
- If you have a set of the keys of records to be removed, use the INPUT BATCH or BATCH procedures.
- If you are using indexed criteria, use the REMOVE procedure in SPIRES, in conjunction with the FOR RESULT command.
- If you are using non-indexed criteria, use the REMOVE procedure in Global FOR in either SPIRES or SPIBILD.
You can use the REMOVE command in SPIRES with any Global FOR class, such as SUBFILE, RESULT and STACK. That means you can remove any and all records in the given class. This procedure is available to any user with update privileges to the subfile.
SPIRES places a removal request for each record into the deferred queue; during subsequent file processing, the records will be removed, and any necessary "un-indexing" will be done. Hence, the indexes aren't updated (except for any immediate indexes) at the time the REMOVE command is issued. File processing costs are picked up by the file owner (or the next account to process the file), not by the user removing the records.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic REMOVE command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.)
See the manual "Sequential Record Processing in SPIRES: Global FOR" for more information about the REMOVE command in Global FOR; online, EXPLAIN REMOVE COMMAND, IN GLOBAL FOR. Information about these two prefixes also appears in Part 6 of this manual, under INPUT BATCH, with which they work the same way in SPIRES.
You can use the INPUT BATCH command to add new records, modify existing records and remove existing records. SPIRES places the data into the file's deferred queue. When updates, merges or removes are involved, the data must be in the standard SPIRES format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.
The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.
Because the removal requests go into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records aren't removed from the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner (or the account that next processes the file), not the user removing the records, who pays for only the input costs.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.
See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES. For online help, EXPLAIN INPUT BATCH.
The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). The input data must be in the standard SPIRES format. It can include adds, updates, merges and removes.
During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile, discarding records that are to be removed. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.
This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.
The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user doesn't know till the next day whether the records were removed, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.
A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched; but the request telling JOBGEN where the data is gets thrown away. This problem makes the BATCH command in SPIRES quite unreliable.
For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".
This procedure is similar to its cousin in SPIRES in that you establish a Global FOR class and then issue a REMOVE command to remove the records in that class.
The Global FOR class can be only SUBFILE or STORED in SPIBILD. (STORED, available in SPIBILD only, means that the records to be examined are in a stored stack or result.) You may include a WHERE clause on the FOR SUBFILE or FOR STORED command, and you may also use the SET FILTER and SET SCAN commands to limit the records removed.
If you are removing only a few of many records in a subfile, using REMOVE under Global FOR in SPIRES is probably more efficient than using it in SPIBILD. On the other hand, the REMOVE command in SPIBILD removes the records from the subfile immediately, unindexing them immediately as well. The user pays for all the processing, including the processing of any data already in the deferred queue. Because the file gets processed, only users with Process access to the file are allowed to use the REMOVE command in SPIBILD. [See 2.5.1.]
You can use the INPUT BATCH command in SPIBILD to add new records, modify existing records and remove existing records. For removes, the INPUT BATCH command removes the records identified by key directly from the tree.
The INPUT BATCH procedure in SPIBILD is more expensive than the similar INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.
The input data may be on disk or tape, though if on tape, the procedure must be done through a Batch SPIBILD job.
The steps of this procedure were described earlier. [See 2.3.1.]
The ZAP RECORD-TYPE command discards all the data in a specified record-type of a file. The ZAP FILE command erases all the data sets of an entire file. Though these sound like the ultimate solutions for removing records from a subfile, they are the best methods to use when you want to remove all the records from a subfile.
Moreover, they are worth considering in situations where you aren't removing all the records of a large subfile but are removing a large majority. In particular, consider saving the few records you want to keep, zapping the record-type (or the file) and then adding the kept records back in to a completely empty record-type.
Several benefits may come from this method, including the release of unneeded blocks for storage, and the savings from not "unindexing" all the records you're removing -- it costs as much to remove a record as to add it, because it takes SPIBILD the same amount of work to unindex a record as to index it in the first place.
Only the file owner can use the ZAP FILE procedure. [See 2.11.] Users with Master access to the file can use the ZAP RECORD-TYPE procedure. [See 2.5.2.]
Procedure: REMOVE in SPIBILD with Global FOR
Purpose: Remove multiple records (determined by Global FOR)
from a subfile
Who can use it: file owner
users with PROCESS access to the file
Basic Steps:
1) Create a stored stack or result, if appropriate.
2) SPIBILD
3) Issue any SPIBILD special-processing commands desired.
4) ESTABLISH subfile.name
5) Establish the Global FOR environment.
6) Issue the REMOVE command.
The REMOVE procedure removes records determined by the established Global FOR environment. The Global FOR environment in SPIBILD is not as flexible as the one in SPIRES -- the only "FOR class" commands available are FOR SUBFILE and FOR STORED.
If you have a search result or stack of records, or can use some other Global FOR class of records in SPIRES, you can remove them using that class in SPIRES, and then process the file if you want. Or, to avoid the extra work of putting the removal requests into the deferred queue, you can store the search result or stack, and then use the REMOVE command under FOR STORED in SPIBILD to remove them, which also processes the file and updates the indexes appropriately.
Either method will almost certainly be cheaper than using a WHERE clause on a FOR SUBFILE command in SPIBILD, especially if the subfile is large. But if identifying the records via indexes isn't available, this other method will do.
Here are the steps of the procedure:
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
Issue the ESTABLISH command, which processes the file and prepares the subfile for direct update operations.
For online SPIBILD, and the BATWYL program: ESTABLISH subfile.name For batch SPIBILD: ESTABLISH 'subfile.name'
Establish the Global FOR environment by issuing the FOR SUBFILE or FOR STORED command, optionally followed by other Global FOR commands mentioned below. The purpose of these commands is to identify the specific records to be removed.
No VIA clause is allowed on the FOR SUBFILE command, but you may add a WHERE clause as needed, to limit the records to be updated using element criteria:
FOR SUBFILE [WHERE clause]
If you have stored a stack or result containing the records you want to update (see step 1), you would use the FOR STORED command:
FOR STORED {RESULT.name|STACK.name} [WHERE clause]
The SET SCAN commands can give you further flexibility (and savings) in determining which records are to be examined for processing. The SET SCAN commands allowed in SPIBILD are:
SET SCAN BACKWARD SET SCAN LIMIT n SET SCAN PREFIX string SET SCAN START string SET SCAN STOP string
Only SET SCAN LIMIT is allowed under FOR STORED. Note that SET SCAN PREFIX has no meaning with slot subfiles.
Details about these commands, including WHERE clauses, appear in the manual "Sequential Processing in SPIRES: Global FOR".
The SET FILTER command, with the TYPE = SCAN option, works similarly to a WHERE clause but is more versatile. You can, for example, use it to locate records when you have complicated criteria involving elements within structures, a situation that WHERE clauses often cannot handle. See section 4.4.2 of the Global FOR manual referenced above; online [EXPLAIN FILTERS, ON WHERE CLAUSES.]
The SKIP command tells SPIBILD to skip the specified number of records meeting the Global FOR criteria:
SKIP [n|FIRST]
where "n" is the number of records to be skipped. If you specify neither "n" nor FIRST (for the FIRST record), the default is FIRST. Except for the limited options, SKIP works the same in SPIBILD as it does in SPIRES. See section 5.10 of the Global FOR manual; online [EXPLAIN SKIP COMMAND.]
This example shows how to use a WHERE clause and SET SCAN processing to identify the records to be removed in the least expensive manner. Suppose you want to remove all the records with MOD.DATE element values before 1986. You also know that all those records have slot keys less than 20000:
-? for subfile where mod.date < 1986 -? set scan stop 20000 -?
These two commands work more efficiently together than would the single command FOR SUBFILE WHERE MOD.DATE < 1986 AND ID < 20000. The SET SCAN command limits the records that are examined to those whose "id" is less than 20000; under only the FOR SUBFILE command, SPIBILD would examine all records, looking for those whose "id" is less than 20000, which might be hundreds or thousands more records. If you will be processing records under Global FOR in SPIBILD, you should be familiar with the SET SCAN commands, which can save you a great deal in processing costs.
Note in the example that SPIBILD does not use the Global FOR prompt that SPIRES uses (+?) after you issue the FOR SUBFILE command.
The REMOVE command in SPIBILD removes the records determined by the Global FOR class.
REMOVE [ALL|NEXT|n|REST]
Here "n" represents the number of records you want processed; the default is REST.
The REMOVE command first locates each record to be removed and places a "remove request" in the deferred queue for it, just as the REMOVE command does in SPIRES. When the Global FOR traversal is complete, SPIBILD processes the deferred queue again, which is when the records are actually removed and the indexes updated.
Here is a sample online SPIBILD session showing typical use of the REMOVE procedure:
-> spibild -Welcome to SPIBILD -? set information messages = 0 -? establish test inquiries -Processing file: GQ.JNK.TEST.INQUIRIES -Compute time: 0.264 seconds -Elapsed time: 3.214 seconds -Core usage: 0065/0933 -? for subfile -? set scan stop 100 -? remove all - Requests/Success: REM 100 100 SUM 100 100 -End of BATCH/BUILD -Passing: 100 records of record-type REC01 -Processed: 100 DEFQ records of record-type REC01 -Compute time: 6.416 seconds -Elapsed time: 19.526 seconds -Core usage: 0256/0933 -?
This example shows a case where all records whose keys are no greater than 100 are removed. The removal requests go into the deferred queue first; the "End of BATCH/BUILD" message indicates the start of the deferred queue processing, when the records will actually be removed.
The simplest way to recover from system failure is to restart the procedure. But there are two situations in which alternate methods might be cheaper to do:
1. If SPIBILD had printed the "End of BATCH/BUILD" message before the system failure, then SPIBILD had finished placing all the removal requests in the deferred queue and was starting to process the deferred queue to complete the job. To recover, you should use the PROCESS/RECOVER procedure described earlier in this manual. [See 2.1.4.2.]
2. If the REMOVE command under FOR SUBFILE had already removed many records before the failure occurred, you might want to consider adding a SET SCAN START command to the procedure. That way, SPIBILD won't have to re-examine all the records it examined the first time that weren't discarded.
It is easy to find out what record was the last one removed:
- 1) select the subfile in SPIRES
- 2) issue the FOR REMOVES command
- 3) issue the SKIP LAST command
- 4) issue the "SHOW KEY *" command
The key value shown will be the last one removed; use that as the start value in the SET SCAN START command.
On the other hand, if you were working under FOR STORED, you would probably resume by restarting the procedure. It will do no harm for SPIBILD to try removing records that are already removed.
Procedures: ZAP RECORD-TYPE
ZAP DATA SET
Purpose: Remove all records from a record-type
Who can use it: file owner
users with Master access to the file
Basic steps:
1) SPIBILD
2) ESTABLISH subfile-name
3) ZAP RECORD-TYPE recname (or ZAP DATA SET dataset.name)
The ZAP RECORD-TYPE procedure erases all the records in a record-type very quickly and very efficiently. If you think of a SPIRES file as being a file cabinet, the ZAP DATA SET or ZAP RECORD-TYPE commands are equivalent to tossing out a drawerful of file folders. In contrast, using the REMOVE ALL command under Global FOR in SPIRES or SPIBILD is akin to pulling out the folders one by one, checking to see if each one is indexed in another drawer of the cabinet and deleting that index entry too.
You will need to choose between the ZAP RECORD-TYPE and ZAP DATA SET commands when executing this procedure. Generally speaking, use the ZAP RECORD-TYPE command; however, choose the ZAP DATA SET command if the record-type is combined with other record-types in a single ORVYL data set, but you want to discard all of them.
The discussion of "Preliminaries" below will tell you how to determine whether the record-type is combined with others or not.
The ZAP RECORD-TYPE procedure pays no heed to indexes -- it simply discards all the data in the record-type or data set you name. But in most cases when you are removing all the records in a goal record-type, either it has no indexes or the indexes are in their own record-types, not connected to any other goal record-type, and they too can be erased with the ZAP DATA SET procedure. (Of course, if you are discarding all the record-types in the file, the ZAP FILE procedure [See 2.5.3.] is even better.) Unconditionally emptying a goal record-type and its associated index record-types will be much more efficient than having SPIBILD remove the goal records one by one, creating and applying pass stack data.
Before beginning this procedure:
- The name of the file is the key of the file definition. If you aren't the file owner, you can get the name by selecting a subfile of the file in SPIRES and issuing the SHOW SUBFILE INFORMATION command.
- The name of the record-type appears in the file definition. Also, if the record-type is a goal record-type of a subfile, you can get the information in SPIRES by selecting the subfile and issuing the SHOW SUBFILE INFORMATION command.
- The SHOW FILE ACTIVITY command in SPIRES can tell you what data set a given record-type is in. The data set name is "REC" followed by the integer (from 1 to 9) in the "DS" column of the display.
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
Issue the ESTABLISH command so that SPIBILD will know which file to use in subsequent commands, and so that SPIBILD will process the deferred queue.
For online SPIBILD and the BATWYL program: ESTABLISH subfile-name For batch SPIBILD: ESTABLISH 'subfile-name'
The deferred queue must be empty for the ZAP DATA SET and ZAP RECORD-TYPE commands to work. (If the deferred queue contains worthless data, you might want to consider issuing the ZAP DEFQ command in SPIRES as a preliminary step. That would save you the expense of processing the deferred queue.)
Specify the name of any subfile in the file containing the record-type or data set.
Issue the command you chose to use, based on the information given earlier; either:
ZAP RECORD-TYPE recname
where "recname" is the name of the record-type to be erased; or
ZAP DATA SET RECn
where "n" is the number/letter of the physical data set you want to erase.
Warning: Both of these commands take destructive action immediately. Double-check that you have correctly typed the data set number or record-type name before pressing the Return key. Recovering the data if you make a mistake is usually very difficult, if indeed it is possible at all (see below).
These two ZAP commands may also affect data in the residual data set. If records from the data set or record-type being zapped have been removed to the residual, that data will be erased too.
You need not recompile your file definition to begin using the zapped data set or record-type again; it is empty but not gone.
Here is a sample SPIBILD session demonstrating this procedure using the ZAP RECORD-TYPE command:
Command> spibild -Welcome to SPIBILD -? establish course catalog -Processing file: GQ.DOC.COURSE.CAT -Compute time: 0.112 seconds -Elapsed time: 1.788 seconds -Core usage: 0065/0933 -? zap record-type zin09 -File detached -?
The worst problem that can occur with this procedure is that you erase the wrong data set or record-type. If that occurs, you will want to try restoring the entire file, using the most recent copy that was made. (If you made a backup copy of the file at the start of the procedure, as suggested in the Preliminaries, you're in excellent shape.) You may want to contact your SPIRES consultant, who can help you make arrangements to restore the file using backup files made by your computer center. [See 2.1.4.3.]
If the system crashes at any point during the procedure, simply restart the procedure at your convenience.
After working with a new file for awhile, you may decide that the goal records for one of its subfiles needs a new element or that it doesn't need some elements it already has. Depending on the original file design, such changes can be very simple or they can be difficult or impossible to do without rebuilding the entire subfile. [See 2.9.]
The chapter is organized as follows:
Procedure: Add an element Purpose: Add new elements to an existing record-type Who can use it: file owner Basic steps: 1) Change the file definition 2) RECOMPILE filename 3) Recompile formats, if necessary
Once you have created a record-type, whether it's the goal record-type of a subfile or an index record-type of another, you do not have much flexibility to change it, unless you are willing to rebuild the entire record-type, in which case you have all the flexibility you need. But rebuilding can be tedious and expensive, and is unnecessary in many situations. [The procedures for rebuilding record-types appear later in this manual. If the record-type is an index record-type, you will need to rebuild the index, but if it is a goal record-type, you will need to rebuild the entire subfile. [See 2.7.3, 2.9.]]
Generally speaking, as file owner, you can add new Optional elements to any existing record-type in a file as long as that record-type already has Optional elements at the level where you want to add them. For example, if you want to add Optional elements to a Required structure, you can do so if the structure already contains Optional elements.
However, you cannot add new Fixed or Required elements to an existing record-type (unless they are inside a new Optional structure) without rebuilding the record-type. [Note though that you can include Inproc and Inclose rules on an Optional element that make it seem Required to the user; only internally, in terms of the way SPIRES stores the element, would it actually be treated as Optional.]
When significant changes are made to a very important file, the file manager customarily takes either or both of two preliminary steps:
- 1) Create a test file based on the file definition of the production file. Make the proposed changes to the test file, test them, debug them, and then make the changes to the production file.
- 2) Make a backup copy of the production file just before making the change, so that the file can be restored in case of serious problems discovered when the change is made. [See 2.1.4.3.]
The rules for what elements you can add without rebuilding the record-type are:
Following these guidelines, you can change the file definition directly, or by using File Definer. Use File Definer only if you still have the File Definer input for the current version of the file definition.
To change the definition with File Definer, read 1a; to change it directly, read 1b.
Add the desired new elements to your File Definer input file, following the rules listed above. For example, you might add another element to the Optional section at the record level:
ELEMENT comments, comment, com
If you want the element to be indexed, you can easily add it to an existing index, by including the INDEX attribute, followed by the name of the element already indexed. On the other hand, to create a new index for the new element, you should follow the instructions given in the next chapter. [See 2.7.1.] DO NOT simply add the INDEX attribute to the end of the new ELEMENT line.
Then, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2 below
The full procedure, with more information about how to make changes to files using File Definer, appears in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile. Find the record definition for the record-type to which you want to add the elements.
Following the guidelines listed above, add new ELEM statements where desired, along with other statements associated with element definitions as desired, such as LEN, OCC, INPROC, OUTPROC, TYPE, etc. See the "File Definition" manual for more help.
Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new Orvyl data sets requested.
RECOMPILE [gg.uuu].filename
For example,
-> recompile gq.doc.filemgmt -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.
You should recompile any formats written for the record-type if either of these situations is true:
- you have added Optional elements at the record-level or in a structure, and they come before any virtual elements that were already defined at the record-level or in that structure; [Internally, the virtual elements will have new element numbers when the file definition is recompiled, which will not match their old numbers that are compiled into the format definitions.] or
- you have added a new Optional structure within an already existing structure.
If you neglect to recompile them, the formats will probably not work properly.
Use the PERFORM FORMAT LIST command to find the formats defined for the record-type. That list shows all compiled formats for the record-type, including any that were defined by other users, whom you might want to notify of your changes so that they can recompile their formats.
At this point, you are strongly urged to return to SPIRES, select the subfile and add or update some records, including the new element. Remember to display some old and new records too. If the element works as you wanted, you are done with this procedure.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Very serious problems can arise if you mistakenly add a new element in a place that breaks the rules presented earlier. The problem can manifest itself in several ways:
To recover, you need to follow these steps:
- remove or dequeue those records containing the new element;
- process the file in SPIBILD;
- correct the file definition in the FILEDEF subfile, continuing with the procedure described earlier in this section, starting with step 1.
In general, SPIRES does not allow you to remove elements defined for a record-type. Once a record-type has a given set of elements, it will always have those elements, plus any additional ones you may be able to add. [See 2.6.]
You can, of course, rebuild the record-type completely, which is really the only way to completely get rid of an element right away. [See 2.8.1.] When you dump out the contents of the record-type (to batch in again later), you would delete the occurrences of the element you were eliminating. You might do that either by deleting those occurrences using the text editor, or by using the SET ELEMENTS command to restrict the output elements to those you were intending to add back in.
A less drastic solution is to add the $DISCARD proc to the Inproc rule and $NULL to the Outproc rule of the element you want to eliminate. This has no immediate effect on the stored records already in the subfile, though it does hide the element value when the record is displayed.
However, as new records get added or old records get updated, the element will gradually disappear. (As part of this method, you might want to rename it, e.g., DUMMY.) It will not go away completely: it will always show up as an element (e.g., in SHOW ELEMENTS commands) and some older records that never get updated or removed may retain it, but it will take less and less room in the file as time goes by.
Alternatively, you could choose to get rid of all occurrences of the element immediately by updating all the old records. [See 2.4.] Again, the element will not go away completely; it will still show up in element lists, for example.
Rules for record/structure changes.
These rules apply to situations were data already exists within the database. Record level changes are not much different from structure changes, except that record level changes are tied to the existence of records within the record-type, and structure changes are tied to structure occurrences within records.
If there are NO occurrences of records, you can change anything within the RECORD-NAME portion corresponding to the record-type except the COMBINED (or not COMBINED), or the SLOT (or not SLOT) nature of the record-type.
If there are NO occurrences of a structure within ANY of the records, you can change anything in the structure's definition.
If there are records existing in the record-type, your changes are limited to the same changes allowed for existing structure occurrences. What is allowed, and when, is covered below in the form of Questions and Answers.
Q1: Can Optional elements be defined in none were defined?
A1: Usually NO, but there are two exceptions when you can add the OPTIONAL section with one or more optional elements.
If there is only a KEY in the REQUIRED section, and no other REQUIRED elements, OR there is no REQUIRED section at all, then you can add an OPTIONAL section. This is true even if there are FIXED and/or VIRTUAL sections.
Q2: Optional elements are defined (by the OPTIONAL section), but none of these elements occur in any of the records, can I delete the OPTIONAL section entirely?
A2: Usually NO, but you can if there are no other REQUIRED elements other than a REQUIRED KEY. This is basically the reverse of Q1. Furthermore, the only penalty for having declared OPTIONAL elements which don't occur is the OPTIONAL element bit-mask, which is 2-byte long, and only occurs if there are REQUIRED elements other than a REQUIRED KEY.
Q3: Can I define new FIXED or REQUIRED elements?
A3: NO, once FIXED or REQUIRED elements occur, you can't define new elements in either the FIXED or REQUIRED sections. But if OPTIONAL elements have been declared, or can be declared (see Q1), then you can add OPTIONAL elements.
Q4: Optional elements are defined, and some of them exist within records, can I define new OPTIONAL elements?
A4: YES, you can add new optional elements to the OPTIONAL section after the latest physically occurring element in any of the records. For example, if you have defined:
OPTIONAL;
ELEM = A;
ELEM = B;
ELEM = C;
and only A and/or B occur in the records, and C never occurs, you can add ELEM = D; after ELEM = B; or ELEM = C; (the end). It is always safe to append to the end of the OPTIONAL section.
Q5: Can I redefine non-occurring OPTIONAL elements?
A5: YES, if an optional element does not occur in any records, you can redefine that element anyway you like, including making it a simple element when it was a structure, or making it into a structure when it was a simple element. LENGTH and/or OCC attributes can be changed, INPROC and OUTPROC rules, even the element's name and ALIASES. The only requirement is that you retain an element in this element's position. You can't delete it or move it. So in Q4 above, since C didn't occur in any of the records, we could have redeclared it as element D, rather than add ELEM = D; statement. In a sense, we are replacing the definition of a non-occurring element (ELEM = C) by a new definition.
Q6: In a structure defined with no KEY, can I change an ELEMENT to a KEY, or vice versa?
A6: You can only have one KEY in a structure, and it must be either the first FIXED element, or the first REQUIRED element. You can change the first FIXED ELEMENT to a KEY at any time, and vice versa. The first REQUIRED ELEMENT can be changed into a KEY only if the structure does not occur in any records. Once the structure occurs, its REQUIRED section must not be changed.
Q7: What can I do with VIRTUAL elements?
A7: You may add or remove VIRTUAL elements at any time.
Q8: After making changes, what must I do?
A8: If you updated a RECDEF, you must recompile that RECDEF and any FILEDEFs that specify that RECDEF in DEFINED-BY statements. If you updated a FILEDEF, you must recompile that FILEDEF. Then, you must recompile all FORMATS that refer to this file/record-type.
After a SPIRES subfile has been used for awhile, you, the file owner, may discover that the existing index scheme isn't quite adequate. An index you thought would be used often has turned out to be unnecessary, for example, or you realize that you've been using Global FOR commands frequently to find records based on an unindexed element. These situations are fairly common -- file owners often use data bases differently than predicted once the data is available to them.
The decision to add or remove an index is usually related to cost and convenience. Specifically, does the convenience of a particular index justify the cost of storing and maintaining this redundant information? For frequently used indexes, the answer is yes; but if an index is seldom or never used, the answer may be no.
This chapter presents the procedures used to handle major tasks relating to indexes:
- to change values in an index when its Passproc rule has been changed (e.g., so that the index will be a word index instead of a phrase index);
- to repair a damaged index (e.g., an index whose Passproc was changed previously without the index being rebuilt, so that old records being removed or updated aren't getting "un-indexed" properly);
- to remove an element's values from the index (e.g., you want to remove the TITLE element values from a SUBJECT index that had both TITLE and SUBJECT element values);
- to add an element's value to an index (e.g., you want to add the TITLE element values to a SUBJECT index so that it will have both TITLE and SUBJECT element values). [You may be able to do this without erasing the current index first, a possibility discussed in section 2.7.3.]
You can also use the FASTBILD program when building or rebuilding an index. You can find the primary documentation for that program later in this manual [See 2.12.] but its context in building indexes will be discussed in this chapter.
Procedure: Add an index Purpose: Add new indexes to an existing SPIRES subfile Who can use it: file owner Basic steps: 1) Change the file definition. 2) RECOMPILE filename 3) SPIBILD 4) ESTABLISH subfile-name 5) Establish the Global FOR environment. 6) Issue the BUILD command.
Though six steps are shown here, the process of adding new indexes to a subfile is fairly simple. The design phase (suggested in step 1) generally takes the most time.
You need to choose a one-to-six-character record-name for each new index record-type you are defining (e.g., REC09, ZIN12). Note: This is crucial even if you use File Definer to create and change the file definition, previously leaving the selection of record-type names up to File Definer.
New record-types being added to an existing file must be added to the end of the existing record-types. Since SPIRES sorts them in alphanumeric order by name before compiling them, it is crucial that the new index record-types you define have names that sort after those that already exist. [There is an exception. If the file has any DEFQ-only ("pseudo-") record-types, whose names make them sort to the end of the record-types, the new ones must precede them at the end. See the "File Definition" manual for more information.]
For example, if the file already has the three record-types REC01, ZIN01 and ZIN05, your new index would need a name that sorted after ZIN05, such as ZIN06. You could not add an index record named ZIN02, unless you are willing to rebuild the entire file.
The names of all the current record-types appear in the RECORD-NAME statements of the file definition as stored in the FILEDEF subfile. Another reliable way to find the names of the current record-types is to select the subfile in SPIRES and issue the command PERFORM FILE SUMMARY.
You can follow this step using File Definer or by changing the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file definition.
WARNING: Whichever method you use, be extremely careful to follow the directions, to ensure that the new index record definitions follow the definitions for those that already exist. Failure to do so may result in data integrity problems that are very tedious and unpleasant to correct.
To change the file definition using File Definer, read 1a; to change it directly, read 1b.
Add the INDEX attribute as appropriate to your File Definer input file, following it in each case with the RECNAME attribute, followed by the name you chose for the new record-type. For example, to add a new index for the element defined here:
ELEMENT author/ name/ single
you might change it to:
ELEMENT author/ name/ single/ index/ recname zin20
Then, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2 below
The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile and add to it:
- a record definition for each new index record-type; and
- appropriate linkage sections for each index.
Remember that a file can have no more than 64 record-types, stored in no more than 9 Orvyl data sets (filename.REC1 through filename.REC9). Use the COMBINE statement to combine record-types in data sets. SPIRES will put each record-type that doesn't have a COMBINE statement into its own data set, to the limit of 9.
Also remember to follow the rules for index record-types and index linkages in the same goal record-type, such as:
- The pointer element must be given the same name in all index record-types for a single goal record-type.
- In general, pointer groups for indexes that apply to the same goal record must be structured identically, e.g., must all be TYPE = LCTR, must all have the same LENGTH statement if specified for any, etc.
For more detailed information about coding index record definitions and linkage sections, see the manual "SPIRES File Definition".
Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new Orvyl data sets requested.
RECOMPILE [gg.uuu].filename
For example,
-> recompile gq.jnk.players -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.
At this point, you are strongly urged to return to SPIRES, select the subfile and issue the BROWSE command to look at the new index. It should be empty, and you will receive the message "Null range". If it's not, then you did not add the new record-type to the end of the existing record-types, and woe to you if you continue this procedure! Instead, you should return to the first step and correct the file definition, making sure the new record-type or -types appear at the end of the record definitions in the file definition.
The first two steps are usually done interactively, but the rest of the procedure may be done as a batch job if desired. [You may either use SPIBILD in a batch job or use FASTBILD. FASTBILD is recommended if the index will be extremely large, e.g., more than 10,000 index entries created. [See 2.12.]]
Issue the SPIBILD command online, or in a BATWYL job.
-> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
Issue the ESTABLISH command to process the file's deferred queue and to tell SPIBILD the name of the subfile containing the index or indexes to be built.
-? establish players -Processing file: GQ.JNK.TEAMS -Compute time: 0.144 seconds -Elapsed time: 2.086 seconds -Core usage: 0068/0933 -?
You can build all the indexes for a single goal record-type at one time. If you are adding indexes for several different sets of goal record-types in the same file, you need to return to this step for each goal record-type.
At this point, you may also want to increase the size of the pass stack to increase passing efficiency, using the SET PASS STACK command. See the description for that command in Part 6. [See 6.3.0a.]
With this step, you tell SPIBILD which goal records to use to build the index. Usually, you will simply issue the FOR SUBFILE command:
-? for subfile -?
Generally speaking, with this particular procedure, you want to build the new index from all the goal records. However, if for some reason you want to limit the records that are indexed, you can add a WHERE clause, or issue SET SCAN, SET FILTER and SKIP commands as needed.
Another alternative, if you want to limit the records that are indexed, is to create a stored stack or result of those records in SPIRES and use the FOR STORED command rather than FOR SUBFILE. FOR STORED is more commonly used in SPIBILD with merging or removing records [See 2.4.2.] or [See 2.5.1.] For more information, see the description of FOR STORED in Part 6. [See 6.1.2b.]
Under Global FOR, the BUILD command has the following basic syntax:
BUILD index.name[, index.name...]
where "index.name" is the name of the index record-type, as it appears in the INDEX-NAME statement of the file definition's linkage section. If you used File Definer to change the definition, "index.name" is the name you specified with the RECNAME attribute.
The "index-name" specifications of the BUILD command allow a number in parenthesis which chooses one occurrence of INDEX-NAME when there are many having the same "index-name". For example, assume your file-definition had the following:
INDEX-NAME = APPLE; INDEX-NAME = ORANGE; INDEX-NAME = ORANGE; INDEX-NAME = PEARS;
If you specified: BUILD APPLE, PEARS, ORANGE(2) then the first INDEX-NAME = ORANGE; would not be built.
If you specified: BUILD ORANGE, PEARS then only INDEX-NAME = APPLE; would not be built.
The BUILD command under Global FOR builds indexes for ALL goal records (that meet WHERE-clause and SET SCAN criteria).
Continuing the above example, if you are adding the two indexes ZIN14 and ZIN15 to this PLAYERS subfile,
-? build zin14, zin15 -Passing: 121 records of record-type REC01 -End of BATCH/BUILD -Compute time: 2.698 seconds -Elapsed time: 26.007 seconds -Core usage: 0211/0939 -?
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Another alternative is to determine whether any of the records being indexed have been indexed: return to SPIRES, select the subfile and try searching for various records using the new index. If no records in the subfile were indexed, start the procedure again at step 3. If any of the records were successfully indexed, try to determine which was the last record to succeed and then restart the procedure at step 3, using a WHERE clause and/or SET SCAN commands to resume the indexing from that point.
Very serious problems will arise if you mistakenly add a new index record-type between existing record-types in the file. The problem can manifest itself in several ways:
It is very tedious to recover from this problem, particularly if you don't discover it till after using the BUILD command. That's because if you used the BUILD command, you passed data to one or more indexes already containing data, not to your empty index record-type. SPIRES added your new record-type to the end -- that's where the empty one is -- so if you didn't name the last record-type in the BUILD command, more data was added to at least one that already existed.
To recover, you will have to determine which record-types have been harmed by the BUILD command and zap them. Correct your file definition, placing the record-types in their proper order, and recompile it; then try rebuilding.
It is certainly possible that rebuilding the file is the quickest solution to this problem. It may not be the cheapest way, but it may be the least harrowing. Moreover, that gives you complete flexibility in adding and removing indexes.
Procedure: Remove an index
Purpose: Remove indexes from an existing SPIRES subfile
Who can use it: file owner
Basic steps:
1) Use the ZAP RECORD-TYPE procedure to remove the index
records.
2) Change the file definition.
3) RECOMPILE filename
This procedure is an extension of the ZAP RECORD-TYPE procedure described earlier. [See 2.5.2.] The steps of that procedure will be listed as part of this procedure, but you should see the earlier section for details if you need them.
You cannot actually delete a record-type of a SPIRES file -- it must always be there until such time as the file is completely rebuilt. [See 2.9.] What you can do is remove all the records from it and change the file definition so that no data is passed to it, leaving it an empty record-type (which you can use again later if you want).
Be sure that the record-type is used only as an index, and that it doesn't contain other data that makes it useful for other reasons besides as an index (e.g., for table-lookup purposes).
First, you must remove the index records, using the ZAP RECORD-TYPE procedure. The basic steps of that procedure, described in section 2.5.2, are:
The procedure usually looks like this:
Command> spibild -Welcome to SPIBILD -? establish my notebook -Processing file: GQ.DOC.NOTEBOOK -Compute time: 0.158 seconds -Elapsed time: 3.702 seconds -Core usage: 0069/0933 -? zap record-type zin03 -File detached -?
At this point, it is important to understand that you aren't actually removing the index record-type from the file but are in fact emptying it and telling SPIBILD not to pass data to it anymore.
You can follow this step by changing the file directly, or by using File Definer. By changing the file definition directly, you can actually disconnect the index record-type from the subfile. By using File Definer, you are slightly more limited (though the differences can be considered trivial) and you must do more work.
Use File Definer only if you still have the File Definer input for the current version of the file definition.
To change the definition using File Definer, read 2a; to change it directly, read 2b.
Locate the input line where the index you want to remove is "defined". It probably looks something like this:
ELEMENT phone.number/ string/ index
Following that line, add the line:
ELEMENT phone.number/ string/ index PASSPROC $NOPASS
which tells SPIBILD not to pass the element's data to that index.
If other elements are passed to the same index, add the same line to their descriptions too.
WARNING: Do not simply remove the INDEX attribute from the input line. This will definitely cause serious problems, unless the moon is in your seventh house.
The $NOPASS Passproc tells SPIBILD not to pass data to the index, but the linkage to the subfile is still there; e.g., the index still shows up under SHOW INDEXES, even though it is empty. Thus, you will probably want to hide the index using the PRIV-TAGS and NOSEARCH attributes in File Definer. See the "File Definer" manual for details.
Next, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 3 below
The full procedure, with more information about how to make changes to files using File Definer, appears in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the index you want to remove, and delete those lines from the file definition. They usually look like this:
INDEX-NAME = ZIN12;
SEARCHTERMS = PHONE.NUMBER, PHONE;
SEARCHPROC = $Search.Trunc;
PASSPROC = $Pass.Elem('PHONE.NUMBER',phrase);
PTR-GROUP = POINTER;
In general, remove the group of lines that starts with the appropriate INDEX-NAME and ends with the PTR-GROUP statement.
WARNING: Do not remove the record definition for the record-type from the file definition. The file will not recompile without it.
Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.
RECOMPILE [gg.uuu].filename
For example,
-> recompile gq.doc.notebook -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary.
As a final step, you might want to verify that the index is gone by returning to SPIRES, selecting the subfile, and issuing the SHOW INDEXES command.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Procedure: Rebuild an index
Purpose: Rebuild an index that already exists
Who can execute it: file owner
Basic steps:
1) Use the ZAP RECORD-TYPE procedure to remove the index
records.
2) Change and recompile the file definition, if needed.
3) SPIBILD
4) ESTABLISH subfile-name
5) Establish the Global FOR environment.
6) Issue the BUILD command.
This procedure is useful in the following circumstances:
- to change the values in an index when the index's Passproc rule has been changed (e.g., so that the index will be a word index instead of a phrase index);
- to repair a damaged index (e.g., an index whose Passproc was changed previously without the index being rebuilt, so that old records being removed or updated aren't getting "un-indexed" properly);
- to remove an element's values from the index (e.g., you want to remove the TITLE element values from a SUBJECT index that had both TITLE and SUBJECT element values);
- to add an element's value to an index (e.g., you want to add the TITLE element values to a SUBJECT index so that it will have both TITLE and SUBJECT element values). [You may be able to do this without erasing the current index first. Here's a summary of the procedure you can try: 1) Change the file definition so that the new element (TITLE in the example above) is passed to that index, but stop any other elements from being passed to it (SUBJECT). 2) Use the BUILD command to pass the TITLE data to the index for the records currently in the subfile. 3) Change the file definition again so that all the elements are passed to the index for future records. WARNING: This procedure can't be done using File Definer to change the file definition; you must change the file definition directly.]
If you need to rebuild several indexes, you can follow this procedure too, zapping each index separately in step 1, changing the file definition all at once in step 2, and then building the indexes simultaneously in step 6 with the BUILD command.
If you are adding a new element to an index, make sure that the types of data you are placing into the single index are compatible. In general, for instance, you wouldn't index an element stored as an integer with one stored as a string. (You could do it if you convert the integer value to a string value in the Passproc rule string.)
For several steps of this procedure, you need to know the name of the index record-type you are rebuilding, which appears in the file definition. Select the FILEDEF subfile and transfer or display the definition for your file.
To find the name, find the SEARCHTERMS statement that lists the index's search terms; just ahead of that statement will be the INDEX-NAME statement, containing the index record-type's name. For example, if you are trying to find the name of the record-type containing the TITLE index, you might find these lines in the file definition:
INDEX-NAME = ZIN02; SEARCHTERMS = TITLE, TI, T; (etc.)
The name of the record-type for the TITLE index is ZIN02.
As another preliminary step, you might consider making a backup copy of the file in case problems arise during the procedure. [See 2.1.4.3.] Problems where a backup copy is useful arise primarily because of carelessness during step 2, when the file definition is being changed. Hence, you might want to create a test version of your file, making the changes there and testing them before committing the changes to your production file.
The basic steps of that procedure, described in section 2.5.2, are:
The procedure usually looks like this:
Command> spibild -Welcome to SPIBILD -? establish my notebook -Processing file: GQ.DOC.NOTEBOOK -Compute time: 0.195 seconds -Elapsed time: 2.117 seconds -Core usage: 0069/0933 -? zap record-type zin02 -File detached -?
If you are rebuilding a damaged index, you probably don't need to make any changes to the file definition at all, and can continue to step 3. However, if you are changing how or what values are passed to the index, you will need to make changes to the file definition -- most likely to the index's linkage section and possibly to the index record-type's record definition.
You can follow this step by changing the file directly, or by using File Definer. Use File Definer only if you still have the File Definer input for the current version of the file definition.
To change the definition using File Definer, read 2a; to change it directly, read 2b.
If you are changing the type of value indexed, rather than the elements being indexed, follow these instructions: Locate the input line where the index you want to change is "defined". It probably looks something like this:
ELEMENT subject/ string/ index
Make the appropriate change to that line. For example, to convert it to a word index:
ELEMENT subject/ string/ index/ word
which tells SPIBILD to split the element's value into individual words for indexing.
If you are adding an element to those for this index, follow these instructions: Locate the input line for the specific element, e.g.:
ELEMENT subject/ string
Add the INDEX attribute, followed by the name of another element whose values are indexed in the relevant index:
ELEMENT subject/ string/ index title
If you are removing an element from those passed to a given index, follow these instructions: Locate the input line for the specific element, e.g.:
ELEMENT subject/ string/ index title
If the INDEX attribute names another element, simply remove the INDEX attribute:
ELEMENT subject/ string
However, if the INDEX attribute does not name another element, DO NOT simply remove the INDEX attribute from the input line. This will invariably cause serious problems, unless you lead a charmed life. You need to find the name of the index record-type created for this element (it's in the real file definition, in the INDEX-NAME statement), and find another element whose values are also being passed to the same index. For example, if you are removing SUBJECT from an index that also contains TITLE, and the index record-type is REC05, you could change the input lines as follows:
(old) ELEMENT subject/ string/ index ELEMENT title/ string/ index subject (new) ELEMENT subject/ string ELEMENT title/ string/ index/ recname rec05
For further help along these lines, see the SPIRES manual "File Definer" or contact your SPIRES consultant.
After making changes to the File Definer input, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2c below
The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the index you want to change, and make the appropriate changes. Then select the FILEDEF subfile in SPIRES and replace the old version with the new. Next, recompile the file definition, using step 2c.
Simply stated, the procedure to follow is:
RECOMPILE [gg.uuu].filename
If the file definition does not compile properly, SPIRES will tell you so; return to step 2a or 2b as appropriate.
If the file compilation is successful, press on to step 3.
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
Issue the ESTABLISH command to process the file's deferred queue and to tell SPIBILD the name of the subfile containing the index or indexes to be built.
-? establish my notebook -Processing file: GQ.DOC.NOTEBOOK -Compute time: 0.288 seconds -Elapsed time: 4.172 seconds -Core usage: 0068/0933 -?
You can build all the indexes for a single goal record-type at one time. If you are adding indexes for several different sets of goal record-types in the same file, you need to return to this step for each goal record-type.
At this point, you may also want to increase the size of the pass stack to increase passing efficiency, using the SET PASS STACK command. See the description for that command in Part 6. [See 6.3.0a.]
With this step, you tell SPIBILD which goal records to use to build the index. Usually, you will simply issue the FOR SUBFILE command:
-? for subfile -?
Generally speaking, with this particular procedure, you want to build the new index from all the goal records. However, if for some reason you want to limit the records that are indexed, you can add a WHERE clause, or issue SET SCAN, SET FILTER and SKIP commands as needed.
Another alternative, if you want to limit the records that are indexed, is to create a stored stack or result of those records in SPIRES and use the FOR STORED command rather than FOR SUBFILE. FOR STORED is more commonly used in SPIBILD with merging or removing records [See 2.4.2.] or [See 2.5.1.] For more information, see the description of FOR STORED in Part 6. [See 6.1.2b.]
Under Global FOR, the BUILD command has the following basic syntax:
BUILD index.name[, index.name...]
where "index.name" is the name of the index record-type, as it appears in the INDEX-NAME statement of the file definition's linkage section.
The "index-name" specifications of the BUILD command allow a number in parenthesis which chooses one occurrence of INDEX-NAME when there are many having the same "index-name". For example, assume your file-definition had the following:
INDEX-NAME = APPLE; INDEX-NAME = ORANGE; INDEX-NAME = ORANGE; INDEX-NAME = PEARS;
If you specified: BUILD APPLE, PEARS, ORANGE(2) then the first INDEX-NAME = ORANGE; would not be built.
If you specified: BUILD ORANGE, PEARS then only INDEX-NAME = APPLE; would not be built.
The BUILD command under Global FOR builds indexes for ALL goal records (that meet WHERE-clause and SET SCAN criteria).
Continuing the above example, if you are rebuilding the index ZIN02 in the MY NOTEBOOK subfile:
-? build zin02 -Passing: 2421 records of record-type REC01 -End of BATCH/BUILD -Compute time: 19.698 seconds -Elapsed time: 122.944 seconds -Core usage: 0211/0939 -?
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
If any of the records were successfully indexed, try to determine which was the last record to succeed (return to SPIRES and use the index to try to find various records); then restart the procedure at step 3, using a WHERE clause and/or SET SCAN commands to resume the indexing from that point in step 5.
Very serious problems can arise if the changes you make to the file definition are not done carefully. Particularly when File Definer is involved, it is easy to create a new file definition that is not compatible with the old. These problems manifest themselves in various ways:
It can be difficult to find the cause of such problems; to recover from one can be even harder, particularly if you don't discover it till after using the BUILD command. You should probably see your SPIRES consultant as soon as possible. But before you do, you should get a copy of the previous file definition out of the FILEDEF subfile (you could use the TREE or TRANSACTIONS class in Global FOR, for example) in case you need to go back to it. Your consultant will probably need it too, along with a copy of the new definition.
If you erase the wrong record-type in step 1, you will want to try restoring the entire file, using the most recent copy made. Usually this means contacting your SPIRES consultant, who can help make arrangements to restore the file using backup copies made recently by your computer center. If you have your own backup copies for your application, you may want to use those instead. [See 2.1.4.3.]
The previous chapter covered how to add, remove or rebuild an index of a subfile. In this chapter, we go a step farther -- we will add, remove and rebuild subfiles of existing files.
To be more specific, this chapter refers to files already in existence that have (or will have) more than one subfile and more than one set of goal records. Here we will add a new subfile to an existing file, or remove one subfile from a file, leaving other subfiles of the file alone, or rebuild one subfile of a file, leaving the rest of the file alone. [Adding a new subfile to a file that doesn't exist is the same as creating a new file, which is covered in the manual "SPIRES File Definition." Removing the only subfile from a file is the same as destroying the entire file, and rebuilding the only subfile of a file is the same as rebuilding the entire file, two procedures discussed later in this manual. [See 2.9, 2.11.]]
The three sections of this chapter are:
Procedure: Add a new subfile
Purpose: Add a new subfile (goal and possibly index
record-types) to an existing file
Who can use it: file owner
Basic steps:
1) Change the file definition.
2) RECOMPILE filename
Adding a new subfile to an existing file is a fairly simple procedure on paper -- what can complicate it is the design of the subfile.
This section describes the procedure when you are adding a new subfile whose goal record-type will be a new record-type in the file. [If the new subfile will use an existing record-type as the goal record-type, your task is very easy: you change the file definition by adding a new Subfile section describing the subfile and then place the new version in the FILEDEF subfile. You can then select the subfile immediately -- you don't need to recompile the file definition. [See 2.6.1, 2.7.1 if you then want to add elements or indexes to it.]]
You need to choose a record-name of no more than 6 characters for each new record-type you are defining (e.g., REC13). This is crucial to you even if you use File Definer to create and change the file definition.
New record-types being added to an existing file must be added to the end of the existing record-types. Since SPIRES sorts them in alphanumeric order by name before compiling them, it is crucial that the new record-types you define follow those that already exist. [The exception to this rule involves DEFQ-only record-types. If the file contains any of these, whose names make them sort to the end of the record-types, the new ones must precede them. See the "File Definition" manual for more information.]
For example, if the file already has the three record-types REC01, ZIN01 and ZIN05, the new goal record-type would need a name that sorted after ZIN05, such as ZIN06 or ZJA01. You could not add a goal record-type named REC02 unless you were rebuilding the entire file. [See 2.9.]
The names of all current record-types appear in the RECORD-NAME statements of the file definition as stored in the FILEDEF subfile. Another way to find their names is to select a subfile of the file in SPIRES and issue the command PERFORM FILEDEF SUMMARY.
Keep in mind that a file can have only 64 record-types, which must be combined into no more than 9 ORVYL data sets (i.e., "filename.REC1" through "filename.REC9"). In addition, no more than 8 of them can be slot record-types, each of which must be in its own data set, alone. Hence, you could not add a slot record-type if you are already using all 9 of the allowed ORVYL data sets.
To execute this step, you'll need to know specifically what changes you want to make to the file. For help, you may want to consult other SPIRES manuals on file definition, such as "File Definition" or "File Definer". As always when you are making major changes to an important file, you should consider making a test version of the file and making your changes to that one first, to be sure that they work before applying them to the real file.
You can follow this step using File Definer or by changing the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file definition.
WARNING: Whichever method you use, be extremely careful to follow the directions, to ensure that the new index record-type definitions follow the definitions for those that already exist. Failure to do so may result in data integrity problems that are very tedious and unpleasant to correct.
To change the definition using File Definer, read 1a; to change it directly, read 1b.
Add the information for your new subfile at the end of your input file. Be sure to include the GOAL keyword with the name you chose for the goal record-type, based on the rules given under "Preliminaries" above.
Then, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2 below
The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile and add to it:
- a record definition for each new goal and index record-type; and
- appropriate linkage sections for each goal record-type.
- a Subfile section for each new subfile.
For more detailed information about coding file definitions, see the manual "SPIRES File Definition".
Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new ORVYL data sets requested.
RECOMPILE [gg.uuu].filename
For example,
-> recompile patron.information -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.
SPIRES may not recompile the file definition if there are records in the deferred queue. It may tell you to process the file in SPIBILD before trying to compile the definition again. If so, follow the PROCESS/ESTABLISH procedure, and then return to SPIRES, reselect the FILEDEF subfile, and continue with step 2.
When SPIRES successfully recompiles the file definition, you are strongly urged to select the subfile and make sure the new subfile has no records in it. If it isn't empty, then you didn't add the new record-type to the end of the existing record-types, which means your file is on the brink of disaster. You should return to the first step and correct the file definition, making sure the new record-type or -types appear at the end of the record definitions in the file definition. (See "Handling Problems" below.)
For similar reasons, it is a good idea to check the other subfiles in the file too: examine one or two records in them to verify that they are in the proper subfile.
If the new subfile is indeed empty, try adding a couple of records to it. If it works as you wanted, congratulations! If not, you need to make more changes to the file definition, starting again at step 1.
Also, if you defined any indexes for the subfile, try processing the file in SPIBILD and then checking the indexes to make sure the data was passed from your new goal record-type as you expected. [See 2.2.]
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Very serious problems will arise if you mistakenly add a new record-type between existing record-types in the file. The problem can manifest itself in several ways:
It can be very tedious to recover from this problem, particularly if you don't discover it right after you compile the file. If you do discover it right away, you should correct the file definition (placing the old record-types in their original order and moving the new ones to the end), returning to step 1.
Of course, contact your SPIRES consultant if you need help solving any problems that arise. If you're reluctant to ask for help, keep in mind that making mistakes as you "solve" the problem can be more disastrous than the original problem was. Your consultant would prefer being called in sooner rather than later.
Procedure: Remove a subfile
Purpose: Remove a subfile from an existing SPIRES file
Who can use it: file owner
Basic steps:
1) Use the ZAP RECORD-TYPE procedure to remove the goal
and index records.
2) Change the file definition.
3) RECOMPILE filename
This section describes the procedure when you are removing a subfile of a file by removing all the records from its goal record-type and indexes. [You can also eliminate a subfile by simply removing its Subfile section from the file definition -- the records will remain in the file, but users will no longer be able to select the subfile.]
It's assumed here that the file contains other record-types you want to save. If you are getting rid of all the record-types of the file, you should follow the procedure for destroying an entire file, which is much simpler and more efficient. [See 2.11.]
Unless you zap and rebuild the entire file, you can't actually delete a record-type of a SPIRES file. What you can do is remove all the records from it and change the file definition so that no data is placed in it or passed to it, leaving it an empty record-type (which you can use again later if you want).
WARNING: The instructions for this procedure presume you can and are willing to change the file definition directly, without using File Definer. Though it isn't inconceivable that you could use File Definer to make the appropriate changes, that method could be extremely complicated, and complete instructions would take several pages. If you absolutely must use File Definer because you want to continue using it for any future changes to the file, you should either rebuild the file completely (removing the unneeded subfile definitions from the File Definer input) or see your SPIRES consultant for help. Do not use File Definer on your own unless you are very certain how to do so without taking away or rearranging any record-types.
Depending on the way the data was distributed in the ORVYL data sets, you may or may not be able to release any ORVYL blocks containing the subfile's data. In other words, it is possible that eliminating a subfile will not make the file use any less storage space, though that space will be used again if more records are added to other record-types in the file. (See step 1 below.)
This procedure is an extension of the ZAP RECORD-TYPE procedure described in detail earlier in this manual. [See 2.5.2.] The steps of that procedure will be listed as part of this procedure, but you should see the earlier section for details if you need them.
The basic steps of that procedure, described in section 2.5.2, are:
In this command, name one of the record-types of the subfile you want to destroy. Use the ZAP DATA SET command if all the record-types in a given data set are record-types you want to eliminate. [See 2.5.2.]
You need to repeat this procedure for each goal and index record-type of the subfile, restarting each time with step ii. Of course, do not zap any record-types of the subfile that are used by other subfiles you are not destroying.
File blocks no longer needed will be released by the ZAP DATA SET command, and also by the ZAP RECORD-TYPE command if no other record-types are in the same data set.
The procedure usually looks like this:
-> spibild -Welcome to SPIBILD -? establish my notebook xrefs -Processing file: GQ.DOC.NOTEBOOK -Compute time: 0.158 seconds -Elapsed time: 3.702 seconds -Core usage: 0069/0933 -? zap record-type rec05 -File detached -? establish my notebook xrefs -? zap record-type zin06 -File detached (etc.)
At this point, be sure you understand that you aren't actually removing the subfile's record-types from the file but are emptying them and telling SPIRES and SPIBILD not to put data into them anymore.
You should follow this step by changing the file directly. (See the earlier warning against using File Definer.)
In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the goal record-type of the subfile you want to remove, and delete those lines from the file definition. It will begin with a GOALREC-NAME statement:
GOALREC-NAME = recname;
where "recname" is the 1-6 character name of the record-type. Delete all the lines from there up to (but not including) the next GOALREC-NAME or EXTDEF-ID or SUBFILE-NAME statement, whichever comes next.
Next, find the Subfile section(s) of the file definition that names the subfile(s) you want to remove. They begin with the SUBFILE-NAME statement:
SUBFILE-NAME = subfile name;
Delete all the lines from there up to (but not including) the next SUBFILE-NAME, FILE-ACCESS, ELEMINFO, CHECKPOINT, INDEXINFO, EXT-REC or EXT-DEF statement.
Though it isn't necessary, you may want to remove most (but not all) of the contents of the definitions of the various record-types you have emptied. If you do this, do not delete the COMBINE statement, and be sure to leave at least a key element defined -- otherwise, the file definition can't be recompiled.
WARNING: Don't remove any record definitions from the file definition completely. You can't recompile the file definition without them.
Once you've made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.
RECOMPILE [gg.uuu].filename
For example,
-> recompile notebook -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary.
As a final step, you might want to verify that the subfile is gone by trying to select it. It is a good idea to select the remaining subfiles of the file and examine them too.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Procedure: Rebuild a subfile
Purpose: Rebuild an existing subfile of a SPIRES file
Who can use it: file owner
Basic steps:
1) Dump and save the current goal-record data.
2) Use the ZAP RECORD-TYPE procedure to remove the goal
and index records.
3) Change the file definition.
4) RECOMPILE filename
5) Reload the saved goal-record data.
If you decide that one of your subfiles needs major changes, you may need to rebuild it -- that is, remove all the records from its goal record-type and indexes, make appropriate changes to the file definition, put the goal records back in, and rebuild the indexes.
Typical reasons for rebuilding a subfile are these:
- You want to make changes to the goal record-type, adding some new elements, removing some old ones you don't need to keep, or changing the types of values stored for others. These changes also affect one or more of the indexes.
- The goal records have slot keys, and you want to renumber them all, eliminating gaps created by removed records.
- "Available space" information indicates that much of the storage space the file is using is empty. By zapping data sets of the file and rebuilding them, unneeded storage space will be released. (But see below.)
It's assumed here that the file contains other record-types you want to leave untouched. If you are rebuilding the only subfile of the file, you should follow the procedure for rebuilding an entire file, which is much simpler and more efficient. [See 2.9.]
Unless you zap and rebuild the entire file, you cannot actually delete any index record-types of a SPIRES subfile. What you can do is remove all the records from it and change the file definition so that no data is passed to it, leaving it an empty record-type (which you can use again later if you want).
Depending on the way the data was distributed in the ORVYL data sets, rebuilding a subfile may or may not release any ORVYL blocks containing the subfile's data. Blocks are released when an entire data set is zapped or when all the record-types in a data set are zapped. So, if a subfile's data is in ORVYL data sets shared by other record-types of the file, rebuilding the subfile will not make the file use any less storage space, though that space will be used again when the records are added back in.
Depending on the amount and intricacies of changes you want to make to your file definition, you may want to begin with step 3, creating a revised file definition, which you then test thoroughly before "starting" the procedure described below.
CAUTION: In this procedure, you will be zapping data. Anytime you are zapping data, you should compare the importance of the data to your confidence that you can complete the procedure. If the data's value outweighs your confidence, consider making a backup copy of the current file and file definition so that you can quickly recover the file to its current state in case of any problems. [See 2.1.4.3.]
You should create a data set of the goal-record data you want to put back into the subfile. There are several techniques to use in SPIRES:
- If you are not changing the goal record-type's definition, you can use the GENERATE LOAD command. It will create a data set of records in their internal form, which can be loaded into the empty goal record-type later. It is the most efficient tool to use, if you can use it. [See 2.3.3.]
- If you are changing the goal record-type's definition, then you want to create a data set that includes only the elements you want to add back in. Select the old subfile, use the SET ELEMENTS command to choose those elements, and using Global FOR commands (such as FOR SUBFILE and IN ACTIVE DISPLAY ALL), create the data set and save it. If the subfile's records have slot keys, you should omit it from the SET ELEMENTS list if you want the records to have new keys. (See example below.)
- If desired, you might want to create a format to use, rather than the SET ELEMENTS list, for both output and later input. If you do have a format, the method is the same as the one just described.
For the examples in this section, suppose we are rebuilding a subfile called ALARM LOCATIONS in the file GQ.DOC.ALARMS. We want to save all the elements except for the slot key, called RECNUM. There is no "SET ELEMENTS ALL -- elem" command to let us directly set all but a few elements, but an indirect way does exist. In brief, you can use a "SET ELEMENTS ALL + element" and a "SET ELEMENTS -- element" command; by using the two of them in combination with a virtual element (or a dynamic element created on the spot -- either way giving us an additional element to set in addition to "all" of them), you can achieve the desired effect with the SET ELEMENTS command.
Two other items worth noticing are that the $RECSEP variable is set so that each record begins with an "add;" line, and that an index is used to limit the records saved to only those added since 1984.
-> select alarm locations -> find date.added after 1983 -Result: 201 LOCATIONS -> clear format -> set recsep 'add;' -> for result +> define element null for recnum as '' +> set elements all + null +> set elements - null recnum +> in active clear display all +> save alarm.location.records ALARM.LOCATION.RECORDS saved on INT001 +>
If the amount of data may be too large to fit in your active file, consider using the batch SPIRES program to write the data directly to a WYLBUR (OS) data set. See the SPIRES manual "Searching and Updating", or online, [EXPLAIN JCL, FOR SPIRES.] for more information about batch SPIRES.
The basic steps of that procedure, described in section 2.5.2, are:
You need to repeat this procedure for each goal and index record-type of the subfile, restarting each time with step ii. Of course, do not zap any record-types of the subfile that are used by other subfiles you are not rebuilding.
The procedure usually looks like this:
Command> spibild -Welcome to SPIBILD -? establish alarm locations -Processing file: GQ.DOC.ALARMS -Compute time: 0.262 seconds -Elapsed time: 1.551 seconds -Core usage: 0069/0933 -? zap record-type rec01 -File detached -? establish alarm locations -? zap record-type zin08 -File detached (etc.)
If you have no plans to change the file definition, continue with step 6.
For major applications, the file definition will usually be revised and tested as part of the Preliminaries. If so, in this step you should replace the old file definition with your newly revised and tested one.
You can always make the changes directly to the file definition. In addition, you may be able to use File Definer to change the file definition if all of the following conditions are true:
- You have the File Definer input for the current version of the file definition.
- You are not deleting any indexes from the definition.
- You are not rearranging any elements that are indexed.
- You are not adding any new indexes to the definition. [If you are adding new indexes, you can use File Definer by carefully following the directions described in section 2.7.1, under "Preliminaries" and step 1.]
If any of the above conditions are false, you should change the file definition directly, as described below in 3b; if you can use File Definer, read 3a.
Make the desired changes to your input file, making certain that your changes are within the guidelines described above. Then use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 4 below
In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Make the desired changes to it, keeping the following rules in mind:
- Don't remove any record definitions from the file definition completely. You must have at least as many as you did before, and the ones that remain in the file from the past must be in the same numerical position after the file definition is recompiled.
- If you add new record-types, be sure to add them at the end of the list of those already in existence. Remember that SPIRES will sort the records alphabetically by name. A list of all the record-types, in sequential order, is available through the PERFORM FILE SUMMARY command.
Once you've made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.
RECOMPILE [gg.uuu].filename
For example,
-> recompile gq.doc.alarms -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 3 to correct them, if necessary.
You are now ready to reload the data you saved in step 1 into the new subfile. Depending on the form in which you saved it, use either the INPUT BATCH, INPUT ADD, LOAD, FAST BATCH or FASTBILD procedures. [See 2.3.]
As a final step, you might want to verify that the subfile is working as you'd expect by returning to SPIRES and trying to use it. It is a good idea to select the remaining subfiles of the file and examine them too.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Procedure: Rebuild a file Purpose: To rebuild an existing SPIRES file Who can use this procedure: file owner Basic steps: 1) Dump and save the current goal-record data. 2) Change the file definition. 3) ZAP FILE filename 4) SELECT FILEDEF 5) COMPILE filename 6) Reload the saved goal-record data.
If you decide to make major changes to a file, you may need to rebuild it -- that is, make copies of all the goal records in its various subfiles, zap the file, compile a new file, and load the goal records back in. In most ways, this procedure is like the one for rebuilding a subfile, but on a larger scale. [See 2.8.3.] Yet in some ways, it's easier -- you'll find it's easier to zap the entire file rather than individual record-types, and you have complete freedom in creating a new file definition, including the use of File Definer.
People rebuild SPIRES files for many different reasons, such as:
- to rebuild a subfile, when the subfile is the only one in the file;
- to completely eliminate unneeded record-types;
- to make major changes to multiple goal record-types, such as adding new elements, removing old ones, etc., anywhere you like;
- to make major changes to the indexing structure (adding new ones, removing old ones) without having to worry about the order in which the old ones were defined;
- to make major changes in several different areas at once, which could perhaps be achieved by using several different procedures but which can be done more quickly and easily with this one single procedure.
As the list above suggests, this procedure is the catch-all technique for repairing a file. You could compare it to repairing a car that won't run by buying another car. The solution is overkill if the problem simply involves the sparkplugs.
But if the engine has many different problems or much more serious ones, buying a new car may solve them more easily and even more inexpensively (in the long run) than solving them each individually. And consider the advantages of "trading up to a newer model" as you decide how to solve the problems. Moreover, in regard to the sparkplug problem, buying a new car solves the problem (albeit expensively) in a way that keeps your hands clean if you don't want to get them dirty. [We hope this is the closest that SPIRES documentation ever comes to sounding like a car dealer.]
Depending on the amount and intricacies of changes you want to make to your file definition, you may want to begin with step 2, creating a revised file definition, which you then test thoroughly before "starting" the procedure described below.
CAUTION: In this procedure, you will be zapping data. Anytime you are zapping data, you should compare the importance of the data to your confidence that you can complete the procedure. If the data's value outweighs your confidence, consider making a backup copy of the current file and file definition so that you can quickly recover the file to its current state in case of any problems. [See 2.1.4.3.]
Create a data set of the current data in each goal record-type you want to put back into the file. There are several techniques to use in SPIRES:
- If you aren't changing that goal record-type's definition, you can use the GENERATE LOAD command. It will create a data set of records in their internal form, which can be loaded into the empty goal record-type later. It is the most efficient tool to use, if you can use it. [See 2.3.3.]
- If you are changing the goal record-type's definition, then you want to create a data set that only includes the elements you want to add back in. Select the old subfile, use the SET ELEMENTS command to choose those elements, and using Global FOR commands (such as FOR SUBFILE and IN ACTIVE DISPLAY ALL), create the data set and save it. If the subfile's records have slot keys, you should omit it from the SET ELEMENTS list if you want the records to have new keys. (See example below.)
- If desired, you might want to create a format to use, rather than the SET ELEMENTS list, for both output and later input. Except for setting the format, the method is the same as the one just described.
You may choose different methods for different goal record-types of the file.
If you aren't making any changes to the file definition, you can skip ahead to step 3.
Because you have complete flexibility in changing your existing file definition or creating a new one, you can make any changes you want, within the limits of general file definition capabilities. (Remember, after step 4, you will no longer have a file with which the revised definition must be compatible, unless you saved the data in its internal SPIRES format using GENERATE LOAD.) That means you can use either File Definer or the file definition directly to make your changes.
Keep in mind the data that you kept in step 1, however. Be sure the new file will accept that data, e.g., that it will have the same element names, that Inproc rules will accept the same element values, etc.
When you are finished with the file definition, put it in the FILEDEF subfile, using the ADDUPDATE command.
The next step is to destroy the old file, using the ZAP FILE command:
ZAP FILE filename
where "filename" is the name of the file to be erased.
For instance, if you want to zap your file FO.RTE.PIANOS:
-> zap file pianos -Do you want to destroy file FO.RTE.PIANOS ? yes ->
You next need to reselect the FILEDEF subfile.
The COMPILE command compiles the named file definition, verifying the logic of the file definition and creating the various ORVYL data sets of the file.
COMPILE [gg.uuu].filename
For example,
-> select filedef -> compile .pianos -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary, and skip step 3 when you try again.
You are now ready to reload the data you saved in step 1 into the various subfiles of the file. Depending on the forms in which you saved it, use either the INPUT BATCH, INPUT ADD, LOAD, FAST BATCH or FASTBILD procedures. [See 2.3.1.]
As a final step, you should verify that the subfiles work as you'd expect by returning to SPIRES and trying to use them.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Procedure: Move a file
Purpose: To move a SPIRES file from one account to another
Who can use this procedure: file owner
users with Copy access
Basic Steps:
1) Make sure the copying account has enough ORVYL blocks
2) Give the copying account Copy access to the file
3) Make sure the copying account has ORVYL access to the file.
4) Use the COPY FILE command to copy the file
5) Create a file definition for the new file.
6) Cleanup: Other considerations
Sometimes it is necessary to move an entire SPIRES file from one account to another. For example, you might want to move a file from a "test account" to a "production account", or from an account you want to close to a new account.
To move a file from one account to another (say, file GQ.DOC.RESTAURANT from account GQ.DOC to account GQ.JNK), follow this procedure:
First, find out how large the file is. If you are the file owner, issue the ORVYL command SHOW FILES: [The copying account can issue the SHOW FILES command if that account has general access (READ or higher) to all ORVYL data sets on the file owner's account, as determined by the ORVYL command SET PERMIT.]
SHOW FILES LIKE filename BLOCKS
For example, from account GQ.DOC:
-> show files like orv.gq.doc.restaurant blocks ORVYL File System ORV.GQ.DOC.RESTAURANT.CKPT -- 43 BLKS (F: 0, L: 42) ORV.GQ.DOC.RESTAURANT.DEFQ -- 50 BLKS (F: 0, L: 49) ORV.GQ.DOC.RESTAURANT.MSTR -- 5 BLKS (F: 0, L: 4) ORV.GQ.DOC.RESTAURANT.REC1 -- 4 BLKS (F: 0, L: 3) ORV.GQ.DOC.RESTAURANT.REC2 -- 35 BLKS (F: 0, L: 34) ORV.GQ.DOC.RESTAURANT.RES -- 108 BLKS (F: 0, L: 107) ->
Add the number of blocks (BLKS) shown to determine the size of the file.
Next, on the copying account, see how many blocks are available, using the ORVYL command SHOW BLOCKS:
-> show blocks 742 Blocks accounted, 2000 Blocks allowed ->
As long as the difference between the number of blocks allowed and the number currently being used ("accounted") is larger than the size of the file being copied, you can copy the file to that account.
In the file definition, the file owner must give the copying account Copy access to the file. If you have the File Definer input for the current version of the file definition, you can use File Definer to change the file definition (2a below); otherwise, change the file definition directly (2b).
To use File Definer, first add the COPY attribute to the FILE statement, listing the accounts to be given Copy access:
FILE/ COPY GQ.JNK
Then use File Definer itself to generate the revised definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2c below
To add the information directly to the file definition, get the most recent copy and add to the end an occurrence of the FILE-PERMITS structure that includes a FILE-ACCESS value of COPY and an ACCOUNTS value that includes the copying account:
FILE-ACCESS = COPY; ACCOUNTS = GQ.JNK;
Then select the FILEDEF subfile in SPIRES and replace the old version of the file definition with this new one.
The procedure to follow is:
RECOMPILE [gg.uuu].filename
If the file definition doesn't compile properly, SPIRES will give you error messages; return to step 2a or 2b as appropriate.
This step can be skipped as long as the file owner has not set different ORVYL permits for the file than the defaults set by SPIRES when the file is established. If desired, use the ORVYL command SHOW PERMIT to verify that each of the file's data sets allows READ access to the copying account. If any do not, use the SET CLP and SET PERMIT commands in combination to change them.
The COPY FILE command, available in SPIRES, copies the ORVYL data sets of a SPIRES file to the copying account:
COPY FILE ORV.gg.uuu.filename
It must be issued from the copying account, NOT from the file owner's account. As it begins copying each data set, SPIRES displays a message naming the data set and the number of blocks to be copied:
-> spires(quiet) -> show eval $account GQ.JNK -> copy file orv.gq.doc.restaurant Start copy of 50 blocks from data set DEFQ Start copy of 5 blocks from data set MSTR Start copy of 4 blocks from data set REC1 Start copy of 35 blocks from data set REC2 Start copy of 108 blocks from data set RES ->
If you have any problems with the COPY FILE command, see Part 6, or online, EXPLAIN COPY FILE COMMAND.
To define subfiles for your copied file, you must have a file definition. In general, it is best (easiest) to get the original file owner's file definition or file definer input and modify it accordingly, changing the account number in the file name, changing the subfile names, etc. (If, for example, you had an older copy of the file definition and the file owner had changed the file definition in the meantime, recompiling your copy with the old definition could cause problems serious enough to force you to begin the procedure again.) Be sure to add your new copy of the file definition to the FILEDEF subfile so that you can issue the SELECT command to use its subfiles.
Depending on the circumstances, you may need to continue with the following procedures:
- If there are any formats associated with the original file, you may want them to work for the new file too. You should consider either:
- getting copies of the format definitions from the format owners, changing the ID and FILE statements within them to reflect your ownership of the format definition and file, and then adding them to the FORMATS subfile and compiling them; or
- getting the format owners to add GEN-FILE statements to their format definitions so that they can be used by both files. (That would mean setting the formats slightly differently, using the "general file format" form of the SET FORMAT command.)
- If special ORVYL permits were set for the data sets you copied (which is likely if the file has immediate indexes, for instance), they were not brought across to the copies. Instead, SPIRES sets the same permits it does for a new file: READ access to the public for all data sets except the DEFQ, which gets public WRITE access. If you want to set the same permits as the original file, you need to find out what they were from the file owner (who can use the "SHOW PERMIT filename.dataset" command to recall them) and then issue the SET CLP and SET PERMIT commands to set them on the data sets of the new file:
SET CLP filename.dataset
SET PERMIT filename.dataset FOR account access-level
SET NOCLP filename.dataset
- If you now want to destroy the original file, follow the procedure "Destroying an Entire File" in the next chapter. [See 2.11.]
Procedure: Destroy a file Purpose: to destroy an unwanted SPIRES file Who can execute this procedure: file owner Basic Steps: 1) Make sure you no longer want the file. 2) Get rid of formats code associated with the file. 3) Issue the ZAP FILE command. 4) Remove file definition from FILEDEF subfile.
When you no longer want to keep a particular SPIRES file on your account, you can destroy it, following a procedure based on the ZAP FILE command. The example session outlined below shows how account GQ.DOC would destroy its file GQ.DOC.ERRATA.
If you are sure you want to destroy the SPIRES file but aren't absolutely sure that you will never again need the data within it, you might want to save copies of the records, storing them on tape, or on another account, etc. Or you might want to consider copying the entire SPIRES file to another account, using the COPY FILE procedure. [See 2.10.]
When you are destroying a file, please remove formats associated with the file from the FORMATS (format definitions) and FORCHAR (compiled format code) subfiles. To do so, follow this procedure:
-> select spires doc notes <-- a subfile of GQ.DOC.ERRATA
-> perform format list
Formats defined for file 'GQ.DOC.ERRATA'
Format-Name Record Source-ID Date Compiled
ERRATA.REPORT REC01 GQ.DOC.ERRATA.REPORT 03/27/86
->
SELECT FORMATS FIND FILE gg.uuu.filename FOR RESULT REMOVE ALL
There may be other records in system files relating to your file, such as EXTDEF, VGROUPS or SYS PROTO records. Please remove any records from those subfiles that are used only by the file you are discarding too.
The ZAP FILE command in SPIRES erases the ORVYL data sets that comprise the SPIRES file.
-> zap file gq.doc.errata -Do you want to destroy file GQ.DOC.ERRATA ? ok ->
Because of the seriousness of the ZAP FILE command, SPIRES asks you to confirm that you want to destroy the named file. Once you confirm it, the data sets are gone.
You should remove the file's file definition from the FILEDEF subfile. Before doing so, you should consider whether to save the file definition elsewhere, just as you did for the data in step 1. You can store old file definitions in a public SPIRES subfile called BACKFILE. Since the file is owned by the computing center, you are not charged any storage costs for keeping the definition there.
In this example, account GQ.DOC decides to move the file definition for GQ.DOC.ERRATA into the BACKFILE subfile:
-> select filedef -> transfer gq.doc.errata clear -> remove gq.doc.errata -> select backfile -> add ->
FASTBILD is a batch program you can use when you need to add large numbers of records to a brand new subfile, to build all the indexes of a large subfile, or to build a new index for a large subfile. Though it does work that SPIBILD can do, FASTBILD does the work much more efficiently under some circumstances, which means it can save you time and money. In addition, the data sets it builds are somewhat better built than SPIBILD's, meaning that record retrieval from them is more efficient.
FASTBILD is an alternative to SPIBILD only in very particular circumstances, however. Specifically, you should consider FASTBILD for building a new subfile when all of the following are true:
- you are adding 10,000 or more records to an empty goal record-type at one time;
- at least one index is being built from the goal records;
- the records are in sequential order by key in the input data set;
- the record-type has never had any records in it since it was compiled or "zapped";
- no goal-to-same-goal passing is involved, i.e., the goal record does not pass data to other goal records in the same record-type.
You should consider using FASTBILD to build all a subfile's indexes if both of the following are true:
- the data currently in the subfile will create 10,000 or more index entries for the new indexes;
- none of the index record-types have ever had any records in them since they were compiled or zapped.
You should consider using FASTBILD to build a single index if both of the following are true:
- you will be adding 10,000 or more index entries to the new index at one time;
- the index record-type has never had any records in it since it was compiled or zapped.
When CPU-usage is an issue and the preceding guidelines are met, you will get better results by carefully using FASTBILD rather than SPIBILD.
In most cases, file managers use FASTBILD to build new subfiles rather than to build new indexes alone. Hence, the discussions in this chapter are centered around the former use, though in general they are discussing both. However, if particular points need to be made about building new indexes, you will see them along the way.
This chapter is arranged as follows:
The basic FASTBILD task consists of two batch jobs:
- The first job adds the input goal records to the empty goal record-type and places index record data into OS files. [If the FASTBILD task is to just build indexes, the goal records currently in the tree are read, and the index record data is put into OS files.] The input data may be in one of several forms:
- in standard SPIRES format; or
- in a form that can be interpreted by a custom input format; or
- in internal SPIRES format, created by a GENERATE LOAD command.
- The second job sorts the index data and then builds the SPIRES index records from the sorted data.
If you are building a subfile of hundreds of thousands of records, the two jobs could take many hours. You can set up the jobs so that the first job is run one day, and the second is run some other time. (You need to alter the JCL so that the index data is not placed on a scratch volume that might be erased before you run the second job.) In addition, the second job can be split into separate smaller jobs (perhaps one for each index being built) to further subset the procedure.
Developing good JCL for your FASTBILD run is very important. Most file managers use JCL that was created for them by a public SPIRES protocol called FASTBILD.JCL. The protocol actually creates the JCL for just the first job; the JCL for the second is created by FASTBILD as it handles the first job. FASTBILD can submit the second job for you automatically as the first job finishes up, though that is not generally recommended.
Unquestionably, FASTBILD uses less CPU than SPIBILD to do a comparable job when the job is large. The difference primarily comes from the indexing. It is much more efficient for FASTBILD's sorting package to sort all the index data at once than for SPIBILD to retrieve, update and replace index records in pass stack after pass stack. If you are building a subfile that has no indexes, FASTBILD offers no advantages over the FAST BATCH command in SPIBILD. [See 2.3.4.]
The fact that FASTBILD does not "checkpoint" the data as SPIBILD does also contributes to the difference. Checkpointing the data as blocks are written to the data sets of the file is crucial to recovery procedures, but it does add significant overhead. FASTBILD gives up the checkpointing in the name of speed, but at the price of having no graceful way to recover if the system fails -- you usually have to start over. (Doing the work in "steps" may allow you to restart from the last completed step.)
However, there are rare situations where you cannot use FASTBILD. First, no terms being passed from a goal record-type to an index record-type (whether the key, pointer, qualifier or sub-index term) can exceed 255 bytes. Also, if the pointer-group in the index record-types being built is a structure, it must begin with the key (the pointer); in other words, it cannot have Fixed element terms before a key in the Required section. If building the file with FASTBILD is important, you may need to change the file definition to avoid these possibilities.
Many file managers who should consider using FASTBILD have traditionally shied away from it for various reasons. Instead, they have used SPIBILD, using much more CPU and creating less efficient files. Here are some of their reasons for not choosing FASTBILD, along with some sympathetic rebuttals.
There's no doubt that FASTBILD can be complicated and foreboding; it's "real programming", involving analysis of your input data and creation of JCL to do the needed processing. But building extremely large files can cost thousands of dollars; doing it less efficiently in SPIBILD will cost even more over the long run. The immediate savings in building costs and the longterm savings in data retrieval from having a more efficient file will make the work worthwhile.
To build a subfile of hundreds of thousands of records in FASTBILD, you should give some thought to the following:
More information about these suggestions appears in the rest of this chapter.
Procedures: FAST BUILD procedure in FASTBILD
LOAD procedure in FASTBILD
SCAN GOAL RECORDS procedure in FASTBILD
BUILD procedure in FASTBILD
Purpose: Add many records to an empty goal record-type;
or build all the index record-types of a subfile;
or build a new index record-type
Who can use it: file owner
users with Process access to the file
Basic steps:
1) Create the JCL for the first FASTBILD job.
2) Prepare the input data.
3) Make sure the record-type is completely empty.
4) Run the first FASTBILD job.
5) Make a backup copy of the file and the index data sets.
6) Retrieve the JCL for the second job.
7) Run the second FASTBILD job.
8) Cleanup: Make a copy of the file.
9) Get file statistics from the STATUS command.
The procedure described here works for three different types of tasks: building a goal record-type and its indexes, building all of the indexes from an existing goal record-type and building just an index from an existing goal record-type.
This procedure is extremely long; it will take you some time just to read it, and in execution it often takes several days to move from step 1 to step 9. It is not meant to be rushed through in a hurry. Please check with your SPIRES consultant if any questions or problems arise that you can't resolve on your own.
Before you begin the procedure described below, be sure that FASTBILD is appropriate for your task, by reading the sections preceding this one. [See 2.12, 2.12.1.]
This step encompasses several preliminary steps. Before creating the JCL, you need to know some information about the input data and about the file definition:
- the number of goal records being added (or passed to the new index or indexes);
- the maximum lengths for all values being passed;
- the average number of entries being passed to each index for each goal record.
The FASTBILD.JCL protocol needs these numbers to estimate how much space to reserve for the index data sets it will create.
Having a copy of the file definition will help you determine these figures and help you answer questions the FASTBILD.JCL protocol will ask you.
You should get a rough number of how many goal records you will be adding to the goal record-type (or how many goal records you will be passing, if you're using FASTBILD to build new indexes only).
Warning: A low estimate could cause trouble, so definitely estimate high. If the value is too low, it is possible that the allocated space for the OS data set containing the data for an index won't be large enough, and the FASTBILD job will abend.
This step is fairly important. It asks you to analyze the input data and the structure of the index record-types being built. Specifically, you should determine the maximum length for each type of value that will be passed to any of the indexes being built. You need to know if the file definition imposes any length restriction and/or if the input data itself has only shorter lengths for those elements.
The FASTBILD.JCL protocol needs this information for another reason besides for its estimates of the space required. The index-data records in the OS data sets that FASTBILD sorts in the second job are in effect fixed-length records, with each element starting in a fixed column. FASTBILD needs to know how much room to leave for each field, so that there is enough room in the field for the longest value.
On the other hand, it is in your interest to supply the lowest possible value for each field, because sorting efficiency increases as the length of the sort records decreases. That's why you should know, for example, if you have an element whose maximum length is allowed to be 50 characters but all the input values have lengths under 10.
Important: The value you are interested in is the length of the value that will be passed, not the element's input length or its stored length. For instance, the maximum length of a text element might be 200 bytes, but if it is word-indexed, then the longest passed value will be the longest word, which is surely much less than 200 bytes long. Similarly, a coded value stored as a 1-byte integer might be passed in its expanded "decoded" form, which might be longer.
There is an absolute limit in FASTBILD of 255 bytes for any value being passed; the element value, the pointer, the sub-index value, any local qualifier, and any global qualifiers must each be no longer than 255 bytes.
You need to get this information together now, because later, in the middle of the FASTBILD.JCL protocol (step 1e), SPIRES will show you the elements to which data is being passed in each index record-type and ask you for the maximum length specifications of each.
To sum up, be as accurate as reasonably possible, but err on the high side rather than the low. The efficiency gained by declaring a 10-byte length rather than 12, for example, is insignificant compared to the trouble of later handling any records with 12-byte values that were truncated in passing if your guess of 10 was wrong.
You can find out the lengths of element values being passed by examining the file definition and the input data.
By looking at the index-record definitions and the linkage section of the file definition, you can determine what elements will be passed to the indexes (those are the only elements you're interested in) and possibly what their maximum passed length will be. For example, most indexes contain only a key value (the value being indexed) and a locator back to the goal record-type. The locator is always a 4-byte hex value, so that length is easily determined.
You should also examine the input data, especially if the lengths of the values being passed to indexes are variable. That is, if you are passing an element to an index via FASTBILD and you don't know how long the longest value to be passed is, you should find out by examining the data.
It may be impossible or too difficult for you to determine the maximum length, especially if you have no easy access to the input data (e.g., it is coming from a tape). In that case, you might want to simply use the default or make a reasonable guess. Remember that, in many cases, these values are values that users may type in search commands or as keys of records. Is it likely that a user will type a search value longer than 25 characters?
Whatever the case, FASTBILD will truncate values that are longer than the maximum you specify and add an error message to the FASTBILD output. The error will not stop FASTBILD.
For indexed elements whose maximum length can't be easily determined, file owners frequently add a $MAX.LEN proc to both the SEARCHPROC and PASSPROC statements for the element, limiting the passed values' lengths to the maximum specified -- longer values are truncated. They then specify the same length in the FASTBILD JCL; that way, the truncation is intentional, and no errors are reported. This also helps searchers, since any long values they enter are truncated to the same length as the stored values.
You need to tell the FASTBILD.JCL protocol the average number of index records that the average goal record will create for each index. For example, an indexed goal-record element that is a required, singularly-occurring numeric value will probably pass that one value to the index. On the other hand, an optional singularly-occurring text value broken into words for passing may pass from zero to hundreds of values to the index. [Technically, the number of records that will be created is the number of combinations of unique index key values, sub-indexes and qualifiers.]
Warning: As in the estimate for the number of records being added or passed (step 1a), this estimate should be on the high side, rather than the low. If the value is too low, it is possible that the allocated space for the OS data set containing the data for an index won't be large enough, and the FASTBILD job will abend.
The less Inproc processing that needs to be done, the less time it will take FASTBILD to build the goal record-type. Eliminating Inproc rules not needed for this particular data can save a significant amount of processing. [Even better is to avoid any Inproc processing by providing the data in SPIRES internal format, via the GENERATE LOAD command. Of course, the data can't be in that format unless it's coming from another SPIRES file having the same internal structure. [See 2.3.3.]]
For example, if you have an Inproc that tests the range of a numeric value and you know for certain that all the records you are building have values within the range, you could save money by removing that Inproc rule from the file definition for the duration of the subfile-building. Similarly, table-lookups done by $LOOKUP and $SUBF.LOOKUP procs that verify values can often be omitted; they are particularly expensive to do. Thus, you might have two file definitions for the file -- one with full editing and indexing for day-to-day use and a special one just for the FASTBILD task. Naturally, the more records being input, the more savings this step can provide.
In SPIRES, issue the command "XEQ FROM FASTBILD.JCL":
Command> call spires -Welcome to SPIRES-3 ... if in trouble, try HELP Command> xeq from fastbild.jcl -OK to clear? yes * Do you wish full tutoring? (RETURN=Explain Tutoring) :-?
If the XEQ command fails with an S256 error ("no such record"), issue the command SET COMPXEQ + PUBLIC PROTOCOLS and then try the XEQ command again.
Answer the questions the protocol asks, a procedure that usually takes two to ten minutes, depending on the complexity of your task and your familiarity with the questions the protocol asks.
Instructions for answering most of the questions in the protocol are given in the next section of this manual. [See 2.12.3.]
When the protocol ends, it leaves the JCL for the first job in your active file. It is a good idea to save it somewhere before continuing.
Few file managers use the JCL created by the FASTBILD.JCL protocol exactly as is. Sometimes changes must be made; for example, if the input data is from a "generated load" in internal SPIRES format, or if you are using the SCAN GOAL RECORDS procedure to build all indexes, the FASTBILD commands in the JCL must be changed. Sometimes you may want to add options that don't affect the processing but affect the output from the jobs. In either case, be very careful about making changes -- make sure you understand the effects of any line you add, change or delete.
When you are finished making changes, save a copy of the JCL somewhere -- you may want to refer to it later, or use it again.
A section later in this chapter will describe the commands you have to work with. [See 2.12.4.] For now, here is a list of some commands that are commonly added:
- LIST ADDS
- SET FORMAT
- ALLOCATE
- SET SEARCH
- SET MESSAGES
- LOAD
- SCAN GOAL RECORDS
By default, FASTBILD will not display individual lines about the added records in the output (in contrast to SPIBILD).
A recommended change is to increase the TIME parameters on the JOB and EXEC statements and on the EST command, perhaps doubling the amount of time FASTBILD uses (expressed in CPU minutes). FASTBILD's estimate is sometimes too low, especially when the input format is doing extra work (e.g., creating a report). Remember that you pay the bills -- if the guess is too low, the job stops in mid-stream. Increasing the time estimate is strongly recommended.
You may also need to change the LINES parameter on the JOBPARM statement, especially if you request that the ADDS be listed. Again, the job will fail if the estimated lines of output are exceeded, so it's generally better to give a high number. [On the other hand, if successful records are not being listed in the output (i.e., you don't have LIST ADDS in the command stream), then if the job fails because of a low LINES number, it's probably because a large number of records are failing to be added and many error entries are being written. In that case, the job failure might be desirable.]
When FASTBILD creates the index data in the first job for the second job to use, it puts the data sets onto a scratch volume for temporary storage. At Stanford, the data on the scratch volumes is erased daily, at about 6 a.m. To avoid the risk that the data will be discarded before the scratch volume is erased, or to run the second job at a later date, you should have FASTBILD put the data into permanent disk storage. [It is also wise to check that the volume where FASTBILD is going to place the index data has plenty of empty space; if not, you need to find one that does.]
First, find a volume with enough free space to hold the data sets. You need to determine from the SPACE allocations in the JCL how much space FASTBILD is asking for. (Your SPIRES consultant can help you with this step.) Then change the lines of JCL that begin with SCRATCH:
SCRATCH DSNAME=WYL.GQ.JNK.REC020,VOL=SYSDA=SCR002,PURGE
changing the VOL specification to "VOL=DISK=volume", where "volume" is the name of the permanent-storage volume you want to use. Likewise, change the DD statements later that actually create those data sets:
//GO.REC02 DD DSN=WYL.GQ.JNK.REC020,VOL=SER=SCR002, // DISP=(NEW,CATLG),UNIT=SYSDA,SPACE=(6148,(20,10),RLSE), // DCB=(BUFNO=2,RECFM=VB,LRECL=1024,BLKSIZE=6148)
Change "VOL=SER=SCR002" to "VOL=SER=volume" in the first line, and change "UNIT=SYSDA" to "UNIT=DISK" in the second. Do this for each DD statement.
Note: By following this procedure exactly, you will also move the FASTJCL data set to a permanent volume -- that's the data set that contains the JCL for the second job, which the first job creates.
If you can't find enough room on any volume for the data sets, you should probably write them to tape. See your SPIRES consultant if you need help changing the JCL to request tapes.
You may also want to delete the SETUP lines at the start of the JCL that tell the operator the job needs the scratch volume. If you are using tapes, replace it with a SETUP card that asks the operator to mount them.
If the input data is stored on a tape that will be read during the FASTBILD job, you need to change the JCL to tell FASTBILD. The changes must be made to the DD statements in the GO step of the job that have the names "GO.INnnnn", where "n" is a digit from 0 to 9, such as:
//GO.IN0000 DD DISP=OLD,UNIT=DISK, // DSN=WYL.gg.uuu.name
A discussion of the information needed appears earlier in this manual, in connection with batch SPIBILD. [See 2.1.3.2.] Be sure to include a SETUP line at the start to ask the operators to mount the tape.
When you have several input data sets (you may have up to 16), the FASTBILD.JCL protocol will create skeletal DD statements for all except the first one, named "GO.IN0000". You will need to flesh them out, with appropriate DISP, UNIT and VOL parameters as needed. The data sets will be concatenated when they are needed in the order in which they are specified.
If you are loading a subfile using input from a GENERATE LOAD command in SPIRES, there are several changes you must make to the JCL:
- Delete the two lines of JCL about the IN0000 data set that look like this:
//GO.IN0000 DD DISP=OLD,UNIT=DISK,
// DSN=WYL.gg.uuu.name
- Replace the command INPUT ON IN0000 with the LOAD command, telling FASTBILD the name of the ORVYL data set containing the data to be loaded:
LOAD ORV.gg.uuu.dsname
If the goal record-type is already built and you are using FASTBILD to build all its indexes, you must make the same kind of changes you would for a load. Specifically:
- Delete the two lines of JCL about the IN0000 data set that look like this:
//GO.IN0000 DD DISP=OLD,UNIT=DISK,
// DSN=WYL.gg.uuu.name
- Replace the command INPUT ON IN0000 with the SCAN GOAL RECORDS command:
SCAN GOAL RECORDS
- If desired, you can break this part of the procedure into several steps, by adding the SET SKIP, SET LIMIT, SET SCAN START and SET SCAN STOP commands just ahead of the SCAN GOAL RECORDS command, as described later in this chapter. [See 2.12.4.]
If you don't have the input data in the proper data sets, get it ready now. Make sure the data can be read by the input format you are using, whether it's a custom format or the standard SPIRES format.
If you are building indexes, skip this step -- the data is already in the subfile.
For your input, make sure the records are in sequential order by key, unless FASTBILD will be assigning slot keys to them. If they aren't in order, do not use FASTBILD -- the job will probably fail and you will be wasting money.
If the data is coming from another SPIRES file via the GENERATE LOAD command, make sure the load has been created.
Whatever the situation, be sure that the account under which the FASTBILD job is running has "read" access to the data.
If you are building a new subfile, the goal record-type and all the index record-types that will be built should be absolutely empty. In fact, they should never have had any data in them at all since the last time the file definition was compiled or the last time the record-types were zapped.
In other words, even if you have removed all records in all of the record-types, you should zap all the record-types before running the FASTBILD jobs. [See 2.5.2.] Alternatively, if the file has no other subfiles with data in them already, zap the entire file and compile it again. [See 2.9.]
If you are using FASTBILD to build indexes alone, the same rules apply to each index. If it's ever been "tainted" by records, it should be zapped before the FASTBILD job is run. [See 2.5.2.]
Make sure you no longer have the file attached (CLEAR SELECT) -- you don't want the FASTBILD job to fail because it can't get exclusive access to the necessary record-types. Then put the JCL in your active file if it's not still there and issue the RUN command to submit the batch FASTBILD job:
-> clear select -> run hold JOB 2154 JNK#BLD EXECUTING CLASS=B PRIO=6 ->
Now, sit back and try to relax. This is the time for a good book (perhaps something other than a SPIRES manual), a tall drink, quiet meditation or a little nap.
When the job finishes executing, fetch its output and examine it to be certain the job ran properly. You may find messages about truncated index values -- that is, values being passed that were longer than the maximum length declared for that index. In general, this isn't a problem at this point, unless there are thousands of them, in which case you might want to start again. It is also a serious problem if the pointer element is being truncated; SPIRES won't be able to use those pointers to find the right records. See "Handling Problems" at the end of this section for more on this subject.
If the job fails at some point, fetch the output and try to figure out what the problem was. Please see your SPIRES consultant if you want help. Depending on the problem, you may need to return to step 1 and alter the JCL, or change the input in step 2, restarting the procedure from that point.
Remember to zap the appropriate data sets to restore their pristine condition before resubmitting the FASTBILD job.
Though you are only halfway through the batch work at this point, this step is very important for exceptionally large builds. It gives you a means of recovery back to this point in case the second job fails (but see step 6 below).
See your SPIRES consultant for help in using data set copy-and-restore utilities such as ORVCOPY. [See 2.1.4.4.]
If you are using FASTBILD only to build indexes, you may just want to copy the index data sets.
During the first job, FASTBILD generates the JCL for the second job. By default, it is placed on the scratch volume. Regardless of the default, however, you can find it by examining the JCL from the first job, looking for lines that look like this (substitute your own account for "GQ.JNK"):
//GO.JCLFILE DD DSN=WYL.GQ.JNK.FASTJCL,VOL=SER=SCR002, // DISP=(NEW,CATLG),UNIT=SYSDA,SPACE=(6320,(2,2),RLSE), // DCB=(BUFNO=1,RECFM=FB,LRECL=80,BLKSIZE=1600)
The first line gives us the information we want: in this case, the JCL is stored on the scratch volume SCR002, as data set WYL.GQ.JNK.FASTJCL. You should get a copy of it from there and save it somewhere more permanent, for future reference.
If you are building a very large subfile, you may want to make changes to the JCL before running the job. Here are some changes to consider:
//SORTOUT DD DSN=WYL.GQ.JNK.REC020,DISP=(OLD,PASS)
and
//GO.REC02 DD DSN=WYL.GQ.JNK.REC020,DISP=(OLD,PASS)
As you did in step 4, be sure you don't have any subfiles of the file selected. Then put the JCL into your active file and issue the RUN command to submit the second FASTBILD job:
-> clear select -> run hold JOB 3265 JNK1FAS EXECUTING CLASS=B PRIO=6 ->
Again, wait for the job to complete. If you have several jobs to run, wait till each has completed before submitting the next.
If a job fails, it means that one of the indexes has not been built properly. Error messages should tell you what went wrong. You may need to restore the file to its state in step 5, or to simply build the failed index later. Your SPIRES consultant can give you some guidance here.
Basically, you have finished the procedure. You should test the file in SPIRES to be sure it is working properly. Select the subfile, issue the SHOW SUBFILE SIZE command to see if its size matches your expectations, and use the new indexes to find and display some records.
Once you are satisfied that the file was built properly, you should make a backup copy of it, for safety's sake, using ORVCOPY. [See 2.1.4.4.]
The STATUS command will give you a set of statistics profiling your file in peak condition. These statistics are very useful for extremely large files, since they can be compared to STATUS command statistics later to see how the file's efficiency is holding up. [See 3.2.3.]
In general, most of the problems that arise during a FASTBILD task were discussed under the appropriate steps.
One common problem that has several solutions is the truncation of index entries, which happens when a value to be indexed is longer than the maximum length specified for entries for that index (see step 1b). Warning messages about these values appear in the output from the first job.
One solution, if the allowed length was fairly long (such as the default length of 25) is to add the $MAX.LEN system proc to the index's Searchproc rule string, telling SPIRES to truncate the user's value at 25 characters for matching purposes.
If that solution doesn't appeal to you, you will need to remove the records with the truncated index values (the output will tell you which ones they are) and add them back in again. You will probably need to add the $MAX.LEN proc to the index's Passproc before you remove the records so that SPIBILD can create the proper index records to be removed. Then process the file, remove the $LSTR proc from the Passproc, recompile the file definition and add the record(s) back in again.
A more serious problem arises if the truncated value is the pointer back to the goal record. In that case, you absolutely must start again, specifying enough space for the longest key that can arise.
With a great deal of skill and experience in writing JCL, you could write your own JCL for FASTBILD jobs. However, just as File Definer takes care of the most tedious parts of creating a file definition, the public protocol FASTBILD.JCL does the same for FASTBILD JCL.
Using the protocol is just one part of the FASTBILD procedure, described in the previous section. [See 2.12.2.] It is not too difficult to use, though it asks you some questions that require you to be very familiar with the data you want to input and the file definition. When it is finished, it leaves the JCL in your active file, ready for you to run or to alter as necessary.
Warning: Pressing the ATTN/BREAK key at a prompt will usually stop the protocol, but it will also discard the work you've done so far. If you want to pause for a moment (e.g., to respond to a message), type a question mark at a prompt and then press ATTN/BREAK. This causes a session break, and you will be under WYLBUR's control. To return to the same place in the protocol, type the command GO. While you are pausing, leave your active file alone; the protocol is putting the JCL there as it executes.
Pressing the Return key in response to most prompts will give you more information about the question being asked, but beware of questions that ask you to press Return to get the default value.
The rest of this section discusses most of the questions the protocol will ask you.
If you answer YES to this question, the protocol will display a few lines of explanation before most of the questions it asks.
The data you want to input must be the goal records for a subfile. Also, the file containing the subfile must already exist, i.e., be compiled.
If you are building an entire subfile or all the indexes of an existing subfile, answer NO. If you are building a single new index for records that already exist in the subfile, answer YES to this question; the protocol will then ask you for the name of the index record-type you are building.
You need to estimate how many goal records you are adding (or passing to the index being built). This estimate doesn't have to be very accurate -- if anything, it should be a bit high. If it is too low, FASTBILD may not allocate enough space for the OS data sets for the index data, which could cause the job to fail.
Steps should be considered if you project that the ultimate build will take longer than 10 hours. [See 2.12.1.] Using steps gives you the capability of backing up to the previous step if the build fails at any point. The technique is not extremely difficult, but it is procedurally demanding.
In essence, the input is broken into as many as nine groups. Each group is associated with one step. A step is a single job that builds the goal record-type using a group of records. It also builds the OS data sets for the index records. When that job concludes, you make a copy of the file and the OS data sets, after which you submit the next job for the next step. That step adds the next group of records to the goal record-type and adds the new index data to more OS data sets, etc.
When the job that sorts and builds the index records begins execution, it combines the various OS data sets for a single index before sorting them.
Use the steps feature only if you are working in close cooperation with your SPIRES consultant. [A variation of FASTBILD that works with steps, called RESDS FASTBILD, was used to build a 4,000,000-record data base belonging to the Research Libraries Group. Essentially, it builds the entire file in OS (including the key tree) and then other jobs are run to move the file into the ORVYL file system.]
FASTBILD is guessing how much CPU time and lines of output the job will use. These numbers are used by the system monitor in scheduling jobs for execution. If the job exceeds the estimates, it will stop, leaving a partially built file. So in both cases, you should estimate high. Remember, you are charged only for the amount of CPU you use, not for your estimate.
The numbers are shown as "n" minutes of CPU and "n-thousand" lines of output. (LINES=3 means 3000 lines.) A good rule of thumb would be to at least double these. (Don't double your bin number though.)
If you plan to follow the procedure described in the previous section, answer NO to this question. Generally, it is a good idea to stop after the first job(s), check the output from it, and examine the created goal record-type before continuing on to the index-building job (i.e., the "step" referred to in the question; it does not mean "steps" as in the earlier question).
Here you name the data set containing the input data for the subfile. [The protocol does not ask this question if you are building a single index rather than a whole subfile, but it will if you are building all the indexes of a subfile using SCAN GOAL RECORDS -- simply make up an answer now and change the JCL later, as described in "Step 1f" in section 2.12.2.] It assumes the input is in a cataloged OS data set on disk for permanent storage (not on a scratch volume). If the data is elsewhere, such as in an ORVYL data set created by the GENERATE LOAD command or on tape, you will need to change the JCL later. [See "Step 1f" in section 2.12.2.] For now, type any value you want, but remember to change the JCL appropriately as the procedure in the previous section describes.
If you have more input data sets to use for this step, answer YES, then respond to the next prompt with the name of the next one; otherwise, answer NO. You can specify up to 16 input data sets. Note, though, that you will need to make some significant changes to the JCL to get FASTBILD to use them properly. [See "Step 1f" in section 2.12.2.]
For the sort step, FASTBILD needs fixed-length fields. Here, and perhaps in the next few questions, you will tell it the lengths to use for each field for each element being passed to the indexes. The system default is very generous for most circumstances; in fact, if 25 bytes is much longer than the length of the longest value, you might waste some processing time sorting empty bytes. That's why it's good to provide more accurate numbers. Answer NO to begin.
If you answer YES, the next question you will be asked is "How many index entries..." below.
The "Common section" are the elements that will each be passed to all the indexes being built, in other words, the pointer group, as defined in the Global Parameters of the linkage section. Thus, you would supply the maximum length for the pointer and for any global qualifiers you are passing. If you are passing locators as pointers, remember that they are always 4-bytes long; but if you are passing keys, you will need to do some analysis to determine their maximum length. It is crucial that you don't declare too small a size for the pointer -- if in doubt, specify a larger number. The maximum allowed size for a value being passed in SPIBILD is 255 bytes.
The protocol will show you the elements of the pointer group so you'll know how to specify the numbers. Enter the numbers in the order shown, separated by commas, with the lengths of any global qualifiers entered in order of their definition.
The protocol will prompt you for field sizes for each index record-type to which the goal record-type passes records. It will show you which elements in the index record-type are receiving passed data, according to the linkage section. Again you will supply numbers representing the maximum length for the values being passed.
The first number always represents the key element of the index. For simple indexes, that's the value being passed. For compound indexes, it's a special internal value that SPIRES creates -- its length is always 2.
At some point after that will come the "Common section" again; that is, you must enter the letter "c" to represent its positioning in the index data. If the index has sub-indexes, they will precede the Common section; if it has local qualifiers, they will follow the Common section. For a compound index, the length of the longest element value being passed to the index is specified after the Common section, like a local qualifier.
The maximum allowed size for a value being passed in SPIBILD is 255 bytes.
This question, also asked for each index, requests the average number of index entries an average record will create for passing to this index. Precise accuracy is not crucial, but again, you should estimate high rather than low. Keep in mind that a word index may take a single element and split it into dozens of index entries.
Note: If you are building a single index, you will be asked no further questions.
If the protocol finds that the subfile has slot keys, it will ask you this question. Answer NO if the input data contains the slot key values; answer YES if the contrary is true.
You can request an exception file, where FASTBILD will place input records that fail to go into the subfile. An exception file simplifies the process of finding the unsuccessful records and correcting them so that they can be put into the subfile. [See 2.1.5.2.] It has the same restriction that it has in SPIBILD: if the input lines exceed 235 characters, they will be truncated.
Be aware that exception file processing requires considerable overhead. In general, it is extremely useful; but it is expensive in terms of both CPU and I/O, and will slow down the processing.
The protocol will finish up, leaving the generated JCL in your active file. Save it somewhere if you're ready for a rest; otherwise, you might want to continue with the procedure for using Fastbild, altering the JCL if necessary, as described in Step 1f of the previous section.
If you examine the JCL generated by the FASTBILD.JCL protocol [See 2.12.3.] you will see about a dozen non-JCL commands at the end, the GO.SYSIN dataset. Those are FASTBILD commands, which you may want or need to change or delete. [See 2.12.2.] You may also find that additional commands available in FASTBILD are useful to you as well.
The entire list of recognized commands appears below, listed in the order in which they may appear in the JCL:
The DEFINE command specifies the name of the file that contains the goal record to be built. This command must be present and must precede the FAST BUILD command.
ESTIMATE is the command that specifies the CPU time estimate and printed output lines estimate for the second job. This command is optional, and defaults apply if it is not specified. The default will generate a 1 minute CPU time estimate and a 2-thousand line estimate (LINES=2 means 2,000 lines of output). Warning: If either of these parameters is exceeded, the job will terminate, leaving a partially built file; be sure to estimate high.
STEP indicates that the goal records are being added in steps. The "n" indicates which step is being specified. This optional command should only be used if very large files are being built. To indicate the last step, the command is STEP=n,L. This feature should be used only with the assistance of the SPIRES consultant.
This command specifies that the input for the goal records is in ascending key order. When this optional command is specified for non-sequential input the job will terminate; if it is specified for slot goal records, a FASTBILD error will occur.
Remember that you shouldn't use FASTBILD unless your data is in sequential order by key.
This optional command requests that a message be printed for each record added; the default is not to list adds, since large numbers of records are usually added in a FASTBILD run. Error messages are always listed, unless you turn them off with a SET MESSAGES command (see below). Warning: If you specify that adds be listed, be certain to allow enough lines in the ESTIMATE command above.
This command, along with its options and alternate forms (SET FORMAT *, or SET FORMAT format.name 'parm', for instance), tells SPIRES to set the named format to be used for input processing. Be aware that, unlike SPIRES but like SPIBILD, FASTBILD will not execute any startup frame in the format except when the SET FORMAT * form of the command is used. [See the manual "SPIRES Formats", chapter C.9, for more information.]
You may also include SET FTRACE or SET FTRACE ALL at this point, if debugging information is needed. This may generate substantial amounts of output, of course.
The following vgroup manipulation commands may also be issued at this point:
ALLOCATE RESTORE STATIC SHOW STATIC VARIABLES SHOW ALLOCATED SHOW EVAL
The syntax of these commands can be found by issuing appropriate EXPLAIN commands, e.g., EXPLAIN ALLOCATE COMMAND. Other vgroup manipulation commands may be issued at the end of the FASTBILD procedure (see below). Note that vgroups established by setting the format will not be included in SHOW STATIC VARIABLES or SHOW ALLOCATED displays because the format is not actually set until the building begins.
This command directs rejected input records to an "exception" sequential ORVYL file where they can later be manipulated for subsequent addition to the SPIRES file. [See 2.1.5.2.]
Several other SET commands may be issued at this point, each of which is described elsewhere in this manual (see the index) or in other SPIRES documents (use the EXPLAIN command). They are:
SET INDEX SET PASS SET FLAG SET NOINDEX SET NOPASS SET IFTEST SET SHARE SET SEARCH SET CRYPT SET NOSHARE SET NOSEARCH SET SEED SET FILE COUNTS SET STACK SIZE nK SET [ALL|INFORMATION|WARNING|SERIOUS] MESSAGES [=] n
Also, the CLEAR FILE COUNTS command is available to turn off the default file count settings.
This required command indicates the name of the subfile to be built. FASTBILD selects the subfile using the privileges of the named account.
This command is used when a single index is being built from existing goal records. BUILD should not be included if FAST BUILD (above) is, and vice versa.
Because index record information is typically of variable length, and OS SORT cannot handle variable length fields, you have the option of a default field length of 25; or, you may enter specification parameters indicating maximum field sizes of a particular index. If data being passed is longer than the lengths specified, it is truncated at the appropriate length and a message is produced.
The SPECIFY GLOBAL command indicates the field sizes of the common global data, that is, the pointers to all goal records (or keys if the records are not Removed) and any global qualifiers. (GLOBAL is referred to as COMMON in the FASTBILD.JCL generator.)
The "p" indicates the maximum length of the data you expect in the global pointer or key field. If pointers are used, this length is 4; otherwise the maximum length of the goal-record key should be specified.
The "q" field(s) indicate the maximum length of the data to be passed to the global qualifier(s). Since there may be several such qualifiers, multiple "q" values may be specified in the order in which the qualifiers appear in the file definition.
The maximum allowed size for a value being passed in SPIBILD is 255 bytes.
The second type of specification must appear once for each index being built, indicating the field sizes of the data to be passed.
"RECn" is the record number of the record being specified. If the index records are the second, third and fifth record-types define in the file definition, then RECn would have the values REC2, REC3, and REC5. If an invalid record type is named, an error message is issued and the build will terminate.
The "k" field is mandatory and specifies the maximum length of the data that will be passed to the key of the index. If a compound index is being specified, then the key length is always 2. The "v" parameter must also be specified, to pass the values of elements to the index. The "s" field appears if and only if a sub-index is defined for the record-type. The "s" value specifies the length of data to be passed to this sub-index.
The word "GLOBAL" instructs FASTBILD to insert the GLOBAL specifications indicated in the first specification card. If GLOBAL is put in the wrong position, an error message is issued and the build is terminated.
The multiple "v" parameters specify local qualifiers. As many "v" parameters should appear as local qualifiers are defined for the index record-type. This parameter is also coded to pass element values to the CINDEX-VALUE element of a compound index. The "v" value is then the size of the longest element value passing to the compound index.
If a simple index has several INDEX-NAME sections passing to it, then several SPECIFY RECn commands must be supplied. One specification is necessary for each INDEX-NAME section that passes to the index; the specifications for the INDEX-NAME sections are made in the same order in which the sections appear in the file definition's linkage section. Other restrictions in this area: those INDEX-NAME sections must appear consecutively in the linkage section, and the first one declared there must pass the most terms (i.e., sub-index and local-qualifier terms, if applicable).
Accuracy in specifying the size of the fields is desirable. A smaller field size than the 25-byte default will speed up the OS sort, since the comparisons are across shorter fields, and will speed up the build, since a greater number of records will fit in the buffer. However, if field size is underestimated, items will be truncated in the indexes, an error message will be issued, and the build will continue.
The maximum allowed size for a value being passed in SPIBILD is 255 bytes.
The INPUT command specifies that the batch data input is on IN0000. IN0000 is the "ddname" on the DD card in which the input dataset is described in the OS JCL; other DD names can be used. Up to 16 data sets may be declared for the single "ddname"; they will be concatenated together during the processing.
If the data being entered into the subfile was created by a GENERATE LOAD command, this command, rather than the INPUT command above, should be issued.
If you are using FASTBILD only to build all the indexes for an existing subfile, you use this command instead of the INPUT command above. (A very specialized command, SCAN GOAL KEYS, retrieves only the record key of each record, useful if all the data for passing will come from the key itself.) This command has some auxiliary commands that can be issued before it that let you break up the task into multiple steps:
More vgroup manipulation commands are allowed here:
SHOW ALLOCATED SHOW STATIC VARIABLES SHOW DYNAMIC VARIABLES SHOW EVAL STORE STATIC DEALLOCATE
This command is generated by the first job if indexes were built, and appears in the second job of the FASTBILD run. If no index records are to be created, this command does not appear, and no second job will be run.
By building the tree data sets carefully, SPIBILD does a good job of maintaining the efficiency of the average SPIRES file. However, because no one file is likely to be maintained with perfect efficiency, it may be worthwhile for a file owner or manager to occasionally check up on SPIBILD by monitoring various statistics available for the file. These aspects of file ownership are usually of concern to owners of large files or of files with many, many update transactions.
In some cases, the statistics may show that the file has empty space in it that SPIBILD may not be able to use properly for some time, or that a high amount of data entry for records with a given key prefix is making a tree lopsided, which could cause some inefficiency when SPIRES retrieves those records. These problems, and others to be covered in this chapter, are seldom critical ones, but they often have a longterm effect on the overall cost of storing and using a file. Using procedures described in this chapter, you can correct these problems if you decide it is worthwhile to.
These descriptions will tell you not only how to correct the problem, but will also try to give you some idea of whether the "problem" is worth correcting. These topics involve some rather esoteric concepts and features in SPIRES, and the effort and the expense of learning about them and solving them may surpass the value of correcting them.
This chapter discusses various ways you can measure and, if desirable, improve the storage efficiency of your SPIRES files. In specific, it presents commands:
- to tell how much storage space the file's data sets are using [See 3.2.1.]
- to tell how much of that space is being used for data [See 3.2.3.]
- to tell whether the empty space in blocks will be used appropriately as transactions occur, and to correct the situation if they will not [See 3.4.]
- to tell how much of the file's storage space is internal overhead [See 3.2.2.]
- to tell whether the record trees are built well for maximum retrieval efficiency, and to correct the situation if they are not [See 3.3.]
These commands are all available in SPIRES. In versions of SPIRES from mid-1987 through mid-1991, they were available only in SPIUTIL, a separate program containing several maintenance utilities for SPIRES files. SPIUTIL has been completely incorporated into SPIRES and is no longer needed; if you invoke it, it just puts you into SPIRES.
To determine whether your file is in good condition, you need statistical information about it: how many physical blocks of storage space is it using; what percentage of each block contains data; what is the average depth of each key tree, etc. This chapter will present several commands that gather and display various sets of statistics, and will give you some guidelines for reading them.
Keep in mind that if you own a small file, you probably do not have to gather and examine this information. And even if you own large files, you may not need to concern yourself with it -- but if you are concerned about your file running as efficiently as possible, you should examine these statistics occasionally.
Specifically, the subsections of this chapter cover:
For budgetary purposes, you may need to know how many blocks of storage your file uses; after all, you are charged for the number of blocks you use.
No single command in SPIRES will tell you how large a file is, but an ORVYL command, SHOW FILES, can tell you how many blocks each data set of a file is using. The example below shows how account GQ.DOC would find out how much space the GQ.PUB.RESTAURANT file is using:
-> show files like restaurant, blocks ORVYL File System RESTAURANT.CKPT -- 43 BLKS (F: 0, L: 42) RESTAURANT.DEFQ -- 50 BLKS (F: 0, L: 49) RESTAURANT.MSTR -- 5 BLKS (F: 0, L: 4) RESTAURANT.REC1 -- 4 BLKS (F: 0, L: 3) RESTAURANT.REC2 -- 36 BLKS (F: 0, L: 35) RESTAURANT.RES -- 112 BLKS (F: 0, L: 111)
By adding up the numbers of blocks (BLKS) shown, you can determine the storage size of the file in ORVYL 2048-byte blocks. To get the blocks information, include either the BLOCKS or ALL option on the SHOW FILES command. See Part 6 for more information, or online, [EXPLAIN SHOW FILES COMMAND.]
To find out how many blocks your account is using as a whole, issue the ORVYL command SHOW BLOCKS:
-> show blocks 6477 Blocks accounted, 10000 Blocks allowed ->
There will be a difference between the total number of blocks accounted and the total blocks listed for all data sets. To that total, you must also add one block for each 660 blocks of each data set; these blocks are not part of the data set per se, but are stored in ORVYL's index blocks and are charged to you. Non-IXR data sets do not have these blocks. Online, [EXPLAIN GENERATE NONIXR COMMAND.] for more information, or see its description in Part 6.
Knowing how much space a goal record is actually using can sometimes be useful in making estimates of storage space requirements for a large file. The SHOW RECORD OVERHEAD command can tell you how many bytes of storage are being used by a single record, by a sample of about 25 records, or by all the records in the selected subfile.
SHOW RECORD OVERHEAD [record-key|ALL]
The example below demonstrates the command:
-> select employees
-> show record overhead
- Overhead: 15.995 Percent
26 Record(s) scanned. Total length 8613 bytes
-> show record overhead 226-82-3846
- Overhead: 16.557 Percent
1 Record(s) scanned. Total length 511 bytes
->
The most significant figure is the length value at the end of each display. The sample group of 26 records (requested by omitting the option) has a total length of 8613 bytes. The single record 226-82-3846 has a length of 511 bytes.
The length value tells you how much space the data in the record (plus the length and occurrence information kept within the record as specified in the file definition) is using. For this command, the length and occurrence information represent the "overhead".
The same information is also available with the SHOW RECORD INFORMATION command, which has the additional advantages that it can be used in Global FOR mode to process records by criteria other than the record key, and that it can process multiple records, rather than only a single record. See Part 6 for more information about these commands, or online, [EXPLAIN SHOW RECORD OVERHEAD.]
Procedure: STATUS Procedure
Purpose: To collect/display status information about the
contents of a file's data sets
Who can use it: file owner
users with See access to the file
Basic steps:
1) ATTACH filename
2) Issue the STATUS command.
3) Examine and evaluate the results.
Though the STATUS command's most important function is arguably to read and validate the contents of all the blocks of one or more data sets, it also gathers and presents a slew of statistics about the file as it proceeds. The statistics are saved in the file's residual data set, where the SHOW FILE STATUS command can retrieve them to present a summary of the status data. [Much of the data you will get from the STATUS command is also accessible from the SHOW FILE STATUS command. However, the data displayed by SHOW FILE STATUS is only as recent as the latest STATUS command.]
The tables may strike you as boring or fascinating, but in either case, interpreting them correctly can give you some insight into how efficiently your file is growing and may suggest changes to the file you might want to consider. In addition, your SPIRES consultant may ask you to "run a STATUS" to provide some additional information if you ever have serious problems with a file.
Though this procedure is quite simple to execute, it is not meant to be done casually. SPIRES must read every block of the file, which may be quite a bit of work -- the procedure may be expensive. Unless you need the results pronto, it is worthwhile to run the STATUS when your computing rates are the cheapest.
Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:
-> attach orv.gq.doc.restaurants ->
This example demonstrates using the ATTACH command with the file GQ.DOC.RESTAURANTS.
Basically, the STATUS command has the following syntax:
IN ACTIVE [CLEAR|CONTINUE] STATUS ALL [AVSP]
Though it is really optional, you are urged to include the IN ACTIVE prefix, so you're not trying to read and interpret the lengthy display of data as it scrolls off your terminal screen. It's preferable to put the statistics in your active file and examine them there or print them.
It's also a good idea to add the AVSP option, which stands for AVailable SPace. Then the STATUS command also displays information about how much empty space exists in the residual's blocks and how much may be "misplaced". [See 3.4.]
You don't have to issue the STATUS command for ALL the data sets of the file -- ALL can be replaced by other options, described in detail in Part 6 of this manual. [ALL does not include the deferred queue, however. If you need to find out the average depth of the deferred queue's tree, you can replace ALL with DEFQ, as described in the back.]
If the STATUS command encounters any problems with the file (e.g., a tree is approaching the maximum depth, or a block is improperly written), it reports them as part of the statistics.
In terms of commands to execute, the procedure is now complete. If you followed the advice here and put the statistics in your active file, you might want to print it now.
The statistics about the residual data appear first, followed by data about each record-type in the file, in sequential order by name (e.g., REC01, ZIN02, ZIN03, etc.). The data shown for each record-type varies depending on whether it is slot or non-slot.
Only SPIRES consultants and the most enthusiastic SPIRES aficionados will be interested in all the details shown, so they will not be explained in detail here.
The most useful data from the STATUS command concerns these areas:
- errors uncovered while validating the blocks
- balanced tree growth
- available space
The errors are discussed below. The other two areas are discussed in their own chapters following this one. [See 3.3, 3.4.]
Remember that some of the most interesting information captured by the STATUS command can be seen through the SHOW FILE STATUS command. See Part 6, or [EXPLAIN SHOW FILE STATUS COMMAND.]
In the STATUS command's output, error messages may appear that identify blocks in a data set that have serious problems detected by the STATUS command. Use the EXPLAIN command to get more information about the errors, and if so advised, contact your SPIRES consultant for help as soon as possible.
Remember that the SPIRES command reads every block of the residual data set in order, validating the block's contents, i.e.,
- verifying that the block's "trailer" (data at the end of the block that identifies the length and starting location for each record or record segment in the block) does not contain overlapping entries, and that the trailer values are valid;
- verifying that the first four bytes of the block match the last four (a "check number" based on an ID for the transaction that last modified the block);
- verifying that other block numbers appearing in the block (e.g., the current block number, the number of a block where the current record continues, etc.) are valid.
Another error message you might receive if you are working with a large Residual data set is:
Processing Table Overflow - The following data is incomplete.
which signals that SPIRES could not verify all the links between blocks in the residual because of memory limitations.
If the system crashes at any point during the procedure, then restart the procedure at your convenience.
If SPIRES detects problems with your file as it compiles the statistics, it will report them to you in the statistics as described above. In other words, problems within the file will not stop the STATUS command.
Occasionally, a tree-structured record-type becomes "unbalanced", that is, the tree has more levels than it needs, resulting in slower retrieval of records in the lower levels. It is usually caused by unusually extensive growth in one particular area of the tree. Suppose your subfile has a DATE index, which started out with a few dates scattered around the twentieth century but which recently only gets current dates. Hence, the tree may be very deep (have many levels) around the current dates and very shallow around the older ones. If that's the case, the record-type may need rebalancing.
If the record-type is quite unbalanced (or extremely large), it may even reach the maximum tree depth of 16 levels. [SPIBILD will warn you during file-processing if any record-types of the file are getting close to the limit, with the message: "Warning: Tree depth of <recname> approaching limit". If SPIBILD processing would surpass the limit, the processing stops with an error, and the record-type must be rebalanced before SPIBILD recovery will work.]
What does a depth of 16 mean, and why should you be concerned? It means that SPIRES must read 16 blocks (i.e., execute 16 I/Os) to retrieve records at that level in the tree. (And if the desired record is removed to the Residual, add at least one more.) If most of the other records of the file reside at much higher levels, requiring less than, say, three I/Os for retrieval, then SPIRES is doing a great deal more work for the lower-level records than is necessary. Rebalancing the tree might mean that all records would require less than three I/Os, which would lead to a significant efficiency savings each time the records were retrieved.
For files containing record-types of, say, less than 2000 records, you should probably just wait till the unlikely day when you receive the SPIBILD warning message about the tree-depth limit. If and when that day comes, you should rebalance the file immediately.
For larger files, you should check the status of the trees occasionally (perhaps every few months) or even frequently for very large files (50,000 records and up).
A formula to help you determine whether a tree is balanced or not uses information displayed by the SHOW FILE STATUS command (as gathered by the most recent STATUS command for the file). [See 3.2.2.] In particular, compare the number of accesses per record to the tree depth for each tree-structured record-type. (The data for slot record-types, for the Residual and for the Deferred Queue is irrelevant in this context.) Note that the number shown for accesses per record includes any access to retrieve any Removed records in the Residual.
Here are the relevant columns and rows from a SHOW FILE STATUS command:
Record ... Accesses Tree Type / Record Depth INDEX2 1.897 2 INDEX3 1.831 2 INDEX4 2.000 1 INDEX5 2.335 5
INDEX5 has a much lower average number of accesses per record than its tree depth, suggesting it as a likely candidate for rebalancing.
A more graphic and definitive indicator that a record-type needs rebalancing appears in the output from the STATUS command, in the second table displayed for each record-type. Here's one for a large index record-type:
12/21/86 Record-type 6, Data Set REC2
Record REC06 of File EC.PRD.PCAT
...
Level Nodes Blocks (Terminal)
1 77 1 0
2 5904 78 44
3 178401 2517 2508
4 12927 186 185
5 5470 91 90
6 1296 26 26
Base block 81
Average Accesses per Record 4.108
The pattern to look for is a rise and fall in the number of nodes and blocks as the tree gets deeper. (The deeper the tree, the higher the level number -- thus, level 1 is the top-most level.) If the fall is greater than 1 level, then the record-type could benefit from rebalancing. In the above example, the numbers drop from level 3 to level 6. Ideally, the numbers should rise, or the last level should be the only one to drop from the previous one.
Note however that of the approximately 200,000 records (nodes) in the example record-type, only about 6700 are in levels 5 and 6, the "unnecessary" levels that rebalancing would eliminate. You need to consider whether the improved retrieval efficiency for 3 percent of the records in the record-type is worth more to you than the savings from not rebalancing the record-type.
Consider these ideas: First, the lower-level records are likely to be newer records (i.e., more recently added) than the higher-level ones. If users will be doing most of their searching for newer records, then the number of I/Os will be higher than necessary for most searches. Second, if the tree is unbalanced now, it is likely to become more unbalanced rather than less as time goes on.
Rebalancing a tree provides other benefits as well:
Though a single record-type may need rebalancing, all the record-types in the data set containing that record-type get rebalanced too. In other words, the REBALANCE command applies to data sets, not to individual record-types.
The remainder of this chapter describes the REBALANCE procedure. The first section describes the steps of the procedure, while the second discusses how to handle unusual situations and problems that can arise during rebalancing.
Procedure: REBALANCE procedure Purpose: Rebalance the record trees within a data set Who can use it: file owner Basic steps: 1) SPIRES 2) ATTACH filename 3) SET NOJOBGEN 4) IN ACTIVE SHOW FILE ACTIVITY 5) REBALANCE DATA SET RECn, RENAME 6) Check the status information of the rebalanced data set. 7) SET JOBGEN
Suppose you decide you need to rebalance the ERRATA.REC2 data set of your file ERRATA. When you issue the REBALANCE command with the RENAME option, SPIRES takes the following three actions:
- 1) makes a copy of each tree in the data set, rebalances it and places the copy in a new data set (ERRATA.REB2);
- 2) renames the old data set containing the tree(s) to "ERRATA.REO2.yyyymmdd"
("yyyymmdd" is the current date, e.g. "19861231" for 12/31/86) and
renames the new data set to ERRATA.REC2. Thus the new replaces the old, but the old is
kept around in case of any problems.
- 3) resets the base-block numbers in the ERRATA.MSTR data set of the file, using the new values established when the data set was rebalanced.
Additional steps may be necessary when a problem occurs during the second part of the REBALANCE procedure. Information on the procedure involved in this circumstance appears in the next section. [See 3.3.2.] Online, EXPLAIN REBALANCING DATA SETS, SPECIAL CIRCUMSTANCES.
Below is the REBALANCE procedure, step by step, to be followed by the file owner. To understand the syntax, assume you are rebalancing the ORVYL data set "filename.RECn" (where "n" is an integer from 1 to 9, or letter from A to F) of the SPIRES file "gg.uuu.filename".
The REBALANCE command will create a data set with the default ORVYL permit setting of Public Read (or Public Write, for data sets containing immediate indexes). That's the same setting SPIRES uses when it creates the "RECn" data set.
However, if you, the file owner, ever changed the permit, you will probably want to reinstate that level of security when the REBALANCE procedure is over. To that end, issue the command:
SHOW PERMITS filename.RECn
If the data set has permits other than the appropriate default, jot them down so that you can reset them at the end of this procedure.
You should not have any ORVYL data set called "myfile.REBn" or "myfile.REOn.yyyymmdd" ("yyyymmdd" being today's date) when you issue the REBALANCE command, since SPIRES will create them. If either does exist, the procedure will stop. To see if you do, issue these ORVYL commands:
SHOW FILE LIKE myfile.REBn SHOW FILE LIKE myfile.REOn.yyyymmdd
If you do have either of these, you should either erase or rename the data set, using the ORVYL commands "ERASE dataset.name" or "RENAME dataset.name TO new.dataset.name". (You will have to issue the ORVYL command "SET CLP dataset.name" prior to the ERASE or RENAME command if the data set was created by SPIRES.)
WARNING: Since the REBALANCE procedure replaces both the data set being rebalanced and the MSTR data set of the file with new copies, at times it needs exclusive access to the file. It is best to rebalance data sets when you are sure no other users have any subfiles of the file selected; the procedure will go much more smoothly for you. In fact, if practical, ask users of your file not to use it during a given period of time when you are rebalancing it. You might even want to temporarily remove the file definition from the FILEDEF subfile so that no users can select any of its subfiles during the event.
But it is crucial that the trees being rebalanced not be changed during the REBALANCE procedure -- that means that the file should not be processed by SPIBILD during it, nor should any users update records during it if any trees being rebalanced are immediate indexes affected by the updates. Step 3 of the procedure will guard you against users updating any records while you are doing your work, but you are on your own to ensure no one processes the file while you are rebalancing.
You must be in SPIRES to use the REBALANCE command:
-> spires -Welcome to SPIRES ->
Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:
-> attach orv.gq.doc.errata ->
This example, which will be followed through this section, demonstrates using the ATTACH command on the file GQ.DOC.ERRATA.
As mentioned above, this command will prevent any users from updating records during the procedure.
-> set nojobgen
It is really necessary only if any users logged on might update any records and if the file has any immediate indexes. But it cannot prevent any users with Process access from processing the file, which would cause major problems. Again, be certain no one would do that while you are rebalancing the data set.
In the unlikely event of a problem with the rebalanced data set, you may need to return to the "filename.REOn.yyyymmdd" data set, in which case you will need the base-block information from this display. Having a copy of this information is very important, so you should put it in your active file and save it somewhere till after you have successfully completed the procedure.
This command will rebalance the trees of all the record-types stored in the named data set, storing them in a new ORVYL data set called "myfile.REBn".
RENAME, though really an option, is highly recommended -- if you omit it, you will have to issue other commands to complete the process, described in the next section. [See 3.3.1.] The remainder of this procedure assumes you have used the RENAME option.
The display from the REBALANCE command (which goes into your active file if you add the IN ACTIVE prefix) varies from file to file, but it should look something like this one for REC2 of GQ.DOC.ERRATA, which contains three record-types:
-> rebalance data set rec2, rename
Rebalance Data Set REC2
-Base block 1 in record-type REC02
-Next block is 20
-Base block 21 in record-type REC03
-Next block is 138
-Base block 139 in record-type REC04
-Next block is 297
- REBALANCE complete - Start RENAME process
FIX rebalanced data set REC2
-REC02 base is 1; old base was 0
-REC03 base is 21; old base was 1
-REC04 base is 139; old base was 2
- Next block is 297; Next block was 346
ORV.GQ.DOC.ERRATA.REC2 is renamed to ORV.GQ.DOC.ERRATA.REO2.19861122
ORV.GQ.DOC.ERRATA.REB2 is renamed to ORV.GQ.DOC.ERRATA.REC2
-Set new data set permits if needed.
-File detached
->
SPIRES will also display a message each time 64K nodes have been processed, to reassure you (especially if the record-types are large) that the rebalancing activity is occurring.
If the display ends with the "File detached" message, continue with step 6. If it did not, there may be several reasons why:
- you received the message "New data set exists; OK to continue?" because there is already a "filename.REBn" data set. You should type NO and return to the "Preliminaries" above.
- the display stopped after the "FIX rebalanced data set" message with an error code of S56, because the file is being used by others. You must have exclusive access to the file before the renaming part of the procedure can succeed. You will need to continue with the FIX REBALANCED DATA SET command when you have exclusive access. [See 3.3.2.]
- the display stopped after the "Next block is, Next block was" line with an error code of S592, because there is already a "filename.REOn.yyyymmdd" data set. You should rename or erase that data set (You may have to SET CLP for it first; see step 4.), and then continue with the FIX REBALANCED DATA SET command. [See 3.3.2.]
- some other error has occurred. Use the EXPLAIN command to get more information about the error.
Issue the ATTACH command again ("ATTACH filename" -- see step 2), followed by the STATUS command:
[IN ACTIVE [CLEAR|CONTINUE]] STATUS RECn
If this command completes successfully with no errors (it almost always does), then the rebalance procedure has been successful. But if error messages do appear, you can replace the new RECn data set with the old, which is still saved as "filename.REOn.yyyymmdd". The "recovery procedure" is discussed in the next section. [See 3.3.2.]
When you are sure the rebalance succeeded, issue the SET JOBGEN command to allow users to update records again. If necessary, select a subfile of the file again first.
-> set jobgen ->
To clean up after successful rebalancing, you should get rid of the old data set, which you no longer need. Issue the ORVYL commands "SET CLP filename.REOn.yyyymmdd" and "ERASE filename.REOn.yyyymmdd".
If there were special permits set for the data set (see "Preliminaries" above), use ORVYL's SET PERMIT command to reset them:
SET CLP filename.RECn SET PERMIT filename.RECn FOR account access-level
The standard rebalancing procedure described in the previous section does a great deal of work; if problems occur in the middle of the procedure, it could be quite expensive to restart at step 1. This section will discuss ways to continue the procedure in the event of problems.
Many parts of the discussions in this section assume you are already familiar with the procedure discussed in the previous section. [EXPLAIN REBALANCE PROCEDURE to see that section online.]
The most common problems occur because either:
- you don't have exclusive access to the file when renaming the "filename.RECn" and "filename.REBn" data sets; or
- a data set named "filename.REOn.yyyymmdd" already exists (probably left from a previous rebalancing that day) when the REBALANCE command is issued.
In both situations, "filename.REBn", the new data set with the rebalanced trees, has been created; SPIRES was unable to replace the old data set with the new one though.
Recovery from both these problems is relatively easy; both problems are handled similarly:
- 1) If the problem is the leftover "filename.REOn.yyyymmdd" data set, get rid of it by either erasing it or renaming it:
- First, SET CLP filename.REOn.yyyymmdd
- Then, either ERASE filename.REOn.yyyymmdd
- Or, RENAME filename.REOn.yyyymmdd TO new.dataset.name
- 2) Issue the FIX REBALANCED DATA SET command to complete the rebalancing:
FIX REBALANCED DATA SET RECn, RENAME
If you do not have exclusive access to the file, the command will fail. However, failure at this stage means simply issuing the FIX REBALANCED DATA SET command again when the file is available. [Remember not to wait too long, however. If the file is processed, or if other activity changes any trees in the rebalanced data set, you will have to start the entire rebalance procedure again.] Be sure to end the procedure as shown in the previous section, continuing with step 5. [See 3.3.1.]
In very rare situations, the REBALANCE or FIX REBALANCED DATA SET command completes with no sign of problems, but the STATUS command fails, indicating a problem with the rebalanced data set. To recover from this problem, you need to get back to the original data set of trees (now stored as "filename.REOn.yyyymmdd"):
- 1) Issue these commands to switch back to the old data set:
SET CLP filename.RECn
RENAME filename.RECn TO filename.REBn
RENAME filename.REOn.yyyymmdd TO filename.RECn
SET PERMITS filename.RECn ....
(set the same permits that were there before)
SET NOCLP filename.RECn
- 2) Reset the base blocks to their original values. To do this, you must have the information from the SHOW FILE ACTIVITY command (issued before you rebalanced the data set) nearby, since you will need to type numbers from that display:
ATTACH filename
FIX BASE OF RECn
The FIX BASE command allows you to reset the base block numbers for each of the trees in the data set. You type the old numbers from the second column of the SHOW FILE ACTIVITY command as they are requested. This part of the session looks like this:
-> FIX BASE OF RECn record-name1 base is a1 - Type new base: b1 record-name2 base is a2 - Type new base: b2 ... ... ... -File detached ->
For example,
-> fix base of rec2 ZIN02 base is 1 - Type new base: 1 ZIN03 base is 20 - Type new base: 2 (etc.) -File detached ->
- 3) Reset the "next block" figure for the data set, using the FIX NEXT BLOCK
command. First, you need to determine the old number by consulting the third column
("Blocks used") of the SHOW FILE ACTIVITY information you got prior to the
rebalancing. The relevant portions of the display might look like this:
Record Blocks Type ... Used -DS ------ -------- REC01 - Slot ... 194 -1 ZIN02 ... 928 -2 ZIN03 ... Comb -2 ...
If REC2 is the data set you are working with, look for the number "2" in the DS column, and pick up the number next to it under "Blocks Used" -- in the example above, that's the number 928. So the "next block" value should be reset to 928. [In the example, REC2 contains 928 blocks, whose block numbers run from 0 to 927; hence, the "next block" is number 928.]
To do that, issue the ATTACH and FIX NEXT BLOCK commands:
ATTACH filename
FIX NEXT BLOCK OF RECn
The FIX NEXT BLOCK command resets the number SPIRES will assign to the next block added for the data set. You will be shown the current number and asked to supply a new one, like this:
-> fix next block of rec2 - Next block is 890; type new block: 928 ->
The recovery process is now complete. However, you should attach the file to perform a STATUS command again. Should difficulties again show up, you should contact your SPIRES consultant immediately.
Procedure: REBALANCE DEFQ procedure Purpose: Condense the deferred queue, or rebalance its tree Who can use it: file owner Basic Steps: 1. SPIRES 2. ATTACH filename 3. REBALANCE DEFQ
Occasionally in a very large file, the deferred queue can approach or even reach its limit of 64K blocks, or, if there are many transactions in a particular key area, the deferred queue's tree could become too deep. (The deferred queue tree can have 16 levels, the same as the normal record tree.) The purpose of REBALANCE DEFQ is to provide a way to get out of some conditions that could bind up a file if the defq gets too full or the tree gets too deep.
The REBALANCE DEFQ procedure will rebuild a Spires file DEFQ and rebalance the defq tree. As a result, the defq will generally be reduced in size because:
- All transactions are packed together in the defq blocks so that there is very little empty space in the defq.
- The defq tree is compacted using the FASTBILD logic so that the tree has about 90% space utilization.
- If several transactions have been processed for a single record, only the latest is kept.
- Transaction group processing records are eliminated. (This has no effect on previously completed transaction groups except that SHOW SUBFILE TRANSACTIONS no longer shows the transactions in groups.)
Note: a special version of the REBALANCE DEFQ command (the TREE option) can be used solely to rebalance the defq's tree, in which case only the second point above is true.
The REBALANCE DEFQ command will usually create a data set with the default ORVYL permit setting of Public Write. That's the same setting SPIRES uses when it creates the DEFQ data set.
However, if you, the file owner, ever changed the permit, you will probably want to reinstate that level of security when the REBALANCE DEFQ procedure is over. To that end, issue the command:
SHOW PERMITS filename.DEFQ
If the data set has permits other than the appropriate default, jot them down so that you can reset them at the end of this procedure.
You should not have any ORVYL data set called "myfile.DEBQ" or "myfile.DEOQ.yyyymmdd" ("yyyymmdd" being today's date) when you issue the REBALANCE DEFQ command, since SPIRES will create them. If either does exist, the procedure will stop. To see if you do, issue these ORVYL commands:
SHOW FILE LIKE myfile
If you do have either of these, you should either erase or rename the data set, using the ORVYL commands "ERASE dataset.name" or "RENAME dataset.name TO new.dataset.name". (You will have to issue the ORVYL command "SET CLP dataset.name" prior to the ERASE or RENAME command if the data set was created by SPIRES.)
WARNING: Since the basic REBALANCE DEFQ procedure replaces the DEFQ with a new copy, at times it needs exclusive access to the file. You might want to use the SET SUBFILE DISABLE FOR ACCESS and SET SUBFILE DISABLE FOR PROCESS commands to prevent any further activity while you are working on the deferred queue.
However, if getting exclusive access to the file is not possible, but you must rebalance the deferred queue because of a tree-depth problem, you can rebalance only the tree, which does not replace the entire deferred queue. See the TREE option on the REBALANCE DEFQ command. Note, however, that it is not generally recommended that you try any form of rebalancing when the file is being used by others.
You must be in SPIRES to use the REBALANCE DEFQ command:
-> spires -Welcome to SPIRES ->
Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:
-> attach orv.gq.doc.errata ->
This example, which will be followed through this section, demonstrates using the ATTACH command on the file GQ.DOC.ERRATA.
There are several flavors to this command:
[IN ACTIVE [CLEAR|CONTINUE]] REBALANCE DEFQ [RENAME|TREE|COPY]
REBALANCE DEFQ by itself will create a new defq called "Filename.DEBQ" on your account. If you wish to replace the primary defq, you must do this manually.
REBALANCE DEFQ RENAME will not only create a new defq (called "filename.DEBQ"), but then it will replace the old DEFQ data set with the new one, saving the old one aside. (If you omit the RENAME option, you can follow this command with the FIX REBALANCED DATA SET DEFQ RENAME command to do the renaming in a separate step.)
REBALANCE DEFQ COPY is similar to RENAME, except that the new DEFQ data set replaces the old one, which is not saved.
REBALANCE DEFQ TREE will rebalance only the defq tree within the existing defq. The process is very fast and is used primarily to rebalance a tree that is too deep. Note that a STATUS DEFQ will give false errors following this process since both the old and new rebuilt trees are in the defq. Since this command does not replace the deferred queue but simply rewrites data into it, and only a relatively small part at that, it is quite quick. It does not generally change the size of the deferred queue but does solve the tree-depth problem and is the best choice if that is the only problem with the deferred queue.
The example continues:
-> rebalance defq copy
Rebalance DEFQ data set
-Base block 1 in DEFQ data set
-Next block is 324
-Begin DEBQ copy.
-Ended DEBQ copy.
->
To clean up after successful rebalancing, you should get rid of the old data set, which you no longer need. Issue the ORVYL commands "SET CLP filename.DEOQ.yyyymmdd" and "ERASE filename.DEOQ.yyyymmdd".
If there were special permits set for the data set (see "Preliminaries" above), use ORVYL's SET PERMIT command to reset them:
SET CLP filename.RECn SET PERMIT filename.RECn FOR account access-level
Turn file access back on with the SET SUBFILE ENABLE FOR ACCESS and SET SUBFILE ENABLE FOR PROCESS command, if you disabled the file with their SET SUBFILE DISABLE counterparts.
This process will be called RESIZEING the database. The steps from start to finish are outlined as follows: The 4K file account should have MASTER File-access permission or at least COPY access to the 2K database.
1. On the 4K account, build a clone File definition of the database to be RESIZEd. Be sure to add the statement FILE-BLOCK = 4K; at the end of the 4K file definition.
2. Logon to the 4K account. Select the 2K database and perform the following command for each RECn data set.
REBALANCE DATA SET RECx RESIZE
This rebalance option can be used to rebalance a tree data set in a 2K file system into a 4K file system using (and modifying) the MSTR data set of the 4K file system account.
3. While logged onto the 4K account, SELECT the 2K database and issue the command:
COPY DATA SET RESIDUAL TO NewFilename RESIZE
This command is a special form of the COPY DATA SET process that is designed to deal with the conversion from a 2K residual to a 4K residual. The 2K blocks will be read from the selected database, expanded and written to the 4K residual.
There are two options available in case the Residual is extremely large and you wish to convert in more "bytesize" chunks.
LIMIT = block-count -- copies the number of blocks specified. CONTINUE -- continues the COPY on subsequent steps.
4. Logon to the 4K account, SELECT the 4K database and issue the following command for each RECn dataset that was rebalanced:
FIX REBALANCED DATA SET RECn RENAME
This will replace all the RECn datasets with clones of the 2K versions. You could do this before step 3 above.
5. Once steps 3 and 4 have been completed you may use the new database if you wish. However, there is another step which will reclaim the unused space and should be carried out as soon as possible. This process is a special version of AVSP RECovery and should be run until the whole file has been dealt with
AVSPREC RESIZE [CONTINUE] [LIMIT = block-count]
Remember that you have the NORESERVE option that you may use if there is no possibility of Immediate Indexing taking place and you want users to be able to update while AVSPREC is occurring.
This process is a special form of the wonderfully named AVSPREC command. This command will read every block in a Residual data set that has been "RESIZEd", looking for records that are split to multiple blocks. These records are essentially updated in place in order to utilize as much available space as possible in the existing blocks.
Be sure to use the CONTINUE and possible LIMIT options the second, third and subsequent times through the process.
Note: This set of commands does not speak about DEFQ updates. Once the REBALANCEs and COPY DATA SET commands have begun you must ensure that no Immediate Indexing or PROCESSes take place on the 2K database. You may do this by DISABLE-ing UPDates. If you need to synchronize the DEFQs at the end of this procedure, you can SELECT the 2K database and issue the
REBALANCE DATA SET DEFQ
command while logged onto the 4K account.
Then SELECT the 4K database and issue the command:
FIX REBALANCED DATA SET DEFQ RENAME
This chapter discusses how SPIBILD stores records in the residual data set of a file, and how that method can cause available storage space in the residual to be "misplaced". By properly reading the AVSP (AVailable SPace) data from the STATUS command, you can tell how much space has been misplaced. Then, if appropriate, you can use the AVSPREC command to make it available to SPIBILD again.
Is the material in this chapter useful to you now? The best way to tell is to examine the "Space Utilized" value for the residual shown in the STATUS command's output. If its value is less than 90 percent and you haven't issued the AVSPREC command recently, it is quite likely that SPIBILD has misplaced some of the available space. Of course, the larger the file, the more space (and cost) that unused percentage represents -- you are being charged for the space, but SPIBILD isn't using it.
The sections of this chapter are:
[Author's note: This is a technical discussion explaining the purpose and implementation of "available-space chains". The sections that follow describe how to determine whether they need repair and how to repair them -- you don't need to know the information here to handle those tasks. It's like changing the oil in a car -- you don't need to understand how the engine works to do it yourself, but you might want to know.]
When SPIBILD puts a new copy of a record into the residual data set, it tries to find an empty spot in the residual into which the record will fit. If the record is too big for the available space in any block of the residual, SPIRES will add another block to the residual and add the record to that.
SPIBILD keeps track of available space in the residual with a scheme called "available-space chains". There is a chain of blocks for each discrete amount of available space (in 64-byte increments) in a block. For example, the five blocks each containing 2048 bytes of available space would form one chain. Another chain would link, say, the three blocks containing 1984 free bytes, etc.
Each block in the residual holds the location of another block in the residual containing the same amount of free space (except for the last block in a given chain). The location of the first block in a chain is kept in a table in one block of the residual, at a place representing the number of bytes of free space in each block of the chain. Thus the chain runs from the first block, whose location is kept in the table, to the second block, whose location is kept in the first block, etc.
When SPIBILD needs to put a record away in the residual, it determines how many bytes of storage are needed and then goes to the chains. It always goes to the chain having the largest amount of space available (even if there is a chain representing the exact amount of space). SPIBILD retrieves the first block of the chain and places the record into it. It then reads the location of the second block of that chain and puts that into the table, thus making the second block the first.
The block into which the new record was placed now has less free space in it. SPIBILD determines how much space that is, examines the table to find the chain having that much space, places the location of its first block in the block with the new record, and then puts the location of that block into the table, thus making it the first record in the new chain.
Here is an example that might help make that more clear:
Available Space Table
---------------------
| Available Block |
| Space Number |
| --------- ------ |
| 960 bytes 10 | --> block 17 --> block 39 --> block 93
| ... |
| 640 bytes 23 | --> block 41 --> block 86
| ... |
---------------------
The table (inside the box) shows the first block number of two chains -- the first chain, starting with block 10, contains 960 free bytes in each block. To the right are shown the other members of the chain. Thus, blocks 10, 17, 39 and 93 are in the "960 bytes" chain.
Assuming there are no blocks containing more free space than 960 bytes, suppose that SPIBILD needs 320 bytes to store a new record. It will go to the table, retrieve block 10, and put the record into it. Since block 10 contains a pointer to block 17, SPIBILD will replace the number 10 with 17 in the table as the first block of the 960 chain.
Since block 10 now contains 640 free bytes, it needs to be placed on the 640 chain. SPIBILD places the number of the first block on that chain (23) into block 10, and then places the number 10 into the table as the first block on the 640 chain. At the end of this work, the diagram would look like this:
Available Space Table
---------------------
| Available Block |
| Space Number |
| --------- ------ |
| 960 bytes 17 | --> block 39 --> block 93
| ... |
| 640 bytes 10 | --> block 23 --> block 41 --> block 86
| ... |
---------------------
This process works very smoothly for added records, but not so smoothly for updated and removed records. When SPIBILD is processing a record that's been updated or removed, it finds the block containing the old version of the record, which may be in the middle of an available-space chain. For example, suppose you have removed a record that resides in block 41, which is in the 640-byte chain shown above.
After removing the record and thus creating a larger amount of available space in block 41, SPIBILD puts the block back but does not move the block to a larger available-space chain, unless the block is the first one on a chain. (Similarly, if an updated record were larger but there was enough free space to put it back in the same block as the old version, SPIBILD would put back the block but would not connect it to a smaller available-space chain unless it heads a chain.) [Why doesn't SPIBILD usually move the block to its correct chain? The reason is that relinking the old chain would add considerable overhead to the process (at least one more block would need to be read and rewritten for each update).]
The result is that some blocks on a chain may have more (or less) available space in them than their presence on that chain indicates. This is not a problem to SPIBILD -- if a block with the incorrect amount of space works its way to the head of a chain, then when SPIBILD tries to use the block and finds that it doesn't contain enough free space, SPIBILD will simply put the block on the correct chain and then get the next block.
In fact, what makes it a problem is that it can degrade the percentage of space used in the residual. Suppose SPIBILD removes a record from a block on the "64-byte" chain so that the block now contains 1000 free bytes. It could be many months or years (if ever) before that block is used -- remember, for added records, SPIBILD always uses the block with the most free space that has room for the entire record. (Chances are slim that SPIBILD would ever have only a 64-byte record to put away and simultaneously have only available-space chains for 64 bytes and fewer.) If other records in the block get updated and grow, the free space within it will get used, but otherwise, the space is wasted.
For many files, it is useful to rebuild the available-space chains occasionally, using the AVSPREC command. [See 3.4.3.] By putting blocks on the correct chain, SPIBILD insures the likelihood that the blocks with the most free space in them will be used to hold more data. [Documentation and the AVSPREC command "verb" itself talk about "recovering" the lost space, but there is no guarantee that the command will cause the space to be used; it simply rebuilds the chains, increasing the likelihood that blocks with lots of free space that were on wrong chains will be used again.]
The AVSPREC command can be quite expensive, since it requires SPIRES to read every block in the residual and write most of them. Before doing it, you should determine as best you can that the available-space chains need rebuilding and that the reclaimed space will be used. The data that will help you with this decision is provided by the STATUS command. [See 3.4.2.]
The previous section described how SPIBILD uses and maintains available-space chains, which keep track of blocks containing given amounts of free space. For example, among the many chains for a given residual data set, there may be a 320-byte chain consisting of blocks that have 320 bytes of available space. [See 3.4.1.]
As explained in that section, the chains may become inaccurate, containing blocks with more or less space than they are supposed to have. This can lead to free space that isn't used when it should be.
This becomes a problem only when a significant portion of subfile transactions are updates and removes; the higher the percentage of updated and removed records, the more likely (and the faster) the chains are to become inaccurate.
By keeping an eye on the percentage of space used and on the available-space table, you will know when to use the AVSPREC command to help increase the percentage of space used. You can get those statistics from the STATUS command, though you must include the AVSP option to get the available-space table. [See 3.2.2.]
The Space Utilized figure given by the STATUS command for the residual data set is a percentage indicating how much of the residual is being used to store data. The difference between that and 100 percent is the amount of "available space", that is, space that can be used to hold data. [The SHOW FILE STATUS command also has the Space Utilized figure, but remember that it is only as current as the most recent STATUS command.]
It is exceedingly unlikely that the Space Utilized value will ever reach 100 percent for your file; for most well-tuned files, the value should be between 90 and 100 percent. If the value is lower than 90 percent, then there's a good chance that some of the free space is in blocks on the wrong available-space chain. (It's not necessarily the case because the number will be exactly the same after you rebuild the chains with the AVSPREC command; it may be awhile before the percentage begins to increase.)
If the Space Utilized value is under 90 percent, you should examine the statistics about the available-space chains produced by the STATUS command when you include the AVSP option.
Here is a condensed example of those statistics, for a file that would benefit from the AVSPREC command:
Available Space Chain Data
Bytes Blocks Blocks Blocks on
Available (Actual) on Chain Wrong Chain
1936 1
1896 1
1880 1 1
1768 2
1720 2 3 3
...
776 2 2
768 2 2
760 1 1
744 4 5 2
...
0 84 83 9
Reading this chart correctly is admittedly a challenging task. [There are plans to revise it in a future version of SPIRES.]
The second column, "Blocks (Actual)", lists the number of blocks that actually have the amount of free space indicated in the first column. Hence, there is "actually" one block in the residual that contains 1936 bytes of free space.
If a chain is accurate, then the second column, "Blocks on Chain", will have the same number as the first column, and nothing will appear in the third. "Blocks on Chain" tells you how many blocks are on the chain for the given amount of space. Since there is no value given in that column for 1936, there is no chain for 1936 bytes. That means that the block with 1936 free bytes must be on the wrong chain.
The last column, "Blocks on Wrong Chain", is a subset of the the "Blocks on Chain" column. For 1720 bytes, the chart shows that 3 blocks are on the chain but all 3 are on the wrong chain -- that is, none of them contain exactly 1720 bytes of free space. That means that the 2 blocks that actually contain 1720 bytes of free space must be on other chains, where they don't belong.
By subtracting the number of "Blocks on Wrong Chain" from the number of "Blocks on Chain", you can determine how many blocks on a given chain belong there. So for the 744-byte chain, the number of blocks on the chain that actually have that much free space is 5-2, i.e., 3. If you then subtract that number from the number of "Blocks (Actual)", you'll know how many blocks with that much space are on wrong chains, which in this case is 4-3, or 1.
If you discover that many of the blocks containing large amounts of empty space (say, as large or larger than the average record size) are on the wrong chains, it is probably worthwhile to rebuild the chains, using the AVSPREC command. [See 3.4.3.]
Procedure: AVSPREC procedure Purpose: Rebuild the available-space chains Who can use it: file owner Basic steps: 1) SPIRES 2) ATTACH filename 3) AVSPREC
This procedure, which rebuilds the available-space chains for the residual data set, can be fairly expensive, since every block in the residual must be read and many blocks must be rewritten in order to correct the chains. You are urged to wait till your computing rates are the cheapest possible (e.g., during the night or on weekends).
To determine whether your file will benefit from rebuilding the chains, see the previous section. [See 3.4.2.]
You must be in SPIRES to use the AVSPREC command:
-> spires -Welcome to SPIRES ->
Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:
-> attach programs ->
The AVSPREC command will rebuild all the available-space chains of the residual data set. Because all of the blocks in the residual must be read (and most must be rewritten), it may take several minutes for SPIRES to complete the command.
-> in active clear avsprec ->
SPIRES then puts a list of all the chains and the number of blocks on each into your active file. The list looks like this:
01/09/1987 File: GQ.JNK.PROGRAMS
Available Space Chain Data
Bytes Blocks
Available on Chain
1984 12
1920 1
1856 1
1792 1
1408 1
...
This list will now match the first two columns of the AVSP display from the STATUS command -- the number of blocks actually containing a given amount of free space will match the number of blocks on that chain.
The most likely problem you would have with the AVSPREC command is a system failure in the middle of it. Even that is not much of a problem -- simply restart the procedure at your earliest convenience, but add the CONTINUE option to the end of the command. See Part 6 for details, or online, [EXPLAIN AVSPREC COMMAND.]
Most often, how a file is being used determines whether and how the file is changed. For instance, your analysis of how your users find records in a given subfile might uncover the fact that one of the indexes is scarcely used at all or that many searches need to look for records using an unindexed element. Knowing these facts, you might decide to eliminate the unnecessary index and create a new index for the other element; in the first case, you will save unnecessary storage costs for the index, and in the second, you will save your users from doing costly and inefficient searches.
This part of the manual discusses various techniques that will help you keep track of what users are doing with your file. The possibilities vary from collecting statistics on the number of records in each record-type to collecting all commands and responses that users type when they have your subfile selected.
Of course, don't forget that one of the best ways to find out how your file is being used is to talk to your users directly, to ask them what changes to the file would make it easier and better to use.
Here are the subjects of the rest of Part 4:
This chapter discusses ways to monitor the record transactions that users are making to your file. Here you are interested in the contents of the data as well as the transactions that create and change it. This contrasts with Part 3, much of which was concerned with the physical size of the data rather than its contents.
There is a wide variety of commands and procedures to help you with this task. Those that are described in this chapter are:
For various reasons, you may want to examine the records involved in subfile transactions before the file is processed, applying the transactions to the tree. For instance, for the sake of security, you might want to save a copy of all records being removed before they actually disappear from the subfile when the file is processed.
This section discusses in general the techniques SPIRES provides to give you access to the records involved in subfile transactions. Most of the details about the individual commands are covered in other manuals, such as "Sequential Processing in SPIRES: Global FOR" and "Searching and Updating".
You may want to include these techniques in a batch SPIRES or BATWYL job that precedes a job to process your file. [See 2.1.3.]
Use the Global FOR class ADDS to retrieve records being added.
For example:
-> select albums
-> for adds
+> set format adds summary
+> in active display all
+> show level
Processed Examined
FOR ADDS 31 31
+>
Here copies of the 31 added records (as seen through a custom format named ADDS SUMMARY) have been put into the active file.
Remember that the ADDS class contains the records in the deferred queue whose keys don't match any records in the tree. That means, for example, that if a user adds a record and then another updates it before the file has been processed, it is still considered an ADD to SPIRES.
For updated records, there may be two different versions of the record that you're interested in: the version in the tree (the "old version") and the version in the deferred queue that's going to replace it.
To get the new version of each updated record in the deferred queue, use the FOR UPDATES command in SPIRES, just as you use the FOR ADDS command for added records. To get the old version of each, create a stack of the updated records and then use the VIA TREE prefix on the TYPE command.
The example below demonstrates both of these techniques:
-> select albums -> for updates +> stack all -Stack: 15 RECORDS +> in active clear type +> save updates.new.versions UPDATES.NEW.VERSIONS saved on INT001 +> via tree in active clear type +> save updates.old.versions UPDATES.OLD.VERSIONS saved on INT001 +>
A slightly different method was used to get copies of the new versions of the updated records than was for the added records. A stack of keys of the updated records was created first. The first TYPE command put copies of the new versions of the updated records in the active file -- remember that the TYPE command by default retrieves the latest copy of a record. The second TYPE command, with VIA TREE, placed a copy of the old version of each record in the active file.
Keep in mind that the Global FOR class of UPDATES consists of records in the deferred queue that are to replace records currently existing in the tree.
When records are removed, SPIRES puts only the keys of the records in the deferred queue. To see the old version of the records being removed, you need to use the same technique just described for the old version of updated records: create a stack of the removed records, and then use the VIA TREE prefix on the TYPE command to see them.
For instance:
-> select albums -> for removes +> stack all -Stack: 3 RECORDS +> via tree in active type +>
Copies of the three records being removed are now in the active file.
The REMOVES class of records consists of records whose keys (as well as the rest of each record) are being removed from tree. Hence, if you remove a record and then add a new record using that key before the file has been processed, the record will appear in the class of UPDATES, since in sum, SPIBILD is going to replace the old version in the tree with a new one.
Several other Global FOR classes may be useful to you as well:
For more information about Global FOR classes, see the manual "Sequential Processing in SPIRES: Global FOR". Further information about the VIA TREE prefix appears in the SPIRES manual "Searching and Updating".
Several commands are available to you when you need basic statistics about a subfile's size and the transactions that got it there.
The basic command to discover how many records are in a subfile is the SHOW SUBFILE SIZE command:
-> select restaurant -> show subfile size -The subfile has 407 records ->
Although called "SHOW SUBFILE SIZE", the command can be issued anytime you have a record-type of a file attached in SPIRES. It will tell you how many records there are in the attached record-type.
The same value is also available through the $SUBFSIZE system variable.
You can get that information for all the record-types in the attached file using the SHOW FILE STATUS command, available to the file owner and users with SEE access to the file.
In fact, that information is there regardless of whether a STATUS command has ever been issued for the file:
-> show file status
01/12/87 Status Record of File GQ.DOC.RESTAURANT
Last Compiled: 07/23/86 at 15:37:08
Record Blocks Space Number Accesses Tree Date
Type Used Utilized Records / Record Depth
ID -Slot 4 60.944 407 2.000 0 12/09/86
INDEXC 1 5.088 8 2.000 1 12/09/86
INDEX2 11 72.624 611 1.897 2 12/09/86
...
Residual 112 83.600 481 0.000 0 12/09/86
Def Queue 2
->
A breakdown of the ADDS and REMOVES that create the "Number Records" figure above appears in the output from the SHOW FILE ACTIVITY command, available to the file owner and users with SEE access to the file.
-> show file activity
01/12/87 Activity Record of File GQ.DOC.RESTAURANT
Last Compiled: 07/23/86 at 15:37:08
Record Base Blocks Adds Deletes Updates Max Rec
Type Block Used -DS Length
ID -Slot 0 4 -1 617 210 1469 1347
INDEXC 0 33 -2 8 0 2330 6129
INDEX2 1 Comb -2 852 241 559 487
...
Residual 2 112 695 214 5264 6133
Def Queue 1 2
->
Every request to add, update, merge, remove, dequeue or unqueue a record in SPIRES creates a transaction record in the deferred queue of the affected file. The transaction record contains information such as the type of transaction, the date and time the transaction was completed and other information used by SPIRES and SPIBILD to process and track the transaction request. Some of this information is displayed by the SHOW SUBFILE TRANSACTIONS command.
File owners occasionally need to manipulate the information in these transaction records in various ways. For example, you might want to trace all updating activity done from a certain account or terminal, or want to sort the transaction records according to a particular element in the transaction records and generate reports. In other words, you may need to handle transaction records just as if they were records in any SPIRES subfile, using the full complement of SPIRES commands for processing them.
This chapter introduces a special record-type for "holding" transaction records. The transaction records in this record-type exist only until the file is processed in SPIBILD -- then they are discarded. Thus, they are not meant to be kept permanently but to represent the current transactions for the file.
This section is organized as follows:
The easiest way to get basic information about the transactions in the deferred queue is with the SHOW SUBFILE TRANSACTIONS command. Available to the file owner and users with SEE access to the file, the command displays several pieces of transaction data for each record of the selected subfile in the deferred queue.
Here is a sample of the output from the SHOW SUBFILE TRANSACTIONS command:
-> select filedef
-> show subfile transactions
01/13/1990 Transaction Log of File GG.SPI.FILEDEF
For Record Type REC1
File last processed: 01/13/1990 at 02:37:38
Date Time Account Id Type Command Grp Key Value
01/12/90 09:10:26 GQ.WCK UPD Update GQ.WCK.IOTEST
01/12/90 09:15:42 GE.BXK S17 UPD Update GE.BXK.VOLUNTEERS
01/12/90 09:31:30 CT.GIB S17 ADD Add CT.GIB.VOLUNTEERS
01/12/90 09:35:48 AN.NSI N33 UPD Update AN.NSI.DOC
01/12/90 09:37:04 AN.NSI N33 UPD Update AN.NSI.DOC
01/12/90 10:00:50 GQ.WCK DEQ Dequeue GQ.WCK.IOTEST
01/12/90 10:30:04 GA.SDS UPD Update GA.SDS.SOFTDOC
01/12/90 11:54:02 GQ.PBT U29 UPD Update GQ.PBT.TEXAS
01/12/90 12:06:33 AU.RAG REM Remove AU.RAG.REPORT
01/12/90 12:11:24 AN.NSI N33 UPD Unqueue AN.NSI.DOC
01/12/90 16:39:12 CN.MVS UPD Update CN.MVS.BIO.SERIALS
01/12/90 16:55:38 GA.DRM CU12 ADD Add GA.DRM.BIO.EXAMP
01/12/90 16:57:02 GA.DRM CU12 ADD Update GA.DRM.BIO.EXAMP
01/12/90 17:01:12 GA.DRM CU12 DEQ Remove GA.DRM.BIO.EXAMP
01/12/90 17:35:00 GQ.JNK ADD Add 1 GQ.JNK.TRANS.GROUP
01/12/90 17:37:00 GQ.JNK Decommit 1
01/12/90 17:45:22 GQ.JNK ADD Add 2 GQ.JNK.TRANS.GROUP
01/12/90 17:45:32 GQ.JNK Commit 2
This display shows the transactions of the FILEDEF subfile (the selected subfile) since the last time the file was processed in SPIBILD. Some of the columns are self-explanatory, though four may need a word of explanation here. All of them are explained in detail in Part 6; online, [EXPLAIN SHOW SUBFILE TRANSACTIONS COMMAND.]
The "Id" column is the terminal ID if the user has one set.
The "Type" column refers to the four basic types of transactions that can appear in the deferred queue for any given record:
- ADD -- a record with a new key that does not exist in the tree;
- UPD -- a record that is going to replace another with the same key in the tree;
- REM -- a request to remove a record in the tree, without replacing it with another record with the same key; and
- DEQ -- a request to ignore any earlier transactions for records with that key.
The next column, "Command", names the command the user issued to cause the transaction to occur. "Addupdate" will not appear -- depending on whether the record was in reality added or updated, "Add" or "Update" will appear instead. It's important to understand the relationship between the Type and the Command. Keep in mind, for example, that a record that is added and then updated before the file is processed is still treated as an ADD since the updated version of the record will go into the tree as a new record: no record with that key exists in the tree.
The next column, "Grp", refers to the number of the transaction group if transaction group processing was in effect for that transaction.
You can use the SHOW SUBFILE TRANSACTIONS command to find out about an individual record by adding its key value to the end of the command. It is also useful in conjunction with the Global FOR command FOR TRANSACTIONS, since it can help you identify a particular copy of a record you're trying to retrieve from the deferred queue.
If the SHOW SUBFILE TRANSACTIONS does not meet all your needs, you will probably need to use the $TRANSACTIONS record-type processing described in the next sections. [See 4.2.3.2.]
To use all the transactions data kept in the deferred queue by SPIRES, you must make some simple changes to your file definition. The basic procedure, which must be done by the file owner, is this:
The transaction data exists in every file, regardless of whether it has a $TRANSACTIONS record-type in it or not. What the $TRANSACTIONS record-type does is reformat the data for use as if it were another record-type of the file, as described above. No new records or additional storage overhead are created by the $TRANSACTIONS record-type -- the data is already kept in the deferred queue until overnight processing anyway.
To treat transaction data as if it were its own record-type in a file, you must code a special record-type in the file definition. It must be done to the file definition directly; File Definer does not support the needed changes.
Get the file definition from the FILEDEF subfile, and look for the end of the last record-type defined, just before the first linkage section. After the last record-type, add the new record definition, which should look like this:
RECORD-NAME = .name; DEFINED-BY = $TRANSACTIONS;
where "name" is a one- to five-character name, such as REC08, preceded by a period. The period declares the record-type to be a DEFQ-only record-type, a record-type whose data is kept only in the deferred queue and discarded when the file is processed. See the manual "SPIRES File Definition" for more information, or online, [EXPLAIN DEFQ-ONLY RECORD-TYPES.]
Because SPIRES arranges the record-types in alphanumeric order, the transactions record-type will always be the last (or one of the last, along with other DEFQ-only record-types) in the file.
Add a Subfile section for the new record-type as well. It names the new record-type as the goal record-type:
SUBFILE-NAME = subfile name;
GOAL-RECORD = .name;
ACCOUNTS = gg.uuu, gg.uuu, ...;
(etc.)
This allows you to select a subfile of the file to use the transaction data. (Alternatively, you can attach the transactions record-type directly to use the transactions data.) Be aware that access to this subfile means access to keys of all records in the deferred queue for all record-types in the file (though not to the records themselves).
The $TRANSACTIONS record-type may not be named in the linkage section either to pass values to other record-types or to receive values. Because it is a DEFQ-only record-type, its records are only temporary -- they are discarded during SPIBILD processing of the file.
Once you have made these changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
To recompile the file definition, issue the RECOMPILE command when you have the FILEDEF subfile selected:
RECOMPILE [gg.uuu].filename
For example,
-> recompile .almanac -File definition compiled ->
If the file definition does not compile, use the error messages to determine the problem; then correct the file definition, and try the procedure again.
The $TRANSACTIONS record-type is available as soon as the file is compiled.
Each record in the $TRANSACTIONS record-type represents a single transaction against a record-type in the file. All updating transactions, regardless of record-type, will have a record in the $TRANSACTIONS record-type.
The $TRANSACTIONS record definition has the following elements:
TREC, TDATE, TTIME and TKEY together form the structured key of the $TRANSACTIONS record-type.
By adding the $TRANSACTIONS record-type to your file, you gain the ability to treat the transaction data almost as if it were any other record-type in the file. For example, as covered in the previous section, you can declare the $TRANSACTIONS record-type to be the goal record-type of a subfile. [See 4.2.3.2.]
There are restrictions on the use of the record-type, however, which are based on these facts:
- No data may be passed to or from the $TRANSACTIONS record-type by SPIBILD or through immediate indexing. The records thus cannot be indexed, nor can they be retrieved through index-searching commands such as FIND.
- The records in the $TRANSACTIONS record-type don't exist in "tree" or "defq" versions but as "transactions". That means that they can only be processed in Global FOR, under the FOR TRANSACTIONS command.
The $TRANSACTIONS record-type is commonly used as a more versatile means for handling transaction data than the SHOW SUBFILE TRANSACTIONS command. For example, you can write custom formats or use system formats such as $REPORT to display the data in a fashion more useful to you. In the sample session below, the $REPORT format is used to display transactions for the first record-type in the file (TREC = 1):
-> select transactions data -> for transactions where trec = 1 +> set format $report Tstime (1,8) Tcmd(+2,7) Tkey (+2,20) +> display all January 14, 1987 Page 1 Tstime Tcmd Tkey -------- ------- -------------------- 11:15:32 Add 415-497-4420 11:15:46 Merge 415-497-4420 14:22:02 Remove 800-325-3535 15:00:00 Dequeue 800-325-3535 +>
To sequence the transactions data, you need to create a stack first, with the STACK command under FOR TRANSACTIONS. You may use the SEQUENCE, TYPE or ALSO commands to process the stack. You may not, however, process it under FOR STACK. For example,
-> for transactions where tacct = gq.jnk +> stack all -Stack: 13 RECORDS +> sequence trec ttime -Stack: 13 RECORDS +> in active clear type
You may want to examine the actual records that were added, updated, etc., creating the transaction records. You can do this in either direction: from the goal records, you can access the transaction records; and from the transaction records, you can access the goal records. Let's start with retrieving transaction records from goal records.
You can retrieve a transaction record from a goal record only when you are processing goal records under a FOR TRANSACTIONS command. Using FOR TRANSACTIONS, often with a VIA clause, you connect to the transactions data in one of four ways:
- using a phantom structure defined in the goal record definition, which points to the $TRANSACTIONS record-type
- defining a dynamic phantom structure which does the same thing, but which can be created whenever the user wants;
- using subgoal processing in a format, invoked through the SET SUBGOAL TRANSACTIONS Uproc;
- retrieving element values from the $TRANSACTIONS record-type with the $LOOKSUBG or $LOOKSUBF function, called from within a format.
This is useful when you need to carefully monitor changes made to records, particularly when they can change several times a day. Under FOR TRANSACTIONS, you can see each copy of the record as it was added to the deferred queue. With FOR TRANSACTIONS VIA DEFQ (or with one of the other VIA clauses) you process the records in key order, so all copies of the same record are processed one after the other, beginning with the original tree copy if the record starts out there (e.g., for an update).
With the tie to $TRANSACTIONS, you can connect additional information about the transaction to the goal record for producing reports. Consider this example, where the connection to the $TRANSACTIONS record-type is made with a dynamic phantom structure:
-> select albums -> for transactions via defq +> define element xyz subfile "album trans" via trans as @key +> set format $report key album.title tcmd tstime +> display all Key Album.Title tcmd tstime ------ -------------------------------- ------- -------- 256 Differsdorf's Greatest Higs 256 Dittersdorf's Greatest Figs Update 09:13:42 256 Dittersdorf's Greatest Hits Update 09:14:14 2644 Rossini's 50th Birthday Bash Add 13:48:28 2644 Remove 13:50:52
Notice in the last line that a removed record doesn't have any data other than the key in the deferred queue (aside from the data about the transaction from $TRANSACTIONS), so the ALBUM.TITLE field isn't available there.
You could add a WHERE clause to the FOR TRANSACTIONS VIA DEFQ command that would examine data values in $TRANSACTIONS, as long as the phantom structure is defined. For instance, if you use the TCOMMENT element [See 4.2.3.2.] to save information about the specific change made by an Update transaction, you could establish a Global FOR class like this:
-> for transactions via updates where tcomment 'Price changed'
That would let you find the different defq versions of a record whose "price" had changed when updates were made.
For more information about the FOR TRANSACTIONS command, see the SPIRES manual "Global FOR"; online, EXPLAIN FOR TRANSACTIONS. For information about defining phantom structures in a file definition, see the SPIRES manual "Technical Notes", section 23.1; online, EXPLAIN PHANTOM STRUCTURES, CODING. Chapter 23 of "Technical Notes" also has information about defining dynamic phantom structures, as we did in the example above; online, EXPLAIN PHANTOM STRUCTURES, DYNAMIC. For information about using the SET SUBGOAL TRANSACTIONS Uproc, see the manual "SPIRES Formats"; online, EXPLAIN SET SUBGOAL TRANSACTIONS. And for information about the $LOOKSUBF and $LOOKSUBG functions, see the manual "SPIRES Protocols"; online, EXPLAIN $LOOKSUBF FUNCTION or $LOOKSUBG FUNCTION.
An alternative, which is perhaps more complicated to understand, is to retrieve the goal records from the transaction records. Depending on the situation, you can use the element TLCTR or TKEY as the mechanism to access (via subgoal processing) the goal records in the other record-types of the file.
Another method uses FOR PATH processing with either TKEY or TLCTR to retrieve those goal records. Currently, phantom structures are not available in the $TRANSACTIONS record-type, though they are likely to be a future enhancement.
Subgoal processing is discussed in detail in the manual "SPIRES Formats", section B.12. FOR PATH processing is discussed in the manual "Global FOR", sections 2.15 and 3.8. An example below demonstrates the use of FOR PATH processing.
If several different versions of a goal record exist in the deferred queue because it has been updated multiple times today, you can retrieve each copy or the tree version or the latest version -- that is, you can connect a $TRANSACTIONS record with its specific goal-record counterpart in the deferred queue.
This example shows how FOR PATH processing could be used to see each copy of records in the deferred queue in the order in which they were created. The subfile XYZ contains the goal records, while the subfile XYZ TRANS contains the transaction records:
-> select xyz -> through 1 select xyz trans -> through 1 for transactions where trec = 1 -> for path 1, pointer=tdlctr +> in active clean display all
SPIRES uses the element TDLCTR in the XYZ TRANS records having a TREC value of 1 as a pointer for retrieving the deferred queue copy of records in XYZ. In other words, SPIRES will retrieve the version of the XYZ record that went into the deferred queue when the transaction occurred, which is not necessarily the latest version of the record. If you are interested in the latest version, replace TDLCTR with TLCTR or TKEY; if you want the tree version, replace FOR PATH with FOR TREE VIA PATH.
In many cases, you may also retrieve the previous deferred queue copy of a record, replacing TDLCTR with TPLCTR in the example above. However, if the value of TPLCTR in a $TRANSACTIONS record is zero (hex 00000000), then that transaction is the first in the deferred queue for the goal record. Since there is no previous deferred queue copy of the goal record, SPIRES would skip this $TRANSACTIONS record in processing.
Suppose, for instance that record A from the tree has been updated three times today:
-> through 1 for transactions where tkey = A
-> for path 1, pointer=tplctr
+> show key
A
+> through 1 show levels
Processed Examined
FOR DEFQ VIA TRANS 2 12
+> show levels
Processed Examined
FOR SUBFILE VIA PATH 1 2
+>
The SHOW KEY command sent SPIRES through the path to find records in the $TRANSACTIONS record type having a TKEY value of "A". The first record it found, the first updated version in the deferred queue, was "transmitted" through the path to the FOR PATH processing. However, because it has a zero value for TPLCTR, it was considered an "examined record" but was not processed. The next record found in $TRANSACTIONS with a TKEY value of "A" did indeed have a non-zero value for TPLCTR -- it is, in fact, a pointer to that first deferred queue record -- and so it is processed.
As a file owner, you can ask SPIRES to log certain information about the access and use of the file's subfiles. You can choose to log only the information that a user has selected and "cleared" a subfile, or you can request that all the user's commands (and even responses to all ASK prompts) be logged too. In addition, with the WITHIN LOG prefix, you can have an application write anything to the file log you want.
One specialized use of the file log is to support "subfile charging", that is, charging people for their use of a particular subfile. SPIRES will inform them of the rates when they select the subfile and then log the charges as they are incurred. The file owner would be responsible for processing, billing and collecting any fees for subfile use.
This chapter will discuss all aspects of SPIRES's logging facility, divvied up as follows:
To create a file log, you add one or more statements to your file definition: LOG, HOURLY-CHARGE, DISPLAY-CHARGE or MINIMUM-CHARGE. You need at least one of these statements in each Subfile section for which you want logging. When the file definition is returned to the FILEDEF subfile, the statements take effect immediately for users who subsequently select the subfile. SPIRES will begin collecting the log data in the file's deferred queue.
An optional statement, STATISTICS, may also be included, which tells SPIBILD what to do with the log when the file is processed. The STATISTICS statement appears at the beginning of a file definition, which must be recompiled for the statement to take effect.
This section describes how to add the LOG and STATISTICS statements to your file definition. The subsection that follows describes the three statements you can add to implement subfile charging: HOURLY-CHARGE, DISPLAY-CHARGE and MINIMUM-CHARGE. [See 4.3.1.1.]
You can add any of the statements by using File Definer or by changing the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file definition.
Either get your file definition from the FILEDEF subfile (or wherever you keep the copy you change), or get your File Definer input.
The LOG statement in the Subfile section tells SPIRES what to log for the users named in the ACCOUNTS statement of the section when they select the subfile. In the file definition, its syntax is:
LOG = n[, n...];
In File Definer input, it's added as an attribute to the SUBFILE keyword:
SUBFILE subfile-name/ ACCOUNTS accounts/ LOG n[,n...]
In each case, "n" is 1, 2 or 3, as determined from these choices:
They may be requested in any combination.
User commands generated through full-screen interaction aren't logged under any LOG value. Prism has its own method of logging those commands, described in the "Prism Applications" manual. If you are using your own full-screen application environment, you can add the WITHIN LOG prefix to the user command, which writes the command to the log with any LOG level set.
Use LOG = 1, 2, 3 in order to cause commands in Folio applications to be sent to your file log.
The LOG statement belongs to the Subfile section, at some point after the ACCOUNTS statement. For example,
In File Definer input:
subfile Petty Cash/ recname rec01/ accounts gq.jbc,gg.jts/ +
log 1, 2, 3
In the file definition:
SUBFILE-NAME = PETTY CASH;
GOAL-RECORD = REC01;
ACCOUNTS = GQ.JBC, GG.JTS;
LOG = 1, 2, 3;
For File Definer, you would then generate a new file definition, and put it into the FILEDEF subfile. (The GENERATE procedure is summarized at the end of this section.) If you are changing the file definition directly, you just put it into the FILEDEF subfile.
When you put the file definition into the FILEDEF subfile, the LOG statement will have an immediate effect: logging will begin the next time a user affected by the change selects the subfile. The log goes into the deferred queue, where it is accessible via the SHOW FILE LOG command. [See 4.3.3.]
By default, SPIBILD discards the log when it processes the file. By adding the STATISTICS statement to the file definition, you tell SPIBILD to display the log and/or to add the log to an archive data set when it processes the file. In the file definition, its syntax is:
STATISTICS = n[,n];
In File Definer input, it's added as an attribute to the FILE keyword:
FILE [filename]/ STATISTICS n[,n]
In both cases, "n" is 1 or 2, chosen by the following criteria:
You may request both displaying and archiving if you code both 1 and 2.
Here is a sample Subfile section from File Definer input that uses the STATISTICS option:
file petty.cash/ statistics 1,2
After changing your input, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key
The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.
If you're changing the file definition directly, put the STATISTICS statement near the top, after the ID statement:
ID = GQ.JNK.PETTY.CASH; STATISTICS = 1, 2;
Unlike the LOG statement, changes to the STATISTICS statement require recompiling the file definition. After you change the file definition and put it into the FILEDEF subfile, issue the RECOMPILE command:
-> recompile .petty.cash -File definition compiled ->
If the file definition doesn't compile, use the error messages to determine the problem; then correct the file definition, and try the procedure again.
Once the STATISTICS statement is compiled, it will be in effect the next time the file is processed in SPIBILD.
[Note: Check with your computer center to be sure that you can legally charge money for the use of your data before you use the facility described here.]
If you intend to charge users for their use of your subfile, you need to log their use of the subfile. In addition, you probably want SPIRES to tell users what the rates are when they select the subfile and to compute the amount you should charge them, reporting that to them when they leave the subfile.
To tell SPIRES about the rates, you add any or all of three statements to the appropriate Subfile section of your file definition: HOURLY-CHARGE, DISPLAY-CHARGE and MINIMUM-CHARGE. Any of these statements causes logging to begin, equivalent to LOG = 1, described in the previous section on the LOG statement. [See 4.3.1.] [If you need to log more information for charging purposes than what's shown here, you might want to add a LOG statement with a higher value than 1.]
You must change the file definition directly -- File Definer does not support files with subfile charging. Follow the same directions you would use to add the LOG statement, described in the previous section.
Use the HOURLY-CHARGE statement to specify the charge for each hour of elapsed time (or fraction thereof) the user has the subfile selected, as a dollar-and-cents amount:
HOURLY-CHARGE = $n.nn;
SPIRES prorates the charge to the user based on the actual elapsed time.
Use the DISPLAY-CHARGE statement to specify the charge for displaying 1000 records in the subfile (i.e., each record processed by a TYPE, OUTPUT, DISPLAY, TRANSFER or REFERENCE command), as a dollar-and-cents amount:
DISPLAY-CHARGE = $n.nn;
SPIRES prorates the charge to the user based on the actual number of records displayed.
Use the MINIMUM-CHARGE statement to specify a minimum charge to the user when the sum of the "hourly charge" and the "display charge" is less than that amount, as a dollar-and-cents value:
MINIMUM-AMOUNT = $n.nn;
Place any or all of these statements after the ACCOUNTS statement in the appropriate Subfile section of the file definition. For example, here is part of a file definition for a matchmaking service:
SUBFILE-NAME = BLIND DATA;
GOAL-RECORD = REC01;
ACCOUNTS = PUBLIC;
HOURLY-CHARGE = $30.00;
DISPLAY-CHARGE = $10.00;
MINIMUM-CHARGE = $5.00;
Remember that statements in the Subfile section take effect as soon as you put the file definition into the FILEDEF subfile; you do not have to recompile the file definition. But remember too that unless you include a STATISTICS statement (which means the file definition must be recompiled), the log statistics will be discarded when the file is processed.
A user who selects the BLIND DATA subfile would get the following message from SPIRES:
-> select blind data -Database Charge: $30.00/hour, $10.00/1000 displays, $5.00 minimum -Continue select? yes ->
Notice that SPIRES asks the user whether he or she wants to continue the selection. For charging subfiles, this occurs if the user's charge message level is set to 2 or 3. [EXPLAIN SET MESSAGES COMMAND online, or see SET MESSAGES in Part 6.]
Twenty minutes later, after examining 500 records in the subfile, the user might see this when leaving:
-> clear select -Approximate database charge: $15.00 ->
The user has been charged $10.00 for having the subfile selected for 20 minutes, and $5.00 for displaying 500 records. The log contains the amounts to be charged to the account, as well as a count of the number of records examined. [See 4.3.4.]
Be aware that the messages shown here appear when the "charging" message level is 2; they may disappear if the level is set differently with the SET [CHARGE] MESSAGES command.
If a subfile has any logging set for it, then log entries will be written if the subfile is selected in SPIRES, Folio or Prism. (Because Prism is a full-screen system that prompts users with the ASK AREA command, only selects and de-selects get logged; Prism provides another method for logging user commands.) N.B.: Logging is associated only with subfiles; if any record-types are accessed via the ATTACH command by users with SEE access to the file, their use of the file is not logged.
When a user selects a subfile with logging turned on, SPIRES begins collecting the log data. SPIRES immediately writes a log entry to the deferred queue about the select, including the date, time, user's account number, selected subfile and charging rates (if applicable).
If the subfile logs more than just selects and de-selects (i.e., more than LOG = 1), then SPIRES creates log entries in memory as the user works with the subfile. The log entries may include commands issued, ASK commands and ASK Uprocs from formats and the user's responses, and any commands preceded by the WITHIN LOG prefix. [The WITHIN LOG prefix may be added to any SPIRES command to cause it to be written to the log, regardless of the log-level setting. By preceding it with a "/", you tell SPIRES to replace any variables in the command with their values, allowing you to write current variable values to the log if you wish. The WITHIN LOG prefix is frequently added to the "-" comment command when the application designer simply wants to write some information to the log. The prefix is described in more detail in the manual "SPIRES Protocols"; online, [EXPLAIN WITHIN LOG PREFIX.]]
Except for the initial select entry, log entries are not immediately written to the deferred queue, however; SPIRES saves them in memory, sending them to the deferred queue when:
- the subfile is de-selected; or
- the WRITE FILE LOG command is issued; or
- a block's worth of log data (about 2000 bytes) has been accumulated for the user.
Once SPIRES has added them to the deferred queue, it begins collecting the next entries in memory. Hence, a user's log entries for a given session may be scattered through the log in clumps; this is most likely when a subfile is used by several people at once and each session includes many commands.
One other aspect about the delay in writing log data to the deferred queue is important to know about. If the system crashes during the user's session, the log data in memory will be lost. Similarly, if the user leaves SPIRES in some unorthodox way (say, via a session break), the log data in memory will be lost. To minimize the chances of losing the log data, the WRITE FILE LOG command can be issued, which writes the user's current log entries to the deferred queue immediately. This command often appears in protocols following commands that must be logged. See the manual "SPIRES Protocols" for more information about the command; online, [EXPLAIN WRITE FILE LOG COMMAND.]
When the file is processed, SPIBILD discards the log. Depending on its disposition as specified by the optional STATISTICS statement, SPIBILD may display the log near the end of the processing and/or may add the log to the end of your ORVYL data set SYSTEM.ULOG before throwing it away.
Unless you issue the SET SEARCH or SET SHARE command in your SPIBILD session, you can't process the file while users have any of its subfiles selected. However, if you SET SHARE, you can process the file, and the user's log entries at the time will get written to the file's deferred queue after the processing. (That means that a single user session may span logs.) On the other hand, if you SET SEARCH, you will be able to process the file, but there is a possibility that log entries will be lost if the user de-selects the subfile during the processing. [See 2.1.5.1.]
This section describes the commands you can use to work with the file log data. [See 4.3.4 to learn how to read the coded data of the log.]
These commands show you log data:
- SHOW FILE LOG
- PROCESS, ESTABLISH, etc.
- SHOW ARCHIVE
To discard log data, use these commands:
- PROCESS, ESTABLISH, etc.
- ZAP DEFQ
- SHOW ARCHIVE
Each command is explained in detail in the reference section at the end of this manual. Below is a brief description of each:
Use the SHOW FILE LOG command in SPIRES to see the current contents of the log in the deferred queue. Though the word FILE appears in the command, it may be replaced by SUBFILE -- but in fact, the entire file log is displayed, not just the log entries for the selected subfile.
The SPIBILD commands that process the file will discard the current contents of the log in the deferred queue. However, if STATISTICS = 1 is coded in the file definition, the log will be displayed at the terminal (or in the job output, in the case of batch processing jobs) before it is discarded. [See 4.3.1.]
If you are processing the file and SPIBILD begins displaying the log, you can press the ATTN/BREAK key to stop it. However, the log is then discarded immediately -- you cannot see or use those log entries again unless they have been archived (STATISTICS = 2).
Use the SHOW ARCHIVE command in SPIRES to see the log entries that have been moved to your SYSTEM.ULOG data set by SPIBILD. The single data set serves as respository for all archived log entries for all your files, so if you have several subfiles from several files, you may have to be careful that you are working with the log data you want.
If you include the RELEASE option, SPIRES will erase the current contents of the SYSTEM.ULOG data set after displaying them.
The ZAP DEFQ command erases all contents of the deferred queue of the file containing the selected subfile, including the log entries in the deferred queue. Only the file owner and users with Master access to the file can issue the ZAP DEFQ command.
Most of the file log's contents are self-explanatory, even without column headings. Both the file log and the file owner's archive have the same format. Below is a sample of a log, followed by a description of each. column and of the codes in the first column (character positions 2 and 3).
C8 15:33:52 01/09/87 GQ.JNK $0.00 $0.00 $0.00 ALMANAC C9 15:34:04 01/09/87 GQ.JNK show indexes C9 15:34:13 01/09/87 GQ.JNK show autogen C9 15:34:43 01/09/87 GQ.JNK exit E8 15:34:43 01/09/87 GQ.JNK 0 00:00:51 C8 10:42:32 01/26/87 GQ.VVN $6.00 $10.00 $1.00 DATES C9 10:42:39 01/26/87 GQ.VVN sho sub siz C9 10:42:56 01/26/87 GQ.VVN for subfile C9 10:43:00 01/26/87 GQ.VVN sho keys all C9 10:43:15 01/26/87 GQ.VVN endfor C9 10:43:18 01/26/87 GQ.VVN find date 7/27 C9 10:43:20 01/26/87 GQ.VVN type C9 10:43:33 01/26/87 GQ.VVN sel restaurants E8 10:43:33 01/26/87 GQ.VVN 12 00:01:01 E9 10:43:33 01/26/87 GQ.VVN $1.00 C8 14:57:39 01/26/87 GQ.BEC $6.00 $10.00 $1.00 DATES C9 15:25:34 01/26/87 GQ.BEC FIND DAY 11/10 C9 15:35:28 01/26/87 GQ.BEC SHOW INDEXES C9 15:35:36 01/26/87 GQ.BEC FIND DATE 11/10 C9 15:41:43 01/26/87 GQ.BEC CLR SEL E8 15:41:44 01/26/87 GQ.BEC 0 00:44:05 EA 15:41:44 01/26/87 GQ.BEC $4.40
The example shows some excerpts from a log for two different subfiles: ALMANAC, which has no charging associated with it, and DATES, which does. You can see the date and time when a specified account issued the commands shown. Commands are listed in the case in which the user typed them.
All records have the same format for the first 34 columns:
Column Contents
------ ---------------------------------------------------------
1 - Blank
2/ 3 - Record "type" code; see below for types
4 - Blank
5 - Environment code: the value of $ENVIRON variable:
blank (1) = interactive (or BATWYL job)
2 = batch-driven interactive process,
usually HLI, Host Language Interface,
or JOBGEN-submitted SPIBILD jobs
3 = Batch SPIRES or SPIBILD
6 - Blank
7/14 - Time of day, 24-hour clock, in HH:MM:SS form
15/16 - Blank
17/24 - Date, in MM/DD/YY form
25/26 - Blank
27/32 - Account number, in GG.UUU form
33/34 - Multiple logon number, left-justified: the value
of $MULTILOG variable. If the same account is
logged on to several terminals simultaneously,
each is assigned a logon number for identification
purposes, a number that follows the account
number. The default, 1, is not printed, so these
columns are usually blank.
The remaining columns vary depending on the code given in columns 2 and 3:
35/46 - the count of the number of records displayed
49/56 - the time elapsed since the subfile was selected
(HH:MM:SS form)
58/65 - the time the subfile was originally selected
(same value as the C8 record)
67 on - the name of the selected subfile, if it was selected
through a subfile path
Some file management tasks cannot be easily categorized -- and sometimes pat procedures to handle the tasks don't exist either. This part of the manual covers a variety of problems you may need to address.
The most general task, and yet one of the most important, is controlling the costs of your SPIRES files. Throughout the manual, cost considerations have played a major role in most of the discussions. However, chapter 5.2 consolidates many of the cost-saving ideas presented earlier, which may lead you to alternative procedures you might not have considered. [See 5.2.]
Other procedures described in Part 5 are not so critical but may solve small problems that face you. For instance, one chapter describes how to reset the "next slot number" that SPIRES assigns to slot records, when you want to skip some numbers or when you want to set the number back. [See 5.3.]
This chapter presents some ideas about how you can save money on SPIRES files that you manage. This is very important for managers of large files, but some of the ideas presented here may be useful to owners of small, personal files too -- after all, no one wants to spend more money on data bases than necessary.
These are primarily ideas about files that already exist. There are many aspects of savings that the file designer should consider before creating a file, but that is a subject for another manual, "SPIRES File Definition". On the other hand, there are some techniques discussed here that may correct problems created by unfortunate file design.
Keep in mind that savings offered to the file owner may be at the expense of the file's users, and vice versa. For example, the file owner may save storage costs by eliminating all indexes, but the users will have to spend much more if they must use Global FOR instead of indexes for searching. Such tradeoffs will be discussed when appropriate.
The money-saving ideas are presented here as rules -- guidelines that file managers should follow if they want costs to be as low as possible. Naturally, all these rules have exceptions; situations arise where it is actually cheaper to break the rule than to follow it, or where you have no choice but to break the rule in order to get your work done. So some attention will be paid here to known exceptions to these rules.
Still, don't be misled by exceptions. A study of documentation for one system found that 90 percent of the documentation discussed 10 percent of the situations users actually found themselves in. Chances are that your situation is not the exception -- if you think it is, it might be wise to discuss it with your SPIRES consultant before breaking the rules precipitously.
In fact, thinking over what you are doing before you do it is always a good rule for saving money. As one SPIRES consultant said, "Think three times, measure twice, and cut once." This rule isn't always followed, and to some extent, SPIRES can share the blame for that.
Because applications in SPIRES are often developed iteratively and because SPIRES offers many tools that let you quickly try different ways to solve problems, it is easy to get in the habit of implementing an approach before you think it through. Naturally this approach has its benefits: you solve problems quickly, and you learn a lot as you work through the problems; but you may use a lot of expensive resources as you work. Clearly, you should try to strike a balance between implementing an idea swiftly (which might save time) and thinking the problem through (which might save money).
Of course, experimentation may be the only way for you to determine what approach will work best for a given situation. A good rule to follow here, particularly if you are working with a large file, is to experiment with a much smaller version of the file: perhaps a file with 1 percent of the records, or one with 10 percent of the records, from a random sampling if possible.
As you experiment, measure the resources you use and the costs you incur by issuing the SHOW CLOCKS and/or SHOW COST commands before and after issuing the controversial commands. Not only will they help you judge the various options you are considering, but they will also give you a sense of how much SPIRES components cost to use in general, which may be useful to you on a daily basis as you use your file.
If the benefits and convenience of SPIRES (including Prism and Folio) don't outweigh the costs, don't keep the data in SPIRES. Many small data bases may be more inexpensively maintained on microcomputers than in SPIRES. On the other hand, the strengths of SPIRES may make its costs seem less important; some strongpoints to consider are:
- its ability to handle large records and large files;
- its ability to share data between many users simultaneously;
- its flexibility in formatting data for online display and printed reports;
- its support of Prism and Folio, which provide easy-to-use interfaces to SPIRES files;
- its flexibility in handling data security at all levels;
- its strong searching support;
- its variety of tools for application development;
- your computer center's automatic backup of SPIRES files in case of system problems.
If none of these matter to you, chances are very good that you are wasting money keeping your files in SPIRES.
Of course, a more critical question must be faced too: Do you need to keep the data in a data base in any computer program? Name-and-address files, for example, make good examples in SPIRES documentation because they are easy to understand, but do they make good files in any computer program? Not always, unless the benefits of putting them on a computer outweigh the costs. Those benefits include:
- the ability to standardize the data collected;
- the ability to sort the data in several ways;
- the ability to find specific records quickly, searching by various criteria;
- the ability to print updated lists and reports as needed;
- the ability to analyze the data statistically.
If all you need to do is find someone's phone number occasionally, it's unlikely that the time spent putting the file together plus the expense of storing it would justify its storage on any computer.
These considerations are important not just when you create a file but throughout its life. A file's usefulness may change over time. For instance, if you find, several years after its development, that your file is not being used much any more, consider closing it down. It might be less expensive to create a new report, use it to run a final copy of the data and then destroy the file than to keep the file online with no one using it.
This guideline, the first of two corollaries to the rule number 1, suggests you verify that all indexes you are maintaining are getting regular use. Indexes are overhead -- they duplicate data already stored in the file, they require additional storage space, and they require computer time to build and maintain. If they are rarely used, they may be costing you more money to maintain than they are saving users to search with them. And they may even be inefficient to use too.
For example, suppose you own an inventory file for a bookstore. You have an index called BINDING whose common values are HARDBACK and PAPERBACK. It is unlikely you keep the index because you need to find all the hardback or paperback books in stock. In fact, you keep this index because customers often ask if you have a particular book in paperback, so you issue a search like FIND TITLE = GERTEL ESCHENBACH AND BINDING = PAPERBACK.
However, a search like that is almost certainly less efficient than FIND TITLE = GERTEL ESCHENBACH, followed by ALSO BINDING = PAPERBACK. The TITLE search would probably only retrieve several records. By ANDing that result with a very large result from the BINDING part of the search (perhaps thousands of pointers), SPIRES must do more work than if it simply examined the few records from the TITLE search directly.
It is usually worthwhile and often quite educational to study how people are using the indexes you have created for a subfile. You can use subfile logging to help you gather this information [See 4.3.] but don't forget that simply asking your users what kind of searches they need or try to do may be the best way to get the information you need.
If you decide to eliminate indexes, keep in mind that the storage space they used may not be released immediately. Rebalancing the data set will release the empty blocks. [See 2.7.2, 3.3.]
The second corollary to the first rule suggests that you should keep files from growing too large with unneeded data. After living with a file for awhile, you may discover that your goal record-type has elements whose values are interesting but of little or no value to the subfile's users. It takes extra work to collect and input these elements, and it takes extra space to store them. If they are Optional elements, you might consider no longer including them in records, or actually updating records to remove them. [See 2.6.2.]
What applies to elements applies to whole records as well. Files that keep regular transactions (mail files, order files, etc.) may keep growing and growing over the years, with the older records having little or no value. You might consider removing them from the file completely, perhaps making a copy of them on tape, on paper or in some other storage medium.
If you remove hundreds of records at once, keep in mind that SPIRES will not release the empty blocks in which they were stored. They will be reused again, though, as new records get added, as long as the available-space chains are correct. [See 3.4.] However, you might want to consider rebuilding the file so that it will use only as many blocks as it really needs. This is especially important to consider if you remove, say, more than half of the records in the record-type or 1000 records, whichever is more. [See 2.8.3, 2.9.]
At Stanford, users are charged lower rates when they do their computing work at night or on weekends. If possible, schedule your most resource-intensive work (e.g., building a file, rebuilding indexes, creating a large report) for those periods of time.
The cheapest way to add records to a SPIRES file is with the GENERATE LOAD procedure. [See 2.3.3.] It avoids all the Inproc processing as the records are added, a large portion of the work SPIRES must do for new records. The technique requires that the records come from another SPIRES file, and specifically from a record-type whose characteristics match the one the records are being added to. The GENERATE LOAD procedure is most commonly used when rebuilding a file [See 2.9.] or when you are creating a small test version of a file from a larger one.
If you cannot use the GENERATE LOAD procedure for input, try to use an input format written especially for the goal record-type. For input, it is generally cheaper to use custom input formats than to use the standard SPIRES format. The reason for this is that most input formats require data to appear in particular positions or within a specific order within the input data set. That is not a requirement of the standard format, which has to determine which element is being processed each time a new element value starts.
However, if your format is so generalized that it too must determine which element appears on each line, it may be no cheaper than the standard SPIRES format. Experimenting with a small sample to check costs might be worthwhile.
If you are building a large file or a large record-type within a file, it is almost always cheaper to build it with FASTBILD rather than SPIBILD. Using FASTBILD is more complicated because it requires you to study your data and give FASTBILD some specific information about it ahead of time. A minor mistake can cause the FASTBILD process to fail, often requiring it to be restarted from the beginning. But FASTBILD is so much more efficient than SPIBILD for most large files that most FASTBILD users find that they still save money over SPIBILD even if they have to restart the FASTBILD process two or three times.
A more detailed comparison of SPIBILD and FASTBILD appears earlier in this manual. [See 2.12.]
If you have decided not to use FASTBILD for one reason or another, consider the FAST BATCH command in SPIBILD rather than simply the INPUT BATCH command. [See 2.3.4.]
Especially for large files, the findings of the STATUS command may far outweigh its cost. [See 3.2.3.] It can tell you that the goal-record tree is badly unbalanced, causing inefficient retrieval of its records, or that storage space in the residual is being wasted. It is easy to correct these problems by using the rebalancing procedure and the AVSPREC procedure. [See 3.3.1, 3.4.3.]
File owners sometimes want more control over the slot numbers doled out by SPIRES as record key values in slot subfiles. The FIX SUBFILE SLOT command, which can be issued by the file owner or accounts having master access to the file, allows you to set the slot number higher (and in some cases, lower) as you deem necessary.
WARNING: Don't use this procedure casually. If you aren't careful, you can cause serious problems, as described below.
The syntax of the FIX SUBFILE SLOT command is simply
FIX SUBFILE SLOT
The appropriate subfile should already be selected. SPIRES will display the current value for the next slot number (from the system variable $NEXTSLOT) and prompt you for a new value.
For example,
-> select almanac -> fix subfile slot -Next slot is 1074; Type new slot: 10001 ->
This command is most useful for setting a higher starting slot number. For example, a file owner might decide to set the starting slot number to 1001 when the subfile is first created so that all records (or at least the first 8999 of them) will have a four-digit key. Setting the slot number higher results in a permanent change.
However, if you set the slot number lower, the lower value remains in effect only while the subfile is currently selected by you. Anyone else selecting the subfile is unaffected by the change.
Moreover, you should be very careful not to overlay new records onto old records, a problem that can arise if you set the slot number lower. This can cause SPIRES to get confused about the correct slot number; for example, it may become convinced that the next slot number is the number of a record that already exists, causing S413 errors each time you try to add a new record.
Here is one scenario in which the command would be useful. Suppose you added 10 records to a subfile, removed them right away, and then wanted to add some more, reusing the same slot numbers again:
-> fix subfile slot -Next slot is 1010; Type new slot: 1000 ->
Any other users who add records will get slot numbers from 1010 on. For you, though, new records will get slot numbers from 1000 on. Note, however, that it is best to set the slot number lower only at a time when no other users will be updating the subfile.
Though most users get access to a record-type of a file via the SELECT command, the file owner and users with See access to the file have an alternate way: they can use the ATTACH command to attach any of the file's record-types directly. The ATTACH command has two important aspects:
- it allows any record-type of the file to be accessed as if it were the goal record-type of a selected subfile without the file owner having to add a Subfile section for it to the file definition; and
- it provides an alternate way to access a goal record-type, avoiding the environmental settings established by statements in the Subfile section of the file definition, such as secure-switches, subfile logging, a default format, etc.
The ATTACH command is simple to use. It's available in most SPIRES programs, but its use in SPIRES is what interests us here. Its syntax is:
ATTACH [{rec-name|rec-number} OF] [ORV.gg.uuu.]filename
You need to specify the entire file name, with the "ORV" prefix, if the file does not belong to you. If you omit the "rec-name" or "rec-number" option, SPIRES will attach the first record-type of the file. To see another, add its name to the command, or use its number, from the order in which it is defined in the file definition, from 1 to 64. So, for example, if the index record-type you want to see is the third record-type defined in GQ.DOC.ERRATA, you would type the command "ATTACH 3 OF ORV.GQ.DOC.ERRATA".
Again, a typical use of the ATTACH command might be to get a complete list of all the values in an index:
-> attach rec3 of almanac -> show select -Record-type REC3 of GQ.JNK.ALMANAC attached -> in active clear show keys all ->
Remember, you can use the attached record-type as if it were the goal record-type of a selected subfile. You can still search for and display records, and you can even update records too, if you are the file owner or you have Update or Master access to the file. WARNING: Be very careful if the records are index records containing pointers created by SPIRES/SPIBILD; do not change any of these values or you will create data integrity problems.
On occasion, you may need to block users from creating record transactions in a file, or even from selecting a subfile of the file at all. For instance, you might discover a file-processing problem that occurred the previous night, and you want to repair it without having to worry about people adding more records to the file, compounding the problem. You may be able to come up with various ways such changes could be made, such as by altering file permits, or removing the Subfile section from the file definition, etc.
But the fastest way (and usually when you need to do this, time is of the essence) is to 1) select a subfile of the file, and 2) issue the SET SUBFILE DISABLE command:
SET SUBFILE DISABLE FOR ACCESS
Under this command, SPIRES will prevent any new users from selecting any subfiles of the file and detach the file from each current user as soon as the user issues a command that accesses the main block of the deferred queue, which includes most record-processing, search and display commands.
These restrictions don't apply to the file owner, nor to users with MASTER access to the file.
When you are ready to "enable" your file for use again, select the subfile again and issue the SET SUBFILE ENABLE command:
SET SUBFILE ENABLE FOR ACCESS
Both commands are discussed in Part 6 of this manual; online, EXPLAIN SET SUBFILE ENABLE COMMAND.
Procedure: Create or Modify Extended Trees
Purpose: To convert a file to have one or more extended tree
data sets, or to work with the TREE-DATA structure
Who can use it: file owner
Basic steps:
1) Change the file definition
2) Recompile the file using the REBALANCE option
3) ATTACH filename
4) Rebalance the specific data set using the CONVERT option
5) FIX REBALANCED DATA SET MSTR
6) ATTACH filename
7) FIX the rebalanced data set with the RENAME and CONVERT
options
A very large file may run into the limit on the size of record trees, which is 64K blocks per tree. By making some changes to the file definition, you can request that the data set containing the current record-type is to be an "extended-tree" data set, one which can grow to 434,280 blocks. After the file definition has been recompiled using a special option described below, you need to follow a special rebalancing procedure to convert the existing trees that are affected by this change (internally speaking, that means they will have 3-byte instead of 2-byte branch block pointers).
You follow the same procedure to take advantage of the TREE-DATA structure in a file definition, which lets you split a record-type into multiple key trees to help manage tree depth and reduce the amount of rebalancing needed when a record-type is vulnerable to intense growth in a specific key area. If you ever need to change the START-KEYs, you will need to go through this procedure again as well.
You need to know which ORVYL data set holds the record-type you are extending or modifying. The easiest way to do that is to ATTACH the file and issue the SHOW FILE TREES command, which will list the record-types of the file along with the data sets they are in. Of course, you need to already know which record-type of the file you are changing.
For the remainder of this procedure, the data set you are going to be working with will be referred to as the "RECn" data set.
The REBALANCE command (step 4) will create a data set with the default ORVYL permit setting of Public Read (or Public Write, for data sets containing immediate indexes). That's the same setting SPIRES uses when it creates the "RECn" data set.
However, if you, the file owner, ever changed the permit, you will probably want to reinstate that level of security when this procedure is over. To that end, issue the command:
SHOW PERMITS filename.RECn
If the data set has permits other than the appropriate default, jot them down so that you can reset them at the end of this procedure.
You should not have any ORVYL data set called "myfile.MREB", "myfile.REBn" or "myfile.REOn.yyyymmdd" ("yyyymmdd" being today's date) when you issue the REBALANCE command, since SPIRES will create them. If any of them do exist, the procedure will stop. To see if you do have any, issue these ORVYL commands:
SHOW FILE LIKE myfile
If you do have any of them, either erase or rename them, using the ORVYL commands "ERASE dataset.name" or "RENAME dataset.name TO new.dataset.name". (You will have to issue the ORVYL command "SET CLP dataset.name" prior to the ERASE or RENAME command if the data set was created by SPIRES.)
WARNING: Since the REBALANCE procedure replaces both the data set being rebalanced and the MSTR data set of the file with new copies, at times it needs exclusive access to the file. You might want to use the SET SUBFILE DISABLE FOR ACCESS and SET SUBFILE DISABLE FOR PROCESS commands to prevent any further activity while you are working on the deferred queue.
The specific changes you need to make to the file definition are discussed in the SPIRES manual "File Definition", in chapter B.6.5a. Online, [EXPLAIN EXTENDED-TREE STATEMENT.]
After you have updated the file definition in the FILEDEF subfile, go on to the next step, which describes a special form of the RECOMPILE command you should use anytime you make changes to the file definition that involve either the EXTENDED-TREE statement or TREE-DATA structures.
With the FILEDEF subfile still selected in SPIRES, issue the following command:
RECOMPILE [gg.uuu].filename REBALANCE
The compilation creates a new master data set called, for the moment, "filename.MREB". It doesn't replace the MSTR data set till step 5; in the meantime, you will rebuild and rebalance the "RECn" data set so that it matches the extended-tree information that the recompilation put into the MREB data set.
Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:
-> attach orv.gq.doc.errata ->
This example, which will be followed through this section, demonstrates using the ATTACH command on the file GQ.DOC.ERRATA.
This command will rebalance the trees of all the record-types stored in the named data set, storing them in a new ORVYL data set called "myfile.REBn". It uses the record and tree descriptions in the MREB data set.
The display from the REBALANCE command (which goes into your active file if you add the IN ACTIVE prefix) varies from file to file, but it should look something like this one for REC2 of GQ.DOC.ERRATA, which contains three record-types:
-> rebalance data set rec2 convert
Rebalance Data Set REC2
-Base block 0 in record-type REC02
-Next block is 972
-Base block 1 in record-type REC03
-Next block is 972
SPIRES will also display a message each time 64K nodes have been processed, to reassure you (especially if the record-types are large) that the rebalancing activity is occurring.
This next step, as mentioned above, renames to old MSTR data set (adding today's date to the name), and renames the new MREB data set as the MSTR:
-> fix rebalanced data set mstr ORV.GQ.DOC.ERRATA.MSTR is renamed to ORV.GQ.DOC.ERRATA.MSTR.19961106 ORV.GQ.DOC.ERRATA.MREB is renamed to ORV.GQ.DOC.ERRATA.MSTR Fix MSTR complete; MREB data set renamed -File detached ->
The FIX REBALANCED DATA SET MSTR command detaches the file so that internal tables used by SPIRES can be properly synchronized.
If the display ends with the "File detached" message, continue with step 6. If it did not, there may be several reasons why:
- you received the message "New data set exists; OK to continue?" because there is already a "filename.REBn" data set. You should type NO and return to the "Preliminaries" above.
- some other error has occurred. Use the EXPLAIN command to get more information about the error.
Issue the ATTACH command to attach the file again.
-> attach orv.gq.doc.errata ->
Complete the rebalancing procedure by replacing the old "RECn" data set with the new "REBn" data set from step 4. Be sure to include the RENAME and CONVERT options on the end of the FIX REBALANCED DATA SET command:
-> fix rebalanced data set rec2 rename convert FIX rebalanced data set REC2 ORV.GQ.DOC.ERRATA.REC2 is renamed to ORV.GQ.DOC.ERRATA.REO2.19920719 ORV.GQ.DOC.ERRATA.REB2 is renamed to ORV.GQ.DOC.ERRATA.REC2 -Set new data set permits if needed. ->
To clean up after successful rebalancing, you should get rid of the old data set, which you no longer need. Issue the ORVYL commands "SET CLP filename.REOn.yyyymmdd" and "ERASE filename.REOn.yyyymmdd".
If there were special permits set for the data set (see "Preliminaries" above), use ORVYL's SET PERMIT command to reset them:
SET CLP filename.RECn SET PERMIT filename.RECn FOR account access-level
If you turned off access to the file, you may want to turn it back on again with the SET SUBFILE ENABLE FOR ACCESS and SET SUBFILE ENABLE FOR PROCESS commands (see preliminaries).
After any rebalancing procedure, it is always a good idea to use the STATUS command to get a "snapshot" of the file in its new state. [See 3.2.3.] You can also see information about your new or changed trees by issuing the command SHOW FILE TREES.
The PERFORM TABLE CREATE command suite has been implemented to facilitate the task of creating Tables which may be useful for various purposes -- primarily for the movement of data from SPIRES to an RDBMS (Relational data base system) or from an RDBMS to SPIRES.
Several commands are included under the PERFORM TABLE CREATE tool set umbrella, each of which may be very useful in different aspects of the various inter-operability functions that may be dealt with to achieve the primary tasks mentioned above.
The central SPIRES structure that is common to these commands is the DECLARE TABLE structure. Each of these PERFORM commands operates on or performs its function based upon SPIRES DECLAREd TABLEs. Another SPIRES process, PERFORM DATA MOVE is the basic goal in mind as the final driving force in the use of these commands. It might be useful to read more about these features. [EXPLAIN DECLARE TABLE COMMAND.] and [See 5.8..]
Syntax of PERFORM TABLE CREATE
PERFORM TABLE CREATE (DECLARE | DDL | DATA MOVE | RECDEF)
SUBFILE = <source of Spires data elements>
KEY = <Record key of source Spires data>
KEY PREFIX= <Key prefix of source Spires data>
INCREMENT = <Record scan increment with ELEMINFO>
TYPE (ORACLE | SYBASE | NSI | SPIRES)
DESTINATION = <destination Table subfile>
FROM = <source Declare Table subfile>
QUIET
OPTIONS = 'multiple, virtual, changes'
So far the command options are:
1. PERFORM TABLE CREATE DECLARE -- Creates SPIRES DEClare Table data. [See 5.7.1.]
2. PERFORM TABLE CREATE DDL -- Creates RDBMS DDL CREATE constructs. [See 5.7.2.]
3. PERFORM TABLE CREATE DATA MOVE -- Generates a DATA MOVE record. [See 5.7.3.]
4. PERFORM TABLE CREATE RECDEF -- Generates SPIRES RECDEF records. [See 5.7.4.]
5. PERFORM TABLE CREATE INPUT -- Generates INPUT CONTROL constructs. [See 5.7.5.]
6. PERFORM TABLE CREATE ELEMINFO -- Creates SPIRES Data Element Information records. [See 5.7.9.]
PERFORM TABLE CREATE DECLARE -- Creates SPIRES Declare Table data
The output from the PERFORM TABLE CREATE DECLARE command is a series of Declare Table constructs which are intended to be used by SPIRES to tell it how to generate relational tables from SPIRES source elements and structures of a particular SPIRES subfile. Once these Tables are generated, they may be used in a variety of ways. For example, they may be used in conjunction with other PERFORM commands to generate RDBMS DDL statements [See 5.7.2,.] to generate a source record for the DATA MOVE process [See 5.7.3,.] or to generate a relational table View of the source SPIRES subfile itself [EXPLAIN DECLARE TABLE COMMAND.]
There is a good chance that the DECLARE TABLE structures that are generated for a particular subfile will need to be modified or supplemented in some way depending on your needs. You may wish to define a "DESTINATION SUBFILE" to simplify this modification (via normal SPIRES update commands) as well as to maintain a convenient database for generated SPIRES tables.
NOTE -- There is a special parameter that could be very useful if you wish to generate tables that form a precise relationship to an existing SPIRES data base. This feature is initiated through the use of the "FROM" parameter and utilizes a specially prepared database that describes the actual usage of source subfile data elements. This feature is described in [See 5.7.9,.] and is shown in an example at [See 5.7.11,.]
Syntax of PERFORM TABLE CREATE DECLARE
PERFORM TABLE CREATE DECLARE
SUBFILE = <source subfile>
TYPE (ORACLE | SYBASE | NSI | SPIRES)
DESTINATION = <destination subfile>
QUIET
OPTIONS = 'multiple, virtual, changes'
FROM = <element info subfile> see (see NOTE)
Command options:
Note: The File Definition for the goal records for this
database should be DEFINED-BY = $TABLE.
MULTiple - Generate tables for multiply occurring data elements.
VIRTual - Generate tables for virtual data elements.
CHANGES - Generate tables that can deal with insert/
delete records of changed row data.
EXISTS - Used in conjunction with the FROM parameter (see NOTE)
Note 1: If you wish to specify more than one option you must
write the options surrounded by single quotes. For example
- Options = 'Mult, Virt'
Note 2: Currently the maximum number of Columns allowed
within a Declare Table structure is 126. Including virtual
elements could increase the number dramatically. Therefore
one should make an effort to eliminate unnecessary Virtual
element columns. The Rdbms Column names for virtuals has
a "_v" suffix.
SYBASE -- Generate Sybase Rdbms info
ORACLE -- Generate Oracle rdbms info
NSI -- Generate Oracle rdbms info using data element
lookups from the NSI NOTEBOOK subfile.
SPIRES -- Generates DECLARE TABLE data with no reference
to Rdbms information.
Examples of command execution in SPIRES
> perform table create declare subfile family typ oracle, opt 'mult, virt'
> list
1. Declare Table FAMILY
2. ID = GQ.WCK.FAMILY;
3. COMMENTS = Table Generation for Record Level Elements;
4. DEFDATE = TUES. JULY 25, 2000;
5. MODDATE = TUES. JULY 25, 2000;
6. MODTIME = 12:37:44;
7. SUBFILE.NAME = FAMILY;
8. DECLARE.SUBFILE = Data Move Declares;
9. COLNAME = ID;
10. ISKEY;
11. COLTYPE = INT;
12. COLWIDTH = 3;
13. RDBMS_COLUMN = id;
14. RDBMS_DATATYPE = VARCHAR2;
15. RDBMS_DATALENGTH = 3;
16. COLNAME = NAME;
17. COLWIDTH = 28;
18. RDBMS_COLUMN = name;
19. RDBMS_DATATYPE = VARCHAR2;
20. RDBMS_DATALENGTH = 28;
21. COLNAME = CALLED;
22. COMMENTS = Multiple;
23. COLWIDTH = 12;
24. SOURCE.SINGLE;
25. RDBMS_COLUMN = called;
26. RDBMS_DATATYPE = VARCHAR2;
27. RDBMS_DATALENGTH = 12;
28. COLNAME = SEX;
29. COLWIDTH = 3;
30. RDBMS_COLUMN = sex;
31. RDBMS_DATATYPE = VARCHAR2;
32. RDBMS_DATALENGTH = 3;
33. COLNAME = PARENT_1;
34. COMMENTS = Multiple;
35. COLTYPE = INT;
36. COLWIDTH = 9;
37. SOURCE.ELEM = PARENT;
38. DECLARE.KEY = $VALUE.EXTERNAL;
39. SOURCE.OCC = 1;
40. RDBMS_COLUMN = parent_1;
41. RDBMS_DATATYPE = VARCHAR2;
42. RDBMS_DATALENGTH = 9;
43. COLNAME = PARENT_2;
44. COMMENTS = Multiple;
45. COLTYPE = INT;
46. COLWIDTH = 9;
47. SOURCE.ELEM = PARENT;
48. DECLARE.KEY = $VALUE.EXTERNAL;
49. SOURCE.OCC = 2;
50. RDBMS_COLUMN = parent_2;
51. RDBMS_DATATYPE = VARCHAR2;
52. RDBMS_DATALENGTH = 9;
53. COLNAME = BIRTH;
54. COLTYPE = DATE;
55. COLWIDTH = 12;
56. SOURCE.ELEM = BIRTH;
57. DECLARE.KEY = $DATEOUT.CCYY;
58. RDBMS_COLUMN = birth;
59. RDBMS_DATATYPE = VARCHAR2;
60. RDBMS_DATALENGTH = 12;
.................................
130. FILE = GQ.WCK.FAMILY;
131. TABLE.NUM = 001;
132. RDBMS_TABLENAME = family;
133. EndDeclare
134.
135. Declare Table CHILD
136. ID = GQ.WCK.CHILD;
137. COMMENTS = Table Generation for Element CHILD;
138. DEFDATE = TUES. JULY 25, 2000;
139. MODDATE = TUES. JULY 25, 2000;
140. MODTIME = 12:37:44;
141. SUBFILE.NAME = FAMILY;
142. DECLARE.SUBFILE = Data Move Declares;
143. COLNAME = ID;
144. COMMENTS = REF FAMILY;
145. ISKEY;
146. COLTYPE = INT;
147. COLWIDTH = 3;
148. RDBMS_COLUMN = id;
149. RDBMS_DATATYPE = VARCHAR2;
150. RDBMS_DATALENGTH = 3;
151. COLNAME = CHILD.OCC;
152. COMMENTS = Occurrence Number;
153. ISKEY;
154. COLTYPE = INT;
155. SOURCE.ELEM = CHILD;
156. DECLARE.KEY = $VALUE.OCC;
157. SOURCE.SINGLE;
158. RDBMS_COLUMN = child_occ;
159. RDBMS_DATATYPE = NUMBER;
160. RDBMS_DATALENGTH = 4;
161. COLNAME = CHILD;
162. COMMENTS = Multiple;
163. COLTYPE = INT;
164. COLWIDTH = 5;
165. RDBMS_COLUMN = child;
166. RDBMS_DATATYPE = VARCHAR2;
167. RDBMS_DATALENGTH = 5;
168. FILE = GQ.WCK.FAMILY;
169. MULTI.ELEMENT = CHILD;
170. TABLE.NUM = 002;
171. RDBMS_TABLENAME = child;
172. EndDeclare
173.
PERFORM TABLE CREATE DDL -- Creates RDBMS DDL CREATE constructs
This form of the PERFORM TABLE CREATE command will generate Rdbms DDL CREATE commands whose values are based directly on the syntax of the PERFORM TABLE CREATE DECLARE command. [See 5.7.1.]
There are two ways of representing this command, the options differing based upon the source of the input:
In this case the DECLARE TABLE structures stored (and possibly
modified) in the Destination Subfile that have the given
Subfile.Name value are generated and returned in your ACTIVE
file. Note that you may choose instead to specify source
table records using the KEY or KEY PREFIX values.
This form of the command is given with the same options that
you would use to create DECLARE statements where the output
is to be presented in the ACTIVE file.
Syntax of PERFORM TABLE CREATE DDL
- This form of the command is the "normal" form since you would generally want to modify the Declare Table records generated during the Create Declare process.
PERFORM TABLE CREATE DDL
FROM = <Declare table subfile>
SUBFILE = <source subfile.name>
KEY = <source record key>
KEY PREFIX <key prefix of source records>
QUIET
Command options:
Note: If the FROM statement is not given then you
should issue the PERFORM TABLE CREATE DDL command
with all the options used with the PERFORM TABLE
CREATE DECLARE statement, with the exception of
the DESTINATION option.
Examples of command execution in SPIRES
> perform table create DDL subfile family type oracle, opt 'mult, virt'
- DDL Statement Totals for Subfile family
Table: family Columns: 22 Row length: 4814
Table: note Columns: 3 Row length: 519
Table: child Columns: 3 Row length: 12
Table: marriage_str Columns: 11 Row length: 2101
------
Totals Tables: 4 Columns: 39 Row length: 7446
> list
1.
2. CREATE TABLE family
3. (id VARCHAR2(3) NOT NULL,
4. name VARCHAR2(28) NULL,
5. called VARCHAR2(12) NULL,
6. sex VARCHAR2(3) NULL,
7. parent_1 VARCHAR2(9) NULL,
8. parent_2 VARCHAR2(9) NULL,
9. birth VARCHAR2(12) NULL,
10. birth_place VARCHAR2(512) NULL,
11. birth_home VARCHAR2(512) NULL,
12. death VARCHAR2(12) NULL,
13. death_place VARCHAR2(512) NULL,
14. name_suffix VARCHAR2(512) NULL,
15. known_name_v VARCHAR2(512) NULL,
16. birth_month_v VARCHAR2(512) NULL,
17. birth_mmdd_v VARCHAR2(512) NULL,
18. child_count_v VARCHAR2(6) NULL,
19. birth_info_v VARCHAR2(34) NULL,
20. birthplace_v VARCHAR2(20) NULL,
21. death_info_v VARCHAR2(34) NULL,
22. sort_name_v VARCHAR2(24) NULL,
23. first_name_v VARCHAR2(512) NULL,
24. full_name_v VARCHAR2(512) NULL,
25. CONSTRAINT family_pk PRIMARY KEY (id));
26.
27. CREATE TABLE note
28. (id VARCHAR2(3) NOT NULL,
29. note_occ NUMBER(4) NOT NULL,
30. note VARCHAR2(512) NULL,
31. CONSTRAINT note_pk PRIMARY KEY (id, note_occ),
32. CONSTRAINT note_family_fk FOREIGN KEY (id) REFERENCES family
33. (id));
..................................
PERFORM TABLE CREATE DATA MOVE -- Creates a DATA MOVE record
This command may be used to generate a DATA MOVE subfile record that may be used as is or through modification to simplify data extracts from the source subfile to multiple Target tables.
There are two ways of representing this command, the options differing based upon the source of the input:
In this case the DECLARE TABLE structures stored (and possibly
modified) in the Destination Subfile that have the given
Subfile.Name value are generated and returned in your ACTIVE
file. Note that you may choose instead to specify source
table records using the KEY or KEY PREFIX values.
This form of the command is given with the same options that
you would use to create DECLARE statements where the output
is to be presented in the ACTIVE file.
[See 5.7.1.]
Syntax of PERFORM TABLE CREATE DATA MOVE
- This form of the command is the "normal" form since you would generally want to modify the Declare Table records generated during the Create Declare process.
PERFORM TABLE CREATE DATA MOVE
FROM = <Declare table subfile>
SUBFILE = <source subfile.name>
KEY = <source record key>
KEY PREFIX <key prefix of source records>
QUIET
Command options:
Note: If the FROM statement is not given then you
should issue the PERFORM TABLE CREATE DATA MOVE
command with all the options used with the PERFORM
TABLE CREATE DECLARE statement, with the exception
of the DESTINATION option.
Example of use:
It may be instructive to show an example of the use of a protocol which goes through all the steps from the generation of Declared Tables to the generation of a DATA MOVE record and to use that record to actually generate a complete set of relational tables for a subfile.
Note that the "Record.Length" and "lrecl" values generated for the target OS files in the DATA MOVE record are much larger than the corresponding "Max Length" values generated during DATA MOVE processing. This could cause serious space problems in a large volume move. Therefore, you should adjust the values accordingly either by adding ELEMINFO width statements in the File definition, COLWIDTH statements in the DECLARE TABLE records, or by altering the DATA MOVE record itself.
*TestDMove
> Declare Vgroup Local
var = KeyValue; Len = 40;
var = DBase; len = 40;
EndDeclare
> set echo
> clear ACTIVE
> let DBase = races
> /perform table create declare subfile #DBase dest 'wck.tables' opt mult
> /perform table create data move from 'wck.tables', subfile #DBase
> list
1.
2. Id = *RACES_TABLE_MOVE;
3. Subfile.Name = &GQ.WCK.RACES RACES;
4. Table.Subfile = &GQ.WCK.TABLES WCK.TABLES;
5. Area.Name = Table1;
6. Device.Type = OS File;
7. File.Name = WYL.GQ.WCK.RACES;
8. Record.Length = 2590;
9. Device.Options = var,lrecl=2590,repl,refor,temp;
10. Area.Name = Table2;
11. Device.Type = OS File;
12. File.Name = WYL.GQ.WCK.NAME;
13. Record.Length = 1030;
14. Device.Options = var,lrecl=1030,repl,refor,temp;
15. Area.Name = Table3;
16. Device.Type = OS File;
17. File.Name = WYL.GQ.WCK.RACES.X;
18. Record.Length = 1066;
19. Device.Options = var,lrecl=1066,repl,refor,temp;
20. Area.Name = Table4;
21. Device.Type = OS File;
22. File.Name = WYL.GQ.WCK.COMMENTS;
23. Record.Length = 1040;
24. Device.Options = var,lrecl=1040,repl,refor,temp;
25. Target.Name = RACES;
26. Generate.Table = GQ.WCK.RACES;
27. Area = Table1;
28. Target.Name = NAME;
29. Generate.Table = GQ.WCK.NAME;
30. Area = Table2;
31. Target.Name = RACES_X;
32. Generate.Table = GQ.WCK.RACES.X;
33. Area = Table3;
34. Target.Name = COMMENTS;
35. Generate.Table = GQ.WCK.COMMENTS;
36. Area = Table4;
> select data move
> addupd
> let KeyValue = $key
> /select #DBase
> for subfile
> /perform data move from #KeyValue
Totals for Subfile RACES Table Move Jul 26, 2000
Table Name Table Row Max Line Total
Type Count Length Count Length
===================== ===== ======= ====== ======= =========
races DELIM 13 83 14 622
name DELIM 13 25 14 286
races_x DELIM 35 51 36 1680
comments DELIM 8 32 9 243
>
PERFORM TABLE CREATE RECDEF -- Creates SPIRES RECDEF records
This form of the PERFORM TABLE CREATE command will generate SPIRES Record Definition (RECDEF) records which mirror the view of data described by Declare Table records generated by the PERFORM TABLE CREATE DECLARE command. [See 5.7.1.]
This command is expressed as follows:
The DECLARE TABLE structure stored (and possibly modified)
in the Table Subfile that has the given key value is used
as the source for RECDEF generation. Note that you must
also code the SUBFILE parameter as SPIRES uses this value
to determine record characteristics when generating the
record definition.
Syntax of PERFORM TABLE CREATE RECDEF
PERFORM TABLE CREATE RECDEF
FROM = <Declare table subfile>
KEY = <source record name>
KEY PREFIX <key prefix of source records>
SUBFILE = <source subfile name>
QUIET
Command options:
Note: If the FROM statement is not given then you
should issue the PERFORM TABLE CREATE RECDEF command
with all the options used with the PERFORM TABLE
CREATE DECLARE statement, with the exception of
the DESTINATION option.
Examples of command execution in SPIRES
> perform table create recdef from wck.tables subfile races key *races
> list
1. ****
2. ID = GQ.WCK.RACES;
3. MODDATE = FRI. AUG. 18, 2000;
4. DEFDATE = FRI. AUG. 18, 2000;
5. REMOVED;
6. REQUIRED;
7. KEY = RUNNER;
8. OCCURS = 1;
9. ELEMINFO;
10. WIDTH = 64;
11. RDBMS_COLUMN = runner;
12. RDBMS_DATATYPE = VARCHAR2;
13. RDBMS_DATALENGTH = 64;
14. OPTIONAL;
15. ELEM = FAC.STAFF;
16. OCCURS = 1;
17. ELEMINFO;
18. WIDTH = 1;
19. RDBMS_COLUMN = fac_staff;
20. RDBMS_DATATYPE = VARCHAR2;
21. RDBMS_DATALENGTH = 1;
22. ELEM = MAIL.ADDR1;
23. OCCURS = 1;
24. ELEMINFO;
25. WIDTH = 64;
26. RDBMS_COLUMN = mail_addr1;
27. RDBMS_DATATYPE = VARCHAR2;
28. RDBMS_DATALENGTH = 64;
29. ELEM = MAIL.ADDR2;
30. OCCURS = 1;
31. ELEMINFO;
32. WIDTH = 64;
33. RDBMS_COLUMN = mail_addr2;
34. RDBMS_DATATYPE = VARCHAR2;
35. RDBMS_DATALENGTH = 64;
36. ELEM = PHONE;
37. OCCURS = 1;
38. ELEMINFO;
39. WIDTH = 16;
40. RDBMS_COLUMN = phone;
41. RDBMS_DATATYPE = VARCHAR2;
42. RDBMS_DATALENGTH = 16;
43. ELEM = DEPARTMENT;
44. OCCURS = 1;
45. ELEMINFO;
46. WIDTH = 16;
47. RDBMS_COLUMN = department;
48. RDBMS_DATATYPE = VARCHAR2;
49. RDBMS_DATALENGTH = 16;
50. ELEM = TITLE;
51. OCCURS = 1;
52. ELEMINFO;
53. RDBMS_COLUMN = title;
54. RDBMS_DATATYPE = VARCHAR2;
55. RDBMS_DATALENGTH = 512;
56. ELEM = BIRTH.DATE;
57. OCCURS = 1;
58. INPROC = $DATEYY;
59. OUTPROC = $DATEYY.OUT(,CCYY);
60. ELEMINFO;
61. WIDTH = 10;
62. RDBMS_COLUMN = birth_date;
63. RDBMS_DATATYPE = VARCHAR2;
64. RDBMS_DATALENGTH = 10;
65. ELEM = SEX;
66. OCCURS = 1;
67. ELEMINFO;
68. WIDTH = 3;
69. RDBMS_COLUMN = sex;
70. RDBMS_DATATYPE = VARCHAR2;
71. RDBMS_DATALENGTH = 3;
72. ;
The PERFORM TABLE CREATE INPUT command suite has been implemented to facilitate the task of creating Tables which may be useful for a specific purpose, that of simplifying the task for generating the structures needed by the INPUT CONTROL command. [EXPLAIN INPUT CONTROL.]
Several commands are included under the PERFORM TABLE CREATE INPUT tool set, each of which are very useful for Input Control operation. In using these commands be aware that they are intended as a means to simplify the generation of the structures. It is important to understand that modification of this data may be necessary in order to achieve the desired result -- the generation of SPIRES hierarchical records from multiple streams of input tables.
The central SPIRES structure that is common to these commands is the DECLARE TABLE structure. Output Tables generated by PERFORM TABLE create declare form the key ingredient to the generation of the DECLARE INPUT TABLE structures. The DECLARE INPUT TABLE structure is then in turn forms the basis for the subsequent commands
Syntax of PERFORM TABLE CREATE INPUT
PERFORM TABLE CREATE INPUT (DECLARE | RECDEF | INPUT CONTROL)
SUBFILE = <source of Spires data elements>
KEY = <Record key of source Spires data>
KEY PREFIX= <Key prefix of source Spires data>
DESTINATION = <destination Table subfile>
FROM = <source Declare Table subfile>
So far the command options are:
1. PERFORM TABLE CREATE INPUT DECLARE -- Creates SPIRES DEClare Input Table data. [See 5.7.6.]
2. PERFORM TABLE CREATE INPUT RECDEF -- Generates SPIRES RECDEF records for data input. [See 5.7.7.]
3. PERFORM TABLE CREATE INPUT CONTROL -- Generates SPIRES commands that execute INPUT CONTROL activity. [See 5.7.8.]
PERFORM TABLE CREATE INPUT DECLARE -- Creates SPIRES DECLARE INPUT TABLE structures
This form of the PERFORM TABLE CREATE command will generate SPIRES DECLARE INPUT TABLE structures which will form a basic piece of the set of commands needed in INPUT CONTROL processing. [EXPLAIN DECLARE INPUT TABLE COMMAND.] [EXPLAIN INPUT CONTROL.]
This command is expressed as follows:
The DECLARE TABLE structure stored (and possibly modified)
in the Table Subfile that has the given key value is used
as the source for DECLARE INPUT TABLE generation.
Syntax of PERFORM TABLE CREATE INPUT DECLARE
PERFORM TABLE CREATE INPUT DECLARE
FROM = <Declare table subfile>
KEY = <source record name>
KEY PREFIX <key prefix of source records>
SUBFILE = <source subfile name>
Command options:
PERFORM TABLE CREATE INPUT RECDEF -- Creates SPIRES RECDEF records for input
This form of the PERFORM TABLE CREATE command will generate SPIRES Record Definition (RECDEF) records which mirror the view of data described by Declare Input Table records generated by the PERFORM TABLE CREATE INPUT DECLARE command. [See 5.7.6.]
This command is expressed as follows:
The DECLARE INPUT TABLE structure stored (and modified)
in the Input Table Subfile that has the given key value is
used as the source for RECDEF generation.
Syntax of PERFORM TABLE CREATE INPUT RECDEF
PERFORM TABLE CREATE INPUT RECDEF
FROM = <Declare input table subfile>
KEY = <source record name>
KEY PREFIX <key prefix of source records>
SUBFILE = <destination subfile name>
Command options:
PERFORM TABLE CREATE INPUT CONTROL -- Creates a SPIRES protocol for input control
This form of the PERFORM TABLE CREATE command will generate a SPIRES protocol which is made up of a sample set of commands that will provide the basis for Input Control activity. [EXPLAIN INPUT CONTROL.]
This command is expressed as follows:
The DECLARE INPUT TABLE structure stored (and modified)
in the Input Table Subfile that has the given key value is
used to provide information used in the generated protocol
commands.
Syntax of PERFORM TABLE CREATE INPUT CONTROL
PERFORM TABLE CREATE INPUT CONTROL
FROM = <Declare input table subfile>
KEY = <source record name>
KEY PREFIX <key prefix of source records>
SUBFILE = <destination subfile name>
Command options:
PERFORM TABLE CREATE ELEMINFO -- Creates SPIRES Data Element Information
The PERFORM TABLE CREATE commands have been built to simplify the Process needed to convert SPIRES hierarchical record structures into a form that is acceptable for the row/column structure of data in the Relational model of data storage. This set of PERFORM commands has proved to be extremely useful for the conversion process but difficulties may arise, especially if those assigned to handle the conversion are not familiar with the data being converted.
The PERFORM TABLE CREATE ELEMINFO command in conjunction with the other PERFORM TABLE commands has been constructed as another, potentially important piece of this tool set. This command is designed to produce a SPIRES database which holds information pertaining to every data element of a subfile which in turn may be used to aid the construction of the Row/Column data descriptions in a more precise manner than has been previously available.
The CREATE ELEMINFO process is able to gain this precision by scanning the actual records of the source SPIRES database in order to determine exact length/occurrence values for each data element of a subfile. The resulting data collection may then be used by the Create declare process [See 5.7.1,.] to generate tables that more accurately reflect the actual data attributes that exist in the source subfile rather than the suggested attributes based upon the data description from the SPIRES File Definition.
This process will prove less useful for SPIRES subfiles that have been designed with great care, those whose File Definitions more precisely define its data element attributes. Many SPIRES files have not had such a history. SPIRES provides an extremely flexible, yet efficient opportunity to be less precise, and this flexibility has led to databases that have grown and evolved over decades of time and yet have never been rebuilt. This flexibility was designed and built into SPIRES from the beginning over thirty years ago. The system was designed as the Spires PUBLIC Information Retrieval System, a system in which any user with a Forsythe account could build his/her own private or departmental database application without any contact with a DBA (there weren't any).
The SPIRES to Relational conversion is hampered by the fact of this flexibility due to several aspects of SPIRES design.
1. Variable Length Data - Users rarely specify fixed length
constraints on data values. In most cases, efficiency is not
served by doing this. Of the tens of thousands of SPIRES
databases, I know of only one which was made up of nothing
but fixed length data. Relational systems thrive on it.
2. Multiply Occurring Data - It does help data elements to be
defined as singly occurring when they are singly occurring,
but this is often neglected. It matters greatly in SPIRES
to Relational conversion.
3. Optional Data - A SPIRES file may have a number of Optional
data elements defined which may in fact never occur, or which
have occurred in the past but have bypassed their usefulness
and have been deleted. Optional data that does not appear in
a record requires but one bit of space (or none) to express
this fact.
The current PERFORM TABLE commands would generate a separate table for a data element defined as optional, variable length, multiply occurring even if such an element never occurs in the data base.
This command eliminates such problems since it can scan an entire database and report the existence or non existence of every piece of data in the database.
Syntax of PERFORM TABLE CREATE ELEMINFO
PERFORM TABLE CREATE ELEMINFO
SUBFILE = <source subfile>
DESTINATION = <destination subfile>
INCREMENT = <source record scan increment>
Command options:
Examples of command execution in SPIRES
> perform table create eleminfo subfile restaurant - Scanned 939 records of 939 in RESTAURANT - Generated 23 Element Info records > list unn 1April 2, 2003 Page 1 name elemid type exists max_len max_occ avg_xoc ---------------- -------- --------- ------ ------- ------- ------- ID 00000000 Integer 939 4 1 1.00 DATE-ADDED 00000001 Hex 939 14 1 1.00 DATE-UPDATED 00000002 Hex 939 14 1 1.00 NAME 00000003 String 939 38 1 1.00 LOCATION 00000004 Structure 939 0 1 1.00 RECOMMENDER 00000005 String 700 83 4 1.01 COMMENTS 00000006 String 936 988 1 1.00 CUISINE 00000007 String 809 80 3 1.01 PRICE-RANGE 00000008 Integer 5 5 1 1.00 PRICE-AVERAGE 00000009 Integer 776 7 1 1.00 FOOD-QUALITY 0000000A Integer 851 9 1 1.00 SERVICE-QUALITY 0000000B Integer 823 9 1 1.00 CREDIT 0000000C Integer 557 5 1 1.00 RESERVATIONS 0000000D String 615 74 1 1.00 HOURS 0000000E String 509 78 1 1.00 ENTERTAINMENT 0000000F String 593 135 1 1.00 BAR 00000010 Integer 557 9 3 1.01 CITY 00000200 String 939 20 1 1.00 STATE 00000201 String 939 2 1 1.00 PHONE 00000202 String 516 14 1 1.00 ADDRESS 00000203 String 870 41 1 1.00 ZIP-CODE 00000204 String 95 5 1 1.00 LOC.D 00000205 String 0 0 0 0
This display gives a snapshot of the information gathered for each of the data elements in the RESTAURANT subfile. The data shown for any particular data element may be interpreted as follows:
name - Data element name.
elemid - SPIRES internal element Identifier
type - SPIRES internal element type
exists - Number of records in which the element exists
max_len - Maximum length of this element within the subfile
(This value is meaningless for type STRUCTURE elements)
max_occ - maximum number of occurrences of the element in any record
avg_xoc - Average number of occurrences of the element in the
records in which the element exists.
This snapshot display may prove useful in that it gives a good idea of data element usage but if you wish to utilize this process to its maximum potential during SPIRES to relational table conversions you should make use of the DESTINATION parameter.
The following is a sample of the usage of this parameter.
> perform table create eleminfo subfile restaurant dest eleminfo
- Scanned 939 records of 939 in RESTAURANT
- Generated 23 Element Info records
> select eleminfo
> for subfile where subfile.name = restaurant
> display 2
----------------------------------------
Element Name: ADDRESS in Structure LOCATION
File GQ.DOC.RESTAURANT Subfile: RESTAURANT
Internal Type: String Optional, Singular
Exists in 870 Records of 939 Scanned
Occurrences: Total - 870 Maximum - 1 Average - 1.00
Elem Length: Total - 15827 Maximum - 41 Average - 18.19
A singly occurring element which gives the street address
for the restaurant. A warning message will be issued if the
value is over 36 characters long.
----------------------------------------
Element Name: BAR
File GQ.DOC.RESTAURANT Subfile: RESTAURANT
Internal Type: Integer Optional, Multiple
Exists in 557 Records of 939 Scanned
Occurrences: Total - 564 Maximum - 3 Average - 1.01
Elem Length: Total - 3231 Maximum - 9 Average - 5.73
Indicates what kind of bar is available. The four
possible choices are:
None, Beer, Wine, Cocktails.
Other values will not be accepted.
The unformatted output for the preceding record is as follows:
> transfer *
> l
1. KGROUP = BAR /RESTAURANT;
2. FILE.NAME = GQ.DOC.RESTAURANT;
3. DATE.ADD = WED. APRIL 9, 2003;
4. TYPE = Integer;
5. OCCURS = Multiple;
6. PART = Optional;
7. ELEMID = 00000010;
8. TOT_RECORDS = 939;
9. MAX_OCC = 3;
10. MAX_OCST = 3;
11. TOT_OCC = 564;
12. MAX_LEN = 9;
13. TOT_LEN = 3231;
14. EXISTS = 557;
15. ELEMINFO;
16. DESCRIPTION = Indicates what kind of bar is available. The four
17. possible choices are:;
18. DESCRIPTION = " None, Beer, Wine, Cocktails.";
19. DESCRIPTION = Other values will not be accepted.;
20. HEADING = Bar;
21. WIDTH = 9;
22. VALUE-TYPE = CODED;
>
As you can see from this example, the information collected during the CREATE ELEMINFO process is much more extensive than the snapshot shown in the preceding example.
For a more complete description of the information being gathered and how you can build your own element information subfile [See 5.7.10,.]
Also [See 5.7.11,.] to see how you can utilize the element information subfile to improve the generation of relational tables in conjunction with the CREATE DECLARE process.
PERFORM TABLE CREATE DECLARE, USING ELEMINFO -- Generate SPIRES tables
You have the ability to build SPIRES Declare structures that are based upon data derived from the actual records in the source subfile database through the utilization of records stored in a second SPIRES database. The generation of this second database is described by [See 5.7.9,.]
The example shown in that explanation shows how to generate records for the data elements of the SPIRES RESTAURANT subfile. You may also [See 5.7.10,.] to see how that "second' subfile whose name is "eleminfo" is defined. Of course, you should create your own "eleminfo" subfile or have access to one that has been set up for you.
The PERFORM TABLE CREATE DECLARE command below depicts how you may use this element information subfile when generating the Declare Table structures.
Note the FROM parameter which gives the name of the subfile holding the gathered information from RESTAURANTS. Also, note that we are using the 'EXISTS" option. The EXISTS option tells SPIRES to ignore generating row/column data for data elements which are defined but which do not exist in the subfile.
It may be instructive to see how this "ELEMINFO" feature generates more efficient table structures. The "cuisine" data element is defined as a multiply occurring, variable length data element. However, the CREATE ELEMINFO command generated the following record.
Element Name: CUISINE
File GQ.DOC.RESTAURANT Subfile: RESTAURANT
Internal Type: String Optional, Multiple
Exists in 809 Records of 939 Scanned
Occurrences: Total - 820 Maximum - 3 Average - 1.01
Elem Length: Total - 12120 Maximum - 80 Average - 14.78
A textual description of the type(s) of cuisine the
restaurant offers.
The description should be fairly brief.
The following command shows that a separate CUISINE table was not generated since the maximum number of occurrences in any record was three. The cuisne field translates into three columns in the RESTAURANT table. In contrast the RECOMMENDER element had a maximum number of four occurrences in a record. Hence the separate table shown below.
> perform table create declare subfile restaurant ...
from eleminfo type oracle options 'mult,exists'
> list
1. Declare Table RESTAURANT
2. ID = GQ.WCK.RESTAURANT;
3. COMMENTS = Table Generation for Record Level Elements;
4. DEFDATE = WED. APRIL 9, 2003;
5. MODDATE = WED. APRIL 9, 2003;
6. MODTIME = 10:13:56;
7. SUBFILE.NAME = RESTAURANT;
8. DECLARE.SUBFILE = Data Move Declares;
9. COLNAME = ID;
10. ISKEY;
11. COLTYPE = INT;
12. COLWIDTH = 4;
13. RDBMS_COLUMN = id;
14. RDBMS_DATATYPE = NUMBER;
15. RDBMS_DATALENGTH = 4;
16. COLNAME = DATE-ADDED;
17. COMMENTS = Required;
18. COLTYPE = DATE;
19. COLWIDTH = 15;
20. SOURCE.ELEM = DATE-ADDED;
21. DECLARE.KEY = $DATEOUT.CCYY;
22. RDBMS_COLUMN = date_added;
23. RDBMS_DATATYPE = DATE;
24. COLNAME = DATE-UPDATED;
25. COMMENTS = Required;
26. COLTYPE = DATE;
27. COLWIDTH = 15;
28. SOURCE.ELEM = DATE-UPDATED;
29. DECLARE.KEY = $DATEOUT.CCYY;
30. RDBMS_COLUMN = date_updated;
31. RDBMS_DATATYPE = DATE;
32. COLNAME = NAME;
33. COMMENTS = Required;
34. COLWIDTH = 38;
35. RDBMS_COLUMN = name;
36. RDBMS_DATATYPE = VARCHAR2;
37. RDBMS_DATALENGTH = 38;
38. COLNAME = COMMENTS;
39. COLWIDTH = 988;
40. RDBMS_COLUMN = comments;
41. RDBMS_DATATYPE = VARCHAR2;
42. RDBMS_DATALENGTH = 988;
43. COLNAME = CUISINE_1;
44. COMMENTS = Multiple;
45. COLWIDTH = 80;
46. SOURCE.ELEM = CUISINE;
47. DECLARE.KEY = $VALUE.EXTERNAL;
48. SOURCE.OCC = 1;
49. RDBMS_COLUMN = cuisine_1;
50. RDBMS_DATATYPE = VARCHAR2;
51. RDBMS_DATALENGTH = 80;
52. COLNAME = CUISINE_2;
53. COMMENTS = Multiple;
54. COLWIDTH = 80;
55. SOURCE.ELEM = CUISINE;
56. DECLARE.KEY = $VALUE.EXTERNAL;
57. SOURCE.OCC = 2;
58. RDBMS_COLUMN = cuisine_2;
59. RDBMS_DATATYPE = VARCHAR2;
60. RDBMS_DATALENGTH = 80;
61. COLNAME = CUISINE_3;
62. COMMENTS = Multiple;
63. COLWIDTH = 80;
64. SOURCE.ELEM = CUISINE;
65. DECLARE.KEY = $VALUE.EXTERNAL;
66. SOURCE.OCC = 3;
67. RDBMS_COLUMN = cuisine_3;
68. RDBMS_DATATYPE = VARCHAR2;
69. RDBMS_DATALENGTH = 80;
70. COLNAME = PRICE-RANGE;
71. COLWIDTH = 5;
72. RDBMS_COLUMN = price_range;
73. RDBMS_DATATYPE = VARCHAR2;
74. RDBMS_DATALENGTH = 5;
..........
105. COLNAME = ENTERTAINMENT;
106. COLWIDTH = 135;
107. RDBMS_COLUMN = entertainment;
108. RDBMS_DATATYPE = VARCHAR2;
109. RDBMS_DATALENGTH = 135;
110. COLNAME = BAR_1;
111. COMMENTS = Multiple;
112. COLWIDTH = 9;
113. SOURCE.ELEM = BAR;
114. DECLARE.KEY = $VALUE.EXTERNAL;
115. SOURCE.OCC = 1;
116. RDBMS_COLUMN = bar_1;
117. RDBMS_DATATYPE = VARCHAR2;
118. RDBMS_DATALENGTH = 9;
119. COLNAME = BAR_2;
120. COMMENTS = Multiple;
121. COLWIDTH = 9;
122. SOURCE.ELEM = BAR;
123. DECLARE.KEY = $VALUE.EXTERNAL;
124. SOURCE.OCC = 2;
125. RDBMS_COLUMN = bar_2;
126. RDBMS_DATATYPE = VARCHAR2;
127. RDBMS_DATALENGTH = 9;
128. COLNAME = BAR_3;
129. COMMENTS = Multiple;
130. COLWIDTH = 9;
131. SOURCE.ELEM = BAR;
132. DECLARE.KEY = $VALUE.EXTERNAL;
133. SOURCE.OCC = 3;
134. RDBMS_COLUMN = bar_3;
135. RDBMS_DATATYPE = VARCHAR2;
136. RDBMS_DATALENGTH = 9;
137. FILE = GQ.DOC.RESTAURANT;
138. TABLE.NUM = 001;
139. RDBMS_TABLENAME = restaurant;
140. EndDeclare
141.
142. Declare Table LOCATION
143. ID = GQ.WCK.LOCATION;
144. COMMENTS = Table Generation for Structure LOCATION Elements;
145. DEFDATE = WED. APRIL 9, 2003;
146. MODDATE = WED. APRIL 9, 2003;
147. MODTIME = 10:13:56;
148. SUBFILE.NAME = RESTAURANT;
149. DECLARE.SUBFILE = Data Move Declares;
150. COLNAME = ID;
151. COMMENTS = REF RESTAURANT;
152. ISKEY;
153. COLTYPE = INT;
154. COLWIDTH = 4;
155. COLUMN.OPTIONS = Dependent, NoSubtree;
156. RDBMS_COLUMN = id;
157. RDBMS_DATATYPE = NUMBER;
158. RDBMS_DATALENGTH = 4;
159. COLNAME = CITY;
160. ISKEY;
161. COLWIDTH = 20;
162. RDBMS_COLUMN = city;
163. RDBMS_DATATYPE = VARCHAR2;
164. RDBMS_DATALENGTH = 20;
165. COLNAME = STATE;
166. COMMENTS = Required;
167. COLWIDTH = 2;
168. RDBMS_COLUMN = state;
169. RDBMS_DATATYPE = VARCHAR2;
170. RDBMS_DATALENGTH = 2;
171. COLNAME = PHONE;
172. COLWIDTH = 15;
173. RDBMS_COLUMN = phone;
174. RDBMS_DATATYPE = VARCHAR2;
175. RDBMS_DATALENGTH = 15;
176. COLNAME = ADDRESS;
177. COLWIDTH = 41;
178. RDBMS_COLUMN = address;
179. RDBMS_DATATYPE = VARCHAR2;
180. RDBMS_DATALENGTH = 41;
181. COLNAME = ZIP-CODE;
182. COMMENTS = Multiple;
183. COLWIDTH = 5;
184. SOURCE.SINGLE;
185. RDBMS_COLUMN = zip_code;
186. RDBMS_DATATYPE = VARCHAR2;
187. RDBMS_DATALENGTH = 5;
188. FILE = GQ.DOC.RESTAURANT;
189. SOURCE.STRUCTURE = LOCATION;
190. TABLE.NUM = 002;
191. RDBMS_TABLENAME = location;
192. EndDeclare
193.
194. Declare Table RECOMMENDER
195. ID = GQ.WCK.RECOMMENDER;
196. COMMENTS = Table Generation for Element RECOMMENDER;
197. DEFDATE = WED. APRIL 9, 2003;
198. MODDATE = WED. APRIL 9, 2003;
199. MODTIME = 10:13:56;
200. SUBFILE.NAME = RESTAURANT;
201. DECLARE.SUBFILE = Data Move Declares;
202. COLNAME = ID;
203. COMMENTS = REF RESTAURANT;
204. ISKEY;
205. COLTYPE = INT;
206. COLWIDTH = 4;
207. RDBMS_COLUMN = id;
208. RDBMS_DATATYPE = NUMBER;
209. RDBMS_DATALENGTH = 4;
210. COLNAME = RECOMMENDER.OCC;
211. COMMENTS = Occurrence Number;
212. ISKEY;
213. COLTYPE = INT;
214. SOURCE.ELEM = RECOMMENDER;
215. DECLARE.KEY = $VALUE.OCC;
216. SOURCE.SINGLE;
217. RDBMS_COLUMN = recommender_occ;
218. RDBMS_DATATYPE = NUMBER;
219. RDBMS_DATALENGTH = 4;
220. COLNAME = RECOMMENDER;
221. COMMENTS = Multiple;
222. COLWIDTH = 83;
223. RDBMS_COLUMN = recommender;
224. RDBMS_DATATYPE = VARCHAR2;
225. RDBMS_DATALENGTH = 83;
226. FILE = GQ.DOC.RESTAURANT;
227. MULTI.ELEMENT = RECOMMENDER;
228. TABLE.NUM = 003;
229. RDBMS_TABLENAME = recommender;
230. EndDeclare
231.
>
The SPIRES Data Move process has been implemented as a PERFORM command which has the advantage of simplifying the input to the original process. The command syntax is as follows:
PERFORM DATA MOVE FROM key-value, FOR record-set, COUNT count,
UPDATE update-type, DEBUG
where:
- key-value is the key of a record in the DATA MOVE subfile.
- record-set gives the source class for Global FOR such as
SUBFILE/ RESULT/ DEFQ/ STACK ...
(if record-set is not given then DATA MOVE will assume that
global FOR command has already been set up. In this way
you can restrict the goal records at the time of the
PERFORM including SET SCAN etc.).
- count specifies the number of records to process.
If this value is not given, ALL is assumed.
- update-type specifies the way that target records are to
be stored into target "subfiles". Values may be ADD,
ADDUPD, MERGE, ADDMERGE, FASTADD, ...
If this value is not given, ADD is assumed.
- DEBUG may be specified to provide a trace of DATA MOVE execution.
To learn more about this feature [EXPLAIN DATA MOVE PROCESSING.]
This chapter presents detailed descriptions of the commands that handle the file management tasks discussed in the earlier part of this manual. The commands appear alphabetically, and each begins with a standard set of information:
- a brief statement defining the purpose of this command
- which SPIRES programs the command is available in (SPIRES, SPIBILD, FASTBILD)
- related commands, i.e., commands often used in conjunction with this command, or discussed in the same breath
- "recovery commands", i.e., commands that reverse or undo what this command does
- procedures described earlier in the manual that use the command, listed with the section number where they can be found
Following that list is a formal description of the command's syntax and a general description of the command. Most of the descriptions also include one or more examples showing how the command is used.
The ATTACH command "attaches" a record-type of the named file for you to use as if it were the goal record-type of a selected subfile.
Command: ATTACH
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: SELECT
Recovery commands: CLEAR SELECT
SELECT
Procedures: Examining Record-Types Directly (5.4)
ATTACH [{rec-name|rec-number} OF] [ORV.gg.uuu.]filename
Issue the ATTACH command when you want or need to use a record-type in a SPIRES file without going through the subfile-selection process, e.g., to use a record-type of the file for which no subfile has been defined, or to avoid the restrictions or select-commands for a subfile. In general, both ATTACH and SELECT have the same net effect, except that ATTACH does not involve the qualifications listed in the Subfile section of the file definition.
Here is an example using the ATTACH command:
-> attach zin04 of orv.gq.doc.errata -> show select -Record-type ZIN04 of GQ.DOC.ERRATA attached ->
At this point, record-type ZIN04, the fourth record-type in the file, can be used almost as if it were a "selected" goal record-type. The following commands, for example, could be issued in SPIRES against the attached record-type:
SHOW INDEXES BROWSE GOAL FIND FOR SUBFILE DISPLAY SHOW SUBFILE SIZE
But on the other hand, when a file is attached, the SPIRES variable $SELECT (the name of the selected subfile) is null, and $SELECTED (a flag indicating whether a subfile is selected or not) is "0" (NO). [Note: The cited examples are not an exhaustive list of what does and does not work with the ATTACH command.]
The ATTACH command is most frequently used when you want to examine records in an index record-type (e.g., get a list of all the indexed values). If the index record-type is defined as the goal record-type of another subfile, you could SELECT the subfile in order to examine the records. But in most cases, the index record-type is not a goal record-type of a subfile, so it cannot be directly accessed through a SELECT command. Instead, you must use the ATTACH command. [See 5.4.]
The AVSPREC command (for AVailable SPace RECovery) rebuilds the available-space chains for the Residual data set.
Command: AVSPREC
Available in: SPIRES
Available to: file owner
accounts with Master access to the file
Related commands: STATUS
Recovery commands: AVSPREC
Procedures: AVSPREC procedure (3.4.3)
[IN ACTIVE [CLEAR|CONTINUE]] AVSPREC [CONTINUE] ... ... [NORESERVE] [LIMIT = block.count]
SPIRES keeps track of the amount of unused space in each block of the Residual data set. SPIBILD creates a "chain" of the blocks having 64 bytes free, another chain of those having 128 bytes free, etc. Each block in each chain points to the next block in the chain. SPIBILD uses the chains when determining where to store a new record, or where to put the continuation of an updated record when there is not enough space in the record's original block.
When any of the records in a block are updated or removed, the amount of available space in the block is likely to change. However, SPIBILD can move the block to the correct chain only if it is the first block on the chain, in which case it is moved to become the first block of the correct chain. If the block is not the first block in the chain, SPIBILD cannot put it on the correct chain, so it becomes a "block on the wrong chain".
For a block to be on the wrong chain is not a terrible problem, by any means; it does not damage your file in any way. It simply means that SPIBILD may not be able to store new and updated records as efficiently as possible. For example, if a block is on the 640-byte available-space chain but record updates and removes have created 2000 bytes of available space in it, that extra space is likely to remain unused for a long time, if not forever. [On its own, SPIBILD may recover the unused space, but in many cases, that would be unlikely.]
The AVSPREC command examines each block of each chain, rebuilding the chains, moving the blocks on wrong chains onto the right ones. By moving blocks with more available space than the chain they are on indicates, SPIBILD will more likely use the space in the future. [See 3.4.3.]
Here is an example showing how to use the AVSPREC command:
-> spires -Welcome to SPIRES -> select restaurant -> avsprec -OK to clear? yes ->
The AVSPREC command may take some time, since it must read all and usually write many of the blocks in the Residual, typically the largest data set in a SPIRES file.
SPIRES will place a list of the chains and the number of blocks on each in your active file. It will match the list of chains and "Actual blocks" shown in the AVSP display of the STATUS command.
If the system crashes before the AVSPREC command finishes its work, you should restart the operation as soon as convenient by using the CONTINUE option, e.g., AVSPREC CONTINUE. However, if the file gets processed before the AVSPREC operation is restarted, no harm will be done.
The BUILD command passes data from goal records of the established subfile to one or more indexes.
Command: BUILD
Available in: SPIBILD, FASTBILD
Available to: file owner
users with Master access to the file
Related commands: ESTABLISH, FOR SUBFILE, FOR STORED
Recovery commands: ZAP RECORD-TYPE, ZAP DATA SET
Procedures: Add an Index (2.7.1)
Rebuild an Index (2.7.3)
FAST BATCH procedure in SPIBILD (2.3.4)
FAST BATCH procedure in FASTBILD (2.12.2)
Under Global FOR in SPIBILD: BUILD index.name[(n)][, index.name[(n)] ...] Outside of Global FOR in SPIBILD, and in FASTBILD: BUILD index.name OF subfile-name
BUILD REC02(2)
If no occurrence is specified, all linkage specifications for the named index are used by SPIRES.
The BUILD command is most commonly used in these four situations:
- when the file owner decides to add a new index to a subfile;
- when the file owner decides to pass another element's value to an existing index;
- when data integrity problems with an index occur, and the index must be zapped and rebuilt.
- after a FAST BATCH command, to build the subfile's indexes.
If you use the command under Global FOR in SPIBILD, you can build one or more than one index with a single command. Before using the BUILD command, issue the ESTABLISH command to select the desired subfile. Then issue the FOR SUBFILE command, with or without a WHERE clause and any SET SCAN commands desired, to tell SPIBILD which records to use in building the index. You are then ready to issue the BUILD command. [You can also use BUILD under FOR STORED, where the records to be used are specified in a stored stack or result.]
BUILD can be used to build a single index in FASTBILD and in SPIBILD without Global FOR. In each case, SPIBILD or FASTBILD examines all the goal records of the subfile to create entries to the index. Note that the procedure in SPIBILD outside of Global FOR is simpler though less versatile than its counterpart in Global FOR: you simply issue the BUILD command, naming the index and subfile, after entering SPIBILD. Be aware that the file's deferred queue must be empty for the command to succeed; if it is not empty, SPIBILD will tell you to process the file before issuing the BUILD command.
Below is a sample SPIBILD session showing the BUILD command used to add more data to an index that already exists. In this case, the file owner originally decided that the data value NONE for the PETS element would not be passed to the PETS index (record-type ZIN03) but has now reversed that decision; records with NONE should be represented in the index.
First, the file owner changes the file definition to allow the value NONE to be passed to the index. Then, the BUILD command is used to pass the appropriate PETS index data for those records with NONE for PETS. The file owner uses a WHERE clause on the FOR SUBFILE command to limit the passed goal records to those with the value NONE.
-? establish homeowners -Processing file: GQ.DOC.HOMES -Compute time: 0.134 seconds -Elapsed time: 1.163 seconds -Core usage: 0064/0939 -? for subfile where pets = none -? build zin03 -Passing: 36 records of record-type REC01 -End of BATCH/BUILD -Compute time: 1.774 seconds -Elapsed time: 14.502 seconds -?
The CLEAR EXCEPTION command detaches the exception file used during multiple-record input.
Command: CLEAR EXCEPTION
Available in: SPIRES, SPIBILD, FASTBILD
Related commands: SET EXCEPTION, ERASE
Recovery commands: SET EXCEPTION
Procedures: Exception File procedure (2.1.5.2)
INPUT BATCH procedure in SPIRES (2.3)
INPUT ADD procedure in SPIRES (2.3)
INPUT MERGE procedure in SPIRES (2.3)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
FAST BATCH procedure in SPIBILD (2.3.4)
FAST BUILD procedure in FASTBILD (2.12.2)
INPUT MERGE procedure in SPIBILD (2.4.1)
CLEAR EXCEPTION
There are no options on the CLEAR EXCEPTION command.
If you decide not to use an exception file after setting one for a multiple-record input procedure, issue the CLEAR EXCEPTION command to detach it. No exception processing will occur till another SET EXCEPTION command is issued.
As soon as SPIRES completes the execution of a command that uses an exception file (such as BATCH), it automatically detaches the file. If further exception processing is to be done in this SPIRES program session, another SET EXCEPTION command must be issued.
After you issue the CLEAR EXCEPTION command, the ORVYL data set created by the SET EXCEPTION command still exists, even though no data has been put into it. Issue the ORVYL command ERASE followed by the name of the exception file to erase it.
COMMAND> spibild -Welcome to SPIBILD -? set exception rec.wrecks ... (you change your mind about using an exception file) -? clear exception -? erase rec.wrecks REC.WRECKS erased in ORVYL File System -?
The CLEAR FILE COUNTS command turns off the display of file I/O statistics in SPIBILD and resets the counters.
Command: CLEAR FILE COUNTS Available in: SPIBILD, FASTBILD, SPIRES Available to: anyone Related commands: SET FILE COUNTS Recovery commands: SET FILE COUNTS Procedures: any procedure
CLEAR FILE COUNTS
There are no options on the CLEAR FILE COUNTS command.
By default, SPIBILD and FASTBILD automatically show file count information as a regular part of their output. The CLEAR FILE COUNTS command can be used to turn the automatic display off. It can be turned on again (with system file count data too) with the SET FILE COUNTS command. Unlike the way it works in SPIRES, however, it does not reset the system file counts.
In SPIRES, since there is no automatic display of file count data, CLEAR FILE COUNTS completely resets the internal tables used by the command.
The COPY DATA SET command copies an ORVYL data set from the attached SPIRES file to another ORVYL data set, possibly as part of another SPIRES file
Command: COPY DATA SET
Available in: SPIRES
Available to: file owner
users with Copy access to the file
Related commands: COPY FILE, COPY DEFQ
Recovery commands: ERASE, COPY DATA SET
COPY DATA SET dsname1 TO dsname2 [REPLACE|SCRATCH] [NOWARN]
[NORESERVE] [NOFIX] [NONIXR [= n]] [RENAME]
- CKPT -- the file's checkpoint data set
- DEFQ -- the file's deferred queue data set
- MSTR -- the file's Master data set
- RES, RES1, ... RESn -- one of the file's residual data sets
- REC1, REC2, ... RECn -- one of the file's record-type data sets
Maximum size (blocks) Type
--------------------- --------------------------------------
256 CKPT, MSTR
425,984 RES, RESn
434,176 extended-tree REC data sets and
and extended-defq data sets
65,536 regular REC and DEFQ data sets
The COPY DATA SET command makes a copy on your account of one ORVYL data set, specified by you, from the attached file. It is very flexible, and you may find many different uses for it. You may use it simply to make a backup copy of one or more data sets before you take some particular file management action, such as rebalancing a tree or creating a non-IXR data set.
Note: File permits are copied only from the source data set to the copy only when the RENAME option is specified.
Here is a sample session showing how you might create a non-IXR DEFQ, hence avoiding the overhead during transactions of requesting file blocks from ORVYL, useful for files with heavy transaction loads:
-> sel restaurant -> copy data set defq to zzzz rename Start copy of 2 blocks from data set DEFQ ORV.GQ.DOC.RESTAURANT.DEFQ is renamed to ORV.GQ.DOC.RESTAURANT.DEFQ.19961201 ORV.GQ.DOC.ZZZZ is renamed to ORV.GQ.DOC.RESTAURANT.DEFQ File detached ->
The value for "dsname2", ZZZZ in this example, was an arbitrary choice, since it is used only temporarily when the RENAME option is used.
The warning messages you can receive from this command include the following:
The COPY FILE command copies the ORVYL data sets that comprise a SPIRES file from another account to the logged-on account. The COPY DEFQ command is the same as COPY FILE but copies only the file's DEFQ data set.
Command: COPY FILE
COPY DEFQ
Available in: SPIRES
Available to: users with Copy access to the file
Related commands: none
Recovery commands: ZAP FILE, COPY FILE
Procedures: Backup the DEFQ
Move a File (2.10)
COPY FILE ORV.gg.uuu.filename [SHARE] COPY DEFQ ORV.gg.uuu.filename [SHARE]
The COPY FILE command makes a copy on your account of each ORVYL data set of the file owner's file. COPY DEFQ makes a copy of only the DEFQ data set of the file owner's file.
The original file and the copy will have the same file name, but with different account prefixes. Note that both commands only make copies; the original file still exists on the file owner's account.
Here is a sample session showing how SPIRES responds to the COPY FILE command:
-> copy file orv.gq.doc.restaurant Start copy of 50 blocks from data set DEFQ Start copy of 5 blocks from data set MSTR Start copy of 4 blocks from data set REC1 Start copy of 35 blocks from data set REC2 Start copy of 108 blocks from data set RES ->
The "Start copy" message appears as SPIRES starts to copy each data set. It tells you which data set is being copied, as well as how many blocks are in it.
Warning: The COPY FILE and COPY DEFQ commands will fail under the following circumstances:
- If you (the file copier) do not have COPY access to the file, as declared in the FILE-PERMITS structure of the file definition, you will receive the message "Privileged command". [Accounts at a SPIRES site that have "master privileges" in SPIRES (such as the SPIRES consultant) automatically have COPY access to all files, except files whose ORVYL-data-set permits have been changed to forbid such access.]
- If the file is being used by another account and the SHARE option has not been specified,
the command will fail with an S56 error ("File is busy, not currently
available."). This in effect prevents you from copying the file while data within the
file may be changing. You should NOT override this situation with the SHARE option unless
you are positive that other users are not updating records, or otherwise changing the
file's contents.
- If your account does not have enough ORVYL blocks allocated to it, the command will fail. If you need more blocks, contact the computer center's accounting office before trying the COPY FILE or COPY DEFQ command again.
- If a file with the same name already exists on your account, the command will fail with an S40 error.
Technical note: The copy of the file is identical to the original file except in two regards:
- the file-owner account information stored in the MSTR data set is changed by SPIRES. If SPIRES did not change the account number from the original file owner's to the copying account's, the copy could never be recompiled or otherwise linked to a new file definition.
- the ORVYL data set permits for the copy are set to the SPIRES defaults rather than those of the original data sets. The defaults are READ access to the public for all data sets except the DEFQ, which has public WRITE access. To set the same permits as the original file (usually necessary for files with immediate indexing), contact the file owner, who can follow the procedure described at the end of the "Move a file" procedure. [See 2.10.]
The ESTABLISH command moves subfile transactions from the file's deferred queue to the tree, and updates the file's indexes accordingly. It also establishes a subfile to be used by the INPUT commands, or by any merge or remove operations using Global FOR techniques that may follow.
Command: ESTABLISH
Available in: SPIBILD
Available to: file owner
users with Process access to the file
Related commands: PROCESS, FOR SUBFILE, FOR STORED
Recovery commands: ESTABLISH, RECOVER, SET JOBGEN
Procedures: PROCESS/ESTABLISH procedure (2.2.2)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
INPUT ADDUPDATE procedure in SPIBILD (2.3.2)
INPUT ADDMERGE procedure in SPIBILD (2.3.2)
INPUT MERGE procedure in SPIBILD (2.4.1)
MERGE procedure in SPIBILD in Global FOR (2.4.2)
REMOVE procedure in SPIBILD in Global FOR (2.5.1)
For online SPIBILD and in BATWYL jobs: ESTABLISH [&gg.uuu.filename] subfile-name For batch SPIBILD: ESTABLISH [&gg.uuu.filename] 'subfile-name' [ACCOUNT gg.uuu]
The ESTABLISH command, like PROCESS, causes a SPIRES file to be processed. However, it has two major differences from PROCESS:
- 1) The ESTABLISH command names a subfile whose file is to be processed; PROCESS names the file directly.
- 2) ESTABLISH does not "release" the file so that users can begin using its subfiles again. ESTABLISH optionally serves as the first part of a procedure to merge or remove records in SPIBILD using Global FOR techniques, so it keeps the file reserved for your use until you leave SPIBILD or otherwise end the procedure. [See 2.1.5.1 for the SET SEARCH/SET SHARE procedure, however.]
Here is an example of the ESTABLISH command in a SPIBILD session:
-? establish restaurant -Passing: 3 records of record-type ID -Processed: 3 DEFQ records of record-type ID -Compute time: 1.062 seconds -Elapsed time: 8.551 seconds -Core usage: 0104/0939 -?
The output from the command is exactly the same as a PROCESS command would be. For information about the messages, see the explanations under the PROCESS command.
The EXIT command clears the current ORVYL program from memory, returning you to WYLBUR.
Command: EXIT Available in: SPIRES, SPIBILD Available to: any user Related commands: BREAK (in SPIRES), LOGOFF Recovery commands: none
EXIT [QUIET]
EXIT is an ORVYL command that clears the current ORVYL program from memory and detaches all input/output devices (e.g., the ORVYL data sets of selected subfiles and subgoals). It returns you to WYLBUR's control.
You don't need to issue the EXIT command to go from one SPIRES program to another (for instance, from SPIRES to SPIBILD). You can simply issue the desired command from within the processor.
Similarly, to logoff from inside a SPIRES program, you can just issue the LOGOFF command; you don't have to EXIT the program first.
The example below shows the EXIT command used in SPIBILD:
-? exit -Exiting SPIBILD Command>
The FAST BATCH command adds multiple records to a subfile very quickly, but without checkpointing or indexing the data.
Command: FAST BATCH
Available in: SPIBILD
Available to: file owner
users with Process or Master access to the file
Related commands: SET NOPASS, BUILD
Recovery commands: ZAP RECORD-TYPE, ZAP FILE
Procedures: FAST BATCH procedure (2.3.4)
For online SPIBILD and in BATWYL jobs:
FAST BATCH [&gg.uuu.filename] subfile-name
For batch SPIBILD jobs:
FAST BATCH [&gg.uuu.filename] 'subfile-name' [ON ddname]
[ACCOUNT gg.uuu]
You can use the FAST BATCH command in SPIBILD to add multiple records in a single "batch" using the standard SPIRES format or a customized input format. (Remember: Be sure to set the format before issuing the FAST BATCH command.) [See 2.3.4.]
FAST BATCH is much less versatile than SPIBILD's BATCH command. In fact, there are several important limitations and restrictions that you should know before using FAST BATCH:
It's usually best to process the file first using the PROCESS/ESTABLISH procedure so that the deferred queue will be empty when you start. [See 2.2.2.] Then issue the SET NOPASS command before issuing the FAST BATCH command. (FAST BATCH would also process the file, but you generally don't want SET NOPASS in effect when you are processing the deferred queue prior to the primary work of the FAST BATCH command.)
Next SPIBILD begins to process the input data. Unless you are using batch SPIBILD with the "ddname" option, SPIBILD reads the input data from your active file, so the data should be there before you issue the BATCH command. SPIBILD will display messages for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]
The processing is quite straightforward -- the data goes directly into the tree without checkpointing or indexing, which speeds the processing considerably. But remember, if a system problem occurs while the records are being added, you will need to zap the record-type and start again.
Here is a sample session demonstrating the FAST BATCH command in SPIBILD. The file is explicitly processed before setting NOPASS in case there is data already in the deferred queue that should be processed.
-> spibild -Welcome to SPIBILD -? process students -Passing: 3 records of record-type REC01 -Processed: 3 DEFQ records of record-type REC01 -Compute time: 0.146 seconds -Elapsed time: 1.912 seconds -Core usage: 0065/0939 -? set nopass -? fast batch classroom tables - ADD 1, 1 @Line 5. Key = ADDISON-100 - ADD 2, 2 @Line 10. Key = ADDISON-101 - ADD 3, 3 @Line 15. Key = ADDISON-102 ... (etc.) - ADD 207, 207 @Line 1035. Key = WESLEY-022 - Requests/Success: ADD 207 207 SUM 207 207 -End of BATCH/BUILD -Compute time: 2.151 seconds -Elapsed time: 12.789 seconds -Core usage: 0257/0939 -?
The FAST LOAD command adds multiple records to a subfile very quickly, using data already in SPIRES internal format, but without checkpointing or indexing the data.
Command: FAST LOAD
Available in: SPIBILD
Available to: file owner
users with Process access to the file
Related commands: LOAD, SET NOPASS, BUILD
Recovery commands: ZAP RECORD-TYPE, ZAP FILE
Procedures: FAST LOAD procedure (2.3.4)
FAST LOAD loadname [RENUMBER] INTO record-name OF filename
The FAST LOAD command takes the data created by the GENERATE LOAD command and loads it into a SPIRES file. It begins by processing the file, and then adds the records in the load to the specified record-type one by one.
The record-type must have the same characteristics as the one from which the data came. For example, it must have the same fixed elements, with the same lengths and occurrences; the same structures, having the same number and type of elements, etc. [See the SPIRES manual "File Definition" for a discussion of "matching record-type characteristics".]
Because the input data is already in the internal format used by SPIRES to store records, the data is not processed through any processing rules; Inprocs and Incloses are ignored. The only data that can possibly change as the data is added is the record key, if they are slot keys, as requested by the RENUMBER option.
FAST LOAD is much less versatile than SPIBILD's LOAD command. In fact, there are several important limitations and restrictions that you should know before using FAST LOAD:
It's usually best to process the file first using the PROCESS/ESTABLISH procedure so that the deferred queue will be empty when you start. [See 2.2.2.] Then issue the SET NOPASS command before issuing the FAST LOAD command. (FAST LOAD would also process the file, but you generally don't want SET NOPASS in effect when you are processing the deferred queue prior to the primary work of the FAST LOAD command.)
Next SPIBILD begins to process the input data. SPIBILD will display messages for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]
The processing is quite straightforward -- the data goes directly into the tree without checkpointing or indexing, which speeds the processing considerably. But remember, if a system problem occurs while the records are being added, you will need to zap the record-type and start again.
Here is a sample session demonstrating the FAST LOAD command in SPIBILD. The file is explicitly processed before setting NOPASS in case there is data already in the deferred queue that should be processed.
-> spibild -Welcome to SPIBILD -? process students -Passing: 3 records of record-type REC01 -Processed: 3 DEFQ records of record-type REC01 -Compute time: 0.146 seconds -Elapsed time: 1.912 seconds -Core usage: 0065/0939 -? set nopass -? fast load tables into rec06 of *students - ADD 1, 1 @Line 5. Key = ADDISON-100 - ADD 2, 2 @Line 10. Key = ADDISON-101 - ADD 3, 3 @Line 15. Key = ADDISON-102 ... (etc.) - ADD 207, 207 @Line 1035. Key = WESLEY-022 - Requests/Success: ADD 207 207 SUM 207 207 -End of BATCH/BUILD -Compute time: 2.151 seconds -Elapsed time: 12.789 seconds -Core usage: 0257/0939 -?
The FIX BASE command lets you reset the base-block data for the record-types of a data set in the MSTR data set of the currently attached file.
Command: FIX BASE
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: REBALANCE DATA SET, FIX REBALANCED DATA SET
Recovery commands: none
Procedures: REBALANCE procedure (recovery) (3.3.2)
FIX BASE OF RECn
This command, primarily for SPIRES consultants, is for use when the base block information stored in the MSTR data set of the attached file does not match the actual base block data in the "RECn" data set. This rare problem most commonly arises when a STATUS command indicates errors in a just-rebalanced RECn data set -- you must return to the old version of the RECn data set and reset the base-block data with the old data. (That's why the procedure includes saving the old base-block data shown in the SHOW FILE ACTIVITY command before rebalancing the data set.)
The FIX BASE command shows you what the MSTR data set currently has for the base block number of each record-type in the data set, and asks you for the "new" number. When the base blocks of all the record-types have been entered, SPIRES replaces the old values in the MSTR with the new ones. In other words, unless the command runs to its conclusion, the base-block data won't be changed in the MSTR. [See 3.3.2.]
Here is an sample session showing the FIX BASE command:
-> attach gi.tar.strings -> fix base of rec1 -REC01 base is 0; type new base: 0 -REC02 base is 1; type new base: 23 -REC03 base is 2; type new base: 26 -File detached ->
If you don't know the correct base block numbers, it is possible to track them down by using the SHOW FILE BLOCK command to find a block from each record-type in the data set and trace each one back to its base block.
The FIX NEXT BLOCK command resets the number representing the "next block" in a tree data set or the residual data set.
Command: FIX NEXT BLOCK
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: SHOW FILE ACTIVITY, FIX REBALANCED DATA SET
Recovery commands: none
Procedures using it: REBALANCE procedure (recovery) (3.3.2)
FIX NEXT BLOCK OF {RECn|RESIDUAL}
This command, primarily for SPIRES consultants, is useful to data base administrators when problems arise during a data-set rebalance. [See 3.3.2.] Like other FIX commands, it can cause serious problems if not used carefully; if you believe you need to use it for reasons other than the rebalancing situation described earlier in this manual, please check with your SPIRES consultant before using it.
Information about what the proper setting should be for a given data set is usually provided by the SHOW FILE ACTIVITY command.
Here is a sample session showing what the command looks like when used:
-> attach sl.imy.insects -> fix next block of rec4 - Next block is 730; type new block: 726 ->
The FIX MAXIMUM BLOCKS command resets the maximum number of blocks that may be utilized in an extended tree data set.
Command: FIX MAXIMUM BLOCKS
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: SHOW FILE LIMITS
Recovery commands: none
Procedures using it: REBALANCE DATA SET procedure
FIX MAXIMUM BLOCKS OF DATA SET RECn
This command, primarily for SPIRES consultants, enables administrators to extend the maximum size of a tree data set which is approaching its default limit. When you issue this command SPIRES will prompt you for the new maximum block size value as shown in the example below. This command can only be used for extended tree data sets which have a default size limit of 434,280 blocks. If the data set is not an extended tree data set and is approaching its default size limit of 65,536 blocks then you should ignore this command and instead convert the data set to extended tree format. [See 5.6.] The maximum size value is restricted. The maximum size value cannot be set to more than three times the default maximum and cannot be set to less than the default maximum (greater than 434,280 and less than 1,302,840).
If you rebalance an extended tree data set the rebalanced data set will retain the max size of the original. [See 6.1.9.]
Information about the current size of a file's data sets is provided by the SHOW FILE LIMITS command. [See 6.3.4a.]
Here is a sample session showing what the command looks like when used:
> attach *mytest
> show file limits
09/17/2003 Limits Record of File GQ.UUU.MYTEST
Last Compiled: 05/30/2000 at 10:02:00
Data Set Max Blocks - Blocks Used - Percent Used Tree Depth
REC1 65536 119 0.18 3
REC2 434280 84 0.01 3
RES 425984 161 0.03 --
MSTR 256 3 1.17 --
DEFQ 434280 2 0.00
File data set block size is 2048 bytes.
> fix maximum blocks of data set rec2
Current dataset maximum size is 434280; type new value: 512000
> show file limits
09/17/2003 Limits Record of File GQ.UUU.MYTEST
Last Compiled: 05/30/2000 at 10:02:00
Data Set Max Blocks - Blocks Used - Percent Used Tree Depth
REC1 65536 119 0.18 3
REC2 512160 84 0.01 3
RES 425984 161 0.03 --
MSTR 256 3 1.17 --
DEFQ 434280 2 0.00
File data set block size is 2048 bytes.
The FIX REBALANCED DATA SET command registers the new base-block information from a rebalanced data set in the MSTR data set of the currently attached file.
Command: FIX REBALANCED DATA SET
Available in: SPIRES
Available to: file owner
Related commands: REBALANCE DATA SET, FIX BASE
Recovery commands: FIX REBALANCED DATA SET
Procedures: REBALANCE procedure (recovery) (3.3.2)
Create or Modify Extended Trees (5.6)
[IN ACTIVE [CLEAR|CONTINUE]] FIX REBALANCED DATA SET {RECn|DEFQ}
[RENAME] [CONVERT]
The FIX REBALANCED DATA SET command is designed for the recovery procedure if the REBALANCE DATA SET command with the RENAME option does not complete successfully.
The procedure describing its use appears earlier in this manual. [See 3.3.2.]
Below is an example of the FIX REBALANCED DATA SET command. The description of the REBALANCE procedure will give you a better sense of the context in which it is used.
-> fix rebalanced data set rec1, rename
FIX rebalanced data set REC1
-REC01 base is 0; old base was 0
- Next block is 1492; Next block was 1821
ORV.GQ.JNK.DRG.REC1 is renamed to ORV.GQ.JNK.DRG.REO1.19870107
ORV.GQ.JNK.DRG.REB1 is renamed to ORV.GQ.JNK.DRG.REC1
-Set new data set permits if needed.
-File detached
->
The FIX SUBFILE NODE command adds the node to the tree for a given record in the residual that has been orphaned, or adds a node to a tree to serve as a place-holder for future monotonic tree growth.
Command: FIX SUBFILE NODE
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: ZAP SUBFILE NODE, SHOW SUBFILE BLOCK
Recovery commands: ZAP SUBFILE NODE
FIX SUBFILE NODE {LCTR block-trailer|SEED key}
block -- The number of the residual block containing
the orphaned record, expressed as an integer
- -- A hyphen
trailer -- The position of the record within the
block, expressed as an integer.
WARNING: These commands are only for use to solve file integrity problems -- careless use can cause more. Seeing your SPIRES consultant before using them is recommended.
The LCTR version of the FIX SUBFILE NODE command is meant for use in situations where a record in the residual somehow loses the node in the tree that points to it. It is the reverse of the ZAP SUBFILE NODE command, which will delete the tree node of a record, though it does not remove the record from the residual.
The "block-trailer" numbers may be available to you in several ways:
- in any locator for the record that might appear in an index record. For most files, the
first 3 bytes of the locator (in hex) specify the block number, and the last byte specifies
the trailer. For example, if the locator is shown as "00002803", then the block
number is 40 ("28" hex) and the trailer is 3 ("3" hex).
- in the information returned from the ZAP SUBFILE NODE command will be the block number and the record key. Using the SHOW SUBFILE RESIDUAL BLOCK command, you can find the trailer number for that key in the specified block.
In the example below, you need to restore a tree node for record 138, which you tracked down to residual block 40:
-> display 138 -No such record, code=S256 -> show subfile residual block 40 Block 40 of Residual Dated: 01/24/93 No. Disp Len Auxiliary data 1 16 471 Rec 1 Key: 136 2 487 350 Rec 1 Key: 137 3 837 369 Rec 1 Key: 138 4 1206 218 Rec 1 Key: 139 5 1424 392 Rec 1 Key: 140 6 1816 204 Avsp -> fix subfile node lctr 40-3 -> display 138 ... record 138 is displayed...
Monotonic record-types, that is, record-types where new records always have keys higher in value than records already in the file, tend to need rebalancing more frequently than normal record-types. The FIX SUBFILE NODE SEED command assigns a specific key not yet in the record-type to serve as a starting point for a future subtree. Aside from its use as a marker, the new node that SPIRES creates in the tree's base block has no effect. If the key values are chosen properly, you may never need to rebalance the tree again. (Remember: talk with your SPIRES consultant before using this command.)
The command should be issued after the record-type has been rebalanced. [See 3.3.1.] Select the subfile for which the record-type is the goal, or else ATTACH the record-type. One or more FIX SUBFILE NODE SEED commands may be issued, each one assigning a key higher than any existing in the file:
-> attach xrec25 of .trecordings -> fix subfile node seed 2500000 -> fix subfile node seed 2600000 ...
You might then issue the SHOW SUBFILE BLOCK BASE command to see the effect of your change. Near the bottom of the displayed list of nodes in the base block, Look for the "asterisked" key values you have entered.
This method can be used instead of the START-KEY method, which requires changing the file definition.
The FIX SUBFILE SLOT command resets the slot number of the current goal record-type.
Command: FIX SUBFILE SLOT
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: SELECT
Recovery commands: FIX SUBFILE SLOT
Procedures: Resetting the Next Slot Number of a
Record-Type (5.3)
FIX SUBFILE SLOT
FIX SUBFILE SLOT allows you to reset the next slot number of the selected subfile to:
- a higher number, which takes effect permanently and immediately for all users of the subfile; or
- a lower number, which is in effect only for you and only as long as the subfile is selected by you.
Warning: Setting the number lower is quite dangerous, since it could allow you to overlay a new record onto an older one that already exists, causing serious problems for your file. [See 5.3.]
When you issue the command, SPIRES will show you the next slot number and then ask you for the new value. Type in the new number, or press the ATTN/BREAK key to cancel the command if you have changed your mind.
Below is an example of the FIX SUBFILE SLOT command:
-> select albums -> fix subfile slot -Next slot is 328; Type new slot: 1001 ->
The FOR STORED command establishes a Global FOR environment whose class of records is a stored stack or result.
Command: FOR STORED
Available in: SPIBILD
Available to: file owner
users with Process access to the file
Related commands: ESTABLISH, FOR SUBFILE, MERGE, REMOVE, BUILD,
SET SCAN, SET FILTER
Recovery commands: none
Procedures MERGE procedure in SPIBILD in Global FOR (2.4.2)
REMOVE procedure in SPIBILD in Global FOR (2.5.1)
Add an index (2.7.1)
Rebuild an index (2.7.3)
FOR STORED {RESULT.name|STACK.name} [WHERE clause]
In SPIBILD, the REMOVE, MERGE and BUILD commands can be issued only under Global FOR, meaning that they must be preceded by the ESTABLISH command and a FOR SUBFILE or FOR STORED command. The FOR STORED command tells SPIBILD that the records to be examined and, if they pass any WHERE clause criteria, processed are in a stored stack or result, which is named in the FOR STORED command.
Creating a stored result or stack is simply a matter of creating a result or stack in SPIRES (e.g., using a FIND command) and then issuing a "STORE SEARCH.name" or "STORE STACK.name" command.
Not all stored results can be used by the FOR STORED command. Specifically, only results stored from non-structured pointer-groups are allowed; if the list of indexes for the subfile shows qualifiers and sub-indexes, then the subfile has structured pointer-groups.
Because of this limitation, if you don't know whether or not the pointer-groups are structured, or if you don't care to find out either ahead of time or the hard way that they are, you may want to simply create a stored stack rather than a stored result, which will avoid the problem. To do that, in SPIRES, restore or recreate the result, and then issue the two commands FOR RESULT and STACK ALL. Then store the stack.
Under FOR STORED processing in SPIBILD, any consecutive duplicate keys will be ignored (i.e., the record will be processed only once). However, if two copies of the same key are separated by others, that record will be processed twice. That won't be a problem if you are using the REMOVE command, but could be if you are using MERGE or BUILD. If you have a stored stack, it is a good idea to sort it (with the SEQUENCE command) to eliminate duplicate keys. (Stored results are already sorted.)
Here is part of a session showing FOR STORED with the REMOVE command, starting with the creation of a stored stack in SPIRES:
-> select restaurants -> find date-updated before 1980 -Result: 134 RESTAURANTS -> sequence -Stack: 134 RESTAURANTS -> store stack.restaurants.to.remove -'STACK.RESTAURANTS.TO.REMOVE' put in ORVYL file system -> spibild -Welcome to SPIBILD -> establish restaurants -Processing file: GQ.DOC.RESTAURANT -Compute time: 0.029 seconds -Elapsed time: 2.956 seconds -Core usage: 0150/1008 -> for stored stack.restaurants.to.remove -> remove all - REM 1, 1 @Line 0. Key = 96 - REM 2, 2 @Line 0. Key = 99 ... - REM 134, 134 @Line 0. Key = 376 - Requests/Success: REM 134 134 SUM 134 134 -End of BATCH/BUILD -Begin passing of 134 records of record-type ID -Completed passing of 134 records total -Processed: 134 DEFQ records of record-type ID -Detaching File: GQ.DOC.RESTAURANT ... ->
The GENERATE CHANGE FILE command creates a SPIRES file containing just the defq and tree data of record updates in the attached file.
Command: GENERATE CHANGE FILE
Available in: SPIRES
Available to: file owner
users with both Copy access to the file and
ORVYL write permit to the owner account
Recovery commands: not applicable
GENERATE CHANGE FILE [filename]
The PROCESS cycle number exists for SPIRES internal use independently of the GENERATE CHANGE FILE command and cannot be directly controlled by the file owner or user of this command. The advantage of creating your own file name is that you can create your own numbering/naming scheme rather than work from an internal SPIRES number outside of your control. For instance, if your file is named GQ.DOC.INVENTORY, you might want to name the change file "INVENTORY.yyyymmdd" putting the current date in place of "yyyymmdd" each time you create a change file.
GENERATE CHANGE FILE creates a snapshot of all the changes in the selected subfile's deferred queue. Specifically, all the goal-record data in the deferred queue for the selected goal record-type is copied, along with the tree copies of any records marked as updates or removes. The copied data is placed into a new SPIRES "file", a partial copy of the original file's structure, including the compiled characteristics of the goal record-type, a "tree" data set, a residual data set, and a deferred queue. The copied deferred queue data is placed in the defq; the copied tree data goes into the tree and the residual. The file is always created on the file owner's account and cannot be directed elsewhere, even if it is being generated by someone else who has Copy access to the file.
Note: the created file does not have the same ORVYL data set permits that the original has (it has the default permits that all new files start with), and it does not have any of the other record-types of the file that might be related to the goal record-type as indexes or for subgoal lookups.
The purpose of the command is to generate all the data needed to capture all modifications to a subfile prior to file processing. Generally speaking, this command would be coupled to the file-processing procedure, with updates blocked for the duration of the copying and the file processing. This could allow other programs to retrieve the updated data when they wanted to, perhaps to generate the changes in a different manner to apply to other data bases that have the same data. That processing could be done anytime, rather than right before the original file was being processed, which might tie up the original file's processing schedule.
By default, outside of Global FOR, all changed records in the subfile will be processed. If you issue the GENERATE CHANGE FILE command under Global FOR, however, only the records in that class will be processed. For instance, you could generate a change subfile of only the added records by issuing the commands FOR ADDS and GENERATE CHANGE FILE. Some Global FOR classes (SUBFILE, RESULT, STACK for instance) make more sense in this context than others (TREE, REMOVES).
One important exception to normal Global FOR processing worth mentioning here, however, is that GENERATE CHANGE FILE under FOR DEFQ processing will include copying the tree versions of records that are marked as removes in the defq. Normally under FOR DEFQ, removed records are not processed.
Here is an example of GENERATE CHANGE FILE in general use outside of Global FOR, so that all defq data will be processed:
-> select inventory
-> generate change file
-Begin Record Build of File GQ.DOC.INVENTORY.G0134
- Requests/Success:
REM 3 3
ADD 1 1
UPD 1 1
SUM 5 5
-> attach inventory.g0134
-> for subfile
+> skip last
+> show levels
Processed Examined
FOR SUBFILE 2 2
+>
The generated change file can thus be attached and used like a regular SPIRES file. Note that the data from five transactions was copied, but three of them were removes (the three removed records are in the tree of the change file), so only two records, the add and the update, show up under FOR SUBFILE.
If you used the SELECT command to select the original subfile, you can also use it to select the change file:
SELECT :nn subfile-name
or
SELECT &gg.uuu.filename :nn subfile-name
where "nn" is the PROCESS cycle number that was assigned to the generated change file when it was created (see above).
You could select the generated change subfile of the example above with the command:
-> select :34 inventory ->
The same subfile privileges that you have for the original subfile are given to you for the change file when you select it this way. Similarly, anyone else who can select the original subfile can use this method to select the change file.
Because other record-types of the file are not created, you will receive the warning message "Invalid subgoal in selected subfile" if the goal record-type has A32, same-file subgoal access. The subfile is still selected, however.
The deferred queue data for a given record in the generated change subfile is only the latest copy of the record from the original subfile. Unlike the defq of the original subfile, which retains all copies of a record if it was changed in multiple transactions since the file was last processed, the change subfile's defq has only a single copy of the record. In other words, the purpose of the change subfile is to display the changes that will be applied to the tree of the original subfile when it is next processed; it is not a copy of the transaction log of the original. You can see this point demonstrated by using the SHOW SUBFILE TRANSACTIONS command on both the original and the generated change subfiles.
The GENERATE FILE command creates a SPIRES file containing copies of the designated records in the selected subfile.
Command: GENERATE FILE
Available in: SPIRES
Available to: file owner
users with both Copy access to the file and
ORVYL write permit to the owner account
Recovery commands: not applicable
GENERATE FILE filename [APPEND]
GENERATE FILE creates a copy of the selected subfile's records (as selected by the Global FOR set that must be established before issuing this command), placing them in a new SPIRES file. The file could be used to take a snapshot of a particular set of SPIRES records, leaving them in SPIRES format where they can be examined as SPIRES records (e.g., processed through formats, used with referenced record processing, etc.).
The copied data is placed into a new SPIRES "file", a partial copy of the original file's structure, including the compiled characteristics of the goal record-type, a "tree" data set, a residual data set, and a deferred queue. The copied data is placed into the tree. The file is always created on the file owner's account and cannot be directed elsewhere, even if it is being generated by someone else who has Copy access to the file.
Note: the created file does not have the same ORVYL data set permits that the original has (it has the default permits that all new files start with), and it does not have any of the other record-types of the file that might be related to the goal record-type as indexes or for subgoal lookups.
Important: When generating the file, choose a Global FOR class that will process the records in ascending key order, such as FOR SUBFILE or FOR TREE. If you use FOR RESULT, FOR STACK or FOR SET, make certain that the records are sequenced in key order before processing them. SPIRES uses FASTBILD file services internally to build the file copy.
-> select inventory
-> for subfile where inv.date = 1990
-> generate file inventory.year1990
-Begin Record Build of File GQ.DOC.INVENTORY.YEAR1990
- Requests/Success:
ADD 33 33
SUM 33 33
-> attach inventory.year1990
-> for tree
+> skip last
+> show levels
Processed Examined
FOR TREE 33 33
+>
The generated file can thus be attached and used like a regular SPIRES file.
The GENERATE LOAD command creates an ORVYL data set containing SPIRES records in their internal form, ready for loading directly into another record-type.
Command: GENERATE LOAD
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: "FOR class" commands, LOAD, INPUT LOAD
Recovery commands: GENERATE LOAD
Procedures: LOAD Procedure in SPIBILD (2.3.3)
INPUT LOAD Procedure (2.3.3a)
LOAD Procedure in FASTBILD (2.12.2)
GENERATE LOAD loadname [REPLACE] [FIRST|*|NEXT|n|REST|LAST|ALL] ... [END = 'command']
FIRST = the first record fitting the criteria
* = the current record
NEXT = the next record fitting the criteria
n = the next "n" records fitting the criteria
REST = the rest of the records fitting the criteria
LAST = the last record fitting the criteria
ALL = all the records fitting the criteria.
GENERATE LOAD, a Global FOR command in SPIRES, creates an ORVYL data set containing SPIRES records in their internal, stored form. Thus, the element values in the records are not processed through their Outproc processing rules. The command's purpose is to create a data set that can later be loaded into the same or a similar record-type, using the LOAD or INPUT LOAD commands.
The data set must be created with a single GENERATE LOAD command; you cannot "build" it using several GENERATE LOAD commands.
See the manual "Sequential Record Processing in SPIRES: Global FOR" for more information about Global FOR in general.
-> select inventory -> for subfile -> generate load records.for.test 100 -Replace? ok ->
SPIRES replaced the existing RECORDS.FOR.TEST data set in ORVYL with the generated load, consisting of the first 100 records in the subfile. The data set name suggests that the records are being moved from a production file into a test one, a common use of the LOAD and INPUT LOAD procedures.
The GENERATE NONIXR command creates a residual data set of pre-allocated size to save processing costs during file-building operations.
Command: GENERATE NONIXR
Available in: SPIRES
Available to: file owner
Related commands: FAST BUILD, FAST BATCH
Recovery commands: none
Procedures: FAST BUILD procedure in FASTBILD (2.12.2)
FAST BATCH procedure in SPIBILD (2.3.4)
GENERATE NONIXR OF size FOR filename
The GENERATE NONIXR command converts the residual data set of the named file from the default "IXR" type to "non-IXR". "IXR" stands for Index Record, specifically, a block at the beginning of an ORVYL data set containing the address of every block in the data set. An IXR data set contains blocks that may be scattered all over the ORVYL file system. A non-IXR data set does not need an IXR because all of its blocks are contiguous.
When you issue the GENERATE NONIXR command, SPIRES allocates a non-IXR data set of the size you specify in the "size" parameter. Then it copies the blocks of the file's residual into the non-IXR data set and then erases the old residual.
Non-IXR residuals are usually created before a file is built in SPIBILD or FASTBILD. It must be done after the file is compiled, however, since it converts an existing residual data set from IXR to non-IXR.
File owners most often use non-IXR residuals for very large files. The file owner determines (by means of smaller test files) how large the residual will be when it is built. When the build is complete, the file owner creates a second residual that is not a non-IXR file to accommodate future growth.
Since the size of the non-IXR residual is fixed, it can neither grow dynamically to accommodate more blocks, nor release blocks it does not need. If necessary, you can issue another GENERATE NONIXR command to create a larger non-IXR residual, but remember that the command must completely recopy the blocks in the old residual into the larger one, which could be very expensive for a large residual.
-> spires -Welcome to SPIRES -> generate nonixr of 5000 for common.tables ->
The returned command prompt indicates the command succeeded. Because GENERATE NONIXR copies your entire residual from one set of blocks to another, it can take a long time to execute.
If the command fails, the two most likely reasons are:
- S70 error = There were not enough contiguous blocks in the ORVYL file system to handle your request.
- Requested block size not valid = The number of blocks requested was too large (over 425,984) or is smaller than the current size of the residual. You cannot use GENERATE NONIXR to make the residual smaller.
You can use the GENERATE NONIXR command to change any ORVYL data set to non-IXR if you want, because the command does not really care whether the data set is actually a residual.
First, use ORVYL's RENAME command to change the name of the data set so that it ends with ".RES". (That way, SPIRES will think it is a residual data set.) Then issue the GENERATE NONIXR command, using the same name but omitting the ".RES", and finally RENAME the data set again:
-> rename broken.records to broken.records.res BROKEN.RECORDS.RES created in ORVYL File System -> generate nonixr of 100 for broken.records -> rename broken.records.res to broken.records BROKEN.RECORDS created in ORVYL File System -> show files like broken.records, blocks BROKEN.RECORDS -- 100 BLKS (NON-IXR) ->
The ORVYL command SHOW FILES (with the BLOCKS or ALL option) confirms that the change was made, stating that the data set is non-IXR.
If your file needs multiple RESiduals, be sure to include the "RESIDUAL = n;" statement in your file definition, and then RE/COMPILE that definition to create the extra RESidual data sets. If you already have the RESIDUAL statement, but need more space, then raise the value of "n" and RECOMPILE to add more RESiduals.
Any data sets that are "No content" should be secondary RESiduals. You can convert them to NONIXR type only once, as follows:
-> clear select
-> SET CLP file.RES <-- Use your "file" name
-> RENAME file.RES TO file.RESK <-- Save aside the
primary RESidual.
-> RENAME file.RESn TO file.RES <-- n >= 2
-> GENERATE NONIXR OF 425984 FOR file <-- Must be this size.
-> RENAME file.RES TO file.RESn <-- n >= 2
Repeat the RENAME, GENERATE, RENAME for as many secondaries as needed. Finally,
-> RENAME file.RESK TO file.RES -> SET NOCLP file.RES
It is strongly recommended you leave the last RESidual as IXR type. For example, if you know you'll need at least RES and RES2, then make "RESIDUAL = 3;" to create an empty RES3 for future expansion.
The primary RESidual (filename.RES) may be converted to NONIXR type of any size larger than its current size as long as there are no secondary RESiduals defined (no RESIDUAL statement). You may continue to expand this single RESidual to larger and larger NONIXR size with subsequent applications of the GENERATE NONIXR command, but only as long as there are no secondaries.
Once you define secondary RESiduals, the primary may be converted from IXR to NONIXR type only once, and the size must be 425984. If the primary RESidual is already NONIXR type, it must be expanded to maximum size (425984) only once if it isn't already that size.
The INPUT ADD command adds multiple new records to a subfile via a custom input format.
Command: INPUT ADD
Available in: SPIRES, SPIBILD
Available to: (in SPIRES) users who can update the subfile
(in SPIBILD) file owner, and
users with Process access to the file
Related commands: ESTABLISH, INPUT BATCH, SET EXCEPTION,
SET FORMAT, SET MESSAGES
Recovery commands: UNQUEUE, ZAP DEFQ, SHOW RECORD RECOVERY, SET SKIP,
INPUT BATCH, REMOVE, ZAP RECORD-TYPE, ZAP FILE
Procedures: INPUT ADD procedure in SPIBILD (2.3.2)
For online SPIBILD and in BATWYL jobs: [USING frame] INPUT ADD For BATSPI and batch SPIBILD jobs: [FROM ddname] [USING frame] INPUT ADD
You can use the INPUT ADD command in SPIRES or SPIBILD to add multiple records to a subfile using a customized input format (not the standard SPIRES format; use the INPUT BATCH command in that case).
In SPIRES, the INPUT ADD command follows a SELECT command to choose the subfile into which the data will be batched, and a SET FORMAT command naming the input format to be used. In SPIBILD, it follows a SET FORMAT command, which follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT ADD command to work properly.
When you issue the INPUT ADD command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT ADD command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]
In SPIRES, the new records end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the new records are added directly to the tree, and all indexes are updated.
Here is a sample session demonstrating the INPUT ADD command in SPIBILD:
-? establish room tables -Processing file: GQ.DOC.STUDENTS -Compute time: 0.246 seconds -Elapsed time: 1.616 seconds -Core usage: 0065/0939 -? set format mult.adds -? input add - ADD 1, 1 @Line 5. Key = B05 - ADD 2, 2 @Line 10. Key = B06 - ADD 3, 3 @Line 15. Key = B07 ... (etc.) - ADD 96, 96 @Line 480. Key = 232 -Passing: 96 records of record-type REC01 - Requests/Success: ADD 96 96 SUM 96 96 -End of BATCH/BUILD -Compute time: 3.446 seconds -Elapsed time: 19.478 seconds -Core usage: 0275/0939 -?
The INPUT ADDMERGE command adds multiple new records and/or merges data into existing records of a subfile via a custom input format.
Command: INPUT ADDMERGE
Available in: SPIRES, SPIBILD
Available to: (in SPIRES) users who can update the subfile
(in SPIBILD) file owner, and
users with Process access to the file
Related commands: ESTABLISH, INPUT ADD, INPUT MERGE,
SET EXCEPTION, SET FORMAT, SET MESSAGES
Recovery commands: UNQUEUE, ZAP DEFQ, SHOW RECORD RECOVERY, SET SKIP,
INPUT BATCH, REMOVE, ZAP RECORD-TYPE, ZAP FILE
Procedures: INPUT ADDMERGE procedure in SPIBILD (2.3.2)
For online SPIBILD and in BATWYL jobs: [USING frame] INPUT ADDMERGE For BATSPI and batch SPIBILD jobs: [FROM ddname] [USING frame] INPUT ADDMERGE
If desired, you can abbreviate the command to INP ADDMER.
When you use the INPUT ADDMERGE command in SPIRES or SPIBILD, the program first treats the data for each record as if it were a new record -- in the input format, it executes frames of usage FULL, not MERGE. After the format processing for the record is done, SPIRES checks to see whether the key was provided in the input and if so, whether it matches the key of an existing record. If it does match, the input data is treated as data to be merged into that record; if it does not, the data is treated as a new record.
The data is always processed via a custom input format. ADDMERGE is a command allowed in the standard SPIRES format, used with the INPUT BATCH command and procedure. [See 2.3.1, 2.3.5.] For information about formats for INPUT ADDMERGE, see the manual "SPIRES Formats"; online, EXPLAIN FORMATS, FOR ADDMERGE COMMAND.
In SPIRES, the INPUT ADDMERGE command follows a SELECT command to choose the subfile into which the data will be batched, and a SET FORMAT command naming the input format to be used. In SPIBILD, it follows a SET FORMAT command,which follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT ADDMERGE command to work properly.
When you issue the INPUT ADDMERGE command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT ADDMERGE command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]
In SPIRES, the new and updated records end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the new data goes into the tree, and all indexes are updated.
Here is a sample session demonstrating the INPUT ADDMERGE command in SPIBILD:
-? establish course tables -Processing file: GQ.DOC.STUDENTS -Compute time: 0.492 seconds -Elapsed time: 2.322 seconds -Core usage: 0065/0939 -? set format mult.addmerge -? input addmerge - ADD 1, 1 @Line 4. Key = ART101 - ADD 2, 2 @Line 8. Key = ART102 - MER 3, 3 @Line 12. Key = ART201 ... (etc.) - MER 125, 125 @Line 500. Key = WRI402 -Passing: 125 records of record-type REC01 - Requests/Success: ADD 86 86 MER 39 39 SUM 125 125 -End of BATCH/BUILD -Compute time: 6.311 seconds -Elapsed time: 22.450 seconds -Core usage: 0311/0939 -?
The INPUT ADDUPDATE command adds multiple new records and/or replaces existing records of a subfile via a custom input format.
Command: INPUT ADDUPDATE
Available in: SPIRES, SPIBILD
Available to: (in SPIRES) users who can update the subfile
(in SPIBILD) file owner, and
users with Process access to the file
Related commands: ESTABLISH, INPUT ADD, INPUT ADDMERGE,
SET EXCEPTION, SET FORMAT, SET MESSAGES
Recovery commands: UNQUEUE, ZAP DEFQ, SHOW RECORD RECOVERY, SET SKIP,
INPUT ADD, REMOVE, ZAP RECORD-TYPE, ZAP FILE
Procedures: INPUT ADDUPDATE procedure in SPIBILD (2.3.2)
For online SPIBILD and in BATWYL jobs: [USING frame] INPUT ADDUPDATE For BATSPI and batch SPIBILD jobs: [FROM ddname] [USING frame] INPUT ADDUPDATE
The command can be abbreviated to INP ADDUPD if desired.
When you use the INPUT ADDUPDATE command in SPIRES or SPIBILD, the program first treats the data for each record as if it were a new record -- in the input format, it executes frames of usage FULL. After the format processing for the record is done, SPIRES checks to see whether the key was provided in the input and if so, whether it matches the key of an existing record. If it does match, the input data is used to replace that record; if it does not, the data is treated as a new record.
The data is always processed via a custom input format. ADDUPDATE is a command allowed in the standard SPIRES format, used with the INPUT BATCH command and procedure. [See 2.3.1, 2.3.5.] For information about formats for INPUT ADDUPDATE, see the manual "SPIRES Formats"; online, EXPLAIN FORMATS, FOR ADDUPDATE COMMAND.
In SPIRES, the INPUT ADDUPDATE command follows a SELECT command to choose the subfile into which the data will be batched, and a SET FORMAT command naming the input format to be used. In SPIBILD, it follows a SET FORMAT command, which follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT ADDUPDATE command to work properly.
When you issue the INPUT ADDUPDATE command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT ADDUPDATE command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]
In SPIRES, the new and updated records end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the new data goes into the tree, and all indexes are updated.
Here is a sample session demonstrating the INPUT ADDUPDATE command in SPIBILD:
-? establish advisor tables -Processing file: GQ.DOC.STUDENTS -Compute time: 0.294 seconds -Elapsed time: 3.233 seconds -Core usage: 0065/0939 -? set format mult.addupd -? input addupdate - UPD 1, 1 @Line 6. Key = 1 - UPD 2, 2 @Line 12. Key = 2 - UPD 3, 3 @Line 18. Key = 3 ... (etc.) - ADD 17, 17 @Line 102. Key = 17 -Passing: 17 records of record-type REC01 - Requests/Success: ADD 15 15 MER 2 2 SUM 17 17 -End of BATCH/BUILD -Compute time: 1.136 seconds -Elapsed time: 5.422 seconds -Core usage: 0311/0939 -?
The INPUT BATCH command batches data (adds, updates, merges, or removes) into a subfile.
Command: INPUT BATCH
Available in: SPIRES, SPIBILD
Available to: (in SPIRES) users who can update the subfile
(in SPIBILD) file owner, and
users with Process access to the file
Related commands: ESTABLISH, INPUT ADD, SET EXCEPTION, SET MESSAGES
Recovery commands: UNQUEUE, ZAP DEFQ, SHOW RECORD RECOVERY, SET SKIP,
INPUT BATCH, REMOVE, ZAP RECORD-TYPE, ZAP FILE
Procedures: INPUT BATCH procedure in SPIBILD (2.3.1)
For online SPIBILD and in BATWYL jobs that call SPIBILD: [USING frame] INPUT BATCH For batch SPIBILD jobs: [FROM ddname] [USING frame] INPUT BATCH For SPIRES: [WITH FAST|WITH SEARCHONLY] [USING frame] INPUT BATCH [USING line-range] For BATSPI jobs: [WITH FAST|WITH SEARCHONLY] [USING frame] [FROM ddname] INPUT BATCH
You can use the INPUT BATCH command in SPIRES or SPIBILD to add, update, merge or remove multiple records in a single "batch" using the standard SPIRES format. [See 2.3.5.] Note: If an input format is set, INPUT BATCH behaves like an INPUT ADD command, i.e., all input is treated as records to be added to the file, using the input format.
In SPIRES, the INPUT BATCH command follows a SELECT command to choose the subfile into which the data will be batched. In SPIBILD, it follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT BATCH command to work properly.
When you issue the INPUT BATCH command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT BATCH command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]
In SPIRES, the transactions end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the transactions are applied to the tree, and all indexes are updated.
Here is a sample session demonstrating the INPUT BATCH command in SPIBILD:
-? establish teacher tables -Processing file: GQ.DOC.STUDENTS -Compute time: 0.123 seconds -Elapsed time: 1.308 seconds -Core usage: 0065/0939 -? input batch - ADD 1, 1 @Line 7. Key = 2038 - ADD 2, 2 @Line 14. Key = 2039 - MER 3, 3 @Line 21. Key = 1182 ... (etc.) - REM 146, 146 @Line 835. Key = 1022 -Passing: 146 records of record-type REC01 - Requests/Success: REM 13 13 ADD 109 109 MER 24 24 SUM 146 146 -End of BATCH/BUILD -Compute time: 4.364 seconds -Elapsed time: 14.987 seconds -Core usage: 0257/0939 -?
The WITH FAST option, available on the INPUT BATCH command in SPIRES, puts the records into the deferred queue three to four times faster, in terms of processing costs, than the normal INPUT BATCH command. You can use it under these conditions:
- You must have Master access to the file.
- The deferred queue must be empty when you issue your WITH FAST request.
- The keys of the records being added or updated must be in ascending sequence, unless a key matches the key of a record added or updated earlier in the process. Note that the maximum speed will not be attained unless all records are in ascending sequence. (If you are only adding records, and the goal record-type has a slot key, and SPIRES will be assigning the keys, then this rule is not relevant.)
- The file must have a CKPT data set of its own. (If the file does not, then adding either a CHECKPOINT or IMMEDIATE statement to the file definition and recompiling will create one. See the manual "SPIRES File Definition" for information about those statements.)
- You are willing to forego the updating of immediate indexes as part of the input process; if the subfile has immediate indexes, you will have to process the file in SPIBILD to get the indexing done, and to allow other users to use the subfile for updates again.
During the processing, the file will be placed in SET SEARCH mode, so that no one else may update the file. Other users attempting to update with file will get S494 errors. [See 6.2.8, 2.1.5.1.]
If the subfile does not have any immediate indexes, then the subfile will be ready for all to use again upon completion of the command. That is, other users will again be able to update the subfile.
However, if the subfile has any immediate indexes, then SPIRES blocks deferred-queue updates when the INPUT BATCH command completes. (Users will get S560 errors if they try; you can test for this condition with the $DEFQTEST function, using the WRITE type.) This is because no index updates (immediate indexes or not) are done by INPUT BATCH with the WITH FAST option. The file must be processed or the DEFQ zapped before any further update requests will be honored.
The WITH FAST option, when used on Multi-record formatted output causes the format processor to retain $RECNO from any prior formatted output command. This allows you to output in bursts, and $RECNO steadly increases. For example, in a protocol you might have:
scratch * set format to.do.output while $true in active continue with fast using frame display 5 end leave wdse ====== separator line ======= endw
Setting the format automatically clears $RECNO. The WITH FAST option retains $RECNO, and DISPLAY increments it for each record displayed. So you get records 1-5, a separator line, records 6-10, a separator line, etc.
The WITH SEARCHONLY prefix for INPUT BATCH in SPIRES also speeds up the input processing, though not as much as WITH FAST. However, it does not have the same requirements and restrictions. Basically, during the processing, SPIRES will reserve the file's CKPT (checkpoint) data set, so that it is not reserved and released for each update as normal INPUT BATCH processing is done. (That's why other users are allowed to do updates during regular INPUT BATCH processing.) Like WITH FAST, WITH SEARCHONLY puts the file into SET SEARCH mode, so that other users of the file are limited to searching and reporting. [See 6.2.8, 2.1.5.1.]
Any immediate indexing is done as normal, and the file is available for normal updating activity upon completion of the WITH SEARCHONLY INPUT BATCH command.
WITH SEARCHONLY, like WITH FAST, is available only to users with Master access to the file.
The INPUT LOAD command adds records to a subfile using data already in SPIRES internal format, from a data set created by the GENERATE LOAD command.
Command: INPUT LOAD
Available in: SPIRES, SPIBILD
Available to: file owner
users with Master access to the file
Related commands: GENERATE LOAD, SET MESSAGES, LOAD
Recovery commands: ZAP DEFQ, INPUT LOAD
Procedures: INPUT LOAD procedure (2.3.3a)
For SPIRES: [WITH FAST|WITH SEARCHONLY] INPUT LOAD loadname [RENUMBER|UPDATE] For SPIBILD: INPUT LOAD loadname [RENUMBER|UPDATE]
The UPDATE option is blocked Secure Switch 10 in SPIRES only.
The INPUT LOAD command takes the data created by the GENERATE LOAD command and loads it into a SPIRES file. It adds the records in the load to the specified record-type one by one.
The record-type must have the same characteristics as the one from which the data came. For example, it must have the same fixed elements, with the same lengths and occurrences; the same structures, having the same number and type of elements, etc. [See the SPIRES manual "File Definition" for a discussion of "matching record-type characteristics".]
Because the input data is already in the internal format used by SPIRES to store records, the data is not processed through any processing rules; Inprocs and Incloses are ignored. The only data that can possibly change as the data is added is the record key, if they are slot keys, as requested by the RENUMBER option.
Here is an example showing the command's use in online SPIRES:
-> input load records.for.test - ADD 1, 1, Key = TV392 - ADD 2, 2, Key = UN901 - ADD 3, 3, Key = AV013 (etc.) - ADD 12, 12, Key = AR232 - Requests/Success: ADD 12 12 SUM 12 12 -End of load ->
The INPUT LOAD command in SPIBILD works almost exactly the same as the LOAD command. The only significant difference is that LOAD begins by processing the file, whereas with INPUT LOAD, an ESTABLISH command must be issued to process the file first. INPUT LOAD hence requires you to "select" the goal record-type by subfile name (through the ESTABLISH command); on the LOAD command, you name the file and goal record-type explicitly.
An example of the INPUT LOAD command in SPIBILD appears earlier in this manual. [See 2.3.3a.]
The WITH FAST option, available on the INPUT LOAD command (as well as INPUT BATCH) in SPIRES, puts the records into the deferred queue three to four times faster, in terms of processing costs, than the normal INPUT LOAD command. You can use it under these conditions:
- You must have Master access to the file.
- The deferred queue must be empty when you issue your WITH FAST request.
- The keys of the records being added or updated must be in ascending sequence, unless a key matches the key of a record added or updated earlier in the process. Note that the maximum speed will not be attained unless all records are in ascending sequence. (If you are only adding records, and the goal record-type has a slot key, and SPIRES will be assigning the keys, then this rule is not relevant.)
- The file must have a CKPT data set of its own. (If the file does not, then adding either a CHECKPOINT or IMMEDIATE statement to the file definition and recompiling will create one. See the manual "SPIRES File Definition" for information about those statements.)
- You are willing to forego the updating of immediate indexes as part of the input process; if the subfile has immediate indexes, you will have to process the file in SPIBILD to get the indexing done, and to allow other users to use the subfile for updates again.
During the processing, the file will be placed in SET SEARCH mode, so that no one else may update the file. Other users attempting to update with file will get S494 errors. [See 6.2.8, 2.1.5.1.]
If the subfile does not have any immediate indexes, then the subfile will be ready for all to use again upon completion of the command. That is, other users will again be able to update the subfile.
However, if the subfile has any immediate indexes, then SPIRES blocks deferred-queue updates when the INPUT LOAD command completes. (Users will get S560 errors if they try; you can test for this condition with the $DEFQTEST function, using the WRITE type.) This is because no index updates (immediate indexes or not) are done by INPUT LOAD with the WITH FAST option. The file must be processed or the DEFQ zapped before any further update requests will be honored.
The WITH SEARCHONLY prefix for INPUT LOAD in SPIRES also speeds up the input processing, though not as much as WITH FAST. However, it does not have the same requirements and restrictions. Basically, during the processing, SPIRES will reserve the file's CKPT (checkpoint) data set, so that it is not reserved and released for each update as normal INPUT LOAD processing is done. (That's why other users are allowed to do updates during regular INPUT LOAD processing.) Like WITH FAST, WITH SEARCHONLY puts the file into SET SEARCH mode, so that other users of the file are limited to searching and reporting. [See 6.2.8, 2.1.5.1.]
Any immediate indexing is done as normal, and the file is available for normal updating activity upon completion of the WITH SEARCHONLY INPUT LOAD command.
WITH SEARCHONLY, like WITH FAST, is available only to users with Master access to the file.
The INPUT MERGE command merges data into existing records of a subfile via a custom input format.
Command: INPUT MERGE
Available in: SPIRES, SPIBILD
Available to: (in SPIRES) users who can update the subfile
(in SPIBILD) file owner, and
users with Process access to the file
Related commands: ESTABLISH, INPUT ADDMERGE, MERGE,
SET EXCEPTION, SET FORMAT, SET MESSAGES
Recovery commands: UNQUEUE, ZAP DEFQ, SHOW RECORD RECOVERY, SET SKIP,
INPUT BATCH, REMOVE, ZAP RECORD-TYPE, ZAP FILE
Procedures: INPUT MERGE procedure in SPIBILD (2.4.1)
For online SPIBILD and in BATWYL jobs: [USING frame] INPUT MERGE For BATSPI and batch SPIBILD jobs: [FROM ddname] [USING frame] INPUT MERGE
When you use the INPUT MERGE command in SPIRES or SPIBILD, the data is read through an input format, executing frames of usage MERGE. The key for each record to be updated is almost invariably found in the input data; when the data for each record is processed by the format, it is merged into the existing record.
The data is always processed via a custom input format. MERGE is a command allowed in the standard SPIRES format, used with the INPUT BATCH command and procedure. [See 2.3.1, 2.3.5.] For information about formats for INPUT MERGE, see the manual "SPIRES Formats"; online, EXPLAIN FORMATS, FOR INPUT MERGE COMMAND.
In SPIRES, the INPUT MERGE command follows a SELECT command to choose the subfile into which the data will be merged and a SET FORMAT command naming the input format to be used. In SPIBILD, it follows a SET FORMAT command, which follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT MERGE command to work properly.
When you issue the INPUT MERGE command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT MERGE command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]
In SPIRES, the updated records end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the new data goes into the tree, and all indexes are updated.
Here is a sample session demonstrating the INPUT MERGE command in SPIBILD:
-? establish students -Processing file: GQ.DOC.STUDENTS -Compute time: 0.249 seconds -Elapsed time: 3.233 seconds -Core usage: 0065/0939 -? set format mult.merge -? input merge - MER 1, 1 @Line 6. Key = 85-1922 - MER 2, 2 @Line 8. Key = 86-0713 - MER 3, 3 @Line 15. Key = 86-0143 ... (etc.) - MER 9128, 9128 @Line 46296. Key = 83-1539 -Passing: 9128 records of record-type REC01 - Requests/Success: MER 9128 9128 SUM 9128 9128 -End of BATCH/BUILD -Compute time: 46.133 seconds -Elapsed time: 327.504 seconds -Core usage: 0311/0939 -?
The LOAD command adds records to a subfile using data already in SPIRES internal format, from a data set created by the GENERATE LOAD command.
Command: LOAD
Available in: SPIBILD, FASTBILD
Available to: file owner
users with Process access to the file
Related commands: LOAD, SET MESSAGES
Recovery commands: GENERATE LOAD
Procedures: LOAD procedure in SPIBILD (2.3.3)
LOAD procedure in FASTBILD (2.12.2)
For SPIBILD: LOAD loadname [RENUMBER] INTO record-name OF filename For FASTBILD: LOAD loadname [RENUMBER]
The LOAD command takes the data created by the GENERATE LOAD command and loads it into a SPIRES file. It begins by processing the file, and then adds the records in the load to the specified record-type one by one.
The record-type must have the same characteristics as the one from which the data came. For example, it must have the same fixed elements, with the same lengths and occurrences; the same structures, having the same number and type of elements, etc. [See the SPIRES manual "File Definition" for a discussion of "matching record-type characteristics".]
Because the input data is already in the internal format used by SPIRES to store records, the data is not processed through any processing rules; Inprocs and Incloses are ignored. The only data that can possibly change as the data is added is the record key, if they are slot keys, as requested by the RENUMBER option.
Here is an example showing the command's use in online SPIBILD:
-> load records.for.test into xrec01 of gq.tes.tinventory -Compute time: 0.079 seconds -Elapsed time: 1.150 seconds -Core usage: 0065/0937 - ADD 1, 1, Key = TV392 - ADD 2, 2, Key = UN901 - ADD 3, 3, Key = AV013 (etc.) - ADD 12, 12, Key = AR232 -Passing: 12 records of record-type REC01 - Requests/Success: ADD 12 12 -End of BATCH/BUILD -Compute time: 0.362 seconds -Elapsed time: 3.041 seconds -Core usage: 0211/0939 -?
The MERGE command in SPIBILD merges data into multiple records in a subfile under Global FOR.
Command: MERGE
Available in: SPIBILD
Available to: file owner
users with Process access to the file
Related commands: ESTABLISH, SET FORMAT, FOR SUBFILE, FOR
STORED, SET SCAN, SET FILTER
Recovery commands: MERGE, REMOVE, ZAP RECORD-TYPE, ZAP FILE
Procedure: MERGE procedure in SPIBILD in Global FOR (2.4.2)
For online SPIBILD:
MERGE [ALL|NEXT|n|REST] [FROM ACTIVE]
For batch SPIBILD:
MERGE [ALL|NEXT|n|REST] [FROM {ACTIVE|ddname}]
You can use the MERGE command to merge data into multiple records with a single command. [See 2.4.] It is used under Global FOR, which determines the records to be updated, usually based on criteria specified in a WHERE clause or SET FILTER commands. The input data may be provided from a data set or from an input format. The MERGE command must be preceded by ESTABLISH and FOR SUBFILE or FOR STORED commands. [See 2.4.2.]
Whether a merged record gets re-indexed depends on which elements were updated. SPIBILD checks the elements of the merged data against a list of elements (the "pass elements list") constructed by SPIRES when the file was compiled. If any elements from the list are updated, then the updated record goes into the deferred queue so that indexes can be updated; otherwise, the updated record is placed directly into the tree, and no superfluous re-indexing occurs. [See 2.4.3.3.]
Here is a sample session demonstrating the MERGE command in SPIBILD under Global FOR. Note that the format execution begins by asking for the data that it will apply to all the records ("What percent markup...").
-? set format new.overhead -? establish inventory -? for subfile -? merge all What percent markup should be added? 25 - MER 1, 1 @Line 0. Key = CR54 - MER 2, 2 @Line 0. Key = PT109 - MER 3, 3 @Line 0. Key = RM222 ... (etc.) - MER 329, 329 @Line 0. Key = RT66 - Requests/Success: MER 329 329 SUM 329 329 -End of BATCH/BUILD -Compute time: 12.169 seconds -Elapsed time: 73.143 seconds -Core usage: 0256/0939 -?
The PERFORM USER LIST command tells you what users currently have selected any subfiles that have subfile logging for a given file.
Command: PERFORM USER LIST
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: SPIBILD processing commands
Recovery commands: not applicable
[IN ACTIVE [CLEAR|CONTINUE]] PERFORM USER LIST
PERFORM USER LIST produces a list of accounts that may have any subfile selected of the file holding your currently selected subfile. It relies on command logging being in effect for the file's subfiles (at least to the level of LOG = 1).
It can be useful when you want to process or recompile a file but want to know who, if anyone, might have subfiles of the file selected. For processing the file, you can SET SHARE or SET SEARCH, and to recompile the file, you can include the SHARE option on the RECOMPILE command in order to do your work while users have subfiles of the file selected. However, for various reasons, you may want to know whom you may be affecting by going ahead with your work.
Under some conditions, the display from this command may not contain all the accounts that have the subfile selected. For example, if a user has selected a subfile of the file and then the file is processed under SET SEARCH, that user would not then show up in the PERFORM USER LIST display as a current user of the subfile. The log entry in which the user's initial selection of the subfile was recorded will have been erased from the current log by the file processing.
-> select dept ledger -> perform user list GQ.PBT GQ.JNK -> to gq.pbt Ready or not, I'm processing the file. ->
The current user, GQ.JNK, selects the subfile and discovers someone else has a subfile of the file selected. He then tells his colleague he plans to process the file.
The PROCESS command moves subfile transactions from the file's deferred queue to the tree and updates the file's indexes accordingly.
Command: PROCESS
Available in: SPIBILD, SPIRES
Available to: file owner
users with Process access to the file
Related commands: ESTABLISH, BATCH, MERGE, LOAD
Recovery commands: PROCESS, RECOVER, SET JOBGEN
Procedures: PROCESS/ESTABLISH procedure (2.2.2)
RECOVER/PROCESS procedure (2.1.4.2)
FAST BATCH procedure (2.3.4)
In SPIBILD: PROCESS filename In SPIRES: PROCESS [NOWARN]
ORV.gg.uuu.filename gg.uuu.filename .filename *filename filename
The PROCESS command has these specific tasks:
- apply record transactions in the deferred queue to the tree:
- add newly added records to the tree;
- replace old copies of records in the tree with new versions of updated records;
- discard records in the tree that have been removed.
- update any non-immediate indexes, synchronizing them with the record transactions in the deferred queue.
- if a subfile log is kept, either discard its contents, or add them to the archive data set called "SYSTEM.ULOG".
WARNING: The PROCESS command discards any batch requests submitted by users via the BATCH command in SPIRES. Those requests can be handled only by JOBGEN-created processing jobs. You may want to issue the SHOW BATCH REQUESTS command in SPIRES before processing the file, if users might have submitted batch requests.
When the processing is completed, the DEFQ data set for the file will be practically empty -- it should be two blocks large (or whatever size the file definition statement QUELEVEL specifies). Extra blocks added to the DEFQ as it grew will be released during file processing.
Here is a sample SPIBILD session demonstrating the PROCESS command:
Command> spibild -Welcome to SPIBILD -? process orv.gq.doc.address.book -Passing: 68 records of record-type REC01 -Passing: 18 records of record-type REC01 -Processed: 86 DEFQ records of record-type REC01 -Passing: 12 records of record-type REC19 -Processed: 12 DEFQ records of record-type REC19 -Compute time: 8.657 seconds -Elapsed time: 69.574 seconds -Core usage: 0267/0939 -?
In SPIRES, you must precede the PROCESS command by a SELECT command, to select a subfile of the file you want to process. (Note from the syntax that you do not include the file name in the PROCESS command in SPIRES.)
SPIBILD and SPIRES process the record-types in the order in which they are defined in the file definition (alphabetic order). Thus, the REC01 records are processed before REC19 records, for example.
Here are descriptions of each processing message shown in the example above:
The PROCESS SUBFILE command moves transactions for the selected subfile from the file's deferred queue to the tree and updates the file's indexes accordingly.
Command: PROCESS SUBFILE
Available in: SPIRES
Available to: file owner
users with Process access to the file
Related commands: PROCESS
Recovery commands: PROCESS, RECOVER, SET JOBGEN
Procedures: PROCESS SUBFILE procedure (2.2.2a)
PROCESS SUBFILE [NOWARN]
The PROCESS SUBFILE command has these specific tasks:
- apply record transactions for only the selected subfile in the deferred queue to the tree:
- add newly added records to the tree;
- replace old copies of records in the tree with new versions of updated records;
- discard records in the tree that have been removed.
- update any non-immediate indexes, synchronizing them with the record transactions in the deferred queue.
- delete those transactions from the deferred queue.
If there are transactions in the deferred queue for other record-types, or if the subfile-processing creates other transactions (goal-to-goal passing), those transactions are ignored, and will not get processed till the next full-file process. If there are no other transactions and no log data, then the deferred queue is re-initialized as if a regular full-file process had taken place.
If a problem occurs that prevents the process from completing successfully, the NOJOBGEN flag will get set, meaning that record updates to any record-type in the file will fail until file recovery takes place. [See 2.1.4.2.]
Note: Unlike the PROCESS command, the PROCESS SUBFILE command does not discard any log data or batch requests submitted by users via the BATCH command in SPIRES.
Here is a sample session demonstrating the PROCESS SUBFILE command:
Command> spires -Welcome to SPIRES -> select addresses -? process subfile -Passing: 68 records of record-type REC01 -Passing: 18 records of record-type REC01 -Processed: 86 DEFQ records of record-type REC01 -Compute time: 6.923 seconds -Elapsed time: 42.776 seconds -Core usage: 0267/0939 -?
In SPIRES, you must precede the PROCESS SUBFILE command by a SELECT command, to select the subfile you want to process.
Descriptions of each processing message shown in the example above appear in the section on the PROCESS command, a page or two back. Online, EXPLAIN PROCESS COMMAND.
The REBALANCE DATA SET command reorganizes the contents of a tree-structured data set to prevent it from becoming too deep and/or to improve the average number of disk I/Os per record.
Command: REBALANCE DATA SET
Available in: SPIRES
Available to: file owner
Related commands: ATTACH
Recovery commands: FIX REBALANCED DATA SET, FIX BASE
Procedures: REBALANCE procedure (3.3.1)
Create or Modify Extended Trees (5.6)
[IN ACTIVE [CLEAR|CONTINUE]] REBALANCE DATA SET {RECn|DEFQ} [,]
[RENAME] [APPEND] [NOCONTINUE] [CONVERT]
The REBALANCE DATA SET command is the main command of the rebalancing procedure, used to improve the structure of record-types that have become "unbalanced", i.e., the tree's branches are very uneven, resulting in inefficient retrieval of many records. The command creates a new version of a data set containing tree-structured record-types, with the records in each record-type more efficiently arranged. [See 3.3.]
If you specify the RENAME option, the command replaces the old data set with the newly rebalanced one, resetting all the base blocks for each record-type. If you omit the RENAME option, you will have to complete that part of the procedure using either the FIX REBALANCED DATA SET command or "by hand" with the RENAME, SET CLP and FIX BASE commands.
A complete example of the procedure for rebalancing data sets appears earlier in the manual. The example below shows only the REBALANCE DATA SET command and its output, which is just one part of the complete procedure. [See 3.3.1.]
-> rebalance data set rec2, rename
Rebalance Data Set REC2
-Base block 1 in record-type REC02
-Next block is 22
-Base block 86 in record-type REC03
-Next block is 98
-Base block 99 in record-type REC04
-Next block is 124
- REBALANCE complete - Start RENAME process
FIX rebalanced data set REC2
-REC02 base is 1; old base was 1
-REC03 base is 86; old base was 21
-REC04 base is 99; old base was 88
- Next block is 124; Next block was 142
ORV.GQ.JNK.CHIPS.REC2 is renamed to ORV.GQ.JNK.CHIPS.REO2.19870107
ORV.GQ.JNK.CHIPS.REB2 is renamed to ORV.GQ.JNK.CHIPS.REC2
-Set new data set permits if needed.
-File detached
->
In addition, SPIRES will display a message each time 64K nodes have been processed, to reassure you (especially if the record-types are large) that the rebalancing work is occurring.
See the discussion about "Handling Problems" in the procedural description if you have any problems. [See 3.3.1, 3.3.2.]
Technical Note: There is a REBALANCE RECORD-TYPE command available to very large files that cannot rebalance an entire data set in a single session. Contact your SPIRES consultant if you think you need to use that command.
The REBALANCE DEFQ command rebuilds a file's DEFQ and rebalances the DEFQ's tree.
Command: REBALANCE DEFQ Available in: SPIRES Available to: file owner Related commands: ATTACH Recovery commands: REBALANCE DEFQ Procedures: REBALANCE DEFQ procedure (3.3.3)
[IN ACTIVE [CLEAR|CONTINUE]] REBALANCE [DATA SET] DEFQ ... ... [RENAME|TREE|COPY]
A complete example of the procedure for rebalancing a deferred queue appears earlier in the manual. The example below shows only the REBALANCE DEFQ command and its output, which is just one part of the complete procedure. [See 3.3.3.]
-> rebalance defq rename
Rebalance DEFQ data set
-Base block 1 in DEFQ data set
-Next block is 3
- REBALANCE complete - Start RENAME process
FIX rebalanced data set DEFQ
ORV.GQ.DOC.ERRATA.DEFQ is renamed to ORV.GQ.DOC.ERRATA.DEOQ.19920717
ORV.GQ.DOC.ERRATA.DEBQ is renamed to ORV.GQ.DOC.ERRATA.DEFQ
-Set new data set permits if needed.
->
The RECOVER command, primarily used to clean up after a SPIBILD command fails, partially or completely processes a SPIRES file.
Command: RECOVER
Available in: SPIBILD
Available to: file owner
users with Process access to the file
Related commands: PROCESS
Recovery commands: RECOVER
Procedures: RECOVER/PROCESS procedure (2.1.4.2)
RECOVER filename
ORV.gg.uuu.filename gg.uuu.filename .filename *filename filename
Syntax note: The RECOVER command cannot be abbreviated.
The RECOVER command can be considered a "mini-PROCESS" command. Its main purpose is to get a file with the NO JOBGEN flag set into a stable state such that normal file activities can resume. Depending on how much data is in the deferred queue, it may or may not process the file completely; however, if all goes well, it will at least stabilize the deferred queue and reset the JOBGEN flag. (Technically speaking, RECOVER will create and process up to two pass stacks worth of index data and then stop.)
If RECOVER processes the entire deferred queue, then you will be left in SPIBILD at the end of the processing. But if data remains in the deferred queue at the end of the RECOVER work, SPIBILD will issue an automatic EXIT command, taking you out of SPIBILD.
In most cases when you are trying to recover from a SPIBILD command failure, the PROCESS command can be used instead. However, when you are working with a file that has many transactions in the deferred queue, a complete process of the file may take too long or cost too much (at daytime rates).
If the RECOVER command fails and you get error messages, you may have a serious problem with your file. Use the EXPLAIN command to get more information about the errors, and contact your SPIRES consultant if you need further help.
Here is a sample SPIBILD session demonstrating the RECOVER command:
-? recover orv.gq.doc.wysiwyg -Passing: 13 records of record-type REC01 -Processed: 13 DEFQ records of record-type REC01 -Compute time: 0.198 seconds -Elapsed time: 2.010 seconds -Core usage: 0210/0939 -?
The data shown is similar to the display provided by the PROCESS command.
The REMOVE command in SPIBILD removes records from a subfile as determined by the established Global FOR environment.
Command: REMOVE
Available in: SPIBILD
Available to: file owner
users with Process access to the file
Related commands: ESTABLISH, FOR SUBFILE, SET SCAN, SET FILTER
Recovery commands: none
Procedures REMOVE procedure in SPIBILD in Global FOR (2.5.1)
REMOVE [ALL|NEXT|n|REST]
In SPIBILD, the REMOVE command can be issued only under Global FOR, meaning that it must be preceded by the ESTABLISH command and a FOR SUBFILE or FOR TREE command. [See 2.5.1.]
As it executes, it does not immediately remove the records. Instead, it places removal requests in the deferred queue, just as the REMOVE command does in SPIRES. The difference is that when SPIBILD is finished processing the removes, it processes the deferred queue, causing the records to be removed from the file and the indexes to be updated.
Here is part of a sample SPIBILD session using the REMOVE command. The ESTABLISH command to process the file and "select" the subfile has already been issued.
-? for subfile where register.date < 1985 -? remove all - REM 1, 1 @Line 0. Key = 0019323 - REM 2, 2 @Line 0. Key = 0365954 ... - REM 154, 154 @Line 0. Key = 8555991 - Requests/Success: REM 154 154 SUM 154 154 -End of BATCH/BUILD -Passing: 154 records of record-type REC01 -Processed: 154 DEFQ records of record-type REC01 -Compute time: 10.201 seconds -Elapsed time: 41.528 seconds -Core usage: 0209/0939 -?
The SET AUTOGEN command turns on automatic processing of the file by JOBGEN on a one-time basis. The SET NOAUTOGEN command turns off JOBGEN processing of the file.
Command: SET AUTOGEN and SET NOAUTOGEN
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: SHOW AUTOGEN, SET JOBGEN
Recovery commands: SET AUTOGEN, SET NOAUTOGEN
Procedures: JOBGEN (2.2.1)
SET AUTOGEN SET NOAUTOGEN
There are no options on either command.
The AUTOGEN flag determines whether a file is to be processed by JOBGEN, the program that automatically submits batch SPIBILD jobs overnight for files with data in their deferred queues. The basic setting for any file is determined by the file definition: if the file definition contains a NOAUTOGEN statement, then the basic setting for that file is NOAUTOGEN. Otherwise, it is AUTOGEN.
By issuing the SET AUTOGEN or SET NOAUTOGEN command, you can change the setting temporarily. The change is temporary in the sense that the next time the file is processed in SPIBILD (either via JOBGEN or manually), the flag will revert back to its basic setting.
Hence, if the basic setting for a file is NOAUTOGEN and you issue the SET AUTOGEN command, then beginning that evening, JOBGEN will check the deferred queue of the file for data and if so, will submit a batch SPIBILD job to process the file. When the file gets processed, the flag will revert to NOAUTOGEN. Notice that if there is no data in the deferred queue, no batch SPIBILD job will be submitted, so the file will not get processed, and the flag will remain set as AUTOGEN.
On the other hand, if the basic setting for a file is AUTOGEN and you issue the SET NOAUTOGEN command, JOBGEN will ignore the file until the flag gets reset to AUTOGEN, which would happen if you issued a SET AUTOGEN command or if you processed the file yourself in SPIBILD.
To set the AUTOGEN flag for a file, you must first select one of its subfiles in SPIRES:
-> select spires doc notes -> show autogen -NOAUTOGEN in effect -> set autogen -> show autogen -AUTOGEN in effect ->
Remember that anytime the file is processed, the AUTOGEN flag reverts to its basic setting. Thus, if a file has a basic setting of NOAUTOGEN and you SET AUTOGEN, then if you (or someone else) process the file yourself before overnight processing occurs, the flag will be reset to NOAUTOGEN, and no JOBGEN processing will occur till you set the AUTOGEN flag again.
The SET EXCEPTION command establishes an ORVYL data set into which invalid input data will be placed during SPIBILD input operations.
Command: SET EXCEPTION
Available in: SPIRES, SPIBILD, FASTBILD
Available to: file owner
users with Process access to the file
Related commands: INPUT BATCH, INPUT MERGE, INPUT ADD,
FAST BATCH, FAST BUILD
Recovery commands: CLEAR EXCEPTION
Procedures: Exception File procedure (2.1.5.2)
INPUT BATCH procedure in SPIRES (2.3)
INPUT ADD procedure in SPIRES (2.3)
INPUT MERGE procedure in SPIRES (2.4)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
FAST BATCH procedure (2.3.4)
INPUT MERGE procedure in SPIBILD (2.4.1)
FAST BUILD procedure (2.12.2)
SET EXCEPTION orvyl-filename [REPLACE]
The exception file is created as soon as you issue the SET EXCEPTION command. During execution of the primary processing command where data is coming from the active file or some other external file (such as the BATCH command in SPIBILD), any data that causes a serious-level error, which prevents that record transaction from succeeding, is placed in the exception file. After the processing is completed, you can issue the ORVYL command "GET filename" to bring the exception file into your active file, allowing you to make changes and try putting the data in again.
If you decide not to use the exception file after setting one, issue the command CLEAR EXCEPTION. As soon as a file has been processed using an exception file, there is an automatic CLEAR EXCEPTION. In any case, whether any data is placed in it or not, an exception file exists as an ORVYL data set as soon as you issue the SET EXCEPTION command.
Here is part of a SPIBILD session demonstrating the SET EXCEPTION command:
-? set exception problem.records -? batch restaurant ... processing is done
A longer example, showing the SET EXCEPTION command in the context of exception processing, appears earlier in this manual. [See 2.1.5.2 for information on the complete SET EXCEPTION procedure.]
The SET FILE COUNTS command displays the number of reads and writes from and to attached ORVYL data sets during SPIBILD processing.
Command: SET FILE COUNTS Available in: SPIBILD Available to: anyone Related commands: SHOW FILE COUNTS, CLEAR FILE COUNTS Recovery commands: CLEAR FILE COUNTS, EXIT Procedures: any SPIBILD procedure
SET FILE COUNTS
The command has no options.
By default, SPIBILD will display a summary of the read/write activity involving the data sets (and specific record-types within them) for each SPIRES file as it finishes with the file. If you issue the SET FILE COUNTS command before you start your file-processing activity, SPIBILD will also present a display of the same information for the system files (such as those containing the FILEDEF and FORMATS subfiles) used during the session when you leave SPIBILD.
SET FILE COUNTS will also turn the user-file statistics summary back on if it has been turned off with the CLEAR FILE COUNTS command.
The command presents the same type of data as does the SHOW FILE COUNTS command in SPIRES. Unlike that command, however, SET FILE COUNTS continues to display statistics during the session.
Here is part of a SPIBILD session demonstrating the statistics that SPIBILD displays:
Command> spibild
-Welcome to SPIBILD
-? process gq.doc.calendar
-Begin passing of 26 records of record-type G1SECT
-Completed passing of 26 records total
-Processed: 26 DEFQ records of record-type G1SECT
08/11/90 File Read/Write Counts Information
File: GQ.DOC.CALENDAR Reads Writes
CKPT data set 4 153
DEFQ data set 251 5
MSTR data set 19 0
Residual data set 279 282
Rec-type G1SECT 2 8
Rec-type REC02 13 72
Rec-type REC03 9 12
Rec-type REC04 16 101
Rec-type REC05 15 18
Rec-type REC06 28 77
Total for file: 636 728
-Compute time: 2.993 seconds
-Elapsed time: 33.714 seconds
-Core usage: 0330/0967
-?
For each file involved, SPIBILD displays the number of ORVYL blocks read and written since the primary file was attached (i.e., the file that was processed). Specifically, the read/write data is displayed for:
- the DEFQ data set, which holds the deferred queue
- the MSTR data set, which contains the file's master characteristics
- the Residual data set (RES)
- the file's private checkpoint data set (CKPT) if it has one
- the individual record-types of the file
So the example above shows, for example, that SPIBILD read 251 blocks of the CALENDAR file's deferred queue, and wrote 5. Keep in mind that the numbers reflect the number of read/write operations, not the numbers of different blocks read or written. Since the counts are cumulative over the SPIBILD session, the same block may be read or written multiple times, with each operation being counted.
For system files when you've requested the additional information from SET FILE COUNTS, SPIBILD displays similar counts for those files, except that they are combined into one file display. That is, the display for "System files" shows counts as if they all came from a single large system file, rather than from individual files. For example, REC2 represents a record-type in the system's FILEDEF file that is read when users establish a subfile, but it also represents a record-type in the system's FORMATS file where compiled formats code is stored.
Here is a list of the record-types you are likely to see in the system files section, along with the types of data being read from them:
REC1 - EXPLAIN explanations
SHOW SPIRES MAIL
REC2 - subfile selection records
compiled formats code
REC4 - compiled vgroups code
EXPLAIN explanations for system messages
REC5 - STORE and RESTORE STATIC commands
The SET INDEX and SET NOINDEX commands limit the index record-types in a file that may receive passed information for a particular SPIBILD or FASTBILD command.
Command: SET INDEX and SET NOINDEX
Available in: SPIBILD, FASTBILD
Available to: file owner
users with Process access to the file
Related commands: SET PASS, SET NOPASS
Recovery commands: SET INDEX, SET NOINDEX
Procedures: Limiting indexes affected by SPIBILD (2.1.5.4)
PROCESS/ESTABLISH procedure (2.2.2)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
LOAD procedure in SPIBILD (2.3.3)
INPUT MERGE procedure in SPIBILD (2.4.1)
MERGE procedure in SPIBILD with Global FOR (2.4.2)
REMOVE procedure in SPIBILD with Global FOR (2.5.1)
FAST BUILD procedure in FASTBILD (2.12.2)
LOAD procedure in FASTBILD (2.12.2)
SET INDEX record-names SET NOINDEX record-names
Technical note on syntax: Any of the "record-names" can be followed by a positive integer in parentheses, e.g.:
-? set noindex zin10, zin11(2)
The number is used to identify a specific occurrence of the INDEX-NAME instructions when the same name appears several times in the linkage section for a given goal record-type. (Warning for complicated files: The counting of "n" begins again with each new linkage section. If, for example, REC03 serves as an index record-type for both goal record-types REC01 and REC02, and REC01's linkage is first in the file definition, then the command SET NOINDEX REC03(2) will tell SPIBILD/FASTBILD not to pass values for the second occurrence of "INDEX-NAME = REC03" in each linkage section. The command does not mean that values should be passed to REC03 from REC01 but not REC02.)
WARNING: These commands are meant to solve unusual data integrity problems, but if not used carefully, can create them. In general, you should use these commands only in extraordinary situations and then only after consultation with your SPIRES consultant.
In rare situations, you may not want a SPIBILD/FASTBILD processing command to pass data to all the index record-types as specified in the file's definition. You can issue the SET INDEX and SET NOINDEX commands to limit the index record-types affected by the command to only those you want. In other words, these two commands turn off or "leave on" passing to the named record-types.
The SET INDEX command names the only index record-types that are to receive passed information. The SET NOINDEX command names the index record-types that should not receive passed information.
Neither command has any effect until you issue a file-processing command, such as PROCESS or BATCH. Note too that it affects only the primary task of the SPIBILD/FASTBILD command, not any preliminary file-processing the command does, in contrast to the SET PASS and SET NOPASS commands, which affect all processing of the subsequent command. [See 2.1.5.4.]
For example, the BATCH command causes the file to be processed before the new records are batched into the subfile. If you issue a SET INDEX/NOINDEX command before the BATCH command, it will have no effect on the preliminary processing, but will affect the records batched in. If you want the records already in the deferred queue to be affected by SET INDEX/NOINDEX, issue both the PROCESS and BATCH commands, preceding each with the appropriate SET INDEX/NOINDEX command.
The SET JOBGEN command resets the JOBGEN flag, allowing JOBGEN to submit batch SPIBILD jobs to process the file and allowing subfile transactions to resume. SET NOJOBGEN has the reverse effect: it prevents JOBGEN from submitting batch SPIBILD jobs to process the file and it prevents subfile transactions from succeeding.
Command: SET JOBGEN and SET NOJOBGEN
Available in: SPIRES
Available to: file owner
users with Process access to the file
Related commands: SET AUTOGEN, SHOW AUTOGEN, RECOVER, PROCESS
Recovery commands: SET NOJOBGEN, SET JOBGEN
Procedures: JOBGEN (2.2.1)
REBALANCE procedure (3.3.1)
SET JOBGEN SET NOJOBGEN
The commands have no options and cannot be abbreviated.
When a SPIBILD command fails, the NO JOBGEN flag is set, which blocks all update activity for the file, and prevents JOBGEN from submitting batch SPIBILD jobs to process the file overnight. The SET JOBGEN command sets the flag to JOBGEN, once again allowing update activity and again allowing JOBGEN to process the file.
WARNING: Do not issue the SET JOBGEN command for a file unless you are positive that no update activity will occur between the time you issue the command and the time the file is processed. If any updates do occur, data integrity of the file could be lost!
The SET NOJOBGEN command prevents JOBGEN from creating a job to process the file. It also unconditionally blocks all updating activity for the entire file, taking effect immediately. Update requests will fail with an S495 error. SET JOBGEN would then reverse the condition.
Remember though, that if a SPIBILD or FASTBILD failure causes the NO JOBGEN flag to be set, you should not simply reset the JOBGEN flag but should try to recover or process the file first, allowing SPIBILD to reset the flag if the file is okay. [See 2.2.1.]
In the example below, no one except the file owner has access to the subfile. She knows that she will not add or update any records after issuing the command until the next day, after the file is processed via JOBGEN. This may save her some money, compared to processing the file herself now, at day rates, to recover from the SPIBILD problem that set the NO JOBGEN flag.
-> select venomous animals -> show autogen -AUTOGEN in effect (NO JOBGEN) -> set jobgen ->
Remember, if there is any chance that users will try to update records between the time you set the JOBGEN flag and the time the file gets processed via JOBGEN, you should not use this command. Instead, follow the PROCESS/RECOVER procedure. [See 2.1.4.2.]
If you change your mind and want NO JOBGEN set, issue the SET NOJOBGEN command.
The SET MESSAGES command controls the presentation of various types of system messages.
Command: SET MESSAGES
Available in: SPIRES, SPIBILD, FASTBILD
Available to: anyone
Related commands: none
Recovery commands: SET MESSAGES
Procedures using it: Any procedures where you need control
of system messages
SET [type] MESSAGES [TO|=] {0|1|2|3|4}
- INFORMATION (INFO) -- controls messages from the program that provide information (such as "-Stack: 15 RECORDS") or confirm an action (such as the line of information about each processed record by the BATCH command in SPIBILD).
- WARNING (WARN) -- controls messages from the program that announce a potential problem, that announce that SPIRES has taken some action to avoid a problem, or that give you more information about a serious error (e.g., "-Error at line 12").
- SERIOUS (SER) -- controls messages from the program that announce a processing problem, such as "-Serious data error".
- CHARGE (CHR) -- controls messages that announce subfile logging and charging is in effect (available only in SPIRES).
- ALL -- controls all of the above.
-? set messages 0 -? set messages = 0 -? set messages to 0
For all types EXCEPT Charge:
Level Explanation Sample
----- ---------------- ---------------------------------
0 no message
1 coded message -127(S)-
2 text message -Forward range exhausted
3 code and text -127(S)- Forward range exhausted
4 code, text and -127(S)- Forward range exhausted
explanation --A BROWSE command has reached
the end of the range specified.
For Charge messages:
Level Explanation
----- -----------------------------------------------------------------
0 no messages
1 messages when the subfile is selected, for a logging subfile:
-Command logging in effect for this subfile
or for a charging subfile:
-Database charge: $n.nn/hour, $n.nn/1000 displays, $n.nn minimum
and when the subfile is cleared, if charging is in effect:
-Approximate database charge: $n.nn
2 same as 1, but when a charging subfile is selected, also:
-Continue select?
3 none of the messages from 1, but the prompt from 2 for a
charging subfile.
4 no messages
Syntax Notes:
- In SPIBILD and FASTBILD, the automatic explanations of level 4 are not available, so level 4 is equivalent to level 3.
- If you type a number higher than those shown (say, 5), the level gets set to the highest setting, 4.
Control over messages is more often needed by application designers than by file managers, so SET MESSAGES isn't as useful here as it is in protocols and formats, where it is often used to turn off the messages SPIRES sends to users.
For file management purposes, SET MESSAGES is probably most often used to suppress information messages during interactive SPIBILD sessions where records are being added, updated or removed.
The level set by a SET MESSAGES command remains in effect throughout the program session unless changed by another SET MESSAGES command or temporarily overridden by a SET MESSAGES Uproc in an executing format.
In this example, the one-line information message for each successfully batched record is suppressed:
-> spibild -Welcome to SPIBILD -? set information messages = 0 -? batch united states -Processing file: GQ.JNK.UNITED.STATES -Compute time: 0.121 seconds -Elapsed time: 1.054 seconds -Core usage: 0085/0939 -Element=ZIPCODE.START: Value must be 5 digits -Serious data error, code=E22 -Error at or before line 818. -Update terminated, code=S261 -Error at or before line 821. -*ADD 48, 47 @Line 821. Key = VA -Passing: 53 records of record-type REC01 - Requests/Success: ADD 54 53 SUM 54 53 -End of BATCH/BUILD -Compute time: 3.182 seconds -Elapsed time: 20.839 seconds -Core usage: 0279/0939 -?
If the information messages hadn't been suppressed, there would have been 53 more lines of information, one for each successful transaction. Those confirmation lines slow down the task because SPIBILD must stop its work to send a message to the terminal for each successful transaction. Notice that information about the one transaction that failed (for the VA record) was not suppressed because that was a Serious message.
The SET PASS and SET NOPASS commands limit the goal record-types in a file that may pass information for a particular SPIBILD or FASTBILD command.
Command: SET PASS and SET NOPASS
Available in: SPIBILD, FASTBILD
Available to: file owner
users with Process access to the file
Related commands: SET INDEX, SET NOINDEX
Recovery commands: SET PASS, SET NOPASS
Procedures: Limiting indexes affected by SPIBILD (2.1.5.4)
PROCESS/ESTABLISH procedure (2.2.2)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
LOAD procedure in SPIBILD (2.3.3)
FAST BATCH procedure in SPIBILD (2.3.4)
INPUT MERGE procedure in SPIBILD (2.4.1)
MERGE procedure in SPIBILD in Global FOR (2.4.2)
REMOVE procedure in SPIBILD in Global FOR (2.5.1)
FAST BUILD procedure in FASTBILD (2.12.2)
LOAD procedure in FASTBILD (2.12.2)
SET PASS [n] SET NOPASS [n]
WARNING: These commands are meant to solve unusual data integrity problems, but if not used carefully, can create them. In general, you should use these commands only in extraordinary situations and then only after consultation with your SPIRES consultant.
The SET PASS command without options represents the default passing condition for a file: it allows passing from all goal record-types in the file. With the "n" option, the command allows passing only from the "nth" logical record-type; passing from all other record-types in the file is blocked.
The SET NOPASS command blocks passing from all goal record-types. With the "n" option, the command allows passing from all goal record-types except the "nth" logical record-type.
Neither command has any effect until a SPIBILD or FASTBILD file-processing command is issued. It affects all subsequent processing of the file, including any preliminary processing done by a command (e.g., BATCH) before the main task of the command begins executing. If that isn't desired, try processing the file first (with the PROCESS/ESTABLISH procedure) before issuing the SET PASS or SET NOPASS command. [See 2.2.2.]
Only one "pass" setting is in effect at a time. A subsequent SET PASS/NOPASS command cancels any previous one, so if you change your mind after issuing the command, issue an opposite command to cancel it. (For example, if you want to retract a "SET PASS 5" command, then a SET PASS command would cancel it.) Also, the default condition (SET PASS) is restored after each file-processing command is executed, with the exception of ESTABLISH.
The SET SEARCH command tells SPIBILD or FASTBILD to let users continue to select a subfile and search and display records while the file is being used for processing tasks. SET NOSEARCH cancels SET SEARCH.
Command: SET SEARCH and SET NOSEARCH
Available in: SPIBILD, FASTBILD
Available to: file owner
users with Process access to the file
Related commands: SET SHARE
Recovery commands: SET SEARCH, SET NOSEARCH
Procedures: SET SEARCH/SET SHARE (2.1.5.1)
PROCESS/ESTABLISH procedure (2.2.2)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
LOAD procedure in SPIBILD (2.3.3)
FAST BATCH procedure in SPIBILD (2.3.4)
INPUT MERGE procedure in SPIBILD (2.4.1)
MERGE procedure in SPIBILD in Global FOR (2.4.2)
REMOVE procedure in SPIBILD in Global FOR (2.5.1)
FAST BUILD procedure in FASTBILD (2.12.2)
LOAD procedure in FASTBILD (2.12.2)
BUILD procedure in FASTBILD (2.12.2)
SET SEARCH SET NOSEARCH
The commands have no options; the abbreviations SEA and NOSEA are allowed.
By default, a file is unavailable to users while it is being processed or being used for other SPIBILD or FASTBILD tasks. For some large applications, where round-the-clock access is necessary, the "down-time" of a file while it is being processed can be a problem. For instance, a very large file, used by dozens of people each day, might take ten minutes or more each night to process.
Moreover, SPIBILD and FASTBILD will not by default even let you process a file if users have any of its subfiles selected.
Both problems can be solved with the SET SEARCH command. It allows you to perform SPIBILD/FASTBILD tasks on a file when users are currently using it, and the command allows users to select subfiles of the file, and search and display records, as file processing goes on. Specifically, all read-only operations using the file can continue; file updates, on the other hand, are not allowed till the processing is finished. (Compare SET SEARCH to SET SHARE, which lets you process the file while users have its subfiles selected, but which does not let the users continue to work with the subfile.)
The commands must be issued before the SPIBILD or FASTBILD processing command they are to affect. Note that the prevailing condition is not cleared at the end of the file-processing but remains in effect for subsequent file-processing commands, unless altered by another SET SEARCH or SET NOSEARCH command.
Notes:
Suppose you need to process a file in SPIBILD during the day. The file contains multiple subfiles used by many people during the day. You need to add some records to one of the subfiles using SPIBILD's BATCH command. Issuing the SET SEARCH command before BATCH allows people searching the other subfiles to continue, unaffected by your work.
-> spibild -Welcome to SPIBILD -? set search -? batch department table ... processing of the file begins
The SET SHARE command tells SPIBILD to let you process a file even though users have it selected. SET NOSHARE cancels SET SHARE.
Command: SET SHARE and SET NOSHARE
Available in: SPIBILD, FASTBILD
Available to: file owner
users with Process access to the file
Related commands: SET SEARCH
Recovery commands: SET SHARE, SET NOSHARE
Procedures: SET SEARCH/SET SHARE procedure (2.1.5.1)
PROCESS/ESTABLISH procedure (2.2.2)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
LOAD procedure in SPIBILD (2.3.3)
FAST BATCH procedure in SPIBILD (2.3.4)
INPUT MERGE procedure in SPIBILD (2.4.1)
MERGE procedure in SPIBILD in Global FOR (2.4.2)
REMOVE procedure in SPIBILD in Global FOR (2.5.1)
FAST BUILD procedure in FASTBILD (2.12.2)
LOAD procedure in FASTBILD (2.12.2)
BUILD procedure in FASTBILD (2.12.2)
SET SHARE SET NOSHARE
The commands have no options; you may use the abbreviations SHA and NOSHA.
When a file is processed, the account doing the processing needs exclusive access to the file. That is, other users cannot be changing records in the file while the processing occurs. Similarly, while the file is being processed, other users cannot search in the file's subfiles because the processing may be changing the indexes.
SPIBILD and FASTBILD generally avoid these problems by not letting file owners process files if users have their subfiles selected, and by not letting users select a subfile if its file is being processed. The file owner who tries to process a file being used receives an S56 error message: the file is "busy". A user who tries to select a subfile of a file being processed does not get another command prompt until the processing is complete -- the terminal "hangs".
Occasionally, however, you may absolutely need to do some SPIBILD work with one of your files, regardless of the fact that some other user has one of its subfiles selected. If you are the file owner or a user with Process access to the file, you can issue the command SET SHARE to request non-exclusive access to the file. The command has no effect until you issue one of these SPIBILD or FASTBILD commands:
PROCESS ESTABLISH BUILD INPUT BATCH BATCH MERGE LOAD FAST BUILD
As soon as one of those commands is issued, users who try to select any subfile of the file or issue searching and updating commands against it will find that their terminal is hanging, waiting for the completion of your SPIBILD task. [Depending on what physical data sets of the file are being used by your processing work, any given user command could succeed at a particular moment. For example, while records are being moved into a subfile's tree, the indexes might still be available so that a search command would go through.] Note that users who have those subfiles selected can still do other work (e.g., text editing) -- only commands directed to the particular subfile will cause the terminal to hang.
When the processing command (except ESTABLISH) finishes executing, the "share" condition ends -- users will start getting responses to commands that use the file. (The ESTABLISH command is exceptional because it is the first step of a SPIBILD task, not the only one. The SPIBILD commands that usually follow it, such as MERGE and REMOVE, will end the "share" condition. Leaving SPIBILD via an EXIT or CALL command will also clear the "share" condition.)
However, ending the condition does not "clear" the condition. If you issue another SPIBILD command from the list above, SET SHARE will remain in effect. To clear the condition, issue the SET NOSHARE command in SPIBILD at any point before the command you want it to affect.
Warning: Use of SET SHARE is advised only in situations where SPIBILD or FASTBILD processing must proceed, as in a batch job where subsequent steps depend on the success of the processing. Users who find their terminals hung usually have no idea what happened, since there is no message telling them the file is being processed. By pressing ATTN/BREAK twice in a row, they can cause session breaks and receive a WYLBUR command prompt, though they are no longer in SPIRES. Typing GO will return them to the place (and the waiting) where they were before.
Here is a sample SPIBILD session showing how you might use the SET SHARE command:
-? establish spires doc notes -System ATCHFILE error, code=S56 -Compute time: 0.019 seconds -Elapsed time: 0.311 seconds -Core usage: 0051/0939 -? explain S56 S56: The file is busy, not currently available. -?
At this point, you might suspect that SPIBILD is mistaken, because you are the only one who ever uses the file. But then you remember that you left a terminal logged on somewhere else, and that it has the subfile selected. That session won't be bothered at all if you "set share":
-? set share -? establish spires doc notes ...the processing of the file begins
SET REINDEX COMMAND, IN SPIBILD
The SET REINDEX command in SPIBILD is used to establish an environment whereby SPIBILD will index records from the DEFQ with special processing. SET REINDEX remains in effect for the duration of the SPIBILD session, or until turned off by a SET NOREINDEX command. The command may not be abbreviated.
SET REINDEX is used primarily for the following purposes:
1) PROCESSing of restored DEFQ's during disaster recovery. That is when a full file restore needs to be done and then saved copies of the DEFQ's need to be applied.
2) Whenever it would normally be required to REMOVE and then re-ADD a set of goal records to force indexes to be up-to-date. That usually occurs when goal records have been ADDed and indexing did not occur because of a flaw in the PASSPROC rules. Once corrected, simply update the goal records (DEFQ updates) and use the SET REINDEX command to force the PROCESS command to reindex all the DEFQ records.
SET REINDEX only affects records being passed from the DEFQ. Therefore, it does not affect LOADed records, or BATCH input that does not go to the DEFQ. SET REINDEX affects the passing of DEFQ records (ADD, UPD, REM) in the following ways:
1) All indexes are processed, even those declared IMMEDIATE. SET REINDEX also ignores any SET NOINDEX list.
2) Goal records which belong to a REMOVED record-type and which pass to Immediate Indexes and are marked as ADDed (or DEQueue of an ADDed record) in the DEFQ will check the Tree. If the record exists in the Tree, the DEFQ entry is ignored. If not found in the Tree, then the RESidual copy of the record is passed for index deletion, and the ADDed record is then passed for index insertion.
3) Insertion/deletion pairs in the pass stack that normally would be cancelled are retained. That means all the index records are re-examined and the insertion/deletion is done. There are two possible outcomes of this process:
a) If the index record does not have "large segments", then the
index pointer is inserted into the index record if it didn't
already exist there. The index record is only rewritten if the
pointer had to be inserted.
b) If the index record does have "large segments", then the index
pointer is inserted into the top segment if it didn't already
exist there, and then the index pointer is deleted from any
secondary segment if it exists in such a segment. The index
segments are only rewritten if they are changed. The effect is
to guarantee that the pointer exists in the top segment.
An UPDATEd record in the DEFQ is what normally creates an "insertion/deletion pair", which is a pair of pass stack entries that match, but one comes from passing of the old version of a record (the deletion), and the other comes from passing of the new version of that same record (the insertion). For example, if TITLE = THE DOG; was in the old record, and TITLE = THE CAT; is in the new (updated) record, then the word THE creates an insertion/deletion pair. DOG and CAT create normal deletion and insertion pass stack entries which do not cancel, and would be applied to indexes regardless of REINDEX.
SET REINDEX
The command has no options, and may not be abbreviated.
The SET SKIP command tells SPIBILD how many lines or records in the input data set to skip before adding them to the subfile.
Command: SET SKIP
Available in: SPIBILD
Available to: file owner
users with Process access to the file
Related commands: INPUT BATCH, INPUT ADD, INPUT ADDUPDATE,
INPUT ADDMERGE, INPUT MERGE, LOAD
Recovery commands: EXIT, SPIBILD
Procedures: SPIBILD Recovery to Resume Input (2.3.6)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
LOAD procedure in SPIBILD (2.3.3)
INPUT MERGE procedure in SPIBILD (2.4.1)
SET SKIP n
Warning: If you are in WYLBUR mode, be sure that "n" is at least two digits (e.g., instead of "1", type "01"), or precede the command with an exclamation point; otherwise, WYLBUR will interpret the command as an instruction to set WYLBUR's skip character and will not pass it to SPIBILD.
The SET SKIP command is most frequently used in recovery situations when a LOAD or BATCH command in SPIBILD didn't complete properly. The number "n" comes from the SHOW RECORD RECOVERY command display, which is available after one of these commands fails to finish. [See 2.3.6.]
It's best to recover the file with the RECOVER/PROCESS procedure in SPIBILD [See 2.1.4.2.] after which you resume the procedure you were following, issuing the SET SKIP command prior to the processing command:
-? set skip 5746 -? use almanac.input -? establish almanac -? input batch ... the processing resumes ...
The SET STACK SIZE command lets you raise the amount of space available in the pass stack for use by SPIBILD or FASTBILD.
Command: SET STACK SIZE
Available in: SPIBILD, FASTBILD
Available to: file owner
users with Process access to the file
Related commands: none
Recovery commands: SET STACK SIZE
Procedures: any SPIBILD or FASTBILD procedure using indexes
SET STACK SIZE nK
The SET STACK SIZE command is used primarily in problem situations. Its typical use is when a record generates so many index entries that they can't fit into the pass stack, in effect getting the record stuck halfway between defq and tree. (The evidence for this problem is the dreaded "Passing: 0 records" message in SPIBILD or FASTBILD.) When you raise the stack size before starting the SPIBILD processing command again, the record will make it through.
The larger the pass stack size, the more efficient passing usually becomes, although this is not generally a significant concern. If two different records in two pass stacks pass to the same index record, then the index record must be fetched, updated and put back twice; but if both records are in the same pass stack, then the index record is handled only once, meaning fewer IOs.
There is very little point in reducing the pass stack size below the default of 256K, unless you get an S469 error. That indicates too many records are being placed in the pass stack, and the index records are growing too large. In such a case, you may wish to try reducing the pass stack size to limit the number of goal records entering the stack. This gives the index records a chance to grow and split.
SET STACK SIZE is not available in SPIRES; hence immediate indexing is limited to the default 256K size for a pass stack. If you get an S469 error passing to an index in SPIRES, you will be forced to use SPIBILD with a reduced stack size.
The SET SUBFILE DISABLE command prevents users from either using or selecting any subfiles of the file involved, or from creating transactions in that file, or from processing the file. It can also be used to disable duplicate deferred-queues. The SET SUBFILE ENABLE command reverses the effect.
Command: SET SUBFILE DISABLE
SET SUBFILE ENABLE
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: SET NOJOBGEN
Recovery commands: SET SUBFILE ENABLE, SET SUBFILE DISABLE
Procedures: Disabling a file temporarily (5.5)
SET SUBFILE DISABLE FOR {ATTACH|UPDATE|ACCESS|PROCESS|DUPLICATE}
SET SUBFILE ENABLE FOR {ATTACH|UPDATE|ACCESS|PROCESS|DUPLICATE}
The SET SUBFILE DISABLE command is used in emergency situations where it is critical to block users from selecting or updating subfiles of the file as quickly as possible. The SET SUBFILE ENABLE command reverses the effect of SET SUBFILE DISABLE, re-establishing normal conditions for the file.
Note that SET SUBFILE DISABLE FOR ATTACH will not affect users who currently have subfiles of the file selected; SET SUBFILE DISABLE FOR UPDATE will not affect users currently in the midst of a transaction (e.g., entering a record using the $PROMPT format). SET SUBFILE DISABLE FOR ACCESS will affect users in both those situations.
None of those three flavors (ATTACH, UPDATE, ACCESS) affect the file owner, who may continue to select the subfile and create transactions. Users with Master access to the file are also exempt.
On the other hand, disabling the subfile for PROCESS or DUPLICATE affects all users, including the file owner and users with Master access to the file.
Though you issue the command when you have a subfile selected, remember that the command affects all subfiles of the file:
-> select albums -> set subfile disable for access ->
When you are ready to allow users back into the file:
-> select albums -> set subfile enable for access ->
You can check the status of a file (if you can attach it or select one of its subfiles) with the "SHOW EVAL $DEFQTEST(code)" command, where "code" is DISABLED, PROCESS, WRITE and DUPLICATE. Except for DUPLICATE, if the value returned is "0" then the particular "disabling" state is NOT in effect. In other words, if SHOW EVAL $DEFQTEST(DISABLED) returns 0, then the file is not disabled; if SHOW EVAL $DEFQTEST(PROCESS) returns 0, then the file is not disabled for processing. For details about how to use the codes and interpret the results, EXPLAIN $DEFQTEST FUNCTION, or see the manual "SPIRES Protocols".
The SET TCOMMENT command contains a string expression to be put into the transaction log for the subfile transaction that follows.
Command: SET TCOMMENT
Available in: SPIRES
Available to: anyone
Related commands: record transaction commands, such as
ADD, UPDATE, REMOVE, DEQUEUE, MERGE ...
Recovery commands: none
Procedures: Using the $TRANSACTIONS Record-Type (4.2.3.3)
SET TCOMMENT string
SET TCOMMENT is also available as a Uproc in SPIRES formats.
The SET TCOMMENT command sets the value of the $TCOMMENT variable, which is passed to the TCOMMENT element in a $TRANSACTIONS record when the next subfile transaction occurs. Thus you can put data of your own choice into the $TRANSACTIONS records. An application might, for example, need to keep special accounting data for each transaction. The value in the system string variable $TCOMMENT will be stored with the transactions data and can be found in the $TRANSACTIONS record for the given transaction as the element TCOMMENT.
The variable $TCOMMENT is cleared after each subfile transaction and when a CLEAR SELECT command is issued for the primary subfile. Hence it must specifically be set for each transaction, rather than one time for many transactions.
Like $ASK and $PROMPT, $TCOMMENT is a global variable with only a single occurrence. There is not a separate occurrence for each subfile path if path processing is used. (Also like $ASK and $PROMPT, $TCOMMENT can be used to pass values back and forth between formats and protocols.)
-> set tcomment = $Version -> add
The SET TCOMMENT command sets $TCOMMENT to the current value of the $VERSION variable. When SPIRES executes the ADD command, the value of $TCOMMENT is passed to the $TRANSACTIONS record for the added record.
The SHOW ARCHIVE command displays the subfile log entries stored in your SYSTEM.ULOG data set.
Command: SHOW ARCHIVE Available in: SPIRES Available to: file owner Related commands: SHOW FILE LOG Recovery commands: not applicable Procedures: Handling the File Log (4.3.3)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW ARCHIVE [RELEASE]
The SHOW ARCHIVE command shows you data stored in the ORVYL data set SYSTEM.ULOG, which is created when any of your files contain the STATISTICS = 2 statement and the file is processed. SHOW ARCHIVE displays the data in the same format as the SHOW FILE LOG command, but contains log data for all your files that archive log data, all mixed together.
Each line of the log data represents one log record. The type code, in columns 2 and 3, specifies the type of activity that caused the record to be written:
- C8: SELECT commands
- C9: COMMANDS issued by the user
- CA: ASK prompts and responses
- CB: WITHIN LOG prefixed commands
- E8: CLEAR SELECT commands that de-select the subfile
- EA: SELECT CHARGE to the user for elapsed time
- EB: DISPLAY CHARGE to the user for records seen
- EC: MINIMUM CHARGE to the user
Details about the log data appear earlier in this manual. [See 4.3.4.]
Here is a sample session showing the use of this command:
-> show archive C8 08:35:20 01/12/87 GQ.DOC2 $0.00 $0.00 $0.00 FISH C9 08:35:22 01/12/87 GQ.DOC2 DIS smallmouth buffalo C9 08:36:10 01/12/87 GQ.DOC2 ADD C9 08:40:28 01/12/87 GQ.DOC2 DIS longear sunfish C9 08:40:30 01/12/87 GQ.DOC2 DIS black crappie C9 08:41:00 01/12/87 GQ.DOC2 ADDUPD C9 08:41:04 01/12/87 GQ.DOC2 /DISPLAY $KEY C9 08:53:42 01/12/87 GQ.DOC2 FIND eating = good C9 08:53:46 01/12/87 GQ.DOC2 TYPE E8 08:55:26 01/12/87 GQ.DOC2 19 00:20:06 08:25:20 FISH (etc.)
The SHOW AUTOGEN command tells you whether AUTOGEN is set for the file of the currently selected subfile.
Command: SHOW AUTOGEN Available in: SPIRES Available to: anyone who can select the subfile Related commands: SET AUTOGEN, SET JOBGEN Recovery commands: not applicable Procedures: None
SHOW AUTOGEN
There are no options. Unlike most SHOW commands, SHOW AUTOGEN will ignore any IN ACTIVE prefix added to it, displaying the result on the terminal screen anyway.
The SHOW AUTOGEN command tells whether AUTOGEN or NOAUTOGEN (i.e., overnight SPIBILD processing) is set for the file containing the currently selected subfile. [See 2.2.1.] In addition, if the NO JOBGEN flag is set, that information is also displayed.
SHOW AUTOGEN does not show the normal, file-definition-specified setting for AUTOGEN; it simply shows the current value, which may have been changed since the last time the file was processed via a SET AUTOGEN or SET NOAUTOGEN command.
Here is a sample session showing the use of this command:
-> select united states -> show autogen -NOAUTOGEN in effect -> set autogen -> show autogen -AUTOGEN in effect ->
After discovering that NOAUTOGEN was in effect, the user decided to set overnight processing by issuing the SET AUTOGEN command.
The SHOW BLOCKS command tells you how many ORVYL file blocks are being used and how many are allowed to be used by your account.
Command: SHOW BLOCKS Available in: SPIRES, SPIBILD, WYLBUR Available to: anyone Related commands: SHOW FILES, COPY FILE Recovery commands: not applicable Procedures: Determining the physical size of a file (3.2.1)
SHOW BLOCKS [ACCOUNT gg.uuu|[FILE] dsname]
The command with no options shows the number of 2048-byte ORVYL blocks being used by your account, and how many your account is allowed. The difference between them is the number of remaining blocks available to you. (Note: You are charged only for the number of blocks used, not for the number you are allowed.) [See 3.2.1.]
-> show blocks 742 Blocks accounted, 2000 Blocks allowed -> show blocks donors.defq DONORS.DEFQ -- 2 BLKS (F: 0, L: 1) ->
The SHOW ERROR COUNTS command displays a summary of information concerning the number and kinds of errors that occurred during a SPIRES or SPIBILD session. The CLEAR ERROR COUNTS command resets the internal table in which the information is kept.
Command: SHOW ERROR COUNTS
CLEAR ERROR COUNTS
Available in: SPIRES, SPIBILD
Available to: any SPIRES user
Related commands: SET EXCEPTION
Recovery commands: none
Procedures: Exception File procedure (2.1.5.2)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW ERROR COUNTS CLEAR ERROR COUNTS
Syntax note: When using the CLEAR ERROR COUNTS command in a session where SET WYLBUR is in effect, you will need to prefix the command with an exclamation point (!) so that it is passed to SPIRES before being passed to ORVYL, which has a CLEAR ERROR command of its own.
The SHOW ERROR COUNTS command summarizes information about errors that have occurred during a SPIRES or SPIBILD session. The statistics show information about the error codes that would be normally put out in error messages.
Since the statistics are cumulative for the session, it may be useful to reset the error count table under some circumstances, which you do by issuing the CLEAR ERROR COUNTS command.
Here is an example of the SHOW ERROR COUNTS command at the end of a SPIBILD session where records were added to a subfile under INPUT BATCH:
-? show error counts
01/17/91 Error Counts Information
Error Code Count
122 (S) 10
192 (S) 3
E31 (W) 2
S261 (S) 2
162 (S) 1
-?
Notice that the errors that have occurred during the session are displayed in descending order by the number of times they occurred. The severity of the error appears in parentheses, "(S)" for Serious, and "(W)" for Warning.
The SHOW FILE ACTIVITY command displays information about past activity for each record-type in the file, as well as information about each record-type's base block within a data set.
Command: SHOW FILE ACTIVITY
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: FIX BASE, SHOW FILE STATUS
Recovery commands: none
Procedures: Rebalancing procedure (3.3.1)
Getting Basic Overall Statistics about
Transactions (4.2.2)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW FILE ACTIVITY
Syntax note: SHOW FILE ACTIVITY may be issued as SHOW SUBFILE ACTIVITY, but because its output relates to the file as a whole, SHOW FILE ACTIVITY is the preferred form.
The SHOW FILE ACTIVITY command summarizes updates that have occurred during SPIBILD processing since the beginning of the attached file's existence. In addition, it tells which data set contains each record-type of the file and tells the number of the base block for each record-type within that data set.
The data in the sample SHOW FILE ACTIVITY display below is described at the end.
01/07/1987 Activity Record of File GQ.JNK.BLOOD.DONORS
Last Compiled: 09/18/1984 at 17:23:24
Record Base Blocks Adds Deletes Updates Max Rec
Type Block Used -DS Length
REC01 -Slot 0 1 -1 59 3 26 446
ZIN02 0 7 -2 54 6 6 62
ZIN03 1 Comb -2 22 0 25 68
ZIN04 2 Comb -2 2 0 27 160
ZIN05 3 Comb -2 8 0 36 49
ZIN06 4 Comb -2 84 3 16 43
ZIN07 5 Comb -2 51 4 58 131
ZIN08 6 Comb -2 12 2 24 129
Residual 2 8 63 3 49 447
Def Queue 1 2
An explanation of the items under each heading follows:
Note: The numbers of adds, deletes and updates maintained by the system may not be precisely accurate since there is no attempt to maintain their integrity in case of system failure -- the expense in doing this is deemed excessive in comparison to their value. However, in all likelihood, the numbers will be accurate.
The SHOW FILE LIMITS command displays the size of data sets and the depths of trees in the file, marking any close to limits.
Command: SHOW FILE LIMITS
Available in: SPIRES
Available to: file owner
users with Master access to the file
Recovery commands: not applicable
[IN ACTIVE [CLEAR|CONTINUE]] SHOW FILE LIMITS
Syntax Note: SHOW FILE LIMITS may be issued as SHOW SUBFILE LIMITS, but because its output relates to the file as a whole, SHOW FILE LIMITS is the preferred form.
The SHOW FILE LIMITS command shows you data set limits for the attached file, showing you how close the individual data sets are getting to their maximum limits. The limits represent the limits of the specific data set as defined; rebalancing a data set can readjust or redefine the data base so that either the limits are raised or the data sets back away from them.
All data sets are shown, including REC1 through RECF (as they exist for the file), DEFQ, MSTR, RES and RESn (where "n" is an integer, when multiple residuals exist). If a data set is approaching one of the two limits shown (size or tree depth), it is flagged with three asterisks (***).
-? sel wcat
-? show file limits
02/16/1998 Limits Record of File GQ.WCK.WCAT
Last Compiled: 07/03/1995 at 14:45:58
Data Set Max Blocks - Blocks Used - Percent Used Tree Depth
REC1 434280 17 0.0 2
REC2 434280 413000 *** 95.09 8
REC3 434280 5 0.0 0
REC4 434280 38 0.0 2
REC5 434280 38 0.0 2
REC6 434280 88 0.0 2
REC7 434280 22 0.0 2
REC8 434280 5 0.0 1
REC9 434280 23 0.0 2
RECA 434280 2 0.0 0
RECB 434280 5 0.0 2
RES 425984 131 0.0 --
MSTR 256 122 47.65 --
DEFQ 65536 2 0.0
File data set block size is 2048 bytes.
*** Warning: Data Set limits approaching maximum value.
The columns are:
- Max Blocks -- maximum number of blocks in the data set
- Blocks Used -- the number of blocks used (derived from the "Next Block" number)
- Percent Used -- Blocks Used divided by Max Blocks. If this value is greater than 94%, warning characters (***) are displayed next to the value.
- Tree Depth -- The number of levels of the deepest tree within the data set. If this value is 12 or more, warning characters (***) are displayed.
The command also shows the block size for the data sets, which will become more relevant as 4096-byte blocks become available and employed in SPIRES data bases.
The SHOW FILE LOG command displays the file log entries in the deferred queue.
Command: SHOW FILE LOG
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: SHOW ARCHIVE
Recovery commands: not applicable
Procedures using it: Handling the File Log (4.3.3)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW FILE LOG
Syntax Note: SHOW FILE LOG may be issued as SHOW SUBFILE LOG, but because its output relates to the file as a whole, SHOW FILE LOG is the preferred form.
The SHOW FILE LOG command shows you data from the deferred queue gathered from sessions where some level of subfile logging is in effect. When the file is processed, the log data may be archived, and then it is discarded. [See 4.3.1.] to learn how to archive it, if desired.
Each line of the log data represents one log record. The type code, in columns 2 and 3, specifies the type of activity that caused the record to be written:
- C8: SELECT commands
- C9: COMMANDS issued by the user
- CA: ASK prompts and responses
- CB: WITHIN LOG prefixed commands
- E8: CLEAR SELECT commands that de-select the subfile
- EA: SELECT CHARGE to the user for elapsed time
- EB: DISPLAY CHARGE to the user for records seen
- EC: MINIMUM CHARGE to the user
Details about the log data appear earlier in this manual. [See 4.3.4.]
Here is a sample session showing the use of this command:
-> show file log C8 16:25:20 01/12/87 GQ.JNK $0.00 $0.00 $0.00 DATES C9 16:25:22 01/12/87 GQ.JNK dis 634 C9 16:26:10 01/12/87 GQ.JNK add C9 16:30:08 01/12/87 GQ.JNK addupd C9 16:30:28 01/12/87 GQ.JNK dis 312 C9 16:30:30 01/12/87 GQ.JNK dis 322 C9 16:31:00 01/12/87 GQ.JNK addupd C9 16:31:04 01/12/87 GQ.JNK /display $key C9 16:43:42 01/12/87 GQ.JNK find interest lanolin C9 16:43:46 01/12/87 GQ.JNK type E8 16:45:26 01/12/87 GQ.JNK 51 00:20:06 16:25:20 DATES (etc.)
The command SHOW FILE STATUS displays a summary of information about file growth gathered by the most recent STATUS command.
Command: SHOW FILE STATUS
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: STATUS
Recovery commands: not applicable
Procedures: Getting Basic Overall Statistics about
Transactions (4.2.2)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW FILE STATUS
Syntax note: SHOW FILE STATUS may be issued as SHOW SUBFILE STATUS, but because its output refers to the contents of the file as a whole, SHOW FILE STATUS is the "more correct" form.
SHOW FILE STATUS provides a capsule summary of some of the storage and retrieval statistics for the attached file. Most of its contents come from statistics gathered by the most recent STATUS command issued for each data set. If no STATUS command has ever been issued for a data set, most of its statistics will be "0".
The most useful information provided by this command is probably the date a STATUS command was last issued for each record-type. Also useful may be the date and time the file definition for the file was last compiled, shown at the start of the display, and the number of records in each record-type. [The number of records is maintained independently of the STATUS command, so that column will have values whether or not a STATUS command has been issued. The number comes from data in the SHOW FILE ACTIVITY command, specifically the difference between the number of adds and the number of removes for each record-type. Note that its accuracy is not absolutely guaranteed.]
But the SHOW FILE STATUS command isn't as good a source for information to use in making decisions about the file (e.g., rebalancing a record-type, rebuilding the available space chains, etc.) as the data from the STATUS command itself, which is more complete and up-to-date. Still, as a summary of that information, the SHOW FILE STATUS command may be useful to you.
The data in the sample SHOW FILE STATUS display below is explained at the end.
01/06/1987 Status Record of File GQ.JNK.BLOOD.DONORS
Last Compiled: 09/18/1984 at 17:23:24
Record- Blocks Space Number Accesses Tree Date
Type Used Utilized Records / Record Depth
REC01 -Slot 1 24.224 56 2.000 0 01/06/87
ZIN02 1 95.264 48 1.000 1 01/06/87
ZIN03 1 33.456 22 1.000 1 01/06/87
ZIN04 1 5.424 2 1.500 1 01/06/87
ZIN05 1 17.296 8 1.000 1 01/06/87
ZIN06 1 80.432 81 1.000 1 01/06/87
ZIN07 1 66.320 47 1.021 1 01/06/87
ZIN08 1 12.560 10 1.100 1 01/06/87
Residual 8 91.360 60 0.000 0 01/06/87
Def Queue 2
An explanation of the items under each heading follows:
The ORVYL command SHOW FILES lists the names of the ORVYL data sets stored on your account or the named account.
Command: SHOW FILES Available in: SPIRES, SPIBILD, WYLBUR Available to: anyone Related commands: COPY FILE, SHOW BLOCKS Recovery commands: not applicable Procedures: Determining the Physical Size of a File (3.2.1)
SHOW FILES [ACCOUNT gg.uuu] [{FROM|LIKE} name]
[[BLOCKS] [FORMAT] [DATED]|ALL]
For example, from account GQ.DOC:
-> show files like orv.gq.doc.restaurant blocks ORVYL File System ORV.GQ.DOC.RESTAURANT.CKPT -- 43 BLKS (F: 0, L: 42) ORV.GQ.DOC.RESTAURANT.DEFQ -- 50 BLKS (F: 0, L: 49) ORV.GQ.DOC.RESTAURANT.MSTR -- 5 BLKS (F: 0, L: 4) ORV.GQ.DOC.RESTAURANT.REC1 -- 4 BLKS (F: 0, L: 3) ORV.GQ.DOC.RESTAURANT.REC2 -- 35 BLKS (F: 0, L: 34) ORV.GQ.DOC.RESTAURANT.RES -- 108 BLKS (F: 0, L: 107) ->
You can determine the size of a SPIRES file in blocks by adding the number of blocks shown. [Technically speaking, you must also add one block for each 660 blocks of each data set; these blocks are not part of the data set per se, but are stored in ORVYL's index blocks and are charged to you. Non-IXR data sets do not have these blocks. [EXPLAIN GENERATE NONIXR COMMAND for more information, or see its reference earlier in Part 6.]]
The command SHOW RECORD INFORMATION reports how much storage space is used for a specific record, including information about any large record segments.
Command: SHOW RECORD INFORMATION
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: STATUS, SHOW RECORD OVERHEAD
Recovery commands: not applicable
Procedures: Determining Storage Space for Individual
Records (3.2.2)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW RECORD INFORMATION key or, under Global FOR: [IN ACTIVE [CLEAR|CONTINUE]] ... SHOW RECORD INFORMATION [FIRST|*|NEXT|n|LAST|REST|ALL]
The SHOW RECORD INFORMATION command will show you statistics about:
- the length in bytes of the stored record or records;
- the locator for each large record segment of the record, if it exists; and
- the length in bytes of each large record segment.
Like SHOW RECORD OVERHEAD, this command shows the record length including the "overhead" data stored with the record, which describes how many occurrences of an element are stored, how long each value is, which optional elements occur, etc.
Here is part of a SPIRES session using the SHOW RECORD INFORMATION command to see information about an index record's large record segments:
-> show record information COLORADO
Record info for Key: 'COLORADO'
Record Length: 1935 bytes
1 Large record info for locator: 2678234-1
Record Length: 14080 bytes
2 Large record info for locator: 2317299-1
Record Length: 14020 bytes
3 Large record info for locator: 1410120-1
Record Length: 13305 bytes
Totals for record key: 4 Parts. 43340 bytes
For records without large record segments, only the first two lines of the display above are shown.
Users who do not have See access to the file but who would like to see the storage size of a record in the selected subfile can use the $RECINFO function to get that value.
The command SHOW RECORD OVERHEAD reports how much storage space is used for a specific record, a sampling of records or all records in a subfile.
Command: SHOW RECORD OVERHEAD
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: STATUS
Recovery commands: not applicable
Procedures: Determining Storage Space for Individual
Records (3.2.2)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW RECORD OVERHEAD [key|ALL]
The SHOW RECORD OVERHEAD command will show you statistics about:
- the length in bytes of the stored record or records;
- the percentage of that space being used to store "overhead" data (see below); and
- the number of records examined by the command.
The overhead this command refers to is that stored in the record itself that precedes element values, the "header" data, which describes how many occurrences of an element are stored, how long each value is, which optional elements occur, etc. To a large extent, this is overhead prescribed by the file owner in the file definition -- Optional elements in general have slightly more "overhead" than Required ones, and Fixed elements require none. [See the manual "SPIRES File Definition", section B.6.2, for further information on the headers.]
Here is part of a SPIRES session using the SHOW RECORD OVERHEAD command:
-> select restaurant
-> show record overhead
-Overhead: 11.796 percent
26 record(s) scanned; total length 11563 bytes
-> show record overhead 300
-Overhead: 16.949 percent
1 record(s) scanned; total length 177 bytes
Users who do not have See access to the file but who would like to see the storage size of a record in the selected subfile can use the $RECINFO function to get that value.
If a BATCH or LOAD command to add records to a file ends abnormally in SPIBILD, the SHOW RECORD RECOVERY command can tell you where to resume the input.
Command: SHOW RECORD RECOVERY
Available in: SPIRES
Available to: file owner
users with Process access to the file
Related commands: SET SKIP
Recovery commands: not applicable
Procedures: SPIBILD Recovery to Resume Input (2.3.6)
INPUT BATCH procedure in SPIBILD (2.3.1)
INPUT ADD procedure in SPIBILD (2.3.2)
LOAD procedure in SPIBILD (2.3.3)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW RECORD RECOVERY
SHOW RECORD RECOVERY displays information useful for recovery when a LOAD, INPUT BATCH, INPUT ADD, INPUT ADDUPDATE or INPUT ADDMERGE command fails to complete in SPIBILD. Specifically, it will tell you the number of lines (for the INPUT commands) or the number of records (for LOAD) successfully input into the file before the failure occurred. The command only has information to show you if index data was being updated by the LOAD or INPUT command that failed; if no indexes were involved, the SHOW RECORD RECOVERY command will simply return a command prompt.
As input data for new records is batched into SPIBILD, SPIBILD keeps a count of the number of input lines or number of records read, storing the current number away each time the pass stack is filled and processed. If the command terminates abnormally, you can get the number from the SHOW RECORD RECOVERY command and use it in a SET SKIP command prior to restarting the procedure, telling SPIBILD to resume the processing after skipping the successful data.
If no SET SKIP information appears, then you should not issue the SET SKIP command when you resume inputting the data. That means that none of the input data was completely processed so none should be skipped.
The SHOW RECORD RECOVERY display also shows the keys of records in the pass stack being built (or processed) when the SPIBILD command failed. During recovery, SPIBILD will discard those records and the data from them in the pass stack, because it doesn't know how far it got in indexing that data -- hence, it will unindex any of that data too. Generally speaking, you don't need to worry about those records because the data for them is in the input data after the starting point specified by the SET SKIP value -- that data will be processed again if you use the SET SKIP value in the procedure. There is one aspect worth noting -- if the subfile has slot keys, then the slot numbers of the records shown by the SHOW RECORD RECOVERY command will be lost, since those records will be discarded. When the records are added again, they will get new, higher numbers.
Here is an example of the display from SHOW RECORD RECOVERY:
-> select almanac -SET SKIP 5746 -Goal record-type = 1, Key count = 7 2874 2875 2876 2877 2878 2879 2980 ->
The number of record keys shown depends on the number of added records in the pass stack at the time of the failure. If you were to return to SPIRES and select the subfile at this point, you could use the DISPLAY command to see each of these records; some of their data may even be indexed. But remember that when recovery takes place (i.e., the next time the file is processed), those records will be discarded. [See 2.3.6.]
The SHOW SUBFILE BLOCK command displays information about the contents of a particular block within the data set containing the the currently attached record-type, or within the deferred queue.
Command: SHOW SUBFILE BLOCK
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: SHOW FILE ACTIVITY
Recovery commands: not applicable
[IN ACTIVE [CLEAR|CONTINUE]] SHOW SUBFILE [DEFQ|RESIDUAL]
BLOCK {n|BASE|NODE key|LCTR locator|KEY key-value}
or
[IN ACTIVE [CLEAR|CONTINUE]] SHOW SUBFILE BLOCK LINKS key
Syntax note: SHOW SUBFILE BLOCK may be issued as SHOW FILE BLOCK, but because its output generally relates to the attached goal record-type, SHOW SUBFILE BLOCK is the preferred form.
The SHOW SUBFILE BLOCK command displays the following information about the requested block:
- the record-type to which the block belongs;
- whether the block is the base block for a record-type, and if not, which block is the next level higher in the tree (i.e., towards the base or root);
- information about the nodes (the records) in the block;
For the deferred queue and the residual, the block may contain data for several record-types.
The following is a sample of the output from this command:
Block 0 of record REC1 - base block No. Branch Disp Len Lctr Node value 1 398 1385 19 29520-1 AC.TUR.AGE 2 72 1043 20 6679-1 AD.MIT.YUR.MISTAKES 3 365 1215 21 39221-1 AL.LOW.SIXWKS4.DELIVRY 4 76 1140 28 9344-2 AP.PLY.YOURSELF 5 57 18 22 * AR.YOU.RIGHT.FORME ... 57 424 1480 28 20737-1 UR.WAT.UEAT 58 62 1536 268
The heading identifies the block being examined, tells which record-type it belongs to, and either identifies the block as a base block or tells from which block this block can be accessed. The entries in the block are numbered at the left. The remainder of the data is as follows:
The SHOW SUBFILE TRANSACTIONS command displays information about the transactions in the deferred queue for the selected subfile.
Command: SHOW SUBFILE TRANSACTIONS
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: FOR TRANSACTIONS
Recovery commands: not applicable
Procedures: Getting Basic Transaction Information (4.2.3.1)
[IN ACTIVE [CLEAR|CONTINUE]] SHOW SUBFILE TRANSACTIONS [key]
The SHOW SUBFILE TRANSACTIONS command shows you data about all the records in the deferred queue for the selected subfile. When the file is processed, the transactions data is discarded. [See 4.2.3.3 to learn how to save it if desired.]
Unless you use the "key" option, the transactions data is displayed in chronological order, the order in which they occurred. Multiple transactions for a single record will appear as several entries.
Transactions that are part of transaction groups are included, as well as a separate entry for the group "commit" or "decommit" as a whole.
Here is a sample of the output from the SHOW SUBFILE TRANSACTIONS command, followed by an explanation for each field:
01/13/1990 Transaction Log of File GQ.DOC.ERRATA
for Record-type REC01
File last processed: 01/13/1990 at 02:37:38
Date Time Account Id Type Command Grp Key Value
01/12/90 16:26:10 GQ.JNK ADD Add 634
01/12/90 16:30:08 GQ.JNK ADD Update 634
01/12/90 16:31:01 GQ.JNK DEQ Remove 634
01/12/90 16:45:49 GQ.JPR S14 UPD Merge 629
01/12/90 16:46:19 GQ.JPR S14 ADD Add 635
01/12/90 16:56:03 GQ.JLS CU13 UPD Update 628
01/12/90 16:56:07 GQ.JLS CU13 REM Remove 628
01/12/90 17:06:46 GQ.JLS CU13 UPD Unqueue 628
01/12/90 17:07:39 GQ.JLS CU13 DEQ Dequeue 628
01/12/90 17:45:50 GQ.JNK ADD Add 1 636
01/12/90 17:46:32 GQ.JNK ADD Add 1 637
01/12/90 17:47:20 GQ.JNK Decommit 1
01/12/90 18:16:57 GQ.JNK ADD Add 2 636
01/12/90 18:22:45 GQ.JNK ADD Add 2 637
01/12/90 18:25:12 GQ.JNK Commit 2
The heading includes the date you issued the command, the name of the file and the name of the goal record-type. Here are descriptions of the other fields:
- ADD: Add the record to the tree.
- UPD: Replace the tree copy of the record having this key with this deferred-queue copy.
- REM: Remove the record from the tree.
- DEQ: Ignore all earlier transactions for this key.
Notice in the sample display that the first "Add" of record 636 was part of a transaction group that was subsequently decommitted. When examining the display for all the records (i.e., the command without the "key" option), be sure to look for a "Commit" or "Decommit" command to tell you whether records processed in a transaction group are really there.
If you issue the command with the "key" option, then a record in a transaction group will only appear in the display if the group was committed. It would not appear if the group were decommitted. For instance, using the example displayed above, the command SHOW SUBFILE TRANSACTIONS 637 would display only the next-to-last line shown above.
The SHOW FILE TREES command displays information about the trees in the various data sets of the file.
Command: SHOW FILE TREES
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: STATUS
Recovery commands: not applicable
[IN ACTIVE [CLEAR|CONTINUE] SHOW FILE TREES
Syntax note: SHOW FILE TREES may be issued as SHOW SUBFILE TREES, but because its output describes the entire file, SHOW FILE TREES is the preferred form.
The SHOW FILE TREES command displays the following information about the attached file: a list of all record-type trees, their base block numbers, the data set they appear in, and overflow tree data.
The following is a sample of the output from this command:
-> show file tree
07/19/1992 Record-type Layout of File GQ.DOC.ERRATA
Last Compiled: 07/19/1992 at 00:14:22
Record-type Data Set Base Block Overflow Pfx Start-key
REC01 REC1 0
REC02 REC2 0
... ... 1 f
... ... 2 g
... ... 3 m
... ... 4 q
REC03 REC2 1
ZIN04 REC3 0
ZIN05 REC3 1
ZIN06 REC3 2
ZIN07 REC3 3
ZIN08 REC3 4
All the record-types in the file are listed, along with the data set they are in and the base block within that data set in which the record-type begins. The extra lines for REC02 above indicate multiple trees created by the TREE-DATA structure in the file definition, used to control tree depth. The START-KEY value for each of those trees is shown in the far right column; if there were a TREE-PREFIX value, that would be shown in the column labelled "Pfx". [See 5.6.]
The SPIBILD command invokes the SPIBILD program.
Command: SPIBILD Available in: SPIRES, SPIBILD, WYLBUR Available to: any user Related commands: PROCESS, BATCH, ESTABLISH Recovery commands: EXIT
SPIBILD [(parameter options)]
- QUIET or Q. SPIBILD will suppress the "Welcome to SPIBILD" message when this option is specified.
- WYLBUR or W. SPIBILD will establish WYLBUR mode when this option is specified. If the SPIBILD command appears in a WYLBUR Exec file, you must include the "W" parameter if you want the Exec file to continue executing. [EXPLAIN SET WYLBUR command for information about WYLBUR mode.]
- ATTENTION, ATTN or A. If this option is specified, then pressing the ATTN/BREAK key multiple times while a SPIBILD command is being processed will not cause a session break. [EXPLAIN SESSION BREAK for more information.]
- SINFO or S. If this option is specified, SPIBILD will establish SINFO mode, which means that the names of all ORVYL data sets SPIBILD uses during the session will be displayed as they are attached. [EXPLAIN SET SINFO command for information about SINFO mode.]
Syntax note: SPIBILD supersedes the CALL SPIBILD command, which may still be issued if desired. However, specify only the 1-character abbreviations of the parameter options, one against the other, e.g., "CALL SPIBILD (QW)".
SPIBILD invokes the ORVYL program SPIBILD, which handles many of the file-building and file-maintenance tasks for SPIRES. You cannot issue SPIBILD commands unless you are in the SPIBILD program.
Command> spibild -Welcome to SPIBILD 87.06 -?
Now you can issue SPIBILD commands, such as PROCESS or ESTABLISH.
Like SPIRES, SPIBILD displays its release number in the welcome message to help you identify which release of the program you are using. The first two digits represent the year, and the second two represent the month of the release ("June 1987" in the sample above). Which release of SPIBILD you are using is particularly useful to your consultant if you are reporting problems.
The STATUS command collects and displays status information about the contents of a file's data sets
Command: STATUS
Available in: SPIRES
Available to: file owner
users with See access to the file
Related commands: ATTACH, SHOW FILE STATUS, AVSPREC
Recovery commands: none
Procedures using it: STATUS procedure (3.2.3)
AVSPREC procedure (3.4.3)
[IN ACTIVE [CLEAR|CONTINUE]] STATUS {ALL|RES}
[START=block-no] [AVSP] [NOUPDate]
or
[IN ACTIVE [CLEAR|CONTINUE]] STATUS dataset-name [NOUPDate]
- ALL the tree-structured data sets (except the DEFQ), plus the residual; or
- RES, the residual alone; or
- an individual tree-structured data set of the file: REC1 through REC9, or DEFQ.
The "dataset-name" and START options are primarily useful for extremely large files where a STATUS command could take many hours if it were to examine all the data sets.
The STATUS command is a file-maintenance command that insures the validity of the blocks (and the linkages between them) in various data sets of a file. It also gathers important data about the contents of those data sets, some of which it always reports to you in the command's output, and some of which is stored within the file for retrieval by the SHOW FILE STATUS command, unless blocked by the NOUPDate option.
The most useful data from the STATUS command concerns these areas:
- errors uncovered while validating the blocks [See 3.2.2.]
- balanced tree growth [See 3.3.]
- available space [See 3.4.]
An earlier section of this manual describes the STATUS procedure in detail, including how to examine and interpret the important statistics the command returns. [See 3.2.2.] Below is a sample set of the statistics from a STATUS command, with explanations of some of them.
In this example, the ALL option was used so that the data for all the data sets (except the DEFQ) would be available to examine. However, not all of them are shown and described; those that are similar to others shown are not included. Basically, there are four types of statistic groups shown:
- status information about the residual data set
- available space data for the residual (from AVSP option)
- status information about slot record-types
- status information about non-slot record-types
In the example below, the display is occasionally interrupted between major sections for commentary. Note that if SPIRES detects errors as it reads the blocks, error messages would interrupt the display for real.
Command> spires
-Welcome to SPIRES
-> attach orv.gq.jnk.trecordings
-> status all avsp
01/05/1987 Residual Data of File GQ.JNK.TRECORDINGS
Number of Blocks 845
Number of Entries 8234
Length of Entries 1567025
Unused Space 143255
Split Entries 75
Deleted Entries 339
Overflow Segments 83
1395 Entries of REMOVED Record ALBUM 802830 bytes
62 Overflow Segments of Record ALBUM
76 Overflow Entries of Record REC02 12006 bytes
309 Overflow Entries of Record REC03 61121 bytes
3 Overflow Segments of Record REC03
22 Overflow Entries of Record REC04 2424 bytes
...
1 Overflow Entries of Record XREC24 1786 bytes
Space Utilized 91.722 percent
Available Space Chain Data
Bytes Blocks Blocks Blocks on
Available (Actual) on Chain Wrong Chain
1984 1
1920 1
...
704 1 1
320 2 2
256 1 1
192 2 2
128 2 2
64 1 1
...
0 84 83 9
The AVSP data gives you an indication of how much empty space in the residual is available for storing records and how well SPIBILD is keeping track of it. Complete instructions on interpreting these statistics appears in an earlier chapter. [See 3.4.2.] For very large residuals, SPIRES may not have enough internal memory to track down all the chains to their final blocks. If it can't complete the work, SPIRES will precede the AVSP table with the message "Processing Table Overflow -- The following data is incomplete". What may be incomplete is the data about segment counts; the other data is correct and complete.
01/05/87 Record-type 1, Data Set REC1
Record ALBUM of File GQ.JNK.TRECORDINGS
Number of Blocks 6
Number of Slots 1396
Length of Slots 11168
Unused Space 976
Empty Slots 121
Deleted Slots 1
Average Accesses per Record 2.000
Space Utilized 92.057 percent
Because slot record-types store data in a different way than non-slot record-types (which are tree-structured), the status information shown for each type is different.
01/05/87 Record-type 2, Data Set REC2
Record REC02 of File GQ.JNK.TRECORDINGS
Non Terminal Terminal Total
Number of Blocks 1 27 28
Number of Nodes 26 784 810
Length of Nodes 1267 36943 38210
Unused Space 703 17705 18408
Split Nodes 1 75 76
Deleted Nodes 0 0 0
Level Nodes Blocks (Terminal)
1 26 1 0
2 784 27 27
Base Block 1
Average Accesses per Record 2.061
Space Utilized 67.898 percent
The first sets of numbers are split into columns representing "Non Terminal" and "Terminal" blocks. Non-terminal blocks are blocks of the tree that contain branch pointers pointing down to other blocks in the tree. Terminal blocks mark the end of a tree branch.
The next data shown may help you determine that the record-type needs rebalancing. It displays the tree growth almost graphically. For each level of the tree, starting with the top-most level (1), the number of nodes and the number of blocks containing those nodes is shown, followed by the number of blocks containing only terminal nodes. [See 3.3.]
The TESTDEFQ command works to restore the deferred queue to a consistent state.
Command: TESTDEFQ Available in: SPIRES Available to: file owner Related commands: RECOVER, PROCESS Recovery commands: RECOVER, PROCESS
TESTDEFQ
The TESTDEFQ command has no options.
When a subfile is selected and certain commands are issued (such as ADD or UPDATE), SPIRES verifies that SPIRES and ORVYL data regarding the deferred queue is consistent. If SPIRES detects the possibility of a problem, it automatically performs a recovery procedure called "testdefq". If the procedure fails, then an error message is displayed to the user, instructing him or her to contact a SPIRES consultant.
A file owner can verify that the deferred queue is intact by invoking the testdefq procedure explicitly, through the TESTDEFQ command. If the system doesn't report any error, then the deferred queue passed the testdefq test.
Since the recovery is attempted automatically when needed, the TESTDEFQ command is seldom issued explicitly.
The TESTDEFQ command applies to the file of the selected subfile:
-> select albums -> testdefq ->
If all goes well, as it almost invariably does, the only response will be the SPIRES prompt, as shown above. If a problem arises, try the RECOVER/PROCESS procedure to clear it up. [See 2.1.4.2.]
The ZAP DATA SET command discards all the records in the record-types of the specified data set of the file.
Command: ZAP DATA SET
Available in: SPIBILD, SPIRES
Available to: file owner
users with Master access to the file
Related commands: ZAP RECORD-TYPE, ZAP FILE
Recovery commands: none
Procedures: ZAP RECORD-TYPE procedure (2.5.2)
Remove an index procedure (2.7.2)
Rebuild an index procedure (2.7.3)
Remove a subfile procedure (2.8.2)
Rebuild a subfile procedure (2.8.3)
ZAP DATA SET RECn
Before issuing ZAP DATA SET, you must tell SPIBILD which file to use, using the ESTABLISH command. ESTABLISH processes the file's deferred queue, which must be empty for ZAP DATA SET to work. In SPIRES, you would SELECT any subfile of the file.
ZAP DATA SET erases all records in all record-types stored in the ORVYL data set. Even records removed from those record-types to the residual will be discarded. All but one block of the data set will be released; however, no residual blocks will be released.
Below is a sample SPIBILD session showing the use of ZAP DATA SET. Note that SPIBILD lists the record-types in the data set as they are zapped.
-? establish team statistics -Processing file: GI.PPR.STATS -Compute time: 0.120 seconds -Elapsed time: 1.916 seconds -Core usage: 0065/0933 -? zap data set rec5 -Record ZIN15 will be zapped -Record ZIN16 will be zapped -File detached -?
The ZAP DEFQ command discards all the contents of the deferred queue of the file containing the selected subfile.
Command: ZAP DEFQ
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: ZAP RECORD-TYPE
Recovery commands: none
Procedures: Handling the File Log (4.3.3)
ZAP DEFQ [NOWARN]
The ZAP DEFQ command discards everything in the file's deferred queue, including log and transaction data. Its effect is drastic and immediate; hence, SPIRES asks you to confirm the command before releasing the data, unless you include NOWARN on the command.
Here is an example demonstrating the command:
-> select almanac -> zap defq -Do you want to clear the DEFQ of GQ.JNK.ALMANAC? yes ->
If you change your mind when SPIRES prompts you to confirm, type NO (or N) or press the ATTN/BREAK key.
The ZAP FILE command erases the ORVYL data sets that comprise a SPIRES file.
Command: ZAP FILE
Available in: SPIRES
Available to: file owner
Related commands: COPY FILE
Recovery commands: none
Procedures: Rebuild a file procedure (2.9)
Destroy a file procedure (2.11)
ZAP FILE filename [NOWARN]
Issue the ZAP FILE command when you want to destroy a file you own, either because you are completely done with it or because you want to rebuild it from scratch. The command erases all the ORVYL data sets associated with the file (e.g., DEFQ, MSTR, RES). It has no effect on the file definition stored in the FILEDEF subfile.
Only the file owner can issue the ZAP FILE command for a file.
WARNING: This command completely destroys the contents of the file, and there is no simple procedure for recovering it, if recovery is possible at all. (Recovery depends on how you and your computer center back up your ORVYL data sets.) Once the command starts executing, your file is gone. NEVER treat the ZAP FILE command casually.
-> zap file gq.doc.blood.donors -Do you want to destroy file GQ.DOC.BLOOD.DONORS ? yes ->
The ZAP RECORD-TYPE command discards all the records in the specified record-type of the file.
Command: ZAP RECORD-TYPE
Available in: SPIBILD, SPIRES
Available to: file owner
users with Master access to the file
Related commands: ZAP DATA SET, ZAP FILE
Recovery commands: none
Procedures: ZAP RECORD-TYPE procedure (2.5.2)
Remove an index procedure (2.7.2)
Rebuild an index procedure (2.7.3)
Remove a subfile procedure (2.8.2)
Rebuild a subfile procedure (2.8.3)
ZAP RECORD-TYPE recname
Before issuing ZAP RECORD-TYPE in SPIBILD, you must tell SPIBILD which file to use, with the ESTABLISH command. ESTABLISH also processes the file's deferred queue, which must be empty for ZAP RECORD-TYPE to work. In SPIRES, you would SELECT any subfile of the file.
The ZAP RECORD-TYPE command erases all the records in the named record-type, including those records removed to the residual data set. This command may or may not release any blocks from the data set the record-type is in. It will if the named record-type is the only one in the data set, but it won't necessarily if other record-types are combined in the same data set with the named record-type.
Here is a sample SPIBILD session showing the use of ZAP RECORD-TYPE:
-? establish team statistics -Processing file: GI.PPR.STATS -Compute time: 0.166 seconds -Elapsed time: 2.004 seconds -Core usage: 0065/0933 -? zap record-type zin15 -Subfile detached -?
The ZAP SUBFILE NODE command deletes the tree node for a given record. The ZAP SUBFILE DEFQ NODE command deletes the deferred queue node for a given record.
Command: ZAP SUBFILE NODE
ZAP SUBFILE DEFQ NODE
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: REMOVE
Recovery commands: FIX SUBFILE NODE
ZAP SUBFILE NODE key ZAP SUBFILE DEFQ NODE key
These two commands are meant to be used in situations where a record in the tree or deferred queue cannot be transferred, removed or displayed because of a file services error and hence must be "removed" by hand.
WARNING: These commands are only for use to solve file integrity problems -- careless use can cause more. Seeing your SPIRES consultant before using them is recommended.
In the example below, the tree node for record 952 is zapped:
-> zap subfile node 952 -Record-type REC01 , Block = 0000004, Key = 952 ->
SPIRES tells you which block of the tree the node was in.
The ZAP SUBFILE TRANSACTION command discards the previous transaction for a given record from the deferred queue.
Command: ZAP SUBFILE TRANSACTION
Available in: SPIRES
Available to: file owner
users with Master access to the file
Related commands: UNQUEUE, ZAP DEFQ
Recovery commands: none
ZAP SUBFILE TRANSACTION key
The ZAP SUBFILE TRANSACTION command throws away the previous transaction for a given record from the deferred queue. Unlike the UNQUEUE command, the ZAP SUBFILE TRANSACTION command does not itself create a transaction, nor can it cause indexing (or "unindexing") to occur when immediate indexing is specified for the subfile.
You may use this command to throw away transactions one by one, back to the first one in the deferred queue for the given record. Compare it with the UNQUEUE command, which itself creates a transaction: repeated UNQUEUE commands affect each other, not the earlier transactions.
Remember, though, that the command is a ZAP command, and is not intended for casual use. Be aware, if your subfile has immediate indexing, that the ZAP SUBFILE TRANSACTION command will not change the indexes that would be affected by the transaction. You will probably have to fix those index records by hand.
WARNING: The SHOW SUBFILE TRANSACTIONS command, issued without the "key" option, may not accurately reflect the changes made by ZAP SUBFILE TRANSACTION commands. However, the changes for a given record will be reflected in the display from the "SHOW SUBFILE TRANSACTIONS key" command: as a transaction is discarded, it is removed from that list.
-> zap subfile transaction 955 ->
The ZAP BAD POINTERS command deletes invalid pointers from an index record.
Command: ZAP BAD POINTERS
Available in: SPIRES
Available to: file owner
users with Master access to the file
ZAP BAD POINTERS <index-name> <key>
This command causes the index pointer values to be tested for certain types of errors. If they are bad, they are "thrown away" from the index record. Each index record that has one or more bad pointers is then replaced in the data base if desired. If bad pointers are found the number of bad pointers is printed and the index record itself is listed (It will go to the ACTIVE file if you precede the command with IN ACTIVE).
You should examine the listed record before allowing any record replacement to occur to ensure that a good index record was built. This is quite critical in case there is some problem with this process. If the data changed is in a regular index record then that record is updated in the DEFQ. If the data is in a large index record segment then the change goes directly into the file. For this reason you should use care with the command. As a byproduct any search result will be cleared out.
A bad pointer is one which can produce one of the following error codes: S96, S421, S411, S520 and S532.
This appendix provides informal definitions of common and important file-management terms, presented here in alphabetical order. Many of them are discussed in greater detail elsewhere in this manual; others are described in other SPIRES manuals. In any case, most of them include cross-references to the source for more information, whether here or in some other manual.
(The following documents are not SPIRES documents per se, but describe utilities and programs that may be useful in developing SPIRES applications.)
The above documents (except any marked "in preparation") may be obtained through the PUBLISH command on the Forsythe computer at Stanford University. If you do not use SPIRES at Stanford, contact your local system administrator to find out how SPIRES documents are made available there.
SPIRES manuals are updated regularly as changes are made to the system. This does not mean that all manuals are out of date with each new version of SPIRES. The changes to the documentation match those made to SPIRES: they are usually minor and/or transparent. Not having the most current version of a manual may mean you do not have all the most recent information about all the latest features, but the information you do have will usually be accurate.
A public subfile, SPIRES DOC NOTES, contains information about changes to SPIRES manuals. Using this subfile, you can determine whether the manual you have has been updated and if so, how significant those updates are. You need to know the date your manual was published, which is printed at the top of each page. For details on the procedure, issue the command SHOW SUBFILE DESCRIPTION SPIRES DOC NOTES.