******************************************************************
* *
* Stanford Data Center *
* Stanford University *
* Stanford, Ca. 94305 *
* *
* (c)Copyright 1994 by the Board of Trustees of the *
* Leland Stanford Junior University *
* All rights reserved *
* Printed in the United States of America *
* *
******************************************************************
SPIRES (TM) is a trademark of Stanford University.
This manual was written for people who own SPIRES files or who are responsible for the maintenance of SPIRES files. If you are one of these people, you should already be familiar with many aspects of SPIRES. We consider the material presented in the SPIRES primer "A Guide to Data Base Development" as the pre-requisite for this manual. For instance, if you do not know the terms "subfile", "deferred queue", "Global FOR", "File Definer" and "file definition", you should begin with that manual, which introduces some very basic concepts of File Management that we will expound in this manual.
Though the Table of Contents may not show it, this document is divided into three major portions:
In SPIRES manuals, examples of sessions are shown with prompts and messages from SPIRES as they actually appear on the terminal screen; SPIRES prompts usually end with a question mark or greater-than sign (">"), while SPIRES messages usually begin with a hyphen. The commands and responses you type are shown in lowercase, following a prompt. (You may generally use upper or lower case interchangeably when you actually use SPIRES.) In some editions of this manual, what you type will be shown in bold, while the computer's responses are shown in normal type. For example:
-OK to clear? ok
Here SPIRES types "-OK to clear?" and you type "ok".
In formal command syntax descriptions, uppercase letters denote command verbs or other command elements to be entered exactly as shown; a value for lowercase terms and characters must be supplied by you. For example:
SELECT subfile-name
To use this particular command, you type the command verb SELECT with the name of the desired subfile, for instance, "Restaurant":
-> select restaurant
where "->" is the prompt from SPIRES.
Brackets ([ ]) denote optional elements. Braces ({ }) indicate that you must specify one and only one of the alternatives within the braces. Within the braces or brackets, a vertical line (|) separates possible choices. Neither brackets nor braces are to be typed as part of the command. For example:
SHOW SPIRES MAIL [CLEAR|NOCLEAR]
could be entered as
SHOW SPIRES MAIL or SHOW SPIRES MAIL CLEAR or SHOW SPIRES MAIL NOCLEAR
depending on whether you want the option. Here is an example from the report writer command language using braces:
AT {START|TOP} OF PAGE ...
You must choose one of the words in the braces when using this command. For example, this command could be entered as
AT START OF PAGE ...
or as
AT TOP OF PAGE ...
Sections and subsections of this manual whose titles are preceded by an asterisk in parentheses -- (*) -- may be considered optional reading at that point. Usually the material provides details that most users will not need about how to handle uncommon situations, or technical information about how SPIRES internally handles some particular command.
If you are familiar with other SPIRES manuals, you will notice many substantial differences in the areas of structure and style between this manual and those. These changes result from a study to evaluate SPIRES documentation in which different contingents of SPIRES users were interviewed. Though we'd like to reprint here all the complimentary comments we received, it's perhaps more useful to discuss some of the criticisms and suggestions, and mention how this manual addresses them.
Most of the users we interviewed suggested that SPIRES documentation would be more useful if it were more "task-oriented". Most SPIRES users come to documentation trying to accomplish a specific task, and may not know or recall the names of the tools used to do it. To address this concern, we have organized most of this manual, the "User Guide" portion, by tasks. In other words, to find out how to perform a certain file management task, you would probably look up the task in the table of contents or index. All the basic information about solving the task will appear together in one chapter or often in one section of a chapter.
Another request, related to the previous one, was to provide a standardized presentation of each command with full reference material. You will find that in Part 6, with all the commands presented in alphabetic order. Hence, if you already know the name of the command you're interested in, you can quickly find the reference material on it in Part 6. Online, you will discover, if you EXPLAIN any of the commands, that the section of documentation you see comes from that part of the manual.
In addition, depending on the source of your copy of this manual, you may notice that this manual has been printed with proportional spacing, guide words in the margins, and other special typographical improvements.
Besides the new style and structure, there is plenty of new material. Though much of the information in this manual has been seen before in the reference manuals "File Definition", "Formats" and "Technical Notes", much of it is new too. We think you will appreciate having the material together in a single manual, despite any duplications between manuals.
If you think of any aspects of SPIRES file management that weren't covered in this manual and which you'd like to see addressed in a future edition, please send them to your SPIRES consultant, or send them directly to SPIRES Documentation, Stanford Data Center, Stanford University, Stanford, California 94305.
This manual is the creation of many contributors, including the earliest designers of SPIRES back in the late 1960's and recent users who suggested improvements to the explanation of this or that feature in SPIBILD. In general tribute to all, we here thank many specific individuals for their significant help.
SPIRES is developed and maintained by the SPIRALS team of Stanford University's Stanford Data Center. Programming responsibility for SPIRES is shared by Dick Guertin and Bill Kiefer, both of whom contributed enormously to the development of this manual. Chuck Boeheim, responsible for the VM/CMS implementation of SPIRES, was very helpful in that area. Sandy Laws, Peter Tuttle, Lynn McRae, Norman Roth and Becky Morton also provided important technical support and review.
Others outside of SPIRALS provided concrete suggestions about the manual's contents, including Ed Begley of I.T.S.'s Community Information Resources and Hannah Kaufmann and Michael Sperberg-McQueen of Princeton University's Center for Information Technology.
The style of this manual is significantly different from SPIRES manuals written in the past, as is its physical design. Major contributors in this area include DRG staff members Lynne Sinclair, Jeff Rensch and Tony Navarrete, as well as Terry Oldano, Director's Office, and Anne Janzer, Systems. A new GML format for this document was patiently and painstakingly created by Greg Bruce.
For printing support, we acknowledge the assistance of staff members Lloyd Warble and Mark Lawrence of Electronic Printing and Lincoln Ong of Operations. Charley Hoyt of Systems deserves special commendation for his customization work on Waterloo Script's Photo processor especially for this document.
The manual was written by John Klemm, who thanks those named above and everyone else who provided ideas, patience, and non-addictive stimuli when they were most needed.
Managing a SPIRES file consists of tasks in three broad categories:
- record-transaction tasks, consisting of adding, updating or removing "batches" of records from the file; [You might consider that adding, updating and removing records individually in SPIRES are file management tasks, albeit on a smaller scale. But in general, the term "file management tasks" refers to work involving groups of records rather than individual ones.]
- routine maintenance tasks, ensuring that the records are being stored efficiently, that the indexes are being updated, that the file has a backup copy in storage, etc.
- repair tasks, perhaps involving the destruction or reconstruction of an index, an entire subfile or an entire file, perhaps done in response to information from a maintenance task.
The same file management capabilities are available to all SPIRES files, regardless of size. But many of them are not usually needed by small files; the test to determine whether SPIRES is storing some data inefficiently could cost more than the inefficiency itself does. But for large files (say, over 100,000 records), such tests could be very cost-effective in the long run, which is how long such files generally last.
The other parts of this manual cover the file management tasks in detail, principally for the benefit of owners of large files. Owners of small files typically do not have the need, interest or incentive to learn so much about SPIRES, and certainly shouldn't have to do so.
In view of that, the rest of this part of the manual introduces the most important aspects of file management for all SPIRES files. If you own a small file, you will probably find enough information here to meet your needs; if not, suggestions for further reading in the rest of the document are made along the way. But even if you own or manage larger files, you may find this introduction useful too.
Many of the terms that show up in discussions of SPIRES file management are informally defined in a glossary at the end of this manual. The index in the back includes a complete list of the file management tasks addressed by this manual, under the heading "file management task", with page number references to the appropriate sections, of course. Online, you can see the list by issuing the command EXPLAIN FILE MANAGEMENT TASK, which also gives you online access to all those sections of the manual too.
For small SPIRES files, file management may be quite simple:
- records are updated one at a time;
- the JOBGEN program causes the file to be processed every night so that the tree is updated using the information in the deferred queue, and the indexes are updated;
- the file is destroyed with the ZAP FILE command when it is no longer needed.
In a sense, such files have little use for "file-management procedures"; SPIRES takes care of the most important task (automatic processing of the file) without any intervention from you.
JOBGEN is a program, running after midnight each night, that examines the deferred queue of each SPIRES file requesting this service. If the deferred queue is empty, JOBGEN does nothing more to that file. But if it is not empty, JOBGEN creates a job to process the file in the SPIBILD program ("SPI" stands for SPIRES; "BILD" stands for "file-building". SPIBILD's tasks center around file-building.); the job usually runs a few minutes later. The file owner's account is charged for the cost of the job. [See 2.2.1.]
JOBGEN runs during the night because computing rates are cheaper then, and because the files are less likely to be in use at that time. (In general, SPIBILD jobs need exclusive access to a file during processing.)
The AUTOGEN flag, which is most often controlled by the file owner, determines whether JOBGEN is responsible for processing the file. The flag is set in one of three ways:
- 1) When the file is defined, the file owner determines a default setting for the AUTOGEN flag. Specifically, if the file definition contains the NOAUTOGEN statement, then NOAUTOGEN is the basic (default) setting for the file, i.e., JOBGEN is not responsible for the file. (File Definer's NOAUTOGEN attribute of the FILE statement has the same effect.) Otherwise, the basic setting is AUTOGEN.
- 2) The file owner, or a user with Master access to the file [The file definition may give other users certain file-owner powers -- this is called "Master access". This, and other forms of file access, such as Process access, are discussed in the glossary at the end of the manual.] may issue the SET AUTOGEN or SET NOAUTOGEN command to change to the desired setting.
- 3) Every time the file is processed, by either JOBGEN or you, the file returns to its basic setting, according to the file definition (see step 1).
To turn off automatic processing for awhile, you can just issue the SET NOAUTOGEN command when you have the subfile selected in SPIRES:
-> select staff addresses -> show autogen -AUTOGEN in effect -> set noautogen ->
On the other hand, if you have NOAUTOGEN in effect but decide that it's time for the file to be processed again, you can issue the SET AUTOGEN command, and the next time JOBGEN finds data in the deferred queue, the file will be processed:
-> show autogen -NOAUTOGEN in effect -> set autogen ->
Remember that the file will revert to its default setting of AUTOGEN or NOAUTOGEN, according to the file definition, after the file is processed.
Occasionally you may not want to wait for overnight processing of a file by JOBGEN. You may need your indexes updated immediately, for one reason or another, so you need the file processed now. (This scenario assumes the indexes aren't "immediate indexes", which are updated each time a record transaction occurs.)
Processing a file is quite a simple task to do, but unless you have compelling reasons to process the file yourself, you should leave it to JOBGEN. [See 2.2.1.]
If you are the file owner, or if you have PROCESS access to the file, you can process a SPIRES file. Processing the file is most commonly done in the SPIBILD program. To process a SPIRES file, you first call SPIBILD and then issue the PROCESS command:
-> spibild
-Welcome to SPIBILD
-? process orv.gq.doc.addresses
-Begin passing of 4 records of record-type REC01
-Completed passing of 4 records total
-Processed: 4 DEFQ records of record-type REC01
-Detaching File: GQ.DOC.ADDRESSES
08/11/90 File Read/Write Counts Information
File: GQ.DOC.ADDRESSES Reads Writes
CKPT data set 4 1
DEFQ data set 7 4
MSTR data set 3 0
Residual data set 3 6
Total for file: 17 11
-Compute time: 0.246 seconds
-Elapsed time: 2.407 seconds
-Core usage: 0267/0939
-? exit
-Exiting SPIBILD
Command>
The commands SPIBILD and EXIT take you in and out of SPIBILD. The SPIBILD program is similar to SPIRES in that you can issue WYLBUR and MILTEN commands within its environment, though you cannot issue SPIRES commands.
The PROCESS command causes the named file to be processed, and the output from the command as the processing occurs usually looks like the example above. If you try it but have problems, try using the EXPLAIN command to find out more about any error messages you received -- the EXPLAIN command is available in SPIBILD.
You can also process the file in SPIRES itself, without having to go into the SPIBILD program. Processing the file in SPIRES has some limitations that may affect you, but in most cases, they will not be significant. The advantages of processing in SPIRES include the ability to put the PROCESS command into a protocol, and being able to direct the error messages into the "trace log" (SET TLOG). [See 2.2.2 for a comparison of file-processing in SPIBILD and SPIRES.]
Most of the time, SPIBILD processes your files without problems. However, sometimes power outages or system problems occur during a SPIBILD session, causing the file processing to stop in mid-stream.
You generally find out that your file has been affected by a SPIBILD problem in one of three ways:
- the computer tells you that a problem occurred as you are processing the file in SPIBILD;
- you get output from JOBGEN that tells you that the file-processing job it created did not run properly;
- users discover that they cannot accomplish any subfile transactions (adds, updates, removes) for the file because the transactions are blocked when a SPIBILD processing error has occurred.
This doesn't mean the file is damaged; it merely means that the file is unstable, and normal subfile transactions cannot resume till SPIBILD cleans up whatever problem it had.
In general, the best way to "recover the file" is to issue the RECOVER command in SPIBILD, naming the file to recover:
Command> spibild
-Welcome to SPIBILD
-? recover gq.doc.addresses
-Begin passing of 12 records of record-type REC01
-Completed passing of 12 records total
-Processed: 12 DEFQ records of record-type REC01
-Detaching File: GQ.DOC.ADDRESSES
08/11/90 File Read/Write Counts Information
File: GQ.DOC.ADDRESSES Reads Writes
CKPT data set 4 1
DEFQ data set 4 4
MSTR data set 2 0
Residual data set 3 12
Rec-type REC01 2 2
Rec-type ZIN02 2 5
Total for file: 17 24
-Compute time: 1.833 seconds
-Elapsed time: 5.900 seconds
-Core usage: 0210/0939
-?
The RECOVER command begins to process the file again, though if there is a large amount of deferred queue data, the RECOVER command may successfully complete without actually finishing the file processing. When the RECOVER command completes its work, the file should be back to normal.
On the other hand, if you receive error messages rather than the messages shown in the example above, your file still has problems that need to be resolved. [See 2.1.4 for a detailed discussion of SPIBILD problems you might encounter.]
In rare cases when a file's problems are so serious that its integrity is questionable, or when a disk-storage problem arises, your computer center may need to use backup copies of data sets to restore the file to its state at the point when the backup was made. If you believe you need to take advantage of this service, contact your SPIRES consultant for help.
One of the most common duties of the file manager is to "build the file" -- that is, add many records to the file at once to get it started. Another common task that arises occasionally is to add, update and/or remove many records of a subfile at one time.
There are quite a few ways to apply multiple record-transactions to a subfile at once. In general, the easiest and least expensive methods (to the user doing the work) use SPIRES, putting the record transactions into the deferred queue. Then, when the file is processed in SPIBILD, perhaps automatically overnight, the deferred queue data will be applied to the tree of the subfile.
On the other hand, if getting the data into the tree and having the indexes reflect the new data immediately is important to you, you can use SPIBILD techniques instead. Remember that they may cost more than the SPIRES techniques, particularly if the processing is not occurring overnight, when rates are likely to be cheaper. On the other hand, if all the charges are being borne by the file owner, then doing all the work at once in SPIBILD is likely to be cheaper than putting the data first into the deferred queue in SPIRES and later having SPIBILD move the data into the tree.
Descriptions and comparisons of all the different procedures you can use for multiple record-transactions appear later, in several chapters of Part 2. [See 2.3, 2.4, 2.5.] Here we will discuss the three simplest and most general procedures of SPIBILD, probably the most useful ones to owners of small files:
- the BATCH procedure in SPIBILD, which allows you to add new records and update, merge and remove records whose keys you know; and
- the MERGE procedure in Global FOR in SPIBILD, which lets you update multiple records at once using the same merge input for each one, choosing the records to be updated with non-key criteria.
- the REMOVE command in Global FOR in SPIBILD, which lets you remove multiple records at once using non-key criteria to determine the records to remove.
The INPUT BATCH command in SPIBILD provides a fast and reliable method of getting data into a SPIRES subfile and getting the indexes updated. It's available to any user who can process the file in SPIBILD.
SPIBILD examines the data for each added and updated record, rejecting it if it breaks any of the processing rules defined for the subfile. Each record that succeeds is placed in the tree of the file. For removes, SPIBILD removes that record from the tree. SPIBILD then updates any indexes defined for the subfile.
For this procedure, your data must be in the standard SPIRES format, with each record preceded by a command followed by a semicolon ("ADD;" or "UPDATE 226-82-3846;") and (except for REMOVE) followed by an additional semicolon. [See 2.3.5.]
Here's a brief example of some data in the standard SPIRES format:
ADD; ITEM = 24-ounce Klein bottles; UNIT = carton of 12; UNIT.PRICE = $10.50; STOCK = 12; ; MERGE 134; ITEM = Mobius film strips; UNIT = "a ""grab bag"" of 10 assorted subjects"; ; REMOVE 19;
There is one caveat about using this procedure: the ESTABLISH command used at the start processes the file, which processes all the transactions currently in the deferred queue. If you don't want that to happen for some reason, consider using the INPUT BATCH command in SPIRES.
Here are the steps of the INPUT BATCH procedure in SPIBILD:
If you aren't already in SPIBILD, issue the SPIBILD command:
Command> spibild -Welcome to SPIBILD -?
You must have the input data in your active file. If you don't already have it there, use the appropriate WYLBUR command to put it there. For example,
-? use wyl.gq.jnk.supply.input -?
You will get one line of informational data about each successful record from the INPUT BATCH command by default. If you are working with dozens of records, those messages can be tedious, and they add a small degree of inefficiency too. You can turn them off by issuing the command SET INFORMATION MESSAGES = 0; it does not suppress any error messages. [See 2.1.5.3.]
It is a good idea to tell SPIRES to put the data for transactions that fail into a special data set called an "exception file". By extracting the bad data from the good data, SPIRES makes it easier for you to find the problems in the data so that you can correct it and submit it again with a subsequent INPUT BATCH command.
The exception file is an ORVYL data set that you name yourself, in the SET EXCEPTION command. For example:
-? set exception bad.supplies.records -?
SPIRES will put any errant records into the ORVYL data set called BAD.SUPPLIES.RECORDS on your account. [See 2.1.5.2.]
Issue the ESTABLISH command to tell SPIBILD which subfile you want to use and to have SPIBILD process the file containing it:
ESTABLISH subfile-name
For example:
-? establish supplies -Processing file: GQ.JNK.SUPPLIES -Begin passing of 2 records of record-type REC01 -Completed passing of 2 records total -Processed: 2 DEFQ records of record-type REC01 -Compute time: 0.196 seconds -Elapsed time: 1.227 seconds -Core usage: 0210/0939 -?
Next issue the INPUT BATCH command to tell SPIBILD to read the input data from your active file and apply it to the subfile:
INPUT BATCH
Continuing the above example:
-? input batch
- ADD 1, 1 @Line 6. Key = 356
- UPD 1, 1 @Line 10. Key = 134
- REM 1, 1 @Line 11. Key = 19
... (etc.) ...
-Begin passing of 94 records of record-type REC01
-Completed passing of 94 records total
- Requests/Success:
ADD 72 72
UPD 16 16
REM 6 6
SUM 94 94
-End of BATCH/BUILD
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 9 7
MSTR data set 6 0
Residual data set 53 51
Rec-type REC01 2 14
Rec-type ZIN02 1 13
Total for file: 75 86
-Compute time: 4.688 seconds
-Elapsed time: 14.448 seconds
-Core usage: 0279/0939
-?
In this example, the information messages were not turned off (step 3); had they been, the first three lines shown would not have appeared, since they indicate successful transactions.
The most common problem is that one or more transactions fail due to input errors, such as an invalid element name, or an improper element value. SPIBILD will display error messages telling you what the problem is. Assuming that you have set an exception file, it is easy to retrieve the unsuccessful data, using the ORVYL command GET:
-? get bad.supplies.records OK to clear? ok -?
The data is now in your active file. After correcting it, you can return to step 3 and continue the procedure.
If the system fails during any step of the procedure except step 6, just restart from the beginning when it is convenient to. If it fails during the INPUT BATCH command, you need to determine the point where it failed, i.e., figure out which was the last record to be fully processed. A command called SHOW RECORD RECOVERY may be useful in this regard. For full details, see the complete discussion on the INPUT BATCH procedure in SPIBILD. [See 2.3.1.]
Frequently the file manager may find that many records sharing some element value need to have that element value changed or that they need to be removed from the subfile. Removing records will be discussed in the next section; here we will discuss updating them.
In the updating situation, SPIBILD provides a MERGE ALL command using Global FOR, which merges the data in your active file into all the records that meet the Global FOR criteria. Like SPIBILD's BATCH procedure, this procedure causes the file to be processed prior to the MERGE command and SPIBILD processes the file again at the end of it, so it can be relatively expensive. [See 2.4.2.]
The procedure for using it is described below. Note: You must be able to process the file to use this procedure.
If you aren't in SPIBILD already, get in:
Command> spibild -Welcome to SPIBILD ->
Tell SPIBILD which subfile you want with the ESTABLISH command, which also processes the file:
-> establish supplies -Processing file: GQ.JNK.SUPPLIES -Compute time: 0.104 seconds -Elapsed time: 1.073 seconds -Core usage: 0130/0939 ->
In this example, the intent is to find records with the value RCA for the singly-occurring element SUPPLIER and replace it with the value "General Electric". See the SPIRES manual "Searching and Updating" for more information on merging in general (for instance, how to add new occurrences, replace multiple existing ones, remove occurrences, etc.).
You must have the input data in your active file. It can be in the standard SPIRES format or in a custom format; if the latter, you should issue the SET FORMAT command as part of this step. [The format must be a merge format. See the manual "SPIRES Formats" for further information.] For example,
-> collect
1. > supplier = General Electric;
2. > *** <- user presses ATTN/BREAK
->
In SPIBILD, the only Global FOR classes available are the SUBFILE and STORED classes. You can, however, use a WHERE clause and SET SCAN commands to further refine the group of records to be updated.
For example,
-? for subfile where supplier = rca -? set scan start 10000 -? set scan stop 19999 -?
The SET SCAN START and STOP commands tell SPIBILD to look only at records with key values from 10000 to 19999. This is considerably more efficient than "FOR SUBFILE WHERE SUPPLIER = RCA" alone, if all the records you want are within the limited range.
The STORED class would be preferable to use if, for example, you could easily create a stored result or stack in SPIRES of the records you wanted to update. Then SPIBILD could retrieve the records for updating directly, rather than have to use WHERE-clause examination techniques to find the records to update. Suppose, for instance, that in SPIRES you could store the search result from the command FIND SUPPLIER = RCA. Using that stored search would be much more efficient than having SPIBILD examine each record in the subfile.
The MERGE ALL command merges the data in your active file into each record fitting the Global FOR class, after which SPIBILD processes the file so that the indexes are all updated.
-? merge all from active
- Requests/Success:
MER 38 38
SUM 38 38
-End of BATCH/BUILD
-Begin passing of 38 records of record-type REC01
-Completed passing of 38 records total
-Processing: 38 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 6 82
MSTR data set 5 0
Residual data set 4 46
Rec-type REC01 1 0
Rec-type ZIN02 1 25
Total for file: 21 154
-Compute time: 7.894 seconds
-Elapsed time: 19.521 seconds
-Core usage: 0256/0933
-?
If an error occurs during step 5, SPIRES will tell you about it and stop. The most likely error comes from bad input data, i.e., an invalid element name or value for that element. That will most likely happen on the very first record -- if it won't go into one record, it probably won't go into any.
To recover from this problem, simply correct the data and start again from the ESTABLISH command in step 2.
If you discover that you updated the wrong records, you will need to determine another way to find those records and perhaps "re-merge" the old data into them.
If the system crashes during the first 4 steps of the procedure, simply restart it at your convenience. If the system crashes during the MERGE processing, see the explanation for handling this problem in the full discussion of the MERGE procedure in Global FOR in SPIBILD later in this manual. [See 2.4.2.] A simple possibility worth mentioning here is to simply restart the procedure and merge the same data again into the records that succeeded. Do not do this, though, if the merge requests are creating new element occurrences or if the data is being used to compute other element values.
Sometime you may have to remove many records from the subfile based on some criteria they share, such as all records added to the subfile more than two years ago, or records that don't have occurrences of some element.
This situation is quite easily handled in SPIRES, using the REMOVE command in Global FOR mode. REMOVE is also available under Global FOR in SPIBILD and has the same effect; in fact, it works more efficiently there than in SPIRES. However, you are limited to the Global FOR classes SUBFILE (with WHERE clauses, SET SCAN and SET FILTER commands to limit the records examined and removed) and STORED (with WHERE clauses, the SET SCAN LIMIT and SET FILTER commands) in SPIBILD.
Suppose, for example, that you want to remove 50 records of a 1000-record subfile, and you can find those 50 records using a simple search:
-> find supplier = woolco -Result: 50 SUPPLIES ->
In that case, you may want to simply remove the records in the search result using SPIRES:
-> for result +> remove all -Removed: 50 record(s) +>
The removal requests are in the deferred queue, so the records won't be removed from the tree till the file is next processed.
Another possibility would be to store the result (or stack) in SPIRES for use under the FOR STORED command in SPIBILD:
-> show result -Result: 50 SUPPLIES -> store result.woolco -'RESULT.WOOLCO' put in ORVYL file system ->
In SPIBILD, you can use the command FOR STORED RESULT.WOOLCO to remove only the records in the result (see step 3 below); under SPIBILD processing the file would be processed, and the records would be removed from the tree and "un-indexed" immediately.
Especially if there are several hundred records to remove (say, more than 15 percent of the subfile), the SPIBILD methods discussed below are preferable.
Warning: If the subfile is fairly large (say, more than 1000 records) and you want to remove more than half of them, you should consider zapping the goal record-type and rebuilding it with only the records you want to keep. [See 2.8.3.]
One final aspect to consider between the SPIRES and SPIBILD methods of removing records in Global FOR -- in SPIRES, the removal requests go into the deferred queue and can be unqueued if you later determine that you removed the wrong records. In SPIBILD, once they are removed, they are gone. Often when file managers remove many records from a subfile using SPIBILD, they make sure they have a backup copy of the file around, just in case. [See 2.1.4.3.]
The procedure for removing multiple records in SPIBILD is described below. If you plan to use FOR STORED processing, as described above, create your stored result or stack in SPIRES before beginning. Note: You must be able to process the file to use this procedure.
If you aren't in SPIBILD already, issue the SPIBILD command:
-> spibild -Welcome to SPIBILD -?
Tell SPIBILD which subfile you want with the ESTABLISH command, which also processes the file:
-? establish supplies -Processing file: GQ.JNK.SUPPLIES -Compute time: 0.208 seconds -Elapsed time: 1.037 seconds -Core usage: 0130/0939 -?
Remember that you are limited here to the FOR SUBFILE or FOR STORED command, with an optional WHERE clause. You should also use SET SCAN commands to limit the records examined, if you can, since that will save unnecessary work.
If you're not familiar with the SET SCAN commands, you might think of them this way: suppose you are asked to look for one book in the library, and you have no card catalog (index) to use. You must look on every shelf in the library till you find the book you want. But if you are told that the book you want must be on the north end of the third floor, the work you must do has been narrowed considerably. The SET SCAN commands do that type of limiting, telling SPIRES what record keys to examine.
For example, here we are removing old records from a subfile with slot keys:
-? for subfile where date-added < 1985 -? set scan stop 500 -?
It happens that we know that no records added before 1985 have keys greater than 500. The SET SCAN STOP command shown ensures that only records whose keys are 500 or less will be examined for removal.
To use FOR STORED instead, you would replace FOR SUBFILE with FOR STORED, along with the name of the stored stack or result.
-? for stored result.woolco
You may also add a WHERE clause to further limit the records that are removed; SET SCAN LIMIT is the only SET SCAN command allowed with FOR STORED, however.
Issue the REMOVE ALL command to tell SPIBILD to remove all the records that match the criteria established by the Global FOR class:
-? remove all
- REM 1, 1 @Line 1. Key = 131
- REM 2, 2 @Line 1. Key = 132
- REM 3, 3 @Line 1. Key = 134
- REM 4, 4 @Line 1. Key = 135
... (etc.) ...
- Requests/Success:
REM 58 58
SUM 58 58
-End of BATCH/BUILD
-Begin passing of 58 records of record-type REC01
-Completed passing of 58 records total
-Processed: 58 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.SUPPLIES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.SUPPLIES Reads Writes
CKPT data set 4 1
DEFQ data set 6 120
MSTR data set 5 0
Residual data set 7 65
Rec-type REC01 2 58
Rec-type ZIN02 1 20
Total for file: 25 264
-Compute time: 0.516 seconds
-Elapsed time: 13.240 seconds
-Core usage: 0204/0933
-?
If you discover that you removed the wrong records, the records are gone and cannot be recovered except via any backup copies of the file or data you may be able to use.
If the system crashes during the procedure, you may simply restart the procedure at your convenience.
As its manager, you have access to many interesting statistics about your SPIRES file, such as the amount of storage space the file is using, or the kinds of commands users are issuing when they select your subfile.
Here are some of the statistical questions you might ask about your file, along with possible answers.
This is a useful question to ask because storage charges are based on the number of blocks the file uses. To find out the answer, issue the ORVYL command SHOW FILES, including the name of the file and the BLOCKS option:
-> show files like almanac, blocks ORVYL File System ALMANAC.CKPT -- 7 BLKS (F: 0, L: 6) ALMANAC.DEFQ -- 2 BLKS (F: 0, L: 1) ALMANAC.MSTR -- 6 BLKS (F: 0, L: 5) ALMANAC.REC1 -- 14 BLKS (F: 0, L: 13) ALMANAC.REC2 -- 108 BLKS (F: 0, L: 107) ALMANAC.RES -- 123 BLKS (F: 0, L: 122) ->
The total of the number of blocks (the "BLKS") is the size of your file (260 in this example). [See 3.2.1.]
One way to get these statistics is to issue the SHOW FILE ACTIVITY command when you have a subfile of the file selected. For example,
-> select almanac
-> show file activity
01/26/87 Activity Record of File GQ.JNK.ALMANAC
Last Compiled: 01/07/87 at 23:55:02
Record Base Blocks Adds Deletes Updates Max Rec
Type Block Used -DS Length
ENTRY -Slot 0 14 -1 3272 0 22 339
REC2 1 105 -2 373 0 1841 72
REC3 8 Comb -2 3180 6 89 18
REC4 25 Comb -2 2564 2 407 417
REC5 103 Comb -2 7 0 21 2004
REC6 104 Comb -2 2 0 10 11063
Residual 2 123 3488 7 414 11067
Def Queue 1 3
->
By subtracting the "Deletes" from the "Adds", you get the number of records in each record-type. The ENTRY record-type, for instance, has 3272 entries (3272 minus 0).
The SHOW FILE ACTIVITY command displays some other useful information too, such as the date and time when the file was last compiled. [See 4.2.2.]
Another way is to use the ATTACH command to "attach" each record-type of the file, followed by the SHOW SUBFILE SIZE command. The SHOW SUBFILE SIZE command does exactly the work you do -- subtracting the "Deletes" from the "Adds":
-> attach 1 of gq.jnk.almanac -> show select -Record-type ENTRY of GQ.JNK.ALMANAC attached -> show subfile size -The subfile has 3272 records -> attach 2 of gq.jnk.almanac (etc.)
The ATTACH command lets you use any record-type of a file as if it were the goal record-type of a selected subfile. [See 5.4.]
If you need more information about a particular transaction in the deferred queue, such as when it happened or who did it, issue the SHOW SUBFILE TRANSACTIONS command, optionally followed by the key of the record you are interested in. For example,
-> show subfile transactions 12933
01/26/87 Transaction Log of File GQ.DOC.ERRATA
for Record-type G1SECT
Date Time Account Id Type Command Grp Key Value
01/26/87 15:45:28 GQ.JNK ADD Update 12933
01/26/87 13:32:35 GQ.JNK ADD Update 12933
01/26/87 10:54:10 GQ.JNK ADD Add 12933
->
Information about the transactions for the record that have been made since the file was last processed are shown, listed in reverse chronological order. This example shows that user GQ.JNK added the record at 10:54 a.m., and updated it twice later in the afternoon.
If you omit a specific key from the command, SPIRES will list all the transactions for the selected subfile that are in the deferred queue, in chronological order. More information about how to read the transaction log appears later in this manual. [See 4.2.3.1.]
This is representative of many questions you may have about how users are taking advantage (or failing to) of your subfile. You may want to know how many users are selecting it, or what commands they are issuing to use it.
You can ask SPIRES to "log" the use of your subfile by adding some statements to your file definition (directly or via your File Definer input). Later you can analyze the log's contents to help you evaluate the usefulness of your file, and to help you look for ways it could be improved, e.g., new indexes it needs, current indexes it doesn't need, etc.
Directions for establishing a log appear later in this manual. [See 4.3.]
Other statistical information can tell the file manager how efficiently the file's blocks are being used for storage, and how many levels deep are the trees of the various data sets of the file. This information, and much more, is available through the STATUS and SHOW FILE STATUS commands. [See 3.2.3.] But in general, those commands are of interest (and of need) to managers of large files.
When you no longer need a SPIRES file, you should destroy it, so that it will stop accruing storage charges. If the data contained in it is important to you, you might want to make a printed listing of the data or perhaps save the data in another data set on disk or tape before you "zap" the file.
The SPIRES command that erases the file is ZAP FILE, which names the file to be erased. It is quite easy to use (almost too easy, considering its drastic effects):
-> zap file gq.jnk.supplies -Do you really want to destroy file GQ.JNK.SUPPLIES ? ok ->
Once you confirm your desire, the ORVYL data sets of the file are quickly erased and the file is gone. Only the file owner can issue the ZAP FILE command for a file. [See 2.11.]
After a ZAP FILE command, the file definition still exists in the FILEDEF subfile; if you no longer need it, you should remove it from there, or consider moving it to the BACKFILE subfile. If you don't do one or the other, the subfiles of the file will continue to show up in the list from the SHOW SUBFILES command.
Also get rid of code for the file from other system subfiles, such as FORMATS, RECDEF or EXTDEF, as part of your cleanup activities. [See 2.11 for the complete description of the procedure, which describes how to find and get rid of any other code.]
The next four parts of this manual are a "User Guide" to file management. In these sections you'll find descriptions of procedures you can use to accomplish various file management tasks.
Here in Part 2, the subject is the building and destroying of record-types, subfiles and files, the most important responsibilities of every SPIRES file manager.
Many of the procedures described in this part of the manual require you to use a program called SPIBILD. The rest of this first chapter describes the SPIBILD program in general, introducing various commands and "sub-procedures" that may be useful to you as you follow the procedures in the remainder of Part 2.
Those remaining chapters of Part 2 are:
SPIBILD is a part of SPIRES that builds the data sets containing a file's records. Most procedures that involve SPIBILD cause the file to be "processed". When you process a file, the records in the file's deferred queue -- that is, those that have been added or updated since the file was last processed -- are moved from the deferred queue to the tree. Also, any records with removal requests in the deferred queue are removed from the tree. Along the way, changes are made to the indexes as appropriate, by adding, updating and removing data from them to reflect the changes to the tree.
Processing a file can be done in SPIRES or SPIBILD, depending sometimes on specific needs but generally on personal preference. [See 2.2.2, 2.2.3 for information about processing a file in SPIRES.]
You can process your own files anytime you want, or you can allow other people to process them. Or, if you want, SPIBILD will process your files automatically each night, saving you the trouble.
You can use SPIBILD as part of many other tasks too, which are discussed later in Part 2:
- to add, update or remove many records at once
- to build new indexes for a subfile that already exists
- to remove indexes you no longer need
- to remove one subfile in a file when you no longer need it but need to keep the file
- to rebuild indexes, or entire subfiles, or entire files
The rest of Chapter 2.1 discusses aspects of SPIBILD in general, which relate to most of the tasks discussed in the rest of Part 2. The remaining sections of this chapter are:
When you want to process one of your files yourself, you must use SPIBILD, the program that builds the data sets containing SPIRES records.
The online version of SPIBILD is an ORVYL program invoked by the SPIBILD command:
SPIBILD [(parameters)]
where the most popular optional parameters are:
- QUIET, to suppress the welcome message, and
- WYLBUR, to request that the command SET WYLBUR be issued automatically
Separate them with a comma or blank to request both. See Part 6 for the full syntax, or online, [EXPLAIN SPIBILD COMMAND.]
For example:
-> spibild -Welcome to SPIBILD 87.06 -?
In this example, 87.06 is the release number, indicating that this version of SPIBILD was released in June of 1987.
Unlike SPIRES, SPIBILD has no "uplow" setting, because none of its commands are case-sensitive. [SPIRES has an uplow setting because the data the user types in some commands, such as DISPLAY or FIND, may be case-sensitive.] You can type commands in uppercase, lowercase or any mixture you like.
Once you have invoked SPIBILD, you can issue any SPIBILD command, or any WYLBUR, ORVYL or MILTEN commands. To use SPIRES commands, or those of any other SPIRES processor, you must invoke that processor, leaving SPIBILD.
To leave SPIBILD, issue the EXIT command:
EXIT [QUIET]
where the QUIET option suppresses the exit message from SPIBILD.
The EXIT command takes you back to WYLBUR:
-? exit -Exiting SPIBILD Command>
You can also leave SPIBILD by issuing a command to invoke another SPIRES processor, e.g., "SPIRES".
Between the SPIBILD and EXIT commands, you will probably issue several SPIBILD commands, as part of procedures discussed in other chapters of Part 2. Other SPIBILD commands useful in many different SPIBILD procedures are discussed later in this chapter. [See 2.1.5.]
The remainder of this section discusses features and commands that the SPIBILD environment has in common with SPIRES. Most are fully documented in chapter A.2 of the SPIRES reference manual "Searching and Updating", with specific sections referenced below. In summary, those that affect SPIBILD are:
- The EXPLAIN command is available in SPIBILD, for explanations of SPIRES terms and error messages. Subfile explanations, available in SPIRES, are not available in SPIBILD, however. Nor are the SHOW EXAMPLE and SHOW SYNTAX commands. You can put an explanation in your active file by preceding the command with the IN ACTIVE prefix (see below).
- The SHOW EVAL command is available in SPIBILD as well as in SPIRES, although many variables (those involving search results or protocol execution, for example) are not available to be evaluated. The command (documented in the "SPIRES Protocols" manual) could also be used for on-the-spot computations in SPIBILD or for evaluation of static variables before and after SPIBILD processing. One likely use of SHOW EVAL in SPIBILD might be to determine which version of the program (i.e., Test or Public) you are currently using:
-? show eval $version
87.01.27 PUBLIC System
-?
- Unless otherwise mentioned in their documentation, command verbs and their options can be abbreviated to their first three characters. [See A.2.7 of "SPIRES Searching and Updating".]
- The SET WYLBUR and SET NOWYLBUR commands can be issued to specify whether WYLBUR or SPIBILD examines commands first, passing unrecognized commands to the other program. WYLBUR Exec files will not execute in SPIBILD unless SET WYLBUR is in effect. Extended Exec files will execute in SPIBILD, however, regardless of SET WYLBUR or SET NOWYLBUR. [See A.2.5.1 of "SPIRES Searching and Updating".]
- You may use the WYLBUR active file during various SPIBILD activities. For example, it may contain data that you want to merge into a SPIRES file. The IN ACTIVE prefix, with the CLEAR and CONTINUE options, may be used on some commands to direct data to your active file. [See A.2.9 of "SPIRES Searching and Updating".] The following commands relating to the active file are applicable in SPIBILD:
- SET LIST and SET NOLIST -- The default is LIST, which means that SPIBILD will automatically list your active file at the end of any command that puts data into it. [See A.2.9.2 of "SPIRES Searching and Updating".]
- SET CLEAR and SET NOCLEAR -- The default is NOCLEAR, which means that SPIBILD will not automatically clear your active file if you use the IN ACTIVE prefix without either the CLEAR or CONTINUE option. [See A.2.9.3 of "SPIRES Searching and Updating".]
- SET LENGTH -- This command can be used to change the default line length of 72, the value of the $LENGTH variable. [See A.2.9.4 of "SPIRES Searching and Updating".]
- The ATTN/BREAK key has several uses in SPIBILD:
- If you press the key in response to a command prompt, WYLBUR will either reprompt or place you in Collect mode.
- If you press the key after you have begun typing a command but before you press Return, the command you were typing is ignored, and you are prompted for a new command.
- If you press the key while SPIBILD is executing a command, you will interrupt the processing. SPIBILD may ask you if it can continue the processing, or it may simply stop altogether. Often in this case, if you press the key twice in succession, a session break will result. Type GO if you want the SPIBILD session to continue.
- If SPIBILD asks a question (such as "-Continue processing?"), pressing the ATTN/BREAK key is equivalent to answering NO.
- You can see how many read and write operations have taken place involving blocks of the files involved in a SPIBILD task by issuing the SET FILE COUNTS command prior to the main commands of the task in SPIBILD. SET FILE COUNTS is quite similar in effect to the SHOW FILE COUNTS command in SPIRES, though it does have some significant differences. See Part 6 for more information, or online, [EXPLAIN SET FILE COUNTS COMMAND.]
You can use SPIBILD through the batch system as well as interactively. That allows you to process SPIRES files, merge records into a subfile, or execute any other SPIBILD task by submitting jobs to the batch system.
There are several reasons why you might want to use SPIBILD in batch:
- 1) The input data you have for a SPIRES file is on tape or is too large to fit in your active file. [If the data is in standard SPIRES format, you can also use the BATCH command in SPIRES. [See the SPIRES manual "Searching and Updating", section D.8.]]
- 2) You want the SPIBILD work to occur at a time when you won't be logged on (in the middle of the night, for instance), so you submit a batch job to run for you at that time.
- 3) You don't want to tie up your terminal while your SPIBILD work is being done.
You can use either of two different programs when you want to use SPIBILD in batch:
The next two sections will provide detail about using these two programs for SPIBILD work and will give you a clearer picture of which to use in particular situations. [See 2.1.3.1, 2.1.3.2.] In general, more people choose to use BATWYL because of its flexibility, but here are some guidelines for choosing between them:
- If you are using SPIBILD to merge some data into a SPIRES file from a tape, you must use batch SPIBILD.
- If the input file is too large to fit into your WYLBUR active file or is in a format that cannot be imported into your active file, you must use batch SPIBILD.
- If you need return codes from the job in the event of SPIBILD problems, you should use batch SPIBILD.
- If you need to use WYLBUR Exec files, or to leave SPIBILD during the job, you should use BATWYL.
A BATWYL job is basically an interactive session controlled from the batch processor. Unlike some batch programs where writing the appropriate JCL can be very tedious and difficult, BATWYL uses very simple JCL. The challenge in constructing BATWYL jobs is to anticipate every prompt that BATWYL will send to the "virtual terminal" and include an appropriate response in the command stream.
Complete details on using BATWYL can be found in the document "Using BATWYL", available through the PUBLISH command.
Use the following JCL to run BATWYL jobs:
// JOB // EXEC BATWYL[,PARM='ATTN=*'] //SYSIN DD * ... interactive commands ...
The statements will be formally described below.
First though, here's an example of a simple BATWYL job that does some SPIBILD work (the numbers on the left are not part of the input but are used for reference in the explanation that follows):
1. // JOB
2. // EXEC BATWYL,PARM='ATTN=*'
3. //SYSIN DD *
4. call spires
5. select manual
6. for adds
7. in active display all
8. print forms=3hol duplex
9. in active show subfile transactions
10. ok
11. print forms=3hol duplex
12. exit
13. call spibild
14. process gq.doc.manual
15. logoff
In the above example, following the JCL (lines 1-3), the job calls SPIRES (4), places all the newly added records in the active file (6,7), and then prints them (8). Next, it puts the transactions display into the active file for printing (9-11). The "ok" (10) is not a command, but is the response to the SPIRES prompt "OK to clear?" that would result from the preceding command. (More likely, you would add the CLEAR option to the IN ACTIVE prefix on the SHOW SUBFILE TRANSACTIONS command, to avoid the prompt altogether.)
Next the job calls SPIBILD, processes the file, and logs off, ending the job (13-15).
Here are descriptions of the JCL statements you use to run BATWYL:
// JOB
You must begin all jobs with the JOB statement. It may be as simple as shown, or may include a job name, account number, bin number, etc. [See the document "Batch Processing" for more information.] Be sure to type blanks as shown in JCL, such as at least one before the word JOB.
// EXEC BATWYL,PARM='ATTN=*'
The EXEC statement calls the BATWYL program. The PARM option shown here allows you to create an ATTN/BREAK response by typing three asterisks at the end of a command line. The character doesn't have to be an asterisk; whatever character you choose should be specified in the ATTN clause instead. If you will not need to simulate an ATTN/BREAK, you do not need the PARM option (nor the comma preceding it). Other options are also available.
//SYSIN DD *[,DCB=LRECL=240]
The SYSIN DD statement shown above indicates that the commands to be executed by WYLBUR follow this statement. The document "Using BATWYL" describes how you can tell the program that the commands are stored elsewhere. If you add the DCB option shown, you raise the allowed length of commands from 80 to 240 characters. Note that SPIBILD commands may not exceed 160 characters, but WYLBUR and Milten commands can be as long as 235 characters.
... Interactive commands ...
Here you list the commands you want executed. If commands exceed 80 characters in length, be sure to add the DCB option to the SYSIN DD statement shown above. Note that SPIBILD commands absolutely may not exceed 160 characters.
All interactive commands are available, with the exception of those that invoke full-screen processes, such as Prism or full-screen SPIRES applications.
To use SPIBILD, you must include a SPIBILD command. After that, all SPIBILD commands are available to you. Note, however, that command options limited to batch SPIBILD, such as the "ON ddname" option on the BATCH command, are unavailable, because you are using interactive SPIBILD through the batch system, not batch SPIBILD.
Finally, be aware that output lines longer than 133 characters will be truncated.
Batch SPIBILD gives you direct access to the SPIBILD program through the batch system. You can, for instance, run jobs that process other user's files if you can do so yourself online. In addition, you may use your WYLBUR active file for input and output, and may issue WYLBUR commands such as DELETE or PRINT. However, there are some differences from online SPIBILD, and those restrictions are described in appropriate places throughout this manual and are summarized in this section.
Use the following JCL to run batch SPIBILD jobs. (Statements marked with an asterisk are optional, and are explained below.)
1. // JOB 2. // EXEC SPIBILD *3. //DDNAME DD DSN='WYL.gg.uuu.dsname',DISP=SHR, *4. // VOL=SER=volume,UNIT=unit *5. //SYSIN DD * 6. ... spibild commands ... 7. LOGOFF
For general information about batch jobs and JCL, see the "Batch Processing" manual, available through the PUBLISH command. Detail information about each of these specific statements appears below.
First, though, here is a sample job that uses batch SPIBILD:
// JOB // EXEC SPIBILD //DDNAME DD DSN='WYL.GQ.DOC.MANUAL.INPUT',DISP=SHR //SYSIN DD * batch 'manual' on ddname logoff clear
This job batches records in the data set named in the DDNAME DD statement into the subfile MANUAL.
Here are descriptions of the JCL statements you use to run batch SPIBILD:
1. // JOB
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number, output bin, region size, maximum number of output lines allowed (5000, by default), etc. See "Batch Processing" for more information.
Note that you never need to change the default region size, no matter how much SPIBILD work you are doing -- the region size of the job controls a small program that invokes interactive SPIBILD, which always has more than enough region for any SPIBILD task.
2. // EXEC SPIBILD[,PARM='parm']
The EXEC statement calls the batch SPIBILD program. The PARM option is occasionally used to set special escape characters, or to request versions of SPIBILD other than the public one, such as internal test versions, if they are available to you. The parameters most commonly specified are:
ESC=x OUT=y ACCOUNT=gg.uuu LIBRARY=[ORV.gg.uuu.]libname
If used, ESC and OUT need to appear in the PARM option before the other two.
ESC and OUT have the same effect in Batch SPIBILD as they do in the BATWYL program. Please see the document "Using BATWYL", Data Center document number 409, for more information.
*3. //DDNAME DD DSN='WYL.gg.uuu.dsname',DISP=SHR, *4. // VOL=SER=volume,UNIT=unit
Use the optional DDNAME statement when you want SPIBILD to use input data from a data set stored on tape or on disk. Generally speaking, it is easier for you to use the USE and GET commands in batch SPIBILD to bring data into your active file from stored data sets, but you must use the DDNAME statement if either of the following is true:
- the input data is on tape;
- the input data is not in a format that can be brought into your WYLBUR active file.
The data set may have a RECFM value of F or FB or U, with a fixed LRECL of up to 4090 bytes.
If the data set is cataloged, you can eliminate the second line of the DD statement shown above, as well as the comma at the end of the first line. Otherwise, specify the tape or disk volume number, and the unit TAPE or DISK. Be sure to include a /*SETUP card if you're using tapes.
*5. //SYSIN DD *[,DCB=LRECL=240]
The SYSIN statement shown above indicates that the commands to be executed by SPIBILD follow this statement. Thus, it should be the last JCL statement before the list of commands.
If you want, you can store the commands separately and then tell batch SPIBILD where to find them for execution. You do this by replacing the * in the SYSIN statement with the same type of values that follow the DD in the DDNAME statement above: the name of the data set (DSN) containing the commands, the disposition (DISP), and the VOL and UNIT information if the data set is not cataloged. The data set must be saved either in EDIT format, or at any fixed length up to LRECL=240.
If you need to issue commands longer than 80 characters (but no longer than 240) in your batch SPIBILD job, add the DCB option shown. Be aware that SPIBILD commands are limited to 160 characters, and WYLBUR and Milten commands are limited to 235 characters. (Remember too that you can abbreviate many command verbs and options, which might shrink long commands to fewer than 80 characters.)
6. SPIBILD commands
Here you list the commands you want executed, each command consisting of one and only one line of input. (See the previous paragraph about commands longer than 80 characters.)
All SPIBILD commands are available and work as you would expect, with the following exceptions:
If a SPIBILD command would prompt you for input if you were in interactive SPIBILD, then batch SPIBILD will assume a response of ATTN (the ATTN/BREAK key).
7. LOGOFF [CLEAR]
If you omit the LOGOFF statement, batch SPIBILD will supply it for you, and give you a return code of 4.
Though command lines may be as long as 150 characters, output from batch SPIBILD cannot exceed 133 characters per line. Lines longer than 133 characters will be truncated on output.
There are other lines of JCL you can add if you want, such as JOBPARM and SYSPRINT. See the "Batch Processing" manual for more information about them.
The following condition codes are supplied by SPIBILD if a job terminates abnormally:
Condition code Meaning
4 - LOGOFF CLEAR was supplied by the
program. The input stream ran out
without the proper LOGOFF command.
20 - Virtual terminal communications
problem.
24 - SUZAN path to Milten disconnected,
probably because ORVYL died.
28 - SUZAN path to SPIBILD disconnected.
32 - No SYSIN or COMMAND input data.
Normally when you use SPIBILD or SPIRES for a file-processing task, the task is completed without problems or complications. However, problems that can cause a file-processing command to go awry do occasionally arise. For example, while you are waiting for a file-processing command to execute:
- the system might fail;
- you might accidentally disconnect your terminal, causing the system to log you off;
- SPIBILD or SPIRES could encounter a serious problem with your file, causing it to stop executing the command immediately.
The results of these problems vary extensively, depending on the command being executed, on the precise point when the processing stopped, etc. In most cases, the file has suffered no ill effects at all; in other cases, there may be data integrity problems that SPIBILD or SPIRES needs to repair.
For example, suppose the system fails while SPIBILD is updating a subfile's indexes with some data from newly added records. At the time of the failure, half the indexes have been updated, but not the other half. As a result, the indexes are not reliable, and they won't be until the problem is repaired.
Occasionally your own programming error can cause problems too. For instance, changing a file definition carelessly can sometimes cause serious data integrity problems the next time the file is processed, requiring you to "back up" to a previous version of the file, stored prior to the change, to restore your file to a usable condition.
The next section describes the most common problems encountered during SPIBILD or SPIRES file-processing, with suggested methods for handling them. [See 2.1.4.1.]
Invariably, a result of interrupted file-processing commands is that the NO JOBGEN flag is set, blocking all file transactions (such as adds and removes) till it is reset.
In almost all situations where file-processing is interrupted, the file is fine; if you return to SPIBILD or SPIRES and issue the PROCESS command, the problem will be cleared up, with the JOBGEN flag reset so that file updates can continue. If you do not want to spend the time or money on a complete processing of the file, you can issue the RECOVER command instead, which does a "mini-process" of the file: just enough work to clear up any minor problems and reset the JOBGEN flag. The complete RECOVER/PROCESS command procedure for file recovery is described in the next section. [See 2.1.4.2.]
Another technique, used very infrequently, is available to files whose file-processing is automatically handled by JOBGEN. [See 2.2.1.] The SET JOBGEN command is at the heart of that procedure. It is used less frequently because all it does is reset the JOBGEN flag, so that file updates are allowed again. That allows the file to be processed again via JOBGEN, starting that night.
Unfortunately, the SET JOBGEN command can create serious data integrity problems, and should not be used unless there will be no file updates between the time you set the JOBGEN flag and the time the JOBGEN-created batch job processes the file. See Part 6 for more information; online, [EXPLAIN SET JOBGEN COMMAND.]
In extremely rare cases, such severe problems can occur that the file is considered destroyed. When a problem like that occurs, it is more likely due to system problems (such as a disk-drive failure) than to a SPIBILD problem. But in any case, backup copies of the file or parts of the file are a vital part of SPIRES applications.
Some basic backup services are provided by your computer center, which are always adequate if JOBGEN handles your file-processing needs. However, you may choose to handle a file's backup yourself. A discussion of the available services appears later in this section. [See 2.1.4.3.]
Though most file-processing tasks in SPIBILD and SPIRES run quite smoothly, this section discusses the most frequently encountered problems (some described by their system error numbers), with suggested solutions. Remember that the EXPLAIN command can be used in SPIBILD as well as SPIRES to get information about system messages and error codes.
This problem could be caused by a system crash or by your terminal becoming disconnected. Note that although the file will be in a somewhat unstable state, users can still use the file for searching, though probably not for updating, until you can execute one of the two recovery procedures to stabilize it completely. In any case, the file is not damaged, and the recovery procedures will return it to normal.
To recover the file, follow the RECOVER/PROCESS procedure. [See 2.1.4.2.]
You may have misspelled the name of the file. If that's not the case, return to SPIRES, select a subfile of the file, and check the value of the $FILENAME system variable (SHOW EVAL $FILENAME) to be sure you were trying to process the correct file.
Another possibility is that you are trying to use a file that hasn't been compiled yet. That sometimes happens when you are rebuilding a file: you issue the ZAP FILE command to destroy the old file and then go to SPIBILD to rebuild it, without first compiling it again. The file doesn't exist till it is compiled.
This error indicates that other people have subfiles of the file selected. You may want to use the SET SEARCH/SET SHARE procedure to circumvent this problem. [See 2.1.5.1.] Note: SPIRES automatically "sets SEARCH" when you issue the PROCESS command in SPIRES, so S56 errors are not a problem there.
To process someone else's file, you must have WRITE access to all the ORVYL data sets of the file except the MSTR, as well as WRITE access to the SYSTEM.CHKPT data set on the file owner's account. If you don't have the proper access, the command will fail with an S116 error. The file owner can correct the problem by issuing SET PERMIT commands for each of the data sets.
This list is not comprehensive -- other problems can stop file-processing too. In general, the error messages, combined with information available through the EXPLAIN command, should lead you to the problem and suggest a solution. Of course, file-processing problems should be taken seriously, so if you need any help, be sure to see your SPIRES consultant as soon as possible.
Procedure: RECOVER/PROCESS
Purpose: General recovery from SPIBILD problems
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Issue the PROCESS or RECOVER command
3. Fix the problem, if step 2 fails
When a SPIBILD command finishes abnormally, it may leave the file in an unstable state. It will definitely leave the NO JOBGEN flag set, meaning that automatic processing of the file by JOBGEN will not occur, and no file updates are allowed.
The RECOVER/PROCESS procedure is a general one for solving such problems; special factors for particular SPIBILD tasks that don't get completed are discussed under individual procedures.
If you aren't already in SPIBILD, issue the SPIBILD command:
-> spibild -Welcome to SPIBILD -?
Note: If you use the PROCESS command (see step 2), you can do this procedure in SPIRES; however, RECOVER and SPIBILD commands that process the file other than PROCESS are available only in SPIBILD.
in SPIBILD, depending on your preference (see below), issue either the RECOVER or PROCESS command:
RECOVER [ORV.gg.uuu.]filename PROCESS [ORV.gg.uuu.]filename
Note that the PROCESS command in this context means any SPIBILD command that processes the file, such as the BATCH, MERGE and ESTABLISH commands -- they will all have the same effect in terms of recovery.
Also note that if you are processing the file in SPIRES, you follow a slightly different procedure here at step 2: first, SELECT a subfile of the file and then issue the PROCESS command in this form:
PROCESS [NOWARN]
NOWARN tells SPIRES not to prompt you for confirmation that you want to process the file.
It may seem odd to try recovering from the problem before determining just what the problem is, but in fact, you can easily waste a lot of time trying to understand the problem, only to discover that you can solve it simply by issuing either the RECOVER or PROCESS command. Since issuing either of these commands cannot make the situation worse, but stands an excellent chance of fixing it, giving that a try is usually worthwhile.
Here are some notes to help you choose between PROCESS and RECOVER:
The PROCESS command:
- tries to completely process the file. If it cleans up the problem, then the deferred queue will be empty when the command is completed.
- may execute for a long time if the deferred queue is large.
- can be done in SPIRES, e.g., under protocol control.
The RECOVER command:
- does only a limited amount of work. (Technically speaking, it processes two pass stacks' worth of data.) For most files, where the deferred queue is not large, that will empty the deferred queue, making RECOVER equivalent to PROCESS. But for files with large deferred queues, the RECOVER command will leave records in the deferred queue, waiting for the next processing of the file.
- will not execute for very long. Thus, it may be preferable to PROCESS in situations when you need to restore the JOBGEN flag as quickly as possible so that users can resume updating records in the file.
- is available only in SPIBILD.
The sample session below shows the successful execution of the RECOVER command:
-> spibild
-Welcome to SPIBILD
-? recover bionotes
-Begin passing of 32 records of record-type REC01
-Completed passing of 32 records total
-Begin passing of 30 records of record-type REC01
-Completed passing of 62 records total
-Processed: 62 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.BIONOTES
08/11/90 File Read/Write Counts Information
File: GQ.JNK.BIONOTES Reads Writes
CKPT data set 4 1
DEFQ data set 6 120
MSTR data set 5 0
Residual data set 7 65
Rec-type REC01 2 58
Rec-type ZIN02 1 20
Total for file: 25 264
-Compute time: 0.258 seconds
-Elapsed time: 2.310 seconds
-Core usage: 0210/0939
-?
If no errors are indicated, then the recovery was successful, users can begin using your file again, and the procedure is complete.
Note: If SPIBILD would create more than two pass stacks' worth of data, then it quits the RECOVER command without processing the file, and exits SPIBILD. Hence, in the example above, there was only enough data in the DEFQ for two pass stacks; had there been more, the message "Processed: ..." and those that follow it would have been replaced by "Exiting SPIBILD". Though the file wouldn't have been completely processed, it would be back to a useable state again.
However, if errors are indicated, as in the example below, you will need to continue to step 3:
-? recover orv.lo.ony.tunes -Action E170 error -While processing record, key = 2 -Failure when goal record-type = REC01 -Exiting SPIBILD Command>
Notice that failure invariably invokes an automatic exit from SPIBILD.
You can find out what went wrong by examining the display from the SPIBILD session, in particular, by examining the error messages and using the EXPLAIN command to find out details about specific errors. The error explanations should help you determine whether you can fix the problems or whether you should contact your SPIRES consultant. In any event, more work will be required before the file can be processed properly.
Everyone knows that having backup copies of computer files is a good idea, but the more reliable computer systems become, the less important it may seem. In fact, few of the thousands of SPIRES files created at Stanford have ever needed restoring from the system-created backup tapes due to system problems. But that fact begs the question -- history shows a number of occasions when file restoration has been necessary. In many cases, though, the problem was caused by a file owner accidentally issuing a destructive command (such as ZAP FILE) or making an unfortunate programming decision that destroys the data integrity of the file.
File backups are thus a crucial ingredient of SPIRES applications. File owners should be aware of what file backup is automatically done, what they themselves can do in addition, and what to do if they need to use the backup data sets to restore a SPIRES file.
Backup of SPIRES files is available in four forms:
If you think you need to restore a SPIRES file using the backup files created by either of those two methods, contact your SPIRES consultant for assistance.
Procedure: ORVCOPY backup Purpose: Make backup copies of data sets of a SPIRES file Who can execute it: file owner Basic steps: 1. Create the ORVCOPY JCL. 2. Run the job.
The batch program ORVCOPY, the ORVYL file dump/restore utility program, lets you back up the data sets of a SPIRES file to tape or disk and later restore them to the ORVYL file system. Besides the data itself, attributes of the file such as account-access permits are copied too.
This procedure describes the technique a file owner would use to make a backup copy of a SPIRES file stored on his or her account, storing it on tape. The reverse, restoring the copy back to the ORVYL file system, is covered briefly in the next section. [The document "ORVYL User's Guide" contains complete information about ORVCOPY, including information about storing the data on disk and dealing with other accounts.]
Note though that an ORVCOPY job can copy only data sets belonging to the account under which the job is run. Similarly, it can restore data sets only to the account under which it is run. But the dump job and the restore job can be run on separate accounts, providing a way in which data sets can be moved from one account to another. If you are moving data between accounts with ORVCOPY, be sure that all accounts have access to the tape or disk data sets to which the data is dumped.
Warning: if you use this procedure to move a SPIRES file from one account to another, the MSTR data set of the file, which contains the name of the file (including the owner's account number), must be changed as well. This change must be done by hand, as described at the end of the next section. [See 2.1.4.5.]
Use the following JCL for an ORVCOPY job to copy data sets:
1. // JOB [TIME=n] 2. /*SETUP T OUTPUT=tapeid 3. // EXEC ORVCOPY 4. //OUTPUT DD DSN=WYL.gg.uuu.name,DISP=(NEW,CATLG), 5. // UNIT=TAPE[,VOL=SER=volume] 6. dump commands
For general information about batch jobs and JCL, see the "Batch Processing" manual, available from Document Sales. Specific information about each statement in the ORVCOPY JCL appears below, and a sample job follow that.
Below are descriptions of the JCL statements you'd use to run an ORVCOPY job to copy ORVYL data sets to a new rental tape or empty tape belonging to you:
1. // JOB [TIME=n]
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number or output bin. See "Batch Processing" for more information.
Depending on how many blocks you are copying, you may need to add the TIME parameter to request more than the default 30 CPU seconds. As a rule of thumb, figure about 450 blocks can be copied per CPU second.
2. /*SETUP T OUTPUT=tapeid
The SETUP statement tells the system that the job requires tapes to be mounted. There are many different values for "tapeid", but the most common are:
3. // EXEC ORVCOPY
The EXEC statement calls the ORVCOPY program.
4. //OUTPUT DD DSN=WYL.gg.uuu.name,DISP=(NEW,CATLG), 5. // UNIT=TAPE[,VOL=SER=volume]
The OUTPUT DD statement tells the program the destination for the copy of the ORVYL data sets. Be sure the name you type for the DSN parameter is unique to your account; it must follow the naming conventions for OS data sets. (If you can't give it a unique name, replace CATLG with KEEP in the DISP parameter.) If you use your own tape rather than a scratch tape, include the VOL=SER parameter with the volser number, as given in the SETUP statement earlier.
6. dump commands
The DUMP command tells ORVCOPY what data sets to copy. You can include as many DUMP commands as you need. A single command can copy from one to all of your ORVYL data sets, depending on the option specified:
DUMP option
where "option" is one of the following:
NAME=filename[,RENAME=name] - copy the single named ORVYL data set,
giving the copy a new name if the
RENAME option is specified
LIKE=string - copy all data sets with names beginning
with the given string
RANGE=name1,name2 - copy data sets "name1" and "name2", and
all those whose names fall between them
in alphanumeric order; "name1" must come
before "name2" in alphanumeric order.
ALL - copy all data sets belonging to your account
Do not include the "ORV.gg.uuu." file prefix in any of these options. It is not necessary since you can only copy files on the account under which the job is run.
Be sure to put a blank between the DUMP command and the option you use; however, no other blanks should appear in the DUMP command.
Here is a sample job run by account GQ.DOC that uses ORVCOPY to copy the data sets of the file GQ.DOC.ERRATA:
// JOB /* SETUP T OUTPUT=SCRTCH // EXEC ORVCOPY //OUTPUT DD DSN=WYL.GQ.DOC.ERRATA.BACKUP,DISP=(NEW,KEEP), // UNIT=TAPE DUMP LIKE=ERRATA
When you want, issue the RUN command to submit the job. After it runs, you should examine the output to get the assigned tape number if you asked for a rental tape. You will need it later if you need to restore the files. [See 2.1.4.5.]
Again, see the document "Batch Processing" for information on running jobs, fetching the output, releasing tapes, etc.
Procedure: ORVCOPY restore Purpose: Restore a SPIRES file using backup data sets Who can execute it: file owner Basic steps: 1. Create the ORVCOPY JCL. 2. Run the job.
In the previous section, the procedure for creating a copy of the ORVYL data sets comprising a SPIRES file using ORVCOPY was discussed. In this section, its less frequently used complement will be covered.
If you need to restore a SPIRES file to a version you created using ORVCOPY, you also use ORVCOPY to get it back. Besides the data itself, attributes of each data set such as account-access permits are reset when ORVCOPY restores a file.
This procedure describes the technique a file owner would use to restore a SPIRES file stored on his or her account, using a copy stored on tape. The reverse, creating the copy, was discussed in the previous section. See the document "ORVYL User's Guide" for full information about ORVCOPY.
Use the following JCL for an ORVCOPY job to restore data sets:
1. // JOB [TIME=n] 2. /*SETUP T INPUT=CAT 3. // EXEC ORVCOPY 4. //INPUT DD DSN=WYL.gg.uuu.name,DISP=OLD 5. restore commands
For general information about batch jobs and JCL, see the "Batch Processing" manual, available from Document Sales. Specific information about each of the statements in the ORVCOPY JCL appears below, and some sample jobs follow that.
Below are descriptions of the JCL statements you'd use to run an ORVCOPY job to restore ORVYL data sets from a tape.
1. // JOB [TIME=n]
You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number or output bin. See "Batch Processing" for more information.
Depending on how many blocks you are restoring, you may need to add the TIME parameter to request more than the default 30 CPU seconds. As a rule of thumb, figure about 450 blocks can be restored per CPU second.
2. /*SETUP T INPUT=CAT
The SETUP statement tells the system that the job requires tapes to be mounted. Specify the CAT value if the data set named in line 4 below is cataloged -- the system will find the tape volume serial number in the catalog. If the data set is not cataloged (e.g., it wasn't uniquely named), replace CAT with the volser number of the tape, which you will also need to add to line 4 (see below).
3. // EXEC ORVCOPY
The EXEC statement calls the ORVCOPY program.
4. //INPUT DD DSN=WYL.gg.uuu.name,DISP=OLD
The INPUT DD statement tells the program the source for the copy of the ORVYL data sets. If the data set is not cataloged, include the UNIT=TAPE parameter and the VOL=SER parameter with the volser number.
5. restore commands
The RESTORE command tells ORVCOPY what data sets to restore. You can include as many RESTORE commands as you need. A single command can restore from one to all of the data sets in the INPUT DD data set, depending on the option specified:
RESTORE option [REPLACE]
where "option" is one of the following:
NAME=filename[,RENAME=name] - restore the single named ORVYL data
set, giving the copy a new name if
the RENAME option is included
LIKE=string - restore all data sets with names
beginning with the given string
RANGE=name1,name2 - restore data sets "name1" and "name2", and
all those whose names fall between them
in alphanumeric order; "name1" must come
before "name2" in alphanumeric order.
ALL - restore all the data sets on the tape
Don't include the "ORV.gg.uuu." file prefix in any of these options; also, don't put blanks within the option (for instance, between LIKE and the equals sign).
Include the REPLACE option if you want ORVCOPY to replace existing data sets with the same names as those being restored. If you leave it off, ORVCOPY will only restore files that don't currently exist in ORVYL.
Below is a sample job run by account GQ.DOC that uses ORVCOPY to restore the data sets of the file GQ.DOC.ERRATA. It is the counterpart to the example in the previous section.
// JOB /* SETUP T INPUT=CAT // EXEC ORVCOPY //INPUT DD DSN=WYL.GQ.DOC.ERRATA.BACKUP,DISP=OLD RESTORE LIKE=ERRATA
When you want, issue the RUN command to submit the job. After it runs, check the file-creation date of each data set by adding the DATE option to the SHOW FILES command. See Part 6, or online, [EXPLAIN SHOW FILES COMMAND.] Those that were restored will have the date they were restored listed for "CR", the creation date.
See the document "Batch Processing" for information on running jobs, fetching the output, releasing tapes, etc.
As mentioned before, this procedure can be used to move data sets from one account to another, which means you could use it to move an entire SPIRES file from one account to another. Generally speaking, it is preferable (and cheaper) to do that by following the procedure described later in this manual. [See 2.10.]
But if you have used this procedure to move a SPIRES file from one account to another, the MSTR data set of the file, which contains the name of the file (including the owner's account number), must be changed as well. This change is best done with a special program called "fixmstr", because "recompiling" the file will not change the account to the correct one, and, in fact, SPIRES will not even allow you to recompile the file since the owning account number listed in the MSTR does not match the account the file is stored on. To change the account number and/or the file's name in the MSTR:
Command> call fixmstr acc gq.rlg (filename) or Command> call fixmstr acc gq.rlg Specify filename: <you supply the name at this prompt>
Note that you must call fixmstr from the account of the new file owner, and filename is just the file's name without the .MSTR suffix.
Procedure: COPY DEFQ
Purpose: Copy the deferred queue of a file belonging to
another account
Who can execute it: users with Copy access
Basic steps:
1. Enter SPIRES.
2. Use the COPY DEFQ command to copy the deferred queue.
Application designers constructing complex applications usually want to make sure there are adequate ways to back out any changes the application has made to files when problems occur. Frequently, doing the application's work using transaction groups or getting rid of changes with the ZAP DEFQ command is an adequate solution.
On the other hand, another method is to make backup copies of as much as possible before starting to make a complicated series of changes. In most cases, copying the entire file would be overkill, though that is certainly possible, thanks to the COPY FILE command. [See 2.10.] Another solution is to copy only the data sets of the file that would be changed by the impending transactions, in particular, the DEFQ data set. [If the file has immediate indexes that would be changed, you would either want to copy them first as well, somehow, or use some other method.]
The COPY DEFQ command lets you copy the deferred queue data set of a file that doesn't belong to you, if you have been granted Copy access to that file by the file owner. [See 2.10, step 2 to see how to do that.]
Restoring the deferred queue, if necessary, must be done by the file owner, of course. This can be accomplished either indirectly, by dumping the data set to tape and restoring it on the file owner's account with the ORVCOPY program [See 2.1.4.4, 2.1.4.5.] or directly, by using ORVYL's SET CLP, ERASE and CREATE commands. See your SPIRES consultant if you need help restoring the deferred queue.
Note that you cannot use those ORVYL commands to make the original copy from the file owner's account because you can't issue the SET CLP command to get copy access to someone else's SPIRES files; only the file owner can do that. (Remember, not being able to run the application on the file owner's account was one of our original assumptions.) The COPY DEFQ command circumvents that problem.
The COPY DEFQ command is available only in SPIRES. Issue one of those commands online, or in a BATWYL job:
Command> spires -Welcome to SPIRES ... if in trouble, try HELP ->
In a batch SPIRES job, you are already in SPIRES, so you omit this command from the command stream.
The COPY DEFQ command makes a copy of the deferred queue data set of the named file, saving it as an ORVYL data set on your account:
COPY DEFQ ORV.gg.uuu.filename [SHARE]
It must be issued from the copying account, NOT from the file owner's account. As it begins copying the data set, SPIRES displays a message showing how many blocks must be copied:
-> show eval $account GQ.JNK -> copy defq orv.gq.doc.restaurant Start copy of 50 blocks from data set DEFQ ->
See Part 6, or online EXPLAIN COPY DEFQ COMMAND, if you have any problems or need more information about the command.
What you do with the copied DEFQ is up to you. Some of the possibilities were discussed above. But chances are good that your application will work successfully, and you will not need the copy. Assuming that's true, at some point, you will want to get rid of it. (WARNING: Be sure your previous copy doesn't still exist the next time you use the COPY DEFQ command for the file -- if it does, the command will fail with an S40 error.)
To discard the copied DEFQ, issue these two commands:
SET CLP filename.DEFQ ERASE filename.DEFQ
where "filename" is the name of the copied file, without any account prefix, unless you'd like to include your own, preceded by "ORV.".
Earlier in this chapter, in section 2.1.2, we discussed some details about the SPIBILD environment, such as the command prompt, the EXPLAIN command, the ATTN/BREAK key, etc. That discussion centered around many SPIRES commands and concepts that carry over into SPIBILD. Here in section 2.1.5 and its subsections, we will consider some other aspects of using SPIBILD, aspects that either differ considerably from SPIRES or have no SPIRES equivalent.
For example, a problem arises when you want to process a file in SPIBILD when it is selected by other users. SPIBILD will not process the file but will instead give you an error message (S56).
SPIBILD has several commands that themselves do little or no work but which affect the way the primary SPIBILD commands work. For instance, the SET SEARCH command does nothing at the time it is issued, but when you issue a subsequent PROCESS command, it lets SPIBILD process the file even though users have the subfile selected.
Problems such as that, and the commands that solve them, are discussed in this section. Specific details about each command, including its syntax, appear at the end of the manual.
Note: All these commands must be issued before the command they are meant to affect -- they establish conditions under which the processing command (e.g., BATCH) is to operate. Also, some of the conditions are cleared after the processing command executes and must be reset if further SPIBILD processing under those conditions is desired. Be sure to see the detail section for each command if you need that information.
Specifically, here are the topics of the subsections of section 2.1.5:
Procedure: SET SEARCH/SET SHARE
Purpose: Get access to a file being used by others so
that you can do SPIBILD tasks; let users
use file for searching during SPIBILD work
Available in: SPIBILD, FASTBILD
Who can use it: file owner
users with Process access to the file
Basic steps:
1. Issue the SET SHARE or SET SEARCH command.
2. Issue the file-processing commands.
You cannot do any SPIBILD task of consequence to a file when it is being used by other accounts, unless you issue the SET SEARCH or SET SHARE command first:
-? process gq.jnk.almanac -System ATCHFILE error, code=S56 -Compute time: 0.019 seconds -Elapsed time: 0.256 seconds -Core usage: 0051/0939 -? explain s56 S56: The file is busy, not currently available. -?
How can SPIBILD tell you, the file owner, that your own file is not available to you? The reason is that the processing activity can interfere with the current users' activities, causing their terminal sessions to be locked up and invalidating their search results. Hence, SPIBILD protects them by blocking the processing commands till the file is free. [If your file has subfile logging in effect, you may be able to determine who is currently using your file. [See Part 6 for PERFORM USER LIST; online, use the EXPLAIN command.] Additionally, ORVYL can provide the information to you through the command "SHOW VFILES LIKE ORV.gg.uuu.filename". Type HELP SHOW VFILES for information about that command.]
However, times arise when it is imperative for you to process a file even though it is in use by others. The SET SEARCH and SET SHARE commands tell SPIBILD (and FASTBILD) to let your processing commands through. (Note: This problem does not occur when you process a file in SPIRES; SET SEARCH, explained below, is automatically in effect.)
In addition, the SET SEARCH command lets users continue to use the file for searching while the file is being processed. This is important for large files that might take a long time (say, over 10 minutes) for processing, especially files that must be available for searching 24 hours a day. [Note: no subfile logging can occur during the processing, so if you are logging subfile use, you will not receive any log entries for that period of time. [See 4.3.]]
The SET SHARE command, on the other hand, simply locks users out of the file till the SPIBILD or FASTBILD processing is completed, which means that users' terminals may get locked till your processing is over. For users trying to use the file for searching, this can be quite irritating.
But oddly enough, for users entering data, particularly with a prompting format or in a full-screen environment such as Prism, it may actually be preferable to SET SHARE. Under SET SEARCH, a record-transaction command such as ADD will cause an error (usually S494) at the end of the processing, discarding the input; under SET SHARE, the terminal would lock up, but eventually the transaction would occur.
Hence, SET SEARCH is most often used for files whose processing takes a considerable amount of time (10 minutes or more). The processing is probably done overnight when users are not trying to enter data; but the file is still needed for searching by nighttime users. SET SEARCH is usually added to the command stream that handles the regular processing.
SET SHARE, on the other hand, tends to be used on an ad hoc basis, when a file's deferred queue is small so the processing won't take long. Any user whose terminal is locked by the processing would only be inconvenienced for a moment or two.
You can issue the SET SEARCH command in SPIBILD or FASTBILD if the file you want to work with has a private checkpoint data set. Any file having any immediate indexes has a private checkpoint data set; many others do too. [The SHOW FILES command can tell you whether a SPIRES file has a private checkpoint data set. Look for the data sets for the specific file (i.e., those whose names begin with the file name, such as "filename.RES"). If there is one called "filename.CKPT", that is the private checkpoint data set.]
-? set search -?
If the file does not have a private checkpoint data set, issuing the SET SEARCH command will have precisely the same effect as SET SHARE:
-? set share -?
But neither command has any effect on anyone till the file-processing command is issued.
If you change your mind after issuing either command, you can undo your request with the SET NOSEARCH or SET NOSHARE command. (In SPIRES, SET SEARCH is automatic and unsettable; SET NOSEARCH is not available to change it.)
For example,
-? process gq.jnk.almanac -Begin passing of 11 records of record-type REC01 ... etc. ... -Compute time: 0.117 seconds -Elapsed time: 1.736 seconds -Core usage: 0069/0939 -?
More details about these two commands can be found in Part 6. Online, EXPLAIN SET SEARCH COMMAND or SET SHARE COMMAND.
Procedure: Exception File
Purpose: Place a copy of bad input data into a separate
data set to make corrections easier
Available in: SPIRES, SPIBILD, FASTBILD
Who can use it: file owner
users with Process access to the file
Basic steps:
1. Issue the SET EXCEPTION command.
2. Issue the command that adds the data into the subfile.
3. GET the exception file, correct the data that failed.
4. Resubmit the corrected data.
5. Cleanup: ERASE the exception file.
When input data causes errors, SPIRES programs generally report them to you so that you can correct them before trying to put the data into the subfile again. When input for multiple records is involved, locating the few problem records in the input file can be tedious, however. An "exception file", into which all the data for failed transactions is placed, makes the correction procedure much easier.
Exception files work with input in the standard SPIRES format or in a custom input format, as long as the format does not execute the HOLD Uproc. See the manual "SPIRES Formats" for information about input formats.
The SET EXCEPTION command creates an empty ORVYL data set that the SPIRES program will use to hold any bad data found by the processing command.
SET EXCEPTION orvyl-filename [REPLACE]
Here is the start of a SPIBILD session in which data will be added to a subfile using the INPUT BATCH command:
Command> spibild -Welcome to SPIBILD -? set exception unruly.records -?
If you change your mind and decide not to use an exception file, you can issue the CLEAR EXCEPTION command at this point. Remember, however, that an empty ORVYL data set has been created; step 5 talks about how to erase it.
Issue the commands that create the subfile transactions, such as:
SPIRES: SELECT subfile-name, and one of the following:
- INPUT BATCH - INPUT ADDUPDATE - INPUT MERGE
- INPUT ADD - INPUT ADDMERGE
SPIBILD: ESTABLISH subfile-name, and one of the following:
- INPUT BATCH - INPUT ADDUPDATE - INPUT MERGE
- INPUT ADD - INPUT ADDMERGE
FAST BATCH subfile-name
FASTBILD: FAST BUILD 'subfile-name'
During the processing of the input command, where data is coming from the active file or some other external file (e.g., from a tape), any data that causes a serious-level error, which prevents that record transaction from succeeding, is placed in the exception file.
The exception file will not work properly if lines of the input file exceed 235 characters, the maximum WYLBUR line length. That problem could arise only in batch SPIBILD and FASTBILD, in which case the lines are truncated.
Another significant limit is that no single record in the input file can exceed 80 blocks (about 160K bytes); that is larger than a record can be in SPIRES, but the limit could be significant if the input data for the record contains data that is discarded during the input process, perhaps by the input format.
The example below, continuing the SPIBILD session started above, begins with a SET MESSAGES command that suppresses messages about individual records that succeed in getting into the subfile; only the failures will get individual messages. [See 2.1.5.3.] In this simple example, we will batch two records into the subfile, the second of which contains an error:
-? set information messages = 0 -? use input.records <-- put the records in our active file -? establish biblio -Processing file: GQ.JNK.BIBLIOGRAPHY -Compute time: 0.123 seconds -Elapsed time: 1.966 seconds -Core usage: 0065/0933 -? input batch -Element=HAVE.READ: -Serious data error, code=E49 -Error at or before line 5. -Update terminated, code=S261 -Error at or before line 6. -*ADD 2, 1 @Line 6 Key=341 Err = S261 -Begin passing of 1 record of record-type REC01 -Completed passing of 1 record total - Requests/Success: ADD 2 1 SUM 2 1 -End of BATCH/BUILD ... etc. ... -Core usage: 0257/0933 -?
It is useful to have a listing of the errors, especially if there are many of them; the line numbers are particularly useful. You might want to consider running a batch SPIBILD job so that the job's output will have the error messages in it, which you can print and keep by your side as you make corrections to the data.
On the other hand, you might want a summary of the number and kinds of errors that occurred during the processing. To display this information, which is accumulated by SPIRES and SPIBILD, you can use the SHOW ERROR COUNTS command. See Part 6, or online, [EXPLAIN SHOW ERROR COUNTS COMMAND.]
Be aware that when the processing command ends, an automatic CLEAR EXCEPTION is executed. Thus, if you issue another processing command where you want an exception file, you should issue another SET EXCEPTION command. You cannot use the same exception file again, however, unless it is empty or you are willing to replace it with the new data.
Issue the GET command to place the exception file data into your active file:
GET exception.filename [CLEAR] [RENUMBER]
(GET is an ORVYL command, similar to WYLBUR's USE command.) You can then correct the data as necessary.
All the input data for each rejected record is placed in the exception file, including any record separators. For example, if the input data is in the standard SPIRES format, rejected records would be put in the exception file with the "command" that preceded them (e.g., "ADD;" or "UPDATE 492;") and the extra semicolon at the end.
In addition, the original line numbering of the input data set is maintained. So, if the first rejected record is on lines 12 through 21 of the input data set, it will have the same line numbers in the exception file. [If several data sets at once are being input, the line numbers could get entangled. The fact that the line numbers are retained could, in fact, cause a problem, with the result that the GET command would not "get" the entire exception file. You'll know the problem occurred if ORVYL responds to the GET command with the warning message "Command does not replace". In that case, issue the GET command again, adding the RENUMBER option.]
The example continues:
-? get unruly.records clear
-? list 5 <-- the line number from the error message
5. have.read = mo;
-? change "m" to "n" in 5
5. have.read = no;
-?
If the exception file is so large that the GET command cannot place the entire file in your active file, contact your SPIRES consultant for help.
Once you have corrected the data, you need to resubmit it for processing. How you do that depends on various factors. For instance, if you used FASTBILD to enter the records, you will not be able to use it for the corrections since FASTBILD requires that the record-type be empty when you start. (Of course, if every single record failed, the record-type is still empty, but in that case, the exception file did you no good.)
But if you used SPIBILD, you will likely use the same command to resubmit the corrections. If so, remember that you can set another exception file in case you have any more problems.
So, next in the example, we batch the corrected records into the subfile:
-? establish biblio -Attached file: GQ.JNK.BIBLIOGRAPHY -? input batch <-- now batch in the corrected records -Begin passing of 1 record of record-type REC01 -Completed passing of 1 record total - Requests/Success: ADD 1 1 SUM 1 1 -End of BATCH/BUILD ... etc. ... -?
When you are finished with an exception file, issue the ORVYL command ERASE to erase it.
-? erase unruly.records UNRULY.RECORDS erased in ORVYL File System -?
You can control the type of messages that SPIRES, SPIBILD and FASTBILD display during a session with the SET MESSAGES command. In most file-management situations, the default setting for each program works fine. In fact, probably the most common use of the command is in protocols and formats, where SET MESSAGES is used to suppress system messages from SPIRES, allowing the application to create and display its own.
Its most common use in SPIBILD is to suppress information messages when you are updating a subfile (e.g., using the BATCH or MERGE commands in SPIBILD). By default, SPIBILD prints a line of information for each record processed:
-> spibild
-Welcome to SPIBILD
-? establish trade stats
-Processing file: GQ.JPR.TRADE.STATS
-Begin passing of 6 records of record-type REC01
-Completed passing of 6 records total
-Processed: 6 DEFQ records of record-type REC01
-Compute time: 0.168 seconds
-Elapsed time: 2.938 seconds
-Core usage: 0210/0939
-? input batch
- ADD 1, 1 @Line 1. Key = USA1936
- ADD 2, 2 @Line 13. Key = USA1937
- ADD 3, 3 @Line 22. Key = USA1938
...
- ADD 47, 47 @Line 800. Key = USA1982
-Element=AG.IMPORTS: Value must be numeric -Serious data
error, code=E49
-Error at or before line 818.
-Update terminated, code=S261
-Error at or before line 821.
-*ADD 48, 47 @Line 821. Key = USA1983 Err = S261
- ADD 49, 48 @Line 835. Key = USA1984
-Begin passing of 48 records of record-type REC01
-Completed passing of 48 records total
- Requests/Success:
ADD 49 48
SUM 49 48
-End of BATCH/BUILD
-Detaching File: GQ.JPR.TRADE.STATS
08/11/90 File Read/Write Counts Information
File: GQ.JPR.TRADE.STATS Reads Writes
CKPT data set 4 1
DEFQ data set 4 3
MSTR data set 4 0
Residual data set 5 16
Rec-type REC01 1 3
Rec-type ZIN02 2 0
Total for file: 20 23
-Compute time: 3.182 seconds
-Elapsed time: 20.839 seconds
-Core usage: 0279/0939
-?
The information about successful transactions can be interesting, but the information about failed transactions, which is usually more important, can easily get overlooked. Moreover, the information messages slow down the processing: each information line for a record requires SPIBILD to stop its current processing, create a message and request a terminal I/O to send you the information.
If you set the information messages to the "0" level from the default level of "2", SPIBILD still gives you the most important information as well as information about any failed transactions. The session above would look like this if information messages were turned off:
-> spibild -Welcome to SPIBILD -? set information messages = 0 -? establish trade stats -Processing file: GQ.JPR.TRADE.STATS -Passing: 6 records of record-type REC01 -Processed: 6 DEFQ records of record-type REC01 -Compute time: 0.168 seconds -Elapsed time: 2.938 seconds -Core usage: 0210/0939 -? input batch -Element=AG.IMPORTS: Value must be numeric -Serious data error, code=E49 -Error at or before line 818. -Update terminated, code=S261 -Error at or before line 821. -*ADD 48, 47 @Line 821. Key = USA1983 Err = S261 -Passing: 48 records of record-type REC01 - Requests/Success: ADD 49 48 SUM 49 48 -End of BATCH/BUILD -Compute time: 3.182 seconds -Elapsed time: 20.839 seconds -Core usage: 0279/0939 -?
The only change is that SPIBILD does not display the information lines for the successful records.
Turning off the other types of messages (Serious, Warning) is not recommended for file-processing in SPIRES nor for SPIBILD and FASTBILD tasks, since serious problems could be masked. And, generally speaking, there is not much reason to set messages to any other level than 0 or 2 in SPIBILD and FASTBILD. See Part 6 for more information, or online, [EXPLAIN SET MESSAGES.]
Occasionally, emergency situations may arise where you don't want to pass data to one or more indexes, or where you only want to pass data from one of the goal record-types in the file. [Production applications may have similar needs that are routine, but such applications are uncommon.] The commands described below, available only in SPIBILD, can be used as temporary measures to control the record-types that are passing data or those that are receiving data:
- SET INDEX record-names. This command names the record-types that can receive passed information, chosen from the file's index record-types. No passing will be done to any other record-types. It and SET NOINDEX below affect only the primary work of any SPIBILD command, not any preliminary file-processing the command may do.
- SET NOINDEX record-names. This command names the record-types that are not to receive passed information as they normally would. All other index record-types will receive passed data as usual.
- SET PASS [n]. This command names the number (1 to 64) of the logical record-type that is to pass data, chosen from the file's goal record-types. No other goal record-types in the file will pass any data. If no number is specified, then all goal record-types will pass data as usual. SET PASS by itself is usually issued to countermand a SET NOPASS command. This command and SET NOPASS below affect all processing work of any SPIBILD command, including any preliminary file-processing the command may do.
- SET NOPASS [n]. This command names the number (1 to 64) of the logical record-type that is not to pass data. All other goal record-types will pass data as usual. If no number is specified, no index passing will occur. This command is necessary when using the FAST BATCH command in SPIBILD, and also has uses when you know that updating activity hasn't caused changes that will affect the indexes. [See 2.4.3.2.]
Intended for use in solving data integrity problems, all four of these commands can make such problems worse if not used very carefully. You should probably discuss the problem with your SPIRES consultant before using any of these commands. See Part 6 for full details about each of these commands, or, online, use the EXPLAIN command.
The SET REINDEX command may also be useful.
[See 6.2.10.]
Note: Another way to prevent any passing at all during a processing operation is to code the NOPASS statement in the file definition. The "nopass" condition would stay in effect for all SPIBILD tasks until the NOPASS statement was removed from the definition. See the SPIRES manual "File Definition" for details.
Procedure: Reporting in SPIBILD
Purpose: Write report data during SPIBILD input
processing
Available in: SPIBILD
Who can use it: file owner
users with Process access to the file
Basic steps:
1. DEFINE AREA areaname(rows,cols) ON FILE
2. ASSIGN AREA areaname TO orvyl.dataset.name [REPLACE]
3. Continue the session, including BATCH or MERGE task
4. CLOSE AREA areaname
When you use the SPIBILD commands BATCH or MERGE for subfile transactions and you use an input format to handle the data, the format can simultaneously produce an output report, containing information about the processed records.
Technical coding information on how to create these versatile formats appears in the manual "SPIRES Formats", section B.16.3.
To use them in SPIBILD, you need one or more "device services areas", areas of memory in the computer used to hold data for particular physical or logical devices. Most useful in this situation is the FILE area, which allows you to send the report to an ORVYL data set.
To use the FILE area in SPIBILD for report output, you issue the following commands before the BATCH or MERGE command in SPIBILD:
This command, documented in full in the SPIRES manual "Device Services", defines an area on the FILE area. The name should be the same name given in the IN-AREA statement in the format definition. If you specified "IN-AREA = FILE", then this command is not necessary.
In this command, you assign the area to an ORVYL data set -- that is where the report will end up. If the data set exists already, you can add the REPLACE option to the command.
Issue any other SPIBILD commands you want, including the SET FORMAT command if you have not set the format already. Then do your BATCH or MERGE task. Afterwards, before leaving SPIBILD or starting another task, continue to step 4.
This command detaches the ORVYL data set, allowing you to issue a GET command ("GET orvyl.dataset.name") to examine the report.
Other device services commands are available in SPIBILD, too. They are fully documented in the "Device Services" manual:
BLANK areaname READ areaname SHOW AREAS SHOW AREA areaname SET AREA areaname functions SET WRITE|NOWRITE SET READ|NOREAD
Processing a SPIRES file has three major effects:
- 1) It moves records from the deferred queue to the tree.
- 2) It updates indexes using the information in the deferred queue (except for immediate indexes, which change at the time of the record transaction).
- 3) It empties the deferred queue, discarding overhead data kept for each transaction (e.g., date, time, account doing transaction) and all but the latest copy of records in the deferred queue.
The first effect is important because records in the deferred queue are handled a bit less efficiently than those in the tree. That's because many commands must scan both the deferred queue and the tree when there are records in the deferred queue; if the deferred queue is empty, they work faster. For example, if you were creating a report using all the records in a large file, it would probably be worth your while to create the report when the deferred queue is empty, right after it is processed.
The specific contents of this chapter are:
Many file owners don't work directly with SPIBILD at all. They let a program called JOBGEN handle the processing of their files in SPIBILD automatically. By default, JOBGEN has that responsibility for a file. The authority is taken away from JOBGEN when NOAUTOGEN is in effect.
Each night after midnight, JOBGEN examines the deferred queues of all SPIRES files that have AUTOGEN set. For any file with data in the deferred queue, JOBGEN creates and submits a simple Batch SPIBILD job that processes the file and handles any batch requests submitted through the BATCH command in SPIRES.
The result is that files with data in the deferred queue are processed overnight, when computer rates are likely to be cheaper and the files are less likely to be used by end-users. Thus, the file is "fresh" the next morning, with updated indexes and no records in the deferred queue.
For files with no data in the deferred queue, no processing is needed, so no SPIBILD job is submitted and no charges are assessed.
Whether a file is to be processed by JOBGEN is indicated by the "AUTOGEN flag". The basic AUTOGEN setting for a file is determined in the file definition. By default, files are established with AUTOGEN, meaning that JOBGEN will be responsible for processing the file overnight. However, if the file owner adds the NOAUTOGEN statement to the file definition and compiles it, then the file's basic setting will be NOAUTOGEN, meaning that the JOBGEN program will not process the file. [WARNING: Account GG.PRD, under which JOBGEN is run, must have Read access to the DEFQ data set of the file; otherwise, JOBGEN can't see if any data exists in the deferred queue. By default, when you compile a new file definition, all accounts have Read access to the DEFQ data set. The ORVYL command "SHOW PERMITS filename.DEFQ" will tell you what accounts have access. To allow account GG.PRD access if it has none, issue the commands "SET CLP filename.DEFQ", "SET PERMIT filename.DEFQ FOR GG.PRD READ" and "SET NOCLP filename.DEFQ".]
Important: The file is always reset to its basic setting whenever the file is processed, whether by you or by JOBGEN.
If you are the file owner, you may change the basic AUTOGEN setting at any time by changing the file definition appropriately. You may make the change directly to the file definition itself or use File Definer to create a new definition with the change. Use File Definer only if you still have File Definer input that created the latest version of the definition.
Find the FILE keyword in your File Definer input and add or remove the NOAUTOGEN attribute, depending on whether you want NOAUTOGEN to be the basic setting. Note: if you don't have a FILE statement, then AUTOGEN is the file's basic setting. You can add a FILE input line like this, if you want NOAUTOGEN:
FILE / NOAUTOGEN
If you need help with this change, see the SPIRES manual "File Definer".
Next use File Definer to generate a revised file definition with these commands:
SPIRES ENTER FILE DEFINER INPUT ACTIVE (save your revised input file for later) GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key (recompile the file definition; see below)
If you prefer to change the file definition directly, get the file definition from the FILEDEF subfile (or wherever you keep your working copy). If you want to change the basic setting from NOAUTOGEN to AUTOGEN, you'll find the NOAUTOGEN statement several lines after the key (FILE). Simply remove it and update the record in FILEDEF.
If you want to change the basic setting from AUTOGEN to NOAUTOGEN, you need to add the NOAUTOGEN statement; you can put it right after the key and FILEDEF will move it to its appropriate location (just after the BIN statement), e.g.:
FILE = GQ.JNK.ALMANAC; NOAUTOGEN; (etc.)
Then update the record in the FILEDEF subfile.
Once you've changed the file definition in FILEDEF, recompile it:
RECOMPILE [gg.uuu].filename
If the file definition compiles properly, the basic AUTOGEN setting has changed. If it does not compile, SPIRES will tell you so. You will need to correct the file definition as SPIRES directs.
The file owner and any users with Master access to the file can still turn JOBGEN processing on or off, on a one-time basis, by issuing one of these commands:
SET AUTOGEN SET NOAUTOGEN
when a subfile of the file is selected in SPIRES.
You can see the current setting of the AUTOGEN flag by issuing the command SHOW AUTOGEN:
-> show autogen -AUTOGEN in effect -> set noautogen ->
If you issue the SET AUTOGEN command, then JOBGEN will begin checking the file's deferred queue that evening. The first evening that JOBGEN finds data to process in the deferred queue, it will submit a job to process the file. The file will then be reset to its basic AUTOGEN setting, as determined in the file definition.
On the other hand, if you issue the SET NOAUTOGEN command, then JOBGEN will not do any work for your file until the flag is reset to AUTOGEN. That happens when the SET AUTOGEN command is issued, or, if the basic setting is AUTOGEN, when you next process the file yourself.
When any SPIBILD task terminates abnormally, SPIBILD may set the "NO JOBGEN flag", which specifies that JOBGEN should not cause the file to be processed until the file owner gives the ok. Perhaps more significantly, the NO JOBGEN flag prohibits any data base transactions -- users who try receive S495 errors.
Thus, the NO JOBGEN flag affects more than simply JOBGEN. It stops all subfile transactions and any subfile logging in effect. [See 2.1.4.]
Most of the time, users discover that a problem occurred with the overnight processing of the file sometime later, when they try to make subfile modifications but get S495 errors. But there are other ways to find out too:
- The "log" from the SPIBILD session (either the online messages displayed or the output from the batch SPIBILD job) will indicate that a problem occurred and ended the SPIBILD session. When that happens, SPIBILD sets the NO JOBGEN flag. [Even if the output from JOBGEN-created jobs is normally purged (requested by the file definition statement "BIN = PURGE"), it will be printed if a SPIBILD problem occurs. In fact, if JOBGEN output unexpectedly appears in your bin, be sure to examine it carefully to see if there was a problem.]
Note also that JOBGEN processing jobs include the SET SEARCH command, which allows searching to continue during the file's processing. This can cause problems of its own if users try to update the file during processing. See Part 6 of this manual for more information about SET SEARCH. Online, EXPLAIN SET SEARCH COMMAND.
- The SHOW AUTOGEN command indicates when the NO JOBGEN flag is set:
-> show autogen
-AUTOGEN in effect (NO JOBGEN)
->
- The $DEFQTEST function can be used within applications to determine whether the NO JOBGEN flag is set. [See the manual "SPIRES Protocols", or EXPLAIN $DEFQTEST FUNCTION online.]
Generally, users who discover that the NO JOBGEN flag is set should notify the file owner, particularly if it is interfering with their work (e.g., they cannot add any records).
What should the file owner do? In most situations, he or she should follow the RECOVER procedure, described earlier. [See 2.1.4.2.]
Procedure: PROCESS/ESTABLISH
Purpose: Move records from the DEFQ to the tree; Clear DEFQ
Who can use it: file owner
users with Process access to the file
Basic steps:
1. SPIBILD
2. Issue any SPIBILD special-processing commands desired.
3. Issue the PROCESS or ESTABLISH command.
or
1. SPIRES
2. SELECT subfile-name
3. Issue the PROCESS command.
When you explicitly want to process a SPIRES file, you may be in either SPIBILD or SPIRES. SPIBILD has more special-processing features than SPIRES does, so there be reasons you will need to use SPIBILD rather than SPIRES. On the other hand, SPIRES has some advantages SPIBILD does not, such as the ability to put the processing under protocol control, and better control over messages.
Specifically, here are the significant differences between processing a file in SPIBILD and in SPIRES:
- SPIBILD exclusively has several commands that handle special file-processing circumstances, including SET INDEX, SET NOINDEX, SET PASS, SET NOPASS and SET STACK SIZE.
- If problems arise, recovery operations involving the RECOVER command must be done in SPIBILD. [See 2.1.4.2.]
- File-processing in SPIRES can be controlled from a protocol, with $NO testing for failure; that means a file can be compiled, built and processed within a SPIRES protocol.
- Under file-processing in SPIRES, messages from the procedure can be handled by SET TLOG, so that they will be logged rather than displayed at the terminal. For more information, [EXPLAIN SET TLOG.]
- File-processing in SPIRES simplifies testing when you are building files and testing indexes in SPIRES; you don't need to enter a different program and then return to SPIRES and re-select the subfile.
- For a small or temporary file that you are building in SPIRES, processing the file in SPIRES is more efficient than having immediate indexes do that work, if you were avoiding SPIBILD processing for some other reason.
- "File count" statistics are shown at the end of the SPIBILD process by default; they are not shown in SPIRES (see discussion below).
In SPIBILD, here is the procedure to follow:
Issue the SPIBILD command to enter that program. [See 2.1.2.]
Command> spibild -Welcome to SPIBILD -?
In batch SPIBILD, you are already in SPIBILD, so you omit this command from the JCL.
Several optional commands or procedures may be useful at this point, such as:
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
The chief file-processing command is PROCESS:
PROCESS filename
where "filename" must be in the form "ORV.gg.uuu.filename" if the file does not belong to you.
An alternative is the ESTABLISH command:
ESTABLISH subfile-name
where "subfile-name" is the name of a subfile in the file. The ESTABLISH command is meant for Global FOR tasks in SPIBILD, where more work with the file will be done after the file-processing is finished. Hence, it does not release the file for use by other accounts when it is done, as does the PROCESS command.
In general, use the PROCESS command, unless either:
- another procedure you're following requires the ESTABLISH command (e.g., using Global FOR in SPIBILD); or
- you have forgotten the name of the file but remember one of its subfile names.
In SPIRES, follow this procedure:
Issue the SPIRES command to enter that program.
Command> spires -Welcome to SPIRES ->
In batch SPIRES, you are already in SPIRES, so you omit this command from the JCL.
Select a subfile of the file you want to process.
SELECT [&gg.uuu.filename] subfile.name
For example, the subfile ALBUMS is part of the file called GQ.JNK.RECORDINGS:
-> select albums ->
At this point, if desired, you may issue SET PASSTRACE commands to enable tracing of Passprocs. See the manual "SPIRES File Definition", or online, [EXPLAIN SET PASSTRACE COMMAND.]
In SPIRES, the syntax for the PROCESS command is:
PROCESS [NOWARN]
The NOWARN option tells SPIRES not to ask you to confirm your request, as it does by default.
For example:
-> process -Do you want to PROCESS file GQ.JNK.RECORDINGS? ok -Processed: 39 DEFQ records of record type REC01 Compute time: 0.044 seconds Elapsed time: 1.583 seconds ->
Depending on the size of the deferred queue and the number of indexes to update, file-processing in either SPIRES or SPIBILD could take only a few seconds or up to several hours. But the "average file" with, say, several dozen records to process requires a minute or two of elapsed time. Messages reporting the progress of the processing will appear occasionally. A description of all the messages you will see appears later in this chapter. [See 2.2.4.] or online, [EXPLAIN FILE PROCESSING, MESSAGES.]
Here is how a typical SPIBILD session involving the PROCESS command might look:
Command> spibild
-Welcome to SPIBILD
-? process orv.gq.jnk.recordings
-Begin passing of 14 records of record-type ALBUM
-Completed passing of 14 records total; Key: 15
-Begin passing of 4 records of record-type ALBUM
-Completed passing of 18 records total
-Processed: 18 DEFQ records of record-type ALBUM
-Begin passing of 118 records of record-type XREC19
-Completed passing of 118 records total
-Processed: 118 DEFQ records of record-type XREC19
-Detaching File: GQ.JNK.TRECORDINGS
08/11/90 File Read/Write Counts Information
File: GQ.JNK.TRECORDINGS Reads Writes
CKPT data set 4 114
DEFQ data set 42 244
MSTR data set 26 0
Residual data set 506 559
Rec-type REC02 8 10
Rec-type REC03 156 157
... etc. ...
Total for file: 1038 1537
-Compute time: 4.989 seconds
-Elapsed time: 45.425 seconds
-Core usage: 0267/0939
-?
The messages tell us that this file has two goal record-types, ALBUM and XREC19, each with its own set of indexes to update. SPIBILD processed the 136 DEFQ records in about 45 seconds, using 4.989 real CPU seconds.
Though most file-processing tasks run quite smoothly, problems do occasionally arise. The most common problems encountered during SPIBILD processing were discussed in chapter 2.1. [See 2.1.4.1.] or [EXPLAIN PROBLEMS IN SPIBILD.]
Procedure: PROCESS SUBFILE
Purpose: Move one subfile's records from the DEFQ to the tree
Who can use it: file owner
users with Process access to the file
Basic steps:
1. SPIRES
2. SELECT subfile-name
3. Issue the PROCESS SUBFILE command.
In some situations, you may not want to process the entire file. For example, suppose you have added some records to a subfile that acts as a lookup table for the primary record-type of the file. You need the lookups to begin using the new records right away, but for efficiency reasons, the lookup actions won't look in the deferred queue for the table records. You could process the file, but that could cause problems for the people currently using the primary subfile.
So the PROCESS SUBFILE procedure lets you process only the selected subfile's worth of deferred-queue data. As soon as the record-type is processed, its deferred-queue data is discarded from the defq, leaving any data from other record-types of the file. If there is no other defq data, including logs or batch requests, then the defq is re-initialized, just as if a full-fledged file process had been done.
Issue the SPIRES command to enter that program.
Command> spires -Welcome to SPIRES ->
In batch SPIRES, you are already in SPIRES, so you omit this command from the JCL.
Select the subfile you want to process.
SELECT [&gg.uuu.filename] subfile.name
For example, the subfile ARTIST TABLE is part of the file called GQ.JNK.RECORDINGS:
-> select artist table ->
In SPIRES, the syntax for the PROCESS SUBFILE command is:
PROCESS SUBFILE [NOWARN]
The NOWARN option tells SPIRES not to ask you to confirm your request, as it does by default. It also tells SPIRES not to warn you if you have a result and/or stack in effect for the selected subfile; the PROCESS SUBFILE command must clear any stack or result in order to work successfully.
For example:
-> process subfile -Do you want to PROCESS file GQ.JNK.RECORDINGS? ok -Processed: 23 DEFQ records of record type REC12 Compute time: 0.031 seconds Elapsed time: 1.285 seconds ->
Depending on the size of the deferred queue and the number of indexes to update, subfile-processing could take only a few seconds or up to several hours. But the "average file" with, say, several dozen records to process requires no more than a minute or two of elapsed time. Messages reporting the progress of the processing will appear occasionally. A description of all the messages you will see appears later in this chapter. [See 2.2.4 or online, EXPLAIN FILE PROCESSING, MESSAGES.]
Though most file-processing tasks run quite smoothly, problems do occasionally arise. The most common problems encountered during file-processing were discussed in chapter 2.1. [See 2.1.4.1 or online, EXPLAIN PROBLEMS IN SPIBILD.]
If the process fails, note that the NOJOBGEN flag will be set, blocking subsequent updates for any record-type of the file until the file is successfully processed or recovery takes place. [See 2.1.4.2 or online, EXPLAIN RECOVERING A FILE.]
In practice then, there are three techniques used by file owners to process a file:
- 1) give JOBGEN the responsibility of overnight processing [See 2.2.1.]
- 2) submit a batch job (batch SPIBILD or BATWYL) to process the file on a regular basis, probably through the SUBMIT program [See 2.1.3, 2.2.2.]
- 3) process the file manually whenever desired, using the PROCESS/ESTABLISH procedure [See 2.1.3.]
As file owner, you can change between these methods whenever you like. For regular processing of a file, most file owners choose either the first or second method, and then manually process the file when special needs arise. Alternatively, you could decide to manually process the file most of the time, but on any given day, choose to have JOBGEN take the responsibility for that night only.
Here are some factors to consider when you are deciding what method to use:
The overhead costs of file processing decrease per record as the number of records processed increases. For example, five jobs that each process one new record will cost more than one job that processes five new records at once. If a file has only one or two transactions per day, it might not be worthwhile to have the file processed each night by JOBGEN; a job that processed the file once a week might be preferable.
Generally, the most important reason to process a file is to update the indexes to reflect the record transactions. [See 2.2 for other reasons.] You should consider how important it is that the indexes accurately reflect the current contents of the subfiles. (If constant accuracy is crucial, you probably want to consider making the indexes "immediate indexes", in which case file processing is not so important.)
JOBGEN makes a backup copy of the deferred queue of each file before it submits the job to process it. In conjunction with "dump tapes" made regularly to copy all ORVYL data sets, the backed-up deferred queues can be used to restore a SPIRES file to its state as of the previous night's processing. However, if the file owner has manually processed the file after the dump tape was last made, the data handled by that process would not be available in JOBGEN's deferred queue dump tapes. Many applications that cannot use JOBGEN processing have jobs to process the file that begin by making a backup copy of the file. [See 2.1.4.3.]
Many applications require other work to be done immediately before or after a file is processed. For example, you might want to print a copy of all records being removed, or save a copy of the transactions log of one of the file's subfiles, or make a backup copy of the file. In such situations, you would probably write your own job to process the file, surrounded by JCL and commands to handle the other tasks too.
The log data for subfiles with command logging is data in the deferred queue. Any data in the deferred queue, even if it's only log data, will cause JOBGEN to process a file.
Users can submit "batch requests" for subfile updates in SPIRES (using the BATCH command) that are processed by JOBGEN when it creates the SPIBILD job during overnight processing. Such batch requests, admittedly quite rare, reside in the deferred queue; they are completely discarded if the file is processed by anyone other than JOBGEN, without warning. If there is a chance that users are entering data via the BATCH command in SPIRES, you should either rely on JOBGEN processing the file, or be sure to check for batch requests before processing it yourself. [See "SPIRES Searching and Updating", section D.8, for more information.]
The above list of points to consider when choosing how your file should be processed are only guidelines. You will have to decide what to do based on your own conditions and preferences. Testing the possibilities is highly recommended, particularly if your application is complex.
Remember, it is easy to switch from one method to the other, either for a single processing of the file, or for a longer period of time.
When you process a file, either by an explicit PROCESS command or as part of a SPIBILD command, SPIRES or SPIBILD displays information about the processing as it proceeds. For example, here is a simple SPIBILD session showing the messages SPIBILD provides in most cases:
-? process workshop
-Begin passing of 84 records of record-type REC01
-Completed passing of 84 records total
-Begin passing of 19 records of record-type REC01
-Completed passing of 103 records total
-Processed: 103 DEFQ records of record-type REC01
-Detaching File: GQ.JNK.WORKSHOP
08/11/90 File Read/Write Counts Information
File: GQ.JNK.WORKSHOP Reads Writes
CKPT data set 4 1
DEFQ data set 13 6
MSTR data set 4 0
Residual data set 48 166
Rec-type REC01 1 128
Rec-type ZIND2 2 208
Total for file: 72 509
-Compute time: 8.221 seconds
-Elapsed time: 56.840 seconds
-Core usage: 0244/0939
-?
This section will describe what those messages mean. In addition, it will discuss the file's command log, which may be displayed when you process a file.
The passing messages appear only if the deferred queue contains data that must be passed to indexes. The first message gives you several pieces of information:
- 1) SPIBILD is now working with the deferred-queue records of the named record-type.
- 2) The record-type is a goal record-type with indexes.
- 3) SPIBILD has moved the specified number of records from the deferred queue into the tree for that record-type.
- 4) SPIBILD is beginning to apply data from the "pass stack", an area in memory in which the index data is collected, into the indexes for the record-type. The pass stack contains the index data for that number of records.
This message appears when SPIBILD has finished applying the pass stack to the index record-types, has discarded the pass stack and is ready to start over with the next goal records, if any. The number "n" is a running total of the records of the goal record-type being worked on that have been successfully processed. The key of the record with which SPIBILD is resuming will also appear at the end if there are more goal records of the same record-type to process, as in the example above.
The pass stack has a limited size. If it fills up with data before processing all the deferred-queue records for that record-type, then SPIBILD will display the first passing message, "pass" that data, empty the pass stack, display the "Completed" passing message, and begin filling it up again. That's why the example above shows two pairs of passing messages for record-type REC01 -- the pass stack filled up from the first 84 records. After the index records were updated using that data, SPIBILD began working with the remaining REC01 records in the deferred queue.
The last section of this chapter describes the pass stack in more detail. [See 2.2.6.] or [EXPLAIN FILE PROCESSING, TECHNICAL DETAILS.]
This message appears when all processing is completed for a given record-type. That is, SPIBILD has applied the data in the deferred queue for that record-type to the tree, and updated all the indexes. This message displays the total number of records processed for the record-type.
The order in which the record-types are processed is the order in which they are defined in the file definition: in alphanumeric order by name. For example, REC01 is processed before REC02, which is processed before REC10, which precedes ZIN01.
If the deferred queue contains records for several different record-types, there will be several sets of passing and processing messages.
Many SPIRES files have subfiles for which log information is collected, invoked by the LOG and STATISTICS statements in the file definition. [See 4.3.] If the STATISTICS statement has the value "1", the log information is displayed at this point in the file processing.
The log data may be thousands of lines long. If you want to stop the display, press the ATTN/BREAK key. The log display should stop, and the file processing will resume. (Pressing ATTN/BREAK too many times may cause a session break; if so, simply type GO as directed to resume the processing.)
SPIBILD is finished working with the file and detaches it, as confirmed by this message. When you process a file is SPIRES, you will not see this message; SPIRES does not "de-select" the subfile, so the file remains attached.
SPIBILD (but not SPIRES) summarizes the I/O counts for each data set of the file that was involved in the processing. These messages can be suppressed with the CLEAR FILE COUNTS command. For further information about them, see the SET FILE COUNTS command in Part 6 of this manual; online, [EXPLAIN SET FILE COUNTS COMMAND.]
For SPIRES, to see this information, use the SHOW FILE COUNTS and CLEAR FILE COUNTS commands, which are described in the manual "SPIRES Technical Notes". Online, [EXPLAIN SHOW FILE COUNTS.]
This message displays the number of real CPU seconds used to process the file. At Stanford, this number must be multiplied by a constant to get the CPU value for which you are charged; see your consultant for more information.
This message displays the actual amount of time passed since the processing began.
This SPIBILD (not SPIRES) message displays the number of kilobytes (1024 bytes) in main memory used to process the file (nnnn) compared to the number available (mmmm). The amount available is often three or four times the amount needed. If the amount used gets very close to the amount available (say, within 10K), contact your SPIRES consultant.
When problems arise during file processing, SPIBILD or SPIRES displays appropriate error messages. You can get more information about them by issuing the EXPLAIN command. More information about specific SPIBILD problems appears elsewhere in this manual. [See 2.1.4.1.] or online, [EXPLAIN SPIBILD PROBLEMS.]
Before another user can process or perform any other SPIBILD task on one of your files, there are two actions you must take as the file owner:
The two steps are described in detail below.
You must change and recompile the file definition for this step to take effect. You can either use File Definer to change the file definition or change the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file.
If you are using File Definer to change the definition, add (or change) the PROCESS attribute onto the FILE statement of your File Definer input:
FILE/ PROCESS gg.uuu
where "gg.uuu" is the user's account number. If you don't already have a FILE statement in your input, add it as the first line of the input. You can include multiple accounts, by separating them with commas, or you can specify them in other standard forms, such as "GG...." or PUBLIC to give access to an entire group or the public at large.
Then use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key recompile the file, as described below...
If you are changing the file definition directly, get it from the FILEDEF subfile in SPIRES and add at the end (or change, if it already exists):
FILE-ACCESS = PROCESS; ACCOUNTS = gg.uuu;
You can include multiple accounts, by separating them with commas, or specify them in other standard forms, such as "GG...." or PUBLIC to give access to an entire group or the public at large.
When you've made the change, replace the old version with the new in the FILEDEF subfile.
Once you've changed the file definition, you must recompile it to effect the changes:
RECOMPILE [gg.uuu].filename
The next step is to tell ORVYL to allow the user to write data to each data set of the file except the MSTR. You must issue three separate ORVYL commands for each data set:
SET CLP filename.dsname SET PERMIT filename.dsname FOR gg.uuu WRITE SET NOCLP filename.dsname
where "filename" is the name of your file and "dsname" is the type of data set: CKPT, DEFQ, REC1, REC2, etc. and RES. (To see which RECn data sets exist for your file, type the command "SHOW FILES LIKE filename.REC".)
The other user may now process your file.
Note: some file-management procedures may cause ORVYL to lose the permits established here. In particular, procedures in which a data set is completely replaced or temporarily erased will require you to reset the ORVYL permits you set here. The most notable examples are the Rebalancing procedure and the Rebuilding a SPIRES File procedure.
[Note: The following section provides technical details on the passing procedure used by SPIBILD. It is unnecessary reading for most SPIRES file managers.]
The SPIRES/SPIBILD/FASTBILD processors make use of the Linkage Sections of a File Definition (GOALREC-NAME sections) in the following ways:
Each record type of a file is processed in ascending order by record-type number (all REC1's, then all REC2's, etc.). Goal records associated with each record-type are processed either in the order in which they are input under BATCH or MERGE command processing, or in ascending key order if through the DEFQ. Note that REMOVE, UPDATE and MERGE commands under BATCH processing normally cause DEFQ entries to be created and later processed in key order. ADD commands under BATCH processing are handled in the order they are received.
Limiting the discussion to just the PROCESS command in SPIBILD will simplify what happens, and make it possible for you to determine what happens on a "recovery" situation. For simplicity, assume all records come from the DEFQ.
If the record-type does NOT have an associated Linkage Section (or all INDEX-NAME sections are marked NOPASS), then skip to Phase 2.
For ADD-type records, a copy of the record is first added to the TREE. If the record fails to add because a copy already exists in the TREE, then it is assumed that "recovery" is taking place, and the TREE copy is replaced. The DEFQ record is now sent to the passing process marked for pointer INSERTION.
For UPDATE-type or REMOVE-type records, the original TREE copy of the record is read up and sent to the passing process marked for pointer DELETION. Then, for the UPDATE-type, the record from the DEFQ is sent to the passing process marked for pointer INSERTION.
Note that the values passed should come from data within the goal records themselves. Values created from $variables, like $UDATE, or retrieved from other records ($LOOKUP) may cause index entries that won't be deleted when the goal record is removed or updated. [See 2.2.7, 2.2.8.]
The passing process is controlled by the linkage section. Each linkage section consists of:
- A Common portion before the first INDEX-NAME that defines the pointer (PTR-ELEM) as well as any global qualifiers (QUAL-ELEM).
- Individual INDEX-NAME portions, in which index record-types are named, and index values are defined. There can also be sub-index and/or local qualifiers defined.
For each record received (marked for pointer INSERTION or DELETION), the PTR-ELEM and global QUAL-ELEM portion of the Linkage is processed first. One and only one item is placed in the "pass stack" for each item in this Common portion. Passproc rules $PASS.OCC or A165 may be used to cause no entry, or something like $PASS.ELEM or A167:1 (multiple passer) can be used to cause the first occurrence of the first existing element in the list to be placed in the pass stack. If no value can be found and $PASS.OCC (A165) or $PASS.DEF (A171) was not used, then a null value is placed in the pass stack.
Once the Common portion of the Linkage Section is finished, processing begins on each INDEX-NAME portion. The INDEX-NAME, SUB-INDEX, and local QUAL-ELEM terms constitute one INDEX-NAME portion. These terms are "traversed" in both a forward and backward manner beginning with a forward scan starting with INDEX-NAME and running through the last term. Each term causes one value to be retrieved from the goal record.
If no value can be found during a forward scan, and $PASS.OCC (A165) or $PASS.EDF (A171) was not used, then NO INDEX ENTRY is defined, and backward scan commenses. If the forward scan can be completed, then an INDEX ENTRY is defined, and backward scan commences.
During backward scan, each term attempts to retrieve a value from the goal record from where it left off during the previous forward scan. If another value can be retrieved, then forward scan resumes from this point, and if it continues forward through the last term, then another INDEX ENTRY is defined. However, if during any forward scan, new terms are within the structural bounds of previous terms, then those structural bounds limit the values retrieved to occurrences within those bounds. Terms may be bound by a "common root", which is some structural level from which separate branches to the terms can be traced.
Here are some examples using a single INDEX-NAME section with different layouts of elements in goal records. Note: only significant elements are shown.
INDEX-NAME = index; GOALREC-ELEM = WORD; PTR-GROUP = ptstr; QUAL-ELEM = qual1; GOALREC-ELEM = DATE; QUAL-ELEM = qual2; GOALREC-ELEM = TIME;
The layout of WORD, DATE, and TIME in the goal records can vary in many ways. Structural boundaries can make a big difference in what happens. Consider the following record:
Record-level WORD(1) WORD(2) DATE(1) DATE(2) TIME(1) TIME(2) ------------------|--------|--------|--------|--------|--------|
All elements are doubly occurring, and all are at the record level. The following INDEX ENTRIES will be constructed:
WORD(1) DATE(1) TIME(1) WORD(1) DATE(1) TIME(2) WORD(1) DATE(2) TIME(1) WORD(1) DATE(2) TIME(2) WORD(2) DATE(1) TIME(1) WORD(2) DATE(1) TIME(2) WORD(2) DATE(2) TIME(1) WORD(2) DATE(2) TIME(2)
Now consider the following layout:
TIME(1) TIME(2)
DATE(1) | TS(1) DATE(2) | TS(2)
|--------| |--------|
WORD(1) WORD(2) | DS(1) | DS(2)
-|--------|--------|-------------------|
Here, TIME is in a structure (TS) which has been defined as an element of another structure (DS) along with DATE, and that structure is defined at the record level along with WORD.
The following INDEX ENTRIES will be created:
WORD(1) DATE(1) TIME(1) WORD(1) DATE(2) TIME(2) WORD(2) DATE(1) TIME(1) WORD(2) DATE(2) TIME(2)
After completing the first forward pass, an attempt to pick up another value of TIME on a backward scan would have required leaving the structural bounds defined by the first DATE. On a backward scan, structural bounds defined by a previous element can't be crossed, so the backward scan continues by trying to get another DATE value. The boundary now is the record-level defined by WORD. Since the DATE structure has its root at that level, we can proceed forward again beginning with DATE(2), and then TIME(2). Scanning backward again takes us all the way back to WORD(1) from which we now pick up WORD(2) and scan forward again.
If there had been other occurrences of the TIME structure associated with DATE(1), then we would have picked all of them up along with DATE(1), but none of them with DATE(2). The same thing would have happened if TIME had been simply a multiply occurring element inside the DATE structure instead of being nested within a multiply occurring structure defined inside the DATE structure.
That is,
DATE(1) TIME(1) DATE(2) TIME(2)
|--------| |--------|
WORD(1) WORD(2) | DS(1) | DS(2)
-|--------|--------|-------------------|
Now, consider the following layout:
WORD(1) WORD(2)
|--------|
DATE(1) DATE(2) | WS(1)
|--------|----------|
TIME(1) TIME(2) | DS(1)
-|--------|--------|
Here, WORD is a multiply occurring element of a structure (WS) defined as an element of another structure (DS) containing a multiply occurring DATE element, and that structure is defined at the record-level along with a multiply occurring TIME element.
The following INDEX ENTRIES will be created:
WORD(1) DATE(1) TIME(1) WORD(1) DATE(1) TIME(2) WORD(1) DATE(2) TIME(1) WORD(1) DATE(2) TIME(2) WORD(2) DATE(1) TIME(1) WORD(2) DATE(1) TIME(2) WORD(2) DATE(2) TIME(1)
That's all possible combinations because the entries have a "common root" at the record level. Although the access path is upside-down, there still are structural bounds, beginning at WORD and working down to TIME at the record level. However, the next example shows how bounds can apply, even in upside-down accessing.
WORD(1) WORD(2)
DATE(1) | WS(1) DATE(2) | WS(2)
) |--------| |--------|
TIME(1) TIME(2) | DS(1) | DS(2)
-|--------|--------|-------------------|
Here, WORD is within a structure (WS) which has been defined as an element of another structure (DS) along with DATE, and that structure is defined at the record level along with TIME.
The following INDEX ENTRIES will be created:
WORD(1) DATE(1) TIME(1) WORD(1) DATE(1) TIME(2) WORD(2) DATE(2) TIME(1) WORD(2) DATE(2) TIME(2)
Once all INDEX ENTRIES for an INDEX-NAME section have been placed in the pass stack, the next INDEX-NAME section is processed. An INDEX-NAME section is consider completed when a backward scan for the INDEX-NAME term fails to retrieve a value. When all INDEX-NAME sections have been completed, passing for the record is finished, but the information concerning index updates only exists in the pass stack.
When the pass stack fills up or no more records are retrieved from the DEFQ, then the pass stack is processed by sorting it down to unique INDEX ENTRIES cancelling duplicate entries and any matching INSERTION/DELETION pairs. The result is a list of INSERTION and/or DELETION entries to be applied to the indexes. The appropriate index records are added, removed, or updated using the the pass stack information. Because of the sort, an index record need only be read once in order to apply all the pass stack information associated with that record.
When a Pass Stack's worth of goal records has been passed, those same goal records are reread from the DEFQ. Each REMOVE-type record causes the TREE copy to be removed; each UPDATE-type record is used to replace the TREE copy. Then the DEFQ record is marked as DEQueued. When the group of records that contributed to the the pass stack are all DEQueued, Phase 1 repeats with another group of goal records until all DEFQ record for this record-type have been processed. When done with all records, continue with Phase 3.
All the records in the DEFQ associated with one record-type are read sequentially. For each ADD-type record, the record is added to the TREE. For each REMOVE-type record, the TREE copy of the record is removed. For each UPDATE-type record, the TREE copy is replaced by the DEFQ copy. When done with all records, continue with Phase 3.
When all the records for a particular record-type have been processed, the next logical record-type is processed starting back at Phase 1. When all record-types have been processed, the DEFQ is cleared and the PROCESS command is finished.
Note that Immediate Indexing in SPIRES does Phase 1 for just the single record being immediately indexed, and only those INDEX-NAME terms marked as IMMEDIATE are sent to the pass stack. These same records are usually processed again later by SPIBILD, but then only the non-IMMEDIATE terms are sent to the pass stack because the IMMEDIATE terms are already done. The TREE copy of the record is not REMOVED or UPDATED during immediate indexing. The DEFQ information becomes the latest copy of the record, and may participate in another Immediate Indexing transaction.
Passing to indexes requires you be able to undo that indexing. Here's a simple example of something that works just fine.
RECORD-NAME = GOAL; ....etc; ELEM = MODDATE; LEN = 4; OCC = 1; INPROC = $DATE(UPD); OUTPROC = $DATE.OUT;
An index record might be defined something like this:
RECORD-NAME = INDX; KEY = MODDATE; LEN = 4; INPROC = $DATE; OUTPROC = $DATE.OUT; ELEM = pointer; ....etc;
The Linkage section would probably look like this:
GOALREC-NAME = GOAL;
EXTERNAL-NAME = GOAL;
PTR-ELEM = pointer; ....etc;
INDEX-NAME = INDX;
SEARCHTERMS = MODDATE;
SEARCHPROC = $DATE(TRUNC);
PASSPROC = $PASS(BINARY);
GOALREC-ELEM = MODDATE;
PTR-GROUP = pointer;
If you ADD a new record to the database, MODDATE gets today's date as its value. The Linkage section for GOAL passes MODDATE to the INDX index. Then, the new record is placed in the GOAL record's TREE. [See 2.2.6.]
Now, several days pass and you decide to update this GOAL record. The updated record get's today's date as the value of MODDATE, and the record is placed in the DEFQ. Of course, "today's date" is different than when the record was originally added. When passing occurs, the TREE copy of the record is read and the value of MODDATE stored in that record is sent to the pass stack marked for DELETION.
The record in the DEFQ then passes its MODDATE value to the pass stack marked for INSERTION. The DELETION and INSERTION values are different, so the old value will be un-indexed, and the new value will be indexed.
But here's a simple example of something that doesn't work. To save space, you decide to make MODDATE into a VIRTUAL element, and have it pass the equivalent of the $UDATE variable. Your MODDATE element in the GOAL record is defined as follows:
RECORD-NAME = GOAL; ....etc; VIRTUAL; ELEM = MODDATE; LEN = 4; OCC = 1; INPROC = $DATE; OUTPROC = $CALL(GENDATE); USERDEFS; USERPROC = GENDATE; UPROC = SET VALUE = $DATE;
Everything else remains the same.
If you ADD a new record to this database, MODDATE has no value because it is VIRTUAL. But the Linkage section obtains a value from MODDATE to pass to the INDX, and that value is today's date. Then, the new record is placed in the GOAL record's TREE.
So far, it seems like everything is fine. But now, several days pass and you decide to update this GOAL record. The record is placed in the DEFQ. Again, MODDATE has no value because it is VIRTUAL.
When passing occurs, the TREE copy of the record is read and the value of MODDATE created is today's date, which is sent to the pass stack marked for DELETION. However, that wasn't the date that was originally indexed.
The record in the DEFQ then passes its MODDATE created value to the pass stack marked for INSERTION. The DELETION and INSERTION values are identical! This is a matching INSERTION/DELETION pair, and it is eliminated from the pass stack. The result is that NO INDX entries are changed.
Even if you tried to REMOVE the GOAL record at some date after it was originally added and processed, you would not be able to un-index the original date from the INDX. This is an example of why you should NOT pass values dependent upon variables.
Indexes are usually built from values contained within the GOAL record itself. There may be some transformation of the value, but it is a consistent transformation which will always produce the same index value(s) for a given GOAL value.
For example, consider GOAL and ZINDX records defined as:
RECORD-NAME = GOAL; KEY = something; ....etc; ELEM = ZIPCODE; RECORD-NAME = ZINDX; KEY = ZIPCODE; ELEM = pointer; ....etc;
The Linkage section might contain something like this:
GOALREC-NAME = GOAL;
PTR-ELEM = pointer; ....etc;
INDEX-NAME = ZINDX;
SEARCHTERMS = ZIPCODE;
PASSPROC = $PASS;
GOALREC-ELEM = ZIPCODE;
PTR-GROUP = pointer;
Occurrences of ZIPCODE in the GOAL records are passed to the ZINDX so you can search for GOAL records by ZIPCODE. There's nothing unusual here.
But now you discover there is a separate subfile (call ZIPTABLE) with records that act like a table pairing ZIPCODE with CITY names. Records in the ZIPTABLE subfile might look like this:
ZIPCODE = 94040; CITY = MTN VIEW; ZIPCODE = 94041; CITY = MTN VIEW; ZIPCODE = 94022; CITY = LOS ALTOS; ZIPCODE = 94023; CITY = LOS ALTOS HILLS;
You decide you want to build a CITY index in your database, using your ZIPCODE to lookup CITY in this separate subfile, and passing that value to a word index. You add the following record-type to your file definition:
RECORD-NAME = ZNAME; KEY = WORD; ELEM = pointer; ....etc;
You also add a VIRTUAL element to your GOAL record definition:
VIRTUAL;
ELEM = CITY;
REDEFINES = ZIPCODE;
OUTPROC = $CALL(GETCITY);
USERDEFS;
USERPROC = GETCITY;
UPROC = SET VALUE = $LOOKSUBF($VAL,REPLACE,ZIPTABLE,CITY);
And finally, you add the following to the Linkage section:
INDEX-NAME = ZNAME;
SEARCHTERMS = CITY;
SEARCHPROC = $TRUNC;
PASSPROC = $PASS(PHRASE) / $BREAK;
GOALREC-ELEM = CITY;
PTR-GROUP = pointer;
The first time you add a GOAL record, ZIPCODE is used to lookup CITY in the ZIPTABLE subfile, and that values is broken into words and sent to the ZNAME index.
But what happens if the table changes after you've built ZNAME entries? For example, assume you used 94040 as a ZIPCODE which built ZNAME index entries for MTN and VIEW. Then the 94040 record in the ZIPTABLE subfile is changed like this:
ZIPCODE = 94040; CITY = MOUNTAIN VIEW;
The next time a GOAL record accesses the table for 94040 it will obtain MOUNTAIN VIEW as the CITY, and that is NOT what was originally sent to the ZNAME index. If you tried to REMOVE a GOAL record which had 94040 as a ZIPCODE, the word VIEW would be unindexed, but not the word MTN. Likewise, if you UPDATE a GOAL record which has 94040 as a ZIPCODE, and changed it to 94022, you would unindex the word VIEW, but NOT the word MTN. And then you would index the words LOS and ALTOS. Your record would still be found by: FIND CITY MTN; even though the ZIPCODE is no longer associated with MTN VIEW.
Even if you updated all GOAL records that contained 94040 as the ZIPCODE, and didn't change the ZIPCODE, you wouldn't affect the ZNAME index. That's because both the TREE and DEFQ copies of the GOAL record would retrieve the same 94040 ZIPTABLE record, and would get MOUNTAIN VIEW as the CITY. These terms cancel in the pass stack, and ZNAME is left unchanged. [See 2.2.6.]
This is the danger in passing values obtained by $LOOKSUBF or $LOOKSUBG. The accessed value from such records may not change once they are stored. You can add new table entries with new codes, and you can change the accessing codes in your GOAL record, but you can't allow the accessed values in the table to change.
A better solution, to get what you want without fear of table changes, is to have the ZIPTABLE subfile build its own CITY word index. Then, instead of building a ZNAME index, you code "Indirect Search". [EXPLAIN INDIRECT SEARCHING.]
SPIRES, SPIBILD and FASTBILD provide several ways for you to add multiple records to a subfile at once. The different methods work under different circumstances -- which one you choose will depend on the particular situation you have.
Below is a comparative summary of the methods. The procedures involving SPIBILD are described in detail in the rest of this chapter; those involving FASTBILD are described later in Part 2. [See 2.12.] Information about the SPIRES procedures appears in the SPIRES manual "Searching and Updating".
This chapter ends with a short section about the format of the data in input data sets used with these procedures [See 2.3.5.] and a section on how to resume data input if a problem terminates your input processing in SPIBILD using one of the methods described in this chapter. [See 2.3.6.]
There are 14 techniques for adding multiple records to a subfile in a single shot:
- 1. (SPIRES) the INPUT BATCH procedure
- 2. (SPIRES) the INPUT ADD procedure
- 3. (SPIRES) the INPUT ADDUPDATE procedure
- 4. (SPIRES) the INPUT ADDMERGE procedure
- 5. (SPIRES) the BATCH procedure
- 6. (SPIBILD) the INPUT BATCH procedure
- 7. (SPIBILD) the INPUT ADD procedure
- 8. (SPIBILD) the INPUT ADDUPDATE procedure
- 9. (SPIBILD) the INPUT ADDMERGE procedure
- 10. (SPIBILD) the LOAD procedure
- 11. (SPIRES or SPIBILD) the INPUT LOAD procedure
- 12. (SPIBILD) the FAST BATCH / FAST LOAD procedure
- 13. (FASTBILD) the FAST BUILD procedure
- 14. (FASTBILD) the LOAD procedure
Which you should use in a given situation depends primarily on these factors:
- If it's in the internal SPIRES format (i.e., coming from another SPIRES file via the GENERATE LOAD command), you must use the LOAD procedure in SPIBILD or FASTBILD, or the INPUT LOAD procedure, or the FAST LOAD procedure in SPIBILD.
- If it's in the standard SPIRES format, you can use the INPUT BATCH procedures in SPIRES or SPIBILD, the BATCH procedure in SPIRES, the FAST BATCH procedure in SPIBILD or the FAST BUILD procedure in FASTBILD.
- If it's in a customized format, you can use the INPUT ADD, INPUT ADDUPDATE or INPUT ADDMERGE procedures in SPIRES or SPIBILD, the FAST BATCH procedure in SPIBILD, or the FAST BUILD procedure in FASTBILD.
Below are descriptions of each of the techniques listed above.
The INPUT BATCH command builds records from the input data set and adds them to the file's deferred queue. The data may be in the standard SPIRES format and may include, besides records to be added, records to be updated, merged or removed. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.
The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. It can be used with data stored on tape through a BATSPI batch job.
Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner, not the user doing the input, who pays for only the input costs.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.
See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES in general. For online help, EXPLAIN INPUT BATCH.
The INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands build records from the input data set and add them to the file's deferred queue. The data must be in a form that can be read by a customized input format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.
INPUT ADD can handle only records being added. INPUT ADDUPDATE and INPUT ADDMERGE check the subfile to see whether a record with the same key as given in the input data already exists, in which case that record is replaced with the new input (INPUT ADDUPDATE) or the input is merged into that record (INPUT ADDMERGE).
These INPUT commands in SPIRES are available to any user with update privileges to the subfile. Each can be used with data stored on tape through a BATSPI batch job.
Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner, not the user doing the input, who pays for only the input costs.
See the SPIRES manual "Searching and Updating" for more information about the INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands in SPIRES. For online help, EXPLAIN INPUT ADD COMMAND, etc.
The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.
This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.
The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user does not know till the next day whether the records were accepted, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.
The input data must be in the standard SPIRES format. It is not limited to added records, but can include updates, merges or removes too.
A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched, of course; but the request which tells JOBGEN where the data is gets thrown away. This problem makes the BATCH command a somewhat unreliable method, particularly if you do not know the file owner.
For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".
The INPUT BATCH command in SPIBILD builds records from the input data set and adds them directly to the tree. As it does that, it builds the pass stack (the changes to be applied to the subfile's indexes) for the added records, which is processed and emptied when it fills up or when the end of the input data is reached.
The data must be in the standard SPIRES format and may include records to be updated, merged or removed. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the records involved.
The INPUT BATCH procedure in SPIBILD is unlike the INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.
The input data may be on disk or tape, though if on tape, the procedure must be done through a batch SPIBILD job.
The steps of this procedure are described in detail in the next section. [EXPLAIN BATCH PROCEDURE IN SPIBILD online.]
The INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands in SPIBILD build records from the input data set and add them directly to the tree. As they do that, they build the pass stack (the changes to be applied to the subfile's indexes) for the added records, which is processed and emptied when it fills up or when the end of the input data is reached.
The data must be in a form that can be read by a customized input format. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the records involved.
INPUT ADD can handle only records being added. INPUT ADDUPDATE and INPUT ADDMERGE check the subfile to see whether a record with the same key as given in the input data already exists, in which case that record is replaced with the new input (INPUT ADDUPDATE) or the input is merged into that record (INPUT ADDMERGE).
These INPUT procedures in SPIBILD are unlike their counterparts in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use these SPIBILD procedures. Again, the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.
The input data may be on disk or tape, though if on tape, the procedure must be done through a batch SPIBILD job.
The steps of this procedure are described in detail later in this chapter. [EXPLAIN INPUT ADD PROCEDURE IN SPIBILD online.]
You can use the LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file).
The input data is the internal SPIRES form of the records, exactly as stored in the ORVYL data sets. To create the data, you use the Global FOR command GENERATE LOAD in SPIRES. Then in SPIBILD, you issue the LOAD command to add the data to the appropriate file.
The major advantage of this method is that it avoids the substantial overhead of executing Outproc rules when the input data set is being created, and of executing Inproc rules when the data set is being loaded into the new file.
The input data for the LOAD procedure must come from a disk data set created by the GENERATE LOAD command; it cannot come directly from a tape.
Again, because the LOAD command causes the file to be processed, you cannot use it unless you have Process access to the file.
The LOAD procedure is covered in detail later in this chapter. [See 2.3.3.]
You can use the INPUT LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file). It is similar to the LOAD procedure and has the same advantages; it is different in that it is available in SPIRES as well as SPIBILD, which allows you to take advantage of deferred-queue processing, to use it within a protocol, or to use it with temporary files, which are available only in SPIRES.
In SPIRES, INPUT LOAD also has the WITH FAST and WITH SEARCHONLY options, which speed up the processing, at the cost of some restrictions.
The input data is the internal SPIRES form of the records, exactly as stored in the ORVYL data sets. To create the data, you use the Global FOR command GENERATE LOAD in SPIRES. Then in SPIRES or SPIBILD, you issue the INPUT LOAD command to add the data to the appropriate file.
The major advantage of this method is that it avoids the substantial overhead of executing Outproc rules when the input data set is being created, and of executing Inproc rules when the data set is being loaded into the new file.
You cannot use it unless you have Master access to the file.
The INPUT LOAD procedure is covered in detail later in this chapter. [See 2.3.3a.]
The FAST BATCH and FAST LOAD procedures for adding records are specialized techniques that can be used only in certain circumstances:
- The record-type into which the records will be added must be completely empty, unless the record-type has a slot key.
- The records in the input data must either be in ascending key order or else have their keys assigned by SPIRES in ascending order (e.g., a slot key).
The records will go into the record-type very quickly and efficiently, many times faster than they probably would with the standard INPUT BATCH or INPUT ADD procedure in SPIBILD (see above). The gain in efficiency comes at the loss of two features of the INPUT procedures:
- The FAST procedures do not pass data to indexes; if the record-type has indexes, you must build them later, using the BUILD procedure. [See 2.7.1.]
- The FAST procedures do not checkpoint the data. That means that the procedure must be started over again if a major problem such as system failure occurs.
FAST BATCH and FAST LOAD are very useful when you have hundreds of thousands of records to add to a new file. They break the file-building process into several pieces (building the goal record-type with the FAST BATCH or FAST LOAD command and building the indexes later with the BUILD command), which can be scheduled independently. For example, you might need many hours to build a large file using the BATCH command in SPIBILD; you would need much less total time to do the same work using FAST BATCH and BUILD, plus you could do the two pieces whenever convenient.
To use the FAST BATCH procedure, you must have Process access to the file. The input data for the FAST BATCH procedure may be on tape, in which case you must use batch SPIBILD, and it may be formatted, as long as you have an input format that can read it.
You can use the FAST LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file). The data must be stored in a load file online; it cannot be on tape. The FAST LOAD procedure is similar to the LOAD procedure and has the same advantages; but it also has the same advantages as the FAST BATCH procedure.
FAST BATCH and FAST LOAD, though similar to FASTBILD in some respects (see below), are much easier to use. The complete procedure is described later in this chapter. [See 2.3.4.]
Of all the procedures, FAST BUILD and LOAD in FASTBILD are the most difficult to use. Like FAST BATCH, they can be used only if the goal record-type is empty to begin with. They also separate building the goal record-type from building the indexes, though you can have FASTBILD take care of the index-building automatically, if you like.
Also like FAST BATCH, FAST BUILD and LOAD do not checkpoint the data; if problems arise, the procedure must be restarted. Problems tend to arise more often in FASTBILD because the JCL and commands you must put together in order to use it can be quite complex. Several limits that are not always easy to determine must be specified in the job stream; errors in these estimates can cause serious problems, requiring you to start the job over again.
But on the other hand, the FASTBILD procedures are the most efficient of all when you are building a record-type of thousands of records. The index-building process is considerably more efficient in FASTBILD than SPIBILD, particularly in I/O usage.
You can use the FASTBILD procedures if you have Process access to the file. The input data may be formatted (though you will need an input format to read it) and may be on tape. If it is formatted in the internal SPIRES format, i.e., if it is coming directly from another SPIRES file, you use LOAD instead of FAST BUILD.
More information about FASTBILD and a complete description of the FAST BUILD and LOAD procedures appear in a later chapter. [See 2.12.]
Procedure: INPUT BATCH procedure in SPIBILD
Purpose: Add, update or remove multiple records in a subfile
using standard SPIRES format
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Prepare the input data set.
3. Issue any SPIBILD special-processing commands desired.
4. ESTABLISH subfile-name
5. Issue the INPUT BATCH command.
This procedure allows you to add multiple records to a subfile at once using SPIBILD. The input data is in the standard SPIRES format, and may also contain requests for record updates and removes as well. [See 2.3.5.]
You may use this procedure either online or in batch. [See 2.1.3.] If using it in batch, you will need to modify the steps of the procedure to fit batch, rather than interactive, work.
To invoke SPIBILD, issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET EXCEPTION command, to name an ORVYL data set into which unacceptable records from the input data set are placed. [See 2.1.5.2.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of lines in the input file. It is used for recovery purposes when a previous SPIBILD task has failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. [See the description for that command in Part 6, or online, EXPLAIN SET STACK SIZE COMMAND.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT BATCH command will work properly.
The INPUT BATCH command in SPIBILD batches the input data into the named subfile:
For online SPIBILD, and the BATWYL program: INPUT BATCH For batch SPIBILD: [FROM ddname] INPUT BATCH
Here is a typical online SPIBILD session using the INPUT BATCH procedure:
Command> spibild
-Welcome to SPIBILD
-? use data.from.bakers
-? set exception crummy.cookies
-? set information messages = 0
-? establish cookies
-Compute time: 0.238 seconds
-Elapsed time: 1.544 seconds
-Core usage: 0065/0933
-? input batch
-Element=OVEN.TEMP: -Must be integer -Serious data error, code=E21
-Error at or before line 396.
-Update terminated, code=S261
-Error at or before line 399.
-*ADD 27, 26 @Line 399 Key = CHOCCHIP Err = S261
-Begin passing of 52 records of record-type REC01
-Completed passing of 52 records total
- Requests/Success:
ADD 53 52
SUM 53 52
-End of BATCH/BUILD
-Detaching File: AM.OSS.COOKIES
08/11/90 File Read/Write Counts Information
File: AM.OSS.COOKIES Reads Writes
CKPT data set 4 1
DEFQ data set 9 7
MSTR data set 6 0
Residual data set 23 29
Rec-type REC01 2 14
Rec-type ZIN02 22 22
Total for file: 66 73
-Compute time: 9.772 seconds
-Elapsed time: 48.202 seconds
-Core usage: 0257/0933
-?
The record that failed to be added (because the value of the OVEN.TEMP element was not an integer) was placed in the exception file, an ORVYL data set called CRUMMY.COOKIES. The other 52 records in the input data set were added successfully.
The most common problem that occurs was shown above: one or more records fail to go into the subfile because an Inproc processing rule rejects them. The solution is to find the rejected records, correct them, and try the procedure again with only those records in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected records there. [See 2.1.5.2.]
System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.
If the failure happens prior to the INPUT BATCH command, you should just start the entire procedure over again when you can.
If it happens during the INPUT BATCH command, you will want to determine the last record that made it through the entire process (i.e., that was added and indexed). All records added before the last "-Begin passing:" message that's not followed by a "Completed passing..." message are in the subfile; all those added after it will be discarded during file recovery. You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. You may either delete the successful records from the front of the input data set or skip them using the SET SKIP command, but in either case, you should restart the procedure, omitting the records that previously went in successfully. [See 2.3.6.]
Procedures: INPUT ADD procedure in SPIBILD
INPUT ADDUPDATE procedure in SPIBILD
INPUT ADDMERGE procedure in SPIBILD
Purpose: Add multiple records to a subfile using an input
format; also, for INPUT ADDUPDATE, update records if
they already exist; also, for INPUT ADDMERGE, merge
data into records if they already exist
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Prepare the input data set.
3. Issue any SPIBILD special-processing commands desired.
4. ESTABLISH subfile-name
5. Set the input format.
6. Issue the appropriate INPUT command.
This procedure allows you to add multiple records to a subfile at once using SPIBILD. The input data will be read by a customized input format. See the manual "SPIRES Formats" for information about input formats for multiple records; online, [EXPLAIN FORMATS, FOR MULTIPLE-RECORD INPUT.]
In addition, by using either the INPUT ADDUPDATE or INPUT ADDMERGE command, you can tell SPIBILD not to reject a record in the input data if its key matches the key of a record already in the subfile, but instead to use the input data to update that record. In the case of INPUT ADDUPDATE, the input data for the record completely replaces the record already in the subfile; for INPUT ADDMERGE, the input data is merged into the record already there. Particularly for INPUT ADDMERGE, you may need to include some special processing techniques within the input format, which are described in the "SPIRES Formats" manual; online, [EXPLAIN FORMATS, FOR ADDUPDATE COMMAND.]
You may use these procedures either online or in batch. [See 2.1.3.] If using one in batch, you will need to modify the steps of the procedure to fit batch, rather than interactive, work.
To invoke SPIBILD, issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET EXCEPTION command, to name an ORVYL data set into which rejected records from the input data set are placed. [See 2.1.5.2.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of lines in the input file. It is used for recovery purposes when a previous SPIBILD task has failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
- the SET PTRACE command, in one or more of its forms, to display tracing information for input or output processing rules (Inproc, Inclose, Outproc) executed during the record input processing.
You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT command will work properly.
Depending on what the format does, any of the commands listed below may be involved. All except SET FORMAT are optional; those you use should be issued in the order shown:
- ALLOCATE. This command allocates a vgroup, usually so that stored values can be restored to it (see RESTORE STATIC).
- RESTORE STATIC. This command restores previously stored values to a vgroup. See "SPIRES Protocols" for more information about these two commands; online, [EXPLAIN ALLOCATE.] or [EXPLAIN RESTORE STATIC.]
- DEFINE AREA and ASSIGN AREA. These commands (as well as CLOSE AREA to be issued after the INPUT command) are needed when the input format is also creating a report. [See 2.1.5.5.]
- SET FORMAT. The SET FORMAT command sets the named format, or, if the "*" form of the command is used, executes the startup frame of the format named in the previous SET FORMAT command. You must start with a "SET FORMAT format-name" command, with one or more SET FORMAT * commands optionally to follow.
SET FORMAT {format-name|*} [,parm|'parm'|"parm"]
- SET FTRACE. SET FTRACE enables format-tracing messages, most often used for debugging. [See "SPIRES Formats", section B.7.2.]
Issue the INPUT ADD, INPUT ADDUPDATE or INPUT ADDMERGE command to batch the input data into the named subfile:
For online SPIBILD, and the BATWYL program:
[USING frame] INPUT {ADD|ADDUPDATE|ADDMERGE}
For batch SPIBILD:
[FROM ddname] [USING frame] INPUT {ADD|ADDUPDATE|ADDMERGE}
Here is a typical online SPIBILD session using the INPUT ADD procedure:
Command> spibild -Welcome to SPIBILD -? use new.students.data -? set information messages = 0 -? clear file counts -? establish students -Compute time: 0.238 seconds -Elapsed time: 1.544 seconds -Core usage: 0065/0933 -? set format new -? input add -Begin passing of 36 records of record-type REC01 -Completed passing of 36 records total - Requests/Success: ADD 36 36 SUM 36 36 -End of BATCH/BUILD -Compute time: 7.229 seconds -Elapsed time: 24.820 seconds -Core usage: 0257/0933 -?
The most common problem that occurs is that one or more records fail to go into the subfile because an Inproc processing rule rejects them. The solution is to find the rejected records, correct them, and try the procedure again with only those records in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected records there. [See 2.1.5.2.]
System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.
If the failure happens prior to the INPUT command, you should just start the entire procedure over again when you can.
If it happens during the INPUT command, you will want to determine the last record that made it through the entire process (i.e., that was added and indexed). All records added before the last "-Begin passing:" message that's not followed by a "-Completed passing..." message are in the subfile; all those added after it will be discarded during file recovery. You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. You may either delete the successful records from the front of the input data set or skip them using the SET SKIP command, but in either case, you should restart the procedure, omitting the records that previously went in successfully. [See 2.3.6.]
Procedure: LOAD procedure in SPIBILD
Purpose: Add multiple records in internal SPIRES format to a
subfile
Who can use it: file owner
users with See access to the file
containing the data, and Process access
to the file receiving the data
Basic Steps:
1. Create the input data set with GENERATE LOAD.
2. SPIBILD
3. Issue any SPIBILD special-processing commands desired.
4. Issue the LOAD command.
The LOAD procedure is useful when the records you want to add to a subfile are already in a SPIRES subfile with identical record characteristics. You may want to move records from the test version of a file to the production version. Or you might want to rebuild a file, using the records that are already in it.
You may want to compare this technique with the INPUT LOAD procedure, described in the next section, which works in SPIRES as well as SPIBILD [See 2.3.3a.] and with the FAST LOAD procedure in SPIBILD. [See 2.3.4.]
With this technique, you create an ORVYL data set containing the internal, stored form of the desired records. From there, the records can be loaded into another subfile in SPIBILD. This technique avoids the overhead of running the element values in the records through Outproc rules (as the data set is created) and Inproc and Inclose rules (as the data set is read back in), which can lead to substantial savings in processing time and costs.
The two record-types must share the same internal structure. That means that the way any given record would be stored in one must be identical to the way it would be stored in the other. One easy way to test the relationship between the two is to select each subfile and compare the displays from the SHOW ELEMENT CHARACTERISTICS command. The element names, displayed in the last column, do not have to match, but everything else shown should match, row for row. [Technically, there are some allowable differences besides element names. In theory, the element types do not have to match (except for the "Struc" (Structure) type). But such theories don't usually make good practice.] If they do not match, then the load will probably fail.
You must have See access or better to the file containing the data in order to use the GENERATE LOAD command, and you must have Process access to the receiving file in order to use the LOAD command.
Here are the steps of the procedure:
To create the load of records, you need to select the subfile containing the data in SPIRES.
Next, establish a Global FOR class containing the records you want. You must begin with a "FOR class" command, such as FOR SUBFILE, that may contain VIA and WHERE clauses. You may also issue SET SCAN commands and SET FILTER commands with "TYPE = SCAN" to further limit the records to be retrieved. [See the manuals "Sequential Record Processing in SPIRES: Global FOR" and "SPIRES Technical Notes" for more information about these commands.]
Next, issue the GENERATE LOAD command:
GENERATE LOAD loadname [REPLACE] [FIRST|*|NEXT|n|REST|LAST|ALL]
where "loadname" is the name of an ORVYL data set. The options at the end are the standard ones for a Global FOR command, specifying which records are to be processed. The default is ALL, for all the records in the established class. See Part 6, or online, [EXPLAIN GENERATE LOAD.]
Here, for instance, is a sample session that creates a load of all Stanford records in a subfile:
-> spires -Welcome to SPIRES-3 ... If in trouble, try 'HELP' -> select consortium members -> for subfile where university = stanford +> generate load stanford.members +>
The most likely problems you could have with GENERATE LOAD are these:
At this point, you could move the load data set to a tape using the ORVCOPY program. Later, when you are ready to load it into the receiving file, you restore it to an online ORVYL data set, again using ORVCOPY. [See 2.1.4.3.]
Note too that if you need to examine the records in the load at a later time, for example, to see the keys of the records there, you can use the FOR LOAD command in Global FOR to process them.
By the way, in step 4, you will need to know the name of the file and the name of the record-type into which the data is being loaded. If you do not know that information and are about to proceed to step 2, you should select the subfile you want to load the data into, and issue the SHOW SUBFILE INFORMATION command, which will tell you the name of the file, and the name of the goal record-type (listed under "Record") that you will specify in the LOAD command.
Issue the SPIBILD command online, or if you want to load the data via a BATWYL job, place it in the command stream.
+> spibild -Welcome to SPIBILD -?
If you are loading the data via a batch SPIBILD job, the program puts you in SPIBILD automatically, so you can omit this command.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add" message displayed for each new record. [See 2.1.5.3.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of records in the load of records. It is used for recovery purposes when a previous LOAD command failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
Note that you can use neither an input format nor an exception file with the LOAD procedure.
The LOAD command loads the data into the desired file:
LOAD loadname [RENUMBER] INTO record-name OF filename
where "loadname" is the name specified in the GENERATE LOAD command (see step 1) and "filename" and "record-name" are the names of the file and the particular record-type into which the data is to be loaded.
The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).
The LOAD command begins by processing the named file. Then it loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN LOAD COMMAND.]
Here is a continuation of the example started above:
-? set information messages = 0 -? clear file counts -? load stanford.members into rec01 of gq.jnk.su.chapter -Compute time: 0.199 seconds -Elapsed time: 6.767 seconds -Core usage: 0065/0933 -Begin passing of 49 records of record-type REC01 -Completed passing of 49 records total - Requests/Success: ADD 49 49 SUM 49 49 -End of BATCH/BUILD -Compute time: 3.012 seconds -Elapsed time: 18.798 seconds -Core usage: 0132/0933 -?
Note that the messages displayed from the LOAD command are identical to those from the BATCH command. The individual "add" messages for the records were suppressed in this example thanks to the SET MESSAGES command just prior to the LOAD command. [See 2.2.4 for information about the other messages shown.]
Below are described some problems that occasionally arise during LOAD processing.
An S311 error message indicates that the characteristics of the record-type into which you are trying to load the data don't match the characteristics of the record-type containing the original data. Though some minor differences may exist, such as differences in element names, the two record-types should have practically identical internal structures, as described at the start of this section. If they do not, the LOAD command will fail.
An S413 error message indicates that an attempt was made to load a record with a key that matches a record already in the record-type. LOAD works only with new records, not updates of existing ones. If this problem occurs with slot records, you can tell SPIBILD to ignore the provided keys of the incoming records and replace them with new slot numbers by adding the RENUMBER option to the LOAD command.
If the system crashes during the initial processing of the file, i.e., after you have issued the LOAD command but before the loading has begun, simply restart the procedure from step 2. Alternatively, you might want to use the PROCESS/RECOVER procedure to recover the file first, and then continue this procedure from step 2. [See 2.1.4.2.]
- you can start over again from step 2, ignoring the S413 errors you would get for the records previously added, [But if you used the RENUMBER option to tell SPIBILD to assign new slot keys to the records, you will not get any S413 errors when you restart the procedure; the same records will go in again.]
- you can determine how many records of the load made it into the subfile and then use the SET SKIP command in SPIBILD to tell SPIBILD how many records in the load to skip before starting the load again. [See 2.2.5.]
- you can determine which records made it into the subfile and then create a new load data set without those records (i.e., start again at step 1). You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. Alternatively, you could return to SPIRES, select the subfile and use various searching techniques to determine which was the last record to get into the new subfile.
Procedure: INPUT LOAD procedure
Purpose: Add multiple records in internal SPIRES format to a
subfile
Who can use it: file owner
users with See access to the file
containing the data, and Master access
to the file receiving the data
Basic Steps:
In SPIRES:
1. Create the input data set with GENERATE LOAD.
2. SELECT subfile.name
3. Issue any special-processing commands desired.
4. Issue the INPUT LOAD command.
5. Cleanup: Other considerations
In SPIBILD:
1. Create the input data set with GENERATE LOAD.
2. SPIBILD
3. ESTABLISH subfile.name
4. Issue any special-processing commands desired.
5. Issue the INPUT LOAD command.
The INPUT LOAD procedure is useful when the records you want to add to a subfile are already in a SPIRES subfile with identical record characteristics. You may want to move records from the test version of a file to the production version, or vice versa. Or you might want to rebuild a file, using the records that are already in it.
This technique is quite similar to the LOAD procedure in SPIBILD, but has these differences:
- It is available in SPIRES as well as SPIBILD, meaning it could be incorporated into a protocol, or used with temporary files.
- In SPIRES, it has the WITH FAST and WITH SEARCHONLY options, which speed up the processing considerably; they are not available to the LOAD procedure.
- It has an UPDATE option, whose effect is to make SPIRES treat the incoming records as "addupdates" rather than just "adds"; the LOAD procedure can handle only adds.
With this technique, you create an ORVYL data set containing the internal, stored form of the desired records. From there, the records can be loaded into another subfile in SPIRES or SPIBILD. This technique avoids the overhead of running the element values in the records through Outproc rules (as the data set is created) and Inproc and Inclose rules (as the data set is read back in), which can lead to substantial savings in processing time and costs.
The two record-types must share the same internal structure. That means that the way any given record would be stored in one must be identical to the way it would be stored in the other. One easy way to test the relationship between the two is to select each subfile and compare the displays from the SHOW ELEMENT CHARACTERISTICS command. The element names, displayed in the last column, do not have to match, but everything else shown should match, row for row. [Technically, there are some allowable differences besides element names. In theory, the element types do not have to match (except for the "Struc" (Structure) type). But such theories don't usually make good practice.] If they do not match, then the load will probably fail.
You must have See access or better to the file containing the data in order to use the GENERATE LOAD command, and you must have Process access to the receiving file in order to use the LOAD command.
This procedure is probably most commonly used in SPIRES because of the first two factors listed above that distinguish it from the LOAD procedure in SPIBILD. In SPIBILD, it is very similar to the LOAD procedure; the only significant difference is that INPUT LOAD has the ability to handle record updates as well as adds. A relatively insignificant difference is that with INPUT LOAD, you only need to know the name of the subfile into which you want to load the records; with LOAD, you must know the name of the file and the goal record-type.
In choosing between SPIRES and SPIBILD, remember that SPIRES will leave the records in the deferred queue for subsequent processing. SPIBILD will process the file, put the new records into the tree and leave the deferred queue empty.
If you choose to follow the procedure in SPIRES, here are the steps. (The steps for the SPIBILD procedure appear afterwards.)
To create the load of records, you need to select the subfile containing the data in SPIRES.
Next, establish a Global FOR class containing the records you want. You must begin with a "FOR class" command, such as FOR SUBFILE, that may contain VIA and WHERE clauses. You may also issue SET SCAN commands and SET FILTER commands with "TYPE = SCAN" to further limit the records to be retrieved. [See the manuals "Sequential Record Processing in SPIRES: Global FOR" and "SPIRES Technical Notes" for more information about these commands.]
Next, issue the GENERATE LOAD command:
GENERATE LOAD loadname [REPLACE] [FIRST|*|NEXT|n|REST|LAST|ALL]
where "loadname" is the name of an ORVYL data set. The options at the end are the standard ones for a Global FOR command, specifying which records are to be processed. The default is ALL, for all the records in the established class. See Part 6, or online [EXPLAIN GENERATE LOAD.]
Here, for instance, is a sample session that creates a load of all Stanford records in a subfile:
-> spires -Welcome to SPIRES-3 ... If in trouble, try 'HELP' -> select consortium members -> for subfile where university = stanford +> generate load stanford.members +>
The most likely problems you could have with GENERATE LOAD are these:
At this point, you could move the load data set to a tape using the ORVCOPY program. Later, when you are ready to load it into the receiving file, you restore it to an online ORVYL data set, again using ORVCOPY. [See 2.1.4.3.]
Note too that if you need to examine the records in the load at a later time, for example, to see the keys of the records there, you can use the FOR LOAD command in Global FOR to process them.
If you intend to use the WITH FAST option on the INPUT LOAD command, you may want to use the PROCESS/ESTABLISH procedure at this point to make sure the deferred queue is empty. [See 2.2.2.] You may also want to take whatever steps are needed to prevent any other users from trying to add records to the subfile at the same time as you are doing so -- WITH FAST requires and empty deferred queue and exclusive update access to the file, and users attempting to add records will be blocked. (This entire paragraph is relevant only under WITH FAST processing. See the description of the INPUT LOAD command in Part 6 for information about its value; online, [EXPLAIN INPUT LOAD COMMAND.] )
If you are continuing immediately from step 1, then you are already in SPIRES; but if not, issue the SPIRES command. Then select the target subfile, that is, the subfile into which you are loading the records.
+> select su chapter ->
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIRES displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add" or "update" message displayed for each new record. [See 2.1.5.3.]
- the SET SKIP command, which skips the specified number of records in the load of records. It is used for recovery purposes when a previous INPUT LOAD command failed during a batch input process. [See 2.3.6.]
Note that you can use neither an input format nor an exception file with the INPUT LOAD procedure.
The INPUT LOAD command loads the data into the desired file:
[WITH FAST|WITH SEARCHONLY] INPUT LOAD loadname [RENUMBER|UPDATE]
where "loadname" is the name specified in the GENERATE LOAD command.
The WITH FAST and WITH SEARCHONLY options request faster, more efficient processing of the data at the cost of you accepting several restrictions. Details are discussed in the reference section on INPUT LOAD in the back of this manual. [See Part 6, or EXPLAIN LOAD COMMAND for more information.]
The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).
By default, only new records are allowed to be loaded with the INPUT LOAD command -- if a record being loaded has the same key as a record already in the subfile, then the new one is rejected. The UPDATE option tells SPIRES to treat the new one as an update of the record already there, in essence turning the "add" into an "addupdate". That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN LOAD COMMAND.]
Here is a continuation of the example started above:
-> set information messages = 0 -> input load stanford.members renumber - Requests/Success: ADD 49 49 SUM 49 49 -End of load ->
Note that the messages displayed from the INPUT LOAD command are basically identical to those from the BATCH command. The individual "add" and "update" messages for the records were suppressed in this example, thanks to the SET MESSAGES command just prior to the INPUT LOAD command. [See 2.2.4 for information about the other messages shown.]
If you have problems with the INPUT LOAD command, see the discussion at the end of this section.
Once you are certain that all went well, you will probably want to discard the load file you generated in step 1.
If you used the WITH FAST option on INPUT LOAD in step 4, then if your file has immediate indexing, no further updates to the subfile will be allowed until the file is processed. You may want to process the file at this point to restore it to its normal state. [See 2.2.2.]
If you choose to follow the procedure in SPIBILD, here are the steps.
Follow the instructions under step 1 above of the SPIRES version of the procedure.
Issue the SPIBILD command online, or to load the data via a BATWYL job, place it in the command stream.
+> spibild -Welcome to SPIBILD -?
If you are loading the data via a batch SPIBILD job, the program puts you in SPIBILD automatically, so you can omit this command.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add" or "update" message displayed for each new record. [See 2.1.5.3.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of records in the load of records. It is used for recovery purposes when a previous INPUT LOAD command failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
Note that you can use neither an input format nor an exception file with the INPUT LOAD procedure.
You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT LOAD command will work properly.
The INPUT LOAD command loads the data into the desired file:
INPUT LOAD loadname [RENUMBER|UPDATE]
where "loadname" is the name specified in the GENERATE LOAD command.
The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).
By default, only new records are allowed to be loaded with the INPUT LOAD command -- if a record being loaded has the same key as a record already in the subfile, then the new one is rejected. The UPDATE option tells SPIBILD to treat the new one as an update of the record already there, in essence turning the "add" into an "addupdate".
The INPUT LOAD command loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN INPUT LOAD COMMAND.]
Here is a continuation of the example started above:
-? set information messages = 0 -? clear file counts -? establish su chapter -Compute time: 0.199 seconds -Elapsed time: 6.767 seconds -Core usage: 0065/0933 -? input load stanford.members -Begin passing of 49 records of record-type REC01 -Completed passing of 49 records total - Requests/Success: ADD 49 49 SUM 49 49 -End of BATCH/BUILD -Compute time: 3.012 seconds -Elapsed time: 18.798 seconds -Core usage: 0132/0933 -?
Note that the messages displayed from the INPUT LOAD command are identical to those from the BATCH command. The individual "add" or "update" messages for the records were suppressed in this example thanks to the SET MESSAGES command prior to the INPUT LOAD command. [See 2.2.4 for information about the other messages shown.]
Below are described some problems that occasionally arise during INPUT LOAD processing.
An S311 error message indicates that the characteristics of the record-type into which you are trying to load the data don't match the characteristics of the record-type containing the original data. Though some minor differences may exist, such as differences in element names, the two record-types should have practically identical internal structures, as described at the start of this section. If they do not, the INPUT LOAD command will fail.
An S413 error message indicates that an attempt was made to load a record with a key that matches a record already in the record-type. By default, INPUT LOAD works only with new records, not updates of existing ones. If this problem occurs with slot records, you can tell SPIRES to ignore the provided keys of the incoming records and replace them with new slot numbers by adding the RENUMBER option to the INPUT LOAD command. You may alternatively want to use the UPDATE option to tell SPIRES to use the load copy of the record as an update to the copy already in the file.
In SPIBILD, if the system crashes during the initial processing of the file, i.e., after you have issued the LOAD command but before the loading has begun, simply restart the procedure from step 2. Alternatively, you might want to use the PROCESS/RECOVER procedure to recover the file first, and then continue this procedure from step 2. [See 2.1.4.2.]
In SPIBILD, if the system crashes during the actual loading, records added before the last "-Begin passing" message that's not followed by a "-Completed passing..." message are in the subfile; those added after it are in the deferred queue but will be discarded during file recovery. You have three choices:
- you can start over again from step 2 of the procedure you were using, ignoring the S413 errors you would get for the records previously added. [But if you used the RENUMBER or UPDATE option to tell SPIBILD to assign new slot keys to the records, you will not get any S413 errors when you restart the procedure; the same records will go in again.]
- you can determine how many records of the load made it into the subfile and then use the SET SKIP command in SPIBILD to tell SPIBILD how many records in the load to skip before starting the load again. [See 2.2.5.]
- you can determine which records made it into the subfile and then create a new load data set without those records (i.e., start again at step 1). You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. Alternatively, you could return to SPIRES, select the subfile and use various searching techniques to determine which was the last record to get into the new subfile.
In SPIRES, if you included the WITH FAST option on the INPUT LOAD command, you may want to simply ZAP DEFQ and start again from step 2; the only records in the deferred queue are the ones you put there. In fact, even if you didn't use the WITH FAST option, it may be worth checking with SHOW SUBFILE TRANSACTIONS to see if yours are the only records in the deferred queue. (Remember though that other record-types in the file may have defq data too, which would be lost with a ZAP DEFQ command.)
Procedure: FAST BATCH or FAST LOAD
Purpose: Add multiple records to an empty subfile very
quickly, under special conditions
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Process the file with the PROCESS/ESTABLISH procedure.
3. Prepare the input data set.
4. SET NOPASS
5. Set the input format, if necessary.
6. Issue any SPIBILD special-processing commands desired.
7. Issue the FAST BATCH or FAST LOAD command.
8. Build the indexes, if any.
The FAST BATCH and FAST LOAD procedures provide the fastest "easy" method of getting new records into a subfile, at the loss of two features common to most other methods:
- FAST BATCH / FAST LOAD does not checkpoint the input data, meaning that the process must be completely redone in the event of a system failure; and
- FAST BATCH / FAST LOAD does not build indexes from the input data. That must be done separately, using the BUILD command.
These procedures are especially useful when you want to build a new goal record-type with thousands of records. They let you split the work into two separate pieces: building the goal record-type and building its indexes. This is critical in situations where the amount of time available in a single computer session is less than the amount needed to build and index the record-type at one time with the INPUT BATCH procedure in SPIBILD.
To use FAST BATCH or FAST LOAD, you must meet several other restrictions too:
- The record-type into which the data is being added must be completely empty, unless it is a slot record-type.
- The records in the input data must either be in ascending key order, or have no keys at all, letting SPIBILD assign them (e.g., slot keys). [If the record-type is a slot record-type and the input data contains slot key values, then the records do not have to be in ascending key order, but the input processing will be less efficient than if they were.]
- All input must be ADD requests; this process cannot handle updates or removes.
- For FAST LOAD, the input data must already be in the internal form of SPIRES records, in a file created by the GENERATE LOAD command; the record characteristics of the record-type the records came from must match those of the record-type into which you want to FAST LOAD them. [See 2.3.3 for further information about the "load" type of batch input in general.]
If you are using the FAST LOAD procedure, you need to prepare the data set containing the input. Follow step 1 in "LOAD Procedure in SPIBILD" to generate the load file. [See 2.3.3.]
Issue the SPIBILD command online or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job you are already in SPIBILD, so you omit this command from the command stream.
This step is only necessary if the file has other record-types in which data already exists. It is important to ensure that no passing between other record-types of the file would be attempted when the FAST BATCH command is executing.
Follow the PROCESS/ESTABLISH procedure described earlier in this manual. To execute its most simple form, simply issue the PROCESS command:
PROCESS [ORV.gg.uuu.]filename
where "gg.uuu" is the file owner's account, which must be specified if you're processing someone else's file. [See 2.2.2.]
For FAST LOAD, you can skip this step; your input is ready to use.
For FAST BATCH, if you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file.
The data may be in the standard SPIRES format, or it may be formatted in some other way, as long as you have an input format that can be used to read it. Always make certain the format works properly before trying to use it to batch hundreds of records into a subfile; a minor failure can be expensive when multiplied by hundreds of records. [See the manual "SPIRES Formats" for information about creating input formats.]
For FAST BATCH in batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set. If the data set is stored online, you may issue a USE command in the JCL at this point; otherwise, be sure to include a DD statement for the data set, to which you will refer in the FAST BATCH command later. [See 2.1.3.2.]
Unless you are planning to use the "ON ddname" option in batch SPIBILD, be sure that the input data set is in your active file at this point.
The records in the input data set should either be in order by key, or have their keys assigned by SPIRES as they enter the data base, e.g., slot keys. Records that are out of order will be rejected (with the exception of records with keys that are going into a slot record-type, which can be handled in any order).
You must specifically tell SPIBILD not to try to pass index data during the FAST BATCH / FAST LOAD process. The best way is to issue the SET NOPASS command in SPIBILD.
-? set nopass -?
Another method is to add the NOPASS statement to the linkage section for the record-type in the file definition and recompile it. When you are done with the FAST BATCH / FAST LOAD procedure, remove the statement from the file definition and recompile it again.
For FAST LOAD, skip this step.
For FAST BATCH, if the data will be read through a SPIRES input format, issue the SET FORMAT command at this point; otherwise, skip to step 6. Depending on what the format does, any of the commands listed below may be involved. All are optional; those you use should be issued in the order shown:
- ALLOCATE. This command allocates a vgroup, usually so that stored values can be restored to it (see RESTORE STATIC).
- RESTORE STATIC. This command restores previously stored values to a vgroup. [See "SPIRES Protocols" for more information about these two commands.]
- DEFINE AREA and ASSIGN AREA. These commands (as well as CLOSE AREA to be issued after the FAST BATCH command) are needed when the input format is also creating a report. [See 2.1.5.5.]
- SET FORMAT. The SET FORMAT command sets the named format, or, if the "*" form of the command is used, executes the startup frame of the format named in the previous SET FORMAT command. You must start with a "SET FORMAT format-name" command, with one or more SET FORMAT * commands optionally to follow.
SET FORMAT {format-name|*} [,parm|'parm'|"parm"]
- SET FTRACE. SET FTRACE enables format-tracing messages, most often used for debugging. [See "SPIRES Formats", section B.7.2.]
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET EXCEPTION command, to name an ORVYL data set into which unacceptable records from the input data set are placed (FAST BATCH only). [See 2.1.5.2.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the SET SKIP command, which skips the specified number of lines (FAST BATCH) or records (FAST LOAD) in the input file. Though normally used for recovery purposes, it can be used here if you simply want to skip a given number of lines or records of the input data set. For FAST BATCH, it is only useful when the input is in the standard SPIRES format. [See 2.3.6.]
- the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn off (CLEAR) the default file count data displayed at the end of the process, or to request full file count data (SET). [See 2.2.4.]
- the SET PTRACE command, in one or more of its forms, to display tracing information for input or output processing rules (Inproc, Inclose, Outproc) executed during the record input processing.
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
The FAST BATCH command processes the file and batches the input data into the named subfile:
For online SPIBILD, and the BATWYL program: FAST BATCH subfile.name For batch SPIBILD: FAST BATCH 'subfile.name' ON ddname
The FAST LOAD command processes the file and loads the records from the named ORVYL data set into the named record-type of the named file:
FAST LOAD loadname [RENUMBER] INTO record-name OF filename
where "loadname" is the name specified in the GENERATE LOAD command (see Preliminary Work above) and "filename" and "record-name" are the names of the file and the particular record-type into which the data is to be loaded.
The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).
The FAST LOAD command begins by processing the named file. Then it loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN FAST LOAD COMMAND.]
Here is a sample session showing typical use of the FAST BATCH procedure:
Command> spibild -Welcome to SPIBILD -? process fish ... the file is processed ... -? use shad.to.add -? set nopass -? set exception bad.shad -? clear file counts -? set information messages = 0 -? fast batch shad specimens -Processing file: GQ.JNK.FISH -Compute time: 0.117 seconds -Elapsed time: 1.423 seconds -Core usage: 0065/0939 - Requests/Success: ADD 500 500 SUM 500 500 -End of BATCH/BUILD -Compute time: 0.850 seconds -Elapsed time: 3.108 seconds -Core usage: 0197/0939 -?
Notice how very little processing time (0.85 seconds) was involved to add 500 records; while the records that generated this example are admittedly small, the "compute time" needed was low because none of the backup work SPIBILD normally does to allow recovery was done here.
The FAST LOAD procedure would look exactly the same except for the USE command, which would not be there, and the FAST BATCH command, which would be replaced by a FAST LOAD command, such as:
-? load shad.to.add into rec01 of gq.jnk.fish
If the record-type has any indexes, you will need to "build" them at some point. The index-building procedure is described in detail later. [See 2.7.1, steps 4 through 7.] Remember if you build the indexes in the same SPIBILD session to "turn on" passing again, e.g., using the SET PASS command.
The example above continues, as we build the indexes.
-? set pass -? establish shad specimens -? for subfile -? build zin02, zin03, zin04 -Begin passing of 500 records of record-type REC01 -Complete passing of 500 records total -End of BATCH/BUILD -Compute time: 5.361 seconds -Elapsed time: 35.639 seconds -Core usage: 0214/0939 -?
The most likely problems you could have with FAST BATCH are these:
Most of the problems described above for FAST BATCH could happen for FAST LOAD as well, and are handled the same. Also, problems related to the load input data could arise; they are the same as the problems described in the previous section on the LOAD procedure. [See 2.3.3.] Remember, however, that restarting FAST LOAD will require you to zap the record-type, so that it will be empty when you start (unless you are working with records having slot keys). Alternatively, you may want to retain the records that did get in, build the indexes with what you have (step 8) and resume the load with the LOAD procedure, which does not require an empty record-type to start with.
Most of the procedures for adding records to a SPIRES subfile allow the input data to be formatted so that it can be read by either the standard SPIRES format or by a custom input format. Custom formats are discussed in detail in the manual "SPIRES Formats", in section C.9 in particular.
This section presents a summary of the standard SPIRES format, discussed in detail in the SPIRES manual "Searching and Updating", sections D.2.2, D.2.3, D.2.6 and D.8.1.
Keep in mind that some of the procedures discussed in this chapter (2.3) limit their scope to added records, to which some of this information may not apply.
The basic syntax of individual element values is:
element-name = value;
where "element-name" is the name of the element, and "value" is the data value to be assigned to that element. The equals sign is optional; the semicolon at the end is required.
Here are other important rules to remember:
COMMENT = SPIRES will put a blank after the comma,
so that the comma and 'so' don't get run together.;
NAME = Merrick Butte; COMMENTS = Breeder;
COMMENTS = "This semicolon; is not really needed.";
COMMENTS = "See ""SPIRES Formats"" for details.";
COMMENTS;
MEMBER;
NAME = Jenny Lake;
STATE = Wyoming;
MEMBER;
NAME = Mitchell Butte;
STATE = Utah;
ADD;
ADDUPDATE;
ADDMERGE;
UPDATE key;
MERGE key;
REMOVE key;
REMOVE 94025;
MERGE 85-194;
NAME = Mrs. Osgood Azamile;
;
REMOVE 83-019;
ADDUPDATE;
ID = 86-028;
NAME = Miss Bea Hayve;
;
When a SPIBILD command such as INPUT BATCH, LOAD or FAST BATCH terminates in the middle of adding new records to a subfile, you must decide how to recover from the problem and finish doing your input. The procedures described earlier in this chapter for those commands describe how to determine the cause of the problem and make some suggestions for how to solve it.
Often, the solution is to somehow determine which record was the last record in the input data to be successfully added to the subfile. Once that is determined, you can delete the input up through that record, or use the SET SKIP command to tell SPIBILD to skip that much input before starting the input procedure again. [Keep in mind for FAST BATCH or FAST LOAD that if you decide to use this method for recovery, you cannot use the FAST procedure when you resume adding records, since the goal record-type must be completely empty in order to use FAST BATCH or FAST LOAD. You can, however, switch to the INPUT BATCH or INPUT ADD or LOAD procedure, or, if not many records were successfully added anyway, consider zapping the record-type and starting over again with FAST BATCH or FAST LOAD. If you do switch to another procedure, build the indexes using the data already in the subfile before you batch more records in. [See 2.7.1, steps 4 through 7.]]
How you determine the last record to be successfully added depends on whether the records were being indexed or not. For instance, they would not be indexed if:
- you were using the FAST BATCH / FAST LOAD procedure in SPIBILD; or
- you had SET NOPASS; or
- the goal record-type had no indexes.
But if they are indexed, you can take advantage of a pair of handy commands, SHOW RECORD RECOVERY and SET SKIP. We'll start with the discussion for the indexed records, followed by the discussion for the non-indexed records.
SPIBILD keeps track of the number of lines of input read (for INPUT BATCH, INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE) or the number of records read (for LOAD), storing that number away each time a group of records have been completely processed, i.e., added to the tree and fully indexed. If the processing completely finishes, then SPIBILD discards the information. But if the processing is interrupted, you can issue the SHOW RECORD RECOVERY command to show you that number:
-> select albums -> show record recovery -SET SKIP 1593 -Goal record-type = 1, Key count = 2 894 895 ->
The significant number here is the SET SKIP value, the number of lines (or records) of input successfully processed. [The other information is useful in some ways. This information is about the records that are only half-processed, i.e., whose index data was not completely processed. During recovery, SPIBILD will discard those records and "unindex" them if any indexing was done. The data for those records appears in the input data after the number of lines given in the "skip" indicator so it will be input again when you use the SET SKIP command. See Part 6, or online, [EXPLAIN SHOW RECORD RECOVERY COMMAND.]]
When you are ready to resume the SPIBILD work, start with the RECOVER/PROCESS procedure. [See 2.1.4.2.] Then start again the procedure you were using when the problem occurred, incorporating the SET SKIP command at the point where it is mentioned. See Part 6 for more information, or online, [EXPLAIN SET SKIP COMMAND.]
This procedure is somewhat more complicated when the input for the INPUT BATCH command is a mixture of transaction types, rather than only new records to be added. SET SKIP only registers added records, so the number will not get updated when non-add transactions are being processed, even though they may indeed get completely indexed. The best way to handle this situation is to get the skip number using the SHOW RECORD RECOVERY command and then use the RECOVER/PROCESS procedure to clean up the file.
Next, examine the input data past the number of lines you can skip and, in SPIRES, use the indexes (or simply display records by key) to look for those records. When you find input data that has no matching record, you have found the point where the processing stopped. Then determine how many more lines past the skip number you should tell SPIBILD to skip to begin at that point.
Note too, in both cases, that you may also discard the input that succeeded rather than use SET SKIP to skip over it. SET SKIP is most handy when the input data is not yours to change or is on a tape.
If the data is not indexed, then SPIBILD does not keep track of the SET SKIP figure. But in fact, recovery is much simpler because there will be no half-processed records that SPIBILD must discard in order to clean up properly.
The solution here is to use the RECOVER/PROCESS procedure to do any file cleanup that might be necessary, and then return to SPIRES to examine the file to determine where the processing stopped. If you had information messages turned on during the processing that failed, you will know from them how far the processing got. If not, you will need to examine records from the input data set, looking for their counterparts in the file. When you can't find one, that is the point where the processing stopped.
You can either delete the input to that point, or determine how many lines of the input should be skipped to begin processing at that point, including that number in a SET SKIP command when you restart the procedure.
SPIRES and SPIBILD provide several ways for you to update multiple records in a subfile in a single operation. The different methods work under different circumstances -- which one you choose will depend on the particular situation you have.
Below is a comparative summary of the methods. The procedures involving SPIRES are described in the "SPIRES Searching and Updating" manual. SPIBILD's INPUT BATCH, INPUT ADDUPDATE and INPUT ADDMERGE procedures were described in the previous chapter, but the other SPIBILD procedures are discussed later in this chapter. This chapter also includes a section about how SPIBILD determines when an updated record requires re-indexing. [See 2.4.3.]
A discussion about the format of the data in input data sets used with these procedures also appeared in the previous chapter. [See 2.3.5.]
There are twelve techniques for updating multiple records in a subfile in one fell swoop:
- 1. (SPIRES) the INPUT BATCH procedure
- 2. (SPIRES) the INPUT MERGE procedure
- 3. (SPIRES) the INPUT ADDMERGE procedure
- 4. (SPIRES) the INPUT ADDUPDATE procedure
- 5. (SPIRES) the BATCH procedure
- 6. (SPIRES) the MERGE procedure in Global FOR
- 7. (SPIRES) the WITH UPDATE procedure
- 8. (SPIBILD) the INPUT BATCH procedure
- 9. (SPIBILD) the INPUT MERGE procedure
- 10. (SPIBILD) the INPUT ADDMERGE procedure
- 11. (SPIBILD) the INPUT ADDUPDATE procedure
- 12. (SPIBILD) the MERGE procedure in Global FOR
With so many choices, it can be challenging to decide which procedure to use. The following list of questions, along with the descriptions that follow, may help you make up your mind:
Below are descriptions of each of the techniques.
You can use the INPUT BATCH command to add new records, modify existing records and remove existing records. SPIRES places the data into the file's deferred queue. When updates, merges or removes are involved, the data must be in the standard SPIRES format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.
The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.
Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner (or the account that next processes the file), not the user doing the input, who pays for only the input costs.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.
See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES. For online help, EXPLAIN INPUT BATCH.
You use the INPUT MERGE procedure to merge data into existing records in the subfile. An input format that handles merge processing does most of the work, usually getting the data, including the key of each record to be updated, from an input data set. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.
SPIRES places each updated record in the deferred queue, so unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. Hence, the cost of file processing is not borne by users of the INPUT MERGE procedure.
See the SPIRES manual "Searching and Updating" for more information about the INPUT MERGE command in SPIRES. For online help, EXPLAIN INPUT MERGE.
The INPUT ADDMERGE and INPUT ADDUPDATE procedures are very similar to the INPUT ADD precedure, using the input to add new records to the subfile. A custom input format does most of the work, usually getting the data, including the key of each record to be added, from an input data set. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set. If SPIRES finds that the input key matches the key of a record already in the subfile, then the input is either used to replace that record (ADDUPDATE) or to be merged into that record (ADDMERGE).
SPIRES places each updated record in the deferred queue, so unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. Hence, the cost of file processing is not borne by users of these INPUT procedures.
See the SPIRES manual "Searching and Updating" for more information about the INPUT ADDMERGE and INPUT ADDUPDATE commands in SPIRES. For online help, EXPLAIN INPUT ADDMERGE or INPUT ADDUPDATE.
The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.
This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.
The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user doesn't know till the next day whether the records were accepted, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.
The input data must be in the standard SPIRES format. It can include adds, updates, merges and removes.
A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched; but the request telling JOBGEN where the data is gets thrown away. This problem makes the BATCH command in SPIRES quite unreliable.
For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".
In SPIRES, the MERGE command can update multiple records, determined by the "FOR class" command in effect. The input data may be provided in several ways:
- it may be in the standard SPIRES format in the active file, with the data read only once and applied to each record being merged;
- it may be formatted input in the active file, with the data read and applied to each record being merged;
- it may be requested from the user by the merge format, or it may be provided by the format itself.
As the MERGE command executes, it merges the data into each record, placing the updated version in the deferred queue. Unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. The cost of file processing is not borne by users of this procedure.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic MERGE command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN WITH FAST PREFIX.
See the manual "Sequential Processing in SPIRES: Global FOR" for more information about the MERGE command in Global FOR.
WITH UPDATE is a command prefix for the DISPLAY and TYPE commands. Using the WITH UPDATE procedure, you can, to a limited degree:
- display records, simultaneously updating them; or
- display records, simultaneously updating other records in other subfiles or record-types; or
- update a record, simultaneously updating other records in other subfiles or record-types.
The most noteworthy limits of the WITH UPDATE procedure are:
- You must use a custom format containing special code to handle the extraordinary updating.
- The user must have secure-switch 14 set.
- You cannot add new occurrences or remove old occurrences of elements in the records being updated; you can only replace existing occurrences, with values that are the same lengths as the values being replaced.
- Inproc rules are executed for the replacement values, but not INCLOSE rules.
For more information about this specialized technique, see the manual "SPIRES Formats", chapter C.13, or "EXPLAIN WITH UPDATE PREFIX, RESTRICTIONS" online.
You can use the INPUT BATCH command in SPIBILD to add new records, modify existing records and remove existing records. The INPUT BATCH command applies the data in the input data set (in the standard SPIRES format) directly to the tree. For merges and updates, the INPUT BATCH command usually puts the data from the input data set (in the standard SPIRES format) into the deferred queue, which is then processed after all the input has been read.
The data may include adds, updates, merges and removes. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the data involved.
The INPUT BATCH procedure in SPIBILD is more expensive than the similar INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.
The input data may be on disk or tape, though if on tape, the procedure must be done through a Batch SPIBILD job.
The steps of this procedure were described in detail in the previous chapter. [See 2.3.1 or EXPLAIN INPUT BATCH PROCEDURE IN SPIBILD online.]
You can use SPIBILD's INPUT MERGE procedure to merge data into existing records in the subfile. [See 2.4.1.] You must use an input format that handles merge processing. It does most of the work, usually getting the data, including the key of each record to be updated, from an input data set. The data to be read may come from your active file or a data set stored on disk or tape.
Processing the file is part of the INPUT MERGE procedure -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. You cannot use this procedure unless your account can process the file.
Like their counterparts in SPIRES, the INPUT ADDMERGE and INPUT ADDUPDATE procedures in SPIBILD are very similar to the INPUT ADD precedure, using the input to add new records to the subfile. A custom input format does most of the work, usually getting the data, including the key of each record to be added, from an input data set. The data to be read must be in your active file or, if you use batch SPIBILD, a data set stored on disk or tape. If SPIRES finds that the input key matches the key of a record already in the subfile, then the input is either used to replace that record (ADDUPDATE) or to be merged into that record (ADDMERGE).
Processing the file is part of these INPUT procedures -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. You can use these procedures only if your account can process the file.
The INPUT ADDMERGE and INPUT ADDUPDATE procedures are discussed along with the INPUT ADD procedure in the previous chapter. [See 2.3.2 or EXPLAIN INPUT ADD PROCEDURE IN SPIBILD online.]
This procedure is similar to the INPUT MERGE procedure described above, except that the records to be updated are determined by the Global FOR criteria specified, not by the input data. The input data usually consists of one or more values that are all to be merged into each record. The input data may be in the standard SPIRES format, or a custom input format may be used. [See 2.4.2.]
The data can come from your active file, or, if you are using batch SPIBILD, from a tape or another WYLBUR data set. There may even be no input data set at all -- a custom format can provide the data itself, or it may reference the existing record to make changes based on its current values (such as reducing all occurrences of a PRICE element by half).
Processing the file is part of this procedure too -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. To use this procedure, your account must be able to process the file.
Procedure: INPUT MERGE in SPIBILD
Purpose: Merge data into multiple records (determined by the
input) using a custom input format
Who can use it: file owner
users with Process access to the file
Basic Steps:
1. SPIBILD
2. Prepare the input data set.
3. Issue any SPIBILD special-processing commands desired.
4. ESTABLISH subfile-name
5. Set the input format.
6. INPUT MERGE
The INPUT MERGE procedure merges data into records using a custom input format. The input data for each record must contain its key, so that SPIRES can retrieve the record for updating. [To merge data into records identified by criteria other than the key, use the MERGE procedure in Global FOR in SPIBILD. [See 2.4.2.]]
Typically, an input data set might contain a line for each record to be updated, containing the key and a new data value for one or more elements. For example, this could be an input data set for a mythical doctor's billing system, where the data in the first column are keys to records already in the subfile:
N1785 9/17/86 Payment $5.00 B7922 9/17/86 Office Visit $4.00 C0018 9/17/86 Payment $8.00
What happens to the input data is determined by the input format. It could be added as new data, or as replacement data, or as data that causes existing data to be removed or recomputed (the data above could cause a BALANCE element in each record to be refigured, for instance). Information on creating merge formats appears in chapters C.5 and C.9 of the manual "SPIRES Formats".
Be aware that you can use this procedure only if you have a custom merge format that reads each key and its associated data. If the data to be merged is in the standard SPIRES format, you should simply use SPIBILD's INPUT BATCH procedure. [See 2.3.1, 2.3.5.]
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL command stream.
If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set on disk.
Be sure the input data set can read the input data and apply it properly to individual SPIRES records. Always make certain the format works properly before trying to use it to merge hundreds of records in a subfile -- a small problem can be quite costly when multiplied by hundreds of records.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET EXCEPTION command, to name an ORVYL data set into which unacceptable records from the input data set are placed. [See 2.1.5.2.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET SKIP command, which skips the specified number of lines in the input file. It is used for recovery purposes when a previous SPIBILD task has failed during a batch input process. [See 2.3.6.]
- the SET STACK SIZE command, to increase the size of the pass stack. [See the description for that command in Part 6, or online, EXPLAIN SET STACK SIZE COMMAND.]
- the SET PTRACE command, in one or more of its forms, to display tracing information for input or output processing rules (Inproc, Inclose, Outproc) executed during the record input processing.
The ESTABLISH command in SPIBILD "selects" the named subfile and processes the file containing it.
For online SPIBILD, and the BATWYL program: ESTABLISH [&gg.uuu.filename] subfile-name For batch SPIBILD: ESTABLISH [&gg.uuu.filename] 'subfile-name'
You must set the desired format. Depending on what the format does, any of the commands listed below may be involved. All are optional except for SET FORMAT; those you use should be issued in the order shown:
- ALLOCATE. This command allocates a vgroup, usually so that stored values can be restored to it (see RESTORE STATIC).
- RESTORE STATIC. This command restores previously stored values to a vgroup. [See "SPIRES Protocols" for more information about these two commands.]
- DEFINE AREA and ASSIGN AREA. These commands (as well as CLOSE AREA to be issued after the FAST BATCH command) are needed when the input format is also creating a report. [See 2.1.5.5.]
- SET FORMAT. The SET FORMAT command sets the named format, or, if the "*" form of the command is used, executes the startup frame of the format named in the previous SET FORMAT command. You must start with a "SET FORMAT format-name" command, with one or more SET FORMAT * commands optionally to follow.
SET FORMAT {format-name|*} [,parm|'parm'|"parm"]
- SET FTRACE. SET FTRACE enables format-tracing messages, most often used for debugging. [See "SPIRES Formats", section B.7.2.]
The INPUT MERGE command in SPIBILD merges the input data into the subfile according to the specifications of the set format.
For online SPIBILD, and the BATWYL program: [USING frame] INPUT MERGE For batch SPIBILD: [FROM ddname] [USING frame] INPUT MERGE
Here is a sample online SPIBILD session showing typical usage of the INPUT MERGE procedure:
Command> spibild -Welcome to SPIBILD -? use grades.on.test6 -? set exception bad.grades replace -? set information messages = 0 -? establish student grades -Compute time: 0.146 seconds -Elapsed time: 1.502 seconds -Core usage: 0065/0933 -? input merge - Requests/Success: MER 462 462 SUM 462 462 -End of BATCH/BUILD -Passing: 462 records of record-type REC01 -Processed: 462 DEFQ records of record-type REC01 -Compute time: 5.762 seconds -Elapsed time: 28.122 seconds -Core usage: 0256/0933 -?
This example shows a case where the merge data caused indexes to be updated; in some cases, no index data is involved, so the records don't need to be re-indexed, which is somewhat cheaper. [See 2.4.3.1.]
The most common problem that occurs is that one or more records fail to get updated because an Inproc processing rule rejects the data. The solution is to find the rejected data, correct it, and try the procedure again with only that data in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected data there. [See 2.1.5.2.]
System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.
- simply restart the procedure using the same data. This is the simplest method, but don't use it if the merge requests are creating new element or structure occurrences, or, say, causing addition into amount elements. Completely resubmitting such requests would cause erroneous data in those records that were successfully processed the first time.
- determine the last record that was updated. SPIBILD's information messages -- if you didn't turn them off -- can be useful here, but you should check the records actually in the file in SPIRES. This might be tedious, but it is the most efficient method (it means you won't update the same records again). You can delete the data for the successful records from the front of the input data set and then restart the procedure. Alternatively, restart the procedure and include the SET SKIP command to tell SPIBILD how many lines of the input to skip in step 3.
Procedure: MERGE in SPIBILD with Global FOR
Purpose: Merge data into multiple records determined by
Global FOR
Who can use it: file owner
users with PROCESS access to the file
Basic Steps:
1) Create a stored stack or result, if appropriate.
2) SPIBILD
3) Prepare the input data set, if any.
4) Issue any SPIBILD special-processing commands desired.
5) ESTABLISH subfile.name
6) Set the input format, if any.
7) Establish the Global FOR environment.
8) Issue the MERGE command.
This MERGE procedure merges data into records determined by the established Global FOR environment. This contrasts with the "normal" MERGE procedure in SPIBILD, where the keys of the records to be updated are contained in the input data set. Under Global FOR, however, an input data set may or may not exist. If it does, it almost always contains pieces of data that are all to be merged into each record, with or without the aid of a custom format. If the input data set doesn't exist, then the data is supplied or computed by a custom format directly.
Information on writing merge formats to be used with Global FOR in SPIBILD appears in section C.9.4 of the manual "SPIRES Formats".
If you are merging into all the records in the subfile, skip ahead to step 2. In SPIBILD, you can use the FOR SUBFILE command with a WHERE clause to determine which records to update. However, it is much more efficient, if you can do this, to search for the records using indexes in SPIRES, store the result, and then use the FOR STORED command in SPIBILD to identify the records in the stored result. This technique also works for stored stacks that were created in SPIRES.
To do this, create your search result or stack in SPIRES, using commands like FIND, SEQUENCE and STACK. Then, depending on whether you have a result or a stack, issue the STORE RESULT.name or STORE STACK.name command to save the result or stack as an ORVYL data set. Don't use the TEMPORARY option on those commands; the result or stack won't be around long enough for SPIBILD to use. [If you have a result, there is a slim possibility that FOR STORED cannot work with it. To avoid this problem, you may want to turn the result into a stack (e.g., by issuing the commands FOR RESULT and STACK ALL) before storing it. See the technical information on the FOR STORED command in Chapter 6 of this manual; online, [EXPLAIN FOR STORED COMMAND.]]
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command.
If the data to be merged will be read from an input data set, prepare it for use. If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set on disk.
If you are using an input format to read the data, be sure it works properly before trying to use it to update hundreds of records in a subfile -- a small problem can be quite costly when multiplied by hundreds of records.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET PTRACE command, in one or more of its forms, to display tracing information for input or output processing rules (Inproc, Inclose, Outproc) executed during the record input processing.
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
Issue the ESTABLISH command, which processes the file and prepares the subfile for direct update operations.
For online SPIBILD, and the BATWYL program: ESTABLISH subfile.name For batch SPIBILD: ESTABLISH 'subfile.name'
If you are using an input format, you set it now; otherwise, skip to step 7. Depending on what the format does, any of the commands listed below may be involved. All are optional except for SET FORMAT; those you use should be issued in the order shown:
- ALLOCATE. This command allocates a vgroup, usually so that stored values can be restored to it (see RESTORE STATIC).
- RESTORE STATIC. This command restores previously stored values to a vgroup. [See "SPIRES Protocols" for more information about these two commands.]
- DEFINE AREA and ASSIGN AREA. These commands (as well as CLOSE AREA to be issued after the FAST BATCH command) are needed when the input format is also creating a report. [See 2.1.5.5.]
- SET FORMAT. The SET FORMAT command sets the named format, or, if the "*" form of the command is used, executes the startup frame of the format named in the previous SET FORMAT command. You must start with a "SET FORMAT format-name" command, with one or more SET FORMAT * commands optionally to follow.
SET FORMAT {format-name|*} [,parm|'parm'|"parm"]
- SET FTRACE. SET FTRACE enables format-tracing messages, most often used for debugging. [See "SPIRES Formats", section B.7.2.]
Establish the Global FOR environment by issuing the FOR SUBFILE or FOR STORED command, optionally followed by other Global FOR commands mentioned below. The purpose of these commands is to identify the specific records to be updated.
No VIA clause is allowed on the FOR SUBFILE command, but you may add a WHERE clause as needed, to limit the records to be updated using element criteria:
FOR SUBFILE [WHERE clause]
If you have stored a stack or result containing the records you want to update (see step 1), you would use the FOR STORED command:
FOR STORED {RESULT.name|STACK.name} [WHERE clause]
The SET SCAN commands can give you further flexibility (and savings) in determining which records are to be examined for processing. The SET SCAN commands allowed in SPIBILD under FOR SUBFILE are:
SET SCAN BACKWARD SET SCAN LIMIT n SET SCAN PREFIX string SET SCAN START string SET SCAN STOP string
Only SET SCAN LIMIT is allowed under FOR STORED. Note that SET SCAN PREFIX has no meaning with slot subfiles.
Details about these commands, including WHERE clauses, appear in the manual "Sequential Processing in SPIRES: Global FOR".
The SET FILTER command, with the TYPE = SCAN option, works similarly to a WHERE clause but is more versatile. You can, for example, use it to locate records when you have complicated criteria involving elements within structures, a situation that WHERE clauses often cannot handle. See section 4.4.2 of the Global FOR manual referenced above; online [EXPLAIN FILTERS, ON WHERE CLAUSES.]
The SKIP command tells SPIBILD to skip the specified number of records meeting the Global FOR criteria:
SKIP [n|FIRST]
where "n" is the number of records to be skipped. If you specify neither "n" nor FIRST (for the FIRST record), the default is FIRST. Except for the limited options, SKIP works the same in SPIBILD as it does in SPIRES. See section 5.10 of the Global FOR manual; online [EXPLAIN SKIP COMMAND.]
This example shows how a WHERE clause and SET SCAN processing may be used to identify the records to be updated in the least expensive manner. Suppose you want to update all the records with ADD.DATE element values of 1986 whose keys begin with the code KL:
-? for subfile where add.date 1986 -? set scan prefix kl -?
These two commands work more efficiently together than would the single command FOR SUBFILE WHERE ADD.DATE 1986 AND ID PREFIX KL. The SET SCAN command limits the records that are examined to those whose "id" begins with KL; but with the single command, SPIBILD would examine all records, looking for those whose "id" begins with KL, which might be hundreds or thousands more records. If you will be using MERGE processing under Global FOR in SPIBILD, you should be familiar with the SET SCAN commands, which can save you a great deal in processing costs.
Note in the example that SPIBILD does not use the Global FOR prompt that SPIRES uses (+?) after you issue the FOR SUBFILE command.
The MERGE command in SPIBILD merges the input data into the named subfile according to the specifications of the set format.
MERGE [ALL|NEXT|n|REST] [FROM {ACTIVE|ddname}]
Again, "n" represents the number of records you want processed; the default is REST. You need to specify the FROM option if you want SPIBILD to read data from the active file (or from a DD data set in batch SPIBILD). But if the input data is being provided by an input format that does not need to read the data from the active file, you should omit the FROM option.
Here is a sample online SPIBILD session showing typical usage of the MERGE procedure in SPIBILD:
Command> list
1. manager = Sandy Claws;
Command> spibild
-Welcome to SPIBILD
-? set information messages = 0
-? establish personnel
-Compute time: 0.138 seconds
-Elapsed time: 1.638 seconds
-Core usage: 0065/0933
-? for subfile where supervisor = ebenezer scrootch
-? merge all from active
- Requests/Success:
MER 24 24
SUM 24 24
-End of BATCH/BUILD
-Passing: 24 records of record-type EMPL01
-Processed: 24 DEFQ records of record-type EMPL01
-Compute time: 2.119 seconds
-Elapsed time: 8.152 seconds
-Core usage: 0256/0933
-?
This example shows a case where the merge data caused indexes to be updated; in some cases, no index data is involved, so the records don't need to be re-indexed, which is somewhat cheaper. [See 2.4.3.1.]
Because this procedure usually applies the same data to all the records it updates, if the data fails to go into one record, it usually fails to go into all of them. In most cases where input failure occurs, you just correct the input data and restart the procedure.
System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.
- simply restart the procedure using the same data. This is the simplest method, but don't use it if the merge requests are creating new element or structure occurrences, or, say, causing addition into amount elements. Completely resubmitting such requests would cause erroneous data in those records that were successfully processed the first time.
- determine the last record that was updated. SPIBILD's information messages -- if you didn't turn them off -- can be useful here, but you should check the records actually in the file in SPIRES. This might be tedious, but it is the most efficient method (it means you won't update the same records again). You should issue the SKIP command (see above) to skip the successful records when you restart the procedure.
- Since SKIP works only with FOR SUBFILE and not FOR STORED, with FOR STORED processing, you might want to recreate your result or stack in SPIRES, eliminating the successfully-updated records, store the new result or stack, and restart the procedure using that one.
In most situations where you are updating multiple records in a single pass, the distinctions between the procedures described at the beginning of this chapter are adequate for choosing the right one. And in terms of cost, for smaller files the differences between the procedures are seldom very significant.
For large files or files with somewhat unusual characteristics, however, the differences can be quite significant. Moreover, some extra thought and work on your part may lead to considerable savings that SPIBILD would not provide you on its own.
The situations covered in the subsections of this section require fairly technical discussions, and this section should be considered optional reading unless:
- you are concerned about cutting costs any way you can; and
- at least one of the following is true:
- you are working with a file containing thousands of records;
- you will be updating hundreds of records at a time;
- you are working with extremely large records, containing hundreds of element values, many of which are indexed.
Techniques are available that alter the standard way SPIBILD processes data, to increase efficiency when the input data or the file is different from the norm. The techniques, described in separate sections that follow, are:
- 1) the SET NOPASS command, useful when merging data in SPIBILD that has no effect on any index records; [See 2.4.3.2.]
- 2) the PASS-ELEM statement in file definitions, which has a similar use but which is built into the file's characteristics; [See 2.4.3.3.]
The first subsection provides a brief overview of how updated records affect indexes, which is useful information to know if you must decide whether to use any of these efficiency techniques.
In SPIRES, whether records are updated or merged, they are placed in the deferred queue as "updates" -- that is, SPIRES places the complete updated record in the deferred queue, so that it can replace the existing tree copy of the record. Merge data by itself is not stored in the deferred queue; SPIRES applies it to the record immediately, and places the updated record in the deferred queue as an "update".
SPIBILD has different ways to handle merge and update requests, i.e., when the data is being merged by BATCH or MERGE commands in SPIBILD. For updates, which are handled by the BATCH command only, SPIBILD places each updated record in the deferred queue. As soon as SPIBILD finishes with the input data, it processes the deferred queue again, moving the updated records into the tree (removing the old copies) and determining what changes need to be made to the indexes. [The technical details of how SPIBILD efficiently re-indexes updated records were presented earlier in this manual. In brief, the index data is generated for the old record, each item marked as a "delete", and the index data is generated for the new copy, with each item marked as an "add". SPIBILD sorts the index data, discarding any pairs that are identical except for opposite "add" and "delete" markers. What isn't discarded then gets applied to the indexes. [See 2.2.6.]]
For updates, SPIBILD needs to check the data to see if the indexes need to be changed because it doesn't know what elements, if any, have been changed from one copy of the record to the next; all it knows is that it has a new copy of the record to replace the old one. [When SPIBILD gets an updated record, it is a new copy of the existing record, which may or may not have been changed. SPIBILD assumes the record has been changed, and replaces the old copy with the new one in the tree. During indexing, however, SPIBILD determines which indexed values have not changed from the old record to the new, and discards that data from the pass stack, as described in the previous note.]
For merges, the circumstances are different: SPIBILD knows exactly what elements are being updated, since it is doing the work. Hence, it may determine that the only elements being updated by a merge request are non-indexed ones, so the merged record doesn't need to go to the deferred queue for re-indexing; instead, it can go directly to the tree. SPIBILD can do this for goal record-types that have a "pass-elements list", created when the file is compiled, which identifies the elements that affect indexes -- if the merge request doesn't change any elements on the list, there's no need to re-index the record.
On the other hand, if the merge request does change an element in the pass-elements list, the record goes to the deferred queue for re-indexing. Naturally, the trip to the deferred queue means more work for SPIBILD (and more cost to you) since all the re-indexing work must be done.
These facts lead to several conclusions:
- In SPIBILD, it's more efficient to merge data that doesn't affect indexes than data that does.
- If you are updating records with data that doesn't affect indexes, the total processing cost (i.e., to get the data into the tree) will be less if you:
- use the SET NOPASS command to turn off index passing; or
- merge the data in SPIBILD, taking advantage of the pass-elements list, rather than update the records in SPIRES or SPIBILD.
The next two sections will discuss the two methods mentioned above:
When updated records in the deferred queue are processed, SPIBILD must generate two sets of index data for each record: the data for the tree copy, to be deleted, and the data for the new copy, to be added. To avoid unnecessary work, SPIBILD sorts them together, so that pairs that are identical except for the "add" or "delete" designation are discarded -- those pairs indicate element values that were not changed, which thus don't need to be re-indexed.
If a record is updated but no changes are made to the elements that affect indexes, then SPIBILD will create all the index data and subsequently discard it. This superfluous work may be worth avoiding in applications where non-indexed data is updated regularly.
As described in the next section, on pass-element lists, SPIBILD tries to avoid this unnecessary work in the case of merges done in SPIBILD (though without some intervention on the file owner's part, SPIBILD may not be able to avoid it).
The SET NOPASS command is a fairly simple command you can use for updates and merges no matter whether you do them in SPIRES or SPIBILD -- it guarantees the extra work won't be done. If you are merging records in SPIBILD, using SET NOPASS can save you the trouble of determining whether a pass-elements list is in effect.
The technique is to add the SET NOPASS command to the SPIBILD procedure you are using, at the time prescribed in the procedure's description. For instance, if you are processing a file in SPIBILD after updating records in SPIRES, you issue the SET NOPASS command before the PROCESS command in SPIBILD. [See 2.2.2.]
WARNING: Do not use this technique unless you are absolutely certain that no data in the deferred queue affects any indexes. If it does, and you use the SET NOPASS command to turn off passing, then data integrity problems (discrepancies between the goal records and the indexes) will occur. If you aren't certain, or if you can't control the update procedure (including file processing) from start to finish, do not use the SET NOPASS command.
You might want to process the file first, before you apply any of the updates, in order to clear any other data from the deferred queue. In that case, don't issue the SET NOPASS command until you are ready to process the new data -- you want the data already in the deferred queue to be processed normally.
Merged record processing in SPIBILD can be more efficient if SPIBILD finds a "pass-elements list", a list of the elements that affect indexing. When only non-indexed elements are merged, SPIBILD can put the record directly into the tree, rather than putting it in the deferred queue for indexing later.
Unfortunately for many subfiles, SPIRES can't always create a pass-elements list on its own, and if you don't help it along, it won't create one. When a subfile doesn't have one, SPIBILD has no idea whether any given element will affect an index, and so must place all merged records back into the deferred queue. To understand why this is true, we need to know more about pass-elements lists.
When it compiles a file definition, SPIRES creates the pass-elements list from the elements it knows are being passed to indexes. Specifically, SPIRES places the following elements into the pass-elements list:
- elements named in the GOALREC-NAME statement; and
- elements named in any of the following Passproc rules:
- $PASS.ELEM proc (A167)
- $PASS.UPPER proc (A162)
- $PASS.BY proc (A163)
- $PASS.COND proc (A168)
SPIRES only builds this list if it is sure that no other elements in the record-type can affect the index values. If such an element existed and if it were changed in a SPIBILD merge request but no elements in the pass-elements list were changed, then the related index values would not be updated, since SPIBILD would only re-index the record if an element on the pass-elements list had been merged too.
SPIRES doesn't build the list if any of the above-listed elements:
- is a virtual element; or
- has the $CALL proc (A62) in its Passproc string; or
- is passed in its output form, i.e., is processed through its Outproc string before being passed.
If SPIRES doesn't build the list, then all merged records handled by SPIBILD are placed in the deferred queue for index processing. This certainly isn't a tragedy, but obviously, if no indexed elements are affected by the merge data, then SPIBILD's having to regenerate index information is extra and unnecessary work. In a large application, such inefficiency could be quite expensive.
You can tell SPIRES to create the pass-elements list anyway, even if the circumstances that would prevent its creation occur, by adding the PASS-ELEM statement to the file definition, at the end of the appropriate linkage section:
PASS-ELEM; or PASS-ELEM = element.list;
You would want to specify in the element list any elements not included in the list as described above that SPIRES should add to the list -- that is, elements whose values affect the passed values of other elements.
The PASS-ELEM statement cannot be added with File Definer; you must change the file definition directly. Select the FILEDEF subfile, retrieve the current copy of your file definition, and add the PASS-ELEM statement as described above to the appropriate linkage section. Then update the record and recompile the file definition in SPIRES.
Here are some examples describing how you might use pass-element lists. Suppose that the only reason SPIRES won't create a pass-element list for your subfile is because one of the elements you are passing is a virtual element but whose value has nothing to do with any other element. You could tell SPIRES to create the list anyway, just by adding "PASS-ELEM;" to the file definition.
Or suppose, for another example, that the indexed virtual element SHIPPING.DATE is created by adding 10 days to the value of the non-indexed element ORDER.DATE. If you code in the file definition:
PASS-ELEM = ORDER.DATE;
then SPIRES will create a pass-elements list, including ORDER.DATE.
WARNING: Don't code the PASS-ELEM statement unless you're sure that either:
- all elements that have any effect on indexes are listed in the PASS-ELEM statement, or will be placed in the list automatically, as described above; or,
- you will never use SPIBILD to merge any elements not on the list that are supposed to affect indexed values. If you do, the indexes won't get updated properly, and data integrity problems will result.
Removing records from a subfile in a single operation is a simple task. The SPIRES and SPIBILD procedures that remove records are easy to use, and much less complex than those for adding or updating records. There are several different methods, each of which shines under separate circumstances.
Below is a set of questions and answers to help you decide which technique to use, followed by a comparative summary of the methods. Two of the techniques are described in detail in this chapter. The others have been described either in Chapter 2.3, or in other SPIRES manuals.
There are seven ways to remove multiple records in a single operation:
- 1. (SPIRES) the REMOVE procedure in Global FOR
- 2. (SPIRES) the INPUT BATCH procedure
- 3. (SPIRES) the BATCH procedure
- 4. (SPIBILD) the REMOVE procedure in Global FOR
- 5. (SPIBILD) the INPUT BATCH procedure
- 6. (SPIBILD) the ZAP RECORD-TYPE procedure
- 7. (SPIRES) the ZAP FILE procedure
Here are some questions to help you determine which method to use, followed by descriptions of each.
- If you have a set of the keys of records to be removed, use the INPUT BATCH or BATCH procedures.
- If you are using indexed criteria, use the REMOVE procedure in SPIRES, in conjunction with the FOR RESULT command.
- If you are using non-indexed criteria, use the REMOVE procedure in Global FOR in either SPIRES or SPIBILD.
You can use the REMOVE command in SPIRES with any Global FOR class, such as SUBFILE, RESULT and STACK. That means you can remove any and all records in the given class. This procedure is available to any user with update privileges to the subfile.
SPIRES places a removal request for each record into the deferred queue; during subsequent file processing, the records will be removed, and any necessary "un-indexing" will be done. Hence, the indexes aren't updated (except for any immediate indexes) at the time the REMOVE command is issued. File processing costs are picked up by the file owner (or the next account to process the file), not by the user removing the records.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic REMOVE command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.)
See the manual "Sequential Record Processing in SPIRES: Global FOR" for more information about the REMOVE command in Global FOR; online, EXPLAIN REMOVE COMMAND, IN GLOBAL FOR. Information about these two prefixes also appears in Part 6 of this manual, under INPUT BATCH, with which they work the same way in SPIRES.
You can use the INPUT BATCH command to add new records, modify existing records and remove existing records. SPIRES places the data into the file's deferred queue. When updates, merges or removes are involved, the data must be in the standard SPIRES format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.
The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.
Because the removal requests go into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records aren't removed from the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner (or the account that next processes the file), not the user removing the records, who pays for only the input costs.
Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.
See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES. For online help, EXPLAIN INPUT BATCH.
The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). The input data must be in the standard SPIRES format. It can include adds, updates, merges and removes.
During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile, discarding records that are to be removed. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.
This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.
The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user doesn't know till the next day whether the records were removed, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.
A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched; but the request telling JOBGEN where the data is gets thrown away. This problem makes the BATCH command in SPIRES quite unreliable.
For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".
This procedure is similar to its cousin in SPIRES in that you establish a Global FOR class and then issue a REMOVE command to remove the records in that class.
The Global FOR class can be only SUBFILE or STORED in SPIBILD. (STORED, available in SPIBILD only, means that the records to be examined are in a stored stack or result.) You may include a WHERE clause on the FOR SUBFILE or FOR STORED command, and you may also use the SET FILTER and SET SCAN commands to limit the records removed.
If you are removing only a few of many records in a subfile, using REMOVE under Global FOR in SPIRES is probably more efficient than using it in SPIBILD. On the other hand, the REMOVE command in SPIBILD removes the records from the subfile immediately, unindexing them immediately as well. The user pays for all the processing, including the processing of any data already in the deferred queue. Because the file gets processed, only users with Process access to the file are allowed to use the REMOVE command in SPIBILD. [See 2.5.1.]
You can use the INPUT BATCH command in SPIBILD to add new records, modify existing records and remove existing records. For removes, the INPUT BATCH command removes the records identified by key directly from the tree.
The INPUT BATCH procedure in SPIBILD is more expensive than the similar INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.
The input data may be on disk or tape, though if on tape, the procedure must be done through a Batch SPIBILD job.
The steps of this procedure were described earlier. [See 2.3.1.]
The ZAP RECORD-TYPE command discards all the data in a specified record-type of a file. The ZAP FILE command erases all the data sets of an entire file. Though these sound like the ultimate solutions for removing records from a subfile, they are the best methods to use when you want to remove all the records from a subfile.
Moreover, they are worth considering in situations where you aren't removing all the records of a large subfile but are removing a large majority. In particular, consider saving the few records you want to keep, zapping the record-type (or the file) and then adding the kept records back in to a completely empty record-type.
Several benefits may come from this method, including the release of unneeded blocks for storage, and the savings from not "unindexing" all the records you're removing -- it costs as much to remove a record as to add it, because it takes SPIBILD the same amount of work to unindex a record as to index it in the first place.
Only the file owner can use the ZAP FILE procedure. [See 2.11.] Users with Master access to the file can use the ZAP RECORD-TYPE procedure. [See 2.5.2.]
Procedure: REMOVE in SPIBILD with Global FOR
Purpose: Remove multiple records (determined by Global FOR)
from a subfile
Who can use it: file owner
users with PROCESS access to the file
Basic Steps:
1) Create a stored stack or result, if appropriate.
2) SPIBILD
3) Issue any SPIBILD special-processing commands desired.
4) ESTABLISH subfile.name
5) Establish the Global FOR environment.
6) Issue the REMOVE command.
The REMOVE procedure removes records determined by the established Global FOR environment. The Global FOR environment in SPIBILD is not as flexible as the one in SPIRES -- the only "FOR class" commands available are FOR SUBFILE and FOR STORED.
If you have a search result or stack of records, or can use some other Global FOR class of records in SPIRES, you can remove them using that class in SPIRES, and then process the file if you want. Or, to avoid the extra work of putting the removal requests into the deferred queue, you can store the search result or stack, and then use the REMOVE command under FOR STORED in SPIBILD to remove them, which also processes the file and updates the indexes appropriately.
Either method will almost certainly be cheaper than using a WHERE clause on a FOR SUBFILE command in SPIBILD, especially if the subfile is large. But if identifying the records via indexes isn't available, this other method will do.
Here are the steps of the procedure:
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
Several optional commands or procedures may be useful at this point, such as:
- the SET MESSAGES command, to control the messages SPIBILD displays. The command "SET INFORMATION MESSAGES = 0" is especially useful, since it turns off the individual "add" message displayed for each new record. [See 2.1.5.3.]
- the SET SEARCH/SET SHARE procedure to share access to the file with other users. [See 2.1.5.1.]
- the commands that limit the record-types passed from and to: SET INDEX, SET NOINDEX, SET PASS and SET NOPASS. Remember, however, that they are primarily meant to solve file integrity problems and should be used only after consultation with your SPIRES consultant. [See 2.1.5.4, 2.4.3.2.]
- the SET STACK SIZE command, to increase the size of the pass stack. See the description for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
Issue the ESTABLISH command, which processes the file and prepares the subfile for direct update operations.
For online SPIBILD, and the BATWYL program: ESTABLISH subfile.name For batch SPIBILD: ESTABLISH 'subfile.name'
Establish the Global FOR environment by issuing the FOR SUBFILE or FOR STORED command, optionally followed by other Global FOR commands mentioned below. The purpose of these commands is to identify the specific records to be removed.
No VIA clause is allowed on the FOR SUBFILE command, but you may add a WHERE clause as needed, to limit the records to be updated using element criteria:
FOR SUBFILE [WHERE clause]
If you have stored a stack or result containing the records you want to update (see step 1), you would use the FOR STORED command:
FOR STORED {RESULT.name|STACK.name} [WHERE clause]
The SET SCAN commands can give you further flexibility (and savings) in determining which records are to be examined for processing. The SET SCAN commands allowed in SPIBILD are:
SET SCAN BACKWARD SET SCAN LIMIT n SET SCAN PREFIX string SET SCAN START string SET SCAN STOP string
Only SET SCAN LIMIT is allowed under FOR STORED. Note that SET SCAN PREFIX has no meaning with slot subfiles.
Details about these commands, including WHERE clauses, appear in the manual "Sequential Processing in SPIRES: Global FOR".
The SET FILTER command, with the TYPE = SCAN option, works similarly to a WHERE clause but is more versatile. You can, for example, use it to locate records when you have complicated criteria involving elements within structures, a situation that WHERE clauses often cannot handle. See section 4.4.2 of the Global FOR manual referenced above; online [EXPLAIN FILTERS, ON WHERE CLAUSES.]
The SKIP command tells SPIBILD to skip the specified number of records meeting the Global FOR criteria:
SKIP [n|FIRST]
where "n" is the number of records to be skipped. If you specify neither "n" nor FIRST (for the FIRST record), the default is FIRST. Except for the limited options, SKIP works the same in SPIBILD as it does in SPIRES. See section 5.10 of the Global FOR manual; online [EXPLAIN SKIP COMMAND.]
This example shows how to use a WHERE clause and SET SCAN processing to identify the records to be removed in the least expensive manner. Suppose you want to remove all the records with MOD.DATE element values before 1986. You also know that all those records have slot keys less than 20000:
-? for subfile where mod.date < 1986 -? set scan stop 20000 -?
These two commands work more efficiently together than would the single command FOR SUBFILE WHERE MOD.DATE < 1986 AND ID < 20000. The SET SCAN command limits the records that are examined to those whose "id" is less than 20000; under only the FOR SUBFILE command, SPIBILD would examine all records, looking for those whose "id" is less than 20000, which might be hundreds or thousands more records. If you will be processing records under Global FOR in SPIBILD, you should be familiar with the SET SCAN commands, which can save you a great deal in processing costs.
Note in the example that SPIBILD does not use the Global FOR prompt that SPIRES uses (+?) after you issue the FOR SUBFILE command.
The REMOVE command in SPIBILD removes the records determined by the Global FOR class.
REMOVE [ALL|NEXT|n|REST]
Here "n" represents the number of records you want processed; the default is REST.
The REMOVE command first locates each record to be removed and places a "remove request" in the deferred queue for it, just as the REMOVE command does in SPIRES. When the Global FOR traversal is complete, SPIBILD processes the deferred queue again, which is when the records are actually removed and the indexes updated.
Here is a sample online SPIBILD session showing typical use of the REMOVE procedure:
-> spibild -Welcome to SPIBILD -? set information messages = 0 -? establish test inquiries -Processing file: GQ.JNK.TEST.INQUIRIES -Compute time: 0.264 seconds -Elapsed time: 3.214 seconds -Core usage: 0065/0933 -? for subfile -? set scan stop 100 -? remove all - Requests/Success: REM 100 100 SUM 100 100 -End of BATCH/BUILD -Passing: 100 records of record-type REC01 -Processed: 100 DEFQ records of record-type REC01 -Compute time: 6.416 seconds -Elapsed time: 19.526 seconds -Core usage: 0256/0933 -?
This example shows a case where all records whose keys are no greater than 100 are removed. The removal requests go into the deferred queue first; the "End of BATCH/BUILD" message indicates the start of the deferred queue processing, when the records will actually be removed.
The simplest way to recover from system failure is to restart the procedure. But there are two situations in which alternate methods might be cheaper to do:
1. If SPIBILD had printed the "End of BATCH/BUILD" message before the system failure, then SPIBILD had finished placing all the removal requests in the deferred queue and was starting to process the deferred queue to complete the job. To recover, you should use the PROCESS/RECOVER procedure described earlier in this manual. [See 2.1.4.2.]
2. If the REMOVE command under FOR SUBFILE had already removed many records before the failure occurred, you might want to consider adding a SET SCAN START command to the procedure. That way, SPIBILD won't have to re-examine all the records it examined the first time that weren't discarded.
It is easy to find out what record was the last one removed:
- 1) select the subfile in SPIRES
- 2) issue the FOR REMOVES command
- 3) issue the SKIP LAST command
- 4) issue the "SHOW KEY *" command
The key value shown will be the last one removed; use that as the start value in the SET SCAN START command.
On the other hand, if you were working under FOR STORED, you would probably resume by restarting the procedure. It will do no harm for SPIBILD to try removing records that are already removed.
Procedures: ZAP RECORD-TYPE
ZAP DATA SET
Purpose: Remove all records from a record-type
Who can use it: file owner
users with Master access to the file
Basic steps:
1) SPIBILD
2) ESTABLISH subfile-name
3) ZAP RECORD-TYPE recname (or ZAP DATA SET dataset.name)
The ZAP RECORD-TYPE procedure erases all the records in a record-type very quickly and very efficiently. If you think of a SPIRES file as being a file cabinet, the ZAP DATA SET or ZAP RECORD-TYPE commands are equivalent to tossing out a drawerful of file folders. In contrast, using the REMOVE ALL command under Global FOR in SPIRES or SPIBILD is akin to pulling out the folders one by one, checking to see if each one is indexed in another drawer of the cabinet and deleting that index entry too.
You will need to choose between the ZAP RECORD-TYPE and ZAP DATA SET commands when executing this procedure. Generally speaking, use the ZAP RECORD-TYPE command; however, choose the ZAP DATA SET command if the record-type is combined with other record-types in a single ORVYL data set, but you want to discard all of them.
The discussion of "Preliminaries" below will tell you how to determine whether the record-type is combined with others or not.
The ZAP RECORD-TYPE procedure pays no heed to indexes -- it simply discards all the data in the record-type or data set you name. But in most cases when you are removing all the records in a goal record-type, either it has no indexes or the indexes are in their own record-types, not connected to any other goal record-type, and they too can be erased with the ZAP DATA SET procedure. (Of course, if you are discarding all the record-types in the file, the ZAP FILE procedure [See 2.5.3.] is even better.) Unconditionally emptying a goal record-type and its associated index record-types will be much more efficient than having SPIBILD remove the goal records one by one, creating and applying pass stack data.
Before beginning this procedure:
- The name of the file is the key of the file definition. If you aren't the file owner, you can get the name by selecting a subfile of the file in SPIRES and issuing the SHOW SUBFILE INFORMATION command.
- The name of the record-type appears in the file definition. Also, if the record-type is a goal record-type of a subfile, you can get the information in SPIRES by selecting the subfile and issuing the SHOW SUBFILE INFORMATION command.
- The SHOW FILE ACTIVITY command in SPIRES can tell you what data set a given record-type is in. The data set name is "REC" followed by the integer (from 1 to 9) in the "DS" column of the display.
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
Issue the ESTABLISH command so that SPIBILD will know which file to use in subsequent commands, and so that SPIBILD will process the deferred queue.
For online SPIBILD and the BATWYL program: ESTABLISH subfile-name For batch SPIBILD: ESTABLISH 'subfile-name'
The deferred queue must be empty for the ZAP DATA SET and ZAP RECORD-TYPE commands to work. (If the deferred queue contains worthless data, you might want to consider issuing the ZAP DEFQ command in SPIRES as a preliminary step. That would save you the expense of processing the deferred queue.)
Specify the name of any subfile in the file containing the record-type or data set.
Issue the command you chose to use, based on the information given earlier; either:
ZAP RECORD-TYPE recname
where "recname" is the name of the record-type to be erased; or
ZAP DATA SET RECn
where "n" is the number/letter of the physical data set you want to erase.
Warning: Both of these commands take destructive action immediately. Double-check that you have correctly typed the data set number or record-type name before pressing the Return key. Recovering the data if you make a mistake is usually very difficult, if indeed it is possible at all (see below).
These two ZAP commands may also affect data in the residual data set. If records from the data set or record-type being zapped have been removed to the residual, that data will be erased too.
You need not recompile your file definition to begin using the zapped data set or record-type again; it is empty but not gone.
Here is a sample SPIBILD session demonstrating this procedure using the ZAP RECORD-TYPE command:
Command> spibild -Welcome to SPIBILD -? establish course catalog -Processing file: GQ.DOC.COURSE.CAT -Compute time: 0.112 seconds -Elapsed time: 1.788 seconds -Core usage: 0065/0933 -? zap record-type zin09 -File detached -?
The worst problem that can occur with this procedure is that you erase the wrong data set or record-type. If that occurs, you will want to try restoring the entire file, using the most recent copy that was made. (If you made a backup copy of the file at the start of the procedure, as suggested in the Preliminaries, you're in excellent shape.) You may want to contact your SPIRES consultant, who can help you make arrangements to restore the file using backup files made by your computer center. [See 2.1.4.3.]
If the system crashes at any point during the procedure, simply restart the procedure at your convenience.
After working with a new file for awhile, you may decide that the goal records for one of its subfiles needs a new element or that it doesn't need some elements it already has. Depending on the original file design, such changes can be very simple or they can be difficult or impossible to do without rebuilding the entire subfile. [See 2.9.]
The chapter is organized as follows:
Procedure: Add an element Purpose: Add new elements to an existing record-type Who can use it: file owner Basic steps: 1) Change the file definition 2) RECOMPILE filename 3) Recompile formats, if necessary
Once you have created a record-type, whether it's the goal record-type of a subfile or an index record-type of another, you do not have much flexibility to change it, unless you are willing to rebuild the entire record-type, in which case you have all the flexibility you need. But rebuilding can be tedious and expensive, and is unnecessary in many situations. [The procedures for rebuilding record-types appear later in this manual. If the record-type is an index record-type, you will need to rebuild the index, but if it is a goal record-type, you will need to rebuild the entire subfile. [See 2.7.3, 2.9.]]
Generally speaking, as file owner, you can add new Optional elements to any existing record-type in a file as long as that record-type already has Optional elements at the level where you want to add them. For example, if you want to add Optional elements to a Required structure, you can do so if the structure already contains Optional elements.
However, you cannot add new Fixed or Required elements to an existing record-type (unless they are inside a new Optional structure) without rebuilding the record-type. [Note though that you can include Inproc and Inclose rules on an Optional element that make it seem Required to the user; only internally, in terms of the way SPIRES stores the element, would it actually be treated as Optional.]
When significant changes are made to a very important file, the file manager customarily takes either or both of two preliminary steps:
- 1) Create a test file based on the file definition of the production file. Make the proposed changes to the test file, test them, debug them, and then make the changes to the production file.
- 2) Make a backup copy of the production file just before making the change, so that the file can be restored in case of serious problems discovered when the change is made. [See 2.1.4.3.]
The rules for what elements you can add without rebuilding the record-type are:
Following these guidelines, you can change the file definition directly, or by using File Definer. Use File Definer only if you still have the File Definer input for the current version of the file definition.
To change the definition with File Definer, read 1a; to change it directly, read 1b.
Add the desired new elements to your File Definer input file, following the rules listed above. For example, you might add another element to the Optional section at the record level:
ELEMENT comments, comment, com
If you want the element to be indexed, you can easily add it to an existing index, by including the INDEX attribute, followed by the name of the element already indexed. On the other hand, to create a new index for the new element, you should follow the instructions given in the next chapter. [See 2.7.1.] DO NOT simply add the INDEX attribute to the end of the new ELEMENT line.
Then, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2 below
The full procedure, with more information about how to make changes to files using File Definer, appears in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile. Find the record definition for the record-type to which you want to add the elements.
Following the guidelines listed above, add new ELEM statements where desired, along with other statements associated with element definitions as desired, such as LEN, OCC, INPROC, OUTPROC, TYPE, etc. See the "File Definition" manual for more help.
Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new Orvyl data sets requested.
RECOMPILE [gg.uuu].filename
For example,
-> recompile gq.doc.filemgmt -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.
You should recompile any formats written for the record-type if either of these situations is true:
- you have added Optional elements at the record-level or in a structure, and they come before any virtual elements that were already defined at the record-level or in that structure; [Internally, the virtual elements will have new element numbers when the file definition is recompiled, which will not match their old numbers that are compiled into the format definitions.] or
- you have added a new Optional structure within an already existing structure.
If you neglect to recompile them, the formats will probably not work properly.
Use the PERFORM FORMAT LIST command to find the formats defined for the record-type. That list shows all compiled formats for the record-type, including any that were defined by other users, whom you might want to notify of your changes so that they can recompile their formats.
At this point, you are strongly urged to return to SPIRES, select the subfile and add or update some records, including the new element. Remember to display some old and new records too. If the element works as you wanted, you are done with this procedure.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Very serious problems can arise if you mistakenly add a new element in a place that breaks the rules presented earlier. The problem can manifest itself in several ways:
To recover, you need to follow these steps:
- remove or dequeue those records containing the new element;
- process the file in SPIBILD;
- correct the file definition in the FILEDEF subfile, continuing with the procedure described earlier in this section, starting with step 1.
In general, SPIRES does not allow you to remove elements defined for a record-type. Once a record-type has a given set of elements, it will always have those elements, plus any additional ones you may be able to add. [See 2.6.]
You can, of course, rebuild the record-type completely, which is really the only way to completely get rid of an element right away. [See 2.8.1.] When you dump out the contents of the record-type (to batch in again later), you would delete the occurrences of the element you were eliminating. You might do that either by deleting those occurrences using the text editor, or by using the SET ELEMENTS command to restrict the output elements to those you were intending to add back in.
A less drastic solution is to add the $DISCARD proc to the Inproc rule and $NULL to the Outproc rule of the element you want to eliminate. This has no immediate effect on the stored records already in the subfile, though it does hide the element value when the record is displayed.
However, as new records get added or old records get updated, the element will gradually disappear. (As part of this method, you might want to rename it, e.g., DUMMY.) It will not go away completely: it will always show up as an element (e.g., in SHOW ELEMENTS commands) and some older records that never get updated or removed may retain it, but it will take less and less room in the file as time goes by.
Alternatively, you could choose to get rid of all occurrences of the element immediately by updating all the old records. [See 2.4.] Again, the element will not go away completely; it will still show up in element lists, for example.
Rules for record/structure changes.
These rules apply to situations were data already exists within the database. Record level changes are not much different from structure changes, except that record level changes are tied to the existence of records within the record-type, and structure changes are tied to structure occurrences within records.
If there are NO occurrences of records, you can change anything within the RECORD-NAME portion corresponding to the record-type except the COMBINED (or not COMBINED), or the SLOT (or not SLOT) nature of the record-type.
If there are NO occurrences of a structure within ANY of the records, you can change anything in the structure's definition.
If there are records existing in the record-type, your changes are limited to the same changes allowed for existing structure occurrences. What is allowed, and when, is covered below in the form of Questions and Answers.
Q1: Can Optional elements be defined in none were defined?
A1: Usually NO, but there are two exceptions when you can add the OPTIONAL section with one or more optional elements.
If there is only a KEY in the REQUIRED section, and no other REQUIRED elements, OR there is no REQUIRED section at all, then you can add an OPTIONAL section. This is true even if there are FIXED and/or VIRTUAL sections.
Q2: Optional elements are defined (by the OPTIONAL section), but none of these elements occur in any of the records, can I delete the OPTIONAL section entirely?
A2: Usually NO, but you can if there are no other REQUIRED elements other than a REQUIRED KEY. This is basically the reverse of Q1. Furthermore, the only penalty for having declared OPTIONAL elements which don't occur is the OPTIONAL element bit-mask, which is 2-byte long, and only occurs if there are REQUIRED elements other than a REQUIRED KEY.
Q3: Can I define new FIXED or REQUIRED elements?
A3: NO, once FIXED or REQUIRED elements occur, you can't define new elements in either the FIXED or REQUIRED sections. But if OPTIONAL elements have been declared, or can be declared (see Q1), then you can add OPTIONAL elements.
Q4: Optional elements are defined, and some of them exist within records, can I define new OPTIONAL elements?
A4: YES, you can add new optional elements to the OPTIONAL section after the latest physically occurring element in any of the records. For example, if you have defined:
OPTIONAL;
ELEM = A;
ELEM = B;
ELEM = C;
and only A and/or B occur in the records, and C never occurs, you can add ELEM = D; after ELEM = B; or ELEM = C; (the end). It is always safe to append to the end of the OPTIONAL section.
Q5: Can I redefine non-occurring OPTIONAL elements?
A5: YES, if an optional element does not occur in any records, you can redefine that element anyway you like, including making it a simple element when it was a structure, or making it into a structure when it was a simple element. LENGTH and/or OCC attributes can be changed, INPROC and OUTPROC rules, even the element's name and ALIASES. The only requirement is that you retain an element in this element's position. You can't delete it or move it. So in Q4 above, since C didn't occur in any of the records, we could have redeclared it as element D, rather than add ELEM = D; statement. In a sense, we are replacing the definition of a non-occurring element (ELEM = C) by a new definition.
Q6: In a structure defined with no KEY, can I change an ELEMENT to a KEY, or vice versa?
A6: You can only have one KEY in a structure, and it must be either the first FIXED element, or the first REQUIRED element. You can change the first FIXED ELEMENT to a KEY at any time, and vice versa. The first REQUIRED ELEMENT can be changed into a KEY only if the structure does not occur in any records. Once the structure occurs, its REQUIRED section must not be changed.
Q7: What can I do with VIRTUAL elements?
A7: You may add or remove VIRTUAL elements at any time.
Q8: After making changes, what must I do?
A8: If you updated a RECDEF, you must recompile that RECDEF and any FILEDEFs that specify that RECDEF in DEFINED-BY statements. If you updated a FILEDEF, you must recompile that FILEDEF. Then, you must recompile all FORMATS that refer to this file/record-type.
After a SPIRES subfile has been used for awhile, you, the file owner, may discover that the existing index scheme isn't quite adequate. An index you thought would be used often has turned out to be unnecessary, for example, or you realize that you've been using Global FOR commands frequently to find records based on an unindexed element. These situations are fairly common -- file owners often use data bases differently than predicted once the data is available to them.
The decision to add or remove an index is usually related to cost and convenience. Specifically, does the convenience of a particular index justify the cost of storing and maintaining this redundant information? For frequently used indexes, the answer is yes; but if an index is seldom or never used, the answer may be no.
This chapter presents the procedures used to handle major tasks relating to indexes:
- to change values in an index when its Passproc rule has been changed (e.g., so that the index will be a word index instead of a phrase index);
- to repair a damaged index (e.g., an index whose Passproc was changed previously without the index being rebuilt, so that old records being removed or updated aren't getting "un-indexed" properly);
- to remove an element's values from the index (e.g., you want to remove the TITLE element values from a SUBJECT index that had both TITLE and SUBJECT element values);
- to add an element's value to an index (e.g., you want to add the TITLE element values to a SUBJECT index so that it will have both TITLE and SUBJECT element values). [You may be able to do this without erasing the current index first, a possibility discussed in section 2.7.3.]
You can also use the FASTBILD program when building or rebuilding an index. You can find the primary documentation for that program later in this manual [See 2.12.] but its context in building indexes will be discussed in this chapter.
Procedure: Add an index Purpose: Add new indexes to an existing SPIRES subfile Who can use it: file owner Basic steps: 1) Change the file definition. 2) RECOMPILE filename 3) SPIBILD 4) ESTABLISH subfile-name 5) Establish the Global FOR environment. 6) Issue the BUILD command.
Though six steps are shown here, the process of adding new indexes to a subfile is fairly simple. The design phase (suggested in step 1) generally takes the most time.
You need to choose a one-to-six-character record-name for each new index record-type you are defining (e.g., REC09, ZIN12). Note: This is crucial even if you use File Definer to create and change the file definition, previously leaving the selection of record-type names up to File Definer.
New record-types being added to an existing file must be added to the end of the existing record-types. Since SPIRES sorts them in alphanumeric order by name before compiling them, it is crucial that the new index record-types you define have names that sort after those that already exist. [There is an exception. If the file has any DEFQ-only ("pseudo-") record-types, whose names make them sort to the end of the record-types, the new ones must precede them at the end. See the "File Definition" manual for more information.]
For example, if the file already has the three record-types REC01, ZIN01 and ZIN05, your new index would need a name that sorted after ZIN05, such as ZIN06. You could not add an index record named ZIN02, unless you are willing to rebuild the entire file.
The names of all the current record-types appear in the RECORD-NAME statements of the file definition as stored in the FILEDEF subfile. Another reliable way to find the names of the current record-types is to select the subfile in SPIRES and issue the command PERFORM FILE SUMMARY.
You can follow this step using File Definer or by changing the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file definition.
WARNING: Whichever method you use, be extremely careful to follow the directions, to ensure that the new index record definitions follow the definitions for those that already exist. Failure to do so may result in data integrity problems that are very tedious and unpleasant to correct.
To change the file definition using File Definer, read 1a; to change it directly, read 1b.
Add the INDEX attribute as appropriate to your File Definer input file, following it in each case with the RECNAME attribute, followed by the name you chose for the new record-type. For example, to add a new index for the element defined here:
ELEMENT author/ name/ single
you might change it to:
ELEMENT author/ name/ single/ index/ recname zin20
Then, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2 below
The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile and add to it:
- a record definition for each new index record-type; and
- appropriate linkage sections for each index.
Remember that a file can have no more than 64 record-types, stored in no more than 9 Orvyl data sets (filename.REC1 through filename.REC9). Use the COMBINE statement to combine record-types in data sets. SPIRES will put each record-type that doesn't have a COMBINE statement into its own data set, to the limit of 9.
Also remember to follow the rules for index record-types and index linkages in the same goal record-type, such as:
- The pointer element must be given the same name in all index record-types for a single goal record-type.
- In general, pointer groups for indexes that apply to the same goal record must be structured identically, e.g., must all be TYPE = LCTR, must all have the same LENGTH statement if specified for any, etc.
For more detailed information about coding index record definitions and linkage sections, see the manual "SPIRES File Definition".
Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new Orvyl data sets requested.
RECOMPILE [gg.uuu].filename
For example,
-> recompile gq.jnk.players -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.
At this point, you are strongly urged to return to SPIRES, select the subfile and issue the BROWSE command to look at the new index. It should be empty, and you will receive the message "Null range". If it's not, then you did not add the new record-type to the end of the existing record-types, and woe to you if you continue this procedure! Instead, you should return to the first step and correct the file definition, making sure the new record-type or -types appear at the end of the record definitions in the file definition.
The first two steps are usually done interactively, but the rest of the procedure may be done as a batch job if desired. [You may either use SPIBILD in a batch job or use FASTBILD. FASTBILD is recommended if the index will be extremely large, e.g., more than 10,000 index entries created. [See 2.12.]]
Issue the SPIBILD command online, or in a BATWYL job.
-> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
Issue the ESTABLISH command to process the file's deferred queue and to tell SPIBILD the name of the subfile containing the index or indexes to be built.
-? establish players -Processing file: GQ.JNK.TEAMS -Compute time: 0.144 seconds -Elapsed time: 2.086 seconds -Core usage: 0068/0933 -?
You can build all the indexes for a single goal record-type at one time. If you are adding indexes for several different sets of goal record-types in the same file, you need to return to this step for each goal record-type.
At this point, you may also want to increase the size of the pass stack to increase passing efficiency, using the SET PASS STACK command. See the description for that command in Part 6. [See 6.3.0a.]
With this step, you tell SPIBILD which goal records to use to build the index. Usually, you will simply issue the FOR SUBFILE command:
-? for subfile -?
Generally speaking, with this particular procedure, you want to build the new index from all the goal records. However, if for some reason you want to limit the records that are indexed, you can add a WHERE clause, or issue SET SCAN, SET FILTER and SKIP commands as needed.
Another alternative, if you want to limit the records that are indexed, is to create a stored stack or result of those records in SPIRES and use the FOR STORED command rather than FOR SUBFILE. FOR STORED is more commonly used in SPIBILD with merging or removing records [See 2.4.2.] or [See 2.5.1.] For more information, see the description of FOR STORED in Part 6. [See 6.1.2b.]
Under Global FOR, the BUILD command has the following basic syntax:
BUILD index.name[, index.name...]
where "index.name" is the name of the index record-type, as it appears in the INDEX-NAME statement of the file definition's linkage section. If you used File Definer to change the definition, "index.name" is the name you specified with the RECNAME attribute.
The "index-name" specifications of the BUILD command allow a number in parenthesis which chooses one occurrence of INDEX-NAME when there are many having the same "index-name". For example, assume your file-definition had the following:
INDEX-NAME = APPLE; INDEX-NAME = ORANGE; INDEX-NAME = ORANGE; INDEX-NAME = PEARS;
If you specified: BUILD APPLE, PEARS, ORANGE(2) then the first INDEX-NAME = ORANGE; would not be built.
If you specified: BUILD ORANGE, PEARS then only INDEX-NAME = APPLE; would not be built.
The BUILD command under Global FOR builds indexes for ALL goal records (that meet WHERE-clause and SET SCAN criteria).
Continuing the above example, if you are adding the two indexes ZIN14 and ZIN15 to this PLAYERS subfile,
-? build zin14, zin15 -Passing: 121 records of record-type REC01 -End of BATCH/BUILD -Compute time: 2.698 seconds -Elapsed time: 26.007 seconds -Core usage: 0211/0939 -?
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Another alternative is to determine whether any of the records being indexed have been indexed: return to SPIRES, select the subfile and try searching for various records using the new index. If no records in the subfile were indexed, start the procedure again at step 3. If any of the records were successfully indexed, try to determine which was the last record to succeed and then restart the procedure at step 3, using a WHERE clause and/or SET SCAN commands to resume the indexing from that point.
Very serious problems will arise if you mistakenly add a new index record-type between existing record-types in the file. The problem can manifest itself in several ways:
It is very tedious to recover from this problem, particularly if you don't discover it till after using the BUILD command. That's because if you used the BUILD command, you passed data to one or more indexes already containing data, not to your empty index record-type. SPIRES added your new record-type to the end -- that's where the empty one is -- so if you didn't name the last record-type in the BUILD command, more data was added to at least one that already existed.
To recover, you will have to determine which record-types have been harmed by the BUILD command and zap them. Correct your file definition, placing the record-types in their proper order, and recompile it; then try rebuilding.
It is certainly possible that rebuilding the file is the quickest solution to this problem. It may not be the cheapest way, but it may be the least harrowing. Moreover, that gives you complete flexibility in adding and removing indexes.
Procedure: Remove an index
Purpose: Remove indexes from an existing SPIRES subfile
Who can use it: file owner
Basic steps:
1) Use the ZAP RECORD-TYPE procedure to remove the index
records.
2) Change the file definition.
3) RECOMPILE filename
This procedure is an extension of the ZAP RECORD-TYPE procedure described earlier. [See 2.5.2.] The steps of that procedure will be listed as part of this procedure, but you should see the earlier section for details if you need them.
You cannot actually delete a record-type of a SPIRES file -- it must always be there until such time as the file is completely rebuilt. [See 2.9.] What you can do is remove all the records from it and change the file definition so that no data is passed to it, leaving it an empty record-type (which you can use again later if you want).
Be sure that the record-type is used only as an index, and that it doesn't contain other data that makes it useful for other reasons besides as an index (e.g., for table-lookup purposes).
First, you must remove the index records, using the ZAP RECORD-TYPE procedure. The basic steps of that procedure, described in section 2.5.2, are:
The procedure usually looks like this:
Command> spibild -Welcome to SPIBILD -? establish my notebook -Processing file: GQ.DOC.NOTEBOOK -Compute time: 0.158 seconds -Elapsed time: 3.702 seconds -Core usage: 0069/0933 -? zap record-type zin03 -File detached -?
At this point, it is important to understand that you aren't actually removing the index record-type from the file but are in fact emptying it and telling SPIBILD not to pass data to it anymore.
You can follow this step by changing the file directly, or by using File Definer. By changing the file definition directly, you can actually disconnect the index record-type from the subfile. By using File Definer, you are slightly more limited (though the differences can be considered trivial) and you must do more work.
Use File Definer only if you still have the File Definer input for the current version of the file definition.
To change the definition using File Definer, read 2a; to change it directly, read 2b.
Locate the input line where the index you want to remove is "defined". It probably looks something like this:
ELEMENT phone.number/ string/ index
Following that line, add the line:
ELEMENT phone.number/ string/ index PASSPROC $NOPASS
which tells SPIBILD not to pass the element's data to that index.
If other elements are passed to the same index, add the same line to their descriptions too.
WARNING: Do not simply remove the INDEX attribute from the input line. This will definitely cause serious problems, unless the moon is in your seventh house.
The $NOPASS Passproc tells SPIBILD not to pass data to the index, but the linkage to the subfile is still there; e.g., the index still shows up under SHOW INDEXES, even though it is empty. Thus, you will probably want to hide the index using the PRIV-TAGS and NOSEARCH attributes in File Definer. See the "File Definer" manual for details.
Next, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 3 below
The full procedure, with more information about how to make changes to files using File Definer, appears in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the index you want to remove, and delete those lines from the file definition. They usually look like this:
INDEX-NAME = ZIN12;
SEARCHTERMS = PHONE.NUMBER, PHONE;
SEARCHPROC = $Search.Trunc;
PASSPROC = $Pass.Elem('PHONE.NUMBER',phrase);
PTR-GROUP = POINTER;
In general, remove the group of lines that starts with the appropriate INDEX-NAME and ends with the PTR-GROUP statement.
WARNING: Do not remove the record definition for the record-type from the file definition. The file will not recompile without it.
Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.
RECOMPILE [gg.uuu].filename
For example,
-> recompile gq.doc.notebook -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary.
As a final step, you might want to verify that the index is gone by returning to SPIRES, selecting the subfile, and issuing the SHOW INDEXES command.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Procedure: Rebuild an index
Purpose: Rebuild an index that already exists
Who can execute it: file owner
Basic steps:
1) Use the ZAP RECORD-TYPE procedure to remove the index
records.
2) Change and recompile the file definition, if needed.
3) SPIBILD
4) ESTABLISH subfile-name
5) Establish the Global FOR environment.
6) Issue the BUILD command.
This procedure is useful in the following circumstances:
- to change the values in an index when the index's Passproc rule has been changed (e.g., so that the index will be a word index instead of a phrase index);
- to repair a damaged index (e.g., an index whose Passproc was changed previously without the index being rebuilt, so that old records being removed or updated aren't getting "un-indexed" properly);
- to remove an element's values from the index (e.g., you want to remove the TITLE element values from a SUBJECT index that had both TITLE and SUBJECT element values);
- to add an element's value to an index (e.g., you want to add the TITLE element values to a SUBJECT index so that it will have both TITLE and SUBJECT element values). [You may be able to do this without erasing the current index first. Here's a summary of the procedure you can try: 1) Change the file definition so that the new element (TITLE in the example above) is passed to that index, but stop any other elements from being passed to it (SUBJECT). 2) Use the BUILD command to pass the TITLE data to the index for the records currently in the subfile. 3) Change the file definition again so that all the elements are passed to the index for future records. WARNING: This procedure can't be done using File Definer to change the file definition; you must change the file definition directly.]
If you need to rebuild several indexes, you can follow this procedure too, zapping each index separately in step 1, changing the file definition all at once in step 2, and then building the indexes simultaneously in step 6 with the BUILD command.
If you are adding a new element to an index, make sure that the types of data you are placing into the single index are compatible. In general, for instance, you wouldn't index an element stored as an integer with one stored as a string. (You could do it if you convert the integer value to a string value in the Passproc rule string.)
For several steps of this procedure, you need to know the name of the index record-type you are rebuilding, which appears in the file definition. Select the FILEDEF subfile and transfer or display the definition for your file.
To find the name, find the SEARCHTERMS statement that lists the index's search terms; just ahead of that statement will be the INDEX-NAME statement, containing the index record-type's name. For example, if you are trying to find the name of the record-type containing the TITLE index, you might find these lines in the file definition:
INDEX-NAME = ZIN02; SEARCHTERMS = TITLE, TI, T; (etc.)
The name of the record-type for the TITLE index is ZIN02.
As another preliminary step, you might consider making a backup copy of the file in case problems arise during the procedure. [See 2.1.4.3.] Problems where a backup copy is useful arise primarily because of carelessness during step 2, when the file definition is being changed. Hence, you might want to create a test version of your file, making the changes there and testing them before committing the changes to your production file.
The basic steps of that procedure, described in section 2.5.2, are:
The procedure usually looks like this:
Command> spibild -Welcome to SPIBILD -? establish my notebook -Processing file: GQ.DOC.NOTEBOOK -Compute time: 0.195 seconds -Elapsed time: 2.117 seconds -Core usage: 0069/0933 -? zap record-type zin02 -File detached -?
If you are rebuilding a damaged index, you probably don't need to make any changes to the file definition at all, and can continue to step 3. However, if you are changing how or what values are passed to the index, you will need to make changes to the file definition -- most likely to the index's linkage section and possibly to the index record-type's record definition.
You can follow this step by changing the file directly, or by using File Definer. Use File Definer only if you still have the File Definer input for the current version of the file definition.
To change the definition using File Definer, read 2a; to change it directly, read 2b.
If you are changing the type of value indexed, rather than the elements being indexed, follow these instructions: Locate the input line where the index you want to change is "defined". It probably looks something like this:
ELEMENT subject/ string/ index
Make the appropriate change to that line. For example, to convert it to a word index:
ELEMENT subject/ string/ index/ word
which tells SPIBILD to split the element's value into individual words for indexing.
If you are adding an element to those for this index, follow these instructions: Locate the input line for the specific element, e.g.:
ELEMENT subject/ string
Add the INDEX attribute, followed by the name of another element whose values are indexed in the relevant index:
ELEMENT subject/ string/ index title
If you are removing an element from those passed to a given index, follow these instructions: Locate the input line for the specific element, e.g.:
ELEMENT subject/ string/ index title
If the INDEX attribute names another element, simply remove the INDEX attribute:
ELEMENT subject/ string
However, if the INDEX attribute does not name another element, DO NOT simply remove the INDEX attribute from the input line. This will invariably cause serious problems, unless you lead a charmed life. You need to find the name of the index record-type created for this element (it's in the real file definition, in the INDEX-NAME statement), and find another element whose values are also being passed to the same index. For example, if you are removing SUBJECT from an index that also contains TITLE, and the index record-type is REC05, you could change the input lines as follows:
(old) ELEMENT subject/ string/ index ELEMENT title/ string/ index subject (new) ELEMENT subject/ string ELEMENT title/ string/ index/ recname rec05
For further help along these lines, see the SPIRES manual "File Definer" or contact your SPIRES consultant.
After making changes to the File Definer input, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2c below
The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the index you want to change, and make the appropriate changes. Then select the FILEDEF subfile in SPIRES and replace the old version with the new. Next, recompile the file definition, using step 2c.
Simply stated, the procedure to follow is:
RECOMPILE [gg.uuu].filename
If the file definition does not compile properly, SPIRES will tell you so; return to step 2a or 2b as appropriate.
If the file compilation is successful, press on to step 3.
Issue the SPIBILD command online, or in a BATWYL job:
Command> spibild -Welcome to SPIBILD -?
In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.
Issue the ESTABLISH command to process the file's deferred queue and to tell SPIBILD the name of the subfile containing the index or indexes to be built.
-? establish my notebook -Processing file: GQ.DOC.NOTEBOOK -Compute time: 0.288 seconds -Elapsed time: 4.172 seconds -Core usage: 0068/0933 -?
You can build all the indexes for a single goal record-type at one time. If you are adding indexes for several different sets of goal record-types in the same file, you need to return to this step for each goal record-type.
At this point, you may also want to increase the size of the pass stack to increase passing efficiency, using the SET PASS STACK command. See the description for that command in Part 6. [See 6.3.0a.]
With this step, you tell SPIBILD which goal records to use to build the index. Usually, you will simply issue the FOR SUBFILE command:
-? for subfile -?
Generally speaking, with this particular procedure, you want to build the new index from all the goal records. However, if for some reason you want to limit the records that are indexed, you can add a WHERE clause, or issue SET SCAN, SET FILTER and SKIP commands as needed.
Another alternative, if you want to limit the records that are indexed, is to create a stored stack or result of those records in SPIRES and use the FOR STORED command rather than FOR SUBFILE. FOR STORED is more commonly used in SPIBILD with merging or removing records [See 2.4.2.] or [See 2.5.1.] For more information, see the description of FOR STORED in Part 6. [See 6.1.2b.]
Under Global FOR, the BUILD command has the following basic syntax:
BUILD index.name[, index.name...]
where "index.name" is the name of the index record-type, as it appears in the INDEX-NAME statement of the file definition's linkage section.
The "index-name" specifications of the BUILD command allow a number in parenthesis which chooses one occurrence of INDEX-NAME when there are many having the same "index-name". For example, assume your file-definition had the following:
INDEX-NAME = APPLE; INDEX-NAME = ORANGE; INDEX-NAME = ORANGE; INDEX-NAME = PEARS;
If you specified: BUILD APPLE, PEARS, ORANGE(2) then the first INDEX-NAME = ORANGE; would not be built.
If you specified: BUILD ORANGE, PEARS then only INDEX-NAME = APPLE; would not be built.
The BUILD command under Global FOR builds indexes for ALL goal records (that meet WHERE-clause and SET SCAN criteria).
Continuing the above example, if you are rebuilding the index ZIN02 in the MY NOTEBOOK subfile:
-? build zin02 -Passing: 2421 records of record-type REC01 -End of BATCH/BUILD -Compute time: 19.698 seconds -Elapsed time: 122.944 seconds -Core usage: 0211/0939 -?
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
If any of the records were successfully indexed, try to determine which was the last record to succeed (return to SPIRES and use the index to try to find various records); then restart the procedure at step 3, using a WHERE clause and/or SET SCAN commands to resume the indexing from that point in step 5.
Very serious problems can arise if the changes you make to the file definition are not done carefully. Particularly when File Definer is involved, it is easy to create a new file definition that is not compatible with the old. These problems manifest themselves in various ways:
It can be difficult to find the cause of such problems; to recover from one can be even harder, particularly if you don't discover it till after using the BUILD command. You should probably see your SPIRES consultant as soon as possible. But before you do, you should get a copy of the previous file definition out of the FILEDEF subfile (you could use the TREE or TRANSACTIONS class in Global FOR, for example) in case you need to go back to it. Your consultant will probably need it too, along with a copy of the new definition.
If you erase the wrong record-type in step 1, you will want to try restoring the entire file, using the most recent copy made. Usually this means contacting your SPIRES consultant, who can help make arrangements to restore the file using backup copies made recently by your computer center. If you have your own backup copies for your application, you may want to use those instead. [See 2.1.4.3.]
The previous chapter covered how to add, remove or rebuild an index of a subfile. In this chapter, we go a step farther -- we will add, remove and rebuild subfiles of existing files.
To be more specific, this chapter refers to files already in existence that have (or will have) more than one subfile and more than one set of goal records. Here we will add a new subfile to an existing file, or remove one subfile from a file, leaving other subfiles of the file alone, or rebuild one subfile of a file, leaving the rest of the file alone. [Adding a new subfile to a file that doesn't exist is the same as creating a new file, which is covered in the manual "SPIRES File Definition." Removing the only subfile from a file is the same as destroying the entire file, and rebuilding the only subfile of a file is the same as rebuilding the entire file, two procedures discussed later in this manual. [See 2.9, 2.11.]]
The three sections of this chapter are:
Procedure: Add a new subfile
Purpose: Add a new subfile (goal and possibly index
record-types) to an existing file
Who can use it: file owner
Basic steps:
1) Change the file definition.
2) RECOMPILE filename
Adding a new subfile to an existing file is a fairly simple procedure on paper -- what can complicate it is the design of the subfile.
This section describes the procedure when you are adding a new subfile whose goal record-type will be a new record-type in the file. [If the new subfile will use an existing record-type as the goal record-type, your task is very easy: you change the file definition by adding a new Subfile section describing the subfile and then place the new version in the FILEDEF subfile. You can then select the subfile immediately -- you don't need to recompile the file definition. [See 2.6.1, 2.7.1 if you then want to add elements or indexes to it.]]
You need to choose a record-name of no more than 6 characters for each new record-type you are defining (e.g., REC13). This is crucial to you even if you use File Definer to create and change the file definition.
New record-types being added to an existing file must be added to the end of the existing record-types. Since SPIRES sorts them in alphanumeric order by name before compiling them, it is crucial that the new record-types you define follow those that already exist. [The exception to this rule involves DEFQ-only record-types. If the file contains any of these, whose names make them sort to the end of the record-types, the new ones must precede them. See the "File Definition" manual for more information.]
For example, if the file already has the three record-types REC01, ZIN01 and ZIN05, the new goal record-type would need a name that sorted after ZIN05, such as ZIN06 or ZJA01. You could not add a goal record-type named REC02 unless you were rebuilding the entire file. [See 2.9.]
The names of all current record-types appear in the RECORD-NAME statements of the file definition as stored in the FILEDEF subfile. Another way to find their names is to select a subfile of the file in SPIRES and issue the command PERFORM FILEDEF SUMMARY.
Keep in mind that a file can have only 64 record-types, which must be combined into no more than 9 ORVYL data sets (i.e., "filename.REC1" through "filename.REC9"). In addition, no more than 8 of them can be slot record-types, each of which must be in its own data set, alone. Hence, you could not add a slot record-type if you are already using all 9 of the allowed ORVYL data sets.
To execute this step, you'll need to know specifically what changes you want to make to the file. For help, you may want to consult other SPIRES manuals on file definition, such as "File Definition" or "File Definer". As always when you are making major changes to an important file, you should consider making a test version of the file and making your changes to that one first, to be sure that they work before applying them to the real file.
You can follow this step using File Definer or by changing the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file definition.
WARNING: Whichever method you use, be extremely careful to follow the directions, to ensure that the new index record-type definitions follow the definitions for those that already exist. Failure to do so may result in data integrity problems that are very tedious and unpleasant to correct.
To change the definition using File Definer, read 1a; to change it directly, read 1b.
Add the information for your new subfile at the end of your input file. Be sure to include the GOAL keyword with the name you chose for the goal record-type, based on the rules given under "Preliminaries" above.
Then, use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2 below
The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.
In SPIRES, get the file definition from the FILEDEF subfile and add to it:
- a record definition for each new goal and index record-type; and
- appropriate linkage sections for each goal record-type.
- a Subfile section for each new subfile.
For more detailed information about coding file definitions, see the manual "SPIRES File Definition".
Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new ORVYL data sets requested.
RECOMPILE [gg.uuu].filename
For example,
-> recompile patron.information -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.
SPIRES may not recompile the file definition if there are records in the deferred queue. It may tell you to process the file in SPIBILD before trying to compile the definition again. If so, follow the PROCESS/ESTABLISH procedure, and then return to SPIRES, reselect the FILEDEF subfile, and continue with step 2.
When SPIRES successfully recompiles the file definition, you are strongly urged to select the subfile and make sure the new subfile has no records in it. If it isn't empty, then you didn't add the new record-type to the end of the existing record-types, which means your file is on the brink of disaster. You should return to the first step and correct the file definition, making sure the new record-type or -types appear at the end of the record definitions in the file definition. (See "Handling Problems" below.)
For similar reasons, it is a good idea to check the other subfiles in the file too: examine one or two records in them to verify that they are in the proper subfile.
If the new subfile is indeed empty, try adding a couple of records to it. If it works as you wanted, congratulations! If not, you need to make more changes to the file definition, starting again at step 1.
Also, if you defined any indexes for the subfile, try processing the file in SPIBILD and then checking the indexes to make sure the data was passed from your new goal record-type as you expected. [See 2.2.]
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Very serious problems will arise if you mistakenly add a new record-type between existing record-types in the file. The problem can manifest itself in several ways:
It can be very tedious to recover from this problem, particularly if you don't discover it right after you compile the file. If you do discover it right away, you should correct the file definition (placing the old record-types in their original order and moving the new ones to the end), returning to step 1.
Of course, contact your SPIRES consultant if you need help solving any problems that arise. If you're reluctant to ask for help, keep in mind that making mistakes as you "solve" the problem can be more disastrous than the original problem was. Your consultant would prefer being called in sooner rather than later.
Procedure: Remove a subfile
Purpose: Remove a subfile from an existing SPIRES file
Who can use it: file owner
Basic steps:
1) Use the ZAP RECORD-TYPE procedure to remove the goal
and index records.
2) Change the file definition.
3) RECOMPILE filename
This section describes the procedure when you are removing a subfile of a file by removing all the records from its goal record-type and indexes. [You can also eliminate a subfile by simply removing its Subfile section from the file definition -- the records will remain in the file, but users will no longer be able to select the subfile.]
It's assumed here that the file contains other record-types you want to save. If you are getting rid of all the record-types of the file, you should follow the procedure for destroying an entire file, which is much simpler and more efficient. [See 2.11.]
Unless you zap and rebuild the entire file, you can't actually delete a record-type of a SPIRES file. What you can do is remove all the records from it and change the file definition so that no data is placed in it or passed to it, leaving it an empty record-type (which you can use again later if you want).
WARNING: The instructions for this procedure presume you can and are willing to change the file definition directly, without using File Definer. Though it isn't inconceivable that you could use File Definer to make the appropriate changes, that method could be extremely complicated, and complete instructions would take several pages. If you absolutely must use File Definer because you want to continue using it for any future changes to the file, you should either rebuild the file completely (removing the unneeded subfile definitions from the File Definer input) or see your SPIRES consultant for help. Do not use File Definer on your own unless you are very certain how to do so without taking away or rearranging any record-types.
Depending on the way the data was distributed in the ORVYL data sets, you may or may not be able to release any ORVYL blocks containing the subfile's data. In other words, it is possible that eliminating a subfile will not make the file use any less storage space, though that space will be used again if more records are added to other record-types in the file. (See step 1 below.)
This procedure is an extension of the ZAP RECORD-TYPE procedure described in detail earlier in this manual. [See 2.5.2.] The steps of that procedure will be listed as part of this procedure, but you should see the earlier section for details if you need them.
The basic steps of that procedure, described in section 2.5.2, are:
In this command, name one of the record-types of the subfile you want to destroy. Use the ZAP DATA SET command if all the record-types in a given data set are record-types you want to eliminate. [See 2.5.2.]
You need to repeat this procedure for each goal and index record-type of the subfile, restarting each time with step ii. Of course, do not zap any record-types of the subfile that are used by other subfiles you are not destroying.
File blocks no longer needed will be released by the ZAP DATA SET command, and also by the ZAP RECORD-TYPE command if no other record-types are in the same data set.
The procedure usually looks like this:
-> spibild -Welcome to SPIBILD -? establish my notebook xrefs -Processing file: GQ.DOC.NOTEBOOK -Compute time: 0.158 seconds -Elapsed time: 3.702 seconds -Core usage: 0069/0933 -? zap record-type rec05 -File detached -? establish my notebook xrefs -? zap record-type zin06 -File detached (etc.)
At this point, be sure you understand that you aren't actually removing the subfile's record-types from the file but are emptying them and telling SPIRES and SPIBILD not to put data into them anymore.
You should follow this step by changing the file directly. (See the earlier warning against using File Definer.)
In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the goal record-type of the subfile you want to remove, and delete those lines from the file definition. It will begin with a GOALREC-NAME statement:
GOALREC-NAME = recname;
where "recname" is the 1-6 character name of the record-type. Delete all the lines from there up to (but not including) the next GOALREC-NAME or EXTDEF-ID or SUBFILE-NAME statement, whichever comes next.
Next, find the Subfile section(s) of the file definition that names the subfile(s) you want to remove. They begin with the SUBFILE-NAME statement:
SUBFILE-NAME = subfile name;
Delete all the lines from there up to (but not including) the next SUBFILE-NAME, FILE-ACCESS, ELEMINFO, CHECKPOINT, INDEXINFO, EXT-REC or EXT-DEF statement.
Though it isn't necessary, you may want to remove most (but not all) of the contents of the definitions of the various record-types you have emptied. If you do this, do not delete the COMBINE statement, and be sure to leave at least a key element defined -- otherwise, the file definition can't be recompiled.
WARNING: Don't remove any record definitions from the file definition completely. You can't recompile the file definition without them.
Once you've made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.
RECOMPILE [gg.uuu].filename
For example,
-> recompile notebook -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary.
As a final step, you might want to verify that the subfile is gone by trying to select it. It is a good idea to select the remaining subfiles of the file and examine them too.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Procedure: Rebuild a subfile
Purpose: Rebuild an existing subfile of a SPIRES file
Who can use it: file owner
Basic steps:
1) Dump and save the current goal-record data.
2) Use the ZAP RECORD-TYPE procedure to remove the goal
and index records.
3) Change the file definition.
4) RECOMPILE filename
5) Reload the saved goal-record data.
If you decide that one of your subfiles needs major changes, you may need to rebuild it -- that is, remove all the records from its goal record-type and indexes, make appropriate changes to the file definition, put the goal records back in, and rebuild the indexes.
Typical reasons for rebuilding a subfile are these:
- You want to make changes to the goal record-type, adding some new elements, removing some old ones you don't need to keep, or changing the types of values stored for others. These changes also affect one or more of the indexes.
- The goal records have slot keys, and you want to renumber them all, eliminating gaps created by removed records.
- "Available space" information indicates that much of the storage space the file is using is empty. By zapping data sets of the file and rebuilding them, unneeded storage space will be released. (But see below.)
It's assumed here that the file contains other record-types you want to leave untouched. If you are rebuilding the only subfile of the file, you should follow the procedure for rebuilding an entire file, which is much simpler and more efficient. [See 2.9.]
Unless you zap and rebuild the entire file, you cannot actually delete any index record-types of a SPIRES subfile. What you can do is remove all the records from it and change the file definition so that no data is passed to it, leaving it an empty record-type (which you can use again later if you want).
Depending on the way the data was distributed in the ORVYL data sets, rebuilding a subfile may or may not release any ORVYL blocks containing the subfile's data. Blocks are released when an entire data set is zapped or when all the record-types in a data set are zapped. So, if a subfile's data is in ORVYL data sets shared by other record-types of the file, rebuilding the subfile will not make the file use any less storage space, though that space will be used again when the records are added back in.
Depending on the amount and intricacies of changes you want to make to your file definition, you may want to begin with step 3, creating a revised file definition, which you then test thoroughly before "starting" the procedure described below.
CAUTION: In this procedure, you will be zapping data. Anytime you are zapping data, you should compare the importance of the data to your confidence that you can complete the procedure. If the data's value outweighs your confidence, consider making a backup copy of the current file and file definition so that you can quickly recover the file to its current state in case of any problems. [See 2.1.4.3.]
You should create a data set of the goal-record data you want to put back into the subfile. There are several techniques to use in SPIRES:
- If you are not changing the goal record-type's definition, you can use the GENERATE LOAD command. It will create a data set of records in their internal form, which can be loaded into the empty goal record-type later. It is the most efficient tool to use, if you can use it. [See 2.3.3.]
- If you are changing the goal record-type's definition, then you want to create a data set that includes only the elements you want to add back in. Select the old subfile, use the SET ELEMENTS command to choose those elements, and using Global FOR commands (such as FOR SUBFILE and IN ACTIVE DISPLAY ALL), create the data set and save it. If the subfile's records have slot keys, you should omit it from the SET ELEMENTS list if you want the records to have new keys. (See example below.)
- If desired, you might want to create a format to use, rather than the SET ELEMENTS list, for both output and later input. If you do have a format, the method is the same as the one just described.
For the examples in this section, suppose we are rebuilding a subfile called ALARM LOCATIONS in the file GQ.DOC.ALARMS. We want to save all the elements except for the slot key, called RECNUM. There is no "SET ELEMENTS ALL -- elem" command to let us directly set all but a few elements, but an indirect way does exist. In brief, you can use a "SET ELEMENTS ALL + element" and a "SET ELEMENTS -- element" command; by using the two of them in combination with a virtual element (or a dynamic element created on the spot -- either way giving us an additional element to set in addition to "all" of them), you can achieve the desired effect with the SET ELEMENTS command.
Two other items worth noticing are that the $RECSEP variable is set so that each record begins with an "add;" line, and that an index is used to limit the records saved to only those added since 1984.
-> select alarm locations -> find date.added after 1983 -Result: 201 LOCATIONS -> clear format -> set recsep 'add;' -> for result +> define element null for recnum as '' +> set elements all + null +> set elements - null recnum +> in active clear display all +> save alarm.location.records ALARM.LOCATION.RECORDS saved on INT001 +>
If the amount of data may be too large to fit in your active file, consider using the batch SPIRES program to write the data directly to a WYLBUR (OS) data set. See the SPIRES manual "Searching and Updating", or online, [EXPLAIN JCL, FOR SPIRES.] for more information about batch SPIRES.
The basic steps of that procedure, described in section 2.5.2, are:
You need to repeat this procedure for each goal and index record-type of the subfile, restarting each time with step ii. Of course, do not zap any record-types of the subfile that are used by other subfiles you are not rebuilding.
The procedure usually looks like this:
Command> spibild -Welcome to SPIBILD -? establish alarm locations -Processing file: GQ.DOC.ALARMS -Compute time: 0.262 seconds -Elapsed time: 1.551 seconds -Core usage: 0069/0933 -? zap record-type rec01 -File detached -? establish alarm locations -? zap record-type zin08 -File detached (etc.)
If you have no plans to change the file definition, continue with step 6.
For major applications, the file definition will usually be revised and tested as part of the Preliminaries. If so, in this step you should replace the old file definition with your newly revised and tested one.
You can always make the changes directly to the file definition. In addition, you may be able to use File Definer to change the file definition if all of the following conditions are true:
- You have the File Definer input for the current version of the file definition.
- You are not deleting any indexes from the definition.
- You are not rearranging any elements that are indexed.
- You are not adding any new indexes to the definition. [If you are adding new indexes, you can use File Definer by carefully following the directions described in section 2.7.1, under "Preliminaries" and step 1.]
If any of the above conditions are false, you should change the file definition directly, as described below in 3b; if you can use File Definer, read 3a.
Make the desired changes to your input file, making certain that your changes are within the guidelines described above. Then use File Definer to generate a revised file definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 4 below
In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Make the desired changes to it, keeping the following rules in mind:
- Don't remove any record definitions from the file definition completely. You must have at least as many as you did before, and the ones that remain in the file from the past must be in the same numerical position after the file definition is recompiled.
- If you add new record-types, be sure to add them at the end of the list of those already in existence. Remember that SPIRES will sort the records alphabetically by name. A list of all the record-types, in sequential order, is available through the PERFORM FILE SUMMARY command.
Once you've made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.
The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.
RECOMPILE [gg.uuu].filename
For example,
-> recompile gq.doc.alarms -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 3 to correct them, if necessary.
You are now ready to reload the data you saved in step 1 into the new subfile. Depending on the form in which you saved it, use either the INPUT BATCH, INPUT ADD, LOAD, FAST BATCH or FASTBILD procedures. [See 2.3.]
As a final step, you might want to verify that the subfile is working as you'd expect by returning to SPIRES and trying to use it. It is a good idea to select the remaining subfiles of the file and examine them too.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Procedure: Rebuild a file Purpose: To rebuild an existing SPIRES file Who can use this procedure: file owner Basic steps: 1) Dump and save the current goal-record data. 2) Change the file definition. 3) ZAP FILE filename 4) SELECT FILEDEF 5) COMPILE filename 6) Reload the saved goal-record data.
If you decide to make major changes to a file, you may need to rebuild it -- that is, make copies of all the goal records in its various subfiles, zap the file, compile a new file, and load the goal records back in. In most ways, this procedure is like the one for rebuilding a subfile, but on a larger scale. [See 2.8.3.] Yet in some ways, it's easier -- you'll find it's easier to zap the entire file rather than individual record-types, and you have complete freedom in creating a new file definition, including the use of File Definer.
People rebuild SPIRES files for many different reasons, such as:
- to rebuild a subfile, when the subfile is the only one in the file;
- to completely eliminate unneeded record-types;
- to make major changes to multiple goal record-types, such as adding new elements, removing old ones, etc., anywhere you like;
- to make major changes to the indexing structure (adding new ones, removing old ones) without having to worry about the order in which the old ones were defined;
- to make major changes in several different areas at once, which could perhaps be achieved by using several different procedures but which can be done more quickly and easily with this one single procedure.
As the list above suggests, this procedure is the catch-all technique for repairing a file. You could compare it to repairing a car that won't run by buying another car. The solution is overkill if the problem simply involves the sparkplugs.
But if the engine has many different problems or much more serious ones, buying a new car may solve them more easily and even more inexpensively (in the long run) than solving them each individually. And consider the advantages of "trading up to a newer model" as you decide how to solve the problems. Moreover, in regard to the sparkplug problem, buying a new car solves the problem (albeit expensively) in a way that keeps your hands clean if you don't want to get them dirty. [We hope this is the closest that SPIRES documentation ever comes to sounding like a car dealer.]
Depending on the amount and intricacies of changes you want to make to your file definition, you may want to begin with step 2, creating a revised file definition, which you then test thoroughly before "starting" the procedure described below.
CAUTION: In this procedure, you will be zapping data. Anytime you are zapping data, you should compare the importance of the data to your confidence that you can complete the procedure. If the data's value outweighs your confidence, consider making a backup copy of the current file and file definition so that you can quickly recover the file to its current state in case of any problems. [See 2.1.4.3.]
Create a data set of the current data in each goal record-type you want to put back into the file. There are several techniques to use in SPIRES:
- If you aren't changing that goal record-type's definition, you can use the GENERATE LOAD command. It will create a data set of records in their internal form, which can be loaded into the empty goal record-type later. It is the most efficient tool to use, if you can use it. [See 2.3.3.]
- If you are changing the goal record-type's definition, then you want to create a data set that only includes the elements you want to add back in. Select the old subfile, use the SET ELEMENTS command to choose those elements, and using Global FOR commands (such as FOR SUBFILE and IN ACTIVE DISPLAY ALL), create the data set and save it. If the subfile's records have slot keys, you should omit it from the SET ELEMENTS list if you want the records to have new keys. (See example below.)
- If desired, you might want to create a format to use, rather than the SET ELEMENTS list, for both output and later input. Except for setting the format, the method is the same as the one just described.
You may choose different methods for different goal record-types of the file.
If you aren't making any changes to the file definition, you can skip ahead to step 3.
Because you have complete flexibility in changing your existing file definition or creating a new one, you can make any changes you want, within the limits of general file definition capabilities. (Remember, after step 4, you will no longer have a file with which the revised definition must be compatible, unless you saved the data in its internal SPIRES format using GENERATE LOAD.) That means you can use either File Definer or the file definition directly to make your changes.
Keep in mind the data that you kept in step 1, however. Be sure the new file will accept that data, e.g., that it will have the same element names, that Inproc rules will accept the same element values, etc.
When you are finished with the file definition, put it in the FILEDEF subfile, using the ADDUPDATE command.
The next step is to destroy the old file, using the ZAP FILE command:
ZAP FILE filename
where "filename" is the name of the file to be erased.
For instance, if you want to zap your file FO.RTE.PIANOS:
-> zap file pianos -Do you want to destroy file FO.RTE.PIANOS ? yes ->
You next need to reselect the FILEDEF subfile.
The COMPILE command compiles the named file definition, verifying the logic of the file definition and creating the various ORVYL data sets of the file.
COMPILE [gg.uuu].filename
For example,
-> select filedef -> compile .pianos -File Definition compiled ->
If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary, and skip step 3 when you try again.
You are now ready to reload the data you saved in step 1 into the various subfiles of the file. Depending on the forms in which you saved it, use either the INPUT BATCH, INPUT ADD, LOAD, FAST BATCH or FASTBILD procedures. [See 2.3.1.]
As a final step, you should verify that the subfiles work as you'd expect by returning to SPIRES and trying to use them.
System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.
Procedure: Move a file
Purpose: To move a SPIRES file from one account to another
Who can use this procedure: file owner
users with Copy access
Basic Steps:
1) Make sure the copying account has enough ORVYL blocks
2) Give the copying account Copy access to the file
3) Make sure the copying account has ORVYL access to the file.
4) Use the COPY FILE command to copy the file
5) Create a file definition for the new file.
6) Cleanup: Other considerations
Sometimes it is necessary to move an entire SPIRES file from one account to another. For example, you might want to move a file from a "test account" to a "production account", or from an account you want to close to a new account.
To move a file from one account to another (say, file GQ.DOC.RESTAURANT from account GQ.DOC to account GQ.JNK), follow this procedure:
First, find out how large the file is. If you are the file owner, issue the ORVYL command SHOW FILES: [The copying account can issue the SHOW FILES command if that account has general access (READ or higher) to all ORVYL data sets on the file owner's account, as determined by the ORVYL command SET PERMIT.]
SHOW FILES LIKE filename BLOCKS
For example, from account GQ.DOC:
-> show files like orv.gq.doc.restaurant blocks ORVYL File System ORV.GQ.DOC.RESTAURANT.CKPT -- 43 BLKS (F: 0, L: 42) ORV.GQ.DOC.RESTAURANT.DEFQ -- 50 BLKS (F: 0, L: 49) ORV.GQ.DOC.RESTAURANT.MSTR -- 5 BLKS (F: 0, L: 4) ORV.GQ.DOC.RESTAURANT.REC1 -- 4 BLKS (F: 0, L: 3) ORV.GQ.DOC.RESTAURANT.REC2 -- 35 BLKS (F: 0, L: 34) ORV.GQ.DOC.RESTAURANT.RES -- 108 BLKS (F: 0, L: 107) ->
Add the number of blocks (BLKS) shown to determine the size of the file.
Next, on the copying account, see how many blocks are available, using the ORVYL command SHOW BLOCKS:
-> show blocks 742 Blocks accounted, 2000 Blocks allowed ->
As long as the difference between the number of blocks allowed and the number currently being used ("accounted") is larger than the size of the file being copied, you can copy the file to that account.
In the file definition, the file owner must give the copying account Copy access to the file. If you have the File Definer input for the current version of the file definition, you can use File Definer to change the file definition (2a below); otherwise, change the file definition directly (2b).
To use File Definer, first add the COPY attribute to the FILE statement, listing the accounts to be given Copy access:
FILE/ COPY GQ.JNK
Then use File Definer itself to generate the revised definition:
SPIRES ENTER FILE DEFINER INPUT ACTIVE save your revised input file for later GENERATE SELECT FILEDEF ADDUPDATE, or UPDATE key go on to step 2c below
To add the information directly to the file definition, get the most recent copy and add to the end an occurrence of the FILE-PERMITS structure that includes a FILE-ACCESS value of COPY and an ACCOUNTS value that includes the copying account:
FILE-ACCESS = COPY; ACCOUNTS = GQ.JNK;
Then select the FILEDEF subfile in SPIRES and replace the old version of the file definition with this new one.
The procedure to follow is:
RECOMPILE [gg.uuu].filename
If the file definition doesn't compile properly, SPIRES will give you error messages; return to step 2a or 2b as appropriate.
This step can be skipped as long as the file owner has not set different ORVYL permits for the file than the defaults set by SPIRES when the file is established. If desired, use the ORVYL command SHOW PERMIT to verify that each of the file's data sets allows READ access to the copying account. If any do not, use the SET CLP and SET PERMIT commands in combination to change them.
The COPY FILE command, available in SPIRES, copies the ORVYL data sets of a SPIRES file to the copying account:
COPY FILE ORV.gg.uuu.filename
It must be issued from the copying account, NOT from the file owner's account. As it begins copying each data set, SPIRES displays a message naming the data set and the number of blocks to be copied:
-> spires(quiet) -> show eval $account GQ.JNK -> copy file orv.gq.doc.restaurant Start copy of 50 blocks from data set DEFQ Start copy of 5 blocks from data set MSTR Start copy of 4 blocks from data set REC1 Start copy of 35 blocks from data set REC2 Start copy of 108 blocks from data set RES ->
If you have any problems with the COPY FILE command, see Part 6, or online, EXPLAIN COPY FILE COMMAND.
To define subfiles for your copied file, you must have a file definition. In general, it is best (easiest) to get the original file owner's file definition or file definer input and modify it accordingly, changing the account number in the file name, changing the subfile names, etc. (If, for example, you had an older copy of the file definition and the file owner had changed the file definition in the meantime, recompiling your copy with the old definition could cause problems serious enough to force you to begin the procedure again.) Be sure to add your new copy of the file definition to the FILEDEF subfile so that you can issue the SELECT command to use its subfiles.
Depending on the circumstances, you may need to continue with the following procedures:
- If there are any formats associated with the original file, you may want them to work for the new file too. You should consider either:
- getting copies of the format definitions from the format owners, changing the ID and FILE statements within them to reflect your ownership of the format definition and file, and then adding them to the FORMATS subfile and compiling them; or
- getting the format owners to add GEN-FILE statements to their format definitions so that they can be used by both files. (That would mean setting the formats slightly differently, using the "general file format" form of the SET FORMAT command.)
- If special ORVYL permits were set for the data sets you copied (which is likely if the file has immediate indexes, for instance), they were not brought across to the copies. Instead, SPIRES sets the same permits it does for a new file: READ access to the public for all data sets except the DEFQ, which gets public WRITE access. If you want to set the same permits as the original file, you need to find out what they were from the file owner (who can use the "SHOW PERMIT filename.dataset" command to recall them) and then issue the SET CLP and SET PERMIT commands to set them on the data sets of the new file:
SET CLP filename.dataset
SET PERMIT filename.dataset FOR account access-level
SET NOCLP filename.dataset
- If you now want to destroy the original file, follow the procedure "Destroying an Entire File" in the next chapter. [See 2.11.]
Procedure: Destroy a file Purpose: to destroy an unwanted SPIRES file Who can execute this procedure: file owner Basic Steps: 1) Make sure you no longer want the file. 2) Get rid of formats code associated with the file. 3) Issue the ZAP FILE command. 4) Remove file definition from FILEDEF subfile.
When you no longer want to keep a particular SPIRES file on your account, you can destroy it, following a procedure based on the ZAP FILE command. The example session outlined below shows how account GQ.DOC would destroy its file GQ.DOC.ERRATA.
If you are sure you want to destroy the SPIRES file but aren't absolutely sure that you will never again need the data within it, you might want to save copies of the records, storing them on tape, or on another account, etc. Or you might want to consider copying the entire SPIRES file to another account, using the COPY FILE procedure. [See 2.10.]
When you are destroying a file, please remove formats associated with the file from the FORMATS (format definitions) and FORCHAR (compiled format code) subfiles. To do so, follow this procedure:
-> select spires doc notes <-- a subfile of GQ.DOC.ERRATA
-> perform format list
Formats defined for file 'GQ.DOC.ERRATA'
Format-Name Record Source-ID Date Compiled
ERRATA.REPORT REC01 GQ.DOC.ERRATA.REPORT 03/27/86
->
SELECT FORMATS FIND FILE gg.uuu.filename FOR RESULT REMOVE ALL
There may be other records in system files relating to your file, such as EXTDEF, VGROUPS or SYS PROTO records. Please remove any records from those subfiles that are used only by the file you are discarding too.
The ZAP FILE command in SPIRES erases the ORVYL data sets that comprise the SPIRES file.
-> zap file gq.doc.errata -Do you want to destroy file GQ.DOC.ERRATA ? ok ->
Because of the seriousness of the ZAP FILE command, SPIRES asks you to confirm that you want to destroy the named file. Once you confirm it, the data sets are gone.
You should remove the file's file definition from the FILEDEF subfile. Before doing so, you should consider whether to save the file definition elsewhere, just as you did for the data in step 1. You can store old file definitions in a public SPIRES subfile called BACKFILE. Since the file is owned by the computing center, you are not charged any storage costs for keeping the definition there.
In this example, account GQ.DOC decides to move the file definition for GQ.DOC.ERRATA into the BACKFILE subfile:
-> select filedef -> transfer gq.doc.errata clear -> remove gq.doc.errata -> select backfile -> add ->
FASTBILD is a batch program you can use when you need to add large numbers of records to a brand new subfile, to build all the indexes of a large subfile, or to build a new index for a large subfile. Though it does work that SPIBILD can do, FASTBILD does the work much more efficiently under some circumstances, which means it can save you time and money. In addition, the data sets it builds are somewhat better built than SPIBILD's, meaning that record retrieval from them is more efficient.
FASTBILD is an alternative to SPIBILD only in very particular circumstances, however. Specifically, you should consider FASTBILD for building a new subfile when all of the following are true:
- you are adding 10,000 or more records to an empty goal record-type at one time;
- at least one index is being built from the goal records;
- the records are in sequential order by key in the input data set;
- the record-type has never had any records in it since it was compiled or "zapped";
- no goal-to-same-goal passing is involved, i.e., the goal record does not pass data to other goal records in the same record-type.
You should consider using FASTBILD to build all a subfile's indexes if both of the following are true:
- the data currently in the subfile will create 10,000 or more index entries for the new indexes;
- none of the index record-types have ever had any records in them since they were compiled or zapped.
You should consider using FASTBILD to build a single index if both of the following are true:
- you will be adding 10,000 or more index entries to the new index at one time;
- the index record-type has never had any records in it since it was compiled or zapped.
When CPU-usage is an issue and the preceding guidelines are met, you will get better results by carefully using FASTBILD rather than SPIBILD.
In most cases, file managers use FASTBILD to build new subfiles rather than to build new indexes alone. Hence, the discussions in this chapter are centered around the former use, though in general they are discussing both. However, if particular points need to be made about building new indexes, you will see them along the way.
This chapter is arranged as follows:
The basic FASTBILD task consists of two batch jobs:
- The first job adds the input goal records to the empty goal record-typ