INDEX
*  SPIRES File Management
+  Introduction
+1  About this Manual
+2  Notation Conventions
+3  Preface to the First Edition
1  File Management Primer
1.1  What is File Management?
1.2  File Management is Handled Automatically for many SPIRES Files
1.3  Processing a SPIRES File Yourself
1.4  Responding to SPIBILD Problems
1.5  Adding, Updating and Removing Multiple Records of a Subfile
1.5.1  The INPUT BATCH Procedure in SPIBILD
1.5.2  Updating Records Chosen by Non-Key Criteria
1.5.3  Removing Records Chosen by Non-Key Criteria
1.6  Getting Statistics about Your SPIRES File
1.7  Destroying a SPIRES File
2  Building and Destroying Files and Record-Types
2.1  Introduction to SPIBILD
2.1.1  The SPIBILD Program
2.1.2  Using Interactive SPIBILD
2.1.3  Using SPIBILD in Batch
2.1.3.1  Using the BATWYL Program for Access to SPIBILD
2.1.3.2  Using the Batch SPIBILD Program
2.1.3.2a  (Condition Codes for Batch SPIBILD Jobs)
2.1.4  Recovering from Problems in SPIBILD
2.1.4.1  Common Problems Encountered during File-Processing Tasks
2.1.4.2  Using the RECOVER/PROCESS Procedure for SPIBILD Problems
2.1.4.3  Backup Copies of SPIRES Files
2.1.4.4  Making Backup Copies of a File with the ORVCOPY Program
2.1.4.5  Restoring a File with the ORVCOPY Program
2.1.4.6  Copying a File's Deferred Queue from Another Account
2.1.5  Controlling the SPIBILD Environment
2.1.5.1  Using the SET SEARCH/SET SHARE Procedure to Share File Access
2.1.5.2  Using the Exception File Procedure to Repair Bad Input
2.1.5.3  Controlling Messages from SPIBILD
2.1.5.4  Limiting the Indexes Affected by a SPIBILD Command
2.1.5.5  Creating Output Reports During SPIBILD Processing
2.2  Moving Records into the Tree: Processing a File
2.2.1  Letting JOBGEN Handle Your File-Processing
2.2.2  Processing a File with the PROCESS/ESTABLISH Procedure
2.2.2a  Processing Only One Subfile of a File: PROCESS SUBFILE
2.2.3  Choosing How the File Should Be Processed
2.2.4  The Messages Displayed During File Processing
2.2.5  Allowing Others to Process Your Files
2.2.6  (*) How Passing Works
2.2.7  Passing Variable Values
2.2.8  Passing Table Values
2.3  Adding Multiple Records to a Subfile
2.3.1  The INPUT BATCH Procedure in SPIBILD
2.3.2  The INPUT ADD, ADDUPDATE and ADDMERGE Procedures
2.3.3  The LOAD Procedure in SPIBILD
2.3.3a  The INPUT LOAD Procedure
2.3.4  The FAST BATCH and FAST LOAD Procedures in SPIBILD
2.3.5  Formats for Input Data
2.3.6  SPIBILD Recovery to Resume Input
2.4  Updating Multiple Records in a Subfile
2.4.1  The INPUT MERGE Procedure in SPIBILD
2.4.2  The MERGE Procedure in SPIBILD with Global FOR
2.4.3  (*) Considering Efficiency for SPIBILD Multiple Update Requests
2.4.3.1  (*) How SPIBILD Updates Indexes for MERGE & UPDATE Requests
2.4.3.2  (*) Using SET NOPASS to Prevent Unnecessary Indexing Work
2.4.3.3  (*) Using the Pass-Elements List for SPIBILD Merge Processing
2.5  Removing Multiple Records from a Subfile
2.5.1  The REMOVE Procedure in SPIBILD with Global FOR
2.5.2  Using the ZAP RECORD-TYPE Procedure to Remove All Records
2.6  Adding and Removing Elements in a Record-Type
2.6.1  Adding a New Element to a Record-Type
2.6.2  Removing Unneeded Elements from a Record-Type
2.6.3  General Rules for a Record-Type
2.7  Adding, Removing and Rebuilding Indexes of a Subfile
2.7.1  Adding an Index to a SPIRES Subfile
2.7.2  Removing an Index from a SPIRES Subfile
2.7.3  Rebuilding an Index
2.8  Adding, Removing and Rebuilding Subfiles in a File
2.8.1  Adding a New Subfile to a SPIRES File
2.8.2  Removing a Subfile from a SPIRES File
2.8.3  Rebuilding a Subfile in a SPIRES File
2.9  Rebuilding an Entire SPIRES File
2.10  Moving a SPIRES File from One Account to Another
2.11  Destroying a SPIRES File
2.12  Using FASTBILD to Build Subfiles and Indexes
2.12.1  Introduction to FASTBILD
2.12.2  The FASTBILD Procedures
2.12.3  Using the FASTBILD.JCL Protocol
2.12.4  FASTBILD Commands
3  Keeping a File Tuned Up
3.1  Introduction
3.2  Gathering Statistics about Tree Growth and Space Usage
3.2.1  Determining the Physical Size of a SPIRES File
3.2.2  Determining the Storage Space for Individual Records
3.2.3  Getting File Statistics from the STATUS Command
3.3  Rebalancing Tree-Structured Record-Types
3.3.1  The REBALANCE Procedure
3.3.2  Handling Unusual Situations and Problems with Rebalancing
3.3.3  Rebalancing the Deferred Queue: The REBALANCE DEFQ Command
3.3.4  Rebalance Procedure, RESIZE
3.4  Ensuring Efficient Use of Storage Space in the Residual
3.4.1  (*) How SPIBILD Keeps Track of Available Storage Space
3.4.2  Determining Whether To Rebuild the Available-Space Chains
3.4.3  Rebuilding Available-Space Chains
4  Keeping Track of How Your File Is Being Used
4.1  Introduction
4.2  Monitoring File Transactions
4.2.1  Examining the Added, Updated and Removed Records
4.2.2  Getting Basic Overall Statistics about Transactions
4.2.3  Getting Information about Specific Subfile Transactions
4.2.3.1  Getting Basic Transaction Information
4.2.3.2  Creating the $TRANSACTIONS Record-Type
4.2.3.3  Using the $TRANSACTIONS Record-Type
4.3  Logging Subfile Use
4.3.1  Establishing a File Log
4.3.1.1  How to Create a Charging Subfile
4.3.2  How File Log Entries are Created
4.3.3  Handling the File Log
4.3.4  Reading the File Log
5  Miscellaneous File Management Tasks
5.1  Introduction
5.2  Saving Money on Existing SPIRES Files
5.3  Resetting the Next Slot Number of a Record-Type
5.4  Examining Record-Types Directly
5.5  Disabling a File Temporarily to Block Selection and Updates
5.6  Converting to or Modifying Files with Extended Trees
5.7  Perform Table Create Commands.
5.7.1  Perform Table Create Declare Command
5.7.2  Perform Table Create DDL Command
5.7.3  Perform Table Create DATA MOVE Command
5.7.4  Perform Table Create RECDEF Command
5.7.5  Perform Table Create Input Commands.
5.7.6  Perform Table Create Input Declare Command
5.7.7  Perform Table Create Input INPUT RECDEF Command
5.7.8  Perform Table Create Input INPUT CONTROL Command
5.7.9  Perform Table Create Eleminfo Command
5.7.11  Perform Table Create Declare, Using ELEMINFO
5.8  Perform Data Move Command
6  File Management Commands: Reference
6.0  (File Management Commands: Reference)
6.0.1  (ATTACH)
6.0.2  (AVSPREC)
6.0.4  (BUILD)
6.0.5  (CLEAR EXCEPTION)
6.0.5a  (CLEAR FILE COUNTS)
6.0.5b  (COPY DATA SET)
6.0.6  (COPY FILE and COPY DEFQ)
6.0.7  (ESTABLISH)
6.0.8  (EXIT)
6.0.9  (FAST BATCH)
6.0.9a  (FAST LOAD)
6.1.0  (FIX BASE)
6.1.0b  (FIX NEXT BLOCK)
6.1.0c  (FIX MAXIMUM BLOCKS)
6.1.1  (FIX REBALANCED DATA SET)
6.1.1b  (FIX SUBFILE NODE)
6.1.2  (FIX SUBFILE SLOT)
6.1.2b  (FOR STORED)
6.1.2c  (GENERATE CHANGE FILE)
6.1.2c.1  (GENERATE FILE)
6.1.3  (GENERATE LOAD)
6.1.4  (GENERATE NONIXR)
6.1.4.1  (INPUT ADD)
6.1.4.2  (INPUT ADDMERGE)
6.1.4.3  (INPUT ADDUPDATE)
6.1.4.4  (INPUT BATCH)
6.1.4.4b  (INPUT LOAD)
6.1.4.5  (INPUT MERGE)
6.1.5  (LOAD)
6.1.6  (MERGE, in SPIBILD)
6.1.7  (PERFORM USER LIST)
6.1.8  (PROCESS)
6.1.8a  (PROCESS SUBFILE)
6.1.9  (REBALANCE DATA SET)
6.1.9a  (REBALANCE DEFQ)
6.2.0  (RECOVER)
6.2.1  (REMOVE, in SPIBILD)
6.2.2  (SET AUTOGEN and SET NOAUTOGEN)
6.2.3  (SET EXCEPTION)
6.2.3a  (SET FILE COUNTS)
6.2.4  (SET INDEX and SET NOINDEX)
6.2.5  (SET JOBGEN and SET NOJOBGEN)
6.2.6  (SET MESSAGES)
6.2.7  (SET PASS and SET NOPASS)
6.2.8  (SET SEARCH and SET NOSEARCH)
6.2.9  (SET SHARE and SET NOSHARE)
6.2.10  (SET REINDEX)
6.3.0  (SET SKIP)
6.3.0a  (SET STACK SIZE)
6.3.0b  (SET SUBFILE DISABLE and SET SUBFILE ENABLE)
6.3.1  (SET TCOMMENT)
6.3.2  (SHOW ARCHIVE)
6.3.2a  (SHOW AUTOGEN)
6.3.3  (SHOW BLOCKS)
6.3.3a  (SHOW ERROR COUNTS)
6.3.4  (SHOW FILE ACTIVITY)
6.3.4a  (SHOW FILE LIMITS)
6.3.5  (SHOW FILE LOG)
6.3.6  (SHOW FILE STATUS)
6.3.7  (SHOW FILES)
6.3.7a  (SHOW RECORD INFORMATION)
6.3.8  (SHOW RECORD OVERHEAD)
6.3.9  (SHOW RECORD RECOVERY)
6.4.0  (SHOW SUBFILE BLOCK)
6.4.1  (SHOW SUBFILE TRANSACTIONS)
6.4.1a  (SHOW FILE TREES)
6.4.2  (SPIBILD)
6.4.4  (STATUS)
6.4.5  (TESTDEFQ)
6.4.6  (ZAP DATA SET)
6.4.7  (ZAP DEFQ)
6.4.8  (ZAP FILE)
6.4.9  (ZAP RECORD-TYPE)
6.5.0  (ZAP SUBFILE NODE and ZAP SUBFILE DEFQ NODE)
6.5.1  (ZAP SUBFILE TRANSACTION)
6.5.2  (ZAP BAD POINTERS)
:  Appendices
:1  Glossary of Important File-Management Terms
:29  SPIRES Documentation

*  SPIRES File Management

******************************************************************
*                                                                *
*                     Stanford Data Center                       *
*                     Stanford University                        *
*                     Stanford, Ca.   94305                      *
*                                                                *
*       (c)Copyright 1994 by the Board of Trustees of the        *
*               Leland Stanford Junior University                *
*                      All rights reserved                       *
*            Printed in the United States of America             *
*                                                                *
******************************************************************

        SPIRES (TM) is a trademark of Stanford University.

+  Introduction

+1  About this Manual

The audience

This manual was written for people who own SPIRES files or who are responsible for the maintenance of SPIRES files. If you are one of these people, you should already be familiar with many aspects of SPIRES. We consider the material presented in the SPIRES primer "A Guide to Data Base Development" as the pre-requisite for this manual. For instance, if you do not know the terms "subfile", "deferred queue", "Global FOR", "File Definer" and "file definition", you should begin with that manual, which introduces some very basic concepts of File Management that we will expound in this manual.

The arrangement of this manual

Though the Table of Contents may not show it, this document is divided into three major portions:

+2  Notation Conventions

Examples

In SPIRES manuals, examples of sessions are shown with prompts and messages from SPIRES as they actually appear on the terminal screen; SPIRES prompts usually end with a question mark or greater-than sign (">"), while SPIRES messages usually begin with a hyphen. The commands and responses you type are shown in lowercase, following a prompt. (You may generally use upper or lower case interchangeably when you actually use SPIRES.) In some editions of this manual, what you type will be shown in bold, while the computer's responses are shown in normal type. For example:

Here SPIRES types "-OK to clear?" and you type "ok".

Formal command syntax descriptions

In formal command syntax descriptions, uppercase letters denote command verbs or other command elements to be entered exactly as shown; a value for lowercase terms and characters must be supplied by you. For example:

To use this particular command, you type the command verb SELECT with the name of the desired subfile, for instance, "Restaurant":

where "->" is the prompt from SPIRES.

Brackets ([ ]) denote optional elements. Braces ({ }) indicate that you must specify one and only one of the alternatives within the braces. Within the braces or brackets, a vertical line (|) separates possible choices. Neither brackets nor braces are to be typed as part of the command. For example:

could be entered as

depending on whether you want the option. Here is an example from the report writer command language using braces:

You must choose one of the words in the braces when using this command. For example, this command could be entered as

or as

(*) = Technical Detail Indicators

Sections and subsections of this manual whose titles are preceded by an asterisk in parentheses -- (*) -- may be considered optional reading at that point. Usually the material provides details that most users will not need about how to handle uncommon situations, or technical information about how SPIRES internally handles some particular command.

+3  Preface to the First Edition

If you are familiar with other SPIRES manuals, you will notice many substantial differences in the areas of structure and style between this manual and those. These changes result from a study to evaluate SPIRES documentation in which different contingents of SPIRES users were interviewed. Though we'd like to reprint here all the complimentary comments we received, it's perhaps more useful to discuss some of the criticisms and suggestions, and mention how this manual addresses them.

Most of the users we interviewed suggested that SPIRES documentation would be more useful if it were more "task-oriented". Most SPIRES users come to documentation trying to accomplish a specific task, and may not know or recall the names of the tools used to do it. To address this concern, we have organized most of this manual, the "User Guide" portion, by tasks. In other words, to find out how to perform a certain file management task, you would probably look up the task in the table of contents or index. All the basic information about solving the task will appear together in one chapter or often in one section of a chapter.

Another request, related to the previous one, was to provide a standardized presentation of each command with full reference material. You will find that in Part 6, with all the commands presented in alphabetic order. Hence, if you already know the name of the command you're interested in, you can quickly find the reference material on it in Part 6. Online, you will discover, if you EXPLAIN any of the commands, that the section of documentation you see comes from that part of the manual.

In addition, depending on the source of your copy of this manual, you may notice that this manual has been printed with proportional spacing, guide words in the margins, and other special typographical improvements.

Besides the new style and structure, there is plenty of new material. Though much of the information in this manual has been seen before in the reference manuals "File Definition", "Formats" and "Technical Notes", much of it is new too. We think you will appreciate having the material together in a single manual, despite any duplications between manuals.

If you think of any aspects of SPIRES file management that weren't covered in this manual and which you'd like to see addressed in a future edition, please send them to your SPIRES consultant, or send them directly to SPIRES Documentation, Stanford Data Center, Stanford University, Stanford, California 94305.

Acknowledgments

This manual is the creation of many contributors, including the earliest designers of SPIRES back in the late 1960's and recent users who suggested improvements to the explanation of this or that feature in SPIBILD. In general tribute to all, we here thank many specific individuals for their significant help.

SPIRES is developed and maintained by the SPIRALS team of Stanford University's Stanford Data Center. Programming responsibility for SPIRES is shared by Dick Guertin and Bill Kiefer, both of whom contributed enormously to the development of this manual. Chuck Boeheim, responsible for the VM/CMS implementation of SPIRES, was very helpful in that area. Sandy Laws, Peter Tuttle, Lynn McRae, Norman Roth and Becky Morton also provided important technical support and review.

Others outside of SPIRALS provided concrete suggestions about the manual's contents, including Ed Begley of I.T.S.'s Community Information Resources and Hannah Kaufmann and Michael Sperberg-McQueen of Princeton University's Center for Information Technology.

The style of this manual is significantly different from SPIRES manuals written in the past, as is its physical design. Major contributors in this area include DRG staff members Lynne Sinclair, Jeff Rensch and Tony Navarrete, as well as Terry Oldano, Director's Office, and Anne Janzer, Systems. A new GML format for this document was patiently and painstakingly created by Greg Bruce.

For printing support, we acknowledge the assistance of staff members Lloyd Warble and Mark Lawrence of Electronic Printing and Lincoln Ong of Operations. Charley Hoyt of Systems deserves special commendation for his customization work on Waterloo Script's Photo processor especially for this document.

The manual was written by John Klemm, who thanks those named above and everyone else who provided ideas, patience, and non-addictive stimuli when they were most needed.

1  File Management Primer

1.1  What is File Management?

Managing a SPIRES file consists of tasks in three broad categories:

 - record-transaction tasks, consisting of adding, updating or removing "batches" of
 records from the file; [You might consider  that  adding,  updating  and  removing  records
 individually  in  SPIRES  are  file  management  tasks,  albeit  on a smaller scale. But in
 general, the term "file management tasks" refers  to  work  involving  groups  of
 records rather than individual ones.]
 - routine maintenance tasks, ensuring that the records are being stored efficiently, that the
 indexes are being updated, that the file has a backup copy in storage, etc.
 - repair tasks, perhaps involving the destruction or reconstruction of an  index,  an  entire
 subfile or an entire file, perhaps done in response to information from a maintenance task.

Who needs to know about file management?

The same file management capabilities are available to all SPIRES files, regardless of size. But many of them are not usually needed by small files; the test to determine whether SPIRES is storing some data inefficiently could cost more than the inefficiency itself does. But for large files (say, over 100,000 records), such tests could be very cost-effective in the long run, which is how long such files generally last.

The other parts of this manual cover the file management tasks in detail, principally for the benefit of owners of large files. Owners of small files typically do not have the need, interest or incentive to learn so much about SPIRES, and certainly shouldn't have to do so.

This primer introduces basic methods of file management

In view of that, the rest of this part of the manual introduces the most important aspects of file management for all SPIRES files. If you own a small file, you will probably find enough information here to meet your needs; if not, suggestions for further reading in the rest of the document are made along the way. But even if you own or manage larger files, you may find this introduction useful too.

Glossary

Many of the terms that show up in discussions of SPIRES file management are informally defined in a glossary at the end of this manual. The index in the back includes a complete list of the file management tasks addressed by this manual, under the heading "file management task", with page number references to the appropriate sections, of course. Online, you can see the list by issuing the command EXPLAIN FILE MANAGEMENT TASK, which also gives you online access to all those sections of the manual too.

1.2  File Management is Handled Automatically for many SPIRES Files

For small SPIRES files, file management may be quite simple:

 - records are updated one at a time;
 - the JOBGEN program causes the file to be processed every night so that the tree is  updated
 using the information in the deferred queue, and the indexes are updated;
 - the file is destroyed with the ZAP FILE command when it is no longer needed.

In a sense, such files have little use for "file-management procedures"; SPIRES takes care of the most important task (automatic processing of the file) without any intervention from you.

Many files are processed each night by JOBGEN, SPIBILD

JOBGEN is a program, running after midnight each night, that examines the deferred queue of each SPIRES file requesting this service. If the deferred queue is empty, JOBGEN does nothing more to that file. But if it is not empty, JOBGEN creates a job to process the file in the SPIBILD program ("SPI" stands for SPIRES; "BILD" stands for "file-building". SPIBILD's tasks center around file-building.); the job usually runs a few minutes later. The file owner's account is charged for the cost of the job. [See 2.2.1.]

JOBGEN runs during the night because computing rates are cheaper then, and because the files are less likely to be in use at that time. (In general, SPIBILD jobs need exclusive access to a file during processing.)

The AUTOGEN flag

The AUTOGEN flag, which is most often controlled by the file owner, determines whether JOBGEN is responsible for processing the file. The flag is set in one of three ways:

 - 1) When the file is defined, the file owner determines a default setting  for  the  AUTOGEN
 flag.   Specifically,  if  the  file  definition  contains  the  NOAUTOGEN  statement, then
 NOAUTOGEN is the basic (default) setting for the file, i.e., JOBGEN is not responsible  for
 the  file.  (File Definer's NOAUTOGEN attribute of the FILE statement has the same effect.)
 Otherwise, the basic setting is AUTOGEN.
 - 2) The file owner, or a user with Master access to the file [The file definition  may  give
 other  users  certain  file-owner powers -- this is called "Master access". This,
 and other forms of file access, such as Process access, are discussed in  the  glossary  at
 the end of the manual.] may issue the SET AUTOGEN or SET NOAUTOGEN command to change to the
 desired setting.
 - 3) Every time the file is processed, by either JOBGEN or you, the file returns to its basic
 setting, according to the file definition (see step 1).

Turning off automatic processing

To turn off automatic processing for awhile, you can just issue the SET NOAUTOGEN command when you have the subfile selected in SPIRES:

Turning it on again

On the other hand, if you have NOAUTOGEN in effect but decide that it's time for the file to be processed again, you can issue the SET AUTOGEN command, and the next time JOBGEN finds data in the deferred queue, the file will be processed:

Remember that the file will revert to its default setting of AUTOGEN or NOAUTOGEN, according to the file definition, after the file is processed.

1.3  Processing a SPIRES File Yourself

Occasionally you may not want to wait for overnight processing of a file by JOBGEN. You may need your indexes updated immediately, for one reason or another, so you need the file processed now. (This scenario assumes the indexes aren't "immediate indexes", which are updated each time a record transaction occurs.)

Process it yourself only if you must

Processing a file is quite a simple task to do, but unless you have compelling reasons to process the file yourself, you should leave it to JOBGEN. [See 2.2.1.]

SPIBILD and the PROCESS command

If you are the file owner, or if you have PROCESS access to the file, you can process a SPIRES file. Processing the file is most commonly done in the SPIBILD program. To process a SPIRES file, you first call SPIBILD and then issue the PROCESS command:

The commands SPIBILD and EXIT take you in and out of SPIBILD. The SPIBILD program is similar to SPIRES in that you can issue WYLBUR and MILTEN commands within its environment, though you cannot issue SPIRES commands.

EXPLAIN command in SPIBILD

The PROCESS command causes the named file to be processed, and the output from the command as the processing occurs usually looks like the example above. If you try it but have problems, try using the EXPLAIN command to find out more about any error messages you received -- the EXPLAIN command is available in SPIBILD.

Processing can be done in SPIRES

You can also process the file in SPIRES itself, without having to go into the SPIBILD program. Processing the file in SPIRES has some limitations that may affect you, but in most cases, they will not be significant. The advantages of processing in SPIRES include the ability to put the PROCESS command into a protocol, and being able to direct the error messages into the "trace log" (SET TLOG). [See 2.2.2 for a comparison of file-processing in SPIBILD and SPIRES.]

1.4  Responding to SPIBILD Problems

Most of the time, SPIBILD processes your files without problems. However, sometimes power outages or system problems occur during a SPIBILD session, causing the file processing to stop in mid-stream.

You generally find out that your file has been affected by a SPIBILD problem in one of three ways:

 - the computer tells you that a problem occurred as you are processing the file  in  SPIBILD;
 - you get output from JOBGEN that tells you that the file-processing job it created  did  not
 run properly;
 - users discover that  they  cannot  accomplish  any  subfile  transactions  (adds,  updates,
 removes)  for the file because the transactions are blocked when a SPIBILD processing error
 has occurred.

File is probably unharmed

This doesn't mean the file is damaged; it merely means that the file is unstable, and normal subfile transactions cannot resume till SPIBILD cleans up whatever problem it had.

The RECOVER command

In general, the best way to "recover the file" is to issue the RECOVER command in SPIBILD, naming the file to recover:

The RECOVER command begins to process the file again, though if there is a large amount of deferred queue data, the RECOVER command may successfully complete without actually finishing the file processing. When the RECOVER command completes its work, the file should be back to normal.

On the other hand, if you receive error messages rather than the messages shown in the example above, your file still has problems that need to be resolved. [See 2.1.4 for a detailed discussion of SPIBILD problems you might encounter.]

Extremely serious problems

In rare cases when a file's problems are so serious that its integrity is questionable, or when a disk-storage problem arises, your computer center may need to use backup copies of data sets to restore the file to its state at the point when the backup was made. If you believe you need to take advantage of this service, contact your SPIRES consultant for help.

1.5  Adding, Updating and Removing Multiple Records of a Subfile

One of the most common duties of the file manager is to "build the file" -- that is, add many records to the file at once to get it started. Another common task that arises occasionally is to add, update and/or remove many records of a subfile at one time.

SPIRES for multiple record-transactions

There are quite a few ways to apply multiple record-transactions to a subfile at once. In general, the easiest and least expensive methods (to the user doing the work) use SPIRES, putting the record transactions into the deferred queue. Then, when the file is processed in SPIBILD, perhaps automatically overnight, the deferred queue data will be applied to the tree of the subfile.

SPIBILD for multiple record-transactions

On the other hand, if getting the data into the tree and having the indexes reflect the new data immediately is important to you, you can use SPIBILD techniques instead. Remember that they may cost more than the SPIRES techniques, particularly if the processing is not occurring overnight, when rates are likely to be cheaper. On the other hand, if all the charges are being borne by the file owner, then doing all the work at once in SPIBILD is likely to be cheaper than putting the data first into the deferred queue in SPIRES and later having SPIBILD move the data into the tree.

Descriptions and comparisons of all the different procedures you can use for multiple record-transactions appear later, in several chapters of Part 2. [See 2.3, 2.4, 2.5.] Here we will discuss the three simplest and most general procedures of SPIBILD, probably the most useful ones to owners of small files:

 - the BATCH procedure in SPIBILD, which allows you to add new records and update,  merge  and
 remove records whose keys you know; and
 - the MERGE procedure in Global FOR in SPIBILD, which lets you  update  multiple  records  at
 once  using  the  same  merge  input  for each one, choosing the records to be updated with
 non-key criteria.
 - the REMOVE command in Global FOR in SPIBILD, which lets you remove multiple records at once
 using non-key criteria to determine the records to remove.

1.5.1  The INPUT BATCH Procedure in SPIBILD

The INPUT BATCH command in SPIBILD provides a fast and reliable method of getting data into a SPIRES subfile and getting the indexes updated. It's available to any user who can process the file in SPIBILD.

SPIBILD examines the data for each added and updated record, rejecting it if it breaks any of the processing rules defined for the subfile. Each record that succeeds is placed in the tree of the file. For removes, SPIBILD removes that record from the tree. SPIBILD then updates any indexes defined for the subfile.

For this procedure, your data must be in the standard SPIRES format, with each record preceded by a command followed by a semicolon ("ADD;" or "UPDATE 226-82-3846;") and (except for REMOVE) followed by an additional semicolon. [See 2.3.5.]

Here's a brief example of some data in the standard SPIRES format:

This procedure processes the file

There is one caveat about using this procedure: the ESTABLISH command used at the start processes the file, which processes all the transactions currently in the deferred queue. If you don't want that to happen for some reason, consider using the INPUT BATCH command in SPIRES.

Steps of the Procedure

Here are the steps of the INPUT BATCH procedure in SPIBILD:

Step 1: SPIBILD

If you aren't already in SPIBILD, issue the SPIBILD command:

Step 2: Put the data into your active file.

You must have the input data in your active file. If you don't already have it there, use the appropriate WYLBUR command to put it there. For example,

Step 3: SET INFORMATION MESSAGES = 0 (if desired)

You will get one line of informational data about each successful record from the INPUT BATCH command by default. If you are working with dozens of records, those messages can be tedious, and they add a small degree of inefficiency too. You can turn them off by issuing the command SET INFORMATION MESSAGES = 0; it does not suppress any error messages. [See 2.1.5.3.]

Step 4: Set an exception file, if desired.

It is a good idea to tell SPIRES to put the data for transactions that fail into a special data set called an "exception file". By extracting the bad data from the good data, SPIRES makes it easier for you to find the problems in the data so that you can correct it and submit it again with a subsequent INPUT BATCH command.

The exception file is an ORVYL data set that you name yourself, in the SET EXCEPTION command. For example:

SPIRES will put any errant records into the ORVYL data set called BAD.SUPPLIES.RECORDS on your account. [See 2.1.5.2.]

Step 5: ESTABLISH subfile-name

Issue the ESTABLISH command to tell SPIBILD which subfile you want to use and to have SPIBILD process the file containing it:

For example:

Step 6: INPUT BATCH

Next issue the INPUT BATCH command to tell SPIBILD to read the input data from your active file and apply it to the subfile:

Continuing the above example:

In this example, the information messages were not turned off (step 3); had they been, the first three lines shown would not have appeared, since they indicate successful transactions.

Handling Problems

Invalid Data

The most common problem is that one or more transactions fail due to input errors, such as an invalid element name, or an improper element value. SPIBILD will display error messages telling you what the problem is. Assuming that you have set an exception file, it is easy to retrieve the unsuccessful data, using the ORVYL command GET:

The data is now in your active file. After correcting it, you can return to step 3 and continue the procedure.

System Failure

If the system fails during any step of the procedure except step 6, just restart from the beginning when it is convenient to. If it fails during the INPUT BATCH command, you need to determine the point where it failed, i.e., figure out which was the last record to be fully processed. A command called SHOW RECORD RECOVERY may be useful in this regard. For full details, see the complete discussion on the INPUT BATCH procedure in SPIBILD. [See 2.3.1.]

1.5.2  Updating Records Chosen by Non-Key Criteria

Frequently the file manager may find that many records sharing some element value need to have that element value changed or that they need to be removed from the subfile. Removing records will be discussed in the next section; here we will discuss updating them.

The MERGE procedure in SPIBILD under Global FOR

In the updating situation, SPIBILD provides a MERGE ALL command using Global FOR, which merges the data in your active file into all the records that meet the Global FOR criteria. Like SPIBILD's BATCH procedure, this procedure causes the file to be processed prior to the MERGE command and SPIBILD processes the file again at the end of it, so it can be relatively expensive. [See 2.4.2.]

Steps of the procedure

The procedure for using it is described below. Note: You must be able to process the file to use this procedure.

Step 1: SPIBILD

If you aren't in SPIBILD already, get in:

Step 2: ESTABLISH the subfile you want to use.

Tell SPIBILD which subfile you want with the ESTABLISH command, which also processes the file:

In this example, the intent is to find records with the value RCA for the singly-occurring element SUPPLIER and replace it with the value "General Electric". See the SPIRES manual "Searching and Updating" for more information on merging in general (for instance, how to add new occurrences, replace multiple existing ones, remove occurrences, etc.).

Step 3: Put the data into your active file.

You must have the input data in your active file. It can be in the standard SPIRES format or in a custom format; if the latter, you should issue the SET FORMAT command as part of this step. [The format must be a merge format. See the manual "SPIRES Formats" for further information.] For example,

Step 4: Establish the Global FOR class of records to update.

In SPIBILD, the only Global FOR classes available are the SUBFILE and STORED classes. You can, however, use a WHERE clause and SET SCAN commands to further refine the group of records to be updated.

For example,

The SET SCAN START and STOP commands tell SPIBILD to look only at records with key values from 10000 to 19999. This is considerably more efficient than "FOR SUBFILE WHERE SUPPLIER = RCA" alone, if all the records you want are within the limited range.

The STORED class would be preferable to use if, for example, you could easily create a stored result or stack in SPIRES of the records you wanted to update. Then SPIBILD could retrieve the records for updating directly, rather than have to use WHERE-clause examination techniques to find the records to update. Suppose, for instance, that in SPIRES you could store the search result from the command FIND SUPPLIER = RCA. Using that stored search would be much more efficient than having SPIBILD examine each record in the subfile.

Step 5: Issue the MERGE ALL command.

The MERGE ALL command merges the data in your active file into each record fitting the Global FOR class, after which SPIBILD processes the file so that the indexes are all updated.

Handling Problems

Invalid data

If an error occurs during step 5, SPIRES will tell you about it and stop. The most likely error comes from bad input data, i.e., an invalid element name or value for that element. That will most likely happen on the very first record -- if it won't go into one record, it probably won't go into any.

To recover from this problem, simply correct the data and start again from the ESTABLISH command in step 2.

Wrong records were updated

If you discover that you updated the wrong records, you will need to determine another way to find those records and perhaps "re-merge" the old data into them.

System failure

If the system crashes during the first 4 steps of the procedure, simply restart it at your convenience. If the system crashes during the MERGE processing, see the explanation for handling this problem in the full discussion of the MERGE procedure in Global FOR in SPIBILD later in this manual. [See 2.4.2.] A simple possibility worth mentioning here is to simply restart the procedure and merge the same data again into the records that succeeded. Do not do this, though, if the merge requests are creating new element occurrences or if the data is being used to compute other element values.

1.5.3  Removing Records Chosen by Non-Key Criteria

Sometime you may have to remove many records from the subfile based on some criteria they share, such as all records added to the subfile more than two years ago, or records that don't have occurrences of some element.

Use REMOVE command in Global FOR

This situation is quite easily handled in SPIRES, using the REMOVE command in Global FOR mode. REMOVE is also available under Global FOR in SPIBILD and has the same effect; in fact, it works more efficiently there than in SPIRES. However, you are limited to the Global FOR classes SUBFILE (with WHERE clauses, SET SCAN and SET FILTER commands to limit the records examined and removed) and STORED (with WHERE clauses, the SET SCAN LIMIT and SET FILTER commands) in SPIBILD.

Suppose, for example, that you want to remove 50 records of a 1000-record subfile, and you can find those 50 records using a simple search:

In that case, you may want to simply remove the records in the search result using SPIRES:

The removal requests are in the deferred queue, so the records won't be removed from the tree till the file is next processed.

Another possibility would be to store the result (or stack) in SPIRES for use under the FOR STORED command in SPIBILD:

In SPIBILD, you can use the command FOR STORED RESULT.WOOLCO to remove only the records in the result (see step 3 below); under SPIBILD processing the file would be processed, and the records would be removed from the tree and "un-indexed" immediately.

Especially if there are several hundred records to remove (say, more than 15 percent of the subfile), the SPIBILD methods discussed below are preferable.

Rebuild if removing many records

Warning: If the subfile is fairly large (say, more than 1000 records) and you want to remove more than half of them, you should consider zapping the goal record-type and rebuilding it with only the records you want to keep. [See 2.8.3.]

If you remove the wrong records

One final aspect to consider between the SPIRES and SPIBILD methods of removing records in Global FOR -- in SPIRES, the removal requests go into the deferred queue and can be unqueued if you later determine that you removed the wrong records. In SPIBILD, once they are removed, they are gone. Often when file managers remove many records from a subfile using SPIBILD, they make sure they have a backup copy of the file around, just in case. [See 2.1.4.3.]

Steps of the SPIBILD procedure

The procedure for removing multiple records in SPIBILD is described below. If you plan to use FOR STORED processing, as described above, create your stored result or stack in SPIRES before beginning. Note: You must be able to process the file to use this procedure.

Step 1: SPIBILD

If you aren't in SPIBILD already, issue the SPIBILD command:

Step 2: ESTABLISH the subfile you want to use.

Tell SPIBILD which subfile you want with the ESTABLISH command, which also processes the file:

Step 3: Establish the Global FOR class of records to remove.

Remember that you are limited here to the FOR SUBFILE or FOR STORED command, with an optional WHERE clause. You should also use SET SCAN commands to limit the records examined, if you can, since that will save unnecessary work.

If you're not familiar with the SET SCAN commands, you might think of them this way: suppose you are asked to look for one book in the library, and you have no card catalog (index) to use. You must look on every shelf in the library till you find the book you want. But if you are told that the book you want must be on the north end of the third floor, the work you must do has been narrowed considerably. The SET SCAN commands do that type of limiting, telling SPIRES what record keys to examine.

For example, here we are removing old records from a subfile with slot keys:

It happens that we know that no records added before 1985 have keys greater than 500. The SET SCAN STOP command shown ensures that only records whose keys are 500 or less will be examined for removal.

To use FOR STORED instead, you would replace FOR SUBFILE with FOR STORED, along with the name of the stored stack or result.

You may also add a WHERE clause to further limit the records that are removed; SET SCAN LIMIT is the only SET SCAN command allowed with FOR STORED, however.

Step 4: REMOVE ALL

Issue the REMOVE ALL command to tell SPIBILD to remove all the records that match the criteria established by the Global FOR class:

Handling Problems

Wrong records were removed

If you discover that you removed the wrong records, the records are gone and cannot be recovered except via any backup copies of the file or data you may be able to use.

System failure

If the system crashes during the procedure, you may simply restart the procedure at your convenience.

1.6  Getting Statistics about Your SPIRES File

As its manager, you have access to many interesting statistics about your SPIRES file, such as the amount of storage space the file is using, or the kinds of commands users are issuing when they select your subfile.

Here are some of the statistical questions you might ask about your file, along with possible answers.

How much storage space is my file using?

The ORVYL command SHOW FILES

This is a useful question to ask because storage charges are based on the number of blocks the file uses. To find out the answer, issue the ORVYL command SHOW FILES, including the name of the file and the BLOCKS option:

The total of the number of blocks (the "BLKS") is the size of your file (260 in this example). [See 3.2.1.]

How many records are in each record-type of the file?

The SHOW FILE ACTIVITY command

One way to get these statistics is to issue the SHOW FILE ACTIVITY command when you have a subfile of the file selected. For example,

-> select almanac
-> show file activity

    01/26/87     Activity Record of File GQ.JNK.ALMANAC
                   Last Compiled: 01/07/87 at 23:55:02

 Record      Base     Blocks     Adds    Deletes   Updates   Max Rec
  Type       Block   Used -DS                                 Length

 ENTRY -Slot     0     14 -1      3272         0        22       339
 REC2            1    105 -2       373         0      1841        72
 REC3            8   Comb -2      3180         6        89        18
 REC4           25   Comb -2      2564         2       407       417
 REC5          103   Comb -2         7         0        21      2004
 REC6          104   Comb -2         2         0        10     11063

Residual         2       123      3488         7       414     11067
Def Queue        1         3
->

By subtracting the "Deletes" from the "Adds", you get the number of records in each record-type. The ENTRY record-type, for instance, has 3272 entries (3272 minus 0).

The SHOW FILE ACTIVITY command displays some other useful information too, such as the date and time when the file was last compiled. [See 4.2.2.]

Another way is to use the ATTACH command to "attach" each record-type of the file, followed by the SHOW SUBFILE SIZE command. The SHOW SUBFILE SIZE command does exactly the work you do -- subtracting the "Deletes" from the "Adds":

The ATTACH command lets you use any record-type of a file as if it were the goal record-type of a selected subfile. [See 5.4.]

Who updated this record in the deferred queue?

The SHOW SUBFILE TRANSACTIONS command

If you need more information about a particular transaction in the deferred queue, such as when it happened or who did it, issue the SHOW SUBFILE TRANSACTIONS command, optionally followed by the key of the record you are interested in. For example,

-> show subfile transactions 12933

    01/26/87     Transaction Log of File GQ.DOC.ERRATA
                         for Record-type G1SECT

  Date      Time      Account   Id   Type  Command  Grp  Key Value

  01/26/87  15:45:28  GQ.JNK         ADD   Update        12933
  01/26/87  13:32:35  GQ.JNK         ADD   Update        12933
  01/26/87  10:54:10  GQ.JNK         ADD   Add           12933
->

Information about the transactions for the record that have been made since the file was last processed are shown, listed in reverse chronological order. This example shows that user GQ.JNK added the record at 10:54 a.m., and updated it twice later in the afternoon.

If you omit a specific key from the command, SPIRES will list all the transactions for the selected subfile that are in the deferred queue, in chronological order. More information about how to read the transaction log appears later in this manual. [See 4.2.3.1.]

What indexes do the users use to search the subfile?

This is representative of many questions you may have about how users are taking advantage (or failing to) of your subfile. You may want to know how many users are selecting it, or what commands they are issuing to use it.

Subfile logging

You can ask SPIRES to "log" the use of your subfile by adding some statements to your file definition (directly or via your File Definer input). Later you can analyze the log's contents to help you evaluate the usefulness of your file, and to help you look for ways it could be improved, e.g., new indexes it needs, current indexes it doesn't need, etc.

Directions for establishing a log appear later in this manual. [See 4.3.]

What other statistics are available?

Other statistical information can tell the file manager how efficiently the file's blocks are being used for storage, and how many levels deep are the trees of the various data sets of the file. This information, and much more, is available through the STATUS and SHOW FILE STATUS commands. [See 3.2.3.] But in general, those commands are of interest (and of need) to managers of large files.

1.7  Destroying a SPIRES File

When you no longer need a SPIRES file, you should destroy it, so that it will stop accruing storage charges. If the data contained in it is important to you, you might want to make a printed listing of the data or perhaps save the data in another data set on disk or tape before you "zap" the file.

ZAP FILE command destroys files

The SPIRES command that erases the file is ZAP FILE, which names the file to be erased. It is quite easy to use (almost too easy, considering its drastic effects):

Once you confirm your desire, the ORVYL data sets of the file are quickly erased and the file is gone. Only the file owner can issue the ZAP FILE command for a file. [See 2.11.]

Remove other code for the file

After a ZAP FILE command, the file definition still exists in the FILEDEF subfile; if you no longer need it, you should remove it from there, or consider moving it to the BACKFILE subfile. If you don't do one or the other, the subfiles of the file will continue to show up in the list from the SHOW SUBFILES command.

Also get rid of code for the file from other system subfiles, such as FORMATS, RECDEF or EXTDEF, as part of your cleanup activities. [See 2.11 for the complete description of the procedure, which describes how to find and get rid of any other code.]

2  Building and Destroying Files and Record-Types

2.1  Introduction to SPIBILD

The next four parts of this manual are a "User Guide" to file management. In these sections you'll find descriptions of procedures you can use to accomplish various file management tasks.

Here in Part 2, the subject is the building and destroying of record-types, subfiles and files, the most important responsibilities of every SPIRES file manager.

The SPIBILD program

Many of the procedures described in this part of the manual require you to use a program called SPIBILD. The rest of this first chapter describes the SPIBILD program in general, introducing various commands and "sub-procedures" that may be useful to you as you follow the procedures in the remainder of Part 2.

Those remaining chapters of Part 2 are:

2.1.1  The SPIBILD Program

Processing a SPIRES File

SPIBILD is a part of SPIRES that builds the data sets containing a file's records. Most procedures that involve SPIBILD cause the file to be "processed". When you process a file, the records in the file's deferred queue -- that is, those that have been added or updated since the file was last processed -- are moved from the deferred queue to the tree. Also, any records with removal requests in the deferred queue are removed from the tree. Along the way, changes are made to the indexes as appropriate, by adding, updating and removing data from them to reflect the changes to the tree.

Processing a file can be done in SPIRES or SPIBILD, depending sometimes on specific needs but generally on personal preference. [See 2.2.2, 2.2.3 for information about processing a file in SPIRES.]

You can process your own files anytime you want, or you can allow other people to process them. Or, if you want, SPIBILD will process your files automatically each night, saving you the trouble.

Other SPIBILD Tasks

You can use SPIBILD as part of many other tasks too, which are discussed later in Part 2:

 - to add, update or remove many records at once
 - to build new indexes for a subfile that already exists
 - to remove indexes you no longer need
 - to remove one subfile in a file when you no longer need  it  but  need  to  keep  the  file
 - to rebuild indexes, or entire subfiles, or entire files

About this Chapter

The rest of Chapter 2.1 discusses aspects of SPIBILD in general, which relate to most of the tasks discussed in the rest of Part 2. The remaining sections of this chapter are:

2.1.2  Using Interactive SPIBILD

When you want to process one of your files yourself, you must use SPIBILD, the program that builds the data sets containing SPIRES records.

The SPIBILD command

The online version of SPIBILD is an ORVYL program invoked by the SPIBILD command:

where the most popular optional parameters are:

 - QUIET, to suppress the welcome message, and
 - WYLBUR, to request that the command SET WYLBUR be issued automatically

Separate them with a comma or blank to request both. See Part 6 for the full syntax, or online, [EXPLAIN SPIBILD COMMAND.]

For example:

In this example, 87.06 is the release number, indicating that this version of SPIBILD was released in June of 1987.

Unlike SPIRES, SPIBILD has no "uplow" setting, because none of its commands are case-sensitive. [SPIRES has an uplow setting because the data the user types in some commands, such as DISPLAY or FIND, may be case-sensitive.] You can type commands in uppercase, lowercase or any mixture you like.

Once you have invoked SPIBILD, you can issue any SPIBILD command, or any WYLBUR, ORVYL or MILTEN commands. To use SPIRES commands, or those of any other SPIRES processor, you must invoke that processor, leaving SPIBILD.

Leaving SPIBILD

The EXIT command

To leave SPIBILD, issue the EXIT command:

where the QUIET option suppresses the exit message from SPIBILD.

The EXIT command takes you back to WYLBUR:

You can also leave SPIBILD by issuing a command to invoke another SPIRES processor, e.g., "SPIRES".

Details about the SPIBILD Environment

Between the SPIBILD and EXIT commands, you will probably issue several SPIBILD commands, as part of procedures discussed in other chapters of Part 2. Other SPIBILD commands useful in many different SPIBILD procedures are discussed later in this chapter. [See 2.1.5.]

The remainder of this section discusses features and commands that the SPIBILD environment has in common with SPIRES. Most are fully documented in chapter A.2 of the SPIRES reference manual "Searching and Updating", with specific sections referenced below. In summary, those that affect SPIBILD are:

Getting online help: The EXPLAIN command

 - The EXPLAIN command is available in SPIBILD, for explanations of  SPIRES  terms  and  error
 messages. Subfile explanations, available in SPIRES, are not available in SPIBILD, however.
 Nor  are  the  SHOW  EXAMPLE  and SHOW SYNTAX commands.  You can put an explanation in your
 active file by preceding the command with the IN ACTIVE prefix (see below).

Examining variable values: The SHOW EVAL command

 - The SHOW EVAL command is available in SPIBILD as well as in SPIRES, although many variables
 (those involving search results or protocol execution, for example) are not available to be
 evaluated. The command (documented in the "SPIRES Protocols" manual)  could  also
 be  used  for  on-the-spot  computations  in  SPIBILD or for evaluation of static variables
 before and after SPIBILD processing. One likely use of SHOW EVAL in  SPIBILD  might  be  to
 determine  which  version  of  the  program (i.e., Test or Public) you are currently using:
       -? show eval $version
       87.01.27 PUBLIC System
       -?

Abbreviations

 - Unless otherwise mentioned in their documentation, command verbs and their options  can  be
 abbreviated  to  their  first  three  characters.  [See A.2.7 of "SPIRES Searching and
 Updating".]

Altering command control: The SET WYLBUR command

 - The SET WYLBUR and SET NOWYLBUR commands can be issued to specify whether WYLBUR or SPIBILD
 examines commands first, passing unrecognized commands to the other  program.  WYLBUR  Exec
 files will not execute in SPIBILD unless SET WYLBUR is in effect.  Extended Exec files will
 execute  in  SPIBILD,  however,  regardless of SET WYLBUR or SET NOWYLBUR.  [See A.2.5.1 of
 "SPIRES Searching and Updating".]

Using the active file: The IN ACTIVE prefix

 - You may use the WYLBUR active file during various SPIBILD activities. For example,  it  may
 contain  data  that  you  want  to merge into a SPIRES file. The IN ACTIVE prefix, with the
 CLEAR and CONTINUE options, may be used on some commands to  direct  data  to  your  active
 file.  [See  A.2.9  of  "SPIRES  Searching and Updating".] The following commands
 relating to the active file are applicable in SPIBILD:

Automatic listing of the active file: SET LIST

 - SET LIST and SET NOLIST  --  The  default  is  LIST,  which  means  that  SPIBILD  will
 automatically  list  your active file at the end of any command that puts data into it.
 [See A.2.9.2 of "SPIRES Searching and Updating".]

Automatic clearing of the active file: SET CLEAR

 - SET CLEAR and SET NOCLEAR -- The default is NOCLEAR, which means that SPIBILD will  not
 automatically clear your active file if you use the IN ACTIVE prefix without either the
 CLEAR  or  CONTINUE option. [See A.2.9.3 of "SPIRES Searching and Updating".]

Setting line length: SET LENGTH

 - SET LENGTH -- This command can be used to change the default line  length  of  72,  the
 value   of   the   $LENGTH   variable.  [See  A.2.9.4  of  "SPIRES  Searching  and
 Updating".]

The ATTN/BREAK key

 - The ATTN/BREAK key has several uses in SPIBILD:
 - If you press the key in response to a command prompt, WYLBUR will  either  reprompt  or
 place you in Collect mode.
 - If you press the key after you have begun typing a command but before you press Return,
 the command you were typing is ignored,  and  you  are  prompted  for  a  new  command.
 - If you press the key while SPIBILD is executing  a  command,  you  will  interrupt  the
 processing.   SPIBILD  may  ask you if it can continue the processing, or it may simply
 stop altogether.  Often in this case, if you press  the  key  twice  in  succession,  a
 session  break  will  result.  Type  GO  if  you  want the SPIBILD session to continue.
 - If SPIBILD asks a question (such as "-Continue  processing?"),  pressing  the
 ATTN/BREAK key is equivalent to answering NO.

Monitoring I/O

 - You can see how many read and write operations have taken place  involving  blocks  of  the
 files  involved  in a SPIBILD task by issuing the SET FILE COUNTS command prior to the main
 commands of the task in SPIBILD.  SET FILE COUNTS is quite similar in effect  to  the  SHOW
 FILE  COUNTS command in SPIRES, though it does have some significant differences.  See Part
 6 for more information, or online, [EXPLAIN SET FILE COUNTS COMMAND.]

2.1.3  Using SPIBILD in Batch

You can use SPIBILD through the batch system as well as interactively. That allows you to process SPIRES files, merge records into a subfile, or execute any other SPIBILD task by submitting jobs to the batch system.

There are several reasons why you might want to use SPIBILD in batch:

 - 1) The input data you have for a SPIRES file is on tape or is too  large  to  fit  in  your
 active file.  [If the data is in standard SPIRES format, you can also use the BATCH command
 in  SPIRES.  [See  the  SPIRES  manual  "Searching  and Updating", section D.8.]]
 - 2) You want the SPIBILD work to occur at a time when you won't be logged on (in the  middle
 of  the  night,  for  instance),  so  you  submit  a batch job to run for you at that time.
 - 3) You don't want to tie up your terminal while your SPIBILD work is being done.

Two batch programs available

You can use either of two different programs when you want to use SPIBILD in batch:

Choosing between BATWYL and batch SPIBILD

The next two sections will provide detail about using these two programs for SPIBILD work and will give you a clearer picture of which to use in particular situations. [See 2.1.3.1, 2.1.3.2.] In general, more people choose to use BATWYL because of its flexibility, but here are some guidelines for choosing between them:

 - If you are using SPIBILD to merge some data into a SPIRES file from a tape,  you  must  use
 batch SPIBILD.
 - If the input file is too large to fit into your WYLBUR active file or is in a  format  that
 cannot be imported into your active file, you must use batch SPIBILD.
 - If you need return codes from the job in the event of  SPIBILD  problems,  you  should  use
 batch SPIBILD.
 - If you need to use WYLBUR Exec files, or to leave SPIBILD during the job,  you  should  use
 BATWYL.

2.1.3.1  Using the BATWYL Program for Access to SPIBILD

A BATWYL job is basically an interactive session controlled from the batch processor. Unlike some batch programs where writing the appropriate JCL can be very tedious and difficult, BATWYL uses very simple JCL. The challenge in constructing BATWYL jobs is to anticipate every prompt that BATWYL will send to the "virtual terminal" and include an appropriate response in the command stream.

Complete details on using BATWYL can be found in the document "Using BATWYL", available through the PUBLISH command.

JCL for BATWYL jobs

Use the following JCL to run BATWYL jobs:

The statements will be formally described below.

A sample BATWYL job

First though, here's an example of a simple BATWYL job that does some SPIBILD work (the numbers on the left are not part of the input but are used for reference in the explanation that follows):

    1.   //  JOB
    2.   //  EXEC BATWYL,PARM='ATTN=*'
    3.   //SYSIN  DD  *
    4.   call spires
    5.   select manual
    6.   for adds
    7.   in active display all
    8.   print forms=3hol duplex
    9.   in active show subfile transactions
   10.   ok
   11.   print forms=3hol duplex
   12.   exit
   13.   call spibild
   14.   process gq.doc.manual
   15.   logoff

In the above example, following the JCL (lines 1-3), the job calls SPIRES (4), places all the newly added records in the active file (6,7), and then prints them (8). Next, it puts the transactions display into the active file for printing (9-11). The "ok" (10) is not a command, but is the response to the SPIRES prompt "OK to clear?" that would result from the preceding command. (More likely, you would add the CLEAR option to the IN ACTIVE prefix on the SHOW SUBFILE TRANSACTIONS command, to avoid the prompt altogether.)

Next the job calls SPIBILD, processes the file, and logs off, ending the job (13-15).

The BATWYL JCL statements

Here are descriptions of the JCL statements you use to run BATWYL:

You must begin all jobs with the JOB statement. It may be as simple as shown, or may include a job name, account number, bin number, etc. [See the document "Batch Processing" for more information.] Be sure to type blanks as shown in JCL, such as at least one before the word JOB.

The EXEC statement calls the BATWYL program. The PARM option shown here allows you to create an ATTN/BREAK response by typing three asterisks at the end of a command line. The character doesn't have to be an asterisk; whatever character you choose should be specified in the ATTN clause instead. If you will not need to simulate an ATTN/BREAK, you do not need the PARM option (nor the comma preceding it). Other options are also available.

The SYSIN DD statement shown above indicates that the commands to be executed by WYLBUR follow this statement. The document "Using BATWYL" describes how you can tell the program that the commands are stored elsewhere. If you add the DCB option shown, you raise the allowed length of commands from 80 to 240 characters. Note that SPIBILD commands may not exceed 160 characters, but WYLBUR and Milten commands can be as long as 235 characters.

Here you list the commands you want executed. If commands exceed 80 characters in length, be sure to add the DCB option to the SYSIN DD statement shown above. Note that SPIBILD commands absolutely may not exceed 160 characters.

All interactive commands are available, with the exception of those that invoke full-screen processes, such as Prism or full-screen SPIRES applications.

To use SPIBILD, you must include a SPIBILD command. After that, all SPIBILD commands are available to you. Note, however, that command options limited to batch SPIBILD, such as the "ON ddname" option on the BATCH command, are unavailable, because you are using interactive SPIBILD through the batch system, not batch SPIBILD.

Finally, be aware that output lines longer than 133 characters will be truncated.

2.1.3.2  Using the Batch SPIBILD Program

Batch SPIBILD gives you direct access to the SPIBILD program through the batch system. You can, for instance, run jobs that process other user's files if you can do so yourself online. In addition, you may use your WYLBUR active file for input and output, and may issue WYLBUR commands such as DELETE or PRINT. However, there are some differences from online SPIBILD, and those restrictions are described in appropriate places throughout this manual and are summarized in this section.

JCL for Batch SPIBILD Jobs

Use the following JCL to run batch SPIBILD jobs. (Statements marked with an asterisk are optional, and are explained below.)

 1.    //  JOB
 2.    //  EXEC SPIBILD
*3.    //DDNAME DD DSN='WYL.gg.uuu.dsname',DISP=SHR,
*4.    //   VOL=SER=volume,UNIT=unit
*5.    //SYSIN  DD  *
 6.    ... spibild commands ...
 7.    LOGOFF

For general information about batch jobs and JCL, see the "Batch Processing" manual, available through the PUBLISH command. Detail information about each of these specific statements appears below.

A batch SPIBILD example

First, though, here is a sample job that uses batch SPIBILD:

This job batches records in the data set named in the DDNAME DD statement into the subfile MANUAL.

The batch SPIBILD JCL statements

Here are descriptions of the JCL statements you use to run batch SPIBILD:

You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number, output bin, region size, maximum number of output lines allowed (5000, by default), etc. See "Batch Processing" for more information.

Note that you never need to change the default region size, no matter how much SPIBILD work you are doing -- the region size of the job controls a small program that invokes interactive SPIBILD, which always has more than enough region for any SPIBILD task.

The EXEC statement calls the batch SPIBILD program. The PARM option is occasionally used to set special escape characters, or to request versions of SPIBILD other than the public one, such as internal test versions, if they are available to you. The parameters most commonly specified are:

If used, ESC and OUT need to appear in the PARM option before the other two.

ESC and OUT have the same effect in Batch SPIBILD as they do in the BATWYL program. Please see the document "Using BATWYL", Data Center document number 409, for more information.

Use the optional DDNAME statement when you want SPIBILD to use input data from a data set stored on tape or on disk. Generally speaking, it is easier for you to use the USE and GET commands in batch SPIBILD to bring data into your active file from stored data sets, but you must use the DDNAME statement if either of the following is true:

 - the input data is on tape;
 - the input data is not in a format that  can  be  brought  into  your  WYLBUR  active  file.

The data set may have a RECFM value of F or FB or U, with a fixed LRECL of up to 4090 bytes.

If the data set is cataloged, you can eliminate the second line of the DD statement shown above, as well as the comma at the end of the first line. Otherwise, specify the tape or disk volume number, and the unit TAPE or DISK. Be sure to include a /*SETUP card if you're using tapes.

The SYSIN statement shown above indicates that the commands to be executed by SPIBILD follow this statement. Thus, it should be the last JCL statement before the list of commands.

If you want, you can store the commands separately and then tell batch SPIBILD where to find them for execution. You do this by replacing the * in the SYSIN statement with the same type of values that follow the DD in the DDNAME statement above: the name of the data set (DSN) containing the commands, the disposition (DISP), and the VOL and UNIT information if the data set is not cataloged. The data set must be saved either in EDIT format, or at any fixed length up to LRECL=240.

If you need to issue commands longer than 80 characters (but no longer than 240) in your batch SPIBILD job, add the DCB option shown. Be aware that SPIBILD commands are limited to 160 characters, and WYLBUR and Milten commands are limited to 235 characters. (Remember too that you can abbreviate many command verbs and options, which might shrink long commands to fewer than 80 characters.)

Here you list the commands you want executed, each command consisting of one and only one line of input. (See the previous paragraph about commands longer than 80 characters.)

All SPIBILD commands are available and work as you would expect, with the following exceptions:

If a SPIBILD command would prompt you for input if you were in interactive SPIBILD, then batch SPIBILD will assume a response of ATTN (the ATTN/BREAK key).

If you omit the LOGOFF statement, batch SPIBILD will supply it for you, and give you a return code of 4.

Output line length limit

Though command lines may be as long as 150 characters, output from batch SPIBILD cannot exceed 133 characters per line. Lines longer than 133 characters will be truncated on output.

Other JCL cards

There are other lines of JCL you can add if you want, such as JOBPARM and SYSPRINT. See the "Batch Processing" manual for more information about them.

2.1.3.2a  (Condition Codes for Batch SPIBILD Jobs)

The following condition codes are supplied by SPIBILD if a job terminates abnormally:

2.1.4  Recovering from Problems in SPIBILD

Normally when you use SPIBILD or SPIRES for a file-processing task, the task is completed without problems or complications. However, problems that can cause a file-processing command to go awry do occasionally arise. For example, while you are waiting for a file-processing command to execute:

 - the system might fail;
 - you might accidentally disconnect your  terminal,  causing  the  system  to  log  you  off;
 - SPIBILD or SPIRES could encounter a serious problem with your  file,  causing  it  to  stop
 executing the command immediately.

The file is usually unharmed

The results of these problems vary extensively, depending on the command being executed, on the precise point when the processing stopped, etc. In most cases, the file has suffered no ill effects at all; in other cases, there may be data integrity problems that SPIBILD or SPIRES needs to repair.

For example, suppose the system fails while SPIBILD is updating a subfile's indexes with some data from newly added records. At the time of the failure, half the indexes have been updated, but not the other half. As a result, the indexes are not reliable, and they won't be until the problem is repaired.

Accidents cause problems too

Occasionally your own programming error can cause problems too. For instance, changing a file definition carelessly can sometimes cause serious data integrity problems the next time the file is processed, requiring you to "back up" to a previous version of the file, stored prior to the change, to restore your file to a usable condition.

Common SPIBILD processing problems

The next section describes the most common problems encountered during SPIBILD or SPIRES file-processing, with suggested methods for handling them. [See 2.1.4.1.]

Invariably, a result of interrupted file-processing commands is that the NO JOBGEN flag is set, blocking all file transactions (such as adds and removes) till it is reset.

The RECOVER/PROCESS procedure

In almost all situations where file-processing is interrupted, the file is fine; if you return to SPIBILD or SPIRES and issue the PROCESS command, the problem will be cleared up, with the JOBGEN flag reset so that file updates can continue. If you do not want to spend the time or money on a complete processing of the file, you can issue the RECOVER command instead, which does a "mini-process" of the file: just enough work to clear up any minor problems and reset the JOBGEN flag. The complete RECOVER/PROCESS command procedure for file recovery is described in the next section. [See 2.1.4.2.]

The SET JOBGEN procedure

Another technique, used very infrequently, is available to files whose file-processing is automatically handled by JOBGEN. [See 2.2.1.] The SET JOBGEN command is at the heart of that procedure. It is used less frequently because all it does is reset the JOBGEN flag, so that file updates are allowed again. That allows the file to be processed again via JOBGEN, starting that night.

Unfortunately, the SET JOBGEN command can create serious data integrity problems, and should not be used unless there will be no file updates between the time you set the JOBGEN flag and the time the JOBGEN-created batch job processes the file. See Part 6 for more information; online, [EXPLAIN SET JOBGEN COMMAND.]

Backup copies of SPIRES files

In extremely rare cases, such severe problems can occur that the file is considered destroyed. When a problem like that occurs, it is more likely due to system problems (such as a disk-drive failure) than to a SPIBILD problem. But in any case, backup copies of the file or parts of the file are a vital part of SPIRES applications.

Some basic backup services are provided by your computer center, which are always adequate if JOBGEN handles your file-processing needs. However, you may choose to handle a file's backup yourself. A discussion of the available services appears later in this section. [See 2.1.4.3.]

2.1.4.1  Common Problems Encountered during File-Processing Tasks

Though most file-processing tasks in SPIBILD and SPIRES run quite smoothly, this section discusses the most frequently encountered problems (some described by their system error numbers), with suggested solutions. Remember that the EXPLAIN command can be used in SPIBILD as well as SPIRES to get information about system messages and error codes.

The processing is interrupted and doesn't complete.

This problem could be caused by a system crash or by your terminal becoming disconnected. Note that although the file will be in a somewhat unstable state, users can still use the file for searching, though probably not for updating, until you can execute one of the two recovery procedures to stabilize it completely. In any case, the file is not damaged, and the recovery procedures will return it to normal.

To recover the file, follow the RECOVER/PROCESS procedure. [See 2.1.4.2.]

S36: The SPIRES file requested does not exist.

You may have misspelled the name of the file. If that's not the case, return to SPIRES, select a subfile of the file, and check the value of the $FILENAME system variable (SHOW EVAL $FILENAME) to be sure you were trying to process the correct file.

Another possibility is that you are trying to use a file that hasn't been compiled yet. That sometimes happens when you are rebuilding a file: you issue the ZAP FILE command to destroy the old file and then go to SPIBILD to rebuild it, without first compiling it again. The file doesn't exist till it is compiled.

S56: The file is busy, not currently available.

This error indicates that other people have subfiles of the file selected. You may want to use the SET SEARCH/SET SHARE procedure to circumvent this problem. [See 2.1.5.1.] Note: SPIRES automatically "sets SEARCH" when you issue the PROCESS command in SPIRES, so S56 errors are not a problem there.

S116: You don't have WRITE access to an ORVYL data set.

To process someone else's file, you must have WRITE access to all the ORVYL data sets of the file except the MSTR, as well as WRITE access to the SYSTEM.CHKPT data set on the file owner's account. If you don't have the proper access, the command will fail with an S116 error. The file owner can correct the problem by issuing SET PERMIT commands for each of the data sets.

Other Problems

This list is not comprehensive -- other problems can stop file-processing too. In general, the error messages, combined with information available through the EXPLAIN command, should lead you to the problem and suggest a solution. Of course, file-processing problems should be taken seriously, so if you need any help, be sure to see your SPIRES consultant as soon as possible.

2.1.4.2  Using the RECOVER/PROCESS Procedure for SPIBILD Problems

When a SPIBILD command finishes abnormally, it may leave the file in an unstable state. It will definitely leave the NO JOBGEN flag set, meaning that automatic processing of the file by JOBGEN will not occur, and no file updates are allowed.

The RECOVER/PROCESS procedure is a general one for solving such problems; special factors for particular SPIBILD tasks that don't get completed are discussed under individual procedures.

Step 1: SPIBILD

If you aren't already in SPIBILD, issue the SPIBILD command:

Note: If you use the PROCESS command (see step 2), you can do this procedure in SPIRES; however, RECOVER and SPIBILD commands that process the file other than PROCESS are available only in SPIBILD.

Step 2: Issue the RECOVER or PROCESS command.

in SPIBILD, depending on your preference (see below), issue either the RECOVER or PROCESS command:

Note that the PROCESS command in this context means any SPIBILD command that processes the file, such as the BATCH, MERGE and ESTABLISH commands -- they will all have the same effect in terms of recovery.

Also note that if you are processing the file in SPIRES, you follow a slightly different procedure here at step 2: first, SELECT a subfile of the file and then issue the PROCESS command in this form:

NOWARN tells SPIRES not to prompt you for confirmation that you want to process the file.

It may seem odd to try recovering from the problem before determining just what the problem is, but in fact, you can easily waste a lot of time trying to understand the problem, only to discover that you can solve it simply by issuing either the RECOVER or PROCESS command. Since issuing either of these commands cannot make the situation worse, but stands an excellent chance of fixing it, giving that a try is usually worthwhile.

Here are some notes to help you choose between PROCESS and RECOVER:

The PROCESS command:

 - tries to completely process the file.  If it cleans up the problem, then the deferred queue
 will be empty when the command is completed.
 - may execute for a long time if the deferred queue is large.
 - can be done in SPIRES, e.g., under protocol control.

The RECOVER command:

 - does only a limited amount of work.  (Technically speaking, it processes two  pass  stacks'
 worth of data.)  For most files, where the deferred queue is not large, that will empty the
 deferred  queue,  making  RECOVER equivalent to PROCESS.  But for files with large deferred
 queues, the RECOVER command will leave records in the deferred queue, waiting for the  next
 processing of the file.
 - will not execute for very long.  Thus, it may be preferable to PROCESS in  situations  when
 you  need  to  restore  the  JOBGEN  flag  as  quickly as possible so that users can resume
 updating records in the file.
 - is available only in SPIBILD.

RECOVER command example

The sample session below shows the successful execution of the RECOVER command:

If no errors are indicated, then the recovery was successful, users can begin using your file again, and the procedure is complete.

Note: If SPIBILD would create more than two pass stacks' worth of data, then it quits the RECOVER command without processing the file, and exits SPIBILD. Hence, in the example above, there was only enough data in the DEFQ for two pass stacks; had there been more, the message "Processed: ..." and those that follow it would have been replaced by "Exiting SPIBILD". Though the file wouldn't have been completely processed, it would be back to a useable state again.

However, if errors are indicated, as in the example below, you will need to continue to step 3:

Notice that failure invariably invokes an automatic exit from SPIBILD.

Step 3: Determine and fix the cause of the problem, if needed.

You can find out what went wrong by examining the display from the SPIBILD session, in particular, by examining the error messages and using the EXPLAIN command to find out details about specific errors. The error explanations should help you determine whether you can fix the problems or whether you should contact your SPIRES consultant. In any event, more work will be required before the file can be processed properly.

2.1.4.3  Backup Copies of SPIRES Files

Everyone knows that having backup copies of computer files is a good idea, but the more reliable computer systems become, the less important it may seem. In fact, few of the thousands of SPIRES files created at Stanford have ever needed restoring from the system-created backup tapes due to system problems. But that fact begs the question -- history shows a number of occasions when file restoration has been necessary. In many cases, though, the problem was caused by a file owner accidentally issuing a destructive command (such as ZAP FILE) or making an unfortunate programming decision that destroys the data integrity of the file.

File backups are thus a crucial ingredient of SPIRES applications. File owners should be aware of what file backup is automatically done, what they themselves can do in addition, and what to do if they need to use the backup data sets to restore a SPIRES file.

Forms of file backup

Backup of SPIRES files is available in four forms:

If you think you need to restore a SPIRES file using the backup files created by either of those two methods, contact your SPIRES consultant for assistance.

2.1.4.4  Making Backup Copies of a File with the ORVCOPY Program

The batch program ORVCOPY, the ORVYL file dump/restore utility program, lets you back up the data sets of a SPIRES file to tape or disk and later restore them to the ORVYL file system. Besides the data itself, attributes of the file such as account-access permits are copied too.

This procedure describes the technique a file owner would use to make a backup copy of a SPIRES file stored on his or her account, storing it on tape. The reverse, restoring the copy back to the ORVYL file system, is covered briefly in the next section. [The document "ORVYL User's Guide" contains complete information about ORVCOPY, including information about storing the data on disk and dealing with other accounts.]

Note though that an ORVCOPY job can copy only data sets belonging to the account under which the job is run. Similarly, it can restore data sets only to the account under which it is run. But the dump job and the restore job can be run on separate accounts, providing a way in which data sets can be moved from one account to another. If you are moving data between accounts with ORVCOPY, be sure that all accounts have access to the tape or disk data sets to which the data is dumped.

Warning: if you use this procedure to move a SPIRES file from one account to another, the MSTR data set of the file, which contains the name of the file (including the owner's account number), must be changed as well. This change must be done by hand, as described at the end of the next section. [See 2.1.4.5.]

Step 1: Create the ORVCOPY JCL.

Use the following JCL for an ORVCOPY job to copy data sets:

 1.    //  JOB  [TIME=n]
 2.    /*SETUP  T  OUTPUT=tapeid
 3.    //  EXEC  ORVCOPY
 4.    //OUTPUT  DD  DSN=WYL.gg.uuu.name,DISP=(NEW,CATLG),
 5.    //    UNIT=TAPE[,VOL=SER=volume]
 6.    dump commands

For general information about batch jobs and JCL, see the "Batch Processing" manual, available from Document Sales. Specific information about each statement in the ORVCOPY JCL appears below, and a sample job follow that.

The ORVCOPY JCL Statements

Below are descriptions of the JCL statements you'd use to run an ORVCOPY job to copy ORVYL data sets to a new rental tape or empty tape belonging to you:

You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number or output bin. See "Batch Processing" for more information.

Depending on how many blocks you are copying, you may need to add the TIME parameter to request more than the default 30 CPU seconds. As a rule of thumb, figure about 450 blocks can be copied per CPU second.

The SETUP statement tells the system that the job requires tapes to be mounted. There are many different values for "tapeid", but the most common are:

The EXEC statement calls the ORVCOPY program.

4.    //OUTPUT  DD  DSN=WYL.gg.uuu.name,DISP=(NEW,CATLG),
5.    //    UNIT=TAPE[,VOL=SER=volume]

The OUTPUT DD statement tells the program the destination for the copy of the ORVYL data sets. Be sure the name you type for the DSN parameter is unique to your account; it must follow the naming conventions for OS data sets. (If you can't give it a unique name, replace CATLG with KEEP in the DISP parameter.) If you use your own tape rather than a scratch tape, include the VOL=SER parameter with the volser number, as given in the SETUP statement earlier.

The DUMP command tells ORVCOPY what data sets to copy. You can include as many DUMP commands as you need. A single command can copy from one to all of your ORVYL data sets, depending on the option specified:

where "option" is one of the following:

NAME=filename[,RENAME=name]  - copy the single named ORVYL data set,
                                giving the copy a new name if the
                                RENAME option is specified

LIKE=string                  - copy all data sets with names beginning
                                with the given string

RANGE=name1,name2            - copy data sets "name1" and "name2", and
                                all those whose names fall between them
                                in alphanumeric order; "name1" must come
                                before "name2" in alphanumeric order.

ALL                          - copy all data sets belonging to your account

Do not include the "ORV.gg.uuu." file prefix in any of these options. It is not necessary since you can only copy files on the account under which the job is run.

Be sure to put a blank between the DUMP command and the option you use; however, no other blanks should appear in the DUMP command.

An ORVCOPY Example

Here is a sample job run by account GQ.DOC that uses ORVCOPY to copy the data sets of the file GQ.DOC.ERRATA:

Step 2: Run the job.

When you want, issue the RUN command to submit the job. After it runs, you should examine the output to get the assigned tape number if you asked for a rental tape. You will need it later if you need to restore the files. [See 2.1.4.5.]

Again, see the document "Batch Processing" for information on running jobs, fetching the output, releasing tapes, etc.

2.1.4.5  Restoring a File with the ORVCOPY Program

In the previous section, the procedure for creating a copy of the ORVYL data sets comprising a SPIRES file using ORVCOPY was discussed. In this section, its less frequently used complement will be covered.

If you need to restore a SPIRES file to a version you created using ORVCOPY, you also use ORVCOPY to get it back. Besides the data itself, attributes of each data set such as account-access permits are reset when ORVCOPY restores a file.

This procedure describes the technique a file owner would use to restore a SPIRES file stored on his or her account, using a copy stored on tape. The reverse, creating the copy, was discussed in the previous section. See the document "ORVYL User's Guide" for full information about ORVCOPY.

Step 1: Create the ORVCOPY JCL.

Use the following JCL for an ORVCOPY job to restore data sets:

 1.    //  JOB  [TIME=n]
 2.    /*SETUP  T  INPUT=CAT
 3.    //  EXEC  ORVCOPY
 4.    //INPUT  DD  DSN=WYL.gg.uuu.name,DISP=OLD
 5.    restore commands

For general information about batch jobs and JCL, see the "Batch Processing" manual, available from Document Sales. Specific information about each of the statements in the ORVCOPY JCL appears below, and some sample jobs follow that.

The ORVCOPY JCL Statements

Below are descriptions of the JCL statements you'd use to run an ORVCOPY job to restore ORVYL data sets from a tape.

You must begin all jobs with the JOB statement. It can be as simple as shown above (Remember to include blanks where shown -- blanks are very significant in JCL.) or may have other options, such as an account number or output bin. See "Batch Processing" for more information.

Depending on how many blocks you are restoring, you may need to add the TIME parameter to request more than the default 30 CPU seconds. As a rule of thumb, figure about 450 blocks can be restored per CPU second.

The SETUP statement tells the system that the job requires tapes to be mounted. Specify the CAT value if the data set named in line 4 below is cataloged -- the system will find the tape volume serial number in the catalog. If the data set is not cataloged (e.g., it wasn't uniquely named), replace CAT with the volser number of the tape, which you will also need to add to line 4 (see below).

The EXEC statement calls the ORVCOPY program.

4.    //INPUT  DD  DSN=WYL.gg.uuu.name,DISP=OLD

The INPUT DD statement tells the program the source for the copy of the ORVYL data sets. If the data set is not cataloged, include the UNIT=TAPE parameter and the VOL=SER parameter with the volser number.

The RESTORE command tells ORVCOPY what data sets to restore. You can include as many RESTORE commands as you need. A single command can restore from one to all of the data sets in the INPUT DD data set, depending on the option specified:

where "option" is one of the following:

NAME=filename[,RENAME=name]  - restore the single named ORVYL data
                                set, giving the copy a new name if
                                the RENAME option is included

LIKE=string                  - restore all data sets with names
                                beginning with the given string

RANGE=name1,name2            - restore data sets "name1" and "name2", and
                                all those whose names fall between them
                                in alphanumeric order; "name1" must come
                                before "name2" in alphanumeric order.

ALL                          - restore all the data sets on the tape

Don't include the "ORV.gg.uuu." file prefix in any of these options; also, don't put blanks within the option (for instance, between LIKE and the equals sign).

Include the REPLACE option if you want ORVCOPY to replace existing data sets with the same names as those being restored. If you leave it off, ORVCOPY will only restore files that don't currently exist in ORVYL.

An ORVCOPY Example

Below is a sample job run by account GQ.DOC that uses ORVCOPY to restore the data sets of the file GQ.DOC.ERRATA. It is the counterpart to the example in the previous section.

Step 2: Run the job.

When you want, issue the RUN command to submit the job. After it runs, check the file-creation date of each data set by adding the DATE option to the SHOW FILES command. See Part 6, or online, [EXPLAIN SHOW FILES COMMAND.] Those that were restored will have the date they were restored listed for "CR", the creation date.

See the document "Batch Processing" for information on running jobs, fetching the output, releasing tapes, etc.

Moving a SPIRES file to another account

As mentioned before, this procedure can be used to move data sets from one account to another, which means you could use it to move an entire SPIRES file from one account to another. Generally speaking, it is preferable (and cheaper) to do that by following the procedure described later in this manual. [See 2.10.]

But if you have used this procedure to move a SPIRES file from one account to another, the MSTR data set of the file, which contains the name of the file (including the owner's account number), must be changed as well. This change is best done with a special program called "fixmstr", because "recompiling" the file will not change the account to the correct one, and, in fact, SPIRES will not even allow you to recompile the file since the owning account number listed in the MSTR does not match the account the file is stored on. To change the account number and/or the file's name in the MSTR:

Note that you must call fixmstr from the account of the new file owner, and filename is just the file's name without the .MSTR suffix.

2.1.4.6  Copying a File's Deferred Queue from Another Account

Application designers constructing complex applications usually want to make sure there are adequate ways to back out any changes the application has made to files when problems occur. Frequently, doing the application's work using transaction groups or getting rid of changes with the ZAP DEFQ command is an adequate solution.

On the other hand, another method is to make backup copies of as much as possible before starting to make a complicated series of changes. In most cases, copying the entire file would be overkill, though that is certainly possible, thanks to the COPY FILE command. [See 2.10.] Another solution is to copy only the data sets of the file that would be changed by the impending transactions, in particular, the DEFQ data set. [If the file has immediate indexes that would be changed, you would either want to copy them first as well, somehow, or use some other method.]

The COPY DEFQ command lets you copy the deferred queue data set of a file that doesn't belong to you, if you have been granted Copy access to that file by the file owner. [See 2.10, step 2 to see how to do that.]

Restoring the deferred queue, if necessary, must be done by the file owner, of course. This can be accomplished either indirectly, by dumping the data set to tape and restoring it on the file owner's account with the ORVCOPY program [See 2.1.4.4, 2.1.4.5.] or directly, by using ORVYL's SET CLP, ERASE and CREATE commands. See your SPIRES consultant if you need help restoring the deferred queue.

Note that you cannot use those ORVYL commands to make the original copy from the file owner's account because you can't issue the SET CLP command to get copy access to someone else's SPIRES files; only the file owner can do that. (Remember, not being able to run the application on the file owner's account was one of our original assumptions.) The COPY DEFQ command circumvents that problem.

Step 1: Enter SPIRES.

The COPY DEFQ command is available only in SPIRES. Issue one of those commands online, or in a BATWYL job:

In a batch SPIRES job, you are already in SPIRES, so you omit this command from the command stream.

Step 2: Issue the COPY DEFQ command.

The COPY DEFQ command makes a copy of the deferred queue data set of the named file, saving it as an ORVYL data set on your account:

It must be issued from the copying account, NOT from the file owner's account. As it begins copying the data set, SPIRES displays a message showing how many blocks must be copied:

See Part 6, or online EXPLAIN COPY DEFQ COMMAND, if you have any problems or need more information about the command.

Getting rid of the copied DEFQ

What you do with the copied DEFQ is up to you. Some of the possibilities were discussed above. But chances are good that your application will work successfully, and you will not need the copy. Assuming that's true, at some point, you will want to get rid of it. (WARNING: Be sure your previous copy doesn't still exist the next time you use the COPY DEFQ command for the file -- if it does, the command will fail with an S40 error.)

To discard the copied DEFQ, issue these two commands:

where "filename" is the name of the copied file, without any account prefix, unless you'd like to include your own, preceded by "ORV.".

2.1.5  Controlling the SPIBILD Environment

Earlier in this chapter, in section 2.1.2, we discussed some details about the SPIBILD environment, such as the command prompt, the EXPLAIN command, the ATTN/BREAK key, etc. That discussion centered around many SPIRES commands and concepts that carry over into SPIBILD. Here in section 2.1.5 and its subsections, we will consider some other aspects of using SPIBILD, aspects that either differ considerably from SPIRES or have no SPIRES equivalent.

For example, a problem arises when you want to process a file in SPIBILD when it is selected by other users. SPIBILD will not process the file but will instead give you an error message (S56).

SPIBILD has several commands that themselves do little or no work but which affect the way the primary SPIBILD commands work. For instance, the SET SEARCH command does nothing at the time it is issued, but when you issue a subsequent PROCESS command, it lets SPIBILD process the file even though users have the subfile selected.

Problems such as that, and the commands that solve them, are discussed in this section. Specific details about each command, including its syntax, appear at the end of the manual.

Note: All these commands must be issued before the command they are meant to affect -- they establish conditions under which the processing command (e.g., BATCH) is to operate. Also, some of the conditions are cleared after the processing command executes and must be reset if further SPIBILD processing under those conditions is desired. Be sure to see the detail section for each command if you need that information.

Subsections of section 2.1.5

Specifically, here are the topics of the subsections of section 2.1.5:

2.1.5.1  Using the SET SEARCH/SET SHARE Procedure to Share File Access

You cannot do any SPIBILD task of consequence to a file when it is being used by other accounts, unless you issue the SET SEARCH or SET SHARE command first:

File users could be affected by processing

How can SPIBILD tell you, the file owner, that your own file is not available to you? The reason is that the processing activity can interfere with the current users' activities, causing their terminal sessions to be locked up and invalidating their search results. Hence, SPIBILD protects them by blocking the processing commands till the file is free. [If your file has subfile logging in effect, you may be able to determine who is currently using your file. [See Part 6 for PERFORM USER LIST; online, use the EXPLAIN command.] Additionally, ORVYL can provide the information to you through the command "SHOW VFILES LIKE ORV.gg.uuu.filename". Type HELP SHOW VFILES for information about that command.]

However, times arise when it is imperative for you to process a file even though it is in use by others. The SET SEARCH and SET SHARE commands tell SPIBILD (and FASTBILD) to let your processing commands through. (Note: This problem does not occur when you process a file in SPIRES; SET SEARCH, explained below, is automatically in effect.)

SET SEARCH permits searching during processing

In addition, the SET SEARCH command lets users continue to use the file for searching while the file is being processed. This is important for large files that might take a long time (say, over 10 minutes) for processing, especially files that must be available for searching 24 hours a day. [Note: no subfile logging can occur during the processing, so if you are logging subfile use, you will not receive any log entries for that period of time. [See 4.3.]]

SET SHARE locks users out during file processing

The SET SHARE command, on the other hand, simply locks users out of the file till the SPIBILD or FASTBILD processing is completed, which means that users' terminals may get locked till your processing is over. For users trying to use the file for searching, this can be quite irritating.

But oddly enough, for users entering data, particularly with a prompting format or in a full-screen environment such as Prism, it may actually be preferable to SET SHARE. Under SET SEARCH, a record-transaction command such as ADD will cause an error (usually S494) at the end of the processing, discarding the input; under SET SHARE, the terminal would lock up, but eventually the transaction would occur.

Hence, SET SEARCH is most often used for files whose processing takes a considerable amount of time (10 minutes or more). The processing is probably done overnight when users are not trying to enter data; but the file is still needed for searching by nighttime users. SET SEARCH is usually added to the command stream that handles the regular processing.

SET SHARE, on the other hand, tends to be used on an ad hoc basis, when a file's deferred queue is small so the processing won't take long. Any user whose terminal is locked by the processing would only be inconvenienced for a moment or two.

Step 1: Issue the SET SEARCH or SET SHARE command.

You can issue the SET SEARCH command in SPIBILD or FASTBILD if the file you want to work with has a private checkpoint data set. Any file having any immediate indexes has a private checkpoint data set; many others do too. [The SHOW FILES command can tell you whether a SPIRES file has a private checkpoint data set. Look for the data sets for the specific file (i.e., those whose names begin with the file name, such as "filename.RES"). If there is one called "filename.CKPT", that is the private checkpoint data set.]

If the file does not have a private checkpoint data set, issuing the SET SEARCH command will have precisely the same effect as SET SHARE:

But neither command has any effect on anyone till the file-processing command is issued.

If you change your mind after issuing either command, you can undo your request with the SET NOSEARCH or SET NOSHARE command. (In SPIRES, SET SEARCH is automatic and unsettable; SET NOSEARCH is not available to change it.)

Step 2: Issue the file-processing commands.

For example,

More details about these two commands can be found in Part 6. Online, EXPLAIN SET SEARCH COMMAND or SET SHARE COMMAND.

2.1.5.2  Using the Exception File Procedure to Repair Bad Input

When input data causes errors, SPIRES programs generally report them to you so that you can correct them before trying to put the data into the subfile again. When input for multiple records is involved, locating the few problem records in the input file can be tedious, however. An "exception file", into which all the data for failed transactions is placed, makes the correction procedure much easier.

Exception files work with input in the standard SPIRES format or in a custom input format, as long as the format does not execute the HOLD Uproc. See the manual "SPIRES Formats" for information about input formats.

Step 1: Issue the SET EXCEPTION command.

The SET EXCEPTION command creates an empty ORVYL data set that the SPIRES program will use to hold any bad data found by the processing command.

Here is the start of a SPIBILD session in which data will be added to a subfile using the INPUT BATCH command:

If you change your mind and decide not to use an exception file, you can issue the CLEAR EXCEPTION command at this point. Remember, however, that an empty ORVYL data set has been created; step 5 talks about how to erase it.

Step 2: Follow the procedure that adds the data into the subfile.

Issue the commands that create the subfile transactions, such as:

During the processing of the input command, where data is coming from the active file or some other external file (e.g., from a tape), any data that causes a serious-level error, which prevents that record transaction from succeeding, is placed in the exception file.

The exception file will not work properly if lines of the input file exceed 235 characters, the maximum WYLBUR line length. That problem could arise only in batch SPIBILD and FASTBILD, in which case the lines are truncated.

Another significant limit is that no single record in the input file can exceed 80 blocks (about 160K bytes); that is larger than a record can be in SPIRES, but the limit could be significant if the input data for the record contains data that is discarded during the input process, perhaps by the input format.

Example

The example below, continuing the SPIBILD session started above, begins with a SET MESSAGES command that suppresses messages about individual records that succeed in getting into the subfile; only the failures will get individual messages. [See 2.1.5.3.] In this simple example, we will batch two records into the subfile, the second of which contains an error:

It is useful to have a listing of the errors, especially if there are many of them; the line numbers are particularly useful. You might want to consider running a batch SPIBILD job so that the job's output will have the error messages in it, which you can print and keep by your side as you make corrections to the data.

On the other hand, you might want a summary of the number and kinds of errors that occurred during the processing. To display this information, which is accumulated by SPIRES and SPIBILD, you can use the SHOW ERROR COUNTS command. See Part 6, or online, [EXPLAIN SHOW ERROR COUNTS COMMAND.]

Be aware that when the processing command ends, an automatic CLEAR EXCEPTION is executed. Thus, if you issue another processing command where you want an exception file, you should issue another SET EXCEPTION command. You cannot use the same exception file again, however, unless it is empty or you are willing to replace it with the new data.

Step 3: GET the exception file, correct the data that failed.

Issue the GET command to place the exception file data into your active file:

(GET is an ORVYL command, similar to WYLBUR's USE command.) You can then correct the data as necessary.

Exception file includes record separators, line numbers

All the input data for each rejected record is placed in the exception file, including any record separators. For example, if the input data is in the standard SPIRES format, rejected records would be put in the exception file with the "command" that preceded them (e.g., "ADD;" or "UPDATE 492;") and the extra semicolon at the end.

In addition, the original line numbering of the input data set is maintained. So, if the first rejected record is on lines 12 through 21 of the input data set, it will have the same line numbers in the exception file. [If several data sets at once are being input, the line numbers could get entangled. The fact that the line numbers are retained could, in fact, cause a problem, with the result that the GET command would not "get" the entire exception file. You'll know the problem occurred if ORVYL responds to the GET command with the warning message "Command does not replace". In that case, issue the GET command again, adding the RENUMBER option.]

The example continues:

If the exception file is so large that the GET command cannot place the entire file in your active file, contact your SPIRES consultant for help.

Step 4: Resubmit the corrected data.

Once you have corrected the data, you need to resubmit it for processing. How you do that depends on various factors. For instance, if you used FASTBILD to enter the records, you will not be able to use it for the corrections since FASTBILD requires that the record-type be empty when you start. (Of course, if every single record failed, the record-type is still empty, but in that case, the exception file did you no good.)

But if you used SPIBILD, you will likely use the same command to resubmit the corrections. If so, remember that you can set another exception file in case you have any more problems.

So, next in the example, we batch the corrected records into the subfile:

Step 5: Cleanup: ERASE the exception file.

When you are finished with an exception file, issue the ORVYL command ERASE to erase it.

2.1.5.3  Controlling Messages from SPIBILD

The SET MESSAGES command

You can control the type of messages that SPIRES, SPIBILD and FASTBILD display during a session with the SET MESSAGES command. In most file-management situations, the default setting for each program works fine. In fact, probably the most common use of the command is in protocols and formats, where SET MESSAGES is used to suppress system messages from SPIRES, allowing the application to create and display its own.

Its most common use in SPIBILD is to suppress information messages when you are updating a subfile (e.g., using the BATCH or MERGE commands in SPIBILD). By default, SPIBILD prints a line of information for each record processed:

The information about successful transactions can be interesting, but the information about failed transactions, which is usually more important, can easily get overlooked. Moreover, the information messages slow down the processing: each information line for a record requires SPIBILD to stop its current processing, create a message and request a terminal I/O to send you the information.

Try turning off the information messages

If you set the information messages to the "0" level from the default level of "2", SPIBILD still gives you the most important information as well as information about any failed transactions. The session above would look like this if information messages were turned off:

The only change is that SPIBILD does not display the information lines for the successful records.

Don't turn off other messages

Turning off the other types of messages (Serious, Warning) is not recommended for file-processing in SPIRES nor for SPIBILD and FASTBILD tasks, since serious problems could be masked. And, generally speaking, there is not much reason to set messages to any other level than 0 or 2 in SPIBILD and FASTBILD. See Part 6 for more information, or online, [EXPLAIN SET MESSAGES.]

2.1.5.4  Limiting the Indexes Affected by a SPIBILD Command

Occasionally, emergency situations may arise where you don't want to pass data to one or more indexes, or where you only want to pass data from one of the goal record-types in the file. [Production applications may have similar needs that are routine, but such applications are uncommon.] The commands described below, available only in SPIBILD, can be used as temporary measures to control the record-types that are passing data or those that are receiving data:

 - SET INDEX record-names. This  command  names  the  record-types  that  can  receive  passed
 information,  chosen  from  the  file's index record-types.  No passing will be done to any
 other record-types.  It and SET NOINDEX below affect only the primary work of  any  SPIBILD
 command, not any preliminary file-processing the command may do.
 - SET NOINDEX record-names. This command names the  record-types  that  are  not  to  receive
 passed  information  as  they  normally  would.   All other index record-types will receive
 passed data as usual.
 - SET PASS [n]. This command names the number (1 to 64) of the logical record-type that is to
 pass data, chosen from the file's goal record-types.  No other  goal  record-types  in  the
 file  will  pass any data.  If no number is specified, then all goal record-types will pass
 data as usual.  SET PASS by itself is usually issued to countermand a SET  NOPASS  command.
 This  command  and  SET  NOPASS  below  affect  all processing work of any SPIBILD command,
 including any preliminary file-processing the command may do.
 - SET NOPASS [n]. This command names the number (1 to 64) of the logical record-type that  is
 not  to  pass  data.  All other goal record-types will pass data as usual.  If no number is
 specified, no index passing will occur.  This command is  necessary  when  using  the  FAST
 BATCH  command  in  SPIBILD,  and also has uses when you know that updating activity hasn't
 caused changes  that  will  affect  the  indexes.   [See  2.4.3.2.]

Intended for use in solving data integrity problems, all four of these commands can make such problems worse if not used very carefully. You should probably discuss the problem with your SPIRES consultant before using any of these commands. See Part 6 for full details about each of these commands, or, online, use the EXPLAIN command.

The SET REINDEX command may also be useful.

[See 6.2.10.]

The NOPASS statement in the file definition

Note: Another way to prevent any passing at all during a processing operation is to code the NOPASS statement in the file definition. The "nopass" condition would stay in effect for all SPIBILD tasks until the NOPASS statement was removed from the definition. See the SPIRES manual "File Definition" for details.

2.1.5.5  Creating Output Reports During SPIBILD Processing

When you use the SPIBILD commands BATCH or MERGE for subfile transactions and you use an input format to handle the data, the format can simultaneously produce an output report, containing information about the processed records.

Technical coding information on how to create these versatile formats appears in the manual "SPIRES Formats", section B.16.3.

FILE area commonly used

To use them in SPIBILD, you need one or more "device services areas", areas of memory in the computer used to hold data for particular physical or logical devices. Most useful in this situation is the FILE area, which allows you to send the report to an ORVYL data set.

To use the FILE area in SPIBILD for report output, you issue the following commands before the BATCH or MERGE command in SPIBILD:

Step 1: DEFINE AREA areaname(rows,cols) ON FILE

This command, documented in full in the SPIRES manual "Device Services", defines an area on the FILE area. The name should be the same name given in the IN-AREA statement in the format definition. If you specified "IN-AREA = FILE", then this command is not necessary.

Step 2: ASSIGN AREA areaname TO orvyl.dataset.name [REPLACE]

In this command, you assign the area to an ORVYL data set -- that is where the report will end up. If the data set exists already, you can add the REPLACE option to the command.

Step 3: Continue the session, including BATCH or MERGE task.

Issue any other SPIBILD commands you want, including the SET FORMAT command if you have not set the format already. Then do your BATCH or MERGE task. Afterwards, before leaving SPIBILD or starting another task, continue to step 4.

Step 4: CLOSE AREA areaname

This command detaches the ORVYL data set, allowing you to issue a GET command ("GET orvyl.dataset.name") to examine the report.

Other useful commands

Other device services commands are available in SPIBILD, too. They are fully documented in the "Device Services" manual:

2.2  Moving Records into the Tree: Processing a File

Effects of processing a file

Processing a SPIRES file has three major effects:

 - 1) It moves records from the deferred queue to the tree.
 - 2) It updates indexes using the information in the deferred  queue  (except  for  immediate
 indexes, which change at the time of the record transaction).
 - 3) It empties the deferred queue, discarding overhead data kept for each transaction (e.g.,
 date, time, account doing transaction) and all but  the  latest  copy  of  records  in  the
 deferred queue.

The first effect is important because records in the deferred queue are handled a bit less efficiently than those in the tree. That's because many commands must scan both the deferred queue and the tree when there are records in the deferred queue; if the deferred queue is empty, they work faster. For example, if you were creating a report using all the records in a large file, it would probably be worth your while to create the report when the deferred queue is empty, right after it is processed.

Contents of Chapter 2.2

The specific contents of this chapter are:

2.2.1  Letting JOBGEN Handle Your File-Processing

Many file owners don't work directly with SPIBILD at all. They let a program called JOBGEN handle the processing of their files in SPIBILD automatically. By default, JOBGEN has that responsibility for a file. The authority is taken away from JOBGEN when NOAUTOGEN is in effect.

How JOBGEN Works

Each night after midnight, JOBGEN examines the deferred queues of all SPIRES files that have AUTOGEN set. For any file with data in the deferred queue, JOBGEN creates and submits a simple Batch SPIBILD job that processes the file and handles any batch requests submitted through the BATCH command in SPIRES.

Processing is done overnight when rates are cheaper

The result is that files with data in the deferred queue are processed overnight, when computer rates are likely to be cheaper and the files are less likely to be used by end-users. Thus, the file is "fresh" the next morning, with updated indexes and no records in the deferred queue.

No deferred queue data, no charge

For files with no data in the deferred queue, no processing is needed, so no SPIBILD job is submitted and no charges are assessed.

The AUTOGEN Flag

Basic setting determined in file definition

Whether a file is to be processed by JOBGEN is indicated by the "AUTOGEN flag". The basic AUTOGEN setting for a file is determined in the file definition. By default, files are established with AUTOGEN, meaning that JOBGEN will be responsible for processing the file overnight. However, if the file owner adds the NOAUTOGEN statement to the file definition and compiles it, then the file's basic setting will be NOAUTOGEN, meaning that the JOBGEN program will not process the file. [WARNING: Account GG.PRD, under which JOBGEN is run, must have Read access to the DEFQ data set of the file; otherwise, JOBGEN can't see if any data exists in the deferred queue. By default, when you compile a new file definition, all accounts have Read access to the DEFQ data set. The ORVYL command "SHOW PERMITS filename.DEFQ" will tell you what accounts have access. To allow account GG.PRD access if it has none, issue the commands "SET CLP filename.DEFQ", "SET PERMIT filename.DEFQ FOR GG.PRD READ" and "SET NOCLP filename.DEFQ".]

Important: The file is always reset to its basic setting whenever the file is processed, whether by you or by JOBGEN.

Changing the Basic AUTOGEN Setting in the File Definition

If you are the file owner, you may change the basic AUTOGEN setting at any time by changing the file definition appropriately. You may make the change directly to the file definition itself or use File Definer to create a new definition with the change. Use File Definer only if you still have File Definer input that created the latest version of the definition.

Changing the file definition with File Definer

Find the FILE keyword in your File Definer input and add or remove the NOAUTOGEN attribute, depending on whether you want NOAUTOGEN to be the basic setting. Note: if you don't have a FILE statement, then AUTOGEN is the file's basic setting. You can add a FILE input line like this, if you want NOAUTOGEN:

If you need help with this change, see the SPIRES manual "File Definer".

Next use File Definer to generate a revised file definition with these commands:

Changing the file definition directly

If you prefer to change the file definition directly, get the file definition from the FILEDEF subfile (or wherever you keep your working copy). If you want to change the basic setting from NOAUTOGEN to AUTOGEN, you'll find the NOAUTOGEN statement several lines after the key (FILE). Simply remove it and update the record in FILEDEF.

If you want to change the basic setting from AUTOGEN to NOAUTOGEN, you need to add the NOAUTOGEN statement; you can put it right after the key and FILEDEF will move it to its appropriate location (just after the BIN statement), e.g.:

Then update the record in the FILEDEF subfile.

Recompile the file definition

Once you've changed the file definition in FILEDEF, recompile it:

If the file definition compiles properly, the basic AUTOGEN setting has changed. If it does not compile, SPIRES will tell you so. You will need to correct the file definition as SPIRES directs.

The SET AUTOGEN and SET NOAUTOGEN Commands

The file owner and any users with Master access to the file can still turn JOBGEN processing on or off, on a one-time basis, by issuing one of these commands:

when a subfile of the file is selected in SPIRES.

The SHOW AUTOGEN command

You can see the current setting of the AUTOGEN flag by issuing the command SHOW AUTOGEN:

If you issue the SET AUTOGEN command, then JOBGEN will begin checking the file's deferred queue that evening. The first evening that JOBGEN finds data to process in the deferred queue, it will submit a job to process the file. The file will then be reset to its basic AUTOGEN setting, as determined in the file definition.

On the other hand, if you issue the SET NOAUTOGEN command, then JOBGEN will not do any work for your file until the flag is reset to AUTOGEN. That happens when the SET AUTOGEN command is issued, or, if the basic setting is AUTOGEN, when you next process the file yourself.

Recovering from JOBGEN Problems

The NO JOBGEN flag

When any SPIBILD task terminates abnormally, SPIBILD may set the "NO JOBGEN flag", which specifies that JOBGEN should not cause the file to be processed until the file owner gives the ok. Perhaps more significantly, the NO JOBGEN flag prohibits any data base transactions -- users who try receive S495 errors.

Thus, the NO JOBGEN flag affects more than simply JOBGEN. It stops all subfile transactions and any subfile logging in effect. [See 2.1.4.]

Most of the time, users discover that a problem occurred with the overnight processing of the file sometime later, when they try to make subfile modifications but get S495 errors. But there are other ways to find out too:

 - The "log" from the SPIBILD session (either the online messages displayed  or  the
 output  from  the  batch  SPIBILD  job) will indicate that a problem occurred and ended the
 SPIBILD session.  When that happens, SPIBILD sets the NO JOBGEN flag.  [Even if the  output
 from  JOBGEN-created  jobs  is  normally purged (requested by the file definition statement
 "BIN = PURGE"), it will be printed if a  SPIBILD  problem  occurs.  In  fact,  if
 JOBGEN  output  unexpectedly appears in your bin, be sure to examine it carefully to see if
 there was a problem.]

Note also that JOBGEN processing jobs include the SET SEARCH command, which allows searching to continue during the file's processing. This can cause problems of its own if users try to update the file during processing. See Part 6 of this manual for more information about SET SEARCH. Online, EXPLAIN SET SEARCH COMMAND.

 - The SHOW AUTOGEN command indicates when the NO JOBGEN flag is set:
       -> show autogen
       -AUTOGEN in effect (NO JOBGEN)
       ->
 - The $DEFQTEST function can be used within applications to determine whether the  NO  JOBGEN
 flag  is  set.  [See the manual "SPIRES Protocols", or EXPLAIN $DEFQTEST FUNCTION
 online.]

Generally, users who discover that the NO JOBGEN flag is set should notify the file owner, particularly if it is interfering with their work (e.g., they cannot add any records).

What should the file owner do? In most situations, he or she should follow the RECOVER procedure, described earlier. [See 2.1.4.2.]

2.2.2  Processing a File with the PROCESS/ESTABLISH Procedure

When you explicitly want to process a SPIRES file, you may be in either SPIBILD or SPIRES. SPIBILD has more special-processing features than SPIRES does, so there be reasons you will need to use SPIBILD rather than SPIRES. On the other hand, SPIRES has some advantages SPIBILD does not, such as the ability to put the processing under protocol control, and better control over messages.

Specifically, here are the significant differences between processing a file in SPIBILD and in SPIRES:

 - SPIBILD exclusively has several commands that handle special file-processing circumstances,
 including SET INDEX, SET NOINDEX, SET PASS, SET NOPASS and SET STACK SIZE.
 - If problems arise, recovery operations involving  the  RECOVER  command  must  be  done  in
 SPIBILD.  [See 2.1.4.2.]
 - File-processing in SPIRES can be controlled from a protocol, with $NO testing for  failure;
 that means a file can be compiled, built and processed within a SPIRES protocol.
 - Under file-processing in SPIRES, messages from the procedure can be handled by SET TLOG, so
 that they will be logged rather than displayed at  the  terminal.   For  more  information,
 [EXPLAIN SET TLOG.]
 - File-processing in SPIRES simplifies testing  when  you  are  building  files  and  testing
 indexes  in  SPIRES;  you don't need to enter a different program and then return to SPIRES
 and re-select the subfile.
 - For a small or temporary file that you are building  in  SPIRES,  processing  the  file  in
 SPIRES  is  more efficient than having immediate indexes do that work, if you were avoiding
 SPIBILD processing for some other reason.
 - "File count" statistics are shown at the end of the SPIBILD process  by  default;
 they are not shown in SPIRES (see discussion below).

The SPIBILD Procedure

In SPIBILD, here is the procedure to follow:

Step 1: SPIBILD

Issue the SPIBILD command to enter that program. [See 2.1.2.]

In batch SPIBILD, you are already in SPIBILD, so you omit this command from the JCL.

Step 2: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the commands that limit the record-types passed from and to: SET INDEX,  SET  NOINDEX,  SET
 PASS  and  SET  NOPASS.   Remember,  however,  that  they are primarily meant to solve file
 integrity problems and should be used only after consultation with your SPIRES  consultant.
 [See 2.1.5.4, 2.4.3.2.]

Step 3: Issue the PROCESS or ESTABLISH command.

The chief file-processing command is PROCESS:

where "filename" must be in the form "ORV.gg.uuu.filename" if the file does not belong to you.

An alternative is the ESTABLISH command:

where "subfile-name" is the name of a subfile in the file. The ESTABLISH command is meant for Global FOR tasks in SPIBILD, where more work with the file will be done after the file-processing is finished. Hence, it does not release the file for use by other accounts when it is done, as does the PROCESS command.

Use PROCESS unless...

In general, use the PROCESS command, unless either:

 - another procedure you're following requires the ESTABLISH command (e.g., using  Global  FOR
 in SPIBILD); or
 - you have forgotten the name of the file but remember one of its subfile names.

The SPIRES Procedure

In SPIRES, follow this procedure:

Step 1: SPIRES

Issue the SPIRES command to enter that program.

In batch SPIRES, you are already in SPIRES, so you omit this command from the JCL.

Step 2: SELECT subfile-name

Select a subfile of the file you want to process.

For example, the subfile ALBUMS is part of the file called GQ.JNK.RECORDINGS:

At this point, if desired, you may issue SET PASSTRACE commands to enable tracing of Passprocs. See the manual "SPIRES File Definition", or online, [EXPLAIN SET PASSTRACE COMMAND.]

Step 3: Issue the PROCESS command.

In SPIRES, the syntax for the PROCESS command is:

The NOWARN option tells SPIRES not to ask you to confirm your request, as it does by default.

For example:

Processing time varies

Depending on the size of the deferred queue and the number of indexes to update, file-processing in either SPIRES or SPIBILD could take only a few seconds or up to several hours. But the "average file" with, say, several dozen records to process requires a minute or two of elapsed time. Messages reporting the progress of the processing will appear occasionally. A description of all the messages you will see appears later in this chapter. [See 2.2.4.] or online, [EXPLAIN FILE PROCESSING, MESSAGES.]

Example

Here is how a typical SPIBILD session involving the PROCESS command might look:

The messages tell us that this file has two goal record-types, ALBUM and XREC19, each with its own set of indexes to update. SPIBILD processed the 136 DEFQ records in about 45 seconds, using 4.989 real CPU seconds.

Handling Problems

Though most file-processing tasks run quite smoothly, problems do occasionally arise. The most common problems encountered during SPIBILD processing were discussed in chapter 2.1. [See 2.1.4.1.] or [EXPLAIN PROBLEMS IN SPIBILD.]

2.2.2a  Processing Only One Subfile of a File: PROCESS SUBFILE

In some situations, you may not want to process the entire file. For example, suppose you have added some records to a subfile that acts as a lookup table for the primary record-type of the file. You need the lookups to begin using the new records right away, but for efficiency reasons, the lookup actions won't look in the deferred queue for the table records. You could process the file, but that could cause problems for the people currently using the primary subfile.

So the PROCESS SUBFILE procedure lets you process only the selected subfile's worth of deferred-queue data. As soon as the record-type is processed, its deferred-queue data is discarded from the defq, leaving any data from other record-types of the file. If there is no other defq data, including logs or batch requests, then the defq is re-initialized, just as if a full-fledged file process had been done.

Step 1: SPIRES

Issue the SPIRES command to enter that program.

In batch SPIRES, you are already in SPIRES, so you omit this command from the JCL.

Step 2: SELECT subfile-name

Select the subfile you want to process.

For example, the subfile ARTIST TABLE is part of the file called GQ.JNK.RECORDINGS:

Step 3: Issue the PROCESS SUBFILE command.

In SPIRES, the syntax for the PROCESS SUBFILE command is:

The NOWARN option tells SPIRES not to ask you to confirm your request, as it does by default. It also tells SPIRES not to warn you if you have a result and/or stack in effect for the selected subfile; the PROCESS SUBFILE command must clear any stack or result in order to work successfully.

For example:

Processing time varies

Depending on the size of the deferred queue and the number of indexes to update, subfile-processing could take only a few seconds or up to several hours. But the "average file" with, say, several dozen records to process requires no more than a minute or two of elapsed time. Messages reporting the progress of the processing will appear occasionally. A description of all the messages you will see appears later in this chapter. [See 2.2.4 or online, EXPLAIN FILE PROCESSING, MESSAGES.]

Handling Problems

Though most file-processing tasks run quite smoothly, problems do occasionally arise. The most common problems encountered during file-processing were discussed in chapter 2.1. [See 2.1.4.1 or online, EXPLAIN PROBLEMS IN SPIBILD.]

If the process fails, note that the NOJOBGEN flag will be set, blocking subsequent updates for any record-type of the file until the file is successfully processed or recovery takes place. [See 2.1.4.2 or online, EXPLAIN RECOVERING A FILE.]

2.2.3  Choosing How the File Should Be Processed

In practice then, there are three techniques used by file owners to process a file:

 - 1) give JOBGEN the responsibility of overnight processing [See 2.2.1.]
 - 2) submit a batch job (batch SPIBILD or BATWYL) to process the file  on  a  regular  basis,
 probably    through    the    SUBMIT    program   [See   2.1.3,   2.2.2.]
 - 3) process the file manually whenever desired, using the PROCESS/ESTABLISH  procedure  [See
 2.1.3.]

As file owner, you can change between these methods whenever you like. For regular processing of a file, most file owners choose either the first or second method, and then manually process the file when special needs arise. Alternatively, you could decide to manually process the file most of the time, but on any given day, choose to have JOBGEN take the responsibility for that night only.

Here are some factors to consider when you are deciding what method to use:

The rate of subfile transactions

The overhead costs of file processing decrease per record as the number of records processed increases. For example, five jobs that each process one new record will cost more than one job that processes five new records at once. If a file has only one or two transactions per day, it might not be worthwhile to have the file processed each night by JOBGEN; a job that processed the file once a week might be preferable.

The need for up-to-date indexes

Generally, the most important reason to process a file is to update the indexes to reflect the record transactions. [See 2.2 for other reasons.] You should consider how important it is that the indexes accurately reflect the current contents of the subfiles. (If constant accuracy is crucial, you probably want to consider making the indexes "immediate indexes", in which case file processing is not so important.)

Recovery in case of problems

JOBGEN makes a backup copy of the deferred queue of each file before it submits the job to process it. In conjunction with "dump tapes" made regularly to copy all ORVYL data sets, the backed-up deferred queues can be used to restore a SPIRES file to its state as of the previous night's processing. However, if the file owner has manually processed the file after the dump tape was last made, the data handled by that process would not be available in JOBGEN's deferred queue dump tapes. Many applications that cannot use JOBGEN processing have jobs to process the file that begin by making a backup copy of the file. [See 2.1.4.3.]

Special processing

Many applications require other work to be done immediately before or after a file is processed. For example, you might want to print a copy of all records being removed, or save a copy of the transactions log of one of the file's subfiles, or make a backup copy of the file. In such situations, you would probably write your own job to process the file, surrounded by JCL and commands to handle the other tasks too.

Command logging

The log data for subfiles with command logging is data in the deferred queue. Any data in the deferred queue, even if it's only log data, will cause JOBGEN to process a file.

Batch requests in the deferred queue

Users can submit "batch requests" for subfile updates in SPIRES (using the BATCH command) that are processed by JOBGEN when it creates the SPIBILD job during overnight processing. Such batch requests, admittedly quite rare, reside in the deferred queue; they are completely discarded if the file is processed by anyone other than JOBGEN, without warning. If there is a chance that users are entering data via the BATCH command in SPIRES, you should either rely on JOBGEN processing the file, or be sure to check for batch requests before processing it yourself. [See "SPIRES Searching and Updating", section D.8, for more information.]

Test Your Decision

The above list of points to consider when choosing how your file should be processed are only guidelines. You will have to decide what to do based on your own conditions and preferences. Testing the possibilities is highly recommended, particularly if your application is complex.

Remember, it is easy to switch from one method to the other, either for a single processing of the file, or for a longer period of time.

2.2.4  The Messages Displayed During File Processing

When you process a file, either by an explicit PROCESS command or as part of a SPIBILD command, SPIRES or SPIBILD displays information about the processing as it proceeds. For example, here is a simple SPIBILD session showing the messages SPIBILD provides in most cases:

This section will describe what those messages mean. In addition, it will discuss the file's command log, which may be displayed when you process a file.

-Begin passing of n records of record-type <rec-type>

The passing messages appear only if the deferred queue contains data that must be passed to indexes. The first message gives you several pieces of information:

 - 1) SPIBILD is now working  with  the  deferred-queue  records  of  the  named  record-type.
 - 2) The record-type is a goal record-type with indexes.
 - 3) SPIBILD has moved the specified number of records from the deferred queue into the  tree
 for that record-type.
 - 4) SPIBILD is beginning to apply data from the "pass stack", an area in memory in
 which the index data is collected, into the indexes for the record-type.   The  pass  stack
 contains the index data for that number of records.

Completed passing of n records total

This message appears when SPIBILD has finished applying the pass stack to the index record-types, has discarded the pass stack and is ready to start over with the next goal records, if any. The number "n" is a running total of the records of the goal record-type being worked on that have been successfully processed. The key of the record with which SPIBILD is resuming will also appear at the end if there are more goal records of the same record-type to process, as in the example above.

SPIBILD may fill up the pass stack several times

The pass stack has a limited size. If it fills up with data before processing all the deferred-queue records for that record-type, then SPIBILD will display the first passing message, "pass" that data, empty the pass stack, display the "Completed" passing message, and begin filling it up again. That's why the example above shows two pairs of passing messages for record-type REC01 -- the pass stack filled up from the first 84 records. After the index records were updated using that data, SPIBILD began working with the remaining REC01 records in the deferred queue.

The last section of this chapter describes the pass stack in more detail. [See 2.2.6.] or [EXPLAIN FILE PROCESSING, TECHNICAL DETAILS.]

-Processed: n DEFQ records of record-type <rec-type>

This message appears when all processing is completed for a given record-type. That is, SPIBILD has applied the data in the deferred queue for that record-type to the tree, and updated all the indexes. This message displays the total number of records processed for the record-type.

Processing is done in record-type order

The order in which the record-types are processed is the order in which they are defined in the file definition: in alphanumeric order by name. For example, REC01 is processed before REC02, which is processed before REC10, which precedes ZIN01.

If the deferred queue contains records for several different record-types, there will be several sets of passing and processing messages.

Subfile logging messages

Many SPIRES files have subfiles for which log information is collected, invoked by the LOG and STATISTICS statements in the file definition. [See 4.3.] If the STATISTICS statement has the value "1", the log information is displayed at this point in the file processing.

The log data may be thousands of lines long. If you want to stop the display, press the ATTN/BREAK key. The log display should stop, and the file processing will resume. (Pressing ATTN/BREAK too many times may cause a session break; if so, simply type GO as directed to resume the processing.)

Detaching File: <filename>

SPIBILD is finished working with the file and detaches it, as confirmed by this message. When you process a file is SPIRES, you will not see this message; SPIRES does not "de-select" the subfile, so the file remains attached.

File Read/Write Counts Information

SPIBILD (but not SPIRES) summarizes the I/O counts for each data set of the file that was involved in the processing. These messages can be suppressed with the CLEAR FILE COUNTS command. For further information about them, see the SET FILE COUNTS command in Part 6 of this manual; online, [EXPLAIN SET FILE COUNTS COMMAND.]

For SPIRES, to see this information, use the SHOW FILE COUNTS and CLEAR FILE COUNTS commands, which are described in the manual "SPIRES Technical Notes". Online, [EXPLAIN SHOW FILE COUNTS.]

Compute time: n.nnn seconds

This message displays the number of real CPU seconds used to process the file. At Stanford, this number must be multiplied by a constant to get the CPU value for which you are charged; see your consultant for more information.

Elapsed time: n.nnn seconds

This message displays the actual amount of time passed since the processing began.

Core usage: nnnn/mmmm

This SPIBILD (not SPIRES) message displays the number of kilobytes (1024 bytes) in main memory used to process the file (nnnn) compared to the number available (mmmm). The amount available is often three or four times the amount needed. If the amount used gets very close to the amount available (say, within 10K), contact your SPIRES consultant.

Error messages

When problems arise during file processing, SPIBILD or SPIRES displays appropriate error messages. You can get more information about them by issuing the EXPLAIN command. More information about specific SPIBILD problems appears elsewhere in this manual. [See 2.1.4.1.] or online, [EXPLAIN SPIBILD PROBLEMS.]

2.2.5  Allowing Others to Process Your Files

Before another user can process or perform any other SPIBILD task on one of your files, there are two actions you must take as the file owner:

The two steps are described in detail below.

Step 1: Give the user Process access to the file.

You must change and recompile the file definition for this step to take effect. You can either use File Definer to change the file definition or change the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file.

To use File Definer

If you are using File Definer to change the definition, add (or change) the PROCESS attribute onto the FILE statement of your File Definer input:

where "gg.uuu" is the user's account number. If you don't already have a FILE statement in your input, add it as the first line of the input. You can include multiple accounts, by separating them with commas, or you can specify them in other standard forms, such as "GG...." or PUBLIC to give access to an entire group or the public at large.

Then use File Definer to generate a revised file definition:

Change the file definition directly

If you are changing the file definition directly, get it from the FILEDEF subfile in SPIRES and add at the end (or change, if it already exists):

You can include multiple accounts, by separating them with commas, or specify them in other standard forms, such as "GG...." or PUBLIC to give access to an entire group or the public at large.

When you've made the change, replace the old version with the new in the FILEDEF subfile.

Recompile the file definition

Once you've changed the file definition, you must recompile it to effect the changes:

Step 2: Set the ORVYL permits for the file's data sets

The next step is to tell ORVYL to allow the user to write data to each data set of the file except the MSTR. You must issue three separate ORVYL commands for each data set:

where "filename" is the name of your file and "dsname" is the type of data set: CKPT, DEFQ, REC1, REC2, etc. and RES. (To see which RECn data sets exist for your file, type the command "SHOW FILES LIKE filename.REC".)

The other user may now process your file.

Permits sometimes need resetting

Note: some file-management procedures may cause ORVYL to lose the permits established here. In particular, procedures in which a data set is completely replaced or temporarily erased will require you to reset the ORVYL permits you set here. The most notable examples are the Rebalancing procedure and the Rebuilding a SPIRES File procedure.

2.2.6  (*) How Passing Works

[Note: The following section provides technical details on the passing procedure used by SPIBILD. It is unnecessary reading for most SPIRES file managers.]

The SPIRES/SPIBILD/FASTBILD processors make use of the Linkage Sections of a File Definition (GOALREC-NAME sections) in the following ways:

Each record type of a file is processed in ascending order by record-type number (all REC1's, then all REC2's, etc.). Goal records associated with each record-type are processed either in the order in which they are input under BATCH or MERGE command processing, or in ascending key order if through the DEFQ. Note that REMOVE, UPDATE and MERGE commands under BATCH processing normally cause DEFQ entries to be created and later processed in key order. ADD commands under BATCH processing are handled in the order they are received.

Limiting the discussion to just the PROCESS command in SPIBILD will simplify what happens, and make it possible for you to determine what happens on a "recovery" situation. For simplicity, assume all records come from the DEFQ.

Phase 1

If the record-type does NOT have an associated Linkage Section (or all INDEX-NAME sections are marked NOPASS), then skip to Phase 2.

For ADD-type records, a copy of the record is first added to the TREE. If the record fails to add because a copy already exists in the TREE, then it is assumed that "recovery" is taking place, and the TREE copy is replaced. The DEFQ record is now sent to the passing process marked for pointer INSERTION.

For UPDATE-type or REMOVE-type records, the original TREE copy of the record is read up and sent to the passing process marked for pointer DELETION. Then, for the UPDATE-type, the record from the DEFQ is sent to the passing process marked for pointer INSERTION.

Note that the values passed should come from data within the goal records themselves. Values created from $variables, like $UDATE, or retrieved from other records ($LOOKUP) may cause index entries that won't be deleted when the goal record is removed or updated. [See 2.2.7, 2.2.8.]

The linkage section describes what passing to do

The passing process is controlled by the linkage section. Each linkage section consists of:

 - A Common portion before the first INDEX-NAME that defines the pointer (PTR-ELEM) as well as
 any global qualifiers (QUAL-ELEM).
 - Individual INDEX-NAME portions, in which index record-types are named, and index values are
 defined.  There can also be sub-index and/or local qualifiers defined.

The global parameters section

For each record received (marked for pointer INSERTION or DELETION), the PTR-ELEM and global QUAL-ELEM portion of the Linkage is processed first. One and only one item is placed in the "pass stack" for each item in this Common portion. Passproc rules $PASS.OCC or A165 may be used to cause no entry, or something like $PASS.ELEM or A167:1 (multiple passer) can be used to cause the first occurrence of the first existing element in the list to be placed in the pass stack. If no value can be found and $PASS.OCC (A165) or $PASS.DEF (A171) was not used, then a null value is placed in the pass stack.

The individual linkage sections

Once the Common portion of the Linkage Section is finished, processing begins on each INDEX-NAME portion. The INDEX-NAME, SUB-INDEX, and local QUAL-ELEM terms constitute one INDEX-NAME portion. These terms are "traversed" in both a forward and backward manner beginning with a forward scan starting with INDEX-NAME and running through the last term. Each term causes one value to be retrieved from the goal record.

If no value can be found during a forward scan, and $PASS.OCC (A165) or $PASS.EDF (A171) was not used, then NO INDEX ENTRY is defined, and backward scan commenses. If the forward scan can be completed, then an INDEX ENTRY is defined, and backward scan commences.

During backward scan, each term attempts to retrieve a value from the goal record from where it left off during the previous forward scan. If another value can be retrieved, then forward scan resumes from this point, and if it continues forward through the last term, then another INDEX ENTRY is defined. However, if during any forward scan, new terms are within the structural bounds of previous terms, then those structural bounds limit the values retrieved to occurrences within those bounds. Terms may be bound by a "common root", which is some structural level from which separate branches to the terms can be traced.

Examples of scanning

Here are some examples using a single INDEX-NAME section with different layouts of elements in goal records. Note: only significant elements are shown.

The layout of WORD, DATE, and TIME in the goal records can vary in many ways. Structural boundaries can make a big difference in what happens. Consider the following record:

All elements are doubly occurring, and all are at the record level. The following INDEX ENTRIES will be constructed:

Now consider the following layout:

                              TIME(1)             TIME(2)
                     DATE(1)  | TS(1)    DATE(2)  | TS(2)
                     |--------|          |--------|
   WORD(1)  WORD(2)  | DS(1)             | DS(2)
  -|--------|--------|-------------------|

Here, TIME is in a structure (TS) which has been defined as an element of another structure (DS) along with DATE, and that structure is defined at the record level along with WORD.

The following INDEX ENTRIES will be created:

After completing the first forward pass, an attempt to pick up another value of TIME on a backward scan would have required leaving the structural bounds defined by the first DATE. On a backward scan, structural bounds defined by a previous element can't be crossed, so the backward scan continues by trying to get another DATE value. The boundary now is the record-level defined by WORD. Since the DATE structure has its root at that level, we can proceed forward again beginning with DATE(2), and then TIME(2). Scanning backward again takes us all the way back to WORD(1) from which we now pick up WORD(2) and scan forward again.

If there had been other occurrences of the TIME structure associated with DATE(1), then we would have picked all of them up along with DATE(1), but none of them with DATE(2). The same thing would have happened if TIME had been simply a multiply occurring element inside the DATE structure instead of being nested within a multiply occurring structure defined inside the DATE structure.

That is,

                     DATE(1)  TIME(1)    DATE(2)  TIME(2)
                     |--------|          |--------|
   WORD(1)  WORD(2)  | DS(1)             | DS(2)
  -|--------|--------|-------------------|

Now, consider the following layout:

                                         WORD(1)  WORD(2)
                                         |--------|
                     DATE(1)  DATE(2)    | WS(1)
                     |--------|----------|
   TIME(1)  TIME(2)  | DS(1)
  -|--------|--------|

Here, WORD is a multiply occurring element of a structure (WS) defined as an element of another structure (DS) containing a multiply occurring DATE element, and that structure is defined at the record-level along with a multiply occurring TIME element.

The following INDEX ENTRIES will be created:

That's all possible combinations because the entries have a "common root" at the record level. Although the access path is upside-down, there still are structural bounds, beginning at WORD and working down to TIME at the record level. However, the next example shows how bounds can apply, even in upside-down accessing.

                              WORD(1)             WORD(2)
                     DATE(1)  | WS(1)    DATE(2)  | WS(2)
                  )  |--------|          |--------|
   TIME(1)  TIME(2)  | DS(1)             | DS(2)
  -|--------|--------|-------------------|

Here, WORD is within a structure (WS) which has been defined as an element of another structure (DS) along with DATE, and that structure is defined at the record level along with TIME.

The following INDEX ENTRIES will be created:

Concluding Phase 1

Once all INDEX ENTRIES for an INDEX-NAME section have been placed in the pass stack, the next INDEX-NAME section is processed. An INDEX-NAME section is consider completed when a backward scan for the INDEX-NAME term fails to retrieve a value. When all INDEX-NAME sections have been completed, passing for the record is finished, but the information concerning index updates only exists in the pass stack.

When the pass stack fills up or no more records are retrieved from the DEFQ, then the pass stack is processed by sorting it down to unique INDEX ENTRIES cancelling duplicate entries and any matching INSERTION/DELETION pairs. The result is a list of INSERTION and/or DELETION entries to be applied to the indexes. The appropriate index records are added, removed, or updated using the the pass stack information. Because of the sort, an index record need only be read once in order to apply all the pass stack information associated with that record.

When a Pass Stack's worth of goal records has been passed, those same goal records are reread from the DEFQ. Each REMOVE-type record causes the TREE copy to be removed; each UPDATE-type record is used to replace the TREE copy. Then the DEFQ record is marked as DEQueued. When the group of records that contributed to the the pass stack are all DEQueued, Phase 1 repeats with another group of goal records until all DEFQ record for this record-type have been processed. When done with all records, continue with Phase 3.

Phase 2

All the records in the DEFQ associated with one record-type are read sequentially. For each ADD-type record, the record is added to the TREE. For each REMOVE-type record, the TREE copy of the record is removed. For each UPDATE-type record, the TREE copy is replaced by the DEFQ copy. When done with all records, continue with Phase 3.

Phase 3

When all the records for a particular record-type have been processed, the next logical record-type is processed starting back at Phase 1. When all record-types have been processed, the DEFQ is cleared and the PROCESS command is finished.

Immediate Indexing

Note that Immediate Indexing in SPIRES does Phase 1 for just the single record being immediately indexed, and only those INDEX-NAME terms marked as IMMEDIATE are sent to the pass stack. These same records are usually processed again later by SPIBILD, but then only the non-IMMEDIATE terms are sent to the pass stack because the IMMEDIATE terms are already done. The TREE copy of the record is not REMOVED or UPDATED during immediate indexing. The DEFQ information becomes the latest copy of the record, and may participate in another Immediate Indexing transaction.

2.2.7  Passing Variable Values

Passing to indexes requires you be able to undo that indexing. Here's a simple example of something that works just fine.

An index record might be defined something like this:

The Linkage section would probably look like this:

If you ADD a new record to the database, MODDATE gets today's date as its value. The Linkage section for GOAL passes MODDATE to the INDX index. Then, the new record is placed in the GOAL record's TREE. [See 2.2.6.]

Now, several days pass and you decide to update this GOAL record. The updated record get's today's date as the value of MODDATE, and the record is placed in the DEFQ. Of course, "today's date" is different than when the record was originally added. When passing occurs, the TREE copy of the record is read and the value of MODDATE stored in that record is sent to the pass stack marked for DELETION.

The record in the DEFQ then passes its MODDATE value to the pass stack marked for INSERTION. The DELETION and INSERTION values are different, so the old value will be un-indexed, and the new value will be indexed.

But here's a simple example of something that doesn't work. To save space, you decide to make MODDATE into a VIRTUAL element, and have it pass the equivalent of the $UDATE variable. Your MODDATE element in the GOAL record is defined as follows:

Everything else remains the same.

If you ADD a new record to this database, MODDATE has no value because it is VIRTUAL. But the Linkage section obtains a value from MODDATE to pass to the INDX, and that value is today's date. Then, the new record is placed in the GOAL record's TREE.

So far, it seems like everything is fine. But now, several days pass and you decide to update this GOAL record. The record is placed in the DEFQ. Again, MODDATE has no value because it is VIRTUAL.

When passing occurs, the TREE copy of the record is read and the value of MODDATE created is today's date, which is sent to the pass stack marked for DELETION. However, that wasn't the date that was originally indexed.

The record in the DEFQ then passes its MODDATE created value to the pass stack marked for INSERTION. The DELETION and INSERTION values are identical! This is a matching INSERTION/DELETION pair, and it is eliminated from the pass stack. The result is that NO INDX entries are changed.

Even if you tried to REMOVE the GOAL record at some date after it was originally added and processed, you would not be able to un-index the original date from the INDX. This is an example of why you should NOT pass values dependent upon variables.

2.2.8  Passing Table Values

Indexes are usually built from values contained within the GOAL record itself. There may be some transformation of the value, but it is a consistent transformation which will always produce the same index value(s) for a given GOAL value.

For example, consider GOAL and ZINDX records defined as:

The Linkage section might contain something like this:

Occurrences of ZIPCODE in the GOAL records are passed to the ZINDX so you can search for GOAL records by ZIPCODE. There's nothing unusual here.

But now you discover there is a separate subfile (call ZIPTABLE) with records that act like a table pairing ZIPCODE with CITY names. Records in the ZIPTABLE subfile might look like this:

You decide you want to build a CITY index in your database, using your ZIPCODE to lookup CITY in this separate subfile, and passing that value to a word index. You add the following record-type to your file definition:

You also add a VIRTUAL element to your GOAL record definition:

And finally, you add the following to the Linkage section:

The first time you add a GOAL record, ZIPCODE is used to lookup CITY in the ZIPTABLE subfile, and that values is broken into words and sent to the ZNAME index.

But what happens if the table changes after you've built ZNAME entries? For example, assume you used 94040 as a ZIPCODE which built ZNAME index entries for MTN and VIEW. Then the 94040 record in the ZIPTABLE subfile is changed like this:

The next time a GOAL record accesses the table for 94040 it will obtain MOUNTAIN VIEW as the CITY, and that is NOT what was originally sent to the ZNAME index. If you tried to REMOVE a GOAL record which had 94040 as a ZIPCODE, the word VIEW would be unindexed, but not the word MTN. Likewise, if you UPDATE a GOAL record which has 94040 as a ZIPCODE, and changed it to 94022, you would unindex the word VIEW, but NOT the word MTN. And then you would index the words LOS and ALTOS. Your record would still be found by: FIND CITY MTN; even though the ZIPCODE is no longer associated with MTN VIEW.

Even if you updated all GOAL records that contained 94040 as the ZIPCODE, and didn't change the ZIPCODE, you wouldn't affect the ZNAME index. That's because both the TREE and DEFQ copies of the GOAL record would retrieve the same 94040 ZIPTABLE record, and would get MOUNTAIN VIEW as the CITY. These terms cancel in the pass stack, and ZNAME is left unchanged. [See 2.2.6.]

This is the danger in passing values obtained by $LOOKSUBF or $LOOKSUBG. The accessed value from such records may not change once they are stored. You can add new table entries with new codes, and you can change the accessing codes in your GOAL record, but you can't allow the accessed values in the table to change.

A better solution, to get what you want without fear of table changes, is to have the ZIPTABLE subfile build its own CITY word index. Then, instead of building a ZNAME index, you code "Indirect Search". [EXPLAIN INDIRECT SEARCHING.]

2.3  Adding Multiple Records to a Subfile

SPIRES, SPIBILD and FASTBILD provide several ways for you to add multiple records to a subfile at once. The different methods work under different circumstances -- which one you choose will depend on the particular situation you have.

FASTBILD procedures in Chapter 2.12

Below is a comparative summary of the methods. The procedures involving SPIBILD are described in detail in the rest of this chapter; those involving FASTBILD are described later in Part 2. [See 2.12.] Information about the SPIRES procedures appears in the SPIRES manual "Searching and Updating".

This chapter ends with a short section about the format of the data in input data sets used with these procedures [See 2.3.5.] and a section on how to resume data input if a problem terminates your input processing in SPIBILD using one of the methods described in this chapter. [See 2.3.6.]

Methods for Adding Multiple Records to a Subfile

There are 14 techniques for adding multiple records to a subfile in a single shot:

 - 1.  (SPIRES) the INPUT BATCH procedure
 - 2.  (SPIRES) the INPUT ADD procedure
 - 3.  (SPIRES) the INPUT ADDUPDATE procedure
 - 4.  (SPIRES) the INPUT ADDMERGE procedure
 - 5.  (SPIRES) the BATCH procedure
 - 6.  (SPIBILD) the INPUT BATCH procedure
 - 7.  (SPIBILD) the INPUT ADD procedure
 - 8.  (SPIBILD) the INPUT ADDUPDATE procedure
 - 9.  (SPIBILD) the INPUT ADDMERGE procedure
 - 10.  (SPIBILD) the LOAD procedure
 - 11.  (SPIRES or SPIBILD) the INPUT LOAD procedure
 - 12.  (SPIBILD) the FAST BATCH / FAST LOAD procedure
 - 13.  (FASTBILD) the FAST BUILD procedure
 - 14.  (FASTBILD) the LOAD procedure

Which you should use in a given situation depends primarily on these factors:

 - If it's in the internal SPIRES format (i.e.,  coming  from  another  SPIRES  file  via  the
 GENERATE  LOAD  command),  you  must  use the LOAD procedure in SPIBILD or FASTBILD, or the
 INPUT LOAD procedure, or the FAST LOAD procedure in SPIBILD.
 - If it's in the standard SPIRES format, you can use the INPUT BATCH procedures in SPIRES  or
 SPIBILD,  the  BATCH  procedure  in SPIRES, the FAST BATCH procedure in SPIBILD or the FAST
 BUILD procedure in FASTBILD.
 - If it's in a customized format, you can  use  the  INPUT  ADD,  INPUT  ADDUPDATE  or  INPUT
 ADDMERGE  procedures in SPIRES or SPIBILD, the FAST BATCH procedure in SPIBILD, or the FAST
 BUILD procedure in FASTBILD.

Below are descriptions of each of the techniques listed above.

1. (SPIRES) The INPUT BATCH Procedure

The INPUT BATCH command builds records from the input data set and adds them to the file's deferred queue. The data may be in the standard SPIRES format and may include, besides records to be added, records to be updated, merged or removed. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.

The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. It can be used with data stored on tape through a BATSPI batch job.

Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner, not the user doing the input, who pays for only the input costs.

Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.

See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES in general. For online help, EXPLAIN INPUT BATCH.

2, 3, 4. (SPIRES) The INPUT ADD, ADDUPDATE and ADDMERGE Procedures

The INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands build records from the input data set and add them to the file's deferred queue. The data must be in a form that can be read by a customized input format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.

INPUT ADD can handle only records being added. INPUT ADDUPDATE and INPUT ADDMERGE check the subfile to see whether a record with the same key as given in the input data already exists, in which case that record is replaced with the new input (INPUT ADDUPDATE) or the input is merged into that record (INPUT ADDMERGE).

These INPUT commands in SPIRES are available to any user with update privileges to the subfile. Each can be used with data stored on tape through a BATSPI batch job.

Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner, not the user doing the input, who pays for only the input costs.

See the SPIRES manual "Searching and Updating" for more information about the INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands in SPIRES. For online help, EXPLAIN INPUT ADD COMMAND, etc.

5. (SPIRES) The BATCH Procedure

The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.

This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.

The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user does not know till the next day whether the records were accepted, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.

The input data must be in the standard SPIRES format. It is not limited to added records, but can include updates, merges or removes too.

A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched, of course; but the request which tells JOBGEN where the data is gets thrown away. This problem makes the BATCH command a somewhat unreliable method, particularly if you do not know the file owner.

For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".

6. (SPIBILD) The INPUT BATCH Procedure

The INPUT BATCH command in SPIBILD builds records from the input data set and adds them directly to the tree. As it does that, it builds the pass stack (the changes to be applied to the subfile's indexes) for the added records, which is processed and emptied when it fills up or when the end of the input data is reached.

The data must be in the standard SPIRES format and may include records to be updated, merged or removed. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the records involved.

The INPUT BATCH procedure in SPIBILD is unlike the INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.

The input data may be on disk or tape, though if on tape, the procedure must be done through a batch SPIBILD job.

The steps of this procedure are described in detail in the next section. [EXPLAIN BATCH PROCEDURE IN SPIBILD online.]

7, 8, 9. (SPIBILD) The INPUT ADD, ADDUPDATE and ADDMERGE Procedures

The INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE commands in SPIBILD build records from the input data set and add them directly to the tree. As they do that, they build the pass stack (the changes to be applied to the subfile's indexes) for the added records, which is processed and emptied when it fills up or when the end of the input data is reached.

The data must be in a form that can be read by a customized input format. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the records involved.

INPUT ADD can handle only records being added. INPUT ADDUPDATE and INPUT ADDMERGE check the subfile to see whether a record with the same key as given in the input data already exists, in which case that record is replaced with the new input (INPUT ADDUPDATE) or the input is merged into that record (INPUT ADDMERGE).

These INPUT procedures in SPIBILD are unlike their counterparts in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use these SPIBILD procedures. Again, the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.

The input data may be on disk or tape, though if on tape, the procedure must be done through a batch SPIBILD job.

The steps of this procedure are described in detail later in this chapter. [EXPLAIN INPUT ADD PROCEDURE IN SPIBILD online.]

10. (SPIBILD) The LOAD Procedure

You can use the LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file).

The input data is the internal SPIRES form of the records, exactly as stored in the ORVYL data sets. To create the data, you use the Global FOR command GENERATE LOAD in SPIRES. Then in SPIBILD, you issue the LOAD command to add the data to the appropriate file.

The major advantage of this method is that it avoids the substantial overhead of executing Outproc rules when the input data set is being created, and of executing Inproc rules when the data set is being loaded into the new file.

The input data for the LOAD procedure must come from a disk data set created by the GENERATE LOAD command; it cannot come directly from a tape.

Again, because the LOAD command causes the file to be processed, you cannot use it unless you have Process access to the file.

The LOAD procedure is covered in detail later in this chapter. [See 2.3.3.]

11. (SPIRES or SPIBILD) The INPUT LOAD Procedure

You can use the INPUT LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file). It is similar to the LOAD procedure and has the same advantages; it is different in that it is available in SPIRES as well as SPIBILD, which allows you to take advantage of deferred-queue processing, to use it within a protocol, or to use it with temporary files, which are available only in SPIRES.

In SPIRES, INPUT LOAD also has the WITH FAST and WITH SEARCHONLY options, which speed up the processing, at the cost of some restrictions.

The input data is the internal SPIRES form of the records, exactly as stored in the ORVYL data sets. To create the data, you use the Global FOR command GENERATE LOAD in SPIRES. Then in SPIRES or SPIBILD, you issue the INPUT LOAD command to add the data to the appropriate file.

The major advantage of this method is that it avoids the substantial overhead of executing Outproc rules when the input data set is being created, and of executing Inproc rules when the data set is being loaded into the new file.

You cannot use it unless you have Master access to the file.

The INPUT LOAD procedure is covered in detail later in this chapter. [See 2.3.3a.]

12. (SPIBILD) The FAST BATCH and FAST LOAD Procedures

The FAST BATCH and FAST LOAD procedures for adding records are specialized techniques that can be used only in certain circumstances:

 - The record-type into which the records will be added must be completely empty,  unless  the
 record-type has a slot key.
 - The records in the input data must either be in ascending key order or else have their keys
 assigned by SPIRES in ascending order (e.g., a slot key).

The records will go into the record-type very quickly and efficiently, many times faster than they probably would with the standard INPUT BATCH or INPUT ADD procedure in SPIBILD (see above). The gain in efficiency comes at the loss of two features of the INPUT procedures:

 - The FAST procedures do not pass data to indexes; if the record-type has indexes,  you  must
 build them later, using the BUILD procedure.  [See 2.7.1.]
 - The FAST procedures do not checkpoint the data.  That means  that  the  procedure  must  be
 started over again if a major problem such as system failure occurs.

FAST BATCH and FAST LOAD are very useful when you have hundreds of thousands of records to add to a new file. They break the file-building process into several pieces (building the goal record-type with the FAST BATCH or FAST LOAD command and building the indexes later with the BUILD command), which can be scheduled independently. For example, you might need many hours to build a large file using the BATCH command in SPIBILD; you would need much less total time to do the same work using FAST BATCH and BUILD, plus you could do the two pieces whenever convenient.

To use the FAST BATCH procedure, you must have Process access to the file. The input data for the FAST BATCH procedure may be on tape, in which case you must use batch SPIBILD, and it may be formatted, as long as you have an input format that can read it.

You can use the FAST LOAD procedure when the input data is coming from either the same record-type (you may be rebuilding a file) or a record-type with the same characteristics (e.g., a test version of the file). The data must be stored in a load file online; it cannot be on tape. The FAST LOAD procedure is similar to the LOAD procedure and has the same advantages; but it also has the same advantages as the FAST BATCH procedure.

FAST BATCH and FAST LOAD, though similar to FASTBILD in some respects (see below), are much easier to use. The complete procedure is described later in this chapter. [See 2.3.4.]

13, 14. (FASTBILD) The FAST BUILD and LOAD Procedures

Of all the procedures, FAST BUILD and LOAD in FASTBILD are the most difficult to use. Like FAST BATCH, they can be used only if the goal record-type is empty to begin with. They also separate building the goal record-type from building the indexes, though you can have FASTBILD take care of the index-building automatically, if you like.

Also like FAST BATCH, FAST BUILD and LOAD do not checkpoint the data; if problems arise, the procedure must be restarted. Problems tend to arise more often in FASTBILD because the JCL and commands you must put together in order to use it can be quite complex. Several limits that are not always easy to determine must be specified in the job stream; errors in these estimates can cause serious problems, requiring you to start the job over again.

But on the other hand, the FASTBILD procedures are the most efficient of all when you are building a record-type of thousands of records. The index-building process is considerably more efficient in FASTBILD than SPIBILD, particularly in I/O usage.

You can use the FASTBILD procedures if you have Process access to the file. The input data may be formatted (though you will need an input format to read it) and may be on tape. If it is formatted in the internal SPIRES format, i.e., if it is coming directly from another SPIRES file, you use LOAD instead of FAST BUILD.

More information about FASTBILD and a complete description of the FAST BUILD and LOAD procedures appear in a later chapter. [See 2.12.]

2.3.1  The INPUT BATCH Procedure in SPIBILD

This procedure allows you to add multiple records to a subfile at once using SPIBILD. The input data is in the standard SPIRES format, and may also contain requests for record updates and removes as well. [See 2.3.5.]

You may use this procedure either online or in batch. [See 2.1.3.] If using it in batch, you will need to modify the steps of the procedure to fit batch, rather than interactive, work.

Step 1: SPIBILD

To invoke SPIBILD, issue the SPIBILD command online, or in a BATWYL job:

In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.

Step 2: Prepare the input data set.

If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set.

Step 3: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIBILD displays.  The command  "SET
 INFORMATION  MESSAGES  =  0"  is  especially useful, since it turns off the individual
 "add"    message    displayed    for    each     new     record.      [See     2.1.5.3.]
 - the SET EXCEPTION command, to name an ORVYL data set into which unacceptable  records  from
 the input data set are placed.  [See 2.1.5.2.]
 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the commands that limit the record-types passed from and to: SET INDEX,  SET  NOINDEX,  SET
 PASS  and  SET  NOPASS.   Remember,  however,  that  they are primarily meant to solve file
 integrity problems and should be used only after consultation with your SPIRES  consultant.
 [See 2.1.5.4, 2.4.3.2.]
 - the SET SKIP command, which skips the specified number of lines in the input file.   It  is
 used  for  recovery  purposes  when a previous SPIBILD task has failed during a batch input
 process.  [See 2.3.6.]
 - the SET STACK SIZE command, to increase the size of the pass stack.  [See  the  description
 for that command in Part 6, or online, EXPLAIN SET STACK SIZE COMMAND.]
 - the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn  off  (CLEAR)  the  default  file
 count  data  displayed at the end of the process, or to request full file count data (SET).
 [See 2.2.4.]

Step 4: ESTABLISH subfile.name

You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT BATCH command will work properly.

Step 5: Issue the INPUT BATCH command.

The INPUT BATCH command in SPIBILD batches the input data into the named subfile:

Example

Here is a typical online SPIBILD session using the INPUT BATCH procedure:

The record that failed to be added (because the value of the OVEN.TEMP element was not an integer) was placed in the exception file, an ORVYL data set called CRUMMY.COOKIES. The other 52 records in the input data set were added successfully.

Handling Problems

Records that fail input processing

The most common problem that occurs was shown above: one or more records fail to go into the subfile because an Inproc processing rule rejects them. The solution is to find the rejected records, correct them, and try the procedure again with only those records in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected records there. [See 2.1.5.2.]

System failure

System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.

If the failure happens prior to the INPUT BATCH command, you should just start the entire procedure over again when you can.

If it happens during the INPUT BATCH command, you will want to determine the last record that made it through the entire process (i.e., that was added and indexed). All records added before the last "-Begin passing:" message that's not followed by a "Completed passing..." message are in the subfile; all those added after it will be discarded during file recovery. You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. You may either delete the successful records from the front of the input data set or skip them using the SET SKIP command, but in either case, you should restart the procedure, omitting the records that previously went in successfully. [See 2.3.6.]

2.3.2  The INPUT ADD, ADDUPDATE and ADDMERGE Procedures

This procedure allows you to add multiple records to a subfile at once using SPIBILD. The input data will be read by a customized input format. See the manual "SPIRES Formats" for information about input formats for multiple records; online, [EXPLAIN FORMATS, FOR MULTIPLE-RECORD INPUT.]

INPUT ADDUPDATE and INPUT ADDMERGE

In addition, by using either the INPUT ADDUPDATE or INPUT ADDMERGE command, you can tell SPIBILD not to reject a record in the input data if its key matches the key of a record already in the subfile, but instead to use the input data to update that record. In the case of INPUT ADDUPDATE, the input data for the record completely replaces the record already in the subfile; for INPUT ADDMERGE, the input data is merged into the record already there. Particularly for INPUT ADDMERGE, you may need to include some special processing techniques within the input format, which are described in the "SPIRES Formats" manual; online, [EXPLAIN FORMATS, FOR ADDUPDATE COMMAND.]

You may use these procedures either online or in batch. [See 2.1.3.] If using one in batch, you will need to modify the steps of the procedure to fit batch, rather than interactive, work.

Step 1: SPIBILD

To invoke SPIBILD, issue the SPIBILD command online, or in a BATWYL job:

In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.

Step 2: Prepare the input data set.

If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set.

Step 3: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIBILD displays.  The command  "SET
 INFORMATION  MESSAGES  =  0"  is  especially useful, since it turns off the individual
 "add"    message    displayed    for    each     new     record.      [See     2.1.5.3.]
 - the SET EXCEPTION command, to name an ORVYL data set into which rejected records  from  the
 input data set are placed.  [See 2.1.5.2.]
 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the commands that limit the record-types passed from and to: SET INDEX,  SET  NOINDEX,  SET
 PASS  and  SET  NOPASS.   Remember,  however,  that  they are primarily meant to solve file
 integrity problems and should be used only after consultation with your SPIRES  consultant.
 [See 2.1.5.4, 2.4.3.2.]
 - the SET SKIP command, which skips the specified number of lines in the input file.   It  is
 used  for  recovery  purposes  when a previous SPIBILD task has failed during a batch input
 process.  [See 2.3.6.]
 - the SET STACK SIZE command, to increase the size of the pass stack.   See  the  description
 for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
 - the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn  off  (CLEAR)  the  default  file
 count  data  displayed at the end of the process, or to request full file count data (SET).
 [See 2.2.4.]
 - the SET PTRACE command, in one or more of its forms, to  display  tracing  information  for
 input  or  output  processing  rules  (Inproc, Inclose, Outproc) executed during the record
 input processing.

Step 4: ESTABLISH subfile.name

You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT command will work properly.

Step 5: Set the input format.

Depending on what the format does, any of the commands listed below may be involved. All except SET FORMAT are optional; those you use should be issued in the order shown:

 - ALLOCATE.   This command allocates a vgroup, usually so that stored values can be  restored
 to it (see RESTORE STATIC).
 - RESTORE STATIC.   This  command  restores  previously  stored  values  to  a  vgroup.   See
 "SPIRES  Protocols"  for  more  information  about  these  two  commands; online,
 [EXPLAIN ALLOCATE.] or [EXPLAIN RESTORE STATIC.]
 - DEFINE AREA and ASSIGN AREA.   These commands (as well as CLOSE AREA to be issued after the
 INPUT command) are needed when the input  format  is  also  creating  a  report.   [See  2.1.5.5.]
 - SET FORMAT.   The SET FORMAT command sets the named format, or, if the  "*"  form
 of  the command is used, executes the startup frame of the format named in the previous SET
 FORMAT command.  You must start with a "SET FORMAT format-name" command, with one
 or more SET FORMAT * commands optionally to follow.
 - SET FTRACE.   SET FTRACE enables format-tracing messages, most often  used  for  debugging.
 [See "SPIRES Formats", section B.7.2.]

Step 6: Issue the appropriate INPUT command.

Issue the INPUT ADD, INPUT ADDUPDATE or INPUT ADDMERGE command to batch the input data into the named subfile:

Example

Here is a typical online SPIBILD session using the INPUT ADD procedure:

Handling Problems

Records that Fail Input Processing

The most common problem that occurs is that one or more records fail to go into the subfile because an Inproc processing rule rejects them. The solution is to find the rejected records, correct them, and try the procedure again with only those records in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected records there. [See 2.1.5.2.]

System Failure

System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.

If the failure happens prior to the INPUT command, you should just start the entire procedure over again when you can.

If it happens during the INPUT command, you will want to determine the last record that made it through the entire process (i.e., that was added and indexed). All records added before the last "-Begin passing:" message that's not followed by a "-Completed passing..." message are in the subfile; all those added after it will be discarded during file recovery. You may find the SHOW RECORD RECOVERY command helpful in determining which records did and didn't succeed in getting into the subfile. You may either delete the successful records from the front of the input data set or skip them using the SET SKIP command, but in either case, you should restart the procedure, omitting the records that previously went in successfully. [See 2.3.6.]

2.3.3  The LOAD Procedure in SPIBILD

The LOAD procedure is useful when the records you want to add to a subfile are already in a SPIRES subfile with identical record characteristics. You may want to move records from the test version of a file to the production version. Or you might want to rebuild a file, using the records that are already in it.

You may want to compare this technique with the INPUT LOAD procedure, described in the next section, which works in SPIRES as well as SPIBILD [See 2.3.3a.] and with the FAST LOAD procedure in SPIBILD. [See 2.3.4.]

Avoids Outproc and Inproc processing

With this technique, you create an ORVYL data set containing the internal, stored form of the desired records. From there, the records can be loaded into another subfile in SPIBILD. This technique avoids the overhead of running the element values in the records through Outproc rules (as the data set is created) and Inproc and Inclose rules (as the data set is read back in), which can lead to substantial savings in processing time and costs.

How "identical" the record-types must be

The two record-types must share the same internal structure. That means that the way any given record would be stored in one must be identical to the way it would be stored in the other. One easy way to test the relationship between the two is to select each subfile and compare the displays from the SHOW ELEMENT CHARACTERISTICS command. The element names, displayed in the last column, do not have to match, but everything else shown should match, row for row. [Technically, there are some allowable differences besides element names. In theory, the element types do not have to match (except for the "Struc" (Structure) type). But such theories don't usually make good practice.] If they do not match, then the load will probably fail.

You must have See access or better to the file containing the data in order to use the GENERATE LOAD command, and you must have Process access to the receiving file in order to use the LOAD command.

Here are the steps of the procedure:

Step 1: Create the input data set with GENERATE LOAD.

To create the load of records, you need to select the subfile containing the data in SPIRES.

Next, establish a Global FOR class containing the records you want. You must begin with a "FOR class" command, such as FOR SUBFILE, that may contain VIA and WHERE clauses. You may also issue SET SCAN commands and SET FILTER commands with "TYPE = SCAN" to further limit the records to be retrieved. [See the manuals "Sequential Record Processing in SPIRES: Global FOR" and "SPIRES Technical Notes" for more information about these commands.]

Next, issue the GENERATE LOAD command:

where "loadname" is the name of an ORVYL data set. The options at the end are the standard ones for a Global FOR command, specifying which records are to be processed. The default is ALL, for all the records in the established class. See Part 6, or online, [EXPLAIN GENERATE LOAD.]

Example

Here, for instance, is a sample session that creates a load of all Stanford records in a subfile:

Handling problems from the GENERATE LOAD command

The most likely problems you could have with GENERATE LOAD are these:

Before continuing with the procedure...

At this point, you could move the load data set to a tape using the ORVCOPY program. Later, when you are ready to load it into the receiving file, you restore it to an online ORVYL data set, again using ORVCOPY. [See 2.1.4.3.]

Note too that if you need to examine the records in the load at a later time, for example, to see the keys of the records there, you can use the FOR LOAD command in Global FOR to process them.

By the way, in step 4, you will need to know the name of the file and the name of the record-type into which the data is being loaded. If you do not know that information and are about to proceed to step 2, you should select the subfile you want to load the data into, and issue the SHOW SUBFILE INFORMATION command, which will tell you the name of the file, and the name of the goal record-type (listed under "Record") that you will specify in the LOAD command.

Step 2: SPIBILD

Issue the SPIBILD command online, or if you want to load the data via a BATWYL job, place it in the command stream.

If you are loading the data via a batch SPIBILD job, the program puts you in SPIBILD automatically, so you can omit this command.

Step 3: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIBILD displays.  The command  "SET
 INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add"
 message displayed for each new record.  [See 2.1.5.3.]
 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the commands that limit the record-types passed from and to: SET INDEX,  SET  NOINDEX,  SET
 PASS  and  SET  NOPASS.   Remember,  however,  that  they are primarily meant to solve file
 integrity problems and should be used only after consultation with your SPIRES  consultant.
 [See 2.1.5.4, 2.4.3.2.]
 - the SET SKIP command, which skips the specified number of records in the load  of  records.
 It  is  used for recovery purposes when a previous LOAD command failed during a batch input
 process.  [See 2.3.6.]
 - the SET STACK SIZE command, to increase the size of the pass stack.   See  the  description
 for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
 - the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn  off  (CLEAR)  the  default  file
 count  data  displayed at the end of the process, or to request full file count data (SET).
 [See 2.2.4.]

Note that you can use neither an input format nor an exception file with the LOAD procedure.

Step 4: Issue the LOAD command.

The LOAD command loads the data into the desired file:

where "loadname" is the name specified in the GENERATE LOAD command (see step 1) and "filename" and "record-name" are the names of the file and the particular record-type into which the data is to be loaded.

The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).

The LOAD command begins by processing the named file. Then it loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN LOAD COMMAND.]

Example

Here is a continuation of the example started above:

Note that the messages displayed from the LOAD command are identical to those from the BATCH command. The individual "add" messages for the records were suppressed in this example thanks to the SET MESSAGES command just prior to the LOAD command. [See 2.2.4 for information about the other messages shown.]

Handling Problems with the LOAD command

Below are described some problems that occasionally arise during LOAD processing.

S311 error

An S311 error message indicates that the characteristics of the record-type into which you are trying to load the data don't match the characteristics of the record-type containing the original data. Though some minor differences may exist, such as differences in element names, the two record-types should have practically identical internal structures, as described at the start of this section. If they do not, the LOAD command will fail.

S413 error

An S413 error message indicates that an attempt was made to load a record with a key that matches a record already in the record-type. LOAD works only with new records, not updates of existing ones. If this problem occurs with slot records, you can tell SPIBILD to ignore the provided keys of the incoming records and replace them with new slot numbers by adding the RENUMBER option to the LOAD command.

System crash during file-processing

If the system crashes during the initial processing of the file, i.e., after you have issued the LOAD command but before the loading has begun, simply restart the procedure from step 2. Alternatively, you might want to use the PROCESS/RECOVER procedure to recover the file first, and then continue this procedure from step 2. [See 2.1.4.2.]

System crash during the record load

 - you can start over again from step 2, ignoring the  S413  errors  you  would  get  for  the
 records  previously  added,  [But if you used the RENUMBER option to tell SPIBILD to assign
 new slot keys to the records, you will not  get  any  S413  errors  when  you  restart  the
 procedure; the same records will go in again.]
 - you can determine how many records of the load made it into the subfile and  then  use  the
 SET  SKIP  command  in  SPIBILD to tell SPIBILD how many records in the load to skip before
 starting the load again.  [See 2.2.5.]
 - you can determine which records made it into the subfile and then create a  new  load  data
 set  without  those  records  (i.e.,  start again at step 1).  You may find the SHOW RECORD
 RECOVERY command helpful in determining which records did and  didn't  succeed  in  getting
 into  the  subfile.   Alternatively, you could return to SPIRES, select the subfile and use
 various searching techniques to determine which was the last record to  get  into  the  new
 subfile.

2.3.3a  The INPUT LOAD Procedure

The INPUT LOAD procedure is useful when the records you want to add to a subfile are already in a SPIRES subfile with identical record characteristics. You may want to move records from the test version of a file to the production version, or vice versa. Or you might want to rebuild a file, using the records that are already in it.

Differences from LOAD procedure

This technique is quite similar to the LOAD procedure in SPIBILD, but has these differences:

 - It is available in SPIRES as well as SPIBILD, meaning  it  could  be  incorporated  into  a
 protocol, or used with temporary files.
 - In SPIRES, it has the WITH FAST and WITH SEARCHONLY options, which speed up the  processing
 considerably; they are not available to the LOAD procedure.
 - It has an UPDATE option, whose effect is to make  SPIRES  treat  the  incoming  records  as
 "addupdates"  rather  than  just  "adds"; the LOAD procedure can handle
 only adds.

Avoids Outproc and Inproc processing

With this technique, you create an ORVYL data set containing the internal, stored form of the desired records. From there, the records can be loaded into another subfile in SPIRES or SPIBILD. This technique avoids the overhead of running the element values in the records through Outproc rules (as the data set is created) and Inproc and Inclose rules (as the data set is read back in), which can lead to substantial savings in processing time and costs.

How "identical" the record-types must be

The two record-types must share the same internal structure. That means that the way any given record would be stored in one must be identical to the way it would be stored in the other. One easy way to test the relationship between the two is to select each subfile and compare the displays from the SHOW ELEMENT CHARACTERISTICS command. The element names, displayed in the last column, do not have to match, but everything else shown should match, row for row. [Technically, there are some allowable differences besides element names. In theory, the element types do not have to match (except for the "Struc" (Structure) type). But such theories don't usually make good practice.] If they do not match, then the load will probably fail.

You must have See access or better to the file containing the data in order to use the GENERATE LOAD command, and you must have Process access to the receiving file in order to use the LOAD command.

Choosing between SPIRES and SPIBILD

This procedure is probably most commonly used in SPIRES because of the first two factors listed above that distinguish it from the LOAD procedure in SPIBILD. In SPIBILD, it is very similar to the LOAD procedure; the only significant difference is that INPUT LOAD has the ability to handle record updates as well as adds. A relatively insignificant difference is that with INPUT LOAD, you only need to know the name of the subfile into which you want to load the records; with LOAD, you must know the name of the file and the goal record-type.

In choosing between SPIRES and SPIBILD, remember that SPIRES will leave the records in the deferred queue for subsequent processing. SPIBILD will process the file, put the new records into the tree and leave the deferred queue empty.

If you choose to follow the procedure in SPIRES, here are the steps. (The steps for the SPIBILD procedure appear afterwards.)

Step 1: Create the input data set with GENERATE LOAD.

To create the load of records, you need to select the subfile containing the data in SPIRES.

Next, establish a Global FOR class containing the records you want. You must begin with a "FOR class" command, such as FOR SUBFILE, that may contain VIA and WHERE clauses. You may also issue SET SCAN commands and SET FILTER commands with "TYPE = SCAN" to further limit the records to be retrieved. [See the manuals "Sequential Record Processing in SPIRES: Global FOR" and "SPIRES Technical Notes" for more information about these commands.]

Next, issue the GENERATE LOAD command:

where "loadname" is the name of an ORVYL data set. The options at the end are the standard ones for a Global FOR command, specifying which records are to be processed. The default is ALL, for all the records in the established class. See Part 6, or online [EXPLAIN GENERATE LOAD.]

Example

Here, for instance, is a sample session that creates a load of all Stanford records in a subfile:

Handling problems from the GENERATE LOAD command

The most likely problems you could have with GENERATE LOAD are these:

Before continuing with the procedure...

At this point, you could move the load data set to a tape using the ORVCOPY program. Later, when you are ready to load it into the receiving file, you restore it to an online ORVYL data set, again using ORVCOPY. [See 2.1.4.3.]

Note too that if you need to examine the records in the load at a later time, for example, to see the keys of the records there, you can use the FOR LOAD command in Global FOR to process them.

If you intend to use the WITH FAST option on the INPUT LOAD command, you may want to use the PROCESS/ESTABLISH procedure at this point to make sure the deferred queue is empty. [See 2.2.2.] You may also want to take whatever steps are needed to prevent any other users from trying to add records to the subfile at the same time as you are doing so -- WITH FAST requires and empty deferred queue and exclusive update access to the file, and users attempting to add records will be blocked. (This entire paragraph is relevant only under WITH FAST processing. See the description of the INPUT LOAD command in Part 6 for information about its value; online, [EXPLAIN INPUT LOAD COMMAND.] )

Step 2: SELECT subfile.name

If you are continuing immediately from step 1, then you are already in SPIRES; but if not, issue the SPIRES command. Then select the target subfile, that is, the subfile into which you are loading the records.

Step 3: Issue any special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIRES displays.  The  command  "SET
 INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add"
 or    "update"    message    displayed    for   each   new   record.    [See   2.1.5.3.]
 - the SET SKIP command, which skips the specified number of records in the load  of  records.
 It  is  used for recovery purposes when a previous INPUT LOAD command failed during a batch
 input process.  [See 2.3.6.]

Note that you can use neither an input format nor an exception file with the INPUT LOAD procedure.

Step 4: Issue the INPUT LOAD command.

The INPUT LOAD command loads the data into the desired file:

where "loadname" is the name specified in the GENERATE LOAD command.

The WITH FAST and WITH SEARCHONLY options request faster, more efficient processing of the data at the cost of you accepting several restrictions. Details are discussed in the reference section on INPUT LOAD in the back of this manual. [See Part 6, or EXPLAIN LOAD COMMAND for more information.]

The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).

By default, only new records are allowed to be loaded with the INPUT LOAD command -- if a record being loaded has the same key as a record already in the subfile, then the new one is rejected. The UPDATE option tells SPIRES to treat the new one as an update of the record already there, in essence turning the "add" into an "addupdate". That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN LOAD COMMAND.]

Example

Here is a continuation of the example started above:

Note that the messages displayed from the INPUT LOAD command are basically identical to those from the BATCH command. The individual "add" and "update" messages for the records were suppressed in this example, thanks to the SET MESSAGES command just prior to the INPUT LOAD command. [See 2.2.4 for information about the other messages shown.]

If you have problems with the INPUT LOAD command, see the discussion at the end of this section.

Step 5: Cleanup: Other considerations

Once you are certain that all went well, you will probably want to discard the load file you generated in step 1.

If you used the WITH FAST option on INPUT LOAD in step 4, then if your file has immediate indexing, no further updates to the subfile will be allowed until the file is processed. You may want to process the file at this point to restore it to its normal state. [See 2.2.2.]

If you choose to follow the procedure in SPIBILD, here are the steps.

Step 1: Create the input data set with GENERATE LOAD.

Follow the instructions under step 1 above of the SPIRES version of the procedure.

Step 2: SPIBILD

Issue the SPIBILD command online, or to load the data via a BATWYL job, place it in the command stream.

If you are loading the data via a batch SPIBILD job, the program puts you in SPIBILD automatically, so you can omit this command.

Step 3: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIBILD displays.  The command  "SET
 INFORMATION MESSAGES = 0" is especially useful, since it turns off the "add"
 or    "update"    message    displayed    for   each   new   record.    [See   2.1.5.3.]
 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the commands that limit the record-types passed from and to: SET INDEX,  SET  NOINDEX,  SET
 PASS  and  SET  NOPASS.   Remember,  however,  that  they are primarily meant to solve file
 integrity problems and should be used only after consultation with your SPIRES  consultant.
 [See 2.1.5.4, 2.4.3.2.]
 - the SET SKIP command, which skips the specified number of records in the load  of  records.
 It  is  used for recovery purposes when a previous INPUT LOAD command failed during a batch
 input process.  [See 2.3.6.]
 - the SET STACK SIZE command, to increase the size of the pass stack.   See  the  description
 for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]
 - the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn  off  (CLEAR)  the  default  file
 count  data  displayed at the end of the process, or to request full file count data (SET).
 [See 2.2.4.]

Note that you can use neither an input format nor an exception file with the INPUT LOAD procedure.

Step 4: ESTABLISH subfile.name

You issue the ESTABLISH command to identify which subfile you will be using, just as you would issue the SELECT command in SPIRES. Besides selecting the subfile, however, the ESTABLISH command also causes the file to be processed, which is necessary so that the subsequent INPUT LOAD command will work properly.

Step 5: Issue the INPUT LOAD command.

The INPUT LOAD command loads the data into the desired file:

where "loadname" is the name specified in the GENERATE LOAD command.

The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).

By default, only new records are allowed to be loaded with the INPUT LOAD command -- if a record being loaded has the same key as a record already in the subfile, then the new one is rejected. The UPDATE option tells SPIBILD to treat the new one as an update of the record already there, in essence turning the "add" into an "addupdate".

The INPUT LOAD command loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN INPUT LOAD COMMAND.]

Example

Here is a continuation of the example started above:

Note that the messages displayed from the INPUT LOAD command are identical to those from the BATCH command. The individual "add" or "update" messages for the records were suppressed in this example thanks to the SET MESSAGES command prior to the INPUT LOAD command. [See 2.2.4 for information about the other messages shown.]

Handling Problems with the INPUT LOAD command

Below are described some problems that occasionally arise during INPUT LOAD processing.

S311 error

An S311 error message indicates that the characteristics of the record-type into which you are trying to load the data don't match the characteristics of the record-type containing the original data. Though some minor differences may exist, such as differences in element names, the two record-types should have practically identical internal structures, as described at the start of this section. If they do not, the INPUT LOAD command will fail.

S413 error

An S413 error message indicates that an attempt was made to load a record with a key that matches a record already in the record-type. By default, INPUT LOAD works only with new records, not updates of existing ones. If this problem occurs with slot records, you can tell SPIRES to ignore the provided keys of the incoming records and replace them with new slot numbers by adding the RENUMBER option to the INPUT LOAD command. You may alternatively want to use the UPDATE option to tell SPIRES to use the load copy of the record as an update to the copy already in the file.

System crash during file-processing

In SPIBILD, if the system crashes during the initial processing of the file, i.e., after you have issued the LOAD command but before the loading has begun, simply restart the procedure from step 2. Alternatively, you might want to use the PROCESS/RECOVER procedure to recover the file first, and then continue this procedure from step 2. [See 2.1.4.2.]

System crash during the record load

In SPIBILD, if the system crashes during the actual loading, records added before the last "-Begin passing" message that's not followed by a "-Completed passing..." message are in the subfile; those added after it are in the deferred queue but will be discarded during file recovery. You have three choices:

 - you can start over again from step 2 of the procedure you were  using,  ignoring  the  S413
 errors  you  would  get for the records previously added.  [But if you used the RENUMBER or
 UPDATE option to tell SPIBILD to assign new slot keys to the records, you will not get  any
 S413 errors when you restart the procedure; the same records will go in again.]
 - you can determine how many records of the load made it into the subfile and  then  use  the
 SET  SKIP  command  in  SPIBILD to tell SPIBILD how many records in the load to skip before
 starting the load again.  [See 2.2.5.]
 - you can determine which records made it into the subfile and then create a  new  load  data
 set  without  those  records  (i.e.,  start again at step 1).  You may find the SHOW RECORD
 RECOVERY command helpful in determining which records did and  didn't  succeed  in  getting
 into  the  subfile.   Alternatively, you could return to SPIRES, select the subfile and use
 various searching techniques to determine which was the last record to  get  into  the  new
 subfile.

In SPIRES, if you included the WITH FAST option on the INPUT LOAD command, you may want to simply ZAP DEFQ and start again from step 2; the only records in the deferred queue are the ones you put there. In fact, even if you didn't use the WITH FAST option, it may be worth checking with SHOW SUBFILE TRANSACTIONS to see if yours are the only records in the deferred queue. (Remember though that other record-types in the file may have defq data too, which would be lost with a ZAP DEFQ command.)

2.3.4  The FAST BATCH and FAST LOAD Procedures in SPIBILD

The FAST BATCH and FAST LOAD procedures provide the fastest "easy" method of getting new records into a subfile, at the loss of two features common to most other methods:

 - FAST BATCH / FAST LOAD does not checkpoint the input data, meaning that the process must be
 completely redone in the event of a system failure; and
 - FAST BATCH / FAST LOAD does not build indexes from the  input  data.   That  must  be  done
 separately, using the BUILD command.

Procedures split the work into smaller pieces

These procedures are especially useful when you want to build a new goal record-type with thousands of records. They let you split the work into two separate pieces: building the goal record-type and building its indexes. This is critical in situations where the amount of time available in a single computer session is less than the amount needed to build and index the record-type at one time with the INPUT BATCH procedure in SPIBILD.

To use FAST BATCH or FAST LOAD, you must meet several other restrictions too:

 - The record-type into which the data is being added must be completely empty, unless it is a
 slot record-type.
 - The records in the input data must either be in ascending key order, or  have  no  keys  at
 all,  letting  SPIBILD  assign  them  (e.g.,  slot  keys).   [If  the record-type is a slot
 record-type and the input data contains slot key values, then the records do not have to be
 in ascending key order, but the input processing will be less efficient than if they were.]
 - All input must be ADD requests; this process cannot handle updates or removes.
 - For FAST LOAD, the input data must already be in the internal form of SPIRES records, in  a
 file  created  by  the GENERATE LOAD command; the record characteristics of the record-type
 the records came from must match those of the record-type into which you want to FAST  LOAD
 them.   [See  2.3.3 for further information about the "load"
 type of batch input in general.]

Preliminary Work for FAST LOAD

If you are using the FAST LOAD procedure, you need to prepare the data set containing the input. Follow step 1 in "LOAD Procedure in SPIBILD" to generate the load file. [See 2.3.3.]

Step 1: SPIBILD

Issue the SPIBILD command online or in a BATWYL job:

In a batch SPIBILD job you are already in SPIBILD, so you omit this command from the command stream.

Step 2: Process the file with the PROCESS/ESTABLISH procedure.

This step is only necessary if the file has other record-types in which data already exists. It is important to ensure that no passing between other record-types of the file would be attempted when the FAST BATCH command is executing.

Follow the PROCESS/ESTABLISH procedure described earlier in this manual. To execute its most simple form, simply issue the PROCESS command:

where "gg.uuu" is the file owner's account, which must be specified if you're processing someone else's file. [See 2.2.2.]

Step 3: Prepare the input data set.

FAST LOAD

For FAST LOAD, you can skip this step; your input is ready to use.

FAST BATCH

For FAST BATCH, if you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file.

The data may be in the standard SPIRES format, or it may be formatted in some other way, as long as you have an input format that can be used to read it. Always make certain the format works properly before trying to use it to batch hundreds of records into a subfile; a minor failure can be expensive when multiplied by hundreds of records. [See the manual "SPIRES Formats" for information about creating input formats.]

For FAST BATCH in batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set. If the data set is stored online, you may issue a USE command in the JCL at this point; otherwise, be sure to include a DD statement for the data set, to which you will refer in the FAST BATCH command later. [See 2.1.3.2.]

Unless you are planning to use the "ON ddname" option in batch SPIBILD, be sure that the input data set is in your active file at this point.

Data should be in ascending key order

The records in the input data set should either be in order by key, or have their keys assigned by SPIRES as they enter the data base, e.g., slot keys. Records that are out of order will be rejected (with the exception of records with keys that are going into a slot record-type, which can be handled in any order).

Step 4: SET NOPASS

You must specifically tell SPIBILD not to try to pass index data during the FAST BATCH / FAST LOAD process. The best way is to issue the SET NOPASS command in SPIBILD.

Another method is to add the NOPASS statement to the linkage section for the record-type in the file definition and recompile it. When you are done with the FAST BATCH / FAST LOAD procedure, remove the statement from the file definition and recompile it again.

Step 5: Set an input format, if necessary.

For FAST LOAD, skip this step.

For FAST BATCH, if the data will be read through a SPIRES input format, issue the SET FORMAT command at this point; otherwise, skip to step 6. Depending on what the format does, any of the commands listed below may be involved. All are optional; those you use should be issued in the order shown:

 - ALLOCATE.   This command allocates a vgroup, usually so that stored values can be  restored
 to it (see RESTORE STATIC).
 - RESTORE STATIC.   This command  restores  previously  stored  values  to  a  vgroup.   [See
 "SPIRES Protocols" for more information about these two commands.]
 - DEFINE AREA and ASSIGN AREA.   These commands (as well as CLOSE AREA to be issued after the
 FAST BATCH command) are needed when the input format is also creating a  report.   [See  2.1.5.5.]
 - SET FORMAT.   The SET FORMAT command sets the named format, or, if the  "*"  form
 of  the command is used, executes the startup frame of the format named in the previous SET
 FORMAT command.  You must start with a "SET FORMAT format-name" command, with one
 or more SET FORMAT * commands optionally to follow.
 - SET FTRACE.   SET FTRACE enables format-tracing messages, most often  used  for  debugging.
 [See "SPIRES Formats", section B.7.2.]

Step 6: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIBILD displays.  The command  "SET
 INFORMATION  MESSAGES  =  0"  is  especially useful, since it turns off the individual
 "add"    message    displayed    for    each     new     record.      [See     2.1.5.3.]
 - the SET EXCEPTION command, to name an ORVYL data set into which unacceptable  records  from
 the  input  data  set  are placed (FAST BATCH only).  [See 2.1.5.2.]
 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the SET SKIP command, which skips the specified number of lines  (FAST  BATCH)  or  records
 (FAST  LOAD) in the input file.  Though normally used for recovery purposes, it can be used
 here if you simply want to skip a given number of lines or records of the input  data  set.
 For FAST BATCH, it is only useful when the input is in the standard SPIRES format.  [See 2.3.6.]
 - the CLEAR FILE COUNTS or SET FILE COUNTS command, to turn  off  (CLEAR)  the  default  file
 count  data  displayed at the end of the process, or to request full file count data (SET).
 [See 2.2.4.]
 - the SET PTRACE command, in one or more of its forms, to  display  tracing  information  for
 input  or  output  processing  rules  (Inproc, Inclose, Outproc) executed during the record
 input processing.
 - the SET STACK SIZE command, to increase the size of the pass stack.   See  the  description
 for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]

Step 7: Issue the FAST BATCH or FAST LOAD command.

The FAST BATCH command processes the file and batches the input data into the named subfile:

The FAST LOAD command processes the file and loads the records from the named ORVYL data set into the named record-type of the named file:

where "loadname" is the name specified in the GENERATE LOAD command (see Preliminary Work above) and "filename" and "record-name" are the names of the file and the particular record-type into which the data is to be loaded.

The RENUMBER option is useful when both record-types are SLOT. When RENUMBER is specified, the original slot numbers of the loaded records are discarded, and new slot numbers are assigned. Otherwise, the original slot numbers are retained. In that case, if any of the new records has the same slot number as a record already there, the new record will be rejected (see below).

The FAST LOAD command begins by processing the named file. Then it loads the records one by one into the record-type, bypassing all Inproc and Inclose processing rules. That means, for instance, that "date.updated" and "time.updated" elements will not be updated; no data in the records will change at all, with the possible exception of slot keys, as described above. See Part 6 for more information, or online, [EXPLAIN FAST LOAD COMMAND.]

Example

Here is a sample session showing typical use of the FAST BATCH procedure:

Notice how very little processing time (0.85 seconds) was involved to add 500 records; while the records that generated this example are admittedly small, the "compute time" needed was low because none of the backup work SPIBILD normally does to allow recovery was done here.

The FAST LOAD procedure would look exactly the same except for the USE command, which would not be there, and the FAST BATCH command, which would be replaced by a FAST LOAD command, such as:

Step 8: Build the indexes, if any.

If the record-type has any indexes, you will need to "build" them at some point. The index-building procedure is described in detail later. [See 2.7.1, steps 4 through 7.] Remember if you build the indexes in the same SPIBILD session to "turn on" passing again, e.g., using the SET PASS command.

The example above continues, as we build the indexes.

Handling Problems with the FAST BATCH command

The most likely problems you could have with FAST BATCH are these:

Handling Problems with the FAST LOAD command

Most of the problems described above for FAST BATCH could happen for FAST LOAD as well, and are handled the same. Also, problems related to the load input data could arise; they are the same as the problems described in the previous section on the LOAD procedure. [See 2.3.3.] Remember, however, that restarting FAST LOAD will require you to zap the record-type, so that it will be empty when you start (unless you are working with records having slot keys). Alternatively, you may want to retain the records that did get in, build the indexes with what you have (step 8) and resume the load with the LOAD procedure, which does not require an empty record-type to start with.

2.3.5  Formats for Input Data

Most of the procedures for adding records to a SPIRES subfile allow the input data to be formatted so that it can be read by either the standard SPIRES format or by a custom input format. Custom formats are discussed in detail in the manual "SPIRES Formats", in section C.9 in particular.

This section presents a summary of the standard SPIRES format, discussed in detail in the SPIRES manual "Searching and Updating", sections D.2.2, D.2.3, D.2.6 and D.8.1.

Keep in mind that some of the procedures discussed in this chapter (2.3) limit their scope to added records, to which some of this information may not apply.

Basic syntax for individual elements

The basic syntax of individual element values is:

     element-name = value;

where "element-name" is the name of the element, and "value" is the data value to be assigned to that element. The equals sign is optional; the semicolon at the end is required.

Here are other important rules to remember:

Long values may wrap

     COMMENT = SPIRES will put a blank after the comma,
     so that the comma and 'so' don't get run together.;

Multiple elements on one line

     NAME = Merrick Butte; COMMENTS = Breeder;

Semicolons

     COMMENTS = "This semicolon; is not really needed.";

Quotation marks

     COMMENTS = "See ""SPIRES Formats"" for details.";

Null values

     COMMENTS;

Structures

     MEMBER;
       NAME = Jenny Lake;
       STATE = Wyoming;
     MEMBER;
       NAME = Mitchell Butte;
       STATE = Utah;

Merge requests

Commands

     ADD;
     ADDUPDATE;
     ADDMERGE;
     UPDATE key;
     MERGE key;
     REMOVE key;

Remove requests

     REMOVE 94025;

End each record with an extra semicolon

     MERGE 85-194;
       NAME = Mrs. Osgood Azamile;
     ;
     REMOVE 83-019;
     ADDUPDATE;
       ID = 86-028;
       NAME = Miss Bea Hayve;
     ;

2.3.6  SPIBILD Recovery to Resume Input

When a SPIBILD command such as INPUT BATCH, LOAD or FAST BATCH terminates in the middle of adding new records to a subfile, you must decide how to recover from the problem and finish doing your input. The procedures described earlier in this chapter for those commands describe how to determine the cause of the problem and make some suggestions for how to solve it.

SET SKIP command

Often, the solution is to somehow determine which record was the last record in the input data to be successfully added to the subfile. Once that is determined, you can delete the input up through that record, or use the SET SKIP command to tell SPIBILD to skip that much input before starting the input procedure again. [Keep in mind for FAST BATCH or FAST LOAD that if you decide to use this method for recovery, you cannot use the FAST procedure when you resume adding records, since the goal record-type must be completely empty in order to use FAST BATCH or FAST LOAD. You can, however, switch to the INPUT BATCH or INPUT ADD or LOAD procedure, or, if not many records were successfully added anyway, consider zapping the record-type and starting over again with FAST BATCH or FAST LOAD. If you do switch to another procedure, build the indexes using the data already in the subfile before you batch more records in. [See 2.7.1, steps 4 through 7.]]

Was the input being indexed?

How you determine the last record to be successfully added depends on whether the records were being indexed or not. For instance, they would not be indexed if:

 - you were using the FAST BATCH / FAST LOAD procedure in SPIBILD; or
 - you had SET NOPASS; or
 - the goal record-type had no indexes.

But if they are indexed, you can take advantage of a pair of handy commands, SHOW RECORD RECOVERY and SET SKIP. We'll start with the discussion for the indexed records, followed by the discussion for the non-indexed records.

The input records are being indexed

SHOW RECORD RECOVERY command

SPIBILD keeps track of the number of lines of input read (for INPUT BATCH, INPUT ADD, INPUT ADDUPDATE and INPUT ADDMERGE) or the number of records read (for LOAD), storing that number away each time a group of records have been completely processed, i.e., added to the tree and fully indexed. If the processing completely finishes, then SPIBILD discards the information. But if the processing is interrupted, you can issue the SHOW RECORD RECOVERY command to show you that number:

The significant number here is the SET SKIP value, the number of lines (or records) of input successfully processed. [The other information is useful in some ways. This information is about the records that are only half-processed, i.e., whose index data was not completely processed. During recovery, SPIBILD will discard those records and "unindex" them if any indexing was done. The data for those records appears in the input data after the number of lines given in the "skip" indicator so it will be input again when you use the SET SKIP command. See Part 6, or online, [EXPLAIN SHOW RECORD RECOVERY COMMAND.]]

When you are ready to resume the SPIBILD work, start with the RECOVER/PROCESS procedure. [See 2.1.4.2.] Then start again the procedure you were using when the problem occurred, incorporating the SET SKIP command at the point where it is mentioned. See Part 6 for more information, or online, [EXPLAIN SET SKIP COMMAND.]

The input is not all ADDs

This procedure is somewhat more complicated when the input for the INPUT BATCH command is a mixture of transaction types, rather than only new records to be added. SET SKIP only registers added records, so the number will not get updated when non-add transactions are being processed, even though they may indeed get completely indexed. The best way to handle this situation is to get the skip number using the SHOW RECORD RECOVERY command and then use the RECOVER/PROCESS procedure to clean up the file.

Next, examine the input data past the number of lines you can skip and, in SPIRES, use the indexes (or simply display records by key) to look for those records. When you find input data that has no matching record, you have found the point where the processing stopped. Then determine how many more lines past the skip number you should tell SPIBILD to skip to begin at that point.

You can discard the successful data

Note too, in both cases, that you may also discard the input that succeeded rather than use SET SKIP to skip over it. SET SKIP is most handy when the input data is not yours to change or is on a tape.

The input records are not being indexed

If the data is not indexed, then SPIBILD does not keep track of the SET SKIP figure. But in fact, recovery is much simpler because there will be no half-processed records that SPIBILD must discard in order to clean up properly.

The solution here is to use the RECOVER/PROCESS procedure to do any file cleanup that might be necessary, and then return to SPIRES to examine the file to determine where the processing stopped. If you had information messages turned on during the processing that failed, you will know from them how far the processing got. If not, you will need to examine records from the input data set, looking for their counterparts in the file. When you can't find one, that is the point where the processing stopped.

You can either delete the input to that point, or determine how many lines of the input should be skipped to begin processing at that point, including that number in a SET SKIP command when you restart the procedure.

2.4  Updating Multiple Records in a Subfile

SPIRES and SPIBILD provide several ways for you to update multiple records in a subfile in a single operation. The different methods work under different circumstances -- which one you choose will depend on the particular situation you have.

Below is a comparative summary of the methods. The procedures involving SPIRES are described in the "SPIRES Searching and Updating" manual. SPIBILD's INPUT BATCH, INPUT ADDUPDATE and INPUT ADDMERGE procedures were described in the previous chapter, but the other SPIBILD procedures are discussed later in this chapter. This chapter also includes a section about how SPIBILD determines when an updated record requires re-indexing. [See 2.4.3.]

A discussion about the format of the data in input data sets used with these procedures also appeared in the previous chapter. [See 2.3.5.]

Methods for Updating Multiple Records in a Subfile

There are twelve techniques for updating multiple records in a subfile in one fell swoop:

 - 1.  (SPIRES) the INPUT BATCH procedure
 - 2.  (SPIRES) the INPUT MERGE procedure
 - 3.  (SPIRES) the INPUT ADDMERGE procedure
 - 4.  (SPIRES) the INPUT ADDUPDATE procedure
 - 5.  (SPIRES) the BATCH procedure
 - 6.  (SPIRES) the MERGE procedure in Global FOR
 - 7.  (SPIRES) the WITH UPDATE procedure
 - 8.  (SPIBILD) the INPUT BATCH procedure
 - 9.  (SPIBILD) the INPUT MERGE procedure
 - 10.  (SPIBILD) the INPUT ADDMERGE procedure
 - 11.  (SPIBILD) the INPUT ADDUPDATE procedure
 - 12.  (SPIBILD) the MERGE procedure in Global FOR

Determining which Procedure to Use

With so many choices, it can be challenging to decide which procedure to use. The following list of questions, along with the descriptions that follow, may help you make up your mind:

Below are descriptions of each of the techniques.

1. (SPIRES) The INPUT BATCH Procedure

You can use the INPUT BATCH command to add new records, modify existing records and remove existing records. SPIRES places the data into the file's deferred queue. When updates, merges or removes are involved, the data must be in the standard SPIRES format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.

The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.

Because the data goes into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records don't go into the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner (or the account that next processes the file), not the user doing the input, who pays for only the input costs.

Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.

See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES. For online help, EXPLAIN INPUT BATCH.

2. (SPIRES) The INPUT MERGE Procedure

You use the INPUT MERGE procedure to merge data into existing records in the subfile. An input format that handles merge processing does most of the work, usually getting the data, including the key of each record to be updated, from an input data set. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.

SPIRES places each updated record in the deferred queue, so unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. Hence, the cost of file processing is not borne by users of the INPUT MERGE procedure.

See the SPIRES manual "Searching and Updating" for more information about the INPUT MERGE command in SPIRES. For online help, EXPLAIN INPUT MERGE.

3, 4. (SPIRES) The INPUT ADDMERGE, ADDUPDATE Procedures

The INPUT ADDMERGE and INPUT ADDUPDATE procedures are very similar to the INPUT ADD precedure, using the input to add new records to the subfile. A custom input format does most of the work, usually getting the data, including the key of each record to be added, from an input data set. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set. If SPIRES finds that the input key matches the key of a record already in the subfile, then the input is either used to replace that record (ADDUPDATE) or to be merged into that record (ADDMERGE).

SPIRES places each updated record in the deferred queue, so unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. Hence, the cost of file processing is not borne by users of these INPUT procedures.

See the SPIRES manual "Searching and Updating" for more information about the INPUT ADDMERGE and INPUT ADDUPDATE commands in SPIRES. For online help, EXPLAIN INPUT ADDMERGE or INPUT ADDUPDATE.

5. (SPIRES) The BATCH Procedure

The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.

This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.

The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user doesn't know till the next day whether the records were accepted, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.

The input data must be in the standard SPIRES format. It can include adds, updates, merges and removes.

A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched; but the request telling JOBGEN where the data is gets thrown away. This problem makes the BATCH command in SPIRES quite unreliable.

For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".

6. (SPIRES) The MERGE procedure in Global FOR

In SPIRES, the MERGE command can update multiple records, determined by the "FOR class" command in effect. The input data may be provided in several ways:

 - it may be in the standard SPIRES format in the active file, with the data  read  only  once
 and applied to each record being merged;
 - it may be formatted input in the active file, with the data read and applied to each record
 being merged;
 - it may be requested from the user by the merge format, or it may be provided by the  format
 itself.

As the MERGE command executes, it merges the data into each record, placing the updated version in the deferred queue. Unless immediate indexes are involved, the indexes are not updated till the next time the file is processed. The cost of file processing is not borne by users of this procedure.

Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic MERGE command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN WITH FAST PREFIX.

See the manual "Sequential Processing in SPIRES: Global FOR" for more information about the MERGE command in Global FOR.

7. (SPIRES) The WITH UPDATE Procedure

WITH UPDATE is a command prefix for the DISPLAY and TYPE commands. Using the WITH UPDATE procedure, you can, to a limited degree:

 - display records, simultaneously updating them; or
 - display records, simultaneously updating other records in other subfiles  or  record-types;
 or
 - update a record, simultaneously updating other records in other subfiles  or  record-types.

The most noteworthy limits of the WITH UPDATE procedure are:

 - You must use a custom format containing special code to handle the extraordinary  updating.
 - The user must have secure-switch 14 set.
 - You cannot add new occurrences or remove old occurrences of elements in the  records  being
 updated;  you  can only replace existing occurrences, with values that are the same lengths
 as the values being replaced.
 - Inproc rules are executed for the replacement values, but not INCLOSE rules.

For more information about this specialized technique, see the manual "SPIRES Formats", chapter C.13, or "EXPLAIN WITH UPDATE PREFIX, RESTRICTIONS" online.

8. (SPIBILD) The INPUT BATCH Procedure

You can use the INPUT BATCH command in SPIBILD to add new records, modify existing records and remove existing records. The INPUT BATCH command applies the data in the input data set (in the standard SPIRES format) directly to the tree. For merges and updates, the INPUT BATCH command usually puts the data from the input data set (in the standard SPIRES format) into the deferred queue, which is then processed after all the input has been read.

The data may include adds, updates, merges and removes. If SPIBILD finds errors in the data, it reports them right away, possibly rejecting the data involved.

The INPUT BATCH procedure in SPIBILD is more expensive than the similar INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.

The input data may be on disk or tape, though if on tape, the procedure must be done through a Batch SPIBILD job.

The steps of this procedure were described in detail in the previous chapter. [See 2.3.1 or EXPLAIN INPUT BATCH PROCEDURE IN SPIBILD online.]

9. (SPIBILD) The INPUT MERGE Procedure

You can use SPIBILD's INPUT MERGE procedure to merge data into existing records in the subfile. [See 2.4.1.] You must use an input format that handles merge processing. It does most of the work, usually getting the data, including the key of each record to be updated, from an input data set. The data to be read may come from your active file or a data set stored on disk or tape.

Processing the file is part of the INPUT MERGE procedure -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. You cannot use this procedure unless your account can process the file.

10, 11. (SPIBILD) The INPUT ADDMERGE, ADDUPDATE Procedures

Like their counterparts in SPIRES, the INPUT ADDMERGE and INPUT ADDUPDATE procedures in SPIBILD are very similar to the INPUT ADD precedure, using the input to add new records to the subfile. A custom input format does most of the work, usually getting the data, including the key of each record to be added, from an input data set. The data to be read must be in your active file or, if you use batch SPIBILD, a data set stored on disk or tape. If SPIRES finds that the input key matches the key of a record already in the subfile, then the input is either used to replace that record (ADDUPDATE) or to be merged into that record (ADDMERGE).

Processing the file is part of these INPUT procedures -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. You can use these procedures only if your account can process the file.

The INPUT ADDMERGE and INPUT ADDUPDATE procedures are discussed along with the INPUT ADD procedure in the previous chapter. [See 2.3.2 or EXPLAIN INPUT ADD PROCEDURE IN SPIBILD online.]

12. (SPIBILD) The MERGE Procedure in Global FOR

This procedure is similar to the INPUT MERGE procedure described above, except that the records to be updated are determined by the Global FOR criteria specified, not by the input data. The input data usually consists of one or more values that are all to be merged into each record. The input data may be in the standard SPIRES format, or a custom input format may be used. [See 2.4.2.]

The data can come from your active file, or, if you are using batch SPIBILD, from a tape or another WYLBUR data set. There may even be no input data set at all -- a custom format can provide the data itself, or it may reference the existing record to make changes based on its current values (such as reducing all occurrences of a PRICE element by half).

Processing the file is part of this procedure too -- the file is processed at the start; and if the data being merged affects any indexes, they are updated too. To use this procedure, your account must be able to process the file.

2.4.1  The INPUT MERGE Procedure in SPIBILD

The INPUT MERGE procedure merges data into records using a custom input format. The input data for each record must contain its key, so that SPIRES can retrieve the record for updating. [To merge data into records identified by criteria other than the key, use the MERGE procedure in Global FOR in SPIBILD. [See 2.4.2.]]

Typically, an input data set might contain a line for each record to be updated, containing the key and a new data value for one or more elements. For example, this could be an input data set for a mythical doctor's billing system, where the data in the first column are keys to records already in the subfile:

What happens to the input data is determined by the input format. It could be added as new data, or as replacement data, or as data that causes existing data to be removed or recomputed (the data above could cause a BALANCE element in each record to be refigured, for instance). Information on creating merge formats appears in chapters C.5 and C.9 of the manual "SPIRES Formats".

Be aware that you can use this procedure only if you have a custom merge format that reads each key and its associated data. If the data to be merged is in the standard SPIRES format, you should simply use SPIBILD's INPUT BATCH procedure. [See 2.3.1, 2.3.5.]

Step 1: SPIBILD

Issue the SPIBILD command online, or in a BATWYL job:

In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL command stream.

Step 2: Prepare the input data set.

If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set on disk.

Be sure the input data set can read the input data and apply it properly to individual SPIRES records. Always make certain the format works properly before trying to use it to merge hundreds of records in a subfile -- a small problem can be quite costly when multiplied by hundreds of records.

Step 3: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIBILD displays.  The command  "SET
 INFORMATION  MESSAGES  =  0"  is  especially useful, since it turns off the individual
 "add"    message    displayed    for    each     new     record.      [See     2.1.5.3.]
 - the SET EXCEPTION command, to name an ORVYL data set into which unacceptable  records  from
 the input data set are placed.  [See 2.1.5.2.]
 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the commands that limit the record-types passed from and to: SET INDEX,  SET  NOINDEX,  SET
 PASS  and  SET  NOPASS.   Remember,  however,  that  they are primarily meant to solve file
 integrity problems and should be used only after consultation with your SPIRES  consultant.
 [See 2.1.5.4, 2.4.3.2.]
 - the SET SKIP command, which skips the specified number of lines in the input file.   It  is
 used  for  recovery  purposes  when a previous SPIBILD task has failed during a batch input
 process.  [See 2.3.6.]
 - the SET STACK SIZE command, to increase the size of the pass stack.  [See  the  description
 for that command in Part 6, or online, EXPLAIN SET STACK SIZE COMMAND.]
 - the SET PTRACE command, in one or more of its forms, to  display  tracing  information  for
 input  or  output  processing  rules  (Inproc, Inclose, Outproc) executed during the record
 input processing.

Step 4: ESTABLISH subfile-name

The ESTABLISH command in SPIBILD "selects" the named subfile and processes the file containing it.

Step 5: Set the input format.

You must set the desired format. Depending on what the format does, any of the commands listed below may be involved. All are optional except for SET FORMAT; those you use should be issued in the order shown:

 - ALLOCATE.   This command allocates a vgroup, usually so that stored values can be  restored
 to it (see RESTORE STATIC).
 - RESTORE STATIC.   This command  restores  previously  stored  values  to  a  vgroup.   [See
 "SPIRES Protocols" for more information about these two commands.]
 - DEFINE AREA and ASSIGN AREA.   These commands (as well as CLOSE AREA to be issued after the
 FAST BATCH command) are needed when the input format is also creating a  report.   [See  2.1.5.5.]
 - SET FORMAT.   The SET FORMAT command sets the named format, or, if the  "*"  form
 of  the command is used, executes the startup frame of the format named in the previous SET
 FORMAT command.  You must start with a "SET FORMAT format-name" command, with one
 or more SET FORMAT * commands optionally to follow.
 - SET FTRACE.   SET FTRACE enables format-tracing messages, most often  used  for  debugging.
 [See "SPIRES Formats", section B.7.2.]

Step 6: INPUT MERGE

The INPUT MERGE command in SPIBILD merges the input data into the subfile according to the specifications of the set format.

Example

Here is a sample online SPIBILD session showing typical usage of the INPUT MERGE procedure:

This example shows a case where the merge data caused indexes to be updated; in some cases, no index data is involved, so the records don't need to be re-indexed, which is somewhat cheaper. [See 2.4.3.1.]

Handling Problems

Records that fail input processing

The most common problem that occurs is that one or more records fail to get updated because an Inproc processing rule rejects the data. The solution is to find the rejected data, correct it, and try the procedure again with only that data in the input data set. This process is made much simpler if you set an exception file so SPIBILD can place the rejected data there. [See 2.1.5.2.]

System failure

System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.

 - simply restart the procedure using the same data.  This is the simplest method,  but  don't
 use  it  if  the merge requests are creating new element or structure occurrences, or, say,
 causing addition into amount elements.  Completely resubmitting such requests  would  cause
 erroneous data in those records that were successfully processed the first time.
 - determine the last record that was updated.   SPIBILD's  information  messages  --  if  you
 didn't  turn  them  off -- can be useful here, but you should check the records actually in
 the file in SPIRES.  This might be tedious, but it is the most efficient method  (it  means
 you  won't  update  the  same  records  again).  You can delete the data for the successful
 records  from  the  front  of  the  input  data  set  and  then  restart   the   procedure.
 Alternatively,  restart  the procedure and include the SET SKIP command to tell SPIBILD how
 many lines of the input to skip in step 3.

2.4.2  The MERGE Procedure in SPIBILD with Global FOR

This MERGE procedure merges data into records determined by the established Global FOR environment. This contrasts with the "normal" MERGE procedure in SPIBILD, where the keys of the records to be updated are contained in the input data set. Under Global FOR, however, an input data set may or may not exist. If it does, it almost always contains pieces of data that are all to be merged into each record, with or without the aid of a custom format. If the input data set doesn't exist, then the data is supplied or computed by a custom format directly.

Information on writing merge formats to be used with Global FOR in SPIBILD appears in section C.9.4 of the manual "SPIRES Formats".

Step 1: Create a stored stack or result, if appropriate.

If you are merging into all the records in the subfile, skip ahead to step 2. In SPIBILD, you can use the FOR SUBFILE command with a WHERE clause to determine which records to update. However, it is much more efficient, if you can do this, to search for the records using indexes in SPIRES, store the result, and then use the FOR STORED command in SPIBILD to identify the records in the stored result. This technique also works for stored stacks that were created in SPIRES.

To do this, create your search result or stack in SPIRES, using commands like FIND, SEQUENCE and STACK. Then, depending on whether you have a result or a stack, issue the STORE RESULT.name or STORE STACK.name command to save the result or stack as an ORVYL data set. Don't use the TEMPORARY option on those commands; the result or stack won't be around long enough for SPIBILD to use. [If you have a result, there is a slim possibility that FOR STORED cannot work with it. To avoid this problem, you may want to turn the result into a stack (e.g., by issuing the commands FOR RESULT and STACK ALL) before storing it. See the technical information on the FOR STORED command in Chapter 6 of this manual; online, [EXPLAIN FOR STORED COMMAND.]]

Step 2: SPIBILD

Issue the SPIBILD command online, or in a BATWYL job:

In a batch SPIBILD job, you are already in SPIBILD, so you omit this command.

Step 3: Prepare the input data set, if any.

If the data to be merged will be read from an input data set, prepare it for use. If you are using online SPIBILD or the batch program BATWYL, you must bring the input data set into your active file. For batch SPIBILD, the data set may be stored on tape or as a WYLBUR data set on disk.

If you are using an input format to read the data, be sure it works properly before trying to use it to update hundreds of records in a subfile -- a small problem can be quite costly when multiplied by hundreds of records.

Step 4: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIBILD displays.  The command  "SET
 INFORMATION  MESSAGES  =  0"  is  especially useful, since it turns off the individual
 "add"    message    displayed    for    each     new     record.      [See     2.1.5.3.]
 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the commands that limit the record-types passed from and to: SET INDEX,  SET  NOINDEX,  SET
 PASS  and  SET  NOPASS.   Remember,  however,  that  they are primarily meant to solve file
 integrity problems and should be used only after consultation with your SPIRES  consultant.
 [See 2.1.5.4, 2.4.3.2.]
 - the SET PTRACE command, in one or more of its forms, to  display  tracing  information  for
 input  or  output  processing  rules  (Inproc, Inclose, Outproc) executed during the record
 input processing.
 - the SET STACK SIZE command, to increase the size of the pass stack.   See  the  description
 for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]

Step 5: ESTABLISH subfile.name

Issue the ESTABLISH command, which processes the file and prepares the subfile for direct update operations.

Step 6: Set the input format, if any.

If you are using an input format, you set it now; otherwise, skip to step 7. Depending on what the format does, any of the commands listed below may be involved. All are optional except for SET FORMAT; those you use should be issued in the order shown:

 - ALLOCATE.   This command allocates a vgroup, usually so that stored values can be  restored
 to it (see RESTORE STATIC).
 - RESTORE STATIC.   This command  restores  previously  stored  values  to  a  vgroup.   [See
 "SPIRES Protocols" for more information about these two commands.]
 - DEFINE AREA and ASSIGN AREA.   These commands (as well as CLOSE AREA to be issued after the
 FAST BATCH command) are needed when the input format is also creating a  report.   [See  2.1.5.5.]
 - SET FORMAT.   The SET FORMAT command sets the named format, or, if the  "*"  form
 of  the command is used, executes the startup frame of the format named in the previous SET
 FORMAT command.  You must start with a "SET FORMAT format-name" command, with one
 or more SET FORMAT * commands optionally to follow.
 - SET FTRACE.   SET FTRACE enables format-tracing messages, most often  used  for  debugging.
 [See "SPIRES Formats", section B.7.2.]

Step 7: Establish the Global FOR environment.

Establish the Global FOR environment by issuing the FOR SUBFILE or FOR STORED command, optionally followed by other Global FOR commands mentioned below. The purpose of these commands is to identify the specific records to be updated.

The FOR SUBFILE command

No VIA clause is allowed on the FOR SUBFILE command, but you may add a WHERE clause as needed, to limit the records to be updated using element criteria:

The FOR STORED command

If you have stored a stack or result containing the records you want to update (see step 1), you would use the FOR STORED command:

The SET SCAN commands

The SET SCAN commands can give you further flexibility (and savings) in determining which records are to be examined for processing. The SET SCAN commands allowed in SPIBILD under FOR SUBFILE are:

Only SET SCAN LIMIT is allowed under FOR STORED. Note that SET SCAN PREFIX has no meaning with slot subfiles.

Details about these commands, including WHERE clauses, appear in the manual "Sequential Processing in SPIRES: Global FOR".

SET FILTER command in SPIBILD

The SET FILTER command, with the TYPE = SCAN option, works similarly to a WHERE clause but is more versatile. You can, for example, use it to locate records when you have complicated criteria involving elements within structures, a situation that WHERE clauses often cannot handle. See section 4.4.2 of the Global FOR manual referenced above; online [EXPLAIN FILTERS, ON WHERE CLAUSES.]

SKIP command in SPIBILD

The SKIP command tells SPIBILD to skip the specified number of records meeting the Global FOR criteria:

where "n" is the number of records to be skipped. If you specify neither "n" nor FIRST (for the FIRST record), the default is FIRST. Except for the limited options, SKIP works the same in SPIBILD as it does in SPIRES. See section 5.10 of the Global FOR manual; online [EXPLAIN SKIP COMMAND.]

Example

This example shows how a WHERE clause and SET SCAN processing may be used to identify the records to be updated in the least expensive manner. Suppose you want to update all the records with ADD.DATE element values of 1986 whose keys begin with the code KL:

These two commands work more efficiently together than would the single command FOR SUBFILE WHERE ADD.DATE 1986 AND ID PREFIX KL. The SET SCAN command limits the records that are examined to those whose "id" begins with KL; but with the single command, SPIBILD would examine all records, looking for those whose "id" begins with KL, which might be hundreds or thousands more records. If you will be using MERGE processing under Global FOR in SPIBILD, you should be familiar with the SET SCAN commands, which can save you a great deal in processing costs.

Note in the example that SPIBILD does not use the Global FOR prompt that SPIRES uses (+?) after you issue the FOR SUBFILE command.

Step 8: Issue the MERGE command.

The MERGE command in SPIBILD merges the input data into the named subfile according to the specifications of the set format.

Again, "n" represents the number of records you want processed; the default is REST. You need to specify the FROM option if you want SPIBILD to read data from the active file (or from a DD data set in batch SPIBILD). But if the input data is being provided by an input format that does not need to read the data from the active file, you should omit the FROM option.

Example

Here is a sample online SPIBILD session showing typical usage of the MERGE procedure in SPIBILD:

This example shows a case where the merge data caused indexes to be updated; in some cases, no index data is involved, so the records don't need to be re-indexed, which is somewhat cheaper. [See 2.4.3.1.]

Handling Problems

Records that fail input processing

Because this procedure usually applies the same data to all the records it updates, if the data fails to go into one record, it usually fails to go into all of them. In most cases where input failure occurs, you just correct the input data and restart the procedure.

System failure

System failure during the procedure can also cause some problems; however, recovery is usually simple, depending on when the failure occurs.

 - simply restart the procedure using the same data.  This is the simplest method,  but  don't
 use  it  if  the merge requests are creating new element or structure occurrences, or, say,
 causing addition into amount elements.  Completely resubmitting such requests  would  cause
 erroneous data in those records that were successfully processed the first time.
 - determine the last record that was updated.   SPIBILD's  information  messages  --  if  you
 didn't  turn  them  off -- can be useful here, but you should check the records actually in
 the file in SPIRES.  This might be tedious, but it is the most efficient method  (it  means
 you won't update the same records again).  You should issue the SKIP command (see above) to
 skip the successful records when you restart the procedure.
 - Since SKIP works only with FOR SUBFILE and not FOR STORED, with FOR STORED processing,  you
 might want to recreate your result or stack in SPIRES, eliminating the successfully-updated
 records,  store  the  new  result  or  stack,  and  restart  the  procedure using that one.

2.4.3  (*) Considering Efficiency for SPIBILD Multiple Update Requests

In most situations where you are updating multiple records in a single pass, the distinctions between the procedures described at the beginning of this chapter are adequate for choosing the right one. And in terms of cost, for smaller files the differences between the procedures are seldom very significant.

For large files or files with somewhat unusual characteristics, however, the differences can be quite significant. Moreover, some extra thought and work on your part may lead to considerable savings that SPIBILD would not provide you on its own.

This section isn't for everybody

The situations covered in the subsections of this section require fairly technical discussions, and this section should be considered optional reading unless:

 - you are concerned about cutting costs any way you can; and
 - at least one of the following is true:
 - you are working with a file containing thousands of records;
 - you will be updating hundreds of records at a time;
 - you are working with extremely large records, containing hundreds of element values, many
 of which are indexed.

The subsections of section 2.4.3

Techniques are available that alter the standard way SPIBILD processes data, to increase efficiency when the input data or the file is different from the norm. The techniques, described in separate sections that follow, are:

 - 1) the SET NOPASS command, useful when merging data in SPIBILD that has no  effect  on  any
 index records; [See 2.4.3.2.]
 - 2) the PASS-ELEM statement in file definitions, which has a similar use but which is  built
 into the file's characteristics; [See 2.4.3.3.]

The first subsection provides a brief overview of how updated records affect indexes, which is useful information to know if you must decide whether to use any of these efficiency techniques.

2.4.3.1  (*) How SPIBILD Updates Indexes for MERGE & UPDATE Requests

SPIRES treats updates and merges the same

In SPIRES, whether records are updated or merged, they are placed in the deferred queue as "updates" -- that is, SPIRES places the complete updated record in the deferred queue, so that it can replace the existing tree copy of the record. Merge data by itself is not stored in the deferred queue; SPIRES applies it to the record immediately, and places the updated record in the deferred queue as an "update".

SPIBILD handles updates and merges differently

SPIBILD has different ways to handle merge and update requests, i.e., when the data is being merged by BATCH or MERGE commands in SPIBILD. For updates, which are handled by the BATCH command only, SPIBILD places each updated record in the deferred queue. As soon as SPIBILD finishes with the input data, it processes the deferred queue again, moving the updated records into the tree (removing the old copies) and determining what changes need to be made to the indexes. [The technical details of how SPIBILD efficiently re-indexes updated records were presented earlier in this manual. In brief, the index data is generated for the old record, each item marked as a "delete", and the index data is generated for the new copy, with each item marked as an "add". SPIBILD sorts the index data, discarding any pairs that are identical except for opposite "add" and "delete" markers. What isn't discarded then gets applied to the indexes. [See 2.2.6.]]

For updates, SPIBILD needs to check the data to see if the indexes need to be changed because it doesn't know what elements, if any, have been changed from one copy of the record to the next; all it knows is that it has a new copy of the record to replace the old one. [When SPIBILD gets an updated record, it is a new copy of the existing record, which may or may not have been changed. SPIBILD assumes the record has been changed, and replaces the old copy with the new one in the tree. During indexing, however, SPIBILD determines which indexed values have not changed from the old record to the new, and discards that data from the pass stack, as described in the previous note.]

When SPIBILD puts merged records in the deferred queue

For merges, the circumstances are different: SPIBILD knows exactly what elements are being updated, since it is doing the work. Hence, it may determine that the only elements being updated by a merge request are non-indexed ones, so the merged record doesn't need to go to the deferred queue for re-indexing; instead, it can go directly to the tree. SPIBILD can do this for goal record-types that have a "pass-elements list", created when the file is compiled, which identifies the elements that affect indexes -- if the merge request doesn't change any elements on the list, there's no need to re-index the record.

On the other hand, if the merge request does change an element in the pass-elements list, the record goes to the deferred queue for re-indexing. Naturally, the trip to the deferred queue means more work for SPIBILD (and more cost to you) since all the re-indexing work must be done.

Conclusions

These facts lead to several conclusions:

 - In SPIBILD, it's more efficient to merge data that doesn't affect indexes  than  data  that
 does.
 - If you are updating records with data that doesn't affect  indexes,  the  total  processing
 cost (i.e., to get the data into the tree) will be less if you:
 - use the SET NOPASS command to turn off index passing; or
 - merge the data in SPIBILD, taking advantage of  the  pass-elements  list,  rather  than
 update the records in SPIRES or SPIBILD.

The next two sections will discuss the two methods mentioned above:

2.4.3.2  (*) Using SET NOPASS to Prevent Unnecessary Indexing Work

When updated records in the deferred queue are processed, SPIBILD must generate two sets of index data for each record: the data for the tree copy, to be deleted, and the data for the new copy, to be added. To avoid unnecessary work, SPIBILD sorts them together, so that pairs that are identical except for the "add" or "delete" designation are discarded -- those pairs indicate element values that were not changed, which thus don't need to be re-indexed.

Index data is created even if indexed elements don't change

If a record is updated but no changes are made to the elements that affect indexes, then SPIBILD will create all the index data and subsequently discard it. This superfluous work may be worth avoiding in applications where non-indexed data is updated regularly.

As described in the next section, on pass-element lists, SPIBILD tries to avoid this unnecessary work in the case of merges done in SPIBILD (though without some intervention on the file owner's part, SPIBILD may not be able to avoid it).

SET NOPASS suppresses index data creation

The SET NOPASS command is a fairly simple command you can use for updates and merges no matter whether you do them in SPIRES or SPIBILD -- it guarantees the extra work won't be done. If you are merging records in SPIBILD, using SET NOPASS can save you the trouble of determining whether a pass-elements list is in effect.

The technique is to add the SET NOPASS command to the SPIBILD procedure you are using, at the time prescribed in the procedure's description. For instance, if you are processing a file in SPIBILD after updating records in SPIRES, you issue the SET NOPASS command before the PROCESS command in SPIBILD. [See 2.2.2.]

WARNING: Do not use this technique unless you are absolutely certain that no data in the deferred queue affects any indexes. If it does, and you use the SET NOPASS command to turn off passing, then data integrity problems (discrepancies between the goal records and the indexes) will occur. If you aren't certain, or if you can't control the update procedure (including file processing) from start to finish, do not use the SET NOPASS command.

Process the file first

You might want to process the file first, before you apply any of the updates, in order to clear any other data from the deferred queue. In that case, don't issue the SET NOPASS command until you are ready to process the new data -- you want the data already in the deferred queue to be processed normally.

2.4.3.3  (*) Using the Pass-Elements List for SPIBILD Merge Processing

Merged record processing in SPIBILD can be more efficient if SPIBILD finds a "pass-elements list", a list of the elements that affect indexing. When only non-indexed elements are merged, SPIBILD can put the record directly into the tree, rather than putting it in the deferred queue for indexing later.

Unfortunately for many subfiles, SPIRES can't always create a pass-elements list on its own, and if you don't help it along, it won't create one. When a subfile doesn't have one, SPIBILD has no idea whether any given element will affect an index, and so must place all merged records back into the deferred queue. To understand why this is true, we need to know more about pass-elements lists.

How SPIRES creates the pass-elements list

When it compiles a file definition, SPIRES creates the pass-elements list from the elements it knows are being passed to indexes. Specifically, SPIRES places the following elements into the pass-elements list:

 - elements named in the GOALREC-NAME statement; and
 - elements named in any of the following Passproc rules:
 - $PASS.ELEM proc (A167)
 - $PASS.UPPER proc (A162)
 - $PASS.BY proc (A163)
 - $PASS.COND proc (A168)

SPIRES doesn't always build the pass-elements list

SPIRES only builds this list if it is sure that no other elements in the record-type can affect the index values. If such an element existed and if it were changed in a SPIBILD merge request but no elements in the pass-elements list were changed, then the related index values would not be updated, since SPIBILD would only re-index the record if an element on the pass-elements list had been merged too.

SPIRES doesn't build the list if any of the above-listed elements:

 - is a virtual element; or
 - has the $CALL proc (A62) in its Passproc string; or
 - is passed in its output form, i.e., is processed through its Outproc  string  before  being
 passed.

If SPIRES doesn't build the list, then all merged records handled by SPIBILD are placed in the deferred queue for index processing. This certainly isn't a tragedy, but obviously, if no indexed elements are affected by the merge data, then SPIBILD's having to regenerate index information is extra and unnecessary work. In a large application, such inefficiency could be quite expensive.

The PASS-ELEM statement in a file definition

You can tell SPIRES to create the pass-elements list anyway, even if the circumstances that would prevent its creation occur, by adding the PASS-ELEM statement to the file definition, at the end of the appropriate linkage section:

You would want to specify in the element list any elements not included in the list as described above that SPIRES should add to the list -- that is, elements whose values affect the passed values of other elements.

The PASS-ELEM statement cannot be added with File Definer; you must change the file definition directly. Select the FILEDEF subfile, retrieve the current copy of your file definition, and add the PASS-ELEM statement as described above to the appropriate linkage section. Then update the record and recompile the file definition in SPIRES.

Examples

Here are some examples describing how you might use pass-element lists. Suppose that the only reason SPIRES won't create a pass-element list for your subfile is because one of the elements you are passing is a virtual element but whose value has nothing to do with any other element. You could tell SPIRES to create the list anyway, just by adding "PASS-ELEM;" to the file definition.

Or suppose, for another example, that the indexed virtual element SHIPPING.DATE is created by adding 10 days to the value of the non-indexed element ORDER.DATE. If you code in the file definition:

then SPIRES will create a pass-elements list, including ORDER.DATE.

WARNING: Don't code the PASS-ELEM statement unless you're sure that either:

 - all elements that have any effect on indexes are listed in the PASS-ELEM statement, or will
 be placed in the list automatically, as described above; or,
 - you will never use SPIBILD to merge any elements not on  the  list  that  are  supposed  to
 affect  indexed  values.   If  you  do,  the  indexes  won't get updated properly, and data
 integrity problems will result.

2.5  Removing Multiple Records from a Subfile

Removing records from a subfile in a single operation is a simple task. The SPIRES and SPIBILD procedures that remove records are easy to use, and much less complex than those for adding or updating records. There are several different methods, each of which shines under separate circumstances.

Below is a set of questions and answers to help you decide which technique to use, followed by a comparative summary of the methods. Two of the techniques are described in detail in this chapter. The others have been described either in Chapter 2.3, or in other SPIRES manuals.

Methods for Removing Multiple Records from a Subfile

There are seven ways to remove multiple records in a single operation:

 - 1. (SPIRES) the REMOVE procedure in Global FOR
 - 2. (SPIRES) the INPUT BATCH procedure
 - 3. (SPIRES) the BATCH procedure
 - 4. (SPIBILD) the REMOVE procedure in Global FOR
 - 5. (SPIBILD) the INPUT BATCH procedure
 - 6. (SPIBILD) the ZAP RECORD-TYPE procedure
 - 7. (SPIRES) the ZAP FILE procedure

Here are some questions to help you determine which method to use, followed by descriptions of each.

 - If you have a set of the keys of records to be removed, use  the  INPUT  BATCH  or  BATCH
 procedures.
 - If you are using indexed criteria, use the REMOVE procedure  in  SPIRES,  in  conjunction
 with the FOR RESULT command.
 - If you are using non-indexed criteria, use the REMOVE procedure in Global FOR  in  either
 SPIRES or SPIBILD.

1. (SPIRES) The REMOVE Procedure in Global FOR

You can use the REMOVE command in SPIRES with any Global FOR class, such as SUBFILE, RESULT and STACK. That means you can remove any and all records in the given class. This procedure is available to any user with update privileges to the subfile.

SPIRES places a removal request for each record into the deferred queue; during subsequent file processing, the records will be removed, and any necessary "un-indexing" will be done. Hence, the indexes aren't updated (except for any immediate indexes) at the time the REMOVE command is issued. File processing costs are picked up by the file owner (or the next account to process the file), not by the user removing the records.

Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic REMOVE command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.)

See the manual "Sequential Record Processing in SPIRES: Global FOR" for more information about the REMOVE command in Global FOR; online, EXPLAIN REMOVE COMMAND, IN GLOBAL FOR. Information about these two prefixes also appears in Part 6 of this manual, under INPUT BATCH, with which they work the same way in SPIRES.

2. (SPIRES) The INPUT BATCH Procedure

You can use the INPUT BATCH command to add new records, modify existing records and remove existing records. SPIRES places the data into the file's deferred queue. When updates, merges or removes are involved, the data must be in the standard SPIRES format. If an error is found in the data, SPIRES reports it right away, possibly rejecting the record.

The INPUT BATCH command in SPIRES is available to any user with update privileges to the subfile. The data may be read from your active file or, via the BATSPI program, from a tape or WYLBUR data set.

Because the removal requests go into the deferred queue, the indexes are not updated (except for any immediate indexes), and the records aren't removed from the tree until the next time the file is processed. Hence, the file processing costs will be charged to the file owner (or the account that next processes the file), not the user removing the records, who pays for only the input costs.

Two prefixes, WITH FAST and WITH SEARCHONLY, can speed up the processing, though at the cost of some restrictions that the basic INPUT BATCH command does not have. WITH FAST, for instance, prevents immediate indexes from being updated; processing the file afterwards will take care of that. (In fact, when immediate indexes are involved, the NOJOBGEN flag is set, meaning that no updating is allowed until the file is processed.) Full details about these two prefixes appear in Part 6, under INPUT BATCH; online, EXPLAIN INPUT BATCH COMMAND, IN FILE MANAGEMENT.

See the SPIRES manual "Searching and Updating" for more information about the INPUT BATCH command in SPIRES. For online help, EXPLAIN INPUT BATCH.

3. (SPIRES) The BATCH Procedure

The BATCH command places a "batch request" in the file's deferred queue. The batch request tells where the input data is stored (on disk or on tape). The input data must be in the standard SPIRES format. It can include adds, updates, merges and removes.

During overnight processing, JOBGEN sets up a batch SPIBILD job that retrieves the input data and then "batches" the data into the subfile, discarding records that are to be removed. In fact, the job invokes SPIBILD's BATCH command (see below) to do the processing.

This technique is, by far, the cheapest method for the user. All processing costs, except for the small charge of placing the batch request in the deferred queue, are borne by the file owner when the overnight processing occurs.

The BATCH command in SPIRES has some severe limitations and disadvantages, however. Because the input is deferred until overnight, the user doesn't know till the next day whether the records were removed, and then only by checking it out for him- or herself; moreover, the "log" describing any records with errors is printed to the file owner, not the user.

A very significant disadvantage is that if the file is processed by any means other than JOBGEN-submitted overnight processing, the batch request is simply discarded without warning. The data, whether on disk or tape, is left untouched; but the request telling JOBGEN where the data is gets thrown away. This problem makes the BATCH command in SPIRES quite unreliable.

For further information about the BATCH command in SPIRES, see the SPIRES manual "Searching and Updating".

4. (SPIBILD) The REMOVE Procedure in Global FOR

This procedure is similar to its cousin in SPIRES in that you establish a Global FOR class and then issue a REMOVE command to remove the records in that class.

The Global FOR class can be only SUBFILE or STORED in SPIBILD. (STORED, available in SPIBILD only, means that the records to be examined are in a stored stack or result.) You may include a WHERE clause on the FOR SUBFILE or FOR STORED command, and you may also use the SET FILTER and SET SCAN commands to limit the records removed.

If you are removing only a few of many records in a subfile, using REMOVE under Global FOR in SPIRES is probably more efficient than using it in SPIBILD. On the other hand, the REMOVE command in SPIBILD removes the records from the subfile immediately, unindexing them immediately as well. The user pays for all the processing, including the processing of any data already in the deferred queue. Because the file gets processed, only users with Process access to the file are allowed to use the REMOVE command in SPIBILD. [See 2.5.1.]

5. (SPIBILD) The INPUT BATCH Procedure in SPIBILD

You can use the INPUT BATCH command in SPIBILD to add new records, modify existing records and remove existing records. For removes, the INPUT BATCH command removes the records identified by key directly from the tree.

The INPUT BATCH procedure in SPIBILD is more expensive than the similar INPUT BATCH procedure in SPIRES because the file is processed before the input data is batched in, and because the records get indexed right away. Because the file gets processed, only users with Process access to the file are allowed to use the INPUT BATCH procedure. Note too that the user of this procedure pays for all the processing -- including the processing of data already in the deferred queue.

The input data may be on disk or tape, though if on tape, the procedure must be done through a Batch SPIBILD job.

The steps of this procedure were described earlier. [See 2.3.1.]

6,7. The ZAP RECORD-TYPE and ZAP FILE Procedures

The ZAP RECORD-TYPE command discards all the data in a specified record-type of a file. The ZAP FILE command erases all the data sets of an entire file. Though these sound like the ultimate solutions for removing records from a subfile, they are the best methods to use when you want to remove all the records from a subfile.

Moreover, they are worth considering in situations where you aren't removing all the records of a large subfile but are removing a large majority. In particular, consider saving the few records you want to keep, zapping the record-type (or the file) and then adding the kept records back in to a completely empty record-type.

Several benefits may come from this method, including the release of unneeded blocks for storage, and the savings from not "unindexing" all the records you're removing -- it costs as much to remove a record as to add it, because it takes SPIBILD the same amount of work to unindex a record as to index it in the first place.

Only the file owner can use the ZAP FILE procedure. [See 2.11.] Users with Master access to the file can use the ZAP RECORD-TYPE procedure. [See 2.5.2.]

2.5.1  The REMOVE Procedure in SPIBILD with Global FOR

The REMOVE procedure removes records determined by the established Global FOR environment. The Global FOR environment in SPIBILD is not as flexible as the one in SPIRES -- the only "FOR class" commands available are FOR SUBFILE and FOR STORED.

If you have a search result or stack of records, or can use some other Global FOR class of records in SPIRES, you can remove them using that class in SPIRES, and then process the file if you want. Or, to avoid the extra work of putting the removal requests into the deferred queue, you can store the search result or stack, and then use the REMOVE command under FOR STORED in SPIBILD to remove them, which also processes the file and updates the indexes appropriately.

Either method will almost certainly be cheaper than using a WHERE clause on a FOR SUBFILE command in SPIBILD, especially if the subfile is large. But if identifying the records via indexes isn't available, this other method will do.

Here are the steps of the procedure:

Step 2: SPIBILD

Issue the SPIBILD command online, or in a BATWYL job:

In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.

Step 3: Issue any SPIBILD special-processing commands desired.

Several optional commands or procedures may be useful at this point, such as:

 - the SET MESSAGES command, to control the messages SPIBILD displays.  The command  "SET
 INFORMATION  MESSAGES  =  0"  is  especially useful, since it turns off the individual
 "add"    message    displayed    for    each     new     record.      [See     2.1.5.3.]
 - the SET SEARCH/SET SHARE procedure to share access to the file with other users.   [See  2.1.5.1.]
 - the commands that limit the record-types passed from and to: SET INDEX,  SET  NOINDEX,  SET
 PASS  and  SET  NOPASS.   Remember,  however,  that  they are primarily meant to solve file
 integrity problems and should be used only after consultation with your SPIRES  consultant.
 [See 2.1.5.4, 2.4.3.2.]
 - the SET STACK SIZE command, to increase the size of the pass stack.   See  the  description
 for that command in Part 6, or online, [EXPLAIN SET STACK SIZE COMMAND.]

Step 4: ESTABLISH subfile.name

Issue the ESTABLISH command, which processes the file and prepares the subfile for direct update operations.

Step 5: Establish the Global FOR environment.

Establish the Global FOR environment by issuing the FOR SUBFILE or FOR STORED command, optionally followed by other Global FOR commands mentioned below. The purpose of these commands is to identify the specific records to be removed.

The FOR SUBFILE command

No VIA clause is allowed on the FOR SUBFILE command, but you may add a WHERE clause as needed, to limit the records to be updated using element criteria:

The FOR STORED command

If you have stored a stack or result containing the records you want to update (see step 1), you would use the FOR STORED command:

The SET SCAN commands

The SET SCAN commands can give you further flexibility (and savings) in determining which records are to be examined for processing. The SET SCAN commands allowed in SPIBILD are:

Only SET SCAN LIMIT is allowed under FOR STORED. Note that SET SCAN PREFIX has no meaning with slot subfiles.

Details about these commands, including WHERE clauses, appear in the manual "Sequential Processing in SPIRES: Global FOR".

SET FILTER command in SPIBILD

The SET FILTER command, with the TYPE = SCAN option, works similarly to a WHERE clause but is more versatile. You can, for example, use it to locate records when you have complicated criteria involving elements within structures, a situation that WHERE clauses often cannot handle. See section 4.4.2 of the Global FOR manual referenced above; online [EXPLAIN FILTERS, ON WHERE CLAUSES.]

SKIP command in SPIBILD

The SKIP command tells SPIBILD to skip the specified number of records meeting the Global FOR criteria:

where "n" is the number of records to be skipped. If you specify neither "n" nor FIRST (for the FIRST record), the default is FIRST. Except for the limited options, SKIP works the same in SPIBILD as it does in SPIRES. See section 5.10 of the Global FOR manual; online [EXPLAIN SKIP COMMAND.]

Example

This example shows how to use a WHERE clause and SET SCAN processing to identify the records to be removed in the least expensive manner. Suppose you want to remove all the records with MOD.DATE element values before 1986. You also know that all those records have slot keys less than 20000:

These two commands work more efficiently together than would the single command FOR SUBFILE WHERE MOD.DATE < 1986 AND ID < 20000. The SET SCAN command limits the records that are examined to those whose "id" is less than 20000; under only the FOR SUBFILE command, SPIBILD would examine all records, looking for those whose "id" is less than 20000, which might be hundreds or thousands more records. If you will be processing records under Global FOR in SPIBILD, you should be familiar with the SET SCAN commands, which can save you a great deal in processing costs.

Note in the example that SPIBILD does not use the Global FOR prompt that SPIRES uses (+?) after you issue the FOR SUBFILE command.

Step 6: Issue the REMOVE command.

The REMOVE command in SPIBILD removes the records determined by the Global FOR class.

Here "n" represents the number of records you want processed; the default is REST.

The REMOVE command first locates each record to be removed and places a "remove request" in the deferred queue for it, just as the REMOVE command does in SPIRES. When the Global FOR traversal is complete, SPIBILD processes the deferred queue again, which is when the records are actually removed and the indexes updated.

Example

Here is a sample online SPIBILD session showing typical use of the REMOVE procedure:

This example shows a case where all records whose keys are no greater than 100 are removed. The removal requests go into the deferred queue first; the "End of BATCH/BUILD" message indicates the start of the deferred queue processing, when the records will actually be removed.

Handling Problems

System failure

The simplest way to recover from system failure is to restart the procedure. But there are two situations in which alternate methods might be cheaper to do:

1. If SPIBILD had printed the "End of BATCH/BUILD" message before the system failure, then SPIBILD had finished placing all the removal requests in the deferred queue and was starting to process the deferred queue to complete the job. To recover, you should use the PROCESS/RECOVER procedure described earlier in this manual. [See 2.1.4.2.]

2. If the REMOVE command under FOR SUBFILE had already removed many records before the failure occurred, you might want to consider adding a SET SCAN START command to the procedure. That way, SPIBILD won't have to re-examine all the records it examined the first time that weren't discarded.

It is easy to find out what record was the last one removed:

 - 1) select the subfile in SPIRES
 - 2) issue the FOR REMOVES command
 - 3) issue the SKIP LAST command
 - 4) issue the "SHOW KEY *" command

The key value shown will be the last one removed; use that as the start value in the SET SCAN START command.

On the other hand, if you were working under FOR STORED, you would probably resume by restarting the procedure. It will do no harm for SPIBILD to try removing records that are already removed.

2.5.2  Using the ZAP RECORD-TYPE Procedure to Remove All Records

The ZAP RECORD-TYPE procedure erases all the records in a record-type very quickly and very efficiently. If you think of a SPIRES file as being a file cabinet, the ZAP DATA SET or ZAP RECORD-TYPE commands are equivalent to tossing out a drawerful of file folders. In contrast, using the REMOVE ALL command under Global FOR in SPIRES or SPIBILD is akin to pulling out the folders one by one, checking to see if each one is indexed in another drawer of the cabinet and deleting that index entry too.

You will need to choose between the ZAP RECORD-TYPE and ZAP DATA SET commands when executing this procedure. Generally speaking, use the ZAP RECORD-TYPE command; however, choose the ZAP DATA SET command if the record-type is combined with other record-types in a single ORVYL data set, but you want to discard all of them.

The discussion of "Preliminaries" below will tell you how to determine whether the record-type is combined with others or not.

The ZAP RECORD-TYPE procedure pays no heed to indexes -- it simply discards all the data in the record-type or data set you name. But in most cases when you are removing all the records in a goal record-type, either it has no indexes or the indexes are in their own record-types, not connected to any other goal record-type, and they too can be erased with the ZAP DATA SET procedure. (Of course, if you are discarding all the record-types in the file, the ZAP FILE procedure [See 2.5.3.] is even better.) Unconditionally emptying a goal record-type and its associated index record-types will be much more efficient than having SPIBILD remove the goal records one by one, creating and applying pass stack data.

Preliminaries

Before beginning this procedure:

 - The name of the file is the key of the file definition.  If you aren't  the  file  owner,
 you  can  get  the name by selecting a subfile of the file in SPIRES and issuing the SHOW
 SUBFILE INFORMATION command.
 - The name of the record-type appears in the file definition.  Also, if the record-type  is
 a  goal  record-type of a subfile, you can get the information in SPIRES by selecting the
 subfile and issuing the SHOW SUBFILE INFORMATION command.
 - The SHOW FILE ACTIVITY command in SPIRES can tell you what data set a  given  record-type
 is in.  The data set name is "REC" followed by the integer (from 1 to 9) in the
 "DS" column of the display.

Step 1: SPIBILD

Issue the SPIBILD command online, or in a BATWYL job:

In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.

Step 2: ESTABLISH subfile-name

Issue the ESTABLISH command so that SPIBILD will know which file to use in subsequent commands, and so that SPIBILD will process the deferred queue.

The deferred queue must be empty for the ZAP DATA SET and ZAP RECORD-TYPE commands to work. (If the deferred queue contains worthless data, you might want to consider issuing the ZAP DEFQ command in SPIRES as a preliminary step. That would save you the expense of processing the deferred queue.)

Specify the name of any subfile in the file containing the record-type or data set.

Step 3: ZAP RECORD-TYPE recname (or ZAP DATA SET data.set.name)

Issue the command you chose to use, based on the information given earlier; either:

where "recname" is the name of the record-type to be erased; or

where "n" is the number/letter of the physical data set you want to erase.

Warning: Both of these commands take destructive action immediately. Double-check that you have correctly typed the data set number or record-type name before pressing the Return key. Recovering the data if you make a mistake is usually very difficult, if indeed it is possible at all (see below).

These two ZAP commands may also affect data in the residual data set. If records from the data set or record-type being zapped have been removed to the residual, that data will be erased too.

You need not recompile your file definition to begin using the zapped data set or record-type again; it is empty but not gone.

Example

Here is a sample SPIBILD session demonstrating this procedure using the ZAP RECORD-TYPE command:

Handling Problems

The worst problem that can occur with this procedure is that you erase the wrong data set or record-type. If that occurs, you will want to try restoring the entire file, using the most recent copy that was made. (If you made a backup copy of the file at the start of the procedure, as suggested in the Preliminaries, you're in excellent shape.) You may want to contact your SPIRES consultant, who can help you make arrangements to restore the file using backup files made by your computer center. [See 2.1.4.3.]

System failure

If the system crashes at any point during the procedure, simply restart the procedure at your convenience.

2.6  Adding and Removing Elements in a Record-Type

After working with a new file for awhile, you may decide that the goal records for one of its subfiles needs a new element or that it doesn't need some elements it already has. Depending on the original file design, such changes can be very simple or they can be difficult or impossible to do without rebuilding the entire subfile. [See 2.9.]

The chapter is organized as follows:

2.6.1  Adding a New Element to a Record-Type

Once you have created a record-type, whether it's the goal record-type of a subfile or an index record-type of another, you do not have much flexibility to change it, unless you are willing to rebuild the entire record-type, in which case you have all the flexibility you need. But rebuilding can be tedious and expensive, and is unnecessary in many situations. [The procedures for rebuilding record-types appear later in this manual. If the record-type is an index record-type, you will need to rebuild the index, but if it is a goal record-type, you will need to rebuild the entire subfile. [See 2.7.3, 2.9.]]

Adding Optional elements may be possible

Generally speaking, as file owner, you can add new Optional elements to any existing record-type in a file as long as that record-type already has Optional elements at the level where you want to add them. For example, if you want to add Optional elements to a Required structure, you can do so if the structure already contains Optional elements.

However, you cannot add new Fixed or Required elements to an existing record-type (unless they are inside a new Optional structure) without rebuilding the record-type. [Note though that you can include Inproc and Inclose rules on an Optional element that make it seem Required to the user; only internally, in terms of the way SPIRES stores the element, would it actually be treated as Optional.]

Preliminaries

When significant changes are made to a very important file, the file manager customarily takes either or both of two preliminary steps:

 - 1) Create a test file based on the file  definition  of  the  production  file.   Make  the
 proposed  changes to the test file, test them, debug them, and then make the changes to the
 production file.
 - 2) Make a backup copy of the production file just before making the  change,  so  that  the
 file  can be restored in case of serious problems discovered when the change is made.  [See
 2.1.4.3.]

1) Change the file definition.

The rules for what elements you can add without rebuilding the record-type are:

Following these guidelines, you can change the file definition directly, or by using File Definer. Use File Definer only if you still have the File Definer input for the current version of the file definition.

To change the definition with File Definer, read 1a; to change it directly, read 1b.

1a) Change the file definition using File Definer.

Add the desired new elements to your File Definer input file, following the rules listed above. For example, you might add another element to the Optional section at the record level:

If you want the element to be indexed, you can easily add it to an existing index, by including the INDEX attribute, followed by the name of the element already indexed. On the other hand, to create a new index for the new element, you should follow the instructions given in the next chapter. [See 2.7.1.] DO NOT simply add the INDEX attribute to the end of the new ELEMENT line.

Then, use File Definer to generate a revised file definition:

The full procedure, with more information about how to make changes to files using File Definer, appears in the "File Definer" manual.

1b) Change the file definition directly.

In SPIRES, get the file definition from the FILEDEF subfile. Find the record definition for the record-type to which you want to add the elements.

Following the guidelines listed above, add new ELEM statements where desired, along with other statements associated with element definitions as desired, such as LEN, OCC, INPROC, OUTPROC, TYPE, etc. See the "File Definition" manual for more help.

Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.

2) RECOMPILE filename

The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new Orvyl data sets requested.

For example,

If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.

3) Recompile formats, if necessary.

You should recompile any formats written for the record-type if either of these situations is true:

 - you have added Optional elements at the record-level or  in  a  structure,  and  they  come
 before  any  virtual  elements  that  were  already  defined at the record-level or in that
 structure; [Internally, the virtual elements will have new element numbers  when  the  file
 definition is recompiled, which will not match their old numbers that are compiled into the
 format definitions.] or
 - you have added a new Optional structure within an already existing structure.

If you neglect to recompile them, the formats will probably not work properly.

Use the PERFORM FORMAT LIST command to find the formats defined for the record-type. That list shows all compiled formats for the record-type, including any that were defined by other users, whom you might want to notify of your changes so that they can recompile their formats.

Checking your work

At this point, you are strongly urged to return to SPIRES, select the subfile and add or update some records, including the new element. Remember to display some old and new records too. If the element works as you wanted, you are done with this procedure.

Handling Problems

System failure

System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.

Gibberish in the records

Very serious problems can arise if you mistakenly add a new element in a place that breaks the rules presented earlier. The problem can manifest itself in several ways:

To recover, you need to follow these steps:

 - remove or dequeue those records containing the new element;
 - process the file in SPIBILD;
 - correct the file definition in the FILEDEF subfile, continuing with the procedure described
 earlier in this section, starting with step 1.

2.6.2  Removing Unneeded Elements from a Record-Type

In general, SPIRES does not allow you to remove elements defined for a record-type. Once a record-type has a given set of elements, it will always have those elements, plus any additional ones you may be able to add. [See 2.6.]

You can, of course, rebuild the record-type completely, which is really the only way to completely get rid of an element right away. [See 2.8.1.] When you dump out the contents of the record-type (to batch in again later), you would delete the occurrences of the element you were eliminating. You might do that either by deleting those occurrences using the text editor, or by using the SET ELEMENTS command to restrict the output elements to those you were intending to add back in.

A less drastic solution is to add the $DISCARD proc to the Inproc rule and $NULL to the Outproc rule of the element you want to eliminate. This has no immediate effect on the stored records already in the subfile, though it does hide the element value when the record is displayed.

However, as new records get added or old records get updated, the element will gradually disappear. (As part of this method, you might want to rename it, e.g., DUMMY.) It will not go away completely: it will always show up as an element (e.g., in SHOW ELEMENTS commands) and some older records that never get updated or removed may retain it, but it will take less and less room in the file as time goes by.

Alternatively, you could choose to get rid of all occurrences of the element immediately by updating all the old records. [See 2.4.] Again, the element will not go away completely; it will still show up in element lists, for example.

2.6.3  General Rules for a Record-Type

Rules for record/structure changes.

These rules apply to situations were data already exists within the database. Record level changes are not much different from structure changes, except that record level changes are tied to the existence of records within the record-type, and structure changes are tied to structure occurrences within records.

If there are NO occurrences of records, you can change anything within the RECORD-NAME portion corresponding to the record-type except the COMBINED (or not COMBINED), or the SLOT (or not SLOT) nature of the record-type.

If there are NO occurrences of a structure within ANY of the records, you can change anything in the structure's definition.

If there are records existing in the record-type, your changes are limited to the same changes allowed for existing structure occurrences. What is allowed, and when, is covered below in the form of Questions and Answers.

Q1: Can Optional elements be defined in none were defined?

A1: Usually NO, but there are two exceptions when you can add the OPTIONAL section with one or more optional elements.

If there is only a KEY in the REQUIRED section, and no other REQUIRED elements, OR there is no REQUIRED section at all, then you can add an OPTIONAL section. This is true even if there are FIXED and/or VIRTUAL sections.

Q2: Optional elements are defined (by the OPTIONAL section), but none of these elements occur in any of the records, can I delete the OPTIONAL section entirely?

A2: Usually NO, but you can if there are no other REQUIRED elements other than a REQUIRED KEY. This is basically the reverse of Q1. Furthermore, the only penalty for having declared OPTIONAL elements which don't occur is the OPTIONAL element bit-mask, which is 2-byte long, and only occurs if there are REQUIRED elements other than a REQUIRED KEY.

Q3: Can I define new FIXED or REQUIRED elements?

A3: NO, once FIXED or REQUIRED elements occur, you can't define new elements in either the FIXED or REQUIRED sections. But if OPTIONAL elements have been declared, or can be declared (see Q1), then you can add OPTIONAL elements.

Q4: Optional elements are defined, and some of them exist within records, can I define new OPTIONAL elements?

A4: YES, you can add new optional elements to the OPTIONAL section after the latest physically occurring element in any of the records. For example, if you have defined:

    OPTIONAL;
      ELEM = A;
      ELEM = B;
      ELEM = C;

and only A and/or B occur in the records, and C never occurs, you can add ELEM = D; after ELEM = B; or ELEM = C; (the end). It is always safe to append to the end of the OPTIONAL section.

Q5: Can I redefine non-occurring OPTIONAL elements?

A5: YES, if an optional element does not occur in any records, you can redefine that element anyway you like, including making it a simple element when it was a structure, or making it into a structure when it was a simple element. LENGTH and/or OCC attributes can be changed, INPROC and OUTPROC rules, even the element's name and ALIASES. The only requirement is that you retain an element in this element's position. You can't delete it or move it. So in Q4 above, since C didn't occur in any of the records, we could have redeclared it as element D, rather than add ELEM = D; statement. In a sense, we are replacing the definition of a non-occurring element (ELEM = C) by a new definition.

Q6: In a structure defined with no KEY, can I change an ELEMENT to a KEY, or vice versa?

A6: You can only have one KEY in a structure, and it must be either the first FIXED element, or the first REQUIRED element. You can change the first FIXED ELEMENT to a KEY at any time, and vice versa. The first REQUIRED ELEMENT can be changed into a KEY only if the structure does not occur in any records. Once the structure occurs, its REQUIRED section must not be changed.

Q7: What can I do with VIRTUAL elements?

A7: You may add or remove VIRTUAL elements at any time.

Q8: After making changes, what must I do?

A8: If you updated a RECDEF, you must recompile that RECDEF and any FILEDEFs that specify that RECDEF in DEFINED-BY statements. If you updated a FILEDEF, you must recompile that FILEDEF. Then, you must recompile all FORMATS that refer to this file/record-type.

2.7  Adding, Removing and Rebuilding Indexes of a Subfile

After a SPIRES subfile has been used for awhile, you, the file owner, may discover that the existing index scheme isn't quite adequate. An index you thought would be used often has turned out to be unnecessary, for example, or you realize that you've been using Global FOR commands frequently to find records based on an unindexed element. These situations are fairly common -- file owners often use data bases differently than predicted once the data is available to them.

The decision to add or remove an index is usually related to cost and convenience. Specifically, does the convenience of a particular index justify the cost of storing and maintaining this redundant information? For frequently used indexes, the answer is yes; but if an index is seldom or never used, the answer may be no.

Organization of Chapter 2.7

This chapter presents the procedures used to handle major tasks relating to indexes:

 - to change values in an index when its Passproc rule has been changed (e.g., so that the
 index will be a word index instead of a phrase index);
 - to repair a damaged index (e.g., an index whose Passproc was changed previously without
 the index being rebuilt, so that old records being removed or  updated  aren't  getting
 "un-indexed" properly);
 - to remove an element's values from the index  (e.g.,  you  want  to  remove  the  TITLE
 element  values  from  a SUBJECT index that had both TITLE and SUBJECT element values);
 - to add an element's value to an index (e.g., you want to add the TITLE  element  values
 to  a  SUBJECT index so that it will have both TITLE and SUBJECT element values).  [You
 may be able to do this without erasing the current index first, a possibility discussed
 in section 2.7.3.]

FASTBILD builds indexes too

You can also use the FASTBILD program when building or rebuilding an index. You can find the primary documentation for that program later in this manual [See 2.12.] but its context in building indexes will be discussed in this chapter.

2.7.1  Adding an Index to a SPIRES Subfile

Though six steps are shown here, the process of adding new indexes to a subfile is fairly simple. The design phase (suggested in step 1) generally takes the most time.

Preliminaries

You need to choose a one-to-six-character record-name for each new index record-type you are defining (e.g., REC09, ZIN12). Note: This is crucial even if you use File Definer to create and change the file definition, previously leaving the selection of record-type names up to File Definer.

Choose a new index record-name carefully

New record-types being added to an existing file must be added to the end of the existing record-types. Since SPIRES sorts them in alphanumeric order by name before compiling them, it is crucial that the new index record-types you define have names that sort after those that already exist. [There is an exception. If the file has any DEFQ-only ("pseudo-") record-types, whose names make them sort to the end of the record-types, the new ones must precede them at the end. See the "File Definition" manual for more information.]

For example, if the file already has the three record-types REC01, ZIN01 and ZIN05, your new index would need a name that sorted after ZIN05, such as ZIN06. You could not add an index record named ZIN02, unless you are willing to rebuild the entire file.

PERFORM FILE SUMMARY command

The names of all the current record-types appear in the RECORD-NAME statements of the file definition as stored in the FILEDEF subfile. Another reliable way to find the names of the current record-types is to select the subfile in SPIRES and issue the command PERFORM FILE SUMMARY.

Step 1: Change the file definition.

You can follow this step using File Definer or by changing the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file definition.

WARNING: Whichever method you use, be extremely careful to follow the directions, to ensure that the new index record definitions follow the definitions for those that already exist. Failure to do so may result in data integrity problems that are very tedious and unpleasant to correct.

To change the file definition using File Definer, read 1a; to change it directly, read 1b.

Step 1a: Change the file definition using File Definer.

Add the INDEX attribute as appropriate to your File Definer input file, following it in each case with the RECNAME attribute, followed by the name you chose for the new record-type. For example, to add a new index for the element defined here:

you might change it to:

Then, use File Definer to generate a revised file definition:

The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.

Step 1b: Change the file definition directly.

In SPIRES, get the file definition from the FILEDEF subfile and add to it:

 - a record definition for each new index record-type; and
 - appropriate linkage sections for each index.

Remember that a file can have no more than 64 record-types, stored in no more than 9 Orvyl data sets (filename.REC1 through filename.REC9). Use the COMBINE statement to combine record-types in data sets. SPIRES will put each record-type that doesn't have a COMBINE statement into its own data set, to the limit of 9.

Also remember to follow the rules for index record-types and index linkages in the same goal record-type, such as:

 - The pointer element must be given the same name in all index record-types for a single goal
 record-type.
 - In general, pointer groups for  indexes  that  apply  to  the  same  goal  record  must  be
 structured  identically,  e.g.,  must  all  be  TYPE  = LCTR, must all have the same LENGTH
 statement if specified for any, etc.

For more detailed information about coding index record definitions and linkage sections, see the manual "SPIRES File Definition".

Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.

Step 2: RECOMPILE filename

The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new Orvyl data sets requested.

For example,

If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.

At this point, you are strongly urged to return to SPIRES, select the subfile and issue the BROWSE command to look at the new index. It should be empty, and you will receive the message "Null range". If it's not, then you did not add the new record-type to the end of the existing record-types, and woe to you if you continue this procedure! Instead, you should return to the first step and correct the file definition, making sure the new record-type or -types appear at the end of the record definitions in the file definition.

Step 3: SPIBILD

The first two steps are usually done interactively, but the rest of the procedure may be done as a batch job if desired. [You may either use SPIBILD in a batch job or use FASTBILD. FASTBILD is recommended if the index will be extremely large, e.g., more than 10,000 index entries created. [See 2.12.]]

Issue the SPIBILD command online, or in a BATWYL job.

In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.

Step 4: ESTABLISH subfile-name

Issue the ESTABLISH command to process the file's deferred queue and to tell SPIBILD the name of the subfile containing the index or indexes to be built.

You can build all the indexes for a single goal record-type at one time. If you are adding indexes for several different sets of goal record-types in the same file, you need to return to this step for each goal record-type.

At this point, you may also want to increase the size of the pass stack to increase passing efficiency, using the SET PASS STACK command. See the description for that command in Part 6. [See 6.3.0a.]

Step 5: Establish the Global FOR environment.

With this step, you tell SPIBILD which goal records to use to build the index. Usually, you will simply issue the FOR SUBFILE command:

Generally speaking, with this particular procedure, you want to build the new index from all the goal records. However, if for some reason you want to limit the records that are indexed, you can add a WHERE clause, or issue SET SCAN, SET FILTER and SKIP commands as needed.

Another alternative, if you want to limit the records that are indexed, is to create a stored stack or result of those records in SPIRES and use the FOR STORED command rather than FOR SUBFILE. FOR STORED is more commonly used in SPIBILD with merging or removing records [See 2.4.2.] or [See 2.5.1.] For more information, see the description of FOR STORED in Part 6. [See 6.1.2b.]

Step 6: Issue the BUILD command.

Under Global FOR, the BUILD command has the following basic syntax:

where "index.name" is the name of the index record-type, as it appears in the INDEX-NAME statement of the file definition's linkage section. If you used File Definer to change the definition, "index.name" is the name you specified with the RECNAME attribute.

The "index-name" specifications of the BUILD command allow a number in parenthesis which chooses one occurrence of INDEX-NAME when there are many having the same "index-name". For example, assume your file-definition had the following:

If you specified: BUILD APPLE, PEARS, ORANGE(2) then the first INDEX-NAME = ORANGE; would not be built.

If you specified: BUILD ORANGE, PEARS then only INDEX-NAME = APPLE; would not be built.

The BUILD command under Global FOR builds indexes for ALL goal records (that meet WHERE-clause and SET SCAN criteria).

Continuing the above example, if you are adding the two indexes ZIN14 and ZIN15 to this PLAYERS subfile,

Handling Problems

System failure

System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.

Another alternative is to determine whether any of the records being indexed have been indexed: return to SPIRES, select the subfile and try searching for various records using the new index. If no records in the subfile were indexed, start the procedure again at step 3. If any of the records were successfully indexed, try to determine which was the last record to succeed and then restart the procedure at step 3, using a WHERE clause and/or SET SCAN commands to resume the indexing from that point.

Gibberish in the indexes

Very serious problems will arise if you mistakenly add a new index record-type between existing record-types in the file. The problem can manifest itself in several ways:

It is very tedious to recover from this problem, particularly if you don't discover it till after using the BUILD command. That's because if you used the BUILD command, you passed data to one or more indexes already containing data, not to your empty index record-type. SPIRES added your new record-type to the end -- that's where the empty one is -- so if you didn't name the last record-type in the BUILD command, more data was added to at least one that already existed.

To recover, you will have to determine which record-types have been harmed by the BUILD command and zap them. Correct your file definition, placing the record-types in their proper order, and recompile it; then try rebuilding.

It is certainly possible that rebuilding the file is the quickest solution to this problem. It may not be the cheapest way, but it may be the least harrowing. Moreover, that gives you complete flexibility in adding and removing indexes.

2.7.2  Removing an Index from a SPIRES Subfile

This procedure is an extension of the ZAP RECORD-TYPE procedure described earlier. [See 2.5.2.] The steps of that procedure will be listed as part of this procedure, but you should see the earlier section for details if you need them.

Do not remove any record-types

You cannot actually delete a record-type of a SPIRES file -- it must always be there until such time as the file is completely rebuilt. [See 2.9.] What you can do is remove all the records from it and change the file definition so that no data is passed to it, leaving it an empty record-type (which you can use again later if you want).

Be sure that the record-type is used only as an index, and that it doesn't contain other data that makes it useful for other reasons besides as an index (e.g., for table-lookup purposes).

Step 1: Use the ZAP RECORD-TYPE procedure.

First, you must remove the index records, using the ZAP RECORD-TYPE procedure. The basic steps of that procedure, described in section 2.5.2, are:

Example

The procedure usually looks like this:

Step 2: Change the file definition.

At this point, it is important to understand that you aren't actually removing the index record-type from the file but are in fact emptying it and telling SPIBILD not to pass data to it anymore.

You can follow this step by changing the file directly, or by using File Definer. By changing the file definition directly, you can actually disconnect the index record-type from the subfile. By using File Definer, you are slightly more limited (though the differences can be considered trivial) and you must do more work.

Use File Definer only if you still have the File Definer input for the current version of the file definition.

To change the definition using File Definer, read 2a; to change it directly, read 2b.

Step 2a: Change the file definition using File Definer.

Locate the input line where the index you want to remove is "defined". It probably looks something like this:

Following that line, add the line:

which tells SPIBILD not to pass the element's data to that index.

If other elements are passed to the same index, add the same line to their descriptions too.

WARNING: Do not simply remove the INDEX attribute from the input line. This will definitely cause serious problems, unless the moon is in your seventh house.

The $NOPASS Passproc tells SPIBILD not to pass data to the index, but the linkage to the subfile is still there; e.g., the index still shows up under SHOW INDEXES, even though it is empty. Thus, you will probably want to hide the index using the PRIV-TAGS and NOSEARCH attributes in File Definer. See the "File Definer" manual for details.

Next, use File Definer to generate a revised file definition:

The full procedure, with more information about how to make changes to files using File Definer, appears in the "File Definer" manual.

Step 2b: Change the file definition directly.

In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the index you want to remove, and delete those lines from the file definition. They usually look like this:

In general, remove the group of lines that starts with the appropriate INDEX-NAME and ends with the PTR-GROUP statement.

WARNING: Do not remove the record definition for the record-type from the file definition. The file will not recompile without it.

Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.

Step 3: RECOMPILE filename

The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.

For example,

If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary.

As a final step, you might want to verify that the index is gone by returning to SPIRES, selecting the subfile, and issuing the SHOW INDEXES command.

Handling Problems

System failure

System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.

2.7.3  Rebuilding an Index

This procedure is useful in the following circumstances:

 - to change the values in an index when the index's Passproc rule has been changed (e.g.,  so
 that the index will be a word index instead of a phrase index);
 - to repair a damaged index (e.g., an index whose Passproc was changed previously without the
 index being  rebuilt,  so  that  old  records  being  removed  or  updated  aren't  getting
 "un-indexed" properly);
 - to remove an element's values from the index (e.g., you want to remove  the  TITLE  element
 values from a SUBJECT index that had both TITLE and SUBJECT element values);
 - to add an element's value to an index (e.g., you want to add the TITLE element values to  a
 SUBJECT  index  so  that  it will have both TITLE and SUBJECT element values).  [You may be
 able to do this without erasing the current index first. Here's a summary of the  procedure
 you  can  try:  1) Change the file definition so that the new element (TITLE in the example
 above) is passed to that index, but stop  any  other  elements  from  being  passed  to  it
 (SUBJECT).  2)  Use  the  BUILD command to pass the TITLE data to the index for the records
 currently in the subfile. 3) Change the file definition again so that all the elements  are
 passed  to  the  index for future records. WARNING: This procedure can't be done using File
 Definer to change the file definition; you  must  change  the  file  definition  directly.]

This procedure can build multiple indexes

If you need to rebuild several indexes, you can follow this procedure too, zapping each index separately in step 1, changing the file definition all at once in step 2, and then building the indexes simultaneously in step 6 with the BUILD command.

If you are adding a new element to an index, make sure that the types of data you are placing into the single index are compatible. In general, for instance, you wouldn't index an element stored as an integer with one stored as a string. (You could do it if you convert the integer value to a string value in the Passproc rule string.)

Preliminaries

For several steps of this procedure, you need to know the name of the index record-type you are rebuilding, which appears in the file definition. Select the FILEDEF subfile and transfer or display the definition for your file.

To find the name, find the SEARCHTERMS statement that lists the index's search terms; just ahead of that statement will be the INDEX-NAME statement, containing the index record-type's name. For example, if you are trying to find the name of the record-type containing the TITLE index, you might find these lines in the file definition:

The name of the record-type for the TITLE index is ZIN02.

As another preliminary step, you might consider making a backup copy of the file in case problems arise during the procedure. [See 2.1.4.3.] Problems where a backup copy is useful arise primarily because of carelessness during step 2, when the file definition is being changed. Hence, you might want to create a test version of your file, making the changes there and testing them before committing the changes to your production file.

Step 1: Use the ZAP RECORD-TYPE procedure.

The basic steps of that procedure, described in section 2.5.2, are:

Example

The procedure usually looks like this:

Step 2: Change and recompile the file definition, if needed.

If you are rebuilding a damaged index, you probably don't need to make any changes to the file definition at all, and can continue to step 3. However, if you are changing how or what values are passed to the index, you will need to make changes to the file definition -- most likely to the index's linkage section and possibly to the index record-type's record definition.

You can follow this step by changing the file directly, or by using File Definer. Use File Definer only if you still have the File Definer input for the current version of the file definition.

To change the definition using File Definer, read 2a; to change it directly, read 2b.

Step 2a: Change the file definition using File Definer.

If you are changing the type of value indexed, rather than the elements being indexed, follow these instructions: Locate the input line where the index you want to change is "defined". It probably looks something like this:

Make the appropriate change to that line. For example, to convert it to a word index:

which tells SPIBILD to split the element's value into individual words for indexing.

If you are adding an element to those for this index, follow these instructions: Locate the input line for the specific element, e.g.:

Add the INDEX attribute, followed by the name of another element whose values are indexed in the relevant index:

If you are removing an element from those passed to a given index, follow these instructions: Locate the input line for the specific element, e.g.:

If the INDEX attribute names another element, simply remove the INDEX attribute:

However, if the INDEX attribute does not name another element, DO NOT simply remove the INDEX attribute from the input line. This will invariably cause serious problems, unless you lead a charmed life. You need to find the name of the index record-type created for this element (it's in the real file definition, in the INDEX-NAME statement), and find another element whose values are also being passed to the same index. For example, if you are removing SUBJECT from an index that also contains TITLE, and the index record-type is REC05, you could change the input lines as follows:

For further help along these lines, see the SPIRES manual "File Definer" or contact your SPIRES consultant.

After making changes to the File Definer input, use File Definer to generate a revised file definition:

The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.

Step 2b: Change the file definition directly.

In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the index you want to change, and make the appropriate changes. Then select the FILEDEF subfile in SPIRES and replace the old version with the new. Next, recompile the file definition, using step 2c.

Step 2c: Recompile the file definition.

Simply stated, the procedure to follow is:

If the file definition does not compile properly, SPIRES will tell you so; return to step 2a or 2b as appropriate.

If the file compilation is successful, press on to step 3.

Step 3: SPIBILD

Issue the SPIBILD command online, or in a BATWYL job:

In a batch SPIBILD job, you are already in SPIBILD, so you omit this command from the JCL.

Step 4: ESTABLISH subfile-name

Issue the ESTABLISH command to process the file's deferred queue and to tell SPIBILD the name of the subfile containing the index or indexes to be built.

You can build all the indexes for a single goal record-type at one time. If you are adding indexes for several different sets of goal record-types in the same file, you need to return to this step for each goal record-type.

At this point, you may also want to increase the size of the pass stack to increase passing efficiency, using the SET PASS STACK command. See the description for that command in Part 6. [See 6.3.0a.]

Step 5: Establish the Global FOR environment.

With this step, you tell SPIBILD which goal records to use to build the index. Usually, you will simply issue the FOR SUBFILE command:

Generally speaking, with this particular procedure, you want to build the new index from all the goal records. However, if for some reason you want to limit the records that are indexed, you can add a WHERE clause, or issue SET SCAN, SET FILTER and SKIP commands as needed.

Another alternative, if you want to limit the records that are indexed, is to create a stored stack or result of those records in SPIRES and use the FOR STORED command rather than FOR SUBFILE. FOR STORED is more commonly used in SPIBILD with merging or removing records [See 2.4.2.] or [See 2.5.1.] For more information, see the description of FOR STORED in Part 6. [See 6.1.2b.]

Step 6: Issue the BUILD command.

Under Global FOR, the BUILD command has the following basic syntax:

where "index.name" is the name of the index record-type, as it appears in the INDEX-NAME statement of the file definition's linkage section.

The "index-name" specifications of the BUILD command allow a number in parenthesis which chooses one occurrence of INDEX-NAME when there are many having the same "index-name". For example, assume your file-definition had the following:

If you specified: BUILD APPLE, PEARS, ORANGE(2) then the first INDEX-NAME = ORANGE; would not be built.

If you specified: BUILD ORANGE, PEARS then only INDEX-NAME = APPLE; would not be built.

The BUILD command under Global FOR builds indexes for ALL goal records (that meet WHERE-clause and SET SCAN criteria).

Continuing the above example, if you are rebuilding the index ZIN02 in the MY NOTEBOOK subfile:

Handling Problems

System failure

System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.

If any of the records were successfully indexed, try to determine which was the last record to succeed (return to SPIRES and use the index to try to find various records); then restart the procedure at step 3, using a WHERE clause and/or SET SCAN commands to resume the indexing from that point in step 5.

Gibberish in the indexes

Very serious problems can arise if the changes you make to the file definition are not done carefully. Particularly when File Definer is involved, it is easy to create a new file definition that is not compatible with the old. These problems manifest themselves in various ways:

It can be difficult to find the cause of such problems; to recover from one can be even harder, particularly if you don't discover it till after using the BUILD command. You should probably see your SPIRES consultant as soon as possible. But before you do, you should get a copy of the previous file definition out of the FILEDEF subfile (you could use the TREE or TRANSACTIONS class in Global FOR, for example) in case you need to go back to it. Your consultant will probably need it too, along with a copy of the new definition.

Zapping the wrong record-type

If you erase the wrong record-type in step 1, you will want to try restoring the entire file, using the most recent copy made. Usually this means contacting your SPIRES consultant, who can help make arrangements to restore the file using backup copies made recently by your computer center. If you have your own backup copies for your application, you may want to use those instead. [See 2.1.4.3.]

2.8  Adding, Removing and Rebuilding Subfiles in a File

The previous chapter covered how to add, remove or rebuild an index of a subfile. In this chapter, we go a step farther -- we will add, remove and rebuild subfiles of existing files.

To be more specific, this chapter refers to files already in existence that have (or will have) more than one subfile and more than one set of goal records. Here we will add a new subfile to an existing file, or remove one subfile from a file, leaving other subfiles of the file alone, or rebuild one subfile of a file, leaving the rest of the file alone. [Adding a new subfile to a file that doesn't exist is the same as creating a new file, which is covered in the manual "SPIRES File Definition." Removing the only subfile from a file is the same as destroying the entire file, and rebuilding the only subfile of a file is the same as rebuilding the entire file, two procedures discussed later in this manual. [See 2.9, 2.11.]]

The organization of Chapter 2.8

The three sections of this chapter are:

2.8.1  Adding a New Subfile to a SPIRES File

Adding a new subfile to an existing file is a fairly simple procedure on paper -- what can complicate it is the design of the subfile.

This section describes the procedure when you are adding a new subfile whose goal record-type will be a new record-type in the file. [If the new subfile will use an existing record-type as the goal record-type, your task is very easy: you change the file definition by adding a new Subfile section describing the subfile and then place the new version in the FILEDEF subfile. You can then select the subfile immediately -- you don't need to recompile the file definition. [See 2.6.1, 2.7.1 if you then want to add elements or indexes to it.]]

Preliminaries

You need to choose a record-name of no more than 6 characters for each new record-type you are defining (e.g., REC13). This is crucial to you even if you use File Definer to create and change the file definition.

New record-types being added to an existing file must be added to the end of the existing record-types. Since SPIRES sorts them in alphanumeric order by name before compiling them, it is crucial that the new record-types you define follow those that already exist. [The exception to this rule involves DEFQ-only record-types. If the file contains any of these, whose names make them sort to the end of the record-types, the new ones must precede them. See the "File Definition" manual for more information.]

For example, if the file already has the three record-types REC01, ZIN01 and ZIN05, the new goal record-type would need a name that sorted after ZIN05, such as ZIN06 or ZJA01. You could not add a goal record-type named REC02 unless you were rebuilding the entire file. [See 2.9.]

The names of all current record-types appear in the RECORD-NAME statements of the file definition as stored in the FILEDEF subfile. Another way to find their names is to select a subfile of the file in SPIRES and issue the command PERFORM FILEDEF SUMMARY.

Keep in mind that a file can have only 64 record-types, which must be combined into no more than 9 ORVYL data sets (i.e., "filename.REC1" through "filename.REC9"). In addition, no more than 8 of them can be slot record-types, each of which must be in its own data set, alone. Hence, you could not add a slot record-type if you are already using all 9 of the allowed ORVYL data sets.

Step 1: Change the file definition.

To execute this step, you'll need to know specifically what changes you want to make to the file. For help, you may want to consult other SPIRES manuals on file definition, such as "File Definition" or "File Definer". As always when you are making major changes to an important file, you should consider making a test version of the file and making your changes to that one first, to be sure that they work before applying them to the real file.

You can follow this step using File Definer or by changing the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file definition.

WARNING: Whichever method you use, be extremely careful to follow the directions, to ensure that the new index record-type definitions follow the definitions for those that already exist. Failure to do so may result in data integrity problems that are very tedious and unpleasant to correct.

To change the definition using File Definer, read 1a; to change it directly, read 1b.

Step 1a: Change the file definition using File Definer.

Add the information for your new subfile at the end of your input file. Be sure to include the GOAL keyword with the name you chose for the goal record-type, based on the rules given under "Preliminaries" above.

Then, use File Definer to generate a revised file definition:

The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.

Step 1b: Change the file definition directly.

In SPIRES, get the file definition from the FILEDEF subfile and add to it:

 - a record definition for each new goal and index record-type; and
 - appropriate linkage sections for each goal record-type.
 - a Subfile section for each new subfile.

For more detailed information about coding file definitions, see the manual "SPIRES File Definition".

Once you have made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.

Step 2: RECOMPILE filename

The RECOMPILE command recompiles the named file definition, replacing the master characteristics and creating any new ORVYL data sets requested.

For example,

If SPIRES detects errors, it will tell you. Return to step 1 to correct them, if necessary.

SPIRES may not recompile the file definition if there are records in the deferred queue. It may tell you to process the file in SPIBILD before trying to compile the definition again. If so, follow the PROCESS/ESTABLISH procedure, and then return to SPIRES, reselect the FILEDEF subfile, and continue with step 2.

Check your work

When SPIRES successfully recompiles the file definition, you are strongly urged to select the subfile and make sure the new subfile has no records in it. If it isn't empty, then you didn't add the new record-type to the end of the existing record-types, which means your file is on the brink of disaster. You should return to the first step and correct the file definition, making sure the new record-type or -types appear at the end of the record definitions in the file definition. (See "Handling Problems" below.)

For similar reasons, it is a good idea to check the other subfiles in the file too: examine one or two records in them to verify that they are in the proper subfile.

If the new subfile is indeed empty, try adding a couple of records to it. If it works as you wanted, congratulations! If not, you need to make more changes to the file definition, starting again at step 1.

Also, if you defined any indexes for the subfile, try processing the file in SPIBILD and then checking the indexes to make sure the data was passed from your new goal record-type as you expected. [See 2.2.]

Handling Problems

System failure

System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.

New record-types in the wrong place

Very serious problems will arise if you mistakenly add a new record-type between existing record-types in the file. The problem can manifest itself in several ways:

It can be very tedious to recover from this problem, particularly if you don't discover it right after you compile the file. If you do discover it right away, you should correct the file definition (placing the old record-types in their original order and moving the new ones to the end), returning to step 1.

Of course, contact your SPIRES consultant if you need help solving any problems that arise. If you're reluctant to ask for help, keep in mind that making mistakes as you "solve" the problem can be more disastrous than the original problem was. Your consultant would prefer being called in sooner rather than later.

2.8.2  Removing a Subfile from a SPIRES File

This section describes the procedure when you are removing a subfile of a file by removing all the records from its goal record-type and indexes. [You can also eliminate a subfile by simply removing its Subfile section from the file definition -- the records will remain in the file, but users will no longer be able to select the subfile.]

Use ZAP FILE to discard the entire file

It's assumed here that the file contains other record-types you want to save. If you are getting rid of all the record-types of the file, you should follow the procedure for destroying an entire file, which is much simpler and more efficient. [See 2.11.]

Unless you zap and rebuild the entire file, you can't actually delete a record-type of a SPIRES file. What you can do is remove all the records from it and change the file definition so that no data is placed in it or passed to it, leaving it an empty record-type (which you can use again later if you want).

You can't use File Definer

WARNING: The instructions for this procedure presume you can and are willing to change the file definition directly, without using File Definer. Though it isn't inconceivable that you could use File Definer to make the appropriate changes, that method could be extremely complicated, and complete instructions would take several pages. If you absolutely must use File Definer because you want to continue using it for any future changes to the file, you should either rebuild the file completely (removing the unneeded subfile definitions from the File Definer input) or see your SPIRES consultant for help. Do not use File Definer on your own unless you are very certain how to do so without taking away or rearranging any record-types.

Depending on the way the data was distributed in the ORVYL data sets, you may or may not be able to release any ORVYL blocks containing the subfile's data. In other words, it is possible that eliminating a subfile will not make the file use any less storage space, though that space will be used again if more records are added to other record-types in the file. (See step 1 below.)

This procedure is an extension of the ZAP RECORD-TYPE procedure described in detail earlier in this manual. [See 2.5.2.] The steps of that procedure will be listed as part of this procedure, but you should see the earlier section for details if you need them.

Step 1: Use the ZAP RECORD-TYPE procedure to remove the goal and index records.

The basic steps of that procedure, described in section 2.5.2, are:

In this command, name one of the record-types of the subfile you want to destroy. Use the ZAP DATA SET command if all the record-types in a given data set are record-types you want to eliminate. [See 2.5.2.]

You need to repeat this procedure for each goal and index record-type of the subfile, restarting each time with step ii. Of course, do not zap any record-types of the subfile that are used by other subfiles you are not destroying.

File blocks no longer needed will be released by the ZAP DATA SET command, and also by the ZAP RECORD-TYPE command if no other record-types are in the same data set.

Example

The procedure usually looks like this:

Step 2: Change the file definition.

At this point, be sure you understand that you aren't actually removing the subfile's record-types from the file but are emptying them and telling SPIRES and SPIBILD not to put data into them anymore.

You should follow this step by changing the file directly. (See the earlier warning against using File Definer.)

In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Find the linkage section for the goal record-type of the subfile you want to remove, and delete those lines from the file definition. It will begin with a GOALREC-NAME statement:

where "recname" is the 1-6 character name of the record-type. Delete all the lines from there up to (but not including) the next GOALREC-NAME or EXTDEF-ID or SUBFILE-NAME statement, whichever comes next.

Next, find the Subfile section(s) of the file definition that names the subfile(s) you want to remove. They begin with the SUBFILE-NAME statement:

Delete all the lines from there up to (but not including) the next SUBFILE-NAME, FILE-ACCESS, ELEMINFO, CHECKPOINT, INDEXINFO, EXT-REC or EXT-DEF statement.

Though it isn't necessary, you may want to remove most (but not all) of the contents of the definitions of the various record-types you have emptied. If you do this, do not delete the COMBINE statement, and be sure to leave at least a key element defined -- otherwise, the file definition can't be recompiled.

WARNING: Don't remove any record definitions from the file definition completely. You can't recompile the file definition without them.

Once you've made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.

Step 3: RECOMPILE filename

The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.

For example,

If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary.

As a final step, you might want to verify that the subfile is gone by trying to select it. It is a good idea to select the remaining subfiles of the file and examine them too.

Handling Problems

System failure

System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.

2.8.3  Rebuilding a Subfile in a SPIRES File

Rebuild to incorporate major subfile changes

If you decide that one of your subfiles needs major changes, you may need to rebuild it -- that is, remove all the records from its goal record-type and indexes, make appropriate changes to the file definition, put the goal records back in, and rebuild the indexes.

Typical reasons for rebuilding a subfile are these:

 - You want to make changes to the goal record-type, adding some new elements,  removing  some
 old  ones you don't need to keep, or changing the types of values stored for others.  These
 changes also affect one or more of the indexes.
 - The goal records have slot keys, and you  want  to  renumber  them  all,  eliminating  gaps
 created by removed records.
 - "Available space" information indicates that much of the storage space  the  file
 is  using is empty.  By zapping data sets of the file and rebuilding them, unneeded storage
 space will be released.  (But see below.)

It's assumed here that the file contains other record-types you want to leave untouched. If you are rebuilding the only subfile of the file, you should follow the procedure for rebuilding an entire file, which is much simpler and more efficient. [See 2.9.]

Unless you zap and rebuild the entire file, you cannot actually delete any index record-types of a SPIRES subfile. What you can do is remove all the records from it and change the file definition so that no data is passed to it, leaving it an empty record-type (which you can use again later if you want).

Rebuilding may release unneeded file blocks

Depending on the way the data was distributed in the ORVYL data sets, rebuilding a subfile may or may not release any ORVYL blocks containing the subfile's data. Blocks are released when an entire data set is zapped or when all the record-types in a data set are zapped. So, if a subfile's data is in ORVYL data sets shared by other record-types of the file, rebuilding the subfile will not make the file use any less storage space, though that space will be used again when the records are added back in.

Preliminaries

Test your changes first

Depending on the amount and intricacies of changes you want to make to your file definition, you may want to begin with step 3, creating a revised file definition, which you then test thoroughly before "starting" the procedure described below.

Make a backup copy of your file

CAUTION: In this procedure, you will be zapping data. Anytime you are zapping data, you should compare the importance of the data to your confidence that you can complete the procedure. If the data's value outweighs your confidence, consider making a backup copy of the current file and file definition so that you can quickly recover the file to its current state in case of any problems. [See 2.1.4.3.]

Step 1: Dump and save the current goal-record data.

You should create a data set of the goal-record data you want to put back into the subfile. There are several techniques to use in SPIRES:

 - If you are not changing the goal record-type's definition, you can use  the  GENERATE  LOAD
 command.   It will create a data set of records in their internal form, which can be loaded
 into the empty goal record-type later.  It is the most efficient tool to use,  if  you  can
 use it.  [See 2.3.3.]
 - If you are changing the goal record-type's definition, then you want to create a  data  set
 that  includes  only the elements you want to add back in.  Select the old subfile, use the
 SET ELEMENTS command to choose those elements, and using Global FOR commands (such  as  FOR
 SUBFILE  and  IN  ACTIVE  DISPLAY  ALL), create the data set and save it.  If the subfile's
 records have slot keys, you should omit it from the SET  ELEMENTS  list  if  you  want  the
 records to have new keys.  (See example below.)
 - If desired, you might want to create a format to use, rather than the  SET  ELEMENTS  list,
 for  both  output  and later input.  If you do have a format, the method is the same as the
 one just described.

Example

For the examples in this section, suppose we are rebuilding a subfile called ALARM LOCATIONS in the file GQ.DOC.ALARMS. We want to save all the elements except for the slot key, called RECNUM. There is no "SET ELEMENTS ALL -- elem" command to let us directly set all but a few elements, but an indirect way does exist. In brief, you can use a "SET ELEMENTS ALL + element" and a "SET ELEMENTS -- element" command; by using the two of them in combination with a virtual element (or a dynamic element created on the spot -- either way giving us an additional element to set in addition to "all" of them), you can achieve the desired effect with the SET ELEMENTS command.

Two other items worth noticing are that the $RECSEP variable is set so that each record begins with an "add;" line, and that an index is used to limit the records saved to only those added since 1984.

If the amount of data may be too large to fit in your active file, consider using the batch SPIRES program to write the data directly to a WYLBUR (OS) data set. See the SPIRES manual "Searching and Updating", or online, [EXPLAIN JCL, FOR SPIRES.] for more information about batch SPIRES.

Step 2: Use the ZAP RECORD-TYPE procedure to remove the goal and index records.

The basic steps of that procedure, described in section 2.5.2, are:

You need to repeat this procedure for each goal and index record-type of the subfile, restarting each time with step ii. Of course, do not zap any record-types of the subfile that are used by other subfiles you are not rebuilding.

Example

The procedure usually looks like this:

Step 3: Change the file definition.

If you have no plans to change the file definition, continue with step 6.

For major applications, the file definition will usually be revised and tested as part of the Preliminaries. If so, in this step you should replace the old file definition with your newly revised and tested one.

To use File Definer...

You can always make the changes directly to the file definition. In addition, you may be able to use File Definer to change the file definition if all of the following conditions are true:

 - You have the File Definer input for the current version of the file definition.
 - You are not deleting any indexes from the definition.
 - You are not rearranging any elements that are indexed.
 - You are not adding any new indexes to the definition.  [If you are adding new indexes,  you
 can  use  File  Definer  by  carefully following the directions described in section 2.7.1,
 under "Preliminaries" and step 1.]

If any of the above conditions are false, you should change the file definition directly, as described below in 3b; if you can use File Definer, read 3a.

Step 3a: Change the file definition using File Definer.

Make the desired changes to your input file, making certain that your changes are within the guidelines described above. Then use File Definer to generate a revised file definition:

Step 3b: Change the file definition directly.

In SPIRES, get the file definition from the FILEDEF subfile (or from wherever you keep your working copy). Make the desired changes to it, keeping the following rules in mind:

 - Don't remove any record definitions from the file definition completely.  You must have  at
 least as many as you did before, and the ones that remain in the file from the past must be
 in the same numerical position after the file definition is recompiled.
 - If you add new record-types, be sure to add them at the end of the list of those already in
 existence.  Remember that SPIRES will sort the records alphabetically by name.  A  list  of
 all  the  record-types,  in sequential order, is available through the PERFORM FILE SUMMARY
 command.

Once you've made the desired changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.

Step 4: RECOMPILE filename

The RECOMPILE command recompiles the named file definition, replacing the master characteristics with the newly compiled information.

For example,

If SPIRES detects errors, it will tell you. Return to step 3 to correct them, if necessary.

Step 5: Reload the saved goal-record data.

You are now ready to reload the data you saved in step 1 into the new subfile. Depending on the form in which you saved it, use either the INPUT BATCH, INPUT ADD, LOAD, FAST BATCH or FASTBILD procedures. [See 2.3.]

As a final step, you might want to verify that the subfile is working as you'd expect by returning to SPIRES and trying to use it. It is a good idea to select the remaining subfiles of the file and examine them too.

Handling Problems

System failure

System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.

2.9  Rebuilding an Entire SPIRES File

If you decide to make major changes to a file, you may need to rebuild it -- that is, make copies of all the goal records in its various subfiles, zap the file, compile a new file, and load the goal records back in. In most ways, this procedure is like the one for rebuilding a subfile, but on a larger scale. [See 2.8.3.] Yet in some ways, it's easier -- you'll find it's easier to zap the entire file rather than individual record-types, and you have complete freedom in creating a new file definition, including the use of File Definer.

People rebuild SPIRES files for many different reasons, such as:

 - to rebuild a subfile, when the subfile is the only one in the file;
 - to completely eliminate unneeded record-types;
 - to make major changes to multiple goal record-types, such as adding new elements,  removing
 old ones, etc., anywhere you like;
 - to make major changes to the indexing  structure  (adding  new  ones,  removing  old  ones)
 without having to worry about the order in which the old ones were defined;
 - to make major changes in several different areas at once, which could perhaps  be  achieved
 by  using  several  different procedures but which can be done more quickly and easily with
 this one single procedure.

As the list above suggests, this procedure is the catch-all technique for repairing a file. You could compare it to repairing a car that won't run by buying another car. The solution is overkill if the problem simply involves the sparkplugs.

But if the engine has many different problems or much more serious ones, buying a new car may solve them more easily and even more inexpensively (in the long run) than solving them each individually. And consider the advantages of "trading up to a newer model" as you decide how to solve the problems. Moreover, in regard to the sparkplug problem, buying a new car solves the problem (albeit expensively) in a way that keeps your hands clean if you don't want to get them dirty. [We hope this is the closest that SPIRES documentation ever comes to sounding like a car dealer.]

Preliminaries

Depending on the amount and intricacies of changes you want to make to your file definition, you may want to begin with step 2, creating a revised file definition, which you then test thoroughly before "starting" the procedure described below.

CAUTION: In this procedure, you will be zapping data. Anytime you are zapping data, you should compare the importance of the data to your confidence that you can complete the procedure. If the data's value outweighs your confidence, consider making a backup copy of the current file and file definition so that you can quickly recover the file to its current state in case of any problems. [See 2.1.4.3.]

Step 1: Dump and save the current goal-record data.

Create a data set of the current data in each goal record-type you want to put back into the file. There are several techniques to use in SPIRES:

 - If you aren't changing that goal record-type's definition, you can use  the  GENERATE  LOAD
 command.   It will create a data set of records in their internal form, which can be loaded
 into the empty goal record-type later.  It is the most efficient tool to use,  if  you  can
 use it.  [See 2.3.3.]
 - If you are changing the goal record-type's definition, then you want to create a  data  set
 that  only  includes the elements you want to add back in.  Select the old subfile, use the
 SET ELEMENTS command to choose those elements, and using Global FOR commands (such  as  FOR
 SUBFILE  and  IN  ACTIVE  DISPLAY  ALL), create the data set and save it.  If the subfile's
 records have slot keys, you should omit it from the SET  ELEMENTS  list  if  you  want  the
 records to have new keys.  (See example below.)
 - If desired, you might want to create a format to use, rather than the  SET  ELEMENTS  list,
 for  both output and later input.  Except for setting the format, the method is the same as
 the one just described.

You may choose different methods for different goal record-types of the file.

Step 2: Change the file definition.

If you aren't making any changes to the file definition, you can skip ahead to step 3.

Because you have complete flexibility in changing your existing file definition or creating a new one, you can make any changes you want, within the limits of general file definition capabilities. (Remember, after step 4, you will no longer have a file with which the revised definition must be compatible, unless you saved the data in its internal SPIRES format using GENERATE LOAD.) That means you can use either File Definer or the file definition directly to make your changes.

Keep in mind the data that you kept in step 1, however. Be sure the new file will accept that data, e.g., that it will have the same element names, that Inproc rules will accept the same element values, etc.

When you are finished with the file definition, put it in the FILEDEF subfile, using the ADDUPDATE command.

Step 3: ZAP FILE filename

The next step is to destroy the old file, using the ZAP FILE command:

where "filename" is the name of the file to be erased.

For instance, if you want to zap your file FO.RTE.PIANOS:

Step 4: SELECT FILEDEF

You next need to reselect the FILEDEF subfile.

Step 5: COMPILE filename

The COMPILE command compiles the named file definition, verifying the logic of the file definition and creating the various ORVYL data sets of the file.

For example,

If SPIRES detects errors, it will tell you. Return to step 2 to correct them, if necessary, and skip step 3 when you try again.

Step 6: Reload the saved goal-record data.

You are now ready to reload the data you saved in step 1 into the various subfiles of the file. Depending on the forms in which you saved it, use either the INPUT BATCH, INPUT ADD, LOAD, FAST BATCH or FASTBILD procedures. [See 2.3.1.]

As a final step, you should verify that the subfiles work as you'd expect by returning to SPIRES and trying to use them.

Handling Problems

System failure

System failure during any steps of the procedure may cause some inconvenience but generally nothing more serious than that.

2.10  Moving a SPIRES File from One Account to Another

Sometimes it is necessary to move an entire SPIRES file from one account to another. For example, you might want to move a file from a "test account" to a "production account", or from an account you want to close to a new account.

To move a file from one account to another (say, file GQ.DOC.RESTAURANT from account GQ.DOC to account GQ.JNK), follow this procedure:

Step 1: Make sure the copying account has enough ORVYL blocks.

First, find out how large the file is. If you are the file owner, issue the ORVYL command SHOW FILES: [The copying account can issue the SHOW FILES command if that account has general access (READ or higher) to all ORVYL data sets on the file owner's account, as determined by the ORVYL command SET PERMIT.]

For example, from account GQ.DOC:

Add the number of blocks (BLKS) shown to determine the size of the file.

Next, on the copying account, see how many blocks are available, using the ORVYL command SHOW BLOCKS:

As long as the difference between the number of blocks allowed and the number currently being used ("accounted") is larger than the size of the file being copied, you can copy the file to that account.

Step 2: Give the copying account Copy access to the file.

In the file definition, the file owner must give the copying account Copy access to the file. If you have the File Definer input for the current version of the file definition, you can use File Definer to change the file definition (2a below); otherwise, change the file definition directly (2b).

Step 2a: Change the file definition using File Definer.

To use File Definer, first add the COPY attribute to the FILE statement, listing the accounts to be given Copy access:

Then use File Definer itself to generate the revised definition:

Step 2b: Change the file definition directly.

To add the information directly to the file definition, get the most recent copy and add to the end an occurrence of the FILE-PERMITS structure that includes a FILE-ACCESS value of COPY and an ACCOUNTS value that includes the copying account:

Then select the FILEDEF subfile in SPIRES and replace the old version of the file definition with this new one.

Step 2c: Recompile the file definition.

The procedure to follow is:

If the file definition doesn't compile properly, SPIRES will give you error messages; return to step 2a or 2b as appropriate.

Step 3: Make sure the copying account has ORVYL access to the file.

This step can be skipped as long as the file owner has not set different ORVYL permits for the file than the defaults set by SPIRES when the file is established. If desired, use the ORVYL command SHOW PERMIT to verify that each of the file's data sets allows READ access to the copying account. If any do not, use the SET CLP and SET PERMIT commands in combination to change them.

Step 4: Use the COPY FILE command to copy the file

The COPY FILE command, available in SPIRES, copies the ORVYL data sets of a SPIRES file to the copying account:

It must be issued from the copying account, NOT from the file owner's account. As it begins copying each data set, SPIRES displays a message naming the data set and the number of blocks to be copied:

If you have any problems with the COPY FILE command, see Part 6, or online, EXPLAIN COPY FILE COMMAND.

Step 5: Create a file definition for the new file.

To define subfiles for your copied file, you must have a file definition. In general, it is best (easiest) to get the original file owner's file definition or file definer input and modify it accordingly, changing the account number in the file name, changing the subfile names, etc. (If, for example, you had an older copy of the file definition and the file owner had changed the file definition in the meantime, recompiling your copy with the old definition could cause problems serious enough to force you to begin the procedure again.) Be sure to add your new copy of the file definition to the FILEDEF subfile so that you can issue the SELECT command to use its subfiles.

Step 6: Cleanup: Other considerations

Depending on the circumstances, you may need to continue with the following procedures:

 - If there are any formats associated with the original file, you may want them to  work  for
 the new file too.  You should consider either:
 - getting copies of the format definitions from the format owners, changing  the  ID  and
 FILE  statements  within  them  to  reflect your ownership of the format definition and
 file, and then adding them to the FORMATS subfile and compiling them; or
 - getting the format owners to add GEN-FILE statements to  their  format  definitions  so
 that  they  can  be  used by both files.  (That would mean setting the formats slightly
 differently, using the "general file format" form of the SET FORMAT command.)
 - If special ORVYL permits were set for the data sets you copied (which is likely if the file
 has immediate indexes, for instance), they were not brought across to the copies.  Instead,
 SPIRES sets the same permits it does for a new file: READ access to the public for all data
 sets  except the DEFQ, which gets public WRITE access.  If you want to set the same permits
 as the original file, you need to find out what they were from the file owner (who can  use
 the "SHOW PERMIT filename.dataset" command to recall them) and then issue the SET
 CLP and SET PERMIT commands to set them on the data sets of the new file:
       SET CLP filename.dataset
       SET PERMIT filename.dataset FOR account access-level
       SET NOCLP filename.dataset
 - If you now want to destroy the original file,  follow  the  procedure  "Destroying  an
 Entire File" in the next chapter.  [See 2.11.]

2.11  Destroying a SPIRES File

When you no longer want to keep a particular SPIRES file on your account, you can destroy it, following a procedure based on the ZAP FILE command. The example session outlined below shows how account GQ.DOC would destroy its file GQ.DOC.ERRATA.

Step 1: Make sure you no longer want the file.

If you are sure you want to destroy the SPIRES file but aren't absolutely sure that you will never again need the data within it, you might want to save copies of the records, storing them on tape, or on another account, etc. Or you might want to consider copying the entire SPIRES file to another account, using the COPY FILE procedure. [See 2.10.]

Step 2: Get rid of formats code associated with the file.

When you are destroying a file, please remove formats associated with the file from the FORMATS (format definitions) and FORCHAR (compiled format code) subfiles. To do so, follow this procedure:

There may be other records in system files relating to your file, such as EXTDEF, VGROUPS or SYS PROTO records. Please remove any records from those subfiles that are used only by the file you are discarding too.

Step 3: ZAP FILE [gg.uuu.]filename

The ZAP FILE command in SPIRES erases the ORVYL data sets that comprise the SPIRES file.

Because of the seriousness of the ZAP FILE command, SPIRES asks you to confirm that you want to destroy the named file. Once you confirm it, the data sets are gone.

Step 4: Cleanup: the File Definition

You should remove the file's file definition from the FILEDEF subfile. Before doing so, you should consider whether to save the file definition elsewhere, just as you did for the data in step 1. You can store old file definitions in a public SPIRES subfile called BACKFILE. Since the file is owned by the computing center, you are not charged any storage costs for keeping the definition there.

In this example, account GQ.DOC decides to move the file definition for GQ.DOC.ERRATA into the BACKFILE subfile:

2.12  Using FASTBILD to Build Subfiles and Indexes

FASTBILD is a batch program you can use when you need to add large numbers of records to a brand new subfile, to build all the indexes of a large subfile, or to build a new index for a large subfile. Though it does work that SPIBILD can do, FASTBILD does the work much more efficiently under some circumstances, which means it can save you time and money. In addition, the data sets it builds are somewhat better built than SPIBILD's, meaning that record retrieval from them is more efficient.

Use FASTBILD to build a subfile if...

FASTBILD is an alternative to SPIBILD only in very particular circumstances, however. Specifically, you should consider FASTBILD for building a new subfile when all of the following are true:

 - you are adding 10,000 or more records to an empty goal record-type at one time;
 - at least one index is being built from the goal records;
 - the records are in sequential order by key in the input data set;
 - the  record-type  has  never  had  any  records  in   it   since   it   was   compiled   or
 "zapped";
 - no goal-to-same-goal passing is involved, i.e., the goal record does not pass data to other
 goal records in the same record-type.

Use FASTBILD to build all the indexes if...

You should consider using FASTBILD to build all a subfile's indexes if both of the following are true:

 - the data currently in the subfile will create 10,000 or more  index  entries  for  the  new
 indexes;
 - none of the index record-types have ever had any records in them since they  were  compiled
 or zapped.

Use FASTBILD to build one index if...

You should consider using FASTBILD to build a single index if both of the following are true:

 - you will be adding 10,000 or more index entries to the new index at one time;
 - the index record-type has never had any records in it since  it  was  compiled  or  zapped.

When CPU-usage is an issue and the preceding guidelines are met, you will get better results by carefully using FASTBILD rather than SPIBILD.

In most cases, file managers use FASTBILD to build new subfiles rather than to build new indexes alone. Hence, the discussions in this chapter are centered around the former use, though in general they are discussing both. However, if particular points need to be made about building new indexes, you will see them along the way.

The contents of Chapter 2.12

This chapter is arranged as follows:

2.12.1  Introduction to FASTBILD

What FASTBILD does

The basic FASTBILD task consists of two batch jobs:

 - The first job adds the input goal records to the empty goal record-type  and  places  index
 record  data  into  OS  files.   [If  the  FASTBILD task is to just build indexes, the goal
 records currently in the tree are read, and the index record data is put  into  OS  files.]
 The input data may be in one of several forms:
 - in standard SPIRES format; or
 - in a form that can be interpreted by a custom input format; or
 - in internal SPIRES format, created by a GENERATE LOAD command.
 - The second job sorts the index data and then builds  the  SPIRES  index  records  from  the
 sorted data.

The two jobs can be run on separate days

If you are building a subfile of hundreds of thousands of records, the two jobs could take many hours. You can set up the jobs so that the first job is run one day, and the second is run some other time. (You need to alter the JCL so that the index data is not placed on a scratch volume that might be erased before you run the second job.) In addition, the second job can be split into separate smaller jobs (perhaps one for each index being built) to further subset the procedure.

Developing good JCL for your FASTBILD run is very important. Most file managers use JCL that was created for them by a public SPIRES protocol called FASTBILD.JCL. The protocol actually creates the JCL for just the first job; the JCL for the second is created by FASTBILD as it handles the first job. FASTBILD can submit the second job for you automatically as the first job finishes up, though that is not generally recommended.

Why use FASTBILD

Unquestionably, FASTBILD uses less CPU than SPIBILD to do a comparable job when the job is large. The difference primarily comes from the indexing. It is much more efficient for FASTBILD's sorting package to sort all the index data at once than for SPIBILD to retrieve, update and replace index records in pass stack after pass stack. If you are building a subfile that has no indexes, FASTBILD offers no advantages over the FAST BATCH command in SPIBILD. [See 2.3.4.]

The fact that FASTBILD does not "checkpoint" the data as SPIBILD does also contributes to the difference. Checkpointing the data as blocks are written to the data sets of the file is crucial to recovery procedures, but it does add significant overhead. FASTBILD gives up the checkpointing in the name of speed, but at the price of having no graceful way to recover if the system fails -- you usually have to start over. (Doing the work in "steps" may allow you to restart from the last completed step.)

However, there are rare situations where you cannot use FASTBILD. First, no terms being passed from a goal record-type to an index record-type (whether the key, pointer, qualifier or sub-index term) can exceed 255 bytes. Also, if the pointer-group in the index record-types being built is a structure, it must begin with the key (the pointer); in other words, it cannot have Fixed element terms before a key in the Required section. If building the file with FASTBILD is important, you may need to change the file definition to avoid these possibilities.

Reasons why FASTBILD is forbidding

Many file managers who should consider using FASTBILD have traditionally shied away from it for various reasons. Instead, they have used SPIBILD, using much more CPU and creating less efficient files. Here are some of their reasons for not choosing FASTBILD, along with some sympathetic rebuttals.

There's no doubt that FASTBILD can be complicated and foreboding; it's "real programming", involving analysis of your input data and creation of JCL to do the needed processing. But building extremely large files can cost thousands of dollars; doing it less efficiently in SPIBILD will cost even more over the long run. The immediate savings in building costs and the longterm savings in data retrieval from having a more efficient file will make the work worthwhile.

Considerations for Very Large Files

To build a subfile of hundreds of thousands of records in FASTBILD, you should give some thought to the following:

More information about these suggestions appears in the rest of this chapter.

2.12.2  The FASTBILD Procedures

The procedure described here works for three different types of tasks: building a goal record-type and its indexes, building all of the indexes from an existing goal record-type and building just an index from an existing goal record-type.

Procedure may take several days

This procedure is extremely long; it will take you some time just to read it, and in execution it often takes several days to move from step 1 to step 9. It is not meant to be rushed through in a hurry. Please check with your SPIRES consultant if any questions or problems arise that you can't resolve on your own.

Before you begin the procedure described below, be sure that FASTBILD is appropriate for your task, by reading the sections preceding this one. [See 2.12, 2.12.1.]

Step 1: Create the JCL for the first FASTBILD job.

This step encompasses several preliminary steps. Before creating the JCL, you need to know some information about the input data and about the file definition:

 - the number of goal records being added (or passed to the new index or indexes);
 - the maximum lengths for all values being passed;
 - the average number of entries being passed to each index for each goal record.

The FASTBILD.JCL protocol needs these numbers to estimate how much space to reserve for the index data sets it will create.

Having a copy of the file definition will help you determine these figures and help you answer questions the FASTBILD.JCL protocol will ask you.

Step 1a: Determine how many goal records are being added.

You should get a rough number of how many goal records you will be adding to the goal record-type (or how many goal records you will be passing, if you're using FASTBILD to build new indexes only).

Warning: A low estimate could cause trouble, so definitely estimate high. If the value is too low, it is possible that the allocated space for the OS data set containing the data for an index won't be large enough, and the FASTBILD job will abend.

Step 1b: Determine the maximum lengths for all values being passed.

This step is fairly important. It asks you to analyze the input data and the structure of the index record-types being built. Specifically, you should determine the maximum length for each type of value that will be passed to any of the indexes being built. You need to know if the file definition imposes any length restriction and/or if the input data itself has only shorter lengths for those elements.

FASTBILD sorts fixed-length index records

The FASTBILD.JCL protocol needs this information for another reason besides for its estimates of the space required. The index-data records in the OS data sets that FASTBILD sorts in the second job are in effect fixed-length records, with each element starting in a fixed column. FASTBILD needs to know how much room to leave for each field, so that there is enough room in the field for the longest value.

On the other hand, it is in your interest to supply the lowest possible value for each field, because sorting efficiency increases as the length of the sort records decreases. That's why you should know, for example, if you have an element whose maximum length is allowed to be 50 characters but all the input values have lengths under 10.

Important: The value you are interested in is the length of the value that will be passed, not the element's input length or its stored length. For instance, the maximum length of a text element might be 200 bytes, but if it is word-indexed, then the longest passed value will be the longest word, which is surely much less than 200 bytes long. Similarly, a coded value stored as a 1-byte integer might be passed in its expanded "decoded" form, which might be longer.

There is an absolute limit in FASTBILD of 255 bytes for any value being passed; the element value, the pointer, the sub-index value, any local qualifier, and any global qualifiers must each be no longer than 255 bytes.

You need to get this information together now, because later, in the middle of the FASTBILD.JCL protocol (step 1e), SPIRES will show you the elements to which data is being passed in each index record-type and ask you for the maximum length specifications of each.

To sum up, be as accurate as reasonably possible, but err on the high side rather than the low. The efficiency gained by declaring a 10-byte length rather than 12, for example, is insignificant compared to the trouble of later handling any records with 12-byte values that were truncated in passing if your guess of 10 was wrong.

You can find out the lengths of element values being passed by examining the file definition and the input data.

Examine the file definition

By looking at the index-record definitions and the linkage section of the file definition, you can determine what elements will be passed to the indexes (those are the only elements you're interested in) and possibly what their maximum passed length will be. For example, most indexes contain only a key value (the value being indexed) and a locator back to the goal record-type. The locator is always a 4-byte hex value, so that length is easily determined.

Examine the input data

You should also examine the input data, especially if the lengths of the values being passed to indexes are variable. That is, if you are passing an element to an index via FASTBILD and you don't know how long the longest value to be passed is, you should find out by examining the data.

It may be impossible or too difficult for you to determine the maximum length, especially if you have no easy access to the input data (e.g., it is coming from a tape). In that case, you might want to simply use the default or make a reasonable guess. Remember that, in many cases, these values are values that users may type in search commands or as keys of records. Is it likely that a user will type a search value longer than 25 characters?

Whatever the case, FASTBILD will truncate values that are longer than the maximum you specify and add an error message to the FASTBILD output. The error will not stop FASTBILD.

Truncate the values before FASTBILD does

For indexed elements whose maximum length can't be easily determined, file owners frequently add a $MAX.LEN proc to both the SEARCHPROC and PASSPROC statements for the element, limiting the passed values' lengths to the maximum specified -- longer values are truncated. They then specify the same length in the FASTBILD JCL; that way, the truncation is intentional, and no errors are reported. This also helps searchers, since any long values they enter are truncated to the same length as the stored values.

Step 1c: Estimate how many index records will be created.

You need to tell the FASTBILD.JCL protocol the average number of index records that the average goal record will create for each index. For example, an indexed goal-record element that is a required, singularly-occurring numeric value will probably pass that one value to the index. On the other hand, an optional singularly-occurring text value broken into words for passing may pass from zero to hundreds of values to the index. [Technically, the number of records that will be created is the number of combinations of unique index key values, sub-indexes and qualifiers.]

Warning: As in the estimate for the number of records being added or passed (step 1a), this estimate should be on the high side, rather than the low. If the value is too low, it is possible that the allocated space for the OS data set containing the data for an index won't be large enough, and the FASTBILD job will abend.

Step 1d: If possible, reduce the input processing to be done.

The less Inproc processing that needs to be done, the less time it will take FASTBILD to build the goal record-type. Eliminating Inproc rules not needed for this particular data can save a significant amount of processing. [Even better is to avoid any Inproc processing by providing the data in SPIRES internal format, via the GENERATE LOAD command. Of course, the data can't be in that format unless it's coming from another SPIRES file having the same internal structure. [See 2.3.3.]]

For example, if you have an Inproc that tests the range of a numeric value and you know for certain that all the records you are building have values within the range, you could save money by removing that Inproc rule from the file definition for the duration of the subfile-building. Similarly, table-lookups done by $LOOKUP and $SUBF.LOOKUP procs that verify values can often be omitted; they are particularly expensive to do. Thus, you might have two file definitions for the file -- one with full editing and indexing for day-to-day use and a special one just for the FASTBILD task. Naturally, the more records being input, the more savings this step can provide.

Step 1e: Use the FASTBILD.JCL protocol to create the JCL.

In SPIRES, issue the command "XEQ FROM FASTBILD.JCL":

If the XEQ command fails with an S256 error ("no such record"), issue the command SET COMPXEQ + PUBLIC PROTOCOLS and then try the XEQ command again.

Answer the questions the protocol asks, a procedure that usually takes two to ten minutes, depending on the complexity of your task and your familiarity with the questions the protocol asks.

Instructions for answering most of the questions in the protocol are given in the next section of this manual. [See 2.12.3.]

Save the JCL

When the protocol ends, it leaves the JCL for the first job in your active file. It is a good idea to save it somewhere before continuing.

Step 1f: Make changes to the JCL if needed.

Few file managers use the JCL created by the FASTBILD.JCL protocol exactly as is. Sometimes changes must be made; for example, if the input data is from a "generated load" in internal SPIRES format, or if you are using the SCAN GOAL RECORDS procedure to build all indexes, the FASTBILD commands in the JCL must be changed. Sometimes you may want to add options that don't affect the processing but affect the output from the jobs. In either case, be very careful about making changes -- make sure you understand the effects of any line you add, change or delete.

When you are finished making changes, save a copy of the JCL somewhere -- you may want to refer to it later, or use it again.

A section later in this chapter will describe the commands you have to work with. [See 2.12.4.] For now, here is a list of some commands that are commonly added:

 - LIST ADDS
 - SET FORMAT
 - ALLOCATE
 - SET SEARCH
 - SET MESSAGES
 - LOAD
 - SCAN GOAL RECORDS

By default, FASTBILD will not display individual lines about the added records in the output (in contrast to SPIBILD).

A recommended change is to increase the TIME parameters on the JOB and EXEC statements and on the EST command, perhaps doubling the amount of time FASTBILD uses (expressed in CPU minutes). FASTBILD's estimate is sometimes too low, especially when the input format is doing extra work (e.g., creating a report). Remember that you pay the bills -- if the guess is too low, the job stops in mid-stream. Increasing the time estimate is strongly recommended.

You may also need to change the LINES parameter on the JOBPARM statement, especially if you request that the ADDS be listed. Again, the job will fail if the estimated lines of output are exceeded, so it's generally better to give a high number. [On the other hand, if successful records are not being listed in the output (i.e., you don't have LIST ADDS in the command stream), then if the job fails because of a low LINES number, it's probably because a large number of records are failing to be added and many error entries are being written. In that case, the job failure might be desirable.]

Changes needed to put index data in permanent data sets

When FASTBILD creates the index data in the first job for the second job to use, it puts the data sets onto a scratch volume for temporary storage. At Stanford, the data on the scratch volumes is erased daily, at about 6 a.m. To avoid the risk that the data will be discarded before the scratch volume is erased, or to run the second job at a later date, you should have FASTBILD put the data into permanent disk storage. [It is also wise to check that the volume where FASTBILD is going to place the index data has plenty of empty space; if not, you need to find one that does.]

First, find a volume with enough free space to hold the data sets. You need to determine from the SPACE allocations in the JCL how much space FASTBILD is asking for. (Your SPIRES consultant can help you with this step.) Then change the lines of JCL that begin with SCRATCH:

changing the VOL specification to "VOL=DISK=volume", where "volume" is the name of the permanent-storage volume you want to use. Likewise, change the DD statements later that actually create those data sets:

Change "VOL=SER=SCR002" to "VOL=SER=volume" in the first line, and change "UNIT=SYSDA" to "UNIT=DISK" in the second. Do this for each DD statement.

Note: By following this procedure exactly, you will also move the FASTJCL data set to a permanent volume -- that's the data set that contains the JCL for the second job, which the first job creates.

If you can't find enough room on any volume for the data sets, you should probably write them to tape. See your SPIRES consultant if you need help changing the JCL to request tapes.

You may also want to delete the SETUP lines at the start of the JCL that tell the operator the job needs the scratch volume. If you are using tapes, replace it with a SETUP card that asks the operator to mount them.

Changes needed when the input is coming from a tape

If the input data is stored on a tape that will be read during the FASTBILD job, you need to change the JCL to tell FASTBILD. The changes must be made to the DD statements in the GO step of the job that have the names "GO.INnnnn", where "n" is a digit from 0 to 9, such as:

A discussion of the information needed appears earlier in this manual, in connection with batch SPIBILD. [See 2.1.3.2.] Be sure to include a SETUP line at the start to ask the operators to mount the tape.

Changes needed for multiple input data sets

When you have several input data sets (you may have up to 16), the FASTBILD.JCL protocol will create skeletal DD statements for all except the first one, named "GO.IN0000". You will need to flesh them out, with appropriate DISP, UNIT and VOL parameters as needed. The data sets will be concatenated when they are needed in the order in which they are specified.

Changes needed for a load

If you are loading a subfile using input from a GENERATE LOAD command in SPIRES, there are several changes you must make to the JCL:

 - Delete the two lines of JCL about the IN0000 data set that look like this:
     //GO.IN0000 DD DISP=OLD,UNIT=DISK,
     //  DSN=WYL.gg.uuu.name
 - Replace the command INPUT ON IN0000 with the LOAD command, telling FASTBILD the name of the
 ORVYL data set containing the data to be loaded:
     LOAD ORV.gg.uuu.dsname

Changes needed to build only the indexes

If the goal record-type is already built and you are using FASTBILD to build all its indexes, you must make the same kind of changes you would for a load. Specifically:

 - Delete the two lines of JCL about the IN0000 data set that look like this:
     //GO.IN0000 DD DISP=OLD,UNIT=DISK,
     //  DSN=WYL.gg.uuu.name
 - Replace the command INPUT ON IN0000 with the SCAN GOAL RECORDS command:
     SCAN GOAL RECORDS
 - If desired, you can break this part of the procedure into several steps, by adding the  SET
 SKIP,  SET  LIMIT,  SET  SCAN  START and SET SCAN STOP commands just ahead of the SCAN GOAL
 RECORDS command, as described later in this chapter.  [See  2.12.4.]

Step 2: Prepare the input data.

If you don't have the input data in the proper data sets, get it ready now. Make sure the data can be read by the input format you are using, whether it's a custom format or the standard SPIRES format.

If you are building indexes, skip this step -- the data is already in the subfile.

Records should be in sequential order

For your input, make sure the records are in sequential order by key, unless FASTBILD will be assigning slot keys to them. If they aren't in order, do not use FASTBILD -- the job will probably fail and you will be wasting money.

If the data is coming from another SPIRES file via the GENERATE LOAD command, make sure the load has been created.

Whatever the situation, be sure that the account under which the FASTBILD job is running has "read" access to the data.

Step 3: Make sure the record-type is completely empty.

If you are building a new subfile, the goal record-type and all the index record-types that will be built should be absolutely empty. In fact, they should never have had any data in them at all since the last time the file definition was compiled or the last time the record-types were zapped.

In other words, even if you have removed all records in all of the record-types, you should zap all the record-types before running the FASTBILD jobs. [See 2.5.2.] Alternatively, if the file has no other subfiles with data in them already, zap the entire file and compile it again. [See 2.9.]

If you are using FASTBILD to build indexes alone, the same rules apply to each index. If it's ever been "tainted" by records, it should be zapped before the FASTBILD job is run. [See 2.5.2.]

Step 4: Run the first FASTBILD job.

Make sure you no longer have the file attached (CLEAR SELECT) -- you don't want the FASTBILD job to fail because it can't get exclusive access to the necessary record-types. Then put the JCL in your active file if it's not still there and issue the RUN command to submit the batch FASTBILD job:

Now, sit back and try to relax. This is the time for a good book (perhaps something other than a SPIRES manual), a tall drink, quiet meditation or a little nap.

The job is finished

When the job finishes executing, fetch its output and examine it to be certain the job ran properly. You may find messages about truncated index values -- that is, values being passed that were longer than the maximum length declared for that index. In general, this isn't a problem at this point, unless there are thousands of them, in which case you might want to start again. It is also a serious problem if the pointer element is being truncated; SPIRES won't be able to use those pointers to find the right records. See "Handling Problems" at the end of this section for more on this subject.

If the job fails at some point, fetch the output and try to figure out what the problem was. Please see your SPIRES consultant if you want help. Depending on the problem, you may need to return to step 1 and alter the JCL, or change the input in step 2, restarting the procedure from that point.

Remember to zap the appropriate data sets to restore their pristine condition before resubmitting the FASTBILD job.

Step 5: Make a backup copy of the file and the index data sets.

Though you are only halfway through the batch work at this point, this step is very important for exceptionally large builds. It gives you a means of recovery back to this point in case the second job fails (but see step 6 below).

See your SPIRES consultant for help in using data set copy-and-restore utilities such as ORVCOPY. [See 2.1.4.4.]

If you are using FASTBILD only to build indexes, you may just want to copy the index data sets.

Step 6: Retrieve the JCL for the second job.

During the first job, FASTBILD generates the JCL for the second job. By default, it is placed on the scratch volume. Regardless of the default, however, you can find it by examining the JCL from the first job, looking for lines that look like this (substitute your own account for "GQ.JNK"):

The first line gives us the information we want: in this case, the JCL is stored on the scratch volume SCR002, as data set WYL.GQ.JNK.FASTJCL. You should get a copy of it from there and save it somewhere more permanent, for future reference.

If you are building a very large subfile, you may want to make changes to the JCL before running the job. Here are some changes to consider:

   //SORTOUT   DD DSN=WYL.GQ.JNK.REC020,DISP=(OLD,PASS)
      and
   //GO.REC02  DD DSN=WYL.GQ.JNK.REC020,DISP=(OLD,PASS)

Step 7: Run the second FASTBILD job(s).

As you did in step 4, be sure you don't have any subfiles of the file selected. Then put the JCL into your active file and issue the RUN command to submit the second FASTBILD job:

Again, wait for the job to complete. If you have several jobs to run, wait till each has completed before submitting the next.

If a job fails, it means that one of the indexes has not been built properly. Error messages should tell you what went wrong. You may need to restore the file to its state in step 5, or to simply build the failed index later. Your SPIRES consultant can give you some guidance here.

Step 8: Cleanup: Make a copy of your file.

Basically, you have finished the procedure. You should test the file in SPIRES to be sure it is working properly. Select the subfile, issue the SHOW SUBFILE SIZE command to see if its size matches your expectations, and use the new indexes to find and display some records.

Once you are satisfied that the file was built properly, you should make a backup copy of it, for safety's sake, using ORVCOPY. [See 2.1.4.4.]

Step 9: Get file statistics from the STATUS command.

The STATUS command will give you a set of statistics profiling your file in peak condition. These statistics are very useful for extremely large files, since they can be compared to STATUS command statistics later to see how the file's efficiency is holding up. [See 3.2.3.]

Handling Problems

In general, most of the problems that arise during a FASTBILD task were discussed under the appropriate steps.

Truncated index entries

One common problem that has several solutions is the truncation of index entries, which happens when a value to be indexed is longer than the maximum length specified for entries for that index (see step 1b). Warning messages about these values appear in the output from the first job.

One solution, if the allowed length was fairly long (such as the default length of 25) is to add the $MAX.LEN system proc to the index's Searchproc rule string, telling SPIRES to truncate the user's value at 25 characters for matching purposes.

If that solution doesn't appeal to you, you will need to remove the records with the truncated index values (the output will tell you which ones they are) and add them back in again. You will probably need to add the $MAX.LEN proc to the index's Passproc before you remove the records so that SPIBILD can create the proper index records to be removed. Then process the file, remove the $LSTR proc from the Passproc, recompile the file definition and add the record(s) back in again.

A more serious problem arises if the truncated value is the pointer back to the goal record. In that case, you absolutely must start again, specifying enough space for the longest key that can arise.

2.12.3  Using the FASTBILD.JCL Protocol

With a great deal of skill and experience in writing JCL, you could write your own JCL for FASTBILD jobs. However, just as File Definer takes care of the most tedious parts of creating a file definition, the public protocol FASTBILD.JCL does the same for FASTBILD JCL.

Using the protocol is just one part of the FASTBILD procedure, described in the previous section. [See 2.12.2.] It is not too difficult to use, though it asks you some questions that require you to be very familiar with the data you want to input and the file definition. When it is finished, it leaves the JCL in your active file, ready for you to run or to alter as necessary.

Stopping the protocol momentarily or for good

Warning: Pressing the ATTN/BREAK key at a prompt will usually stop the protocol, but it will also discard the work you've done so far. If you want to pause for a moment (e.g., to respond to a message), type a question mark at a prompt and then press ATTN/BREAK. This causes a session break, and you will be under WYLBUR's control. To return to the same place in the protocol, type the command GO. While you are pausing, leave your active file alone; the protocol is putting the JCL there as it executes.

Pressing the Return key in response to most prompts will give you more information about the question being asked, but beware of questions that ask you to press Return to get the default value.

The rest of this section discusses most of the questions the protocol will ask you.

Do you wish full tutoring?

If you answer YES to this question, the protocol will display a few lines of explanation before most of the questions it asks.

For your goal records, what is the name of your SPIRES subfile?

The data you want to input must be the goal records for a subfile. Also, the file containing the subfile must already exist, i.e., be compiled.

Is this a BUILD of a single index from existing records?

If you are building an entire subfile or all the indexes of an existing subfile, answer NO. If you are building a single new index for records that already exist in the subfile, answer YES to this question; the protocol will then ask you for the name of the index record-type you are building.

How many records are you adding (passing)?

You need to estimate how many goal records you are adding (or passing to the index being built). This estimate doesn't have to be very accurate -- if anything, it should be a bit high. If it is too low, FASTBILD may not allocate enough space for the OS data sets for the index data, which could cause the job to fail.

Do you want to build your file in steps?

Steps should be considered if you project that the ultimate build will take longer than 10 hours. [See 2.12.1.] Using steps gives you the capability of backing up to the previous step if the build fails at any point. The technique is not extremely difficult, but it is procedurally demanding.

In essence, the input is broken into as many as nine groups. Each group is associated with one step. A step is a single job that builds the goal record-type using a group of records. It also builds the OS data sets for the index records. When that job concludes, you make a copy of the file and the OS data sets, after which you submit the next job for the next step. That step adds the next group of records to the goal record-type and adds the new index data to more OS data sets, etc.

When the job that sorts and builds the index records begins execution, it combines the various OS data sets for a single index before sorting them.

Use the steps feature only if you are working in close cooperation with your SPIRES consultant. [A variation of FASTBILD that works with steps, called RESDS FASTBILD, was used to build a 4,000,000-record data base belonging to the Research Libraries Group. Essentially, it builds the entire file in OS (including the key tree) and then other jobs are run to move the file into the ORVYL file system.]

Your job estimates: TIME=n, LINES=n, BIN=Default

FASTBILD is guessing how much CPU time and lines of output the job will use. These numbers are used by the system monitor in scheduling jobs for execution. If the job exceeds the estimates, it will stop, leaving a partially built file. So in both cases, you should estimate high. Remember, you are charged only for the amount of CPU you use, not for your estimate.

The numbers are shown as "n" minutes of CPU and "n-thousand" lines of output. (LINES=3 means 3000 lines.) A good rule of thumb would be to at least double these. (Don't double your bin number though.)

Do you want your second job step automatically submitted?

If you plan to follow the procedure described in the previous section, answer NO to this question. Generally, it is a good idea to stop after the first job(s), check the output from it, and examine the created goal record-type before continuing on to the index-building job (i.e., the "step" referred to in the question; it does not mean "steps" as in the earlier question).

Indicate your input data set name in the form GG.UUU.NAME.

Here you name the data set containing the input data for the subfile. [The protocol does not ask this question if you are building a single index rather than a whole subfile, but it will if you are building all the indexes of a subfile using SCAN GOAL RECORDS -- simply make up an answer now and change the JCL later, as described in "Step 1f" in section 2.12.2.] It assumes the input is in a cataloged OS data set on disk for permanent storage (not on a scratch volume). If the data is elsewhere, such as in an ORVYL data set created by the GENERATE LOAD command or on tape, you will need to change the JCL later. [See "Step 1f" in section 2.12.2.] For now, type any value you want, but remember to change the JCL appropriately as the procedure in the previous section describes.

Any more input data sets?

If you have more input data sets to use for this step, answer YES, then respond to the next prompt with the name of the next one; otherwise, answer NO. You can specify up to 16 input data sets. Note, though, that you will need to make some significant changes to the JCL to get FASTBILD to use them properly. [See "Step 1f" in section 2.12.2.]

Do you wish the system default of 25 for your field sizes?

For the sort step, FASTBILD needs fixed-length fields. Here, and perhaps in the next few questions, you will tell it the lengths to use for each field for each element being passed to the indexes. The system default is very generous for most circumstances; in fact, if 25 bytes is much longer than the length of the longest value, you might waste some processing time sorting empty bytes. That's why it's good to provide more accurate numbers. Answer NO to begin.

If you answer YES, the next question you will be asked is "How many index entries..." below.

Specify the field sizes of your Common section:

The "Common section" are the elements that will each be passed to all the indexes being built, in other words, the pointer group, as defined in the Global Parameters of the linkage section. Thus, you would supply the maximum length for the pointer and for any global qualifiers you are passing. If you are passing locators as pointers, remember that they are always 4-bytes long; but if you are passing keys, you will need to do some analysis to determine their maximum length. It is crucial that you don't declare too small a size for the pointer -- if in doubt, specify a larger number. The maximum allowed size for a value being passed in SPIBILD is 255 bytes.

The protocol will show you the elements of the pointer group so you'll know how to specify the numbers. Enter the numbers in the order shown, separated by commas, with the lengths of any global qualifiers entered in order of their definition.

Specify your field sizes for <rec-name>:

The protocol will prompt you for field sizes for each index record-type to which the goal record-type passes records. It will show you which elements in the index record-type are receiving passed data, according to the linkage section. Again you will supply numbers representing the maximum length for the values being passed.

The first number always represents the key element of the index. For simple indexes, that's the value being passed. For compound indexes, it's a special internal value that SPIRES creates -- its length is always 2.

At some point after that will come the "Common section" again; that is, you must enter the letter "c" to represent its positioning in the index data. If the index has sub-indexes, they will precede the Common section; if it has local qualifiers, they will follow the Common section. For a compound index, the length of the longest element value being passed to the index is specified after the Common section, like a local qualifier.

The maximum allowed size for a value being passed in SPIBILD is 255 bytes.

How many index entries per record will be passed to <recname>?

This question, also asked for each index, requests the average number of index entries an average record will create for passing to this index. Precise accuracy is not crucial, but again, you should estimate high rather than low. Keep in mind that a word index may take a single element and split it into dozens of index entries.

Note: If you are building a single index, you will be asked no further questions.

Will FASTBILD be assigning the keys for your slot record?

If the protocol finds that the subfile has slot keys, it will ask you this question. Answer NO if the input data contains the slot key values; answer YES if the contrary is true.

Indicate Exception file name:

You can request an exception file, where FASTBILD will place input records that fail to go into the subfile. An exception file simplifies the process of finding the unsuccessful records and correcting them so that they can be put into the subfile. [See 2.1.5.2.] It has the same restriction that it has in SPIBILD: if the input lines exceed 235 characters, they will be truncated.

Be aware that exception file processing requires considerable overhead. In general, it is extremely useful; but it is expensive in terms of both CPU and I/O, and will slow down the processing.

All done...

The protocol will finish up, leaving the generated JCL in your active file. Save it somewhere if you're ready for a rest; otherwise, you might want to continue with the procedure for using Fastbild, altering the JCL if necessary, as described in Step 1f of the previous section.

2.12.4  FASTBILD Commands

If you examine the JCL generated by the FASTBILD.JCL protocol [See 2.12.3.] you will see about a dozen non-JCL commands at the end, the GO.SYSIN dataset. Those are FASTBILD commands, which you may want or need to change or delete. [See 2.12.2.] You may also find that additional commands available in FASTBILD are useful to you as well.

The entire list of recognized commands appears below, listed in the order in which they may appear in the JCL:

DEFINE filename

The DEFINE command specifies the name of the file that contains the goal record to be built. This command must be present and must precede the FAST BUILD command.

ESTIMATE TIME=n LINES=m

ESTIMATE is the command that specifies the CPU time estimate and printed output lines estimate for the second job. This command is optional, and defaults apply if it is not specified. The default will generate a 1 minute CPU time estimate and a 2-thousand line estimate (LINES=2 means 2,000 lines of output). Warning: If either of these parameters is exceeded, the job will terminate, leaving a partially built file; be sure to estimate high.

STEP=n[,L]

STEP indicates that the goal records are being added in steps. The "n" indicates which step is being specified. This optional command should only be used if very large files are being built. To indicate the last step, the command is STEP=n,L. This feature should be used only with the assistance of the SPIRES consultant.

SEQUENTIAL

This command specifies that the input for the goal records is in ascending key order. When this optional command is specified for non-sequential input the job will terminate; if it is specified for slot goal records, a FASTBILD error will occur.

Remember that you shouldn't use FASTBILD unless your data is in sequential order by key.

LIST ADDS

This optional command requests that a message be printed for each record added; the default is not to list adds, since large numbers of records are usually added in a FASTBILD run. Error messages are always listed, unless you turn them off with a SET MESSAGES command (see below). Warning: If you specify that adds be listed, be certain to allow enough lines in the ESTIMATE command above.

SET FORMAT format.name

This command, along with its options and alternate forms (SET FORMAT *, or SET FORMAT format.name 'parm', for instance), tells SPIRES to set the named format to be used for input processing. Be aware that, unlike SPIRES but like SPIBILD, FASTBILD will not execute any startup frame in the format except when the SET FORMAT * form of the command is used. [See the manual "SPIRES Formats", chapter C.9, for more information.]

You may also include SET FTRACE or SET FTRACE ALL at this point, if debugging information is needed. This may generate substantial amounts of output, of course.

Vgroup manipulation commands

The following vgroup manipulation commands may also be issued at this point:

The syntax of these commands can be found by issuing appropriate EXPLAIN commands, e.g., EXPLAIN ALLOCATE COMMAND. Other vgroup manipulation commands may be issued at the end of the FASTBILD procedure (see below). Note that vgroups established by setting the format will not be included in SHOW STATIC VARIABLES or SHOW ALLOCATED displays because the format is not actually set until the building begins.

SET EXCEPTION ORV.gg.uuu.name [REPLACE]

This command directs rejected input records to an "exception" sequential ORVYL file where they can later be manipulated for subsequent addition to the SPIRES file. [See 2.1.5.2.]

Other SET commands

Several other SET commands may be issued at this point, each of which is described elsewhere in this manual (see the index) or in other SPIRES documents (use the EXPLAIN command). They are:

Also, the CLEAR FILE COUNTS command is available to turn off the default file count settings.

FAST BUILD 'subfile-name' ACC gg.uuu

This required command indicates the name of the subfile to be built. FASTBILD selects the subfile using the privileges of the named account.

BUILD index-name OF 'subfile-name'

This command is used when a single index is being built from existing goal records. BUILD should not be included if FAST BUILD (above) is, and vice versa.

SPECIFY GLOBAL p,q1,q2,...

Because index record information is typically of variable length, and OS SORT cannot handle variable length fields, you have the option of a default field length of 25; or, you may enter specification parameters indicating maximum field sizes of a particular index. If data being passed is longer than the lengths specified, it is truncated at the appropriate length and a message is produced.

The SPECIFY GLOBAL command indicates the field sizes of the common global data, that is, the pointers to all goal records (or keys if the records are not Removed) and any global qualifiers. (GLOBAL is referred to as COMMON in the FASTBILD.JCL generator.)

The "p" indicates the maximum length of the data you expect in the global pointer or key field. If pointers are used, this length is 4; otherwise the maximum length of the goal-record key should be specified.

The "q" field(s) indicate the maximum length of the data to be passed to the global qualifier(s). Since there may be several such qualifiers, multiple "q" values may be specified in the order in which the qualifiers appear in the file definition.

The maximum allowed size for a value being passed in SPIBILD is 255 bytes.

SPECIFY RECn k,s,GLOBAL,v1,v2,...

The second type of specification must appear once for each index being built, indicating the field sizes of the data to be passed.

"RECn" is the record number of the record being specified. If the index records are the second, third and fifth record-types define in the file definition, then RECn would have the values REC2, REC3, and REC5. If an invalid record type is named, an error message is issued and the build will terminate.

The "k" field is mandatory and specifies the maximum length of the data that will be passed to the key of the index. If a compound index is being specified, then the key length is always 2. The "v" parameter must also be specified, to pass the values of elements to the index. The "s" field appears if and only if a sub-index is defined for the record-type. The "s" value specifies the length of data to be passed to this sub-index.

The word "GLOBAL" instructs FASTBILD to insert the GLOBAL specifications indicated in the first specification card. If GLOBAL is put in the wrong position, an error message is issued and the build is terminated.

The multiple "v" parameters specify local qualifiers. As many "v" parameters should appear as local qualifiers are defined for the index record-type. This parameter is also coded to pass element values to the CINDEX-VALUE element of a compound index. The "v" value is then the size of the longest element value passing to the compound index.

If a simple index has several INDEX-NAME sections passing to it, then several SPECIFY RECn commands must be supplied. One specification is necessary for each INDEX-NAME section that passes to the index; the specifications for the INDEX-NAME sections are made in the same order in which the sections appear in the file definition's linkage section. Other restrictions in this area: those INDEX-NAME sections must appear consecutively in the linkage section, and the first one declared there must pass the most terms (i.e., sub-index and local-qualifier terms, if applicable).

Accuracy in specifying the size of the fields is desirable. A smaller field size than the 25-byte default will speed up the OS sort, since the comparisons are across shorter fields, and will speed up the build, since a greater number of records will fit in the buffer. However, if field size is underestimated, items will be truncated in the indexes, an error message will be issued, and the build will continue.

The maximum allowed size for a value being passed in SPIBILD is 255 bytes.

INPUT ON IN0000

The INPUT command specifies that the batch data input is on IN0000. IN0000 is the "ddname" on the DD card in which the input dataset is described in the OS JCL; other DD names can be used. Up to 16 data sets may be declared for the single "ddname"; they will be concatenated together during the processing.

LOAD ORV.gg.uuu.dsname

If the data being entered into the subfile was created by a GENERATE LOAD command, this command, rather than the INPUT command above, should be issued.

SCAN GOAL RECORDS

If you are using FASTBILD only to build all the indexes for an existing subfile, you use this command instead of the INPUT command above. (A very specialized command, SCAN GOAL KEYS, retrieves only the record key of each record, useful if all the data for passing will come from the key itself.) This command has some auxiliary commands that can be issued before it that let you break up the task into multiple steps:

Vgroup manipulation commands

More vgroup manipulation commands are allowed here:

INDEX subfile-name

This command is generated by the first job if indexes were built, and appears in the second job of the FASTBILD run. If no index records are to be created, this command does not appear, and no second job will be run.

3  Keeping a File Tuned Up

3.1  Introduction

By building the tree data sets carefully, SPIBILD does a good job of maintaining the efficiency of the average SPIRES file. However, because no one file is likely to be maintained with perfect efficiency, it may be worthwhile for a file owner or manager to occasionally check up on SPIBILD by monitoring various statistics available for the file. These aspects of file ownership are usually of concern to owners of large files or of files with many, many update transactions.

In some cases, the statistics may show that the file has empty space in it that SPIBILD may not be able to use properly for some time, or that a high amount of data entry for records with a given key prefix is making a tree lopsided, which could cause some inefficiency when SPIRES retrieves those records. These problems, and others to be covered in this chapter, are seldom critical ones, but they often have a longterm effect on the overall cost of storing and using a file. Using procedures described in this chapter, you can correct these problems if you decide it is worthwhile to.

These descriptions will tell you not only how to correct the problem, but will also try to give you some idea of whether the "problem" is worth correcting. These topics involve some rather esoteric concepts and features in SPIRES, and the effort and the expense of learning about them and solving them may surpass the value of correcting them.

This chapter discusses various ways you can measure and, if desirable, improve the storage efficiency of your SPIRES files. In specific, it presents commands:

 - to  tell  how  much  storage   space   the   file's   data   sets   are   using   [See   3.2.1.]
 - to tell how much of that space is being used for  data  [See  3.2.3.]
 - to tell whether the empty space in blocks will be used appropriately as transactions occur,
 and to correct the situation if they will not [See 3.4.]
 - to  tell  how  much  of  the  file's  storage  space   is   internal   overhead   [See   3.2.2.]
 - to tell whether the record trees are built well for maximum retrieval  efficiency,  and  to
 correct the situation if they are not [See 3.3.]

These commands are all available in SPIRES. In versions of SPIRES from mid-1987 through mid-1991, they were available only in SPIUTIL, a separate program containing several maintenance utilities for SPIRES files. SPIUTIL has been completely incorporated into SPIRES and is no longer needed; if you invoke it, it just puts you into SPIRES.

3.2  Gathering Statistics about Tree Growth and Space Usage

To determine whether your file is in good condition, you need statistical information about it: how many physical blocks of storage space is it using; what percentage of each block contains data; what is the average depth of each key tree, etc. This chapter will present several commands that gather and display various sets of statistics, and will give you some guidelines for reading them.

Keep in mind that if you own a small file, you probably do not have to gather and examine this information. And even if you own large files, you may not need to concern yourself with it -- but if you are concerned about your file running as efficiently as possible, you should examine these statistics occasionally.

Organization of Chapter 3.2

Specifically, the subsections of this chapter cover:

3.2.1  Determining the Physical Size of a SPIRES File

For budgetary purposes, you may need to know how many blocks of storage your file uses; after all, you are charged for the number of blocks you use.

No single command in SPIRES will tell you how large a file is, but an ORVYL command, SHOW FILES, can tell you how many blocks each data set of a file is using. The example below shows how account GQ.DOC would find out how much space the GQ.PUB.RESTAURANT file is using:

By adding up the numbers of blocks (BLKS) shown, you can determine the storage size of the file in ORVYL 2048-byte blocks. To get the blocks information, include either the BLOCKS or ALL option on the SHOW FILES command. See Part 6 for more information, or online, [EXPLAIN SHOW FILES COMMAND.]

To find out how many blocks your account is using as a whole, issue the ORVYL command SHOW BLOCKS:

There will be a difference between the total number of blocks accounted and the total blocks listed for all data sets. To that total, you must also add one block for each 660 blocks of each data set; these blocks are not part of the data set per se, but are stored in ORVYL's index blocks and are charged to you. Non-IXR data sets do not have these blocks. Online, [EXPLAIN GENERATE NONIXR COMMAND.] for more information, or see its description in Part 6.

3.2.2  Determining the Storage Space for Individual Records

Knowing how much space a goal record is actually using can sometimes be useful in making estimates of storage space requirements for a large file. The SHOW RECORD OVERHEAD command can tell you how many bytes of storage are being used by a single record, by a sample of about 25 records, or by all the records in the selected subfile.

The example below demonstrates the command:

The most significant figure is the length value at the end of each display. The sample group of 26 records (requested by omitting the option) has a total length of 8613 bytes. The single record 226-82-3846 has a length of 511 bytes.

The length value tells you how much space the data in the record (plus the length and occurrence information kept within the record as specified in the file definition) is using. For this command, the length and occurrence information represent the "overhead".

The same information is also available with the SHOW RECORD INFORMATION command, which has the additional advantages that it can be used in Global FOR mode to process records by criteria other than the record key, and that it can process multiple records, rather than only a single record. See Part 6 for more information about these commands, or online, [EXPLAIN SHOW RECORD OVERHEAD.]

3.2.3  Getting File Statistics from the STATUS Command

Though the STATUS command's most important function is arguably to read and validate the contents of all the blocks of one or more data sets, it also gathers and presents a slew of statistics about the file as it proceeds. The statistics are saved in the file's residual data set, where the SHOW FILE STATUS command can retrieve them to present a summary of the status data. [Much of the data you will get from the STATUS command is also accessible from the SHOW FILE STATUS command. However, the data displayed by SHOW FILE STATUS is only as recent as the latest STATUS command.]

The tables may strike you as boring or fascinating, but in either case, interpreting them correctly can give you some insight into how efficiently your file is growing and may suggest changes to the file you might want to consider. In addition, your SPIRES consultant may ask you to "run a STATUS" to provide some additional information if you ever have serious problems with a file.

Though this procedure is quite simple to execute, it is not meant to be done casually. SPIRES must read every block of the file, which may be quite a bit of work -- the procedure may be expensive. Unless you need the results pronto, it is worthwhile to run the STATUS when your computing rates are the cheapest.

1) ATTACH filename

Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:

This example demonstrates using the ATTACH command with the file GQ.DOC.RESTAURANTS.

2) Issue the STATUS command.

Basically, the STATUS command has the following syntax:

Though it is really optional, you are urged to include the IN ACTIVE prefix, so you're not trying to read and interpret the lengthy display of data as it scrolls off your terminal screen. It's preferable to put the statistics in your active file and examine them there or print them.

Use the AVSP option

It's also a good idea to add the AVSP option, which stands for AVailable SPace. Then the STATUS command also displays information about how much empty space exists in the residual's blocks and how much may be "misplaced". [See 3.4.]

You don't have to issue the STATUS command for ALL the data sets of the file -- ALL can be replaced by other options, described in detail in Part 6 of this manual. [ALL does not include the deferred queue, however. If you need to find out the average depth of the deferred queue's tree, you can replace ALL with DEFQ, as described in the back.]

STATUS reports any file problems

If the STATUS command encounters any problems with the file (e.g., a tree is approaching the maximum depth, or a block is improperly written), it reports them as part of the statistics.

4) Examine and evaluate the results.

In terms of commands to execute, the procedure is now complete. If you followed the advice here and put the statistics in your active file, you might want to print it now.

Residual statistics first

The statistics about the residual data appear first, followed by data about each record-type in the file, in sequential order by name (e.g., REC01, ZIN02, ZIN03, etc.). The data shown for each record-type varies depending on whether it is slot or non-slot.

Only SPIRES consultants and the most enthusiastic SPIRES aficionados will be interested in all the details shown, so they will not be explained in detail here.

The most useful data from the STATUS command concerns these areas:

 - errors uncovered while validating the blocks
 - balanced tree growth
 - available space

The errors are discussed below. The other two areas are discussed in their own chapters following this one. [See 3.3, 3.4.]

Remember that some of the most interesting information captured by the STATUS command can be seen through the SHOW FILE STATUS command. See Part 6, or [EXPLAIN SHOW FILE STATUS COMMAND.]

Errors Uncovered While Validating the Blocks

In the STATUS command's output, error messages may appear that identify blocks in a data set that have serious problems detected by the STATUS command. Use the EXPLAIN command to get more information about the errors, and if so advised, contact your SPIRES consultant for help as soon as possible.

Errors the STATUS command may uncover

Remember that the SPIRES command reads every block of the residual data set in order, validating the block's contents, i.e.,

 - verifying that the  block's  "trailer"  (data  at  the  end  of  the  block  that
 identifies the length and starting location for each record or record segment in the block)
 does not contain overlapping entries, and that the trailer values are valid;
 - verifying that the first four bytes of  the  block  match  the  last  four  (a  "check
 number" based on an ID for the transaction that last modified the block);
 - verifying that other block numbers appearing in the block (e.g., the current block  number,
 the number of a block where the current record continues, etc.) are valid.

Another error message you might receive if you are working with a large Residual data set is:

which signals that SPIRES could not verify all the links between blocks in the residual because of memory limitations.

Handling Problems

If the system crashes at any point during the procedure, then restart the procedure at your convenience.

If SPIRES detects problems with your file as it compiles the statistics, it will report them to you in the statistics as described above. In other words, problems within the file will not stop the STATUS command.

3.3  Rebalancing Tree-Structured Record-Types

Occasionally, a tree-structured record-type becomes "unbalanced", that is, the tree has more levels than it needs, resulting in slower retrieval of records in the lower levels. It is usually caused by unusually extensive growth in one particular area of the tree. Suppose your subfile has a DATE index, which started out with a few dates scattered around the twentieth century but which recently only gets current dates. Hence, the tree may be very deep (have many levels) around the current dates and very shallow around the older ones. If that's the case, the record-type may need rebalancing.

Maximum tree depth is 16 levels

If the record-type is quite unbalanced (or extremely large), it may even reach the maximum tree depth of 16 levels. [SPIBILD will warn you during file-processing if any record-types of the file are getting close to the limit, with the message: "Warning: Tree depth of <recname> approaching limit". If SPIBILD processing would surpass the limit, the processing stops with an error, and the record-type must be rebalanced before SPIBILD recovery will work.]

What does a depth of 16 mean, and why should you be concerned? It means that SPIRES must read 16 blocks (i.e., execute 16 I/Os) to retrieve records at that level in the tree. (And if the desired record is removed to the Residual, add at least one more.) If most of the other records of the file reside at much higher levels, requiring less than, say, three I/Os for retrieval, then SPIRES is doing a great deal more work for the lower-level records than is necessary. Rebalancing the tree might mean that all records would require less than three I/Os, which would lead to a significant efficiency savings each time the records were retrieved.

How to Determine Whether a Tree Needs Rebalancing

For files containing record-types of, say, less than 2000 records, you should probably just wait till the unlikely day when you receive the SPIBILD warning message about the tree-depth limit. If and when that day comes, you should rebalance the file immediately.

For larger files, you should check the status of the trees occasionally (perhaps every few months) or even frequently for very large files (50,000 records and up).

The SHOW FILE STATUS command

A formula to help you determine whether a tree is balanced or not uses information displayed by the SHOW FILE STATUS command (as gathered by the most recent STATUS command for the file). [See 3.2.2.] In particular, compare the number of accesses per record to the tree depth for each tree-structured record-type. (The data for slot record-types, for the Residual and for the Deferred Queue is irrelevant in this context.) Note that the number shown for accesses per record includes any access to retrieve any Removed records in the Residual.

Here are the relevant columns and rows from a SHOW FILE STATUS command:

INDEX5 has a much lower average number of accesses per record than its tree depth, suggesting it as a likely candidate for rebalancing.

The STATUS Command

A more graphic and definitive indicator that a record-type needs rebalancing appears in the output from the STATUS command, in the second table displayed for each record-type. Here's one for a large index record-type:

The pattern to look for is a rise and fall in the number of nodes and blocks as the tree gets deeper. (The deeper the tree, the higher the level number -- thus, level 1 is the top-most level.) If the fall is greater than 1 level, then the record-type could benefit from rebalancing. In the above example, the numbers drop from level 3 to level 6. Ideally, the numbers should rise, or the last level should be the only one to drop from the previous one.

Is it worth rebalancing?

Note however that of the approximately 200,000 records (nodes) in the example record-type, only about 6700 are in levels 5 and 6, the "unnecessary" levels that rebalancing would eliminate. You need to consider whether the improved retrieval efficiency for 3 percent of the records in the record-type is worth more to you than the savings from not rebalancing the record-type.

Consider these ideas: First, the lower-level records are likely to be newer records (i.e., more recently added) than the higher-level ones. If users will be doing most of their searching for newer records, then the number of I/Os will be higher than necessary for most searches. Second, if the tree is unbalanced now, it is likely to become more unbalanced rather than less as time goes on.

Other Benefits of Rebalancing

Rebalancing a tree provides other benefits as well:

Though a single record-type may need rebalancing, all the record-types in the data set containing that record-type get rebalanced too. In other words, the REBALANCE command applies to data sets, not to individual record-types.

Organization of this chapter

The remainder of this chapter describes the REBALANCE procedure. The first section describes the steps of the procedure, while the second discusses how to handle unusual situations and problems that can arise during rebalancing.

3.3.1  The REBALANCE Procedure

Suppose you decide you need to rebalance the ERRATA.REC2 data set of your file ERRATA. When you issue the REBALANCE command with the RENAME option, SPIRES takes the following three actions:

 - 1) makes a copy of each tree in the data set, rebalances it and places the copy in a  new
 data set (ERRATA.REB2);
 - 2) renames the old data set containing the  tree(s)  to  "ERRATA.REO2.yyyymmdd"
 ("yyyymmdd"  is  the current date, e.g.  "19861231" for 12/31/86) and
 renames the new data set to ERRATA.REC2.  Thus the new replaces the old, but the  old  is
 kept around in case of any problems.
 - 3) resets the base-block numbers in the ERRATA.MSTR data set of the file, using  the  new
 values established when the data set was rebalanced.

Additional steps may be necessary when a problem occurs during the second part of the REBALANCE procedure. Information on the procedure involved in this circumstance appears in the next section. [See 3.3.2.] Online, EXPLAIN REBALANCING DATA SETS, SPECIAL CIRCUMSTANCES.

Steps of the REBALANCE Procedure

Below is the REBALANCE procedure, step by step, to be followed by the file owner. To understand the syntax, assume you are rebalancing the ORVYL data set "filename.RECn" (where "n" is an integer from 1 to 9, or letter from A to F) of the SPIRES file "gg.uuu.filename".

Preliminaries

Check the data set's ORVYL permit

The REBALANCE command will create a data set with the default ORVYL permit setting of Public Read (or Public Write, for data sets containing immediate indexes). That's the same setting SPIRES uses when it creates the "RECn" data set.

However, if you, the file owner, ever changed the permit, you will probably want to reinstate that level of security when the REBALANCE procedure is over. To that end, issue the command:

If the data set has permits other than the appropriate default, jot them down so that you can reset them at the end of this procedure.

Get rid of any interim rebalancing data sets

You should not have any ORVYL data set called "myfile.REBn" or "myfile.REOn.yyyymmdd" ("yyyymmdd" being today's date) when you issue the REBALANCE command, since SPIRES will create them. If either does exist, the procedure will stop. To see if you do, issue these ORVYL commands:

If you do have either of these, you should either erase or rename the data set, using the ORVYL commands "ERASE dataset.name" or "RENAME dataset.name TO new.dataset.name". (You will have to issue the ORVYL command "SET CLP dataset.name" prior to the ERASE or RENAME command if the data set was created by SPIRES.)

Get exclusive access to the file

WARNING: Since the REBALANCE procedure replaces both the data set being rebalanced and the MSTR data set of the file with new copies, at times it needs exclusive access to the file. It is best to rebalance data sets when you are sure no other users have any subfiles of the file selected; the procedure will go much more smoothly for you. In fact, if practical, ask users of your file not to use it during a given period of time when you are rebalancing it. You might even want to temporarily remove the file definition from the FILEDEF subfile so that no users can select any of its subfiles during the event.

But it is crucial that the trees being rebalanced not be changed during the REBALANCE procedure -- that means that the file should not be processed by SPIBILD during it, nor should any users update records during it if any trees being rebalanced are immediate indexes affected by the updates. Step 3 of the procedure will guard you against users updating any records while you are doing your work, but you are on your own to ensure no one processes the file while you are rebalancing.

Step 1: SPIRES

You must be in SPIRES to use the REBALANCE command:

Step 2: ATTACH filename

Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:

This example, which will be followed through this section, demonstrates using the ATTACH command on the file GQ.DOC.ERRATA.

Step 3: SET NOJOBGEN

As mentioned above, this command will prevent any users from updating records during the procedure.

It is really necessary only if any users logged on might update any records and if the file has any immediate indexes. But it cannot prevent any users with Process access from processing the file, which would cause major problems. Again, be certain no one would do that while you are rebalancing the data set.

Step 4: IN ACTIVE [CLEAR|CONTINUE] SHOW FILE ACTIVITY

In the unlikely event of a problem with the rebalanced data set, you may need to return to the "filename.REOn.yyyymmdd" data set, in which case you will need the base-block information from this display. Having a copy of this information is very important, so you should put it in your active file and save it somewhere till after you have successfully completed the procedure.

Step 5: [IN ACTIVE] REBALANCE DATA SET RECn, RENAME

This command will rebalance the trees of all the record-types stored in the named data set, storing them in a new ORVYL data set called "myfile.REBn".

RENAME, though really an option, is highly recommended -- if you omit it, you will have to issue other commands to complete the process, described in the next section. [See 3.3.1.] The remainder of this procedure assumes you have used the RENAME option.

The display from the REBALANCE command (which goes into your active file if you add the IN ACTIVE prefix) varies from file to file, but it should look something like this one for REC2 of GQ.DOC.ERRATA, which contains three record-types:

-> rebalance data set rec2, rename

    Rebalance Data Set REC2
-Base block     1 in record-type REC02
-Next block is    20
-Base block    21 in record-type REC03
-Next block is   138
-Base block   139 in record-type REC04
-Next block is   297
- REBALANCE complete - Start RENAME process

    FIX rebalanced data set REC2
-REC02  base is     1;   old base was     0
-REC03  base is    21;   old base was     1
-REC04  base is   139;   old base was     2
- Next block is   297; Next block was   346
ORV.GQ.DOC.ERRATA.REC2 is renamed to ORV.GQ.DOC.ERRATA.REO2.19861122
ORV.GQ.DOC.ERRATA.REB2 is renamed to ORV.GQ.DOC.ERRATA.REC2
-Set new data set permits if needed.
-File detached
->

SPIRES will also display a message each time 64K nodes have been processed, to reassure you (especially if the record-types are large) that the rebalancing activity is occurring.

Problems during the rebalancing

If the display ends with the "File detached" message, continue with step 6. If it did not, there may be several reasons why:

 - you received the message "New data set exists; OK to continue?"  because there is
 already a "filename.REBn" data set.   You  should  type  NO  and  return  to  the
 "Preliminaries" above.
 - the display stopped after the "FIX rebalanced data set"  message  with  an  error
 code  of  S56, because the file is being used by others.  You must have exclusive access to
 the file before the renaming part of the procedure can succeed.  You will need to  continue
 with  the  FIX  REBALANCED  DATA  SET  command  when  you  have  exclusive access.  [See 3.3.2.]
 - the display stopped after the "Next block is, Next block was" line with an  error
 code  of S592, because there is already a "filename.REOn.yyyymmdd" data set.  You
 should rename or erase that data set (You may have to SET CLP for it first; see  step  4.),
 and    then   continue   with   the   FIX   REBALANCED   DATA   SET   command.    [See   3.3.2.]
 - some other error has occurred.  Use the EXPLAIN command to get more information  about  the
 error.

Step 6: Check the status information of the rebalanced data set

Issue the ATTACH command again ("ATTACH filename" -- see step 2), followed by the STATUS command:

If this command completes successfully with no errors (it almost always does), then the rebalance procedure has been successful. But if error messages do appear, you can replace the new RECn data set with the old, which is still saved as "filename.REOn.yyyymmdd". The "recovery procedure" is discussed in the next section. [See 3.3.2.]

Step 7: SET JOBGEN

When you are sure the rebalance succeeded, issue the SET JOBGEN command to allow users to update records again. If necessary, select a subfile of the file again first.

Cleanup

To clean up after successful rebalancing, you should get rid of the old data set, which you no longer need. Issue the ORVYL commands "SET CLP filename.REOn.yyyymmdd" and "ERASE filename.REOn.yyyymmdd".

If there were special permits set for the data set (see "Preliminaries" above), use ORVYL's SET PERMIT command to reset them:

3.3.2  Handling Unusual Situations and Problems with Rebalancing

The standard rebalancing procedure described in the previous section does a great deal of work; if problems occur in the middle of the procedure, it could be quite expensive to restart at step 1. This section will discuss ways to continue the procedure in the event of problems.

Many parts of the discussions in this section assume you are already familiar with the procedure discussed in the previous section. [EXPLAIN REBALANCE PROCEDURE to see that section online.]

Handling Problems

The most common problems occur because either:

 - you don't have exclusive access to the file when renaming the "filename.RECn" and
 "filename.REBn" data sets; or
 - a data set named "filename.REOn.yyyymmdd" already exists (probably  left  from  a
 previous rebalancing that day) when the REBALANCE command is issued.

In both situations, "filename.REBn", the new data set with the rebalanced trees, has been created; SPIRES was unable to replace the old data set with the new one though.

Recovery from both these problems is relatively easy; both problems are handled similarly:

 - 1) If the problem is the leftover "filename.REOn.yyyymmdd" data set, get rid of
 it by either erasing it or renaming it:
 - First, SET CLP filename.REOn.yyyymmdd
 - Then, either ERASE filename.REOn.yyyymmdd
 - Or, RENAME filename.REOn.yyyymmdd TO new.dataset.name
 - 2) Issue the FIX REBALANCED DATA SET command to complete the rebalancing:
        FIX REBALANCED DATA SET RECn, RENAME

If you do not have exclusive access to the file, the command will fail. However, failure at this stage means simply issuing the FIX REBALANCED DATA SET command again when the file is available. [Remember not to wait too long, however. If the file is processed, or if other activity changes any trees in the rebalanced data set, you will have to start the entire rebalance procedure again.] Be sure to end the procedure as shown in the previous section, continuing with step 5. [See 3.3.1.]

In very rare situations, the REBALANCE or FIX REBALANCED DATA SET command completes with no sign of problems, but the STATUS command fails, indicating a problem with the rebalanced data set. To recover from this problem, you need to get back to the original data set of trees (now stored as "filename.REOn.yyyymmdd"):

 - 1) Issue these commands to switch back to the old data set:
     SET CLP filename.RECn
     RENAME filename.RECn TO filename.REBn
     RENAME filename.REOn.yyyymmdd TO filename.RECn
     SET PERMITS filename.RECn ....
          (set the same permits that were there before)
     SET NOCLP filename.RECn
 - 2) Reset the base blocks to their original  values.   To  do  this,  you  must  have  the
 information  from  the  SHOW FILE ACTIVITY command (issued before you rebalanced the data
 set) nearby, since you will need to type numbers from that display:
     ATTACH filename
     FIX BASE OF RECn

The FIX BASE command allows you to reset the base block numbers for each of the trees in the data set. You type the old numbers from the second column of the SHOW FILE ACTIVITY command as they are requested. This part of the session looks like this:

For example,

 - 3) Reset the "next block" figure for the data set, using  the  FIX  NEXT  BLOCK
 command.   First,  you  need  to  determine the old number by consulting the third column
 ("Blocks used") of the SHOW FILE ACTIVITY information  you  got  prior  to  the
 rebalancing.  The relevant portions of the display might look like this:

If REC2 is the data set you are working with, look for the number "2" in the DS column, and pick up the number next to it under "Blocks Used" -- in the example above, that's the number 928. So the "next block" value should be reset to 928. [In the example, REC2 contains 928 blocks, whose block numbers run from 0 to 927; hence, the "next block" is number 928.]

To do that, issue the ATTACH and FIX NEXT BLOCK commands:

     ATTACH filename
     FIX NEXT BLOCK OF RECn

The FIX NEXT BLOCK command resets the number SPIRES will assign to the next block added for the data set. You will be shown the current number and asked to supply a new one, like this:

The recovery process is now complete. However, you should attach the file to perform a STATUS command again. Should difficulties again show up, you should contact your SPIRES consultant immediately.

3.3.3  Rebalancing the Deferred Queue: The REBALANCE DEFQ Command

Occasionally in a very large file, the deferred queue can approach or even reach its limit of 64K blocks, or, if there are many transactions in a particular key area, the deferred queue's tree could become too deep. (The deferred queue tree can have 16 levels, the same as the normal record tree.) The purpose of REBALANCE DEFQ is to provide a way to get out of some conditions that could bind up a file if the defq gets too full or the tree gets too deep.

The REBALANCE DEFQ procedure will rebuild a Spires file DEFQ and rebalance the defq tree. As a result, the defq will generally be reduced in size because:

 - All transactions are packed together in the defq blocks so that there is very little  empty
 space in the defq.
 - The defq tree is compacted using the FASTBILD logic so that the tree has  about  90%  space
 utilization.
 - If several transactions have been processed for a single record, only the latest  is  kept.
 - Transaction group processing records are eliminated.  (This has  no  effect  on  previously
 completed  transaction  groups  except  that  SHOW SUBFILE TRANSACTIONS no longer shows the
 transactions in groups.)

Note: a special version of the REBALANCE DEFQ command (the TREE option) can be used solely to rebalance the defq's tree, in which case only the second point above is true.

Preliminaries

Check the DEFQ data set's ORVYL permit

The REBALANCE DEFQ command will usually create a data set with the default ORVYL permit setting of Public Write. That's the same setting SPIRES uses when it creates the DEFQ data set.

However, if you, the file owner, ever changed the permit, you will probably want to reinstate that level of security when the REBALANCE DEFQ procedure is over. To that end, issue the command:

If the data set has permits other than the appropriate default, jot them down so that you can reset them at the end of this procedure.

Get rid of any interim rebalancing data sets

You should not have any ORVYL data set called "myfile.DEBQ" or "myfile.DEOQ.yyyymmdd" ("yyyymmdd" being today's date) when you issue the REBALANCE DEFQ command, since SPIRES will create them. If either does exist, the procedure will stop. To see if you do, issue these ORVYL commands:

If you do have either of these, you should either erase or rename the data set, using the ORVYL commands "ERASE dataset.name" or "RENAME dataset.name TO new.dataset.name". (You will have to issue the ORVYL command "SET CLP dataset.name" prior to the ERASE or RENAME command if the data set was created by SPIRES.)

Try to get exclusive access to the file

WARNING: Since the basic REBALANCE DEFQ procedure replaces the DEFQ with a new copy, at times it needs exclusive access to the file. You might want to use the SET SUBFILE DISABLE FOR ACCESS and SET SUBFILE DISABLE FOR PROCESS commands to prevent any further activity while you are working on the deferred queue.

However, if getting exclusive access to the file is not possible, but you must rebalance the deferred queue because of a tree-depth problem, you can rebalance only the tree, which does not replace the entire deferred queue. See the TREE option on the REBALANCE DEFQ command. Note, however, that it is not generally recommended that you try any form of rebalancing when the file is being used by others.

Step 1: SPIRES

You must be in SPIRES to use the REBALANCE DEFQ command:

Step 2: ATTACH filename

Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:

This example, which will be followed through this section, demonstrates using the ATTACH command on the file GQ.DOC.ERRATA.

Step 3: REBALANCE DEFQ

There are several flavors to this command:

REBALANCE DEFQ by itself will create a new defq called "Filename.DEBQ" on your account. If you wish to replace the primary defq, you must do this manually.

REBALANCE DEFQ RENAME will not only create a new defq (called "filename.DEBQ"), but then it will replace the old DEFQ data set with the new one, saving the old one aside. (If you omit the RENAME option, you can follow this command with the FIX REBALANCED DATA SET DEFQ RENAME command to do the renaming in a separate step.)

REBALANCE DEFQ COPY is similar to RENAME, except that the new DEFQ data set replaces the old one, which is not saved.

REBALANCE DEFQ TREE will rebalance only the defq tree within the existing defq. The process is very fast and is used primarily to rebalance a tree that is too deep. Note that a STATUS DEFQ will give false errors following this process since both the old and new rebuilt trees are in the defq. Since this command does not replace the deferred queue but simply rewrites data into it, and only a relatively small part at that, it is quite quick. It does not generally change the size of the deferred queue but does solve the tree-depth problem and is the best choice if that is the only problem with the deferred queue.

The example continues:

Cleanup

To clean up after successful rebalancing, you should get rid of the old data set, which you no longer need. Issue the ORVYL commands "SET CLP filename.DEOQ.yyyymmdd" and "ERASE filename.DEOQ.yyyymmdd".

If there were special permits set for the data set (see "Preliminaries" above), use ORVYL's SET PERMIT command to reset them:

Turn file access back on with the SET SUBFILE ENABLE FOR ACCESS and SET SUBFILE ENABLE FOR PROCESS command, if you disabled the file with their SET SUBFILE DISABLE counterparts.

3.3.4  Rebalance Procedure, RESIZE

This process will be called RESIZEING the database. The steps from start to finish are outlined as follows: The 4K file account should have MASTER File-access permission or at least COPY access to the 2K database.

1. On the 4K account, build a clone File definition of the database to be RESIZEd. Be sure to add the statement FILE-BLOCK = 4K; at the end of the 4K file definition.

2. Logon to the 4K account. Select the 2K database and perform the following command for each RECn data set.

This rebalance option can be used to rebalance a tree data set in a 2K file system into a 4K file system using (and modifying) the MSTR data set of the 4K file system account.

3. While logged onto the 4K account, SELECT the 2K database and issue the command:

This command is a special form of the COPY DATA SET process that is designed to deal with the conversion from a 2K residual to a 4K residual. The 2K blocks will be read from the selected database, expanded and written to the 4K residual.

There are two options available in case the Residual is extremely large and you wish to convert in more "bytesize" chunks.

 LIMIT = block-count   -- copies the number of blocks specified.
 CONTINUE     -- continues the COPY on subsequent steps.

4. Logon to the 4K account, SELECT the 4K database and issue the following command for each RECn dataset that was rebalanced:

This will replace all the RECn datasets with clones of the 2K versions. You could do this before step 3 above.

5. Once steps 3 and 4 have been completed you may use the new database if you wish. However, there is another step which will reclaim the unused space and should be carried out as soon as possible. This process is a special version of AVSP RECovery and should be run until the whole file has been dealt with

Remember that you have the NORESERVE option that you may use if there is no possibility of Immediate Indexing taking place and you want users to be able to update while AVSPREC is occurring.

This process is a special form of the wonderfully named AVSPREC command. This command will read every block in a Residual data set that has been "RESIZEd", looking for records that are split to multiple blocks. These records are essentially updated in place in order to utilize as much available space as possible in the existing blocks.

Be sure to use the CONTINUE and possible LIMIT options the second, third and subsequent times through the process.

Note: This set of commands does not speak about DEFQ updates. Once the REBALANCEs and COPY DATA SET commands have begun you must ensure that no Immediate Indexing or PROCESSes take place on the 2K database. You may do this by DISABLE-ing UPDates. If you need to synchronize the DEFQs at the end of this procedure, you can SELECT the 2K database and issue the

    REBALANCE DATA SET DEFQ

command while logged onto the 4K account.

Then SELECT the 4K database and issue the command:

3.4  Ensuring Efficient Use of Storage Space in the Residual

This chapter discusses how SPIBILD stores records in the residual data set of a file, and how that method can cause available storage space in the residual to be "misplaced". By properly reading the AVSP (AVailable SPace) data from the STATUS command, you can tell how much space has been misplaced. Then, if appropriate, you can use the AVSPREC command to make it available to SPIBILD again.

Is the material in this chapter useful to you now? The best way to tell is to examine the "Space Utilized" value for the residual shown in the STATUS command's output. If its value is less than 90 percent and you haven't issued the AVSPREC command recently, it is quite likely that SPIBILD has misplaced some of the available space. Of course, the larger the file, the more space (and cost) that unused percentage represents -- you are being charged for the space, but SPIBILD isn't using it.

The sections of this chapter are:

3.4.1  (*) How SPIBILD Keeps Track of Available Storage Space

[Author's note: This is a technical discussion explaining the purpose and implementation of "available-space chains". The sections that follow describe how to determine whether they need repair and how to repair them -- you don't need to know the information here to handle those tasks. It's like changing the oil in a car -- you don't need to understand how the engine works to do it yourself, but you might want to know.]

When SPIBILD puts a new copy of a record into the residual data set, it tries to find an empty spot in the residual into which the record will fit. If the record is too big for the available space in any block of the residual, SPIRES will add another block to the residual and add the record to that.

Available-space chains

SPIBILD keeps track of available space in the residual with a scheme called "available-space chains". There is a chain of blocks for each discrete amount of available space (in 64-byte increments) in a block. For example, the five blocks each containing 2048 bytes of available space would form one chain. Another chain would link, say, the three blocks containing 1984 free bytes, etc.

How the chains are linked

Each block in the residual holds the location of another block in the residual containing the same amount of free space (except for the last block in a given chain). The location of the first block in a chain is kept in a table in one block of the residual, at a place representing the number of bytes of free space in each block of the chain. Thus the chain runs from the first block, whose location is kept in the table, to the second block, whose location is kept in the first block, etc.

How SPIBILD uses the chains

When SPIBILD needs to put a record away in the residual, it determines how many bytes of storage are needed and then goes to the chains. It always goes to the chain having the largest amount of space available (even if there is a chain representing the exact amount of space). SPIBILD retrieves the first block of the chain and places the record into it. It then reads the location of the second block of that chain and puts that into the table, thus making the second block the first.

The block into which the new record was placed now has less free space in it. SPIBILD determines how much space that is, examines the table to find the chain having that much space, places the location of its first block in the block with the new record, and then puts the location of that block into the table, thus making it the first record in the new chain.

Here is an example that might help make that more clear:

     Available Space Table
     ---------------------
    | Available   Block   |
    |   Space     Number  |
    | ---------   ------  |
    | 960 bytes     10    | --> block 17 --> block 39 --> block 93
    |    ...              |
    | 640 bytes     23    | --> block 41 --> block 86
    |    ...              |
     ---------------------

The table (inside the box) shows the first block number of two chains -- the first chain, starting with block 10, contains 960 free bytes in each block. To the right are shown the other members of the chain. Thus, blocks 10, 17, 39 and 93 are in the "960 bytes" chain.

Assuming there are no blocks containing more free space than 960 bytes, suppose that SPIBILD needs 320 bytes to store a new record. It will go to the table, retrieve block 10, and put the record into it. Since block 10 contains a pointer to block 17, SPIBILD will replace the number 10 with 17 in the table as the first block of the 960 chain.

Since block 10 now contains 640 free bytes, it needs to be placed on the 640 chain. SPIBILD places the number of the first block on that chain (23) into block 10, and then places the number 10 into the table as the first block on the 640 chain. At the end of this work, the diagram would look like this:

     Available Space Table
     ---------------------
    | Available   Block   |
    |   Space     Number  |
    | ---------   ------  |
    | 960 bytes     17    | --> block 39 --> block 93
    |    ...              |
    | 640 bytes     10    | --> block 23 --> block 41 --> block 86
    |    ...              |
     ---------------------

How blocks get on the wrong chain

This process works very smoothly for added records, but not so smoothly for updated and removed records. When SPIBILD is processing a record that's been updated or removed, it finds the block containing the old version of the record, which may be in the middle of an available-space chain. For example, suppose you have removed a record that resides in block 41, which is in the 640-byte chain shown above.

After removing the record and thus creating a larger amount of available space in block 41, SPIBILD puts the block back but does not move the block to a larger available-space chain, unless the block is the first one on a chain. (Similarly, if an updated record were larger but there was enough free space to put it back in the same block as the old version, SPIBILD would put back the block but would not connect it to a smaller available-space chain unless it heads a chain.) [Why doesn't SPIBILD usually move the block to its correct chain? The reason is that relinking the old chain would add considerable overhead to the process (at least one more block would need to be read and rewritten for each update).]

Why this can be a problem

The result is that some blocks on a chain may have more (or less) available space in them than their presence on that chain indicates. This is not a problem to SPIBILD -- if a block with the incorrect amount of space works its way to the head of a chain, then when SPIBILD tries to use the block and finds that it doesn't contain enough free space, SPIBILD will simply put the block on the correct chain and then get the next block.

In fact, what makes it a problem is that it can degrade the percentage of space used in the residual. Suppose SPIBILD removes a record from a block on the "64-byte" chain so that the block now contains 1000 free bytes. It could be many months or years (if ever) before that block is used -- remember, for added records, SPIBILD always uses the block with the most free space that has room for the entire record. (Chances are slim that SPIBILD would ever have only a 64-byte record to put away and simultaneously have only available-space chains for 64 bytes and fewer.) If other records in the block get updated and grow, the free space within it will get used, but otherwise, the space is wasted.

For many files, it is useful to rebuild the available-space chains occasionally, using the AVSPREC command. [See 3.4.3.] By putting blocks on the correct chain, SPIBILD insures the likelihood that the blocks with the most free space in them will be used to hold more data. [Documentation and the AVSPREC command "verb" itself talk about "recovering" the lost space, but there is no guarantee that the command will cause the space to be used; it simply rebuilds the chains, increasing the likelihood that blocks with lots of free space that were on wrong chains will be used again.]

The AVSPREC command can be quite expensive, since it requires SPIRES to read every block in the residual and write most of them. Before doing it, you should determine as best you can that the available-space chains need rebuilding and that the reclaimed space will be used. The data that will help you with this decision is provided by the STATUS command. [See 3.4.2.]

3.4.2  Determining Whether To Rebuild the Available-Space Chains

The previous section described how SPIBILD uses and maintains available-space chains, which keep track of blocks containing given amounts of free space. For example, among the many chains for a given residual data set, there may be a 320-byte chain consisting of blocks that have 320 bytes of available space. [See 3.4.1.]

As explained in that section, the chains may become inaccurate, containing blocks with more or less space than they are supposed to have. This can lead to free space that isn't used when it should be.

This becomes a problem only when a significant portion of subfile transactions are updates and removes; the higher the percentage of updated and removed records, the more likely (and the faster) the chains are to become inaccurate.

By keeping an eye on the percentage of space used and on the available-space table, you will know when to use the AVSPREC command to help increase the percentage of space used. You can get those statistics from the STATUS command, though you must include the AVSP option to get the available-space table. [See 3.2.2.]

Space Utilized

The Space Utilized figure given by the STATUS command for the residual data set is a percentage indicating how much of the residual is being used to store data. The difference between that and 100 percent is the amount of "available space", that is, space that can be used to hold data. [The SHOW FILE STATUS command also has the Space Utilized figure, but remember that it is only as current as the most recent STATUS command.]

It is exceedingly unlikely that the Space Utilized value will ever reach 100 percent for your file; for most well-tuned files, the value should be between 90 and 100 percent. If the value is lower than 90 percent, then there's a good chance that some of the free space is in blocks on the wrong available-space chain. (It's not necessarily the case because the number will be exactly the same after you rebuild the chains with the AVSPREC command; it may be awhile before the percentage begins to increase.)

Available-Space Tables

If the Space Utilized value is under 90 percent, you should examine the statistics about the available-space chains produced by the STATUS command when you include the AVSP option.

Here is a condensed example of those statistics, for a file that would benefit from the AVSPREC command:

                Available Space Chain Data

          Bytes      Blocks      Blocks    Blocks on
        Available   (Actual)    on Chain   Wrong Chain
          1936           1
          1896           1
          1880           1           1
          1768           2
          1720           2           3           3
           ...
           776           2           2
           768                       2           2
           760           1           1
           744           4           5           2
           ...
             0          84          83           9

Reading this chart correctly is admittedly a challenging task. [There are plans to revise it in a future version of SPIRES.]

The second column, "Blocks (Actual)", lists the number of blocks that actually have the amount of free space indicated in the first column. Hence, there is "actually" one block in the residual that contains 1936 bytes of free space.

If a chain is accurate, then the second column, "Blocks on Chain", will have the same number as the first column, and nothing will appear in the third. "Blocks on Chain" tells you how many blocks are on the chain for the given amount of space. Since there is no value given in that column for 1936, there is no chain for 1936 bytes. That means that the block with 1936 free bytes must be on the wrong chain.

The last column, "Blocks on Wrong Chain", is a subset of the the "Blocks on Chain" column. For 1720 bytes, the chart shows that 3 blocks are on the chain but all 3 are on the wrong chain -- that is, none of them contain exactly 1720 bytes of free space. That means that the 2 blocks that actually contain 1720 bytes of free space must be on other chains, where they don't belong.

By subtracting the number of "Blocks on Wrong Chain" from the number of "Blocks on Chain", you can determine how many blocks on a given chain belong there. So for the 744-byte chain, the number of blocks on the chain that actually have that much free space is 5-2, i.e., 3. If you then subtract that number from the number of "Blocks (Actual)", you'll know how many blocks with that much space are on wrong chains, which in this case is 4-3, or 1.

If you discover that many of the blocks containing large amounts of empty space (say, as large or larger than the average record size) are on the wrong chains, it is probably worthwhile to rebuild the chains, using the AVSPREC command. [See 3.4.3.]

3.4.3  Rebuilding Available-Space Chains

This procedure, which rebuilds the available-space chains for the residual data set, can be fairly expensive, since every block in the residual must be read and many blocks must be rewritten in order to correct the chains. You are urged to wait till your computing rates are the cheapest possible (e.g., during the night or on weekends).

To determine whether your file will benefit from rebuilding the chains, see the previous section. [See 3.4.2.]

1) SPIRES

You must be in SPIRES to use the AVSPREC command:

2) ATTACH filename

Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:

3) [IN ACTIVE [CLEAR|CONTINUE]] AVSPREC [CONTINUE]

The AVSPREC command will rebuild all the available-space chains of the residual data set. Because all of the blocks in the residual must be read (and most must be rewritten), it may take several minutes for SPIRES to complete the command.

The AVSPREC output

SPIRES then puts a list of all the chains and the number of blocks on each into your active file. The list looks like this:

This list will now match the first two columns of the AVSP display from the STATUS command -- the number of blocks actually containing a given amount of free space will match the number of blocks on that chain.

Handling Problems

The most likely problem you would have with the AVSPREC command is a system failure in the middle of it. Even that is not much of a problem -- simply restart the procedure at your earliest convenience, but add the CONTINUE option to the end of the command. See Part 6 for details, or online, [EXPLAIN AVSPREC COMMAND.]

4  Keeping Track of How Your File Is Being Used

4.1  Introduction

Most often, how a file is being used determines whether and how the file is changed. For instance, your analysis of how your users find records in a given subfile might uncover the fact that one of the indexes is scarcely used at all or that many searches need to look for records using an unindexed element. Knowing these facts, you might decide to eliminate the unnecessary index and create a new index for the other element; in the first case, you will save unnecessary storage costs for the index, and in the second, you will save your users from doing costly and inefficient searches.

This part of the manual discusses various techniques that will help you keep track of what users are doing with your file. The possibilities vary from collecting statistics on the number of records in each record-type to collecting all commands and responses that users type when they have your subfile selected.

Of course, don't forget that one of the best ways to find out how your file is being used is to talk to your users directly, to ask them what changes to the file would make it easier and better to use.

Contents of Part 4

Here are the subjects of the rest of Part 4:

4.2  Monitoring File Transactions

This chapter discusses ways to monitor the record transactions that users are making to your file. Here you are interested in the contents of the data as well as the transactions that create and change it. This contrasts with Part 3, much of which was concerned with the physical size of the data rather than its contents.

The contents of Chapter 4.2

There is a wide variety of commands and procedures to help you with this task. Those that are described in this chapter are:

4.2.1  Examining the Added, Updated and Removed Records

For various reasons, you may want to examine the records involved in subfile transactions before the file is processed, applying the transactions to the tree. For instance, for the sake of security, you might want to save a copy of all records being removed before they actually disappear from the subfile when the file is processed.

This section discusses in general the techniques SPIRES provides to give you access to the records involved in subfile transactions. Most of the details about the individual commands are covered in other manuals, such as "Sequential Processing in SPIRES: Global FOR" and "Searching and Updating".

You may want to include these techniques in a batch SPIRES or BATWYL job that precedes a job to process your file. [See 2.1.3.]

Added Records

Use the Global FOR class ADDS to retrieve records being added.

Example

For example:

Here copies of the 31 added records (as seen through a custom format named ADDS SUMMARY) have been put into the active file.

Remember that the ADDS class contains the records in the deferred queue whose keys don't match any records in the tree. That means, for example, that if a user adds a record and then another updates it before the file has been processed, it is still considered an ADD to SPIRES.

Updated Records

For updated records, there may be two different versions of the record that you're interested in: the version in the tree (the "old version") and the version in the deferred queue that's going to replace it.

To get the new version of each updated record in the deferred queue, use the FOR UPDATES command in SPIRES, just as you use the FOR ADDS command for added records. To get the old version of each, create a stack of the updated records and then use the VIA TREE prefix on the TYPE command.

Example

The example below demonstrates both of these techniques:

A slightly different method was used to get copies of the new versions of the updated records than was for the added records. A stack of keys of the updated records was created first. The first TYPE command put copies of the new versions of the updated records in the active file -- remember that the TYPE command by default retrieves the latest copy of a record. The second TYPE command, with VIA TREE, placed a copy of the old version of each record in the active file.

Keep in mind that the Global FOR class of UPDATES consists of records in the deferred queue that are to replace records currently existing in the tree.

Removed Records

When records are removed, SPIRES puts only the keys of the records in the deferred queue. To see the old version of the records being removed, you need to use the same technique just described for the old version of updated records: create a stack of the removed records, and then use the VIA TREE prefix on the TYPE command to see them.

Example

For instance:

Copies of the three records being removed are now in the active file.

The REMOVES class of records consists of records whose keys (as well as the rest of each record) are being removed from tree. Hence, if you remove a record and then add a new record using that key before the file has been processed, the record will appear in the class of UPDATES, since in sum, SPIBILD is going to replace the old version in the tree with a new one.

Other Global FOR Classes

Several other Global FOR classes may be useful to you as well:

For more information about Global FOR classes, see the manual "Sequential Processing in SPIRES: Global FOR". Further information about the VIA TREE prefix appears in the SPIRES manual "Searching and Updating".

4.2.2  Getting Basic Overall Statistics about Transactions

Several commands are available to you when you need basic statistics about a subfile's size and the transactions that got it there.

SHOW SUBFILE SIZE command

The basic command to discover how many records are in a subfile is the SHOW SUBFILE SIZE command:

Although called "SHOW SUBFILE SIZE", the command can be issued anytime you have a record-type of a file attached in SPIRES. It will tell you how many records there are in the attached record-type.

The same value is also available through the $SUBFSIZE system variable.

SHOW FILE STATUS command

You can get that information for all the record-types in the attached file using the SHOW FILE STATUS command, available to the file owner and users with SEE access to the file.

In fact, that information is there regardless of whether a STATUS command has ever been issued for the file:

-> show file status

    01/12/87       Status Record of File GQ.DOC.RESTAURANT
                   Last Compiled: 07/23/86 at 15:37:08

 Record     Blocks    Space     Number   Accesses   Tree     Date
  Type       Used    Utilized   Records  / Record   Depth

 ID    -Slot     4    60.944       407     2.000       0    12/09/86
 INDEXC          1     5.088         8     2.000       1    12/09/86
 INDEX2         11    72.624       611     1.897       2    12/09/86
 ...

Residual       112    83.600       481     0.000       0    12/09/86
Def Queue        2
->

SHOW FILE ACTIVITY command

A breakdown of the ADDS and REMOVES that create the "Number Records" figure above appears in the output from the SHOW FILE ACTIVITY command, available to the file owner and users with SEE access to the file.

-> show file activity

    01/12/87     Activity Record of File GQ.DOC.RESTAURANT
                   Last Compiled: 07/23/86 at 15:37:08

 Record      Base     Blocks     Adds    Deletes   Updates   Max Rec
  Type       Block   Used -DS                                 Length

 ID    -Slot     0      4 -1       617       210      1469      1347
 INDEXC          0     33 -2         8         0      2330      6129
 INDEX2          1   Comb -2       852       241       559       487
 ...
Residual         2       112       695       214      5264      6133
Def Queue        1         2
->

4.2.3  Getting Information about Specific Subfile Transactions

Every request to add, update, merge, remove, dequeue or unqueue a record in SPIRES creates a transaction record in the deferred queue of the affected file. The transaction record contains information such as the type of transaction, the date and time the transaction was completed and other information used by SPIRES and SPIBILD to process and track the transaction request. Some of this information is displayed by the SHOW SUBFILE TRANSACTIONS command.

File owners can use transaction records

File owners occasionally need to manipulate the information in these transaction records in various ways. For example, you might want to trace all updating activity done from a certain account or terminal, or want to sort the transaction records according to a particular element in the transaction records and generate reports. In other words, you may need to handle transaction records just as if they were records in any SPIRES subfile, using the full complement of SPIRES commands for processing them.

This chapter introduces a special record-type for "holding" transaction records. The transaction records in this record-type exist only until the file is processed in SPIBILD -- then they are discarded. Thus, they are not meant to be kept permanently but to represent the current transactions for the file.

Contents of Section 4.2.3

This section is organized as follows:

4.2.3.1  Getting Basic Transaction Information

SHOW SUBFILE TRANSACTIONS command

The easiest way to get basic information about the transactions in the deferred queue is with the SHOW SUBFILE TRANSACTIONS command. Available to the file owner and users with SEE access to the file, the command displays several pieces of transaction data for each record of the selected subfile in the deferred queue.

Example

Here is a sample of the output from the SHOW SUBFILE TRANSACTIONS command:

-> select filedef
-> show subfile transactions
  01/13/1990     Transaction Log of File GG.SPI.FILEDEF
                       For Record Type REC1

                 File last processed: 01/13/1990 at 02:37:38

Date      Time      Account   Id   Type  Command  Grp  Key Value

01/12/90  09:10:26  GQ.WCK         UPD   Update        GQ.WCK.IOTEST
01/12/90  09:15:42  GE.BXK   S17   UPD   Update        GE.BXK.VOLUNTEERS
01/12/90  09:31:30  CT.GIB   S17   ADD   Add           CT.GIB.VOLUNTEERS
01/12/90  09:35:48  AN.NSI   N33   UPD   Update        AN.NSI.DOC
01/12/90  09:37:04  AN.NSI   N33   UPD   Update        AN.NSI.DOC
01/12/90  10:00:50  GQ.WCK         DEQ   Dequeue       GQ.WCK.IOTEST
01/12/90  10:30:04  GA.SDS         UPD   Update        GA.SDS.SOFTDOC
01/12/90  11:54:02  GQ.PBT   U29   UPD   Update        GQ.PBT.TEXAS
01/12/90  12:06:33  AU.RAG         REM   Remove        AU.RAG.REPORT
01/12/90  12:11:24  AN.NSI   N33   UPD   Unqueue       AN.NSI.DOC
01/12/90  16:39:12  CN.MVS         UPD   Update        CN.MVS.BIO.SERIALS
01/12/90  16:55:38  GA.DRM   CU12  ADD   Add           GA.DRM.BIO.EXAMP
01/12/90  16:57:02  GA.DRM   CU12  ADD   Update        GA.DRM.BIO.EXAMP
01/12/90  17:01:12  GA.DRM   CU12  DEQ   Remove        GA.DRM.BIO.EXAMP
01/12/90  17:35:00  GQ.JNK         ADD   Add      1    GQ.JNK.TRANS.GROUP
01/12/90  17:37:00  GQ.JNK               Decommit 1
01/12/90  17:45:22  GQ.JNK         ADD   Add      2    GQ.JNK.TRANS.GROUP
01/12/90  17:45:32  GQ.JNK               Commit   2

This display shows the transactions of the FILEDEF subfile (the selected subfile) since the last time the file was processed in SPIBILD. Some of the columns are self-explanatory, though four may need a word of explanation here. All of them are explained in detail in Part 6; online, [EXPLAIN SHOW SUBFILE TRANSACTIONS COMMAND.]

ID

The "Id" column is the terminal ID if the user has one set.

Type

The "Type" column refers to the four basic types of transactions that can appear in the deferred queue for any given record:

 - ADD -- a record with a new key that does not exist in the tree;
 - UPD -- a record that is going to replace another with the same key in the tree;
 - REM -- a request to remove a record in the tree, without replacing it with  another  record
 with the same key; and
 - DEQ -- a request to ignore any earlier transactions for records with that key.

Command

The next column, "Command", names the command the user issued to cause the transaction to occur. "Addupdate" will not appear -- depending on whether the record was in reality added or updated, "Add" or "Update" will appear instead. It's important to understand the relationship between the Type and the Command. Keep in mind, for example, that a record that is added and then updated before the file is processed is still treated as an ADD since the updated version of the record will go into the tree as a new record: no record with that key exists in the tree.

Grp

The next column, "Grp", refers to the number of the transaction group if transaction group processing was in effect for that transaction.

Transaction data for one record

You can use the SHOW SUBFILE TRANSACTIONS command to find out about an individual record by adding its key value to the end of the command. It is also useful in conjunction with the Global FOR command FOR TRANSACTIONS, since it can help you identify a particular copy of a record you're trying to retrieve from the deferred queue.

If the SHOW SUBFILE TRANSACTIONS does not meet all your needs, you will probably need to use the $TRANSACTIONS record-type processing described in the next sections. [See 4.2.3.2.]

4.2.3.2  Creating the $TRANSACTIONS Record-Type

To use all the transactions data kept in the deferred queue by SPIRES, you must make some simple changes to your file definition. The basic procedure, which must be done by the file owner, is this:

$TRANSACTIONS record-type incurs no overhead

The transaction data exists in every file, regardless of whether it has a $TRANSACTIONS record-type in it or not. What the $TRANSACTIONS record-type does is reformat the data for use as if it were another record-type of the file, as described above. No new records or additional storage overhead are created by the $TRANSACTIONS record-type -- the data is already kept in the deferred queue until overnight processing anyway.

Step 1: Add the $TRANSACTIONS record-type to the file definition.

To treat transaction data as if it were its own record-type in a file, you must code a special record-type in the file definition. It must be done to the file definition directly; File Definer does not support the needed changes.

Get the file definition from the FILEDEF subfile, and look for the end of the last record-type defined, just before the first linkage section. After the last record-type, add the new record definition, which should look like this:

where "name" is a one- to five-character name, such as REC08, preceded by a period. The period declares the record-type to be a DEFQ-only record-type, a record-type whose data is kept only in the deferred queue and discarded when the file is processed. See the manual "SPIRES File Definition" for more information, or online, [EXPLAIN DEFQ-ONLY RECORD-TYPES.]

Because SPIRES arranges the record-types in alphanumeric order, the transactions record-type will always be the last (or one of the last, along with other DEFQ-only record-types) in the file.

Step 2: Add a Subfile section to the file definition.

Add a Subfile section for the new record-type as well. It names the new record-type as the goal record-type:

This allows you to select a subfile of the file to use the transaction data. (Alternatively, you can attach the transactions record-type directly to use the transactions data.) Be aware that access to this subfile means access to keys of all records in the deferred queue for all record-types in the file (though not to the records themselves).

Transaction records are discarded when the file is processed

The $TRANSACTIONS record-type may not be named in the linkage section either to pass values to other record-types or to receive values. Because it is a DEFQ-only record-type, its records are only temporary -- they are discarded during SPIBILD processing of the file.

Once you have made these changes, select the FILEDEF subfile in SPIRES and replace the old version with the new.

Step 3: Recompile the file definition.

To recompile the file definition, issue the RECOMPILE command when you have the FILEDEF subfile selected:

For example,

If the file definition does not compile, use the error messages to determine the problem; then correct the file definition, and try the procedure again.

The $TRANSACTIONS record-type is available as soon as the file is compiled.

The Contents of the $TRANSACTIONS Record-Type

Each record in the $TRANSACTIONS record-type represents a single transaction against a record-type in the file. All updating transactions, regardless of record-type, will have a record in the $TRANSACTIONS record-type.

The $TRANSACTIONS record definition has the following elements:

TREC, TDATE, TTIME and TKEY together form the structured key of the $TRANSACTIONS record-type.

4.2.3.3  Using the $TRANSACTIONS Record-Type

By adding the $TRANSACTIONS record-type to your file, you gain the ability to treat the transaction data almost as if it were any other record-type in the file. For example, as covered in the previous section, you can declare the $TRANSACTIONS record-type to be the goal record-type of a subfile. [See 4.2.3.2.]

Restrictions

There are restrictions on the use of the record-type, however, which are based on these facts:

 - No data may be passed to or from  the  $TRANSACTIONS  record-type  by  SPIBILD  or  through
 immediate  indexing.  The records thus cannot be indexed, nor can they be retrieved through
 index-searching commands such as FIND.
 - The  records  in  the  $TRANSACTIONS  record-type  don't  exist  in   "tree"   or
 "defq"  versions  but as "transactions".  That means that they can only
 be processed in Global FOR, under the FOR TRANSACTIONS command.

More versatile than SHOW SUBFILE TRANSACTIONS

The $TRANSACTIONS record-type is commonly used as a more versatile means for handling transaction data than the SHOW SUBFILE TRANSACTIONS command. For example, you can write custom formats or use system formats such as $REPORT to display the data in a fashion more useful to you. In the sample session below, the $REPORT format is used to display transactions for the first record-type in the file (TREC = 1):

Sequencing transaction data

To sequence the transactions data, you need to create a stack first, with the STACK command under FOR TRANSACTIONS. You may use the SEQUENCE, TYPE or ALSO commands to process the stack. You may not, however, process it under FOR STACK. For example,

(*) Retrieving the Transaction Records from the Goal Records

You may want to examine the actual records that were added, updated, etc., creating the transaction records. You can do this in either direction: from the goal records, you can access the transaction records; and from the transaction records, you can access the goal records. Let's start with retrieving transaction records from goal records.

You can retrieve a transaction record from a goal record only when you are processing goal records under a FOR TRANSACTIONS command. Using FOR TRANSACTIONS, often with a VIA clause, you connect to the transactions data in one of four ways:

 - using a phantom structure defined in the  goal  record  definition,  which  points  to  the
 $TRANSACTIONS record-type
 - defining a dynamic phantom structure which does the same thing, but which  can  be  created
 whenever the user wants;
 - using subgoal processing in a format, invoked through the SET SUBGOAL  TRANSACTIONS  Uproc;
 - retrieving element  values  from  the  $TRANSACTIONS  record-type  with  the  $LOOKSUBG  or
 $LOOKSUBF function, called from within a format.

This is useful when you need to carefully monitor changes made to records, particularly when they can change several times a day. Under FOR TRANSACTIONS, you can see each copy of the record as it was added to the deferred queue. With FOR TRANSACTIONS VIA DEFQ (or with one of the other VIA clauses) you process the records in key order, so all copies of the same record are processed one after the other, beginning with the original tree copy if the record starts out there (e.g., for an update).

With the tie to $TRANSACTIONS, you can connect additional information about the transaction to the goal record for producing reports. Consider this example, where the connection to the $TRANSACTIONS record-type is made with a dynamic phantom structure:

Notice in the last line that a removed record doesn't have any data other than the key in the deferred queue (aside from the data about the transaction from $TRANSACTIONS), so the ALBUM.TITLE field isn't available there.

You could add a WHERE clause to the FOR TRANSACTIONS VIA DEFQ command that would examine data values in $TRANSACTIONS, as long as the phantom structure is defined. For instance, if you use the TCOMMENT element [See 4.2.3.2.] to save information about the specific change made by an Update transaction, you could establish a Global FOR class like this:

That would let you find the different defq versions of a record whose "price" had changed when updates were made.

For more information about the FOR TRANSACTIONS command, see the SPIRES manual "Global FOR"; online, EXPLAIN FOR TRANSACTIONS. For information about defining phantom structures in a file definition, see the SPIRES manual "Technical Notes", section 23.1; online, EXPLAIN PHANTOM STRUCTURES, CODING. Chapter 23 of "Technical Notes" also has information about defining dynamic phantom structures, as we did in the example above; online, EXPLAIN PHANTOM STRUCTURES, DYNAMIC. For information about using the SET SUBGOAL TRANSACTIONS Uproc, see the manual "SPIRES Formats"; online, EXPLAIN SET SUBGOAL TRANSACTIONS. And for information about the $LOOKSUBF and $LOOKSUBG functions, see the manual "SPIRES Protocols"; online, EXPLAIN $LOOKSUBF FUNCTION or $LOOKSUBG FUNCTION.

(*) Retrieving the Goal Records from the Transaction Records

An alternative, which is perhaps more complicated to understand, is to retrieve the goal records from the transaction records. Depending on the situation, you can use the element TLCTR or TKEY as the mechanism to access (via subgoal processing) the goal records in the other record-types of the file.

Another method uses FOR PATH processing with either TKEY or TLCTR to retrieve those goal records. Currently, phantom structures are not available in the $TRANSACTIONS record-type, though they are likely to be a future enhancement.

Subgoal processing is discussed in detail in the manual "SPIRES Formats", section B.12. FOR PATH processing is discussed in the manual "Global FOR", sections 2.15 and 3.8. An example below demonstrates the use of FOR PATH processing.

You can retrieve all deferred queue copies of a record

If several different versions of a goal record exist in the deferred queue because it has been updated multiple times today, you can retrieve each copy or the tree version or the latest version -- that is, you can connect a $TRANSACTIONS record with its specific goal-record counterpart in the deferred queue.

Retrieving the record for this transaction

This example shows how FOR PATH processing could be used to see each copy of records in the deferred queue in the order in which they were created. The subfile XYZ contains the goal records, while the subfile XYZ TRANS contains the transaction records:

SPIRES uses the element TDLCTR in the XYZ TRANS records having a TREC value of 1 as a pointer for retrieving the deferred queue copy of records in XYZ. In other words, SPIRES will retrieve the version of the XYZ record that went into the deferred queue when the transaction occurred, which is not necessarily the latest version of the record. If you are interested in the latest version, replace TDLCTR with TLCTR or TKEY; if you want the tree version, replace FOR PATH with FOR TREE VIA PATH.

Retrieving the previous copy for this transaction

In many cases, you may also retrieve the previous deferred queue copy of a record, replacing TDLCTR with TPLCTR in the example above. However, if the value of TPLCTR in a $TRANSACTIONS record is zero (hex 00000000), then that transaction is the first in the deferred queue for the goal record. Since there is no previous deferred queue copy of the goal record, SPIRES would skip this $TRANSACTIONS record in processing.

Suppose, for instance that record A from the tree has been updated three times today:

The SHOW KEY command sent SPIRES through the path to find records in the $TRANSACTIONS record type having a TKEY value of "A". The first record it found, the first updated version in the deferred queue, was "transmitted" through the path to the FOR PATH processing. However, because it has a zero value for TPLCTR, it was considered an "examined record" but was not processed. The next record found in $TRANSACTIONS with a TKEY value of "A" did indeed have a non-zero value for TPLCTR -- it is, in fact, a pointer to that first deferred queue record -- and so it is processed.

4.3  Logging Subfile Use

As a file owner, you can ask SPIRES to log certain information about the access and use of the file's subfiles. You can choose to log only the information that a user has selected and "cleared" a subfile, or you can request that all the user's commands (and even responses to all ASK prompts) be logged too. In addition, with the WITHIN LOG prefix, you can have an application write anything to the file log you want.

One specialized use of the file log is to support "subfile charging", that is, charging people for their use of a particular subfile. SPIRES will inform them of the rates when they select the subfile and then log the charges as they are incurred. The file owner would be responsible for processing, billing and collecting any fees for subfile use.

The contents of Chapter 4.3

This chapter will discuss all aspects of SPIRES's logging facility, divvied up as follows:

4.3.1  Establishing a File Log

To create a file log, you add one or more statements to your file definition: LOG, HOURLY-CHARGE, DISPLAY-CHARGE or MINIMUM-CHARGE. You need at least one of these statements in each Subfile section for which you want logging. When the file definition is returned to the FILEDEF subfile, the statements take effect immediately for users who subsequently select the subfile. SPIRES will begin collecting the log data in the file's deferred queue.

An optional statement, STATISTICS, may also be included, which tells SPIBILD what to do with the log when the file is processed. The STATISTICS statement appears at the beginning of a file definition, which must be recompiled for the statement to take effect.

This section describes how to add the LOG and STATISTICS statements to your file definition. The subsection that follows describes the three statements you can add to implement subfile charging: HOURLY-CHARGE, DISPLAY-CHARGE and MINIMUM-CHARGE. [See 4.3.1.1.]

You can add any of the statements by using File Definer or by changing the file definition directly. Use File Definer only if you still have the File Definer input for the current version of the file definition.

Either get your file definition from the FILEDEF subfile (or wherever you keep the copy you change), or get your File Definer input.

The LOG Statement

The LOG statement in the Subfile section tells SPIRES what to log for the users named in the ACCOUNTS statement of the section when they select the subfile. In the file definition, its syntax is:

In File Definer input, it's added as an attribute to the SUBFILE keyword:

In each case, "n" is 1, 2 or 3, as determined from these choices:

They may be requested in any combination.

Prism and other full-screen applications

User commands generated through full-screen interaction aren't logged under any LOG value. Prism has its own method of logging those commands, described in the "Prism Applications" manual. If you are using your own full-screen application environment, you can add the WITHIN LOG prefix to the user command, which writes the command to the log with any LOG level set.

Folio applications

Use LOG = 1, 2, 3 in order to cause commands in Folio applications to be sent to your file log.

Example

The LOG statement belongs to the Subfile section, at some point after the ACCOUNTS statement. For example,

For File Definer, you would then generate a new file definition, and put it into the FILEDEF subfile. (The GENERATE procedure is summarized at the end of this section.) If you are changing the file definition directly, you just put it into the FILEDEF subfile.

Recompiling the file definition for LOG is not necessary

When you put the file definition into the FILEDEF subfile, the LOG statement will have an immediate effect: logging will begin the next time a user affected by the change selects the subfile. The log goes into the deferred queue, where it is accessible via the SHOW FILE LOG command. [See 4.3.3.]

The STATISTICS Statement

By default, SPIBILD discards the log when it processes the file. By adding the STATISTICS statement to the file definition, you tell SPIBILD to display the log and/or to add the log to an archive data set when it processes the file. In the file definition, its syntax is:

In File Definer input, it's added as an attribute to the FILE keyword:

In both cases, "n" is 1 or 2, chosen by the following criteria:

You may request both displaying and archiving if you code both 1 and 2.

Here is a sample Subfile section from File Definer input that uses the STATISTICS option:

After changing your input, use File Definer to generate a revised file definition:

The full procedure, and more information about how to make changes to files using File Definer, appear in the "File Definer" manual.

If you're changing the file definition directly, put the STATISTICS statement near the top, after the ID statement:

Recompiling the file definition for STATISTICS is necessary

Unlike the LOG statement, changes to the STATISTICS statement require recompiling the file definition. After you change the file definition and put it into the FILEDEF subfile, issue the RECOMPILE command:

If the file definition doesn't compile, use the error messages to determine the problem; then correct the file definition, and try the procedure again.

Once the STATISTICS statement is compiled, it will be in effect the next time the file is processed in SPIBILD.

4.3.1.1  How to Create a Charging Subfile

[Note: Check with your computer center to be sure that you can legally charge money for the use of your data before you use the facility described here.]

If you intend to charge users for their use of your subfile, you need to log their use of the subfile. In addition, you probably want SPIRES to tell users what the rates are when they select the subfile and to compute the amount you should charge them, reporting that to them when they leave the subfile.

Change the Subfile section of the file definition

To tell SPIRES about the rates, you add any or all of three statements to the appropriate Subfile section of your file definition: HOURLY-CHARGE, DISPLAY-CHARGE and MINIMUM-CHARGE. Any of these statements causes logging to begin, equivalent to LOG = 1, described in the previous section on the LOG statement. [See 4.3.1.] [If you need to log more information for charging purposes than what's shown here, you might want to add a LOG statement with a higher value than 1.]

You must change the file definition directly -- File Definer does not support files with subfile charging. Follow the same directions you would use to add the LOG statement, described in the previous section.

The HOURLY-CHARGE Statement

Use the HOURLY-CHARGE statement to specify the charge for each hour of elapsed time (or fraction thereof) the user has the subfile selected, as a dollar-and-cents amount:

SPIRES prorates the charge to the user based on the actual elapsed time.

The DISPLAY-CHARGE Statement

Use the DISPLAY-CHARGE statement to specify the charge for displaying 1000 records in the subfile (i.e., each record processed by a TYPE, OUTPUT, DISPLAY, TRANSFER or REFERENCE command), as a dollar-and-cents amount:

SPIRES prorates the charge to the user based on the actual number of records displayed.

The MINIMUM-CHARGE Statement

Use the MINIMUM-CHARGE statement to specify a minimum charge to the user when the sum of the "hourly charge" and the "display charge" is less than that amount, as a dollar-and-cents value:

Place any or all of these statements after the ACCOUNTS statement in the appropriate Subfile section of the file definition. For example, here is part of a file definition for a matchmaking service:

Recompiling is unnecessary

Remember that statements in the Subfile section take effect as soon as you put the file definition into the FILEDEF subfile; you do not have to recompile the file definition. But remember too that unless you include a STATISTICS statement (which means the file definition must be recompiled), the log statistics will be discarded when the file is processed.

A user who selects the BLIND DATA subfile would get the following message from SPIRES:

-> select blind data
-Database Charge: $30.00/hour, $10.00/1000 displays, $5.00 minimum
-Continue select? yes
->

Notice that SPIRES asks the user whether he or she wants to continue the selection. For charging subfiles, this occurs if the user's charge message level is set to 2 or 3. [EXPLAIN SET MESSAGES COMMAND online, or see SET MESSAGES in Part 6.]

Twenty minutes later, after examining 500 records in the subfile, the user might see this when leaving:

-> clear select
-Approximate database charge: $15.00
->

The user has been charged $10.00 for having the subfile selected for 20 minutes, and $5.00 for displaying 500 records. The log contains the amounts to be charged to the account, as well as a count of the number of records examined. [See 4.3.4.]

Be aware that the messages shown here appear when the "charging" message level is 2; they may disappear if the level is set differently with the SET [CHARGE] MESSAGES command.

4.3.2  How File Log Entries are Created

If a subfile has any logging set for it, then log entries will be written if the subfile is selected in SPIRES, Folio or Prism. (Because Prism is a full-screen system that prompts users with the ASK AREA command, only selects and de-selects get logged; Prism provides another method for logging user commands.) N.B.: Logging is associated only with subfiles; if any record-types are accessed via the ATTACH command by users with SEE access to the file, their use of the file is not logged.

Log gets written when a user selects the subfile

When a user selects a subfile with logging turned on, SPIRES begins collecting the log data. SPIRES immediately writes a log entry to the deferred queue about the select, including the date, time, user's account number, selected subfile and charging rates (if applicable).

If the subfile logs more than just selects and de-selects (i.e., more than LOG = 1), then SPIRES creates log entries in memory as the user works with the subfile. The log entries may include commands issued, ASK commands and ASK Uprocs from formats and the user's responses, and any commands preceded by the WITHIN LOG prefix. [The WITHIN LOG prefix may be added to any SPIRES command to cause it to be written to the log, regardless of the log-level setting. By preceding it with a "/", you tell SPIRES to replace any variables in the command with their values, allowing you to write current variable values to the log if you wish. The WITHIN LOG prefix is frequently added to the "-" comment command when the application designer simply wants to write some information to the log. The prefix is described in more detail in the manual "SPIRES Protocols"; online, [EXPLAIN WITHIN LOG PREFIX.]]

Except for the initial select entry, log entries are not immediately written to the deferred queue, however; SPIRES saves them in memory, sending them to the deferred queue when:

 - the subfile is de-selected; or
 - the WRITE FILE LOG command is issued; or
 - a block's worth of log  data  (about  2000  bytes)  has  been  accumulated  for  the  user.

Once SPIRES has added them to the deferred queue, it begins collecting the next entries in memory. Hence, a user's log entries for a given session may be scattered through the log in clumps; this is most likely when a subfile is used by several people at once and each session includes many commands.

System crashes

One other aspect about the delay in writing log data to the deferred queue is important to know about. If the system crashes during the user's session, the log data in memory will be lost. Similarly, if the user leaves SPIRES in some unorthodox way (say, via a session break), the log data in memory will be lost. To minimize the chances of losing the log data, the WRITE FILE LOG command can be issued, which writes the user's current log entries to the deferred queue immediately. This command often appears in protocols following commands that must be logged. See the manual "SPIRES Protocols" for more information about the command; online, [EXPLAIN WRITE FILE LOG COMMAND.]

SPIBILD discards the log

When the file is processed, SPIBILD discards the log. Depending on its disposition as specified by the optional STATISTICS statement, SPIBILD may display the log near the end of the processing and/or may add the log to the end of your ORVYL data set SYSTEM.ULOG before throwing it away.

Unless you issue the SET SEARCH or SET SHARE command in your SPIBILD session, you can't process the file while users have any of its subfiles selected. However, if you SET SHARE, you can process the file, and the user's log entries at the time will get written to the file's deferred queue after the processing. (That means that a single user session may span logs.) On the other hand, if you SET SEARCH, you will be able to process the file, but there is a possibility that log entries will be lost if the user de-selects the subfile during the processing. [See 2.1.5.1.]

4.3.3  Handling the File Log

This section describes the commands you can use to work with the file log data. [See 4.3.4 to learn how to read the coded data of the log.]

These commands show you log data:

 - SHOW FILE LOG
 - PROCESS, ESTABLISH, etc.
 - SHOW ARCHIVE

To discard log data, use these commands:

 - PROCESS, ESTABLISH, etc.
 - ZAP DEFQ
 - SHOW ARCHIVE

Each command is explained in detail in the reference section at the end of this manual. Below is a brief description of each:

[IN ACTIVE [CLEAR|CONTINUE]] SHOW FILE LOG

Use the SHOW FILE LOG command in SPIRES to see the current contents of the log in the deferred queue. Though the word FILE appears in the command, it may be replaced by SUBFILE -- but in fact, the entire file log is displayed, not just the log entries for the selected subfile.

PROCESS, ESTABLISH, INPUT BATCH, MERGE, etc. in SPIBILD

The SPIBILD commands that process the file will discard the current contents of the log in the deferred queue. However, if STATISTICS = 1 is coded in the file definition, the log will be displayed at the terminal (or in the job output, in the case of batch processing jobs) before it is discarded. [See 4.3.1.]

If you are processing the file and SPIBILD begins displaying the log, you can press the ATTN/BREAK key to stop it. However, the log is then discarded immediately -- you cannot see or use those log entries again unless they have been archived (STATISTICS = 2).

[IN ACTIVE [CLEAR|CONTINUE]] SHOW ARCHIVE [RELEASE]

Use the SHOW ARCHIVE command in SPIRES to see the log entries that have been moved to your SYSTEM.ULOG data set by SPIBILD. The single data set serves as respository for all archived log entries for all your files, so if you have several subfiles from several files, you may have to be careful that you are working with the log data you want.

If you include the RELEASE option, SPIRES will erase the current contents of the SYSTEM.ULOG data set after displaying them.

ZAP DEFQ

The ZAP DEFQ command erases all contents of the deferred queue of the file containing the selected subfile, including the log entries in the deferred queue. Only the file owner and users with Master access to the file can issue the ZAP DEFQ command.

4.3.4  Reading the File Log

Most of the file log's contents are self-explanatory, even without column headings. Both the file log and the file owner's archive have the same format. Below is a sample of a log, followed by a description of each. column and of the codes in the first column (character positions 2 and 3).

A Sample File Log

C8   15:33:52  01/09/87  GQ.JNK       $0.00       $0.00       $0.00  ALMANAC
C9   15:34:04  01/09/87  GQ.JNK  show indexes
C9   15:34:13  01/09/87  GQ.JNK  show autogen
C9   15:34:43  01/09/87  GQ.JNK  exit
E8   15:34:43  01/09/87  GQ.JNK             0  00:00:51
C8   10:42:32  01/26/87  GQ.VVN       $6.00      $10.00       $1.00  DATES
C9   10:42:39  01/26/87  GQ.VVN  sho sub siz
C9   10:42:56  01/26/87  GQ.VVN  for subfile
C9   10:43:00  01/26/87  GQ.VVN  sho keys all
C9   10:43:15  01/26/87  GQ.VVN  endfor
C9   10:43:18  01/26/87  GQ.VVN  find date 7/27
C9   10:43:20  01/26/87  GQ.VVN  type
C9   10:43:33  01/26/87  GQ.VVN  sel restaurants
E8   10:43:33  01/26/87  GQ.VVN            12  00:01:01
E9   10:43:33  01/26/87  GQ.VVN       $1.00
C8   14:57:39  01/26/87  GQ.BEC       $6.00      $10.00       $1.00  DATES
C9   15:25:34  01/26/87  GQ.BEC  FIND DAY 11/10
C9   15:35:28  01/26/87  GQ.BEC  SHOW INDEXES
C9   15:35:36  01/26/87  GQ.BEC  FIND DATE 11/10
C9   15:41:43  01/26/87  GQ.BEC  CLR SEL
E8   15:41:44  01/26/87  GQ.BEC             0  00:44:05
EA   15:41:44  01/26/87  GQ.BEC       $4.40

The example shows some excerpts from a log for two different subfiles: ALMANAC, which has no charging associated with it, and DATES, which does. You can see the date and time when a specified account issued the commands shown. Commands are listed in the case in which the user typed them.

Contents of the Columns

All records have the same format for the first 34 columns:

Type Codes

The remaining columns vary depending on the code given in columns 2 and 3:

     35/46  - the count of the number of records displayed
     49/56  - the time elapsed since the subfile was selected
               (HH:MM:SS form)
     58/65  - the time the subfile was originally selected
               (same value as the C8 record)
     67 on  - the name of the selected subfile, if it was selected
               through a subfile path

5  Miscellaneous File Management Tasks

5.1  Introduction

Some file management tasks cannot be easily categorized -- and sometimes pat procedures to handle the tasks don't exist either. This part of the manual covers a variety of problems you may need to address.

Controlling your costs

The most general task, and yet one of the most important, is controlling the costs of your SPIRES files. Throughout the manual, cost considerations have played a major role in most of the discussions. However, chapter 5.2 consolidates many of the cost-saving ideas presented earlier, which may lead you to alternative procedures you might not have considered. [See 5.2.]

Resetting the slot number

Other procedures described in Part 5 are not so critical but may solve small problems that face you. For instance, one chapter describes how to reset the "next slot number" that SPIRES assigns to slot records, when you want to skip some numbers or when you want to set the number back. [See 5.3.]

5.2  Saving Money on Existing SPIRES Files

This chapter presents some ideas about how you can save money on SPIRES files that you manage. This is very important for managers of large files, but some of the ideas presented here may be useful to owners of small, personal files too -- after all, no one wants to spend more money on data bases than necessary.

These are primarily ideas about files that already exist. There are many aspects of savings that the file designer should consider before creating a file, but that is a subject for another manual, "SPIRES File Definition". On the other hand, there are some techniques discussed here that may correct problems created by unfortunate file design.

File-owner savings vs. user savings

Keep in mind that savings offered to the file owner may be at the expense of the file's users, and vice versa. For example, the file owner may save storage costs by eliminating all indexes, but the users will have to spend much more if they must use Global FOR instead of indexes for searching. Such tradeoffs will be discussed when appropriate.

Rules and exceptions

The money-saving ideas are presented here as rules -- guidelines that file managers should follow if they want costs to be as low as possible. Naturally, all these rules have exceptions; situations arise where it is actually cheaper to break the rule than to follow it, or where you have no choice but to break the rule in order to get your work done. So some attention will be paid here to known exceptions to these rules.

Still, don't be misled by exceptions. A study of documentation for one system found that 90 percent of the documentation discussed 10 percent of the situations users actually found themselves in. Chances are that your situation is not the exception -- if you think it is, it might be wise to discuss it with your SPIRES consultant before breaking the rules precipitously.

Think 3 times, measure twice, cut once

In fact, thinking over what you are doing before you do it is always a good rule for saving money. As one SPIRES consultant said, "Think three times, measure twice, and cut once." This rule isn't always followed, and to some extent, SPIRES can share the blame for that.

Because applications in SPIRES are often developed iteratively and because SPIRES offers many tools that let you quickly try different ways to solve problems, it is easy to get in the habit of implementing an approach before you think it through. Naturally this approach has its benefits: you solve problems quickly, and you learn a lot as you work through the problems; but you may use a lot of expensive resources as you work. Clearly, you should try to strike a balance between implementing an idea swiftly (which might save time) and thinking the problem through (which might save money).

Experiment with smaller versions of the file

Of course, experimentation may be the only way for you to determine what approach will work best for a given situation. A good rule to follow here, particularly if you are working with a large file, is to experiment with a much smaller version of the file: perhaps a file with 1 percent of the records, or one with 10 percent of the records, from a random sampling if possible.

Use SHOW CLOCKS and SHOW COST commands

As you experiment, measure the resources you use and the costs you incur by issuing the SHOW CLOCKS and/or SHOW COST commands before and after issuing the controversial commands. Not only will they help you judge the various options you are considering, but they will also give you a sense of how much SPIRES components cost to use in general, which may be useful to you on a daily basis as you use your file.

1) Create and maintain a SPIRES file only if you need it.

If the benefits and convenience of SPIRES (including Prism and Folio) don't outweigh the costs, don't keep the data in SPIRES. Many small data bases may be more inexpensively maintained on microcomputers than in SPIRES. On the other hand, the strengths of SPIRES may make its costs seem less important; some strongpoints to consider are:

 - its ability to handle large records and large files;
 - its ability to share data between many users simultaneously;
 - its flexibility in formatting data for online display and printed reports;
 - its support of Prism and Folio, which  provide  easy-to-use  interfaces  to  SPIRES  files;
 - its flexibility in handling data security at all levels;
 - its strong searching support;
 - its variety of tools for application development;
 - your computer center's automatic backup  of  SPIRES  files  in  case  of  system  problems.

If none of these matter to you, chances are very good that you are wasting money keeping your files in SPIRES.

Is the data worth keeping on a computer?

Of course, a more critical question must be faced too: Do you need to keep the data in a data base in any computer program? Name-and-address files, for example, make good examples in SPIRES documentation because they are easy to understand, but do they make good files in any computer program? Not always, unless the benefits of putting them on a computer outweigh the costs. Those benefits include:

 - the ability to standardize the data collected;
 - the ability to sort the data in several ways;
 - the ability to find specific records quickly, searching by various criteria;
 - the ability to print updated lists and reports as needed;
 - the ability to analyze the data statistically.

If all you need to do is find someone's phone number occasionally, it's unlikely that the time spent putting the file together plus the expense of storing it would justify its storage on any computer.

Evaluate the file's value regularly

These considerations are important not just when you create a file but throughout its life. A file's usefulness may change over time. For instance, if you find, several years after its development, that your file is not being used much any more, consider closing it down. It might be less expensive to create a new report, use it to run a final copy of the data and then destroy the file than to keep the file online with no one using it.

2) Create and keep indexes only if you will use them.

This guideline, the first of two corollaries to the rule number 1, suggests you verify that all indexes you are maintaining are getting regular use. Indexes are overhead -- they duplicate data already stored in the file, they require additional storage space, and they require computer time to build and maintain. If they are rarely used, they may be costing you more money to maintain than they are saving users to search with them. And they may even be inefficient to use too.

For example, suppose you own an inventory file for a bookstore. You have an index called BINDING whose common values are HARDBACK and PAPERBACK. It is unlikely you keep the index because you need to find all the hardback or paperback books in stock. In fact, you keep this index because customers often ask if you have a particular book in paperback, so you issue a search like FIND TITLE = GERTEL ESCHENBACH AND BINDING = PAPERBACK.

However, a search like that is almost certainly less efficient than FIND TITLE = GERTEL ESCHENBACH, followed by ALSO BINDING = PAPERBACK. The TITLE search would probably only retrieve several records. By ANDing that result with a very large result from the BINDING part of the search (perhaps thousands of pointers), SPIRES must do more work than if it simply examined the few records from the TITLE search directly.

It is usually worthwhile and often quite educational to study how people are using the indexes you have created for a subfile. You can use subfile logging to help you gather this information [See 4.3.] but don't forget that simply asking your users what kind of searches they need or try to do may be the best way to get the information you need.

If you decide to eliminate indexes, keep in mind that the storage space they used may not be released immediately. Rebalancing the data set will release the empty blocks. [See 2.7.2, 3.3.]

3) Remove or archive data you do not need.

The second corollary to the first rule suggests that you should keep files from growing too large with unneeded data. After living with a file for awhile, you may discover that your goal record-type has elements whose values are interesting but of little or no value to the subfile's users. It takes extra work to collect and input these elements, and it takes extra space to store them. If they are Optional elements, you might consider no longer including them in records, or actually updating records to remove them. [See 2.6.2.]

What applies to elements applies to whole records as well. Files that keep regular transactions (mail files, order files, etc.) may keep growing and growing over the years, with the older records having little or no value. You might consider removing them from the file completely, perhaps making a copy of them on tape, on paper or in some other storage medium.

If you remove hundreds of records at once, keep in mind that SPIRES will not release the empty blocks in which they were stored. They will be reused again, though, as new records get added, as long as the available-space chains are correct. [See 3.4.] However, you might want to consider rebuilding the file so that it will use only as many blocks as it really needs. This is especially important to consider if you remove, say, more than half of the records in the record-type or 1000 records, whichever is more. [See 2.8.3, 2.9.]

4) Do your work when the rates are cheapest.

At Stanford, users are charged lower rates when they do their computing work at night or on weekends. If possible, schedule your most resource-intensive work (e.g., building a file, rebuilding indexes, creating a large report) for those periods of time.

5) When possible, use GENERATE LOAD to build files.

The cheapest way to add records to a SPIRES file is with the GENERATE LOAD procedure. [See 2.3.3.] It avoids all the Inproc processing as the records are added, a large portion of the work SPIRES must do for new records. The technique requires that the records come from another SPIRES file, and specifically from a record-type whose characteristics match the one the records are being added to. The GENERATE LOAD procedure is most commonly used when rebuilding a file [See 2.9.] or when you are creating a small test version of a file from a larger one.

6) Use custom input formats, not the standard SPIRES format.

If you cannot use the GENERATE LOAD procedure for input, try to use an input format written especially for the goal record-type. For input, it is generally cheaper to use custom input formats than to use the standard SPIRES format. The reason for this is that most input formats require data to appear in particular positions or within a specific order within the input data set. That is not a requirement of the standard format, which has to determine which element is being processed each time a new element value starts.

However, if your format is so generalized that it too must determine which element appears on each line, it may be no cheaper than the standard SPIRES format. Experimenting with a small sample to check costs might be worthwhile.

7) Use FASTBILD rather than SPIBILD for large files.

If you are building a large file or a large record-type within a file, it is almost always cheaper to build it with FASTBILD rather than SPIBILD. Using FASTBILD is more complicated because it requires you to study your data and give FASTBILD some specific information about it ahead of time. A minor mistake can cause the FASTBILD process to fail, often requiring it to be restarted from the beginning. But FASTBILD is so much more efficient than SPIBILD for most large files that most FASTBILD users find that they still save money over SPIBILD even if they have to restart the FASTBILD process two or three times.

A more detailed comparison of SPIBILD and FASTBILD appears earlier in this manual. [See 2.12.]

If you have decided not to use FASTBILD for one reason or another, consider the FAST BATCH command in SPIBILD rather than simply the INPUT BATCH command. [See 2.3.4.]

8) Use the STATUS command.

Especially for large files, the findings of the STATUS command may far outweigh its cost. [See 3.2.3.] It can tell you that the goal-record tree is badly unbalanced, causing inefficient retrieval of its records, or that storage space in the residual is being wasted. It is easy to correct these problems by using the rebalancing procedure and the AVSPREC procedure. [See 3.3.1, 3.4.3.]

5.3  Resetting the Next Slot Number of a Record-Type

File owners sometimes want more control over the slot numbers doled out by SPIRES as record key values in slot subfiles. The FIX SUBFILE SLOT command, which can be issued by the file owner or accounts having master access to the file, allows you to set the slot number higher (and in some cases, lower) as you deem necessary.

WARNING: Don't use this procedure casually. If you aren't careful, you can cause serious problems, as described below.

FIX SUBFILE SLOT command

The syntax of the FIX SUBFILE SLOT command is simply

The appropriate subfile should already be selected. SPIRES will display the current value for the next slot number (from the system variable $NEXTSLOT) and prompt you for a new value.

For example,

This command is most useful for setting a higher starting slot number. For example, a file owner might decide to set the starting slot number to 1001 when the subfile is first created so that all records (or at least the first 8999 of them) will have a four-digit key. Setting the slot number higher results in a permanent change.

Setting the slot number lower

However, if you set the slot number lower, the lower value remains in effect only while the subfile is currently selected by you. Anyone else selecting the subfile is unaffected by the change.

Moreover, you should be very careful not to overlay new records onto old records, a problem that can arise if you set the slot number lower. This can cause SPIRES to get confused about the correct slot number; for example, it may become convinced that the next slot number is the number of a record that already exists, causing S413 errors each time you try to add a new record.

Here is one scenario in which the command would be useful. Suppose you added 10 records to a subfile, removed them right away, and then wanted to add some more, reusing the same slot numbers again:

Any other users who add records will get slot numbers from 1010 on. For you, though, new records will get slot numbers from 1000 on. Note, however, that it is best to set the slot number lower only at a time when no other users will be updating the subfile.

5.4  Examining Record-Types Directly

Though most users get access to a record-type of a file via the SELECT command, the file owner and users with See access to the file have an alternate way: they can use the ATTACH command to attach any of the file's record-types directly. The ATTACH command has two important aspects:

 - it allows any record-type of the file to be accessed as if it were the goal record-type  of
 a  selected  subfile  without  the file owner having to add a Subfile section for it to the
 file definition; and
 - it provides an alternate way to access  a  goal  record-type,  avoiding  the  environmental
 settings  established  by statements in the Subfile section of the file definition, such as
 secure-switches, subfile logging, a default format, etc.

ATTACH command

The ATTACH command is simple to use. It's available in most SPIRES programs, but its use in SPIRES is what interests us here. Its syntax is:

You need to specify the entire file name, with the "ORV" prefix, if the file does not belong to you. If you omit the "rec-name" or "rec-number" option, SPIRES will attach the first record-type of the file. To see another, add its name to the command, or use its number, from the order in which it is defined in the file definition, from 1 to 64. So, for example, if the index record-type you want to see is the third record-type defined in GQ.DOC.ERRATA, you would type the command "ATTACH 3 OF ORV.GQ.DOC.ERRATA".

Example

Again, a typical use of the ATTACH command might be to get a complete list of all the values in an index:

Remember, you can use the attached record-type as if it were the goal record-type of a selected subfile. You can still search for and display records, and you can even update records too, if you are the file owner or you have Update or Master access to the file. WARNING: Be very careful if the records are index records containing pointers created by SPIRES/SPIBILD; do not change any of these values or you will create data integrity problems.

5.5  Disabling a File Temporarily to Block Selection and Updates

Disabling a file

On occasion, you may need to block users from creating record transactions in a file, or even from selecting a subfile of the file at all. For instance, you might discover a file-processing problem that occurred the previous night, and you want to repair it without having to worry about people adding more records to the file, compounding the problem. You may be able to come up with various ways such changes could be made, such as by altering file permits, or removing the Subfile section from the file definition, etc.

But the fastest way (and usually when you need to do this, time is of the essence) is to 1) select a subfile of the file, and 2) issue the SET SUBFILE DISABLE command:

Under this command, SPIRES will prevent any new users from selecting any subfiles of the file and detach the file from each current user as soon as the user issues a command that accesses the main block of the deferred queue, which includes most record-processing, search and display commands.

These restrictions don't apply to the file owner, nor to users with MASTER access to the file.

Re-enabling a file for use

When you are ready to "enable" your file for use again, select the subfile again and issue the SET SUBFILE ENABLE command:

Both commands are discussed in Part 6 of this manual; online, EXPLAIN SET SUBFILE ENABLE COMMAND.

5.6  Converting to or Modifying Files with Extended Trees

A very large file may run into the limit on the size of record trees, which is 64K blocks per tree. By making some changes to the file definition, you can request that the data set containing the current record-type is to be an "extended-tree" data set, one which can grow to 434,280 blocks. After the file definition has been recompiled using a special option described below, you need to follow a special rebalancing procedure to convert the existing trees that are affected by this change (internally speaking, that means they will have 3-byte instead of 2-byte branch block pointers).

You follow the same procedure to take advantage of the TREE-DATA structure in a file definition, which lets you split a record-type into multiple key trees to help manage tree depth and reduce the amount of rebalancing needed when a record-type is vulnerable to intense growth in a specific key area. If you ever need to change the START-KEYs, you will need to go through this procedure again as well.

Preliminaries

Determine which data set the record-type is in

You need to know which ORVYL data set holds the record-type you are extending or modifying. The easiest way to do that is to ATTACH the file and issue the SHOW FILE TREES command, which will list the record-types of the file along with the data sets they are in. Of course, you need to already know which record-type of the file you are changing.

For the remainder of this procedure, the data set you are going to be working with will be referred to as the "RECn" data set.

Check the data set's ORVYL permit

The REBALANCE command (step 4) will create a data set with the default ORVYL permit setting of Public Read (or Public Write, for data sets containing immediate indexes). That's the same setting SPIRES uses when it creates the "RECn" data set.

However, if you, the file owner, ever changed the permit, you will probably want to reinstate that level of security when this procedure is over. To that end, issue the command:

If the data set has permits other than the appropriate default, jot them down so that you can reset them at the end of this procedure.

Get rid of any interim rebalancing data sets

You should not have any ORVYL data set called "myfile.MREB", "myfile.REBn" or "myfile.REOn.yyyymmdd" ("yyyymmdd" being today's date) when you issue the REBALANCE command, since SPIRES will create them. If any of them do exist, the procedure will stop. To see if you do have any, issue these ORVYL commands:

If you do have any of them, either erase or rename them, using the ORVYL commands "ERASE dataset.name" or "RENAME dataset.name TO new.dataset.name". (You will have to issue the ORVYL command "SET CLP dataset.name" prior to the ERASE or RENAME command if the data set was created by SPIRES.)

Get exclusive access to the file

WARNING: Since the REBALANCE procedure replaces both the data set being rebalanced and the MSTR data set of the file with new copies, at times it needs exclusive access to the file. You might want to use the SET SUBFILE DISABLE FOR ACCESS and SET SUBFILE DISABLE FOR PROCESS commands to prevent any further activity while you are working on the deferred queue.

Step 1: Change the file definition

The specific changes you need to make to the file definition are discussed in the SPIRES manual "File Definition", in chapter B.6.5a. Online, [EXPLAIN EXTENDED-TREE STATEMENT.]

After you have updated the file definition in the FILEDEF subfile, go on to the next step, which describes a special form of the RECOMPILE command you should use anytime you make changes to the file definition that involve either the EXTENDED-TREE statement or TREE-DATA structures.

Step 2: Recompile the file definition using the REBALANCE option

With the FILEDEF subfile still selected in SPIRES, issue the following command:

The compilation creates a new master data set called, for the moment, "filename.MREB". It doesn't replace the MSTR data set till step 5; in the meantime, you will rebuild and rebalance the "RECn" data set so that it matches the extended-tree information that the recompilation put into the MREB data set.

Step 3: ATTACH filename

Issue the ATTACH command to tell SPIRES which file to use for subsequent commands:

This example, which will be followed through this section, demonstrates using the ATTACH command on the file GQ.DOC.ERRATA.

Step 4: [IN ACTIVE] REBALANCE DATA SET RECn CONVERT

This command will rebalance the trees of all the record-types stored in the named data set, storing them in a new ORVYL data set called "myfile.REBn". It uses the record and tree descriptions in the MREB data set.

The display from the REBALANCE command (which goes into your active file if you add the IN ACTIVE prefix) varies from file to file, but it should look something like this one for REC2 of GQ.DOC.ERRATA, which contains three record-types:

SPIRES will also display a message each time 64K nodes have been processed, to reassure you (especially if the record-types are large) that the rebalancing activity is occurring.

Step 5: FIX REBALANCED DATA SET MSTR

This next step, as mentioned above, renames to old MSTR data set (adding today's date to the name), and renames the new MREB data set as the MSTR:

The FIX REBALANCED DATA SET MSTR command detaches the file so that internal tables used by SPIRES can be properly synchronized.

Problems during the rebalancing

If the display ends with the "File detached" message, continue with step 6. If it did not, there may be several reasons why:

 - you received the message "New data set exists; OK to continue?"  because there is
 already a "filename.REBn" data set.   You  should  type  NO  and  return  to  the
 "Preliminaries" above.
 - some other error has occurred.  Use the EXPLAIN command to get more information  about  the
 error.

Step 6: ATTACH filename

Issue the ATTACH command to attach the file again.

Step 7: FIX the rebalanced data set with the RENAME, CONVERT options

Complete the rebalancing procedure by replacing the old "RECn" data set with the new "REBn" data set from step 4. Be sure to include the RENAME and CONVERT options on the end of the FIX REBALANCED DATA SET command:

Cleanup

To clean up after successful rebalancing, you should get rid of the old data set, which you no longer need. Issue the ORVYL commands "SET CLP filename.REOn.yyyymmdd" and "ERASE filename.REOn.yyyymmdd".

If there were special permits set for the data set (see "Preliminaries" above), use ORVYL's SET PERMIT command to reset them:

If you turned off access to the file, you may want to turn it back on again with the SET SUBFILE ENABLE FOR ACCESS and SET SUBFILE ENABLE FOR PROCESS commands (see preliminaries).

After any rebalancing procedure, it is always a good idea to use the STATUS command to get a "snapshot" of the file in its new state. [See 3.2.3.] You can also see information about your new or changed trees by issuing the command SHOW FILE TREES.

5.7  Perform Table Create Commands.

The PERFORM TABLE CREATE command suite has been implemented to facilitate the task of creating Tables which may be useful for various purposes -- primarily for the movement of data from SPIRES to an RDBMS (Relational data base system) or from an RDBMS to SPIRES.

Several commands are included under the PERFORM TABLE CREATE tool set umbrella, each of which may be very useful in different aspects of the various inter-operability functions that may be dealt with to achieve the primary tasks mentioned above.

The central SPIRES structure that is common to these commands is the DECLARE TABLE structure. Each of these PERFORM commands operates on or performs its function based upon SPIRES DECLAREd TABLEs. Another SPIRES process, PERFORM DATA MOVE is the basic goal in mind as the final driving force in the use of these commands. It might be useful to read more about these features. [EXPLAIN DECLARE TABLE COMMAND.] and [See 5.8..]

Syntax of PERFORM TABLE CREATE

So far the command options are:

1. PERFORM TABLE CREATE DECLARE -- Creates SPIRES DEClare Table data. [See 5.7.1.]

2. PERFORM TABLE CREATE DDL -- Creates RDBMS DDL CREATE constructs. [See 5.7.2.]

3. PERFORM TABLE CREATE DATA MOVE -- Generates a DATA MOVE record. [See 5.7.3.]

4. PERFORM TABLE CREATE RECDEF -- Generates SPIRES RECDEF records. [See 5.7.4.]

5. PERFORM TABLE CREATE INPUT -- Generates INPUT CONTROL constructs. [See 5.7.5.]

6. PERFORM TABLE CREATE ELEMINFO -- Creates SPIRES Data Element Information records. [See 5.7.9.]

5.7.1  Perform Table Create Declare Command

PERFORM TABLE CREATE DECLARE -- Creates SPIRES Declare Table data

The output from the PERFORM TABLE CREATE DECLARE command is a series of Declare Table constructs which are intended to be used by SPIRES to tell it how to generate relational tables from SPIRES source elements and structures of a particular SPIRES subfile. Once these Tables are generated, they may be used in a variety of ways. For example, they may be used in conjunction with other PERFORM commands to generate RDBMS DDL statements [See 5.7.2,.] to generate a source record for the DATA MOVE process [See 5.7.3,.] or to generate a relational table View of the source SPIRES subfile itself [EXPLAIN DECLARE TABLE COMMAND.]

There is a good chance that the DECLARE TABLE structures that are generated for a particular subfile will need to be modified or supplemented in some way depending on your needs. You may wish to define a "DESTINATION SUBFILE" to simplify this modification (via normal SPIRES update commands) as well as to maintain a convenient database for generated SPIRES tables.

NOTE -- There is a special parameter that could be very useful if you wish to generate tables that form a precise relationship to an existing SPIRES data base. This feature is initiated through the use of the "FROM" parameter and utilizes a specially prepared database that describes the actual usage of source subfile data elements. This feature is described in [See 5.7.9,.] and is shown in an example at [See 5.7.11,.]

Syntax of PERFORM TABLE CREATE DECLARE

      Note: The File Definition for the goal records for this
      database should be DEFINED-BY = $TABLE.
       MULTiple - Generate tables for multiply occurring data elements.
       VIRTual  - Generate tables for virtual data elements.
       CHANGES  - Generate tables that can deal with insert/
                   delete records of changed row data.
       EXISTS   - Used in conjunction with the FROM parameter (see NOTE)

       Note 1: If you wish to specify more than one option you must
       write the options surrounded by single quotes.  For example
       - Options = 'Mult, Virt'

       Note 2: Currently the maximum number of Columns allowed
       within a Declare Table structure is 126.  Including virtual
       elements could increase the number dramatically.  Therefore
       one should make an effort to eliminate unnecessary Virtual
       element columns.  The Rdbms Column names for virtuals has
       a "_v" suffix.
       SYBASE -- Generate Sybase Rdbms info
       ORACLE -- Generate Oracle rdbms info
       NSI -- Generate Oracle rdbms info using data element
          lookups from the NSI NOTEBOOK subfile.
       SPIRES -- Generates DECLARE TABLE data with no reference
          to Rdbms information.

Examples of command execution in SPIRES

  > perform table create declare subfile family typ oracle, opt 'mult, virt'
  > list
      1.      Declare Table FAMILY
      2.        ID = GQ.WCK.FAMILY;
      3.        COMMENTS = Table Generation for Record Level Elements;
      4.        DEFDATE = TUES. JULY 25, 2000;
      5.        MODDATE = TUES. JULY 25, 2000;
      6.        MODTIME = 12:37:44;
      7.        SUBFILE.NAME = FAMILY;
      8.        DECLARE.SUBFILE = Data Move Declares;
      9.        COLNAME = ID;
     10.          ISKEY;
     11.          COLTYPE = INT;
     12.          COLWIDTH = 3;
     13.          RDBMS_COLUMN = id;
     14.          RDBMS_DATATYPE = VARCHAR2;
     15.          RDBMS_DATALENGTH = 3;
     16.        COLNAME = NAME;
     17.          COLWIDTH = 28;
     18.          RDBMS_COLUMN = name;
     19.          RDBMS_DATATYPE = VARCHAR2;
     20.          RDBMS_DATALENGTH = 28;
     21.        COLNAME = CALLED;
     22.          COMMENTS = Multiple;
     23.          COLWIDTH = 12;
     24.          SOURCE.SINGLE;
     25.          RDBMS_COLUMN = called;
     26.          RDBMS_DATATYPE = VARCHAR2;
     27.          RDBMS_DATALENGTH = 12;
     28.        COLNAME = SEX;
     29.          COLWIDTH = 3;
     30.          RDBMS_COLUMN = sex;
     31.          RDBMS_DATATYPE = VARCHAR2;
     32.          RDBMS_DATALENGTH = 3;
     33.        COLNAME = PARENT_1;
     34.          COMMENTS = Multiple;
     35.          COLTYPE = INT;
     36.          COLWIDTH = 9;
     37.          SOURCE.ELEM = PARENT;
     38.          DECLARE.KEY = $VALUE.EXTERNAL;
     39.          SOURCE.OCC = 1;
     40.          RDBMS_COLUMN = parent_1;
     41.          RDBMS_DATATYPE = VARCHAR2;
     42.          RDBMS_DATALENGTH = 9;
     43.        COLNAME = PARENT_2;
     44.          COMMENTS = Multiple;
     45.          COLTYPE = INT;
     46.          COLWIDTH = 9;
     47.          SOURCE.ELEM = PARENT;
     48.          DECLARE.KEY = $VALUE.EXTERNAL;
     49.          SOURCE.OCC = 2;
     50.          RDBMS_COLUMN = parent_2;
     51.          RDBMS_DATATYPE = VARCHAR2;
     52.          RDBMS_DATALENGTH = 9;
     53.        COLNAME = BIRTH;
     54.          COLTYPE = DATE;
     55.          COLWIDTH = 12;
     56.          SOURCE.ELEM = BIRTH;
     57.          DECLARE.KEY = $DATEOUT.CCYY;
     58.          RDBMS_COLUMN = birth;
     59.          RDBMS_DATATYPE = VARCHAR2;
     60.          RDBMS_DATALENGTH = 12;
             .................................

    130.        FILE = GQ.WCK.FAMILY;
    131.        TABLE.NUM = 001;
    132.        RDBMS_TABLENAME = family;
    133.      EndDeclare
    134.
    135.      Declare Table CHILD
    136.        ID = GQ.WCK.CHILD;
    137.        COMMENTS = Table Generation for Element CHILD;
    138.        DEFDATE = TUES. JULY 25, 2000;
    139.        MODDATE = TUES. JULY 25, 2000;
    140.        MODTIME = 12:37:44;
    141.        SUBFILE.NAME = FAMILY;
    142.        DECLARE.SUBFILE = Data Move Declares;
    143.        COLNAME = ID;
    144.          COMMENTS = REF FAMILY;
    145.          ISKEY;
    146.          COLTYPE = INT;
    147.          COLWIDTH = 3;
    148.          RDBMS_COLUMN = id;
    149.          RDBMS_DATATYPE = VARCHAR2;
    150.          RDBMS_DATALENGTH = 3;
    151.        COLNAME = CHILD.OCC;
    152.          COMMENTS = Occurrence Number;
    153.          ISKEY;
    154.          COLTYPE = INT;
    155.          SOURCE.ELEM = CHILD;
    156.          DECLARE.KEY = $VALUE.OCC;
    157.          SOURCE.SINGLE;
    158.          RDBMS_COLUMN = child_occ;
    159.          RDBMS_DATATYPE = NUMBER;
    160.          RDBMS_DATALENGTH = 4;
    161.        COLNAME = CHILD;
    162.          COMMENTS = Multiple;
    163.          COLTYPE = INT;
    164.          COLWIDTH = 5;
    165.          RDBMS_COLUMN = child;
    166.          RDBMS_DATATYPE = VARCHAR2;
    167.          RDBMS_DATALENGTH = 5;
    168.        FILE = GQ.WCK.FAMILY;
    169.        MULTI.ELEMENT = CHILD;
    170.        TABLE.NUM = 002;
    171.        RDBMS_TABLENAME = child;
    172.      EndDeclare
    173.

5.7.2  Perform Table Create DDL Command

PERFORM TABLE CREATE DDL -- Creates RDBMS DDL CREATE constructs

This form of the PERFORM TABLE CREATE command will generate Rdbms DDL CREATE commands whose values are based directly on the syntax of the PERFORM TABLE CREATE DECLARE command. [See 5.7.1.]

There are two ways of representing this command, the options differing based upon the source of the input:

      In this case the DECLARE TABLE structures stored (and possibly
      modified) in the Destination Subfile that have the given
      Subfile.Name value are generated and returned in your ACTIVE
      file.  Note that you may choose instead to specify source
      table records using the KEY or KEY PREFIX values.
      This form of the command is given with the same options that
      you would use to create DECLARE statements where the output
      is to be presented in the ACTIVE file.

Syntax of PERFORM TABLE CREATE DDL

 - This form of the command is the "normal" form since you would generally  want  to
 modify the Declare Table records generated during the Create Declare process.
      Note: If the FROM statement is not given then you
      should issue the PERFORM TABLE CREATE DDL command
      with all the options used with the PERFORM TABLE
      CREATE DECLARE statement, with the exception of
      the DESTINATION option.

Examples of command execution in SPIRES

 > perform table create DDL subfile family type oracle, opt 'mult, virt'
  - DDL Statement Totals for Subfile family
    Table: family                     Columns: 22     Row length: 4814
    Table: note                       Columns: 3      Row length: 519
    Table: child                      Columns: 3      Row length: 12
    Table: marriage_str               Columns: 11     Row length: 2101
    ------
    Totals      Tables: 4             Columns: 39     Row length: 7446
 > list
     1.
     2.      CREATE TABLE family
     3.        (id                   VARCHAR2(3) NOT NULL,
     4.         name                 VARCHAR2(28) NULL,
     5.         called               VARCHAR2(12) NULL,
     6.         sex                  VARCHAR2(3) NULL,
     7.         parent_1             VARCHAR2(9) NULL,
     8.         parent_2             VARCHAR2(9) NULL,
     9.         birth                VARCHAR2(12) NULL,
    10.         birth_place          VARCHAR2(512) NULL,
    11.         birth_home           VARCHAR2(512) NULL,
    12.         death                VARCHAR2(12) NULL,
    13.         death_place          VARCHAR2(512) NULL,
    14.         name_suffix          VARCHAR2(512) NULL,
    15.         known_name_v         VARCHAR2(512) NULL,
    16.         birth_month_v        VARCHAR2(512) NULL,
    17.         birth_mmdd_v         VARCHAR2(512) NULL,
    18.         child_count_v        VARCHAR2(6) NULL,
    19.         birth_info_v         VARCHAR2(34) NULL,
    20.         birthplace_v         VARCHAR2(20) NULL,
    21.         death_info_v         VARCHAR2(34) NULL,
    22.         sort_name_v          VARCHAR2(24) NULL,
    23.         first_name_v         VARCHAR2(512) NULL,
    24.         full_name_v          VARCHAR2(512) NULL,
    25.            CONSTRAINT family_pk PRIMARY KEY (id));
    26.
    27.      CREATE TABLE note
    28.        (id                   VARCHAR2(3) NOT NULL,
    29.         note_occ             NUMBER(4) NOT NULL,
    30.         note                 VARCHAR2(512) NULL,
    31.            CONSTRAINT note_pk PRIMARY KEY (id, note_occ),
    32.            CONSTRAINT note_family_fk FOREIGN KEY (id) REFERENCES family
    33.     (id));
           ..................................

5.7.3  Perform Table Create DATA MOVE Command

PERFORM TABLE CREATE DATA MOVE -- Creates a DATA MOVE record

This command may be used to generate a DATA MOVE subfile record that may be used as is or through modification to simplify data extracts from the source subfile to multiple Target tables.

There are two ways of representing this command, the options differing based upon the source of the input:

      In this case the DECLARE TABLE structures stored (and possibly
      modified) in the Destination Subfile that have the given
      Subfile.Name value are generated and returned in your ACTIVE
      file.  Note that you may choose instead to specify source
      table records using the KEY or KEY PREFIX values.
      This form of the command is given with the same options that
      you would use to create DECLARE statements where the output
      is to be presented in the ACTIVE file.

[See 5.7.1.]

Syntax of PERFORM TABLE CREATE DATA MOVE

 - This form of the command is the "normal" form since you would generally  want  to
 modify the Declare Table records generated during the Create Declare process.
      Note: If the FROM statement is not given then you
      should issue the PERFORM TABLE CREATE DATA MOVE
      command with all the options used with the PERFORM
      TABLE CREATE DECLARE statement, with the exception
      of the DESTINATION option.

Example of use:

It may be instructive to show an example of the use of a protocol which goes through all the steps from the generation of Declared Tables to the generation of a DATA MOVE record and to use that record to actually generate a complete set of relational tables for a subfile.

Note that the "Record.Length" and "lrecl" values generated for the target OS files in the DATA MOVE record are much larger than the corresponding "Max Length" values generated during DATA MOVE processing. This could cause serious space problems in a large volume move. Therefore, you should adjust the values accordingly either by adding ELEMINFO width statements in the File definition, COLWIDTH statements in the DECLARE TABLE records, or by altering the DATA MOVE record itself.

 *TestDMove
 > Declare Vgroup Local
     var = KeyValue;  Len = 40;
     var = DBase;  len = 40;
   EndDeclare

 > set echo
 > clear ACTIVE
 > let DBase = races
 > /perform table create declare subfile #DBase dest 'wck.tables' opt mult
 > /perform table create data move from 'wck.tables', subfile #DBase
 > list
     1.
     2.      Id = *RACES_TABLE_MOVE;
     3.        Subfile.Name = &GQ.WCK.RACES RACES;
     4.        Table.Subfile = &GQ.WCK.TABLES WCK.TABLES;
     5.        Area.Name = Table1;
     6.          Device.Type = OS File;
     7.          File.Name = WYL.GQ.WCK.RACES;
     8.          Record.Length = 2590;
     9.          Device.Options = var,lrecl=2590,repl,refor,temp;
    10.        Area.Name = Table2;
    11.          Device.Type = OS File;
    12.          File.Name = WYL.GQ.WCK.NAME;
    13.          Record.Length = 1030;
    14.          Device.Options = var,lrecl=1030,repl,refor,temp;
    15.        Area.Name = Table3;
    16.          Device.Type = OS File;
    17.          File.Name = WYL.GQ.WCK.RACES.X;
    18.          Record.Length = 1066;
    19.          Device.Options = var,lrecl=1066,repl,refor,temp;
    20.        Area.Name = Table4;
    21.          Device.Type = OS File;
    22.          File.Name = WYL.GQ.WCK.COMMENTS;
    23.          Record.Length = 1040;
    24.          Device.Options = var,lrecl=1040,repl,refor,temp;
    25.        Target.Name = RACES;
    26.          Generate.Table = GQ.WCK.RACES;
    27.          Area = Table1;
    28.        Target.Name = NAME;
    29.          Generate.Table = GQ.WCK.NAME;
    30.          Area = Table2;
    31.        Target.Name = RACES_X;
    32.          Generate.Table = GQ.WCK.RACES.X;
    33.          Area = Table3;
    34.        Target.Name = COMMENTS;
    35.          Generate.Table = GQ.WCK.COMMENTS;
    36.          Area = Table4;
 > select data move
 > addupd
 > let KeyValue = $key
 > /select #DBase
 > for subfile
 > /perform data move from #KeyValue

  Totals for Subfile RACES Table Move                  Jul 26, 2000
  Table Name              Table     Row     Max     Line      Total
                          Type     Count   Length   Count     Length
  =====================   =====  =======   ======  =======  =========
  races                   DELIM       13      83       14        622
  name                    DELIM       13      25       14        286
  races_x                 DELIM       35      51       36       1680
  comments                DELIM        8      32        9        243
 >

5.7.4  Perform Table Create RECDEF Command

PERFORM TABLE CREATE RECDEF -- Creates SPIRES RECDEF records

This form of the PERFORM TABLE CREATE command will generate SPIRES Record Definition (RECDEF) records which mirror the view of data described by Declare Table records generated by the PERFORM TABLE CREATE DECLARE command. [See 5.7.1.]

This command is expressed as follows:

      The DECLARE TABLE structure stored (and possibly modified)
      in the Table Subfile that has the given key value is used
      as the source for RECDEF generation.  Note that you must
      also code the SUBFILE parameter as SPIRES uses this value
      to determine record characteristics when generating the
      record definition.

Syntax of PERFORM TABLE CREATE RECDEF

      Note: If the FROM statement is not given then you
      should issue the PERFORM TABLE CREATE RECDEF command
      with all the options used with the PERFORM TABLE
      CREATE DECLARE statement, with the exception of
      the DESTINATION option.

Examples of command execution in SPIRES

 > perform table create recdef from wck.tables subfile races key *races
 > list
     1.     ****
     2.      ID = GQ.WCK.RACES;
     3.      MODDATE = FRI. AUG. 18, 2000;
     4.      DEFDATE = FRI. AUG. 18, 2000;
     5.      REMOVED;
     6.      REQUIRED;
     7.        KEY = RUNNER;
     8.          OCCURS = 1;
     9.          ELEMINFO;
    10.            WIDTH = 64;
    11.            RDBMS_COLUMN = runner;
    12.            RDBMS_DATATYPE = VARCHAR2;
    13.            RDBMS_DATALENGTH = 64;
    14.      OPTIONAL;
    15.        ELEM = FAC.STAFF;
    16.          OCCURS = 1;
    17.          ELEMINFO;
    18.            WIDTH = 1;
    19.            RDBMS_COLUMN = fac_staff;
    20.            RDBMS_DATATYPE = VARCHAR2;
    21.            RDBMS_DATALENGTH = 1;
    22.        ELEM = MAIL.ADDR1;
    23.          OCCURS = 1;
    24.          ELEMINFO;
    25.            WIDTH = 64;
    26.            RDBMS_COLUMN = mail_addr1;
    27.            RDBMS_DATATYPE = VARCHAR2;
    28.            RDBMS_DATALENGTH = 64;
    29.        ELEM = MAIL.ADDR2;
    30.          OCCURS = 1;
    31.          ELEMINFO;
    32.            WIDTH = 64;
    33.            RDBMS_COLUMN = mail_addr2;
    34.            RDBMS_DATATYPE = VARCHAR2;
    35.            RDBMS_DATALENGTH = 64;
    36.        ELEM = PHONE;
    37.          OCCURS = 1;
    38.          ELEMINFO;
    39.            WIDTH = 16;
    40.            RDBMS_COLUMN = phone;
    41.            RDBMS_DATATYPE = VARCHAR2;
    42.            RDBMS_DATALENGTH = 16;
    43.        ELEM = DEPARTMENT;
    44.          OCCURS = 1;
    45.          ELEMINFO;
    46.            WIDTH = 16;
    47.            RDBMS_COLUMN = department;
    48.            RDBMS_DATATYPE = VARCHAR2;
    49.            RDBMS_DATALENGTH = 16;
    50.        ELEM = TITLE;
    51.          OCCURS = 1;
    52.          ELEMINFO;
    53.            RDBMS_COLUMN = title;
    54.            RDBMS_DATATYPE = VARCHAR2;
    55.            RDBMS_DATALENGTH = 512;
    56.        ELEM = BIRTH.DATE;
    57.          OCCURS = 1;
    58.          INPROC = $DATEYY;
    59.          OUTPROC = $DATEYY.OUT(,CCYY);
    60.          ELEMINFO;
    61.            WIDTH = 10;
    62.            RDBMS_COLUMN = birth_date;
    63.            RDBMS_DATATYPE = VARCHAR2;
    64.            RDBMS_DATALENGTH = 10;
    65.        ELEM = SEX;
    66.          OCCURS = 1;
    67.          ELEMINFO;
    68.            WIDTH = 3;
    69.            RDBMS_COLUMN = sex;
    70.            RDBMS_DATATYPE = VARCHAR2;
    71.            RDBMS_DATALENGTH = 3;
    72.     ;

5.7.5  Perform Table Create Input Commands.

The PERFORM TABLE CREATE INPUT command suite has been implemented to facilitate the task of creating Tables which may be useful for a specific purpose, that of simplifying the task for generating the structures needed by the INPUT CONTROL command. [EXPLAIN INPUT CONTROL.]

Several commands are included under the PERFORM TABLE CREATE INPUT tool set, each of which are very useful for Input Control operation. In using these commands be aware that they are intended as a means to simplify the generation of the structures. It is important to understand that modification of this data may be necessary in order to achieve the desired result -- the generation of SPIRES hierarchical records from multiple streams of input tables.

The central SPIRES structure that is common to these commands is the DECLARE TABLE structure. Output Tables generated by PERFORM TABLE create declare form the key ingredient to the generation of the DECLARE INPUT TABLE structures. The DECLARE INPUT TABLE structure is then in turn forms the basis for the subsequent commands

Syntax of PERFORM TABLE CREATE INPUT

So far the command options are:

1. PERFORM TABLE CREATE INPUT DECLARE -- Creates SPIRES DEClare Input Table data. [See 5.7.6.]

2. PERFORM TABLE CREATE INPUT RECDEF -- Generates SPIRES RECDEF records for data input. [See 5.7.7.]

3. PERFORM TABLE CREATE INPUT CONTROL -- Generates SPIRES commands that execute INPUT CONTROL activity. [See 5.7.8.]

5.7.6  Perform Table Create Input Declare Command

PERFORM TABLE CREATE INPUT DECLARE -- Creates SPIRES DECLARE INPUT TABLE structures

This form of the PERFORM TABLE CREATE command will generate SPIRES DECLARE INPUT TABLE structures which will form a basic piece of the set of commands needed in INPUT CONTROL processing. [EXPLAIN DECLARE INPUT TABLE COMMAND.] [EXPLAIN INPUT CONTROL.]

This command is expressed as follows:

      The DECLARE TABLE structure stored (and possibly modified)
      in the Table Subfile that has the given key value is used
      as the source for DECLARE INPUT TABLE generation.

Syntax of PERFORM TABLE CREATE INPUT DECLARE

5.7.7  Perform Table Create Input INPUT RECDEF Command

PERFORM TABLE CREATE INPUT RECDEF -- Creates SPIRES RECDEF records for input

This form of the PERFORM TABLE CREATE command will generate SPIRES Record Definition (RECDEF) records which mirror the view of data described by Declare Input Table records generated by the PERFORM TABLE CREATE INPUT DECLARE command. [See 5.7.6.]

This command is expressed as follows:

      The DECLARE INPUT TABLE structure stored (and modified)
      in the Input Table Subfile that has the given key value is
      used as the source for RECDEF generation.

Syntax of PERFORM TABLE CREATE INPUT RECDEF

5.7.8  Perform Table Create Input INPUT CONTROL Command

PERFORM TABLE CREATE INPUT CONTROL -- Creates a SPIRES protocol for input control

This form of the PERFORM TABLE CREATE command will generate a SPIRES protocol which is made up of a sample set of commands that will provide the basis for Input Control activity. [EXPLAIN INPUT CONTROL.]

This command is expressed as follows:

      The DECLARE INPUT TABLE structure stored (and modified)
      in the Input Table Subfile that has the given key value is
      used to provide information used in the generated protocol
      commands.

Syntax of PERFORM TABLE CREATE INPUT CONTROL

5.7.9  Perform Table Create Eleminfo Command

PERFORM TABLE CREATE ELEMINFO -- Creates SPIRES Data Element Information

The PERFORM TABLE CREATE commands have been built to simplify the Process needed to convert SPIRES hierarchical record structures into a form that is acceptable for the row/column structure of data in the Relational model of data storage. This set of PERFORM commands has proved to be extremely useful for the conversion process but difficulties may arise, especially if those assigned to handle the conversion are not familiar with the data being converted.

The PERFORM TABLE CREATE ELEMINFO command in conjunction with the other PERFORM TABLE commands has been constructed as another, potentially important piece of this tool set. This command is designed to produce a SPIRES database which holds information pertaining to every data element of a subfile which in turn may be used to aid the construction of the Row/Column data descriptions in a more precise manner than has been previously available.

The CREATE ELEMINFO process is able to gain this precision by scanning the actual records of the source SPIRES database in order to determine exact length/occurrence values for each data element of a subfile. The resulting data collection may then be used by the Create declare process [See 5.7.1,.] to generate tables that more accurately reflect the actual data attributes that exist in the source subfile rather than the suggested attributes based upon the data description from the SPIRES File Definition.

This process will prove less useful for SPIRES subfiles that have been designed with great care, those whose File Definitions more precisely define its data element attributes. Many SPIRES files have not had such a history. SPIRES provides an extremely flexible, yet efficient opportunity to be less precise, and this flexibility has led to databases that have grown and evolved over decades of time and yet have never been rebuilt. This flexibility was designed and built into SPIRES from the beginning over thirty years ago. The system was designed as the Spires PUBLIC Information Retrieval System, a system in which any user with a Forsythe account could build his/her own private or departmental database application without any contact with a DBA (there weren't any).

The SPIRES to Relational conversion is hampered by the fact of this flexibility due to several aspects of SPIRES design.

 1.  Variable Length Data - Users rarely specify fixed length
     constraints on data values.  In most cases, efficiency is not
     served by doing this.  Of the tens of thousands of SPIRES
     databases, I know of only one which was made up of nothing
     but fixed length data.  Relational systems thrive on it.

 2.  Multiply Occurring Data - It does help data elements to be
     defined as singly occurring when they are singly occurring,
     but this is often neglected.  It matters greatly in SPIRES
     to Relational conversion.

 3.  Optional Data - A SPIRES file may have a number of Optional
     data elements defined which may in fact never occur, or which
     have occurred in the past but have bypassed their usefulness
     and have been deleted.  Optional data that does not appear in
     a record requires but one bit of space (or none) to express
     this fact.

The current PERFORM TABLE commands would generate a separate table for a data element defined as optional, variable length, multiply occurring even if such an element never occurs in the data base.

This command eliminates such problems since it can scan an entire database and report the existence or non existence of every piece of data in the database.

Syntax of PERFORM TABLE CREATE ELEMINFO

Examples of command execution in SPIRES

  > perform table create eleminfo subfile restaurant
  - Scanned 939 records of 939 in RESTAURANT
  - Generated 23 Element Info records
  > list unn
  1April 2, 2003                                                     Page 1

   name               elemid   type       exists  max_len  max_occ  avg_xoc
   ----------------  --------  ---------  ------  -------  -------  -------
   ID                00000000  Integer       939        4        1     1.00
   DATE-ADDED        00000001  Hex           939       14        1     1.00
   DATE-UPDATED      00000002  Hex           939       14        1     1.00
   NAME              00000003  String        939       38        1     1.00
   LOCATION          00000004  Structure     939        0        1     1.00
   RECOMMENDER       00000005  String        700       83        4     1.01
   COMMENTS          00000006  String        936      988        1     1.00
   CUISINE           00000007  String        809       80        3     1.01
   PRICE-RANGE       00000008  Integer         5        5        1     1.00
   PRICE-AVERAGE     00000009  Integer       776        7        1     1.00
   FOOD-QUALITY      0000000A  Integer       851        9        1     1.00
   SERVICE-QUALITY   0000000B  Integer       823        9        1     1.00
   CREDIT            0000000C  Integer       557        5        1     1.00
   RESERVATIONS      0000000D  String        615       74        1     1.00
   HOURS             0000000E  String        509       78        1     1.00
   ENTERTAINMENT     0000000F  String        593      135        1     1.00
   BAR               00000010  Integer       557        9        3     1.01
   CITY              00000200  String        939       20        1     1.00
   STATE             00000201  String        939        2        1     1.00
   PHONE             00000202  String        516       14        1     1.00
   ADDRESS           00000203  String        870       41        1     1.00
   ZIP-CODE          00000204  String         95        5        1     1.00
   LOC.D             00000205  String          0        0        0        0

This display gives a snapshot of the information gathered for each of the data elements in the RESTAURANT subfile. The data shown for any particular data element may be interpreted as follows:

 name    - Data element name.
 elemid  - SPIRES internal element Identifier
 type    - SPIRES internal element type
 exists  - Number of records in which the element exists
 max_len - Maximum length of this element within the subfile
           (This value is meaningless for type STRUCTURE elements)
 max_occ - maximum number of occurrences of the element in any record
 avg_xoc - Average number of occurrences of the element in the
           records in which the element exists.

This snapshot display may prove useful in that it gives a good idea of data element usage but if you wish to utilize this process to its maximum potential during SPIRES to relational table conversions you should make use of the DESTINATION parameter.

The following is a sample of the usage of this parameter.

 > perform table create eleminfo subfile restaurant dest eleminfo
 - Scanned 939 records of 939 in RESTAURANT
 - Generated 23 Element Info records
 > select eleminfo
 > for subfile where subfile.name = restaurant
 > display 2
                     ----------------------------------------

  Element Name: ADDRESS              in Structure LOCATION
    File GQ.DOC.RESTAURANT              Subfile: RESTAURANT
    Internal Type: String               Optional, Singular
      Exists in 870 Records of 939 Scanned
      Occurrences: Total - 870          Maximum - 1         Average - 1.00
      Elem Length: Total - 15827        Maximum - 41        Average - 18.19

             A singly occurring element which gives the street address
          for the restaurant.  A warning message will be issued if the
          value is over 36 characters long.
                     ----------------------------------------

  Element Name: BAR
    File GQ.DOC.RESTAURANT              Subfile: RESTAURANT
    Internal Type: Integer              Optional, Multiple
      Exists in 557 Records of 939 Scanned
      Occurrences: Total - 564          Maximum - 3         Average - 1.01
      Elem Length: Total - 3231         Maximum - 9         Average - 5.73

             Indicates what kind of bar is available.  The four
          possible choices are:
                None, Beer, Wine, Cocktails.
             Other values will not be accepted.

The unformatted output for the preceding record is as follows:

 > transfer *
 > l
     1.      KGROUP = BAR             /RESTAURANT;
     2.      FILE.NAME = GQ.DOC.RESTAURANT;
     3.      DATE.ADD = WED. APRIL 9, 2003;
     4.      TYPE = Integer;
     5.      OCCURS = Multiple;
     6.      PART = Optional;
     7.      ELEMID = 00000010;
     8.      TOT_RECORDS = 939;
     9.      MAX_OCC = 3;
    10.      MAX_OCST = 3;
    11.      TOT_OCC = 564;
    12.      MAX_LEN = 9;
    13.      TOT_LEN = 3231;
    14.      EXISTS = 557;
    15.      ELEMINFO;
    16.        DESCRIPTION = Indicates what kind of bar is available.  The four
    17.     possible choices are:;
    18.        DESCRIPTION = "   None, Beer, Wine, Cocktails.";
    19.        DESCRIPTION = Other values will not be accepted.;
    20.        HEADING = Bar;
    21.        WIDTH = 9;
    22.        VALUE-TYPE = CODED;
 >

As you can see from this example, the information collected during the CREATE ELEMINFO process is much more extensive than the snapshot shown in the preceding example.

For a more complete description of the information being gathered and how you can build your own element information subfile [See 5.7.10,.]

Also [See 5.7.11,.] to see how you can utilize the element information subfile to improve the generation of relational tables in conjunction with the CREATE DECLARE process.

5.7.11  Perform Table Create Declare, Using ELEMINFO

PERFORM TABLE CREATE DECLARE, USING ELEMINFO -- Generate SPIRES tables

You have the ability to build SPIRES Declare structures that are based upon data derived from the actual records in the source subfile database through the utilization of records stored in a second SPIRES database. The generation of this second database is described by [See 5.7.9,.]

The example shown in that explanation shows how to generate records for the data elements of the SPIRES RESTAURANT subfile. You may also [See 5.7.10,.] to see how that "second' subfile whose name is "eleminfo" is defined. Of course, you should create your own "eleminfo" subfile or have access to one that has been set up for you.

The PERFORM TABLE CREATE DECLARE command below depicts how you may use this element information subfile when generating the Declare Table structures.

Note the FROM parameter which gives the name of the subfile holding the gathered information from RESTAURANTS. Also, note that we are using the 'EXISTS" option. The EXISTS option tells SPIRES to ignore generating row/column data for data elements which are defined but which do not exist in the subfile.

It may be instructive to see how this "ELEMINFO" feature generates more efficient table structures. The "cuisine" data element is defined as a multiply occurring, variable length data element. However, the CREATE ELEMINFO command generated the following record.

 Element Name: CUISINE
   File GQ.DOC.RESTAURANT              Subfile: RESTAURANT
   Internal Type: String               Optional, Multiple
     Exists in 809 Records of 939 Scanned
     Occurrences: Total - 820          Maximum - 3         Average - 1.01
     Elem Length: Total - 12120        Maximum - 80        Average - 14.78

            A textual description of the type(s) of cuisine the
         restaurant offers.
            The description should be fairly brief.

The following command shows that a separate CUISINE table was not generated since the maximum number of occurrences in any record was three. The cuisne field translates into three columns in the RESTAURANT table. In contrast the RECOMMENDER element had a maximum number of four occurrences in a record. Hence the separate table shown below.

 > perform table create declare subfile restaurant ...
           from eleminfo type oracle options 'mult,exists'
 > list
     1.      Declare Table RESTAURANT
     2.        ID = GQ.WCK.RESTAURANT;
     3.        COMMENTS = Table Generation for Record Level Elements;
     4.        DEFDATE = WED. APRIL 9, 2003;
     5.        MODDATE = WED. APRIL 9, 2003;
     6.        MODTIME = 10:13:56;
     7.        SUBFILE.NAME = RESTAURANT;
     8.        DECLARE.SUBFILE = Data Move Declares;
     9.        COLNAME = ID;
    10.          ISKEY;
    11.          COLTYPE = INT;
    12.          COLWIDTH = 4;
    13.          RDBMS_COLUMN = id;
    14.          RDBMS_DATATYPE = NUMBER;
    15.          RDBMS_DATALENGTH = 4;
    16.        COLNAME = DATE-ADDED;
    17.          COMMENTS = Required;
    18.          COLTYPE = DATE;
    19.          COLWIDTH = 15;
    20.          SOURCE.ELEM = DATE-ADDED;
    21.          DECLARE.KEY = $DATEOUT.CCYY;
    22.          RDBMS_COLUMN = date_added;
    23.          RDBMS_DATATYPE = DATE;
    24.        COLNAME = DATE-UPDATED;
    25.          COMMENTS = Required;
    26.          COLTYPE = DATE;
    27.          COLWIDTH = 15;
    28.          SOURCE.ELEM = DATE-UPDATED;
    29.          DECLARE.KEY = $DATEOUT.CCYY;
    30.          RDBMS_COLUMN = date_updated;
    31.          RDBMS_DATATYPE = DATE;
    32.        COLNAME = NAME;
    33.          COMMENTS = Required;
    34.          COLWIDTH = 38;
    35.          RDBMS_COLUMN = name;
    36.          RDBMS_DATATYPE = VARCHAR2;
    37.          RDBMS_DATALENGTH = 38;
    38.        COLNAME = COMMENTS;
    39.          COLWIDTH = 988;
    40.          RDBMS_COLUMN = comments;
    41.          RDBMS_DATATYPE = VARCHAR2;
    42.          RDBMS_DATALENGTH = 988;
    43.        COLNAME = CUISINE_1;
    44.          COMMENTS = Multiple;
    45.          COLWIDTH = 80;
    46.          SOURCE.ELEM = CUISINE;
    47.          DECLARE.KEY = $VALUE.EXTERNAL;
    48.          SOURCE.OCC = 1;
    49.          RDBMS_COLUMN = cuisine_1;
    50.          RDBMS_DATATYPE = VARCHAR2;
    51.          RDBMS_DATALENGTH = 80;
    52.        COLNAME = CUISINE_2;
    53.          COMMENTS = Multiple;
    54.          COLWIDTH = 80;
    55.          SOURCE.ELEM = CUISINE;
    56.          DECLARE.KEY = $VALUE.EXTERNAL;
    57.          SOURCE.OCC = 2;
    58.          RDBMS_COLUMN = cuisine_2;
    59.          RDBMS_DATATYPE = VARCHAR2;
    60.          RDBMS_DATALENGTH = 80;
    61.        COLNAME = CUISINE_3;
    62.          COMMENTS = Multiple;
    63.          COLWIDTH = 80;
    64.          SOURCE.ELEM = CUISINE;
    65.          DECLARE.KEY = $VALUE.EXTERNAL;
    66.          SOURCE.OCC = 3;
    67.          RDBMS_COLUMN = cuisine_3;
    68.          RDBMS_DATATYPE = VARCHAR2;
    69.          RDBMS_DATALENGTH = 80;
    70.        COLNAME = PRICE-RANGE;
    71.          COLWIDTH = 5;
    72.          RDBMS_COLUMN = price_range;
    73.          RDBMS_DATATYPE = VARCHAR2;
    74.          RDBMS_DATALENGTH = 5;
                    ..........
   105.        COLNAME = ENTERTAINMENT;
   106.          COLWIDTH = 135;
   107.          RDBMS_COLUMN = entertainment;
   108.          RDBMS_DATATYPE = VARCHAR2;
   109.          RDBMS_DATALENGTH = 135;
   110.        COLNAME = BAR_1;
   111.          COMMENTS = Multiple;
   112.          COLWIDTH = 9;
   113.          SOURCE.ELEM = BAR;
   114.          DECLARE.KEY = $VALUE.EXTERNAL;
   115.          SOURCE.OCC = 1;
   116.          RDBMS_COLUMN = bar_1;
   117.          RDBMS_DATATYPE = VARCHAR2;
   118.          RDBMS_DATALENGTH = 9;
   119.        COLNAME = BAR_2;
   120.          COMMENTS = Multiple;
   121.          COLWIDTH = 9;
   122.          SOURCE.ELEM = BAR;
   123.          DECLARE.KEY = $VALUE.EXTERNAL;
   124.          SOURCE.OCC = 2;
   125.          RDBMS_COLUMN = bar_2;
   126.          RDBMS_DATATYPE = VARCHAR2;
   127.          RDBMS_DATALENGTH = 9;
   128.        COLNAME = BAR_3;
   129.          COMMENTS = Multiple;
   130.          COLWIDTH = 9;
   131.          SOURCE.ELEM = BAR;
   132.          DECLARE.KEY = $VALUE.EXTERNAL;
   133.          SOURCE.OCC = 3;
   134.          RDBMS_COLUMN = bar_3;
   135.          RDBMS_DATATYPE = VARCHAR2;
   136.          RDBMS_DATALENGTH = 9;
   137.        FILE = GQ.DOC.RESTAURANT;
   138.        TABLE.NUM = 001;
   139.        RDBMS_TABLENAME = restaurant;
   140.      EndDeclare
   141.
   142.      Declare Table LOCATION
   143.        ID = GQ.WCK.LOCATION;
   144.        COMMENTS = Table Generation for Structure LOCATION Elements;
   145.        DEFDATE = WED. APRIL 9, 2003;
   146.        MODDATE = WED. APRIL 9, 2003;
   147.        MODTIME = 10:13:56;
   148.        SUBFILE.NAME = RESTAURANT;
   149.        DECLARE.SUBFILE = Data Move Declares;
   150.        COLNAME = ID;
   151.          COMMENTS = REF RESTAURANT;
   152.          ISKEY;
   153.          COLTYPE = INT;
   154.          COLWIDTH = 4;
   155.          COLUMN.OPTIONS = Dependent, NoSubtree;
   156.          RDBMS_COLUMN = id;
   157.          RDBMS_DATATYPE = NUMBER;
   158.          RDBMS_DATALENGTH = 4;
   159.        COLNAME = CITY;
   160.          ISKEY;
   161.          COLWIDTH = 20;
   162.          RDBMS_COLUMN = city;
   163.          RDBMS_DATATYPE = VARCHAR2;
   164.          RDBMS_DATALENGTH = 20;
   165.        COLNAME = STATE;
   166.          COMMENTS = Required;
   167.          COLWIDTH = 2;
   168.          RDBMS_COLUMN = state;
   169.          RDBMS_DATATYPE = VARCHAR2;
   170.          RDBMS_DATALENGTH = 2;
   171.        COLNAME = PHONE;
   172.          COLWIDTH = 15;
   173.          RDBMS_COLUMN = phone;
   174.          RDBMS_DATATYPE = VARCHAR2;
   175.          RDBMS_DATALENGTH = 15;
   176.        COLNAME = ADDRESS;
   177.          COLWIDTH = 41;
   178.          RDBMS_COLUMN = address;
   179.          RDBMS_DATATYPE = VARCHAR2;
   180.          RDBMS_DATALENGTH = 41;
   181.        COLNAME = ZIP-CODE;
   182.          COMMENTS = Multiple;
   183.          COLWIDTH = 5;
   184.          SOURCE.SINGLE;
   185.          RDBMS_COLUMN = zip_code;
   186.          RDBMS_DATATYPE = VARCHAR2;
   187.          RDBMS_DATALENGTH = 5;
   188.        FILE = GQ.DOC.RESTAURANT;
   189.        SOURCE.STRUCTURE = LOCATION;
   190.        TABLE.NUM = 002;
   191.        RDBMS_TABLENAME = location;
   192.      EndDeclare
   193.
   194.      Declare Table RECOMMENDER
   195.        ID = GQ.WCK.RECOMMENDER;
   196.        COMMENTS = Table Generation for Element RECOMMENDER;
   197.        DEFDATE = WED. APRIL 9, 2003;
   198.        MODDATE = WED. APRIL 9, 2003;
   199.        MODTIME = 10:13:56;
   200.        SUBFILE.NAME = RESTAURANT;
   201.        DECLARE.SUBFILE = Data Move Declares;
   202.        COLNAME = ID;
   203.          COMMENTS = REF RESTAURANT;
   204.          ISKEY;
   205.          COLTYPE = INT;
   206.          COLWIDTH = 4;
   207.          RDBMS_COLUMN = id;
   208.          RDBMS_DATATYPE = NUMBER;
   209.          RDBMS_DATALENGTH = 4;
   210.        COLNAME = RECOMMENDER.OCC;
   211.          COMMENTS = Occurrence Number;
   212.          ISKEY;
   213.          COLTYPE = INT;
   214.          SOURCE.ELEM = RECOMMENDER;
   215.          DECLARE.KEY = $VALUE.OCC;
   216.          SOURCE.SINGLE;
   217.          RDBMS_COLUMN = recommender_occ;
   218.          RDBMS_DATATYPE = NUMBER;
   219.          RDBMS_DATALENGTH = 4;
   220.        COLNAME = RECOMMENDER;
   221.          COMMENTS = Multiple;
   222.          COLWIDTH = 83;
   223.          RDBMS_COLUMN = recommender;
   224.          RDBMS_DATATYPE = VARCHAR2;
   225.          RDBMS_DATALENGTH = 83;
   226.        FILE = GQ.DOC.RESTAURANT;
   227.        MULTI.ELEMENT = RECOMMENDER;
   228.        TABLE.NUM = 003;
   229.        RDBMS_TABLENAME = recommender;
   230.      EndDeclare
   231.
 >

5.8  Perform Data Move Command

The SPIRES Data Move process has been implemented as a PERFORM command which has the advantage of simplifying the input to the original process. The command syntax is as follows:

To learn more about this feature [EXPLAIN DATA MOVE PROCESSING.]

6  File Management Commands: Reference

6.0  (File Management Commands: Reference)

This chapter presents detailed descriptions of the commands that handle the file management tasks discussed in the earlier part of this manual. The commands appear alphabetically, and each begins with a standard set of information:

 - a brief statement defining the purpose of this command
 - which SPIRES programs the command is available in (SPIRES, SPIBILD, FASTBILD)
 - related commands, i.e., commands often used in conjunction with this command, or  discussed
 in the same breath
 - "recovery commands", i.e., commands that reverse or undo what this  command  does
 - procedures described earlier in the manual that use the command, listed  with  the  section
 number where they can be found

Following that list is a formal description of the command's syntax and a general description of the command. Most of the descriptions also include one or more examples showing how the command is used.

6.0.1  (ATTACH)

The ATTACH command "attaches" a record-type of the named file for you to use as if it were the goal record-type of a selected subfile.

Syntax

General information

Issue the ATTACH command when you want or need to use a record-type in a SPIRES file without going through the subfile-selection process, e.g., to use a record-type of the file for which no subfile has been defined, or to avoid the restrictions or select-commands for a subfile. In general, both ATTACH and SELECT have the same net effect, except that ATTACH does not involve the qualifications listed in the Subfile section of the file definition.

Example

Here is an example using the ATTACH command:

At this point, record-type ZIN04, the fourth record-type in the file, can be used almost as if it were a "selected" goal record-type. The following commands, for example, could be issued in SPIRES against the attached record-type:

But on the other hand, when a file is attached, the SPIRES variable $SELECT (the name of the selected subfile) is null, and $SELECTED (a flag indicating whether a subfile is selected or not) is "0" (NO). [Note: The cited examples are not an exhaustive list of what does and does not work with the ATTACH command.]

The ATTACH command is most frequently used when you want to examine records in an index record-type (e.g., get a list of all the indexed values). If the index record-type is defined as the goal record-type of another subfile, you could SELECT the subfile in order to examine the records. But in most cases, the index record-type is not a goal record-type of a subfile, so it cannot be directly accessed through a SELECT command. Instead, you must use the ATTACH command. [See 5.4.]

6.0.2  (AVSPREC)

The AVSPREC command (for AVailable SPace RECovery) rebuilds the available-space chains for the Residual data set.

Syntax

General information

SPIRES keeps track of the amount of unused space in each block of the Residual data set. SPIBILD creates a "chain" of the blocks having 64 bytes free, another chain of those having 128 bytes free, etc. Each block in each chain points to the next block in the chain. SPIBILD uses the chains when determining where to store a new record, or where to put the continuation of an updated record when there is not enough space in the record's original block.

When any of the records in a block are updated or removed, the amount of available space in the block is likely to change. However, SPIBILD can move the block to the correct chain only if it is the first block on the chain, in which case it is moved to become the first block of the correct chain. If the block is not the first block in the chain, SPIBILD cannot put it on the correct chain, so it becomes a "block on the wrong chain".

For a block to be on the wrong chain is not a terrible problem, by any means; it does not damage your file in any way. It simply means that SPIBILD may not be able to store new and updated records as efficiently as possible. For example, if a block is on the 640-byte available-space chain but record updates and removes have created 2000 bytes of available space in it, that extra space is likely to remain unused for a long time, if not forever. [On its own, SPIBILD may recover the unused space, but in many cases, that would be unlikely.]

The AVSPREC command examines each block of each chain, rebuilding the chains, moving the blocks on wrong chains onto the right ones. By moving blocks with more available space than the chain they are on indicates, SPIBILD will more likely use the space in the future. [See 3.4.3.]

Example

Here is an example showing how to use the AVSPREC command:

The AVSPREC command may take some time, since it must read all and usually write many of the blocks in the Residual, typically the largest data set in a SPIRES file.

SPIRES will place a list of the chains and the number of blocks on each in your active file. It will match the list of chains and "Actual blocks" shown in the AVSP display of the STATUS command.

Problems and recovery

If the system crashes before the AVSPREC command finishes its work, you should restart the operation as soon as convenient by using the CONTINUE option, e.g., AVSPREC CONTINUE. However, if the file gets processed before the AVSPREC operation is restarted, no harm will be done.

6.0.4  (BUILD)

The BUILD command passes data from goal records of the established subfile to one or more indexes.

Syntax

If no occurrence is specified, all linkage specifications for the named index are used by SPIRES.

General information

The BUILD command is most commonly used in these four situations:

 - when the file owner decides to add a new index to a subfile;
 - when the file owner  decides  to  pass  another  element's  value  to  an  existing  index;
 - when data integrity problems with an index occur, and the index must be zapped and rebuilt.
 - after a FAST BATCH command, to build the subfile's indexes.

In SPIBILD under Global FOR

If you use the command under Global FOR in SPIBILD, you can build one or more than one index with a single command. Before using the BUILD command, issue the ESTABLISH command to select the desired subfile. Then issue the FOR SUBFILE command, with or without a WHERE clause and any SET SCAN commands desired, to tell SPIBILD which records to use in building the index. You are then ready to issue the BUILD command. [You can also use BUILD under FOR STORED, where the records to be used are specified in a stored stack or result.]

In SPIBILD outside of Global FOR and in FASTBILD

BUILD can be used to build a single index in FASTBILD and in SPIBILD without Global FOR. In each case, SPIBILD or FASTBILD examines all the goal records of the subfile to create entries to the index. Note that the procedure in SPIBILD outside of Global FOR is simpler though less versatile than its counterpart in Global FOR: you simply issue the BUILD command, naming the index and subfile, after entering SPIBILD. Be aware that the file's deferred queue must be empty for the command to succeed; if it is not empty, SPIBILD will tell you to process the file before issuing the BUILD command.

Example

Below is a sample SPIBILD session showing the BUILD command used to add more data to an index that already exists. In this case, the file owner originally decided that the data value NONE for the PETS element would not be passed to the PETS index (record-type ZIN03) but has now reversed that decision; records with NONE should be represented in the index.

First, the file owner changes the file definition to allow the value NONE to be passed to the index. Then, the BUILD command is used to pass the appropriate PETS index data for those records with NONE for PETS. The file owner uses a WHERE clause on the FOR SUBFILE command to limit the passed goal records to those with the value NONE.

6.0.5  (CLEAR EXCEPTION)

The CLEAR EXCEPTION command detaches the exception file used during multiple-record input.

Syntax

There are no options on the CLEAR EXCEPTION command.

General information

If you decide not to use an exception file after setting one for a multiple-record input procedure, issue the CLEAR EXCEPTION command to detach it. No exception processing will occur till another SET EXCEPTION command is issued.

As soon as SPIRES completes the execution of a command that uses an exception file (such as BATCH), it automatically detaches the file. If further exception processing is to be done in this SPIRES program session, another SET EXCEPTION command must be issued.

After you issue the CLEAR EXCEPTION command, the ORVYL data set created by the SET EXCEPTION command still exists, even though no data has been put into it. Issue the ORVYL command ERASE followed by the name of the exception file to erase it.

Example

6.0.5a  (CLEAR FILE COUNTS)

The CLEAR FILE COUNTS command turns off the display of file I/O statistics in SPIBILD and resets the counters.

Syntax

There are no options on the CLEAR FILE COUNTS command.

General information

By default, SPIBILD and FASTBILD automatically show file count information as a regular part of their output. The CLEAR FILE COUNTS command can be used to turn the automatic display off. It can be turned on again (with system file count data too) with the SET FILE COUNTS command. Unlike the way it works in SPIRES, however, it does not reset the system file counts.

In SPIRES, since there is no automatic display of file count data, CLEAR FILE COUNTS completely resets the internal tables used by the command.

6.0.5b  (COPY DATA SET)

The COPY DATA SET command copies an ORVYL data set from the attached SPIRES file to another ORVYL data set, possibly as part of another SPIRES file

Syntax

 - CKPT -- the file's checkpoint data set
 - DEFQ -- the file's deferred queue data set
 - MSTR -- the file's Master data set
 - RES, RES1, ... RESn -- one of the file's residual data sets
 - REC1, REC2, ... RECn -- one of the file's record-type data sets

General information

The COPY DATA SET command makes a copy on your account of one ORVYL data set, specified by you, from the attached file. It is very flexible, and you may find many different uses for it. You may use it simply to make a backup copy of one or more data sets before you take some particular file management action, such as rebalancing a tree or creating a non-IXR data set.

Note: File permits are copied only from the source data set to the copy only when the RENAME option is specified.

Example

Here is a sample session showing how you might create a non-IXR DEFQ, hence avoiding the overhead during transactions of requesting file blocks from ORVYL, useful for files with heavy transaction loads:

-> sel restaurant
-> copy data set defq to zzzz rename
Start copy of 2 blocks from data set DEFQ
ORV.GQ.DOC.RESTAURANT.DEFQ is renamed to ORV.GQ.DOC.RESTAURANT.DEFQ.19961201
ORV.GQ.DOC.ZZZZ is renamed to ORV.GQ.DOC.RESTAURANT.DEFQ
File detached
->

The value for "dsname2", ZZZZ in this example, was an arbitrary choice, since it is used only temporarily when the RENAME option is used.

Warning messages

The warning messages you can receive from this command include the following:

6.0.6  (COPY FILE and COPY DEFQ)

The COPY FILE command copies the ORVYL data sets that comprise a SPIRES file from another account to the logged-on account. The COPY DEFQ command is the same as COPY FILE but copies only the file's DEFQ data set.

Syntax

General information

The COPY FILE command makes a copy on your account of each ORVYL data set of the file owner's file. COPY DEFQ makes a copy of only the DEFQ data set of the file owner's file.

The original file and the copy will have the same file name, but with different account prefixes. Note that both commands only make copies; the original file still exists on the file owner's account.

Example

Here is a sample session showing how SPIRES responds to the COPY FILE command:

The "Start copy" message appears as SPIRES starts to copy each data set. It tells you which data set is being copied, as well as how many blocks are in it.

Warning: The COPY FILE and COPY DEFQ commands will fail under the following circumstances:

 - If you (the file copier) do  not  have  COPY  access  to  the  file,  as  declared  in  the
 FILE-PERMITS   structure   of   the   file   definition,   you  will  receive  the  message
 "Privileged  command".   [Accounts  at  a  SPIRES  site  that  have  "master
 privileges"  in  SPIRES (such as the SPIRES consultant) automatically have COPY access
 to all files, except files whose ORVYL-data-set permits have been changed  to  forbid  such
 access.]
 - If the file is being used by another account and the SHARE option has not  been  specified,
 the   command   will   fail   with   an  S56  error  ("File  is  busy,  not  currently
 available.").  This in effect prevents you from copying the file while data within the
 file may be changing.  You should NOT override this situation with the SHARE option  unless
 you  are  positive  that  other  users  are not updating records, or otherwise changing the
 file's contents.
 - If your account does not have enough ORVYL blocks allocated to it, the command  will  fail.
 If  you need more blocks, contact the computer center's accounting office before trying the
 COPY FILE or COPY DEFQ command again.
 - If a file with the same name already exists on your account, the command will fail with  an
 S40 error.

Technical note: The copy of the file is identical to the original file except in two regards:

 - the file-owner account information stored in the MSTR data set is changed  by  SPIRES.   If
 SPIRES  did  not  change  the  account number from the original file owner's to the copying
 account's, the copy could never be recompiled or otherwise linked to a new file definition.
 - the ORVYL data set permits for the copy are set to the SPIRES defaults rather than those of
 the original data sets.  The defaults are READ access to  the  public  for  all  data  sets
 except  the  DEFQ,  which has public WRITE access.  To set the same permits as the original
 file (usually necessary for files with immediate indexing), contact the file owner, who can
 follow the procedure described at the end of the "Move a file"  procedure.   [See
 2.10.]

6.0.7  (ESTABLISH)

The ESTABLISH command moves subfile transactions from the file's deferred queue to the tree, and updates the file's indexes accordingly. It also establishes a subfile to be used by the INPUT commands, or by any merge or remove operations using Global FOR techniques that may follow.

Syntax

General information

The ESTABLISH command, like PROCESS, causes a SPIRES file to be processed. However, it has two major differences from PROCESS:

 - 1) The ESTABLISH command names a subfile whose file is to be processed; PROCESS  names  the
 file directly.
 - 2) ESTABLISH does not "release" the file  so  that  users  can  begin  using  its
 subfiles  again.   ESTABLISH optionally serves as the first part of a procedure to merge or
 remove records in SPIBILD using Global FOR techniques, so it keeps the  file  reserved  for
 your   use   until   you   leave   SPIBILD   or  otherwise  end  the  procedure.   [See  2.1.5.1 for the SET SEARCH/SET SHARE procedure, however.]

Example

Here is an example of the ESTABLISH command in a SPIBILD session:

The output from the command is exactly the same as a PROCESS command would be. For information about the messages, see the explanations under the PROCESS command.

6.0.8  (EXIT)

The EXIT command clears the current ORVYL program from memory, returning you to WYLBUR.

Syntax

General information

EXIT is an ORVYL command that clears the current ORVYL program from memory and detaches all input/output devices (e.g., the ORVYL data sets of selected subfiles and subgoals). It returns you to WYLBUR's control.

You don't need to issue the EXIT command to go from one SPIRES program to another (for instance, from SPIRES to SPIBILD). You can simply issue the desired command from within the processor.

Similarly, to logoff from inside a SPIRES program, you can just issue the LOGOFF command; you don't have to EXIT the program first.

Example

The example below shows the EXIT command used in SPIBILD:

6.0.9  (FAST BATCH)

The FAST BATCH command adds multiple records to a subfile very quickly, but without checkpointing or indexing the data.

Syntax

General information

You can use the FAST BATCH command in SPIBILD to add multiple records in a single "batch" using the standard SPIRES format or a customized input format. (Remember: Be sure to set the format before issuing the FAST BATCH command.) [See 2.3.4.]

FAST BATCH is much less versatile than SPIBILD's BATCH command. In fact, there are several important limitations and restrictions that you should know before using FAST BATCH:

The workings of the FAST BATCH command

It's usually best to process the file first using the PROCESS/ESTABLISH procedure so that the deferred queue will be empty when you start. [See 2.2.2.] Then issue the SET NOPASS command before issuing the FAST BATCH command. (FAST BATCH would also process the file, but you generally don't want SET NOPASS in effect when you are processing the deferred queue prior to the primary work of the FAST BATCH command.)

Next SPIBILD begins to process the input data. Unless you are using batch SPIBILD with the "ddname" option, SPIBILD reads the input data from your active file, so the data should be there before you issue the BATCH command. SPIBILD will display messages for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]

The processing is quite straightforward -- the data goes directly into the tree without checkpointing or indexing, which speeds the processing considerably. But remember, if a system problem occurs while the records are being added, you will need to zap the record-type and start again.

Example

Here is a sample session demonstrating the FAST BATCH command in SPIBILD. The file is explicitly processed before setting NOPASS in case there is data already in the deferred queue that should be processed.

6.0.9a  (FAST LOAD)

The FAST LOAD command adds multiple records to a subfile very quickly, using data already in SPIRES internal format, but without checkpointing or indexing the data.

Syntax

General information

The FAST LOAD command takes the data created by the GENERATE LOAD command and loads it into a SPIRES file. It begins by processing the file, and then adds the records in the load to the specified record-type one by one.

The record-type must have the same characteristics as the one from which the data came. For example, it must have the same fixed elements, with the same lengths and occurrences; the same structures, having the same number and type of elements, etc. [See the SPIRES manual "File Definition" for a discussion of "matching record-type characteristics".]

Because the input data is already in the internal format used by SPIRES to store records, the data is not processed through any processing rules; Inprocs and Incloses are ignored. The only data that can possibly change as the data is added is the record key, if they are slot keys, as requested by the RENUMBER option.

FAST LOAD is much less versatile than SPIBILD's LOAD command. In fact, there are several important limitations and restrictions that you should know before using FAST LOAD:

The workings of the FAST LOAD command

It's usually best to process the file first using the PROCESS/ESTABLISH procedure so that the deferred queue will be empty when you start. [See 2.2.2.] Then issue the SET NOPASS command before issuing the FAST LOAD command. (FAST LOAD would also process the file, but you generally don't want SET NOPASS in effect when you are processing the deferred queue prior to the primary work of the FAST LOAD command.)

Next SPIBILD begins to process the input data. SPIBILD will display messages for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]

The processing is quite straightforward -- the data goes directly into the tree without checkpointing or indexing, which speeds the processing considerably. But remember, if a system problem occurs while the records are being added, you will need to zap the record-type and start again.

Example

Here is a sample session demonstrating the FAST LOAD command in SPIBILD. The file is explicitly processed before setting NOPASS in case there is data already in the deferred queue that should be processed.

6.1.0  (FIX BASE)

The FIX BASE command lets you reset the base-block data for the record-types of a data set in the MSTR data set of the currently attached file.

Syntax

General information

This command, primarily for SPIRES consultants, is for use when the base block information stored in the MSTR data set of the attached file does not match the actual base block data in the "RECn" data set. This rare problem most commonly arises when a STATUS command indicates errors in a just-rebalanced RECn data set -- you must return to the old version of the RECn data set and reset the base-block data with the old data. (That's why the procedure includes saving the old base-block data shown in the SHOW FILE ACTIVITY command before rebalancing the data set.)

The FIX BASE command shows you what the MSTR data set currently has for the base block number of each record-type in the data set, and asks you for the "new" number. When the base blocks of all the record-types have been entered, SPIRES replaces the old values in the MSTR with the new ones. In other words, unless the command runs to its conclusion, the base-block data won't be changed in the MSTR. [See 3.3.2.]

Example

Here is an sample session showing the FIX BASE command:

If you don't know the correct base block numbers, it is possible to track them down by using the SHOW FILE BLOCK command to find a block from each record-type in the data set and trace each one back to its base block.

6.1.0b  (FIX NEXT BLOCK)

The FIX NEXT BLOCK command resets the number representing the "next block" in a tree data set or the residual data set.

Syntax

General Information

This command, primarily for SPIRES consultants, is useful to data base administrators when problems arise during a data-set rebalance. [See 3.3.2.] Like other FIX commands, it can cause serious problems if not used carefully; if you believe you need to use it for reasons other than the rebalancing situation described earlier in this manual, please check with your SPIRES consultant before using it.

Information about what the proper setting should be for a given data set is usually provided by the SHOW FILE ACTIVITY command.

Example

Here is a sample session showing what the command looks like when used:

6.1.0c  (FIX MAXIMUM BLOCKS)

The FIX MAXIMUM BLOCKS command resets the maximum number of blocks that may be utilized in an extended tree data set.

Syntax

General Information

This command, primarily for SPIRES consultants, enables administrators to extend the maximum size of a tree data set which is approaching its default limit. When you issue this command SPIRES will prompt you for the new maximum block size value as shown in the example below. This command can only be used for extended tree data sets which have a default size limit of 434,280 blocks. If the data set is not an extended tree data set and is approaching its default size limit of 65,536 blocks then you should ignore this command and instead convert the data set to extended tree format. [See 5.6.] The maximum size value is restricted. The maximum size value cannot be set to more than three times the default maximum and cannot be set to less than the default maximum (greater than 434,280 and less than 1,302,840).

If you rebalance an extended tree data set the rebalanced data set will retain the max size of the original. [See 6.1.9.]

Information about the current size of a file's data sets is provided by the SHOW FILE LIMITS command. [See 6.3.4a.]

Example

Here is a sample session showing what the command looks like when used:

6.1.1  (FIX REBALANCED DATA SET)

The FIX REBALANCED DATA SET command registers the new base-block information from a rebalanced data set in the MSTR data set of the currently attached file.

Syntax

General information

The FIX REBALANCED DATA SET command is designed for the recovery procedure if the REBALANCE DATA SET command with the RENAME option does not complete successfully.

The procedure describing its use appears earlier in this manual. [See 3.3.2.]

Example

Below is an example of the FIX REBALANCED DATA SET command. The description of the REBALANCE procedure will give you a better sense of the context in which it is used.

6.1.1b  (FIX SUBFILE NODE)

The FIX SUBFILE NODE command adds the node to the tree for a given record in the residual that has been orphaned, or adds a node to a tree to serve as a place-holder for future monotonic tree growth.

Syntax

General information about LCTR

WARNING: These commands are only for use to solve file integrity problems -- careless use can cause more. Seeing your SPIRES consultant before using them is recommended.

The LCTR version of the FIX SUBFILE NODE command is meant for use in situations where a record in the residual somehow loses the node in the tree that points to it. It is the reverse of the ZAP SUBFILE NODE command, which will delete the tree node of a record, though it does not remove the record from the residual.

The "block-trailer" numbers may be available to you in several ways:

 - in any locator for the record that might appear in an index record.  For  most  files,  the
 first 3 bytes of the locator (in hex) specify the block number, and the last byte specifies
 the  trailer.  For example, if the locator is shown as "00002803", then the block
 number is 40 ("28" hex) and the trailer is 3 ("3" hex).
 - in the information returned from the ZAP SUBFILE NODE command will be the block number  and
 the  record  key.   Using the SHOW SUBFILE RESIDUAL BLOCK command, you can find the trailer
 number for that key in the specified block.

Example

In the example below, you need to restore a tree node for record 138, which you tracked down to residual block 40:

General information about SEED

Monotonic record-types, that is, record-types where new records always have keys higher in value than records already in the file, tend to need rebalancing more frequently than normal record-types. The FIX SUBFILE NODE SEED command assigns a specific key not yet in the record-type to serve as a starting point for a future subtree. Aside from its use as a marker, the new node that SPIRES creates in the tree's base block has no effect. If the key values are chosen properly, you may never need to rebalance the tree again. (Remember: talk with your SPIRES consultant before using this command.)

The command should be issued after the record-type has been rebalanced. [See 3.3.1.] Select the subfile for which the record-type is the goal, or else ATTACH the record-type. One or more FIX SUBFILE NODE SEED commands may be issued, each one assigning a key higher than any existing in the file:

You might then issue the SHOW SUBFILE BLOCK BASE command to see the effect of your change. Near the bottom of the displayed list of nodes in the base block, Look for the "asterisked" key values you have entered.

This method can be used instead of the START-KEY method, which requires changing the file definition.

6.1.2  (FIX SUBFILE SLOT)

The FIX SUBFILE SLOT command resets the slot number of the current goal record-type.

Syntax

General information

FIX SUBFILE SLOT allows you to reset the next slot number of the selected subfile to:

 - a higher number, which takes effect permanently  and  immediately  for  all  users  of  the
 subfile; or
 - a lower number, which is in effect only for you and only as long as the subfile is selected
 by you.

Warning: Setting the number lower is quite dangerous, since it could allow you to overlay a new record onto an older one that already exists, causing serious problems for your file. [See 5.3.]

When you issue the command, SPIRES will show you the next slot number and then ask you for the new value. Type in the new number, or press the ATTN/BREAK key to cancel the command if you have changed your mind.

Example

Below is an example of the FIX SUBFILE SLOT command:

6.1.2b  (FOR STORED)

The FOR STORED command establishes a Global FOR environment whose class of records is a stored stack or result.

Syntax

General information

In SPIBILD, the REMOVE, MERGE and BUILD commands can be issued only under Global FOR, meaning that they must be preceded by the ESTABLISH command and a FOR SUBFILE or FOR STORED command. The FOR STORED command tells SPIBILD that the records to be examined and, if they pass any WHERE clause criteria, processed are in a stored stack or result, which is named in the FOR STORED command.

Creating a stored result or stack is simply a matter of creating a result or stack in SPIRES (e.g., using a FIND command) and then issuing a "STORE SEARCH.name" or "STORE STACK.name" command.

Not all stored results can be used by the FOR STORED command. Specifically, only results stored from non-structured pointer-groups are allowed; if the list of indexes for the subfile shows qualifiers and sub-indexes, then the subfile has structured pointer-groups.

Because of this limitation, if you don't know whether or not the pointer-groups are structured, or if you don't care to find out either ahead of time or the hard way that they are, you may want to simply create a stored stack rather than a stored result, which will avoid the problem. To do that, in SPIRES, restore or recreate the result, and then issue the two commands FOR RESULT and STACK ALL. Then store the stack.

Under FOR STORED processing in SPIBILD, any consecutive duplicate keys will be ignored (i.e., the record will be processed only once). However, if two copies of the same key are separated by others, that record will be processed twice. That won't be a problem if you are using the REMOVE command, but could be if you are using MERGE or BUILD. If you have a stored stack, it is a good idea to sort it (with the SEQUENCE command) to eliminate duplicate keys. (Stored results are already sorted.)

Example

Here is part of a session showing FOR STORED with the REMOVE command, starting with the creation of a stored stack in SPIRES:

6.1.2c  (GENERATE CHANGE FILE)

The GENERATE CHANGE FILE command creates a SPIRES file containing just the defq and tree data of record updates in the attached file.

Syntax

The PROCESS cycle number exists for SPIRES internal use independently of the GENERATE CHANGE FILE command and cannot be directly controlled by the file owner or user of this command. The advantage of creating your own file name is that you can create your own numbering/naming scheme rather than work from an internal SPIRES number outside of your control. For instance, if your file is named GQ.DOC.INVENTORY, you might want to name the change file "INVENTORY.yyyymmdd" putting the current date in place of "yyyymmdd" each time you create a change file.

General information

GENERATE CHANGE FILE creates a snapshot of all the changes in the selected subfile's deferred queue. Specifically, all the goal-record data in the deferred queue for the selected goal record-type is copied, along with the tree copies of any records marked as updates or removes. The copied data is placed into a new SPIRES "file", a partial copy of the original file's structure, including the compiled characteristics of the goal record-type, a "tree" data set, a residual data set, and a deferred queue. The copied deferred queue data is placed in the defq; the copied tree data goes into the tree and the residual. The file is always created on the file owner's account and cannot be directed elsewhere, even if it is being generated by someone else who has Copy access to the file.

Note: the created file does not have the same ORVYL data set permits that the original has (it has the default permits that all new files start with), and it does not have any of the other record-types of the file that might be related to the goal record-type as indexes or for subgoal lookups.

The purpose of the command is to generate all the data needed to capture all modifications to a subfile prior to file processing. Generally speaking, this command would be coupled to the file-processing procedure, with updates blocked for the duration of the copying and the file processing. This could allow other programs to retrieve the updated data when they wanted to, perhaps to generate the changes in a different manner to apply to other data bases that have the same data. That processing could be done anytime, rather than right before the original file was being processed, which might tie up the original file's processing schedule.

By default, outside of Global FOR, all changed records in the subfile will be processed. If you issue the GENERATE CHANGE FILE command under Global FOR, however, only the records in that class will be processed. For instance, you could generate a change subfile of only the added records by issuing the commands FOR ADDS and GENERATE CHANGE FILE. Some Global FOR classes (SUBFILE, RESULT, STACK for instance) make more sense in this context than others (TREE, REMOVES).

One important exception to normal Global FOR processing worth mentioning here, however, is that GENERATE CHANGE FILE under FOR DEFQ processing will include copying the tree versions of records that are marked as removes in the defq. Normally under FOR DEFQ, removed records are not processed.

Example

Here is an example of GENERATE CHANGE FILE in general use outside of Global FOR, so that all defq data will be processed:

The generated change file can thus be attached and used like a regular SPIRES file. Note that the data from five transactions was copied, but three of them were removes (the three removed records are in the tree of the change file), so only two records, the add and the update, show up under FOR SUBFILE.

If you used the SELECT command to select the original subfile, you can also use it to select the change file:

where "nn" is the PROCESS cycle number that was assigned to the generated change file when it was created (see above).

You could select the generated change subfile of the example above with the command:

The same subfile privileges that you have for the original subfile are given to you for the change file when you select it this way. Similarly, anyone else who can select the original subfile can use this method to select the change file.

Because other record-types of the file are not created, you will receive the warning message "Invalid subgoal in selected subfile" if the goal record-type has A32, same-file subgoal access. The subfile is still selected, however.

The deferred queue data for a given record in the generated change subfile is only the latest copy of the record from the original subfile. Unlike the defq of the original subfile, which retains all copies of a record if it was changed in multiple transactions since the file was last processed, the change subfile's defq has only a single copy of the record. In other words, the purpose of the change subfile is to display the changes that will be applied to the tree of the original subfile when it is next processed; it is not a copy of the transaction log of the original. You can see this point demonstrated by using the SHOW SUBFILE TRANSACTIONS command on both the original and the generated change subfiles.

6.1.2c.1  (GENERATE FILE)

The GENERATE FILE command creates a SPIRES file containing copies of the designated records in the selected subfile.

Syntax

General information

GENERATE FILE creates a copy of the selected subfile's records (as selected by the Global FOR set that must be established before issuing this command), placing them in a new SPIRES file. The file could be used to take a snapshot of a particular set of SPIRES records, leaving them in SPIRES format where they can be examined as SPIRES records (e.g., processed through formats, used with referenced record processing, etc.).

The copied data is placed into a new SPIRES "file", a partial copy of the original file's structure, including the compiled characteristics of the goal record-type, a "tree" data set, a residual data set, and a deferred queue. The copied data is placed into the tree. The file is always created on the file owner's account and cannot be directed elsewhere, even if it is being generated by someone else who has Copy access to the file.

Note: the created file does not have the same ORVYL data set permits that the original has (it has the default permits that all new files start with), and it does not have any of the other record-types of the file that might be related to the goal record-type as indexes or for subgoal lookups.

Important: When generating the file, choose a Global FOR class that will process the records in ascending key order, such as FOR SUBFILE or FOR TREE. If you use FOR RESULT, FOR STACK or FOR SET, make certain that the records are sequenced in key order before processing them. SPIRES uses FASTBILD file services internally to build the file copy.

Example

The generated file can thus be attached and used like a regular SPIRES file.

6.1.3  (GENERATE LOAD)

The GENERATE LOAD command creates an ORVYL data set containing SPIRES records in their internal form, ready for loading directly into another record-type.

Syntax

     FIRST = the first record fitting the criteria
     *     = the current record
     NEXT  = the next record fitting the criteria
     n     = the next "n" records fitting the criteria
     REST  = the rest of the records fitting the criteria
     LAST  = the last record fitting the criteria
     ALL   = all the records fitting the criteria.

General information

GENERATE LOAD, a Global FOR command in SPIRES, creates an ORVYL data set containing SPIRES records in their internal, stored form. Thus, the element values in the records are not processed through their Outproc processing rules. The command's purpose is to create a data set that can later be loaded into the same or a similar record-type, using the LOAD or INPUT LOAD commands.

The data set must be created with a single GENERATE LOAD command; you cannot "build" it using several GENERATE LOAD commands.

See the manual "Sequential Record Processing in SPIRES: Global FOR" for more information about Global FOR in general.

Example

SPIRES replaced the existing RECORDS.FOR.TEST data set in ORVYL with the generated load, consisting of the first 100 records in the subfile. The data set name suggests that the records are being moved from a production file into a test one, a common use of the LOAD and INPUT LOAD procedures.

6.1.4  (GENERATE NONIXR)

The GENERATE NONIXR command creates a residual data set of pre-allocated size to save processing costs during file-building operations.

Syntax

General information

The GENERATE NONIXR command converts the residual data set of the named file from the default "IXR" type to "non-IXR". "IXR" stands for Index Record, specifically, a block at the beginning of an ORVYL data set containing the address of every block in the data set. An IXR data set contains blocks that may be scattered all over the ORVYL file system. A non-IXR data set does not need an IXR because all of its blocks are contiguous.

When you issue the GENERATE NONIXR command, SPIRES allocates a non-IXR data set of the size you specify in the "size" parameter. Then it copies the blocks of the file's residual into the non-IXR data set and then erases the old residual.

Non-IXR residuals are usually created before a file is built in SPIBILD or FASTBILD. It must be done after the file is compiled, however, since it converts an existing residual data set from IXR to non-IXR.

File owners most often use non-IXR residuals for very large files. The file owner determines (by means of smaller test files) how large the residual will be when it is built. When the build is complete, the file owner creates a second residual that is not a non-IXR file to accommodate future growth.

Since the size of the non-IXR residual is fixed, it can neither grow dynamically to accommodate more blocks, nor release blocks it does not need. If necessary, you can issue another GENERATE NONIXR command to create a larger non-IXR residual, but remember that the command must completely recopy the blocks in the old residual into the larger one, which could be very expensive for a large residual.

Example

The returned command prompt indicates the command succeeded. Because GENERATE NONIXR copies your entire residual from one set of blocks to another, it can take a long time to execute.

If the command fails, the two most likely reasons are:

 - S70 error = There were not enough contiguous blocks in the ORVYL file system to handle your
 request.
 - Requested block size not valid = The  number  of  blocks  requested  was  too  large  (over
 425,984)  or  is  smaller  than  the current size of the residual.  You cannot use GENERATE
 NONIXR to make the residual smaller.

Converting any ORVYL Data Set from IXR to Non-IXR

You can use the GENERATE NONIXR command to change any ORVYL data set to non-IXR if you want, because the command does not really care whether the data set is actually a residual.

First, use ORVYL's RENAME command to change the name of the data set so that it ends with ".RES". (That way, SPIRES will think it is a residual data set.) Then issue the GENERATE NONIXR command, using the same name but omitting the ".RES", and finally RENAME the data set again:

The ORVYL command SHOW FILES (with the BLOCKS or ALL option) confirms that the change was made, stating that the data set is non-IXR.

Files with Multiple Residuals

If your file needs multiple RESiduals, be sure to include the "RESIDUAL = n;" statement in your file definition, and then RE/COMPILE that definition to create the extra RESidual data sets. If you already have the RESIDUAL statement, but need more space, then raise the value of "n" and RECOMPILE to add more RESiduals.

Any data sets that are "No content" should be secondary RESiduals. You can convert them to NONIXR type only once, as follows:

Repeat the RENAME, GENERATE, RENAME for as many secondaries as needed. Finally,

It is strongly recommended you leave the last RESidual as IXR type. For example, if you know you'll need at least RES and RES2, then make "RESIDUAL = 3;" to create an empty RES3 for future expansion.

The primary RESidual (filename.RES) may be converted to NONIXR type of any size larger than its current size as long as there are no secondary RESiduals defined (no RESIDUAL statement). You may continue to expand this single RESidual to larger and larger NONIXR size with subsequent applications of the GENERATE NONIXR command, but only as long as there are no secondaries.

Once you define secondary RESiduals, the primary may be converted from IXR to NONIXR type only once, and the size must be 425984. If the primary RESidual is already NONIXR type, it must be expanded to maximum size (425984) only once if it isn't already that size.

6.1.4.1  (INPUT ADD)

The INPUT ADD command adds multiple new records to a subfile via a custom input format.

Syntax

General information

You can use the INPUT ADD command in SPIRES or SPIBILD to add multiple records to a subfile using a customized input format (not the standard SPIRES format; use the INPUT BATCH command in that case).

In SPIRES, the INPUT ADD command follows a SELECT command to choose the subfile into which the data will be batched, and a SET FORMAT command naming the input format to be used. In SPIBILD, it follows a SET FORMAT command, which follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT ADD command to work properly.

When you issue the INPUT ADD command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT ADD command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]

In SPIRES, the new records end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the new records are added directly to the tree, and all indexes are updated.

Example

Here is a sample session demonstrating the INPUT ADD command in SPIBILD:

6.1.4.2  (INPUT ADDMERGE)

The INPUT ADDMERGE command adds multiple new records and/or merges data into existing records of a subfile via a custom input format.

Syntax

If desired, you can abbreviate the command to INP ADDMER.

General information

When you use the INPUT ADDMERGE command in SPIRES or SPIBILD, the program first treats the data for each record as if it were a new record -- in the input format, it executes frames of usage FULL, not MERGE. After the format processing for the record is done, SPIRES checks to see whether the key was provided in the input and if so, whether it matches the key of an existing record. If it does match, the input data is treated as data to be merged into that record; if it does not, the data is treated as a new record.

The data is always processed via a custom input format. ADDMERGE is a command allowed in the standard SPIRES format, used with the INPUT BATCH command and procedure. [See 2.3.1, 2.3.5.] For information about formats for INPUT ADDMERGE, see the manual "SPIRES Formats"; online, EXPLAIN FORMATS, FOR ADDMERGE COMMAND.

In SPIRES, the INPUT ADDMERGE command follows a SELECT command to choose the subfile into which the data will be batched, and a SET FORMAT command naming the input format to be used. In SPIBILD, it follows a SET FORMAT command,which follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT ADDMERGE command to work properly.

When you issue the INPUT ADDMERGE command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT ADDMERGE command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]

In SPIRES, the new and updated records end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the new data goes into the tree, and all indexes are updated.

Example

Here is a sample session demonstrating the INPUT ADDMERGE command in SPIBILD:

6.1.4.3  (INPUT ADDUPDATE)

The INPUT ADDUPDATE command adds multiple new records and/or replaces existing records of a subfile via a custom input format.

Syntax

The command can be abbreviated to INP ADDUPD if desired.

General information

When you use the INPUT ADDUPDATE command in SPIRES or SPIBILD, the program first treats the data for each record as if it were a new record -- in the input format, it executes frames of usage FULL. After the format processing for the record is done, SPIRES checks to see whether the key was provided in the input and if so, whether it matches the key of an existing record. If it does match, the input data is used to replace that record; if it does not, the data is treated as a new record.

The data is always processed via a custom input format. ADDUPDATE is a command allowed in the standard SPIRES format, used with the INPUT BATCH command and procedure. [See 2.3.1, 2.3.5.] For information about formats for INPUT ADDUPDATE, see the manual "SPIRES Formats"; online, EXPLAIN FORMATS, FOR ADDUPDATE COMMAND.

In SPIRES, the INPUT ADDUPDATE command follows a SELECT command to choose the subfile into which the data will be batched, and a SET FORMAT command naming the input format to be used. In SPIBILD, it follows a SET FORMAT command, which follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT ADDUPDATE command to work properly.

When you issue the INPUT ADDUPDATE command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT ADDUPDATE command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]

In SPIRES, the new and updated records end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the new data goes into the tree, and all indexes are updated.

Example

Here is a sample session demonstrating the INPUT ADDUPDATE command in SPIBILD:

6.1.4.4  (INPUT BATCH)

The INPUT BATCH command batches data (adds, updates, merges, or removes) into a subfile.

Syntax

General information

You can use the INPUT BATCH command in SPIRES or SPIBILD to add, update, merge or remove multiple records in a single "batch" using the standard SPIRES format. [See 2.3.5.] Note: If an input format is set, INPUT BATCH behaves like an INPUT ADD command, i.e., all input is treated as records to be added to the file, using the input format.

In SPIRES, the INPUT BATCH command follows a SELECT command to choose the subfile into which the data will be batched. In SPIBILD, it follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT BATCH command to work properly.

When you issue the INPUT BATCH command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT BATCH command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]

In SPIRES, the transactions end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the transactions are applied to the tree, and all indexes are updated.

Example

Here is a sample session demonstrating the INPUT BATCH command in SPIBILD:

The WITH FAST Option on Input

The WITH FAST option, available on the INPUT BATCH command in SPIRES, puts the records into the deferred queue three to four times faster, in terms of processing costs, than the normal INPUT BATCH command. You can use it under these conditions:

 - You must have Master access to the file.
 - The deferred queue must be empty when you issue your WITH FAST request.
 - The keys of the records being added or updated must be in ascending sequence, unless a  key
 matches the key of a record added or updated earlier in the process.  Note that the maximum
 speed  will not be attained unless all records are in ascending sequence.  (If you are only
 adding records, and the goal record-type has a slot key, and SPIRES will be  assigning  the
 keys, then this rule is not relevant.)
 - The file must have a CKPT data set of its own.  (If the file does not, then adding either a
 CHECKPOINT or IMMEDIATE statement to the file definition and recompiling will  create  one.
 See  the manual "SPIRES File Definition" for information about those statements.)
 - You are willing to forego the updating of immediate indexes as part of the  input  process;
 if  the  subfile has immediate indexes, you will have to process the file in SPIBILD to get
 the indexing done, and to  allow  other  users  to  use  the  subfile  for  updates  again.

During the processing, the file will be placed in SET SEARCH mode, so that no one else may update the file. Other users attempting to update with file will get S494 errors. [See 6.2.8, 2.1.5.1.]

If the subfile does not have any immediate indexes, then the subfile will be ready for all to use again upon completion of the command. That is, other users will again be able to update the subfile.

However, if the subfile has any immediate indexes, then SPIRES blocks deferred-queue updates when the INPUT BATCH command completes. (Users will get S560 errors if they try; you can test for this condition with the $DEFQTEST function, using the WRITE type.) This is because no index updates (immediate indexes or not) are done by INPUT BATCH with the WITH FAST option. The file must be processed or the DEFQ zapped before any further update requests will be honored.

The WITH FAST Option on Multi-record output

The WITH FAST option, when used on Multi-record formatted output causes the format processor to retain $RECNO from any prior formatted output command. This allows you to output in bursts, and $RECNO steadly increases. For example, in a protocol you might have:

Setting the format automatically clears $RECNO. The WITH FAST option retains $RECNO, and DISPLAY increments it for each record displayed. So you get records 1-5, a separator line, records 6-10, a separator line, etc.

The WITH SEARCHONLY Option

The WITH SEARCHONLY prefix for INPUT BATCH in SPIRES also speeds up the input processing, though not as much as WITH FAST. However, it does not have the same requirements and restrictions. Basically, during the processing, SPIRES will reserve the file's CKPT (checkpoint) data set, so that it is not reserved and released for each update as normal INPUT BATCH processing is done. (That's why other users are allowed to do updates during regular INPUT BATCH processing.) Like WITH FAST, WITH SEARCHONLY puts the file into SET SEARCH mode, so that other users of the file are limited to searching and reporting. [See 6.2.8, 2.1.5.1.]

Any immediate indexing is done as normal, and the file is available for normal updating activity upon completion of the WITH SEARCHONLY INPUT BATCH command.

WITH SEARCHONLY, like WITH FAST, is available only to users with Master access to the file.

6.1.4.4b  (INPUT LOAD)

The INPUT LOAD command adds records to a subfile using data already in SPIRES internal format, from a data set created by the GENERATE LOAD command.

Syntax

The UPDATE option is blocked Secure Switch 10 in SPIRES only.

General information

The INPUT LOAD command takes the data created by the GENERATE LOAD command and loads it into a SPIRES file. It adds the records in the load to the specified record-type one by one.

The record-type must have the same characteristics as the one from which the data came. For example, it must have the same fixed elements, with the same lengths and occurrences; the same structures, having the same number and type of elements, etc. [See the SPIRES manual "File Definition" for a discussion of "matching record-type characteristics".]

Because the input data is already in the internal format used by SPIRES to store records, the data is not processed through any processing rules; Inprocs and Incloses are ignored. The only data that can possibly change as the data is added is the record key, if they are slot keys, as requested by the RENUMBER option.

Example

Here is an example showing the command's use in online SPIRES:

In SPIBILD

The INPUT LOAD command in SPIBILD works almost exactly the same as the LOAD command. The only significant difference is that LOAD begins by processing the file, whereas with INPUT LOAD, an ESTABLISH command must be issued to process the file first. INPUT LOAD hence requires you to "select" the goal record-type by subfile name (through the ESTABLISH command); on the LOAD command, you name the file and goal record-type explicitly.

An example of the INPUT LOAD command in SPIBILD appears earlier in this manual. [See 2.3.3a.]

The WITH FAST Option

The WITH FAST option, available on the INPUT LOAD command (as well as INPUT BATCH) in SPIRES, puts the records into the deferred queue three to four times faster, in terms of processing costs, than the normal INPUT LOAD command. You can use it under these conditions:

 - You must have Master access to the file.
 - The deferred queue must be empty when you issue your WITH FAST request.
 - The keys of the records being added or updated must be in ascending sequence, unless a  key
 matches the key of a record added or updated earlier in the process.  Note that the maximum
 speed  will not be attained unless all records are in ascending sequence.  (If you are only
 adding records, and the goal record-type has a slot key, and SPIRES will be  assigning  the
 keys, then this rule is not relevant.)
 - The file must have a CKPT data set of its own.  (If the file does not, then adding either a
 CHECKPOINT or IMMEDIATE statement to the file definition and recompiling will  create  one.
 See  the manual "SPIRES File Definition" for information about those statements.)
 - You are willing to forego the updating of immediate indexes as part of the  input  process;
 if  the  subfile has immediate indexes, you will have to process the file in SPIBILD to get
 the indexing done, and to  allow  other  users  to  use  the  subfile  for  updates  again.

During the processing, the file will be placed in SET SEARCH mode, so that no one else may update the file. Other users attempting to update with file will get S494 errors. [See 6.2.8, 2.1.5.1.]

If the subfile does not have any immediate indexes, then the subfile will be ready for all to use again upon completion of the command. That is, other users will again be able to update the subfile.

However, if the subfile has any immediate indexes, then SPIRES blocks deferred-queue updates when the INPUT LOAD command completes. (Users will get S560 errors if they try; you can test for this condition with the $DEFQTEST function, using the WRITE type.) This is because no index updates (immediate indexes or not) are done by INPUT LOAD with the WITH FAST option. The file must be processed or the DEFQ zapped before any further update requests will be honored.

The WITH SEARCHONLY Option

The WITH SEARCHONLY prefix for INPUT LOAD in SPIRES also speeds up the input processing, though not as much as WITH FAST. However, it does not have the same requirements and restrictions. Basically, during the processing, SPIRES will reserve the file's CKPT (checkpoint) data set, so that it is not reserved and released for each update as normal INPUT LOAD processing is done. (That's why other users are allowed to do updates during regular INPUT LOAD processing.) Like WITH FAST, WITH SEARCHONLY puts the file into SET SEARCH mode, so that other users of the file are limited to searching and reporting. [See 6.2.8, 2.1.5.1.]

Any immediate indexing is done as normal, and the file is available for normal updating activity upon completion of the WITH SEARCHONLY INPUT LOAD command.

WITH SEARCHONLY, like WITH FAST, is available only to users with Master access to the file.

6.1.4.5  (INPUT MERGE)

The INPUT MERGE command merges data into existing records of a subfile via a custom input format.

Syntax

General information

When you use the INPUT MERGE command in SPIRES or SPIBILD, the data is read through an input format, executing frames of usage MERGE. The key for each record to be updated is almost invariably found in the input data; when the data for each record is processed by the format, it is merged into the existing record.

The data is always processed via a custom input format. MERGE is a command allowed in the standard SPIRES format, used with the INPUT BATCH command and procedure. [See 2.3.1, 2.3.5.] For information about formats for INPUT MERGE, see the manual "SPIRES Formats"; online, EXPLAIN FORMATS, FOR INPUT MERGE COMMAND.

In SPIRES, the INPUT MERGE command follows a SELECT command to choose the subfile into which the data will be merged and a SET FORMAT command naming the input format to be used. In SPIBILD, it follows a SET FORMAT command, which follows an ESTABLISH command, which not only names the subfile but also processes the file containing it, because in SPIBILD the deferred queue must be empty for the INPUT MERGE command to work properly.

When you issue the INPUT MERGE command, SPIRES or SPIBILD begins to process the input data. Unless you are using BATSPI or batch SPIBILD with the "ddname" option, the input data is read from your active file, so the data should be there before you issue the INPUT MERGE command. Messages will be displayed for each record as it is processed; if you want, you can suppress these messages using the SET MESSAGES command. [See 2.1.5.3.]

In SPIRES, the updated records end up in the deferred queue; any immediate indexes are updated appropriately, but regular indexes are not. In SPIBILD, the new data goes into the tree, and all indexes are updated.

Example

Here is a sample session demonstrating the INPUT MERGE command in SPIBILD:

6.1.5  (LOAD)

The LOAD command adds records to a subfile using data already in SPIRES internal format, from a data set created by the GENERATE LOAD command.

Syntax

General information

The LOAD command takes the data created by the GENERATE LOAD command and loads it into a SPIRES file. It begins by processing the file, and then adds the records in the load to the specified record-type one by one.

The record-type must have the same characteristics as the one from which the data came. For example, it must have the same fixed elements, with the same lengths and occurrences; the same structures, having the same number and type of elements, etc. [See the SPIRES manual "File Definition" for a discussion of "matching record-type characteristics".]

Because the input data is already in the internal format used by SPIRES to store records, the data is not processed through any processing rules; Inprocs and Incloses are ignored. The only data that can possibly change as the data is added is the record key, if they are slot keys, as requested by the RENUMBER option.

Example

Here is an example showing the command's use in online SPIBILD:

6.1.6  (MERGE, in SPIBILD)

The MERGE command in SPIBILD merges data into multiple records in a subfile under Global FOR.

Syntax

General information

You can use the MERGE command to merge data into multiple records with a single command. [See 2.4.] It is used under Global FOR, which determines the records to be updated, usually based on criteria specified in a WHERE clause or SET FILTER commands. The input data may be provided from a data set or from an input format. The MERGE command must be preceded by ESTABLISH and FOR SUBFILE or FOR STORED commands. [See 2.4.2.]

Whether a merged record gets re-indexed depends on which elements were updated. SPIBILD checks the elements of the merged data against a list of elements (the "pass elements list") constructed by SPIRES when the file was compiled. If any elements from the list are updated, then the updated record goes into the deferred queue so that indexes can be updated; otherwise, the updated record is placed directly into the tree, and no superfluous re-indexing occurs. [See 2.4.3.3.]

Example

Here is a sample session demonstrating the MERGE command in SPIBILD under Global FOR. Note that the format execution begins by asking for the data that it will apply to all the records ("What percent markup...").

6.1.7  (PERFORM USER LIST)

The PERFORM USER LIST command tells you what users currently have selected any subfiles that have subfile logging for a given file.

Syntax

General information

PERFORM USER LIST produces a list of accounts that may have any subfile selected of the file holding your currently selected subfile. It relies on command logging being in effect for the file's subfiles (at least to the level of LOG = 1).

It can be useful when you want to process or recompile a file but want to know who, if anyone, might have subfiles of the file selected. For processing the file, you can SET SHARE or SET SEARCH, and to recompile the file, you can include the SHARE option on the RECOMPILE command in order to do your work while users have subfiles of the file selected. However, for various reasons, you may want to know whom you may be affecting by going ahead with your work.

Under some conditions, the display from this command may not contain all the accounts that have the subfile selected. For example, if a user has selected a subfile of the file and then the file is processed under SET SEARCH, that user would not then show up in the PERFORM USER LIST display as a current user of the subfile. The log entry in which the user's initial selection of the subfile was recorded will have been erased from the current log by the file processing.

Example

The current user, GQ.JNK, selects the subfile and discovers someone else has a subfile of the file selected. He then tells his colleague he plans to process the file.

6.1.8  (PROCESS)

The PROCESS command moves subfile transactions from the file's deferred queue to the tree and updates the file's indexes accordingly.

Syntax

General information

The PROCESS command has these specific tasks:

 - apply record transactions in the deferred queue to the tree:
 - add newly added records to the tree;
 - replace old copies of  records  in  the  tree  with  new  versions  of  updated  records;
 - discard records in the tree that have been removed.
 - update any non-immediate indexes, synchronizing them with the record  transactions  in  the
 deferred queue.
 - if a subfile log is kept, either discard its contents, or add them to the archive data  set
 called "SYSTEM.ULOG".

WARNING: The PROCESS command discards any batch requests submitted by users via the BATCH command in SPIRES. Those requests can be handled only by JOBGEN-created processing jobs. You may want to issue the SHOW BATCH REQUESTS command in SPIRES before processing the file, if users might have submitted batch requests.

When the processing is completed, the DEFQ data set for the file will be practically empty -- it should be two blocks large (or whatever size the file definition statement QUELEVEL specifies). Extra blocks added to the DEFQ as it grew will be released during file processing.

Example

Here is a sample SPIBILD session demonstrating the PROCESS command:

In SPIRES, you must precede the PROCESS command by a SELECT command, to select a subfile of the file you want to process. (Note from the syntax that you do not include the file name in the PROCESS command in SPIRES.)

SPIBILD and SPIRES process the record-types in the order in which they are defined in the file definition (alphabetic order). Thus, the REC01 records are processed before REC19 records, for example.

Processing Messages

Here are descriptions of each processing message shown in the example above:

6.1.8a  (PROCESS SUBFILE)

The PROCESS SUBFILE command moves transactions for the selected subfile from the file's deferred queue to the tree and updates the file's indexes accordingly.

Syntax

General information

The PROCESS SUBFILE command has these specific tasks:

 - apply record transactions for only the selected subfile in the deferred queue to the  tree:
 - add newly added records to the tree;
 - replace old copies of  records  in  the  tree  with  new  versions  of  updated  records;
 - discard records in the tree that have been removed.
 - update any non-immediate indexes, synchronizing them with the record  transactions  in  the
 deferred queue.
 - delete those transactions from the deferred queue.

If there are transactions in the deferred queue for other record-types, or if the subfile-processing creates other transactions (goal-to-goal passing), those transactions are ignored, and will not get processed till the next full-file process. If there are no other transactions and no log data, then the deferred queue is re-initialized as if a regular full-file process had taken place.

If a problem occurs that prevents the process from completing successfully, the NOJOBGEN flag will get set, meaning that record updates to any record-type in the file will fail until file recovery takes place. [See 2.1.4.2.]

Note: Unlike the PROCESS command, the PROCESS SUBFILE command does not discard any log data or batch requests submitted by users via the BATCH command in SPIRES.

Example

Here is a sample session demonstrating the PROCESS SUBFILE command:

In SPIRES, you must precede the PROCESS SUBFILE command by a SELECT command, to select the subfile you want to process.

Processing Messages

Descriptions of each processing message shown in the example above appear in the section on the PROCESS command, a page or two back. Online, EXPLAIN PROCESS COMMAND.

6.1.9  (REBALANCE DATA SET)

The REBALANCE DATA SET command reorganizes the contents of a tree-structured data set to prevent it from becoming too deep and/or to improve the average number of disk I/Os per record.

Syntax

General information

The REBALANCE DATA SET command is the main command of the rebalancing procedure, used to improve the structure of record-types that have become "unbalanced", i.e., the tree's branches are very uneven, resulting in inefficient retrieval of many records. The command creates a new version of a data set containing tree-structured record-types, with the records in each record-type more efficiently arranged. [See 3.3.]

If you specify the RENAME option, the command replaces the old data set with the newly rebalanced one, resetting all the base blocks for each record-type. If you omit the RENAME option, you will have to complete that part of the procedure using either the FIX REBALANCED DATA SET command or "by hand" with the RENAME, SET CLP and FIX BASE commands.

Example

A complete example of the procedure for rebalancing data sets appears earlier in the manual. The example below shows only the REBALANCE DATA SET command and its output, which is just one part of the complete procedure. [See 3.3.1.]

In addition, SPIRES will display a message each time 64K nodes have been processed, to reassure you (especially if the record-types are large) that the rebalancing work is occurring.

See the discussion about "Handling Problems" in the procedural description if you have any problems. [See 3.3.1, 3.3.2.]

Rebalancing extremely large data sets

Technical Note: There is a REBALANCE RECORD-TYPE command available to very large files that cannot rebalance an entire data set in a single session. Contact your SPIRES consultant if you think you need to use that command.

6.1.9a  (REBALANCE DEFQ)

The REBALANCE DEFQ command rebuilds a file's DEFQ and rebalances the DEFQ's tree.

Syntax

Example

A complete example of the procedure for rebalancing a deferred queue appears earlier in the manual. The example below shows only the REBALANCE DEFQ command and its output, which is just one part of the complete procedure. [See 3.3.3.]

6.2.0  (RECOVER)

The RECOVER command, primarily used to clean up after a SPIBILD command fails, partially or completely processes a SPIRES file.

Syntax

Syntax note: The RECOVER command cannot be abbreviated.

General information

The RECOVER command can be considered a "mini-PROCESS" command. Its main purpose is to get a file with the NO JOBGEN flag set into a stable state such that normal file activities can resume. Depending on how much data is in the deferred queue, it may or may not process the file completely; however, if all goes well, it will at least stabilize the deferred queue and reset the JOBGEN flag. (Technically speaking, RECOVER will create and process up to two pass stacks worth of index data and then stop.)

If RECOVER processes the entire deferred queue, then you will be left in SPIBILD at the end of the processing. But if data remains in the deferred queue at the end of the RECOVER work, SPIBILD will issue an automatic EXIT command, taking you out of SPIBILD.

In most cases when you are trying to recover from a SPIBILD command failure, the PROCESS command can be used instead. However, when you are working with a file that has many transactions in the deferred queue, a complete process of the file may take too long or cost too much (at daytime rates).

If the RECOVER command fails and you get error messages, you may have a serious problem with your file. Use the EXPLAIN command to get more information about the errors, and contact your SPIRES consultant if you need further help.

Example

Here is a sample SPIBILD session demonstrating the RECOVER command:

The data shown is similar to the display provided by the PROCESS command.

6.2.1  (REMOVE, in SPIBILD)

The REMOVE command in SPIBILD removes records from a subfile as determined by the established Global FOR environment.

Syntax

General information

In SPIBILD, the REMOVE command can be issued only under Global FOR, meaning that it must be preceded by the ESTABLISH command and a FOR SUBFILE or FOR TREE command. [See 2.5.1.]

As it executes, it does not immediately remove the records. Instead, it places removal requests in the deferred queue, just as the REMOVE command does in SPIRES. The difference is that when SPIBILD is finished processing the removes, it processes the deferred queue, causing the records to be removed from the file and the indexes to be updated.

Example

Here is part of a sample SPIBILD session using the REMOVE command. The ESTABLISH command to process the file and "select" the subfile has already been issued.

6.2.2  (SET AUTOGEN and SET NOAUTOGEN)

The SET AUTOGEN command turns on automatic processing of the file by JOBGEN on a one-time basis. The SET NOAUTOGEN command turns off JOBGEN processing of the file.

Syntax

There are no options on either command.

General information

The AUTOGEN flag determines whether a file is to be processed by JOBGEN, the program that automatically submits batch SPIBILD jobs overnight for files with data in their deferred queues. The basic setting for any file is determined by the file definition: if the file definition contains a NOAUTOGEN statement, then the basic setting for that file is NOAUTOGEN. Otherwise, it is AUTOGEN.

By issuing the SET AUTOGEN or SET NOAUTOGEN command, you can change the setting temporarily. The change is temporary in the sense that the next time the file is processed in SPIBILD (either via JOBGEN or manually), the flag will revert back to its basic setting.

Hence, if the basic setting for a file is NOAUTOGEN and you issue the SET AUTOGEN command, then beginning that evening, JOBGEN will check the deferred queue of the file for data and if so, will submit a batch SPIBILD job to process the file. When the file gets processed, the flag will revert to NOAUTOGEN. Notice that if there is no data in the deferred queue, no batch SPIBILD job will be submitted, so the file will not get processed, and the flag will remain set as AUTOGEN.

On the other hand, if the basic setting for a file is AUTOGEN and you issue the SET NOAUTOGEN command, JOBGEN will ignore the file until the flag gets reset to AUTOGEN, which would happen if you issued a SET AUTOGEN command or if you processed the file yourself in SPIBILD.

Example

To set the AUTOGEN flag for a file, you must first select one of its subfiles in SPIRES:

Remember that anytime the file is processed, the AUTOGEN flag reverts to its basic setting. Thus, if a file has a basic setting of NOAUTOGEN and you SET AUTOGEN, then if you (or someone else) process the file yourself before overnight processing occurs, the flag will be reset to NOAUTOGEN, and no JOBGEN processing will occur till you set the AUTOGEN flag again.

6.2.3  (SET EXCEPTION)

The SET EXCEPTION command establishes an ORVYL data set into which invalid input data will be placed during SPIBILD input operations.

Syntax

General information

The exception file is created as soon as you issue the SET EXCEPTION command. During execution of the primary processing command where data is coming from the active file or some other external file (such as the BATCH command in SPIBILD), any data that causes a serious-level error, which prevents that record transaction from succeeding, is placed in the exception file. After the processing is completed, you can issue the ORVYL command "GET filename" to bring the exception file into your active file, allowing you to make changes and try putting the data in again.

If you decide not to use the exception file after setting one, issue the command CLEAR EXCEPTION. As soon as a file has been processed using an exception file, there is an automatic CLEAR EXCEPTION. In any case, whether any data is placed in it or not, an exception file exists as an ORVYL data set as soon as you issue the SET EXCEPTION command.

Example

Here is part of a SPIBILD session demonstrating the SET EXCEPTION command:

A longer example, showing the SET EXCEPTION command in the context of exception processing, appears earlier in this manual. [See 2.1.5.2 for information on the complete SET EXCEPTION procedure.]

6.2.3a  (SET FILE COUNTS)

The SET FILE COUNTS command displays the number of reads and writes from and to attached ORVYL data sets during SPIBILD processing.

Syntax

The command has no options.

General information

By default, SPIBILD will display a summary of the read/write activity involving the data sets (and specific record-types within them) for each SPIRES file as it finishes with the file. If you issue the SET FILE COUNTS command before you start your file-processing activity, SPIBILD will also present a display of the same information for the system files (such as those containing the FILEDEF and FORMATS subfiles) used during the session when you leave SPIBILD.

SET FILE COUNTS will also turn the user-file statistics summary back on if it has been turned off with the CLEAR FILE COUNTS command.

The command presents the same type of data as does the SHOW FILE COUNTS command in SPIRES. Unlike that command, however, SET FILE COUNTS continues to display statistics during the session.

Example

Here is part of a SPIBILD session demonstrating the statistics that SPIBILD displays:

For each file involved, SPIBILD displays the number of ORVYL blocks read and written since the primary file was attached (i.e., the file that was processed). Specifically, the read/write data is displayed for:

 - the DEFQ data set, which holds the deferred queue
 - the MSTR data set, which contains the file's master characteristics
 - the Residual data set (RES)
 - the file's private checkpoint data set (CKPT) if it has one
 - the individual record-types of the file

So the example above shows, for example, that SPIBILD read 251 blocks of the CALENDAR file's deferred queue, and wrote 5. Keep in mind that the numbers reflect the number of read/write operations, not the numbers of different blocks read or written. Since the counts are cumulative over the SPIBILD session, the same block may be read or written multiple times, with each operation being counted.

System files

For system files when you've requested the additional information from SET FILE COUNTS, SPIBILD displays similar counts for those files, except that they are combined into one file display. That is, the display for "System files" shows counts as if they all came from a single large system file, rather than from individual files. For example, REC2 represents a record-type in the system's FILEDEF file that is read when users establish a subfile, but it also represents a record-type in the system's FORMATS file where compiled formats code is stored.

Here is a list of the record-types you are likely to see in the system files section, along with the types of data being read from them:

6.2.4  (SET INDEX and SET NOINDEX)

The SET INDEX and SET NOINDEX commands limit the index record-types in a file that may receive passed information for a particular SPIBILD or FASTBILD command.

Syntax

Technical note on syntax: Any of the "record-names" can be followed by a positive integer in parentheses, e.g.:

The number is used to identify a specific occurrence of the INDEX-NAME instructions when the same name appears several times in the linkage section for a given goal record-type. (Warning for complicated files: The counting of "n" begins again with each new linkage section. If, for example, REC03 serves as an index record-type for both goal record-types REC01 and REC02, and REC01's linkage is first in the file definition, then the command SET NOINDEX REC03(2) will tell SPIBILD/FASTBILD not to pass values for the second occurrence of "INDEX-NAME = REC03" in each linkage section. The command does not mean that values should be passed to REC03 from REC01 but not REC02.)

WARNING: These commands are meant to solve unusual data integrity problems, but if not used carefully, can create them. In general, you should use these commands only in extraordinary situations and then only after consultation with your SPIRES consultant.

General information

In rare situations, you may not want a SPIBILD/FASTBILD processing command to pass data to all the index record-types as specified in the file's definition. You can issue the SET INDEX and SET NOINDEX commands to limit the index record-types affected by the command to only those you want. In other words, these two commands turn off or "leave on" passing to the named record-types.

The SET INDEX command names the only index record-types that are to receive passed information. The SET NOINDEX command names the index record-types that should not receive passed information.

Neither command has any effect until you issue a file-processing command, such as PROCESS or BATCH. Note too that it affects only the primary task of the SPIBILD/FASTBILD command, not any preliminary file-processing the command does, in contrast to the SET PASS and SET NOPASS commands, which affect all processing of the subsequent command. [See 2.1.5.4.]

For example, the BATCH command causes the file to be processed before the new records are batched into the subfile. If you issue a SET INDEX/NOINDEX command before the BATCH command, it will have no effect on the preliminary processing, but will affect the records batched in. If you want the records already in the deferred queue to be affected by SET INDEX/NOINDEX, issue both the PROCESS and BATCH commands, preceding each with the appropriate SET INDEX/NOINDEX command.

6.2.5  (SET JOBGEN and SET NOJOBGEN)

The SET JOBGEN command resets the JOBGEN flag, allowing JOBGEN to submit batch SPIBILD jobs to process the file and allowing subfile transactions to resume. SET NOJOBGEN has the reverse effect: it prevents JOBGEN from submitting batch SPIBILD jobs to process the file and it prevents subfile transactions from succeeding.

Syntax

The commands have no options and cannot be abbreviated.

General information

When a SPIBILD command fails, the NO JOBGEN flag is set, which blocks all update activity for the file, and prevents JOBGEN from submitting batch SPIBILD jobs to process the file overnight. The SET JOBGEN command sets the flag to JOBGEN, once again allowing update activity and again allowing JOBGEN to process the file.

WARNING: Do not issue the SET JOBGEN command for a file unless you are positive that no update activity will occur between the time you issue the command and the time the file is processed. If any updates do occur, data integrity of the file could be lost!

The SET NOJOBGEN command prevents JOBGEN from creating a job to process the file. It also unconditionally blocks all updating activity for the entire file, taking effect immediately. Update requests will fail with an S495 error. SET JOBGEN would then reverse the condition.

Remember though, that if a SPIBILD or FASTBILD failure causes the NO JOBGEN flag to be set, you should not simply reset the JOBGEN flag but should try to recover or process the file first, allowing SPIBILD to reset the flag if the file is okay. [See 2.2.1.]

Example

In the example below, no one except the file owner has access to the subfile. She knows that she will not add or update any records after issuing the command until the next day, after the file is processed via JOBGEN. This may save her some money, compared to processing the file herself now, at day rates, to recover from the SPIBILD problem that set the NO JOBGEN flag.

Remember, if there is any chance that users will try to update records between the time you set the JOBGEN flag and the time the file gets processed via JOBGEN, you should not use this command. Instead, follow the PROCESS/RECOVER procedure. [See 2.1.4.2.]

If you change your mind and want NO JOBGEN set, issue the SET NOJOBGEN command.

6.2.6  (SET MESSAGES)

The SET MESSAGES command controls the presentation of various types of system messages.

Syntax

 - INFORMATION (INFO) -- controls messages from the program that provide  information  (such
 as  "-Stack: 15 RECORDS") or confirm an action (such as the line of information
 about each processed record by the BATCH command in SPIBILD).
 - WARNING (WARN) -- controls messages from the program that announce a  potential  problem,
 that announce that SPIRES has taken some action to avoid a problem, or that give you more
 information about a serious error (e.g., "-Error at line 12").
 - SERIOUS (SER) -- controls messages from the program that announce a  processing  problem,
 such as "-Serious data error".
 - CHARGE (CHR) -- controls messages that announce subfile logging and charging is in effect
 (available only in SPIRES).
 - ALL -- controls all of the above.
For all types EXCEPT Charge:

Level  Explanation       Sample
-----  ----------------  ---------------------------------
  0    no message

  1    coded message     -127(S)-

  2    text message      -Forward range exhausted

  3    code and text     -127(S)- Forward range exhausted

  4    code, text and    -127(S)- Forward range exhausted
        explanation      --A BROWSE command has reached
                         the end of the range specified.

For Charge messages:

Level  Explanation
-----  -----------------------------------------------------------------
  0    no messages

  1    messages when the subfile is selected, for a logging subfile:
        -Command logging in effect for this subfile
             or for a charging subfile:
        -Database charge: $n.nn/hour, $n.nn/1000 displays, $n.nn minimum
       and when the subfile is cleared, if charging is in effect:
        -Approximate database charge: $n.nn

  2    same as 1, but when a charging subfile is selected, also:
        -Continue select?

  3    none of the messages from 1, but the prompt from 2 for a
        charging subfile.

  4    no messages

Syntax Notes:

 - In SPIBILD and FASTBILD, the automatic explanations of level 4 are not available, so  level
 4 is equivalent to level 3.
 - If you type a number higher than those shown (say, 5), the level gets set  to  the  highest
 setting, 4.

General information

Control over messages is more often needed by application designers than by file managers, so SET MESSAGES isn't as useful here as it is in protocols and formats, where it is often used to turn off the messages SPIRES sends to users.

For file management purposes, SET MESSAGES is probably most often used to suppress information messages during interactive SPIBILD sessions where records are being added, updated or removed.

The level set by a SET MESSAGES command remains in effect throughout the program session unless changed by another SET MESSAGES command or temporarily overridden by a SET MESSAGES Uproc in an executing format.

Example

In this example, the one-line information message for each successfully batched record is suppressed:

If the information messages hadn't been suppressed, there would have been 53 more lines of information, one for each successful transaction. Those confirmation lines slow down the task because SPIBILD must stop its work to send a message to the terminal for each successful transaction. Notice that information about the one transaction that failed (for the VA record) was not suppressed because that was a Serious message.

6.2.7  (SET PASS and SET NOPASS)

The SET PASS and SET NOPASS commands limit the goal record-types in a file that may pass information for a particular SPIBILD or FASTBILD command.

Syntax

WARNING: These commands are meant to solve unusual data integrity problems, but if not used carefully, can create them. In general, you should use these commands only in extraordinary situations and then only after consultation with your SPIRES consultant.

General information

The SET PASS command without options represents the default passing condition for a file: it allows passing from all goal record-types in the file. With the "n" option, the command allows passing only from the "nth" logical record-type; passing from all other record-types in the file is blocked.

The SET NOPASS command blocks passing from all goal record-types. With the "n" option, the command allows passing from all goal record-types except the "nth" logical record-type.

Neither command has any effect until a SPIBILD or FASTBILD file-processing command is issued. It affects all subsequent processing of the file, including any preliminary processing done by a command (e.g., BATCH) before the main task of the command begins executing. If that isn't desired, try processing the file first (with the PROCESS/ESTABLISH procedure) before issuing the SET PASS or SET NOPASS command. [See 2.2.2.]

Only one "pass" setting is in effect at a time. A subsequent SET PASS/NOPASS command cancels any previous one, so if you change your mind after issuing the command, issue an opposite command to cancel it. (For example, if you want to retract a "SET PASS 5" command, then a SET PASS command would cancel it.) Also, the default condition (SET PASS) is restored after each file-processing command is executed, with the exception of ESTABLISH.

6.2.8  (SET SEARCH and SET NOSEARCH)

The SET SEARCH command tells SPIBILD or FASTBILD to let users continue to select a subfile and search and display records while the file is being used for processing tasks. SET NOSEARCH cancels SET SEARCH.

Syntax

The commands have no options; the abbreviations SEA and NOSEA are allowed.

General information

By default, a file is unavailable to users while it is being processed or being used for other SPIBILD or FASTBILD tasks. For some large applications, where round-the-clock access is necessary, the "down-time" of a file while it is being processed can be a problem. For instance, a very large file, used by dozens of people each day, might take ten minutes or more each night to process.

Moreover, SPIBILD and FASTBILD will not by default even let you process a file if users have any of its subfiles selected.

Both problems can be solved with the SET SEARCH command. It allows you to perform SPIBILD/FASTBILD tasks on a file when users are currently using it, and the command allows users to select subfiles of the file, and search and display records, as file processing goes on. Specifically, all read-only operations using the file can continue; file updates, on the other hand, are not allowed till the processing is finished. (Compare SET SEARCH to SET SHARE, which lets you process the file while users have its subfiles selected, but which does not let the users continue to work with the subfile.)

The commands must be issued before the SPIBILD or FASTBILD processing command they are to affect. Note that the prevailing condition is not cleared at the end of the file-processing but remains in effect for subsequent file-processing commands, unless altered by another SET SEARCH or SET NOSEARCH command.

Notes:

Example

Suppose you need to process a file in SPIBILD during the day. The file contains multiple subfiles used by many people during the day. You need to add some records to one of the subfiles using SPIBILD's BATCH command. Issuing the SET SEARCH command before BATCH allows people searching the other subfiles to continue, unaffected by your work.

6.2.9  (SET SHARE and SET NOSHARE)

The SET SHARE command tells SPIBILD to let you process a file even though users have it selected. SET NOSHARE cancels SET SHARE.

Syntax

The commands have no options; you may use the abbreviations SHA and NOSHA.

General information

When a file is processed, the account doing the processing needs exclusive access to the file. That is, other users cannot be changing records in the file while the processing occurs. Similarly, while the file is being processed, other users cannot search in the file's subfiles because the processing may be changing the indexes.

SPIBILD and FASTBILD generally avoid these problems by not letting file owners process files if users have their subfiles selected, and by not letting users select a subfile if its file is being processed. The file owner who tries to process a file being used receives an S56 error message: the file is "busy". A user who tries to select a subfile of a file being processed does not get another command prompt until the processing is complete -- the terminal "hangs".

Occasionally, however, you may absolutely need to do some SPIBILD work with one of your files, regardless of the fact that some other user has one of its subfiles selected. If you are the file owner or a user with Process access to the file, you can issue the command SET SHARE to request non-exclusive access to the file. The command has no effect until you issue one of these SPIBILD or FASTBILD commands:

As soon as one of those commands is issued, users who try to select any subfile of the file or issue searching and updating commands against it will find that their terminal is hanging, waiting for the completion of your SPIBILD task. [Depending on what physical data sets of the file are being used by your processing work, any given user command could succeed at a particular moment. For example, while records are being moved into a subfile's tree, the indexes might still be available so that a search command would go through.] Note that users who have those subfiles selected can still do other work (e.g., text editing) -- only commands directed to the particular subfile will cause the terminal to hang.

When the processing command (except ESTABLISH) finishes executing, the "share" condition ends -- users will start getting responses to commands that use the file. (The ESTABLISH command is exceptional because it is the first step of a SPIBILD task, not the only one. The SPIBILD commands that usually follow it, such as MERGE and REMOVE, will end the "share" condition. Leaving SPIBILD via an EXIT or CALL command will also clear the "share" condition.)

However, ending the condition does not "clear" the condition. If you issue another SPIBILD command from the list above, SET SHARE will remain in effect. To clear the condition, issue the SET NOSHARE command in SPIBILD at any point before the command you want it to affect.

Warning: Use of SET SHARE is advised only in situations where SPIBILD or FASTBILD processing must proceed, as in a batch job where subsequent steps depend on the success of the processing. Users who find their terminals hung usually have no idea what happened, since there is no message telling them the file is being processed. By pressing ATTN/BREAK twice in a row, they can cause session breaks and receive a WYLBUR command prompt, though they are no longer in SPIRES. Typing GO will return them to the place (and the waiting) where they were before.

Example

Here is a sample SPIBILD session showing how you might use the SET SHARE command:

At this point, you might suspect that SPIBILD is mistaken, because you are the only one who ever uses the file. But then you remember that you left a terminal logged on somewhere else, and that it has the subfile selected. That session won't be bothered at all if you "set share":

6.2.10  (SET REINDEX)

SET REINDEX COMMAND, IN SPIBILD

The SET REINDEX command in SPIBILD is used to establish an environment whereby SPIBILD will index records from the DEFQ with special processing. SET REINDEX remains in effect for the duration of the SPIBILD session, or until turned off by a SET NOREINDEX command. The command may not be abbreviated.

SET REINDEX is used primarily for the following purposes:

1) PROCESSing of restored DEFQ's during  disaster  recovery.   That  is
   when  a  full file restore needs to be done and then saved copies of
   the DEFQ's need to be applied.
2) Whenever it would normally be required to REMOVE and then  re-ADD  a
   set of goal records to force indexes to be up-to-date.  That usually
   occurs  when goal records have been ADDed and indexing did not occur
   because of a flaw in the PASSPROC  rules.   Once  corrected,  simply
   update  the  goal  records  (DEFQ  updates)  and use the SET REINDEX
   command to force  the  PROCESS  command  to  reindex  all  the  DEFQ
   records.

SET REINDEX only affects records being passed from the DEFQ. Therefore, it does not affect LOADed records, or BATCH input that does not go to the DEFQ. SET REINDEX affects the passing of DEFQ records (ADD, UPD, REM) in the following ways:

1) All indexes are processed, even those declared IMMEDIATE.
   SET REINDEX also ignores any SET NOINDEX list.
2) Goal records which belong to a REMOVED record-type and which pass to
   Immediate Indexes and are marked as ADDed (or DEQueue  of  an  ADDed
   record)  in  the  DEFQ will check the Tree.  If the record exists in
   the Tree, the DEFQ entry is ignored.  If not found in the Tree, then
   the RESidual copy of the record is passed for  index  deletion,  and
   the ADDed record is then passed for index insertion.
3) Insertion/deletion pairs in the pass stack that  normally  would  be
   cancelled  are  retained.   That  means  all  the  index records are
   re-examined and the  insertion/deletion  is  done.   There  are  two
   possible outcomes of this process:
    a) If the index record does not have  "large  segments",  then  the
       index  pointer  is  inserted  into the index record if it didn't
       already exist there.  The index record is only rewritten if  the
       pointer had to be inserted.
    b) If the index record does have "large segments", then  the  index
       pointer  is  inserted  into the top segment if it didn't already
       exist there, and then the index  pointer  is  deleted  from  any
       secondary  segment  if  it  exists in such a segment.  The index
       segments are only rewritten if they are changed.  The effect  is
       to guarantee that the pointer exists in the top segment.

An UPDATEd record in the DEFQ is what normally creates an "insertion/deletion pair", which is a pair of pass stack entries that match, but one comes from passing of the old version of a record (the deletion), and the other comes from passing of the new version of that same record (the insertion). For example, if TITLE = THE DOG; was in the old record, and TITLE = THE CAT; is in the new (updated) record, then the word THE creates an insertion/deletion pair. DOG and CAT create normal deletion and insertion pass stack entries which do not cancel, and would be applied to indexes regardless of REINDEX.

Syntax

The command has no options, and may not be abbreviated.

6.3.0  (SET SKIP)

The SET SKIP command tells SPIBILD how many lines or records in the input data set to skip before adding them to the subfile.

Syntax

Warning: If you are in WYLBUR mode, be sure that "n" is at least two digits (e.g., instead of "1", type "01"), or precede the command with an exclamation point; otherwise, WYLBUR will interpret the command as an instruction to set WYLBUR's skip character and will not pass it to SPIBILD.

General information

The SET SKIP command is most frequently used in recovery situations when a LOAD or BATCH command in SPIBILD didn't complete properly. The number "n" comes from the SHOW RECORD RECOVERY command display, which is available after one of these commands fails to finish. [See 2.3.6.]

Example

It's best to recover the file with the RECOVER/PROCESS procedure in SPIBILD [See 2.1.4.2.] after which you resume the procedure you were following, issuing the SET SKIP command prior to the processing command:

6.3.0a  (SET STACK SIZE)

The SET STACK SIZE command lets you raise the amount of space available in the pass stack for use by SPIBILD or FASTBILD.

Syntax

General information

The SET STACK SIZE command is used primarily in problem situations. Its typical use is when a record generates so many index entries that they can't fit into the pass stack, in effect getting the record stuck halfway between defq and tree. (The evidence for this problem is the dreaded "Passing: 0 records" message in SPIBILD or FASTBILD.) When you raise the stack size before starting the SPIBILD processing command again, the record will make it through.

The larger the pass stack size, the more efficient passing usually becomes, although this is not generally a significant concern. If two different records in two pass stacks pass to the same index record, then the index record must be fetched, updated and put back twice; but if both records are in the same pass stack, then the index record is handled only once, meaning fewer IOs.

There is very little point in reducing the pass stack size below the default of 256K, unless you get an S469 error. That indicates too many records are being placed in the pass stack, and the index records are growing too large. In such a case, you may wish to try reducing the pass stack size to limit the number of goal records entering the stack. This gives the index records a chance to grow and split.

SET STACK SIZE is not available in SPIRES; hence immediate indexing is limited to the default 256K size for a pass stack. If you get an S469 error passing to an index in SPIRES, you will be forced to use SPIBILD with a reduced stack size.

6.3.0b  (SET SUBFILE DISABLE and SET SUBFILE ENABLE)

The SET SUBFILE DISABLE command prevents users from either using or selecting any subfiles of the file involved, or from creating transactions in that file, or from processing the file. It can also be used to disable duplicate deferred-queues. The SET SUBFILE ENABLE command reverses the effect.

Syntax

General information

The SET SUBFILE DISABLE command is used in emergency situations where it is critical to block users from selecting or updating subfiles of the file as quickly as possible. The SET SUBFILE ENABLE command reverses the effect of SET SUBFILE DISABLE, re-establishing normal conditions for the file.

Note that SET SUBFILE DISABLE FOR ATTACH will not affect users who currently have subfiles of the file selected; SET SUBFILE DISABLE FOR UPDATE will not affect users currently in the midst of a transaction (e.g., entering a record using the $PROMPT format). SET SUBFILE DISABLE FOR ACCESS will affect users in both those situations.

None of those three flavors (ATTACH, UPDATE, ACCESS) affect the file owner, who may continue to select the subfile and create transactions. Users with Master access to the file are also exempt.

On the other hand, disabling the subfile for PROCESS or DUPLICATE affects all users, including the file owner and users with Master access to the file.

Example

Though you issue the command when you have a subfile selected, remember that the command affects all subfiles of the file:

When you are ready to allow users back into the file:

You can check the status of a file (if you can attach it or select one of its subfiles) with the "SHOW EVAL $DEFQTEST(code)" command, where "code" is DISABLED, PROCESS, WRITE and DUPLICATE. Except for DUPLICATE, if the value returned is "0" then the particular "disabling" state is NOT in effect. In other words, if SHOW EVAL $DEFQTEST(DISABLED) returns 0, then the file is not disabled; if SHOW EVAL $DEFQTEST(PROCESS) returns 0, then the file is not disabled for processing. For details about how to use the codes and interpret the results, EXPLAIN $DEFQTEST FUNCTION, or see the manual "SPIRES Protocols".

6.3.1  (SET TCOMMENT)

The SET TCOMMENT command contains a string expression to be put into the transaction log for the subfile transaction that follows.

Syntax

SET TCOMMENT is also available as a Uproc in SPIRES formats.

General information

The SET TCOMMENT command sets the value of the $TCOMMENT variable, which is passed to the TCOMMENT element in a $TRANSACTIONS record when the next subfile transaction occurs. Thus you can put data of your own choice into the $TRANSACTIONS records. An application might, for example, need to keep special accounting data for each transaction. The value in the system string variable $TCOMMENT will be stored with the transactions data and can be found in the $TRANSACTIONS record for the given transaction as the element TCOMMENT.

The variable $TCOMMENT is cleared after each subfile transaction and when a CLEAR SELECT command is issued for the primary subfile. Hence it must specifically be set for each transaction, rather than one time for many transactions.

Like $ASK and $PROMPT, $TCOMMENT is a global variable with only a single occurrence. There is not a separate occurrence for each subfile path if path processing is used. (Also like $ASK and $PROMPT, $TCOMMENT can be used to pass values back and forth between formats and protocols.)

Example

The SET TCOMMENT command sets $TCOMMENT to the current value of the $VERSION variable. When SPIRES executes the ADD command, the value of $TCOMMENT is passed to the $TRANSACTIONS record for the added record.

6.3.2  (SHOW ARCHIVE)

The SHOW ARCHIVE command displays the subfile log entries stored in your SYSTEM.ULOG data set.

Syntax

General information

The SHOW ARCHIVE command shows you data stored in the ORVYL data set SYSTEM.ULOG, which is created when any of your files contain the STATISTICS = 2 statement and the file is processed. SHOW ARCHIVE displays the data in the same format as the SHOW FILE LOG command, but contains log data for all your files that archive log data, all mixed together.

Each line of the log data represents one log record. The type code, in columns 2 and 3, specifies the type of activity that caused the record to be written:

 - C8: SELECT commands
 - C9: COMMANDS issued by the user
 - CA: ASK prompts and responses
 - CB: WITHIN LOG prefixed commands
 - E8: CLEAR SELECT commands that de-select the subfile
 - EA: SELECT CHARGE to the user for elapsed time
 - EB: DISPLAY CHARGE to the user for records seen
 - EC: MINIMUM CHARGE to the user

Details about the log data appear earlier in this manual. [See 4.3.4.]

Example

Here is a sample session showing the use of this command:

-> show archive
 C8   08:35:20  01/12/87  GQ.DOC2      $0.00       $0.00       $0.00  FISH
 C9   08:35:22  01/12/87  GQ.DOC2 DIS smallmouth buffalo
 C9   08:36:10  01/12/87  GQ.DOC2 ADD
 C9   08:40:28  01/12/87  GQ.DOC2 DIS longear sunfish
 C9   08:40:30  01/12/87  GQ.DOC2 DIS black crappie
 C9   08:41:00  01/12/87  GQ.DOC2 ADDUPD
 C9   08:41:04  01/12/87  GQ.DOC2 /DISPLAY $KEY
 C9   08:53:42  01/12/87  GQ.DOC2 FIND eating = good
 C9   08:53:46  01/12/87  GQ.DOC2 TYPE
 E8   08:55:26  01/12/87  GQ.DOC2           19  00:20:06 08:25:20 FISH
   (etc.)

6.3.2a  (SHOW AUTOGEN)

The SHOW AUTOGEN command tells you whether AUTOGEN is set for the file of the currently selected subfile.

Syntax

There are no options. Unlike most SHOW commands, SHOW AUTOGEN will ignore any IN ACTIVE prefix added to it, displaying the result on the terminal screen anyway.

General Information

The SHOW AUTOGEN command tells whether AUTOGEN or NOAUTOGEN (i.e., overnight SPIBILD processing) is set for the file containing the currently selected subfile. [See 2.2.1.] In addition, if the NO JOBGEN flag is set, that information is also displayed.

SHOW AUTOGEN does not show the normal, file-definition-specified setting for AUTOGEN; it simply shows the current value, which may have been changed since the last time the file was processed via a SET AUTOGEN or SET NOAUTOGEN command.

Example

Here is a sample session showing the use of this command:

After discovering that NOAUTOGEN was in effect, the user decided to set overnight processing by issuing the SET AUTOGEN command.

6.3.3  (SHOW BLOCKS)

The SHOW BLOCKS command tells you how many ORVYL file blocks are being used and how many are allowed to be used by your account.

Syntax

General information

The command with no options shows the number of 2048-byte ORVYL blocks being used by your account, and how many your account is allowed. The difference between them is the number of remaining blocks available to you. (Note: You are charged only for the number of blocks used, not for the number you are allowed.) [See 3.2.1.]

Example

6.3.3a  (SHOW ERROR COUNTS)

The SHOW ERROR COUNTS command displays a summary of information concerning the number and kinds of errors that occurred during a SPIRES or SPIBILD session. The CLEAR ERROR COUNTS command resets the internal table in which the information is kept.

Syntax

Syntax note: When using the CLEAR ERROR COUNTS command in a session where SET WYLBUR is in effect, you will need to prefix the command with an exclamation point (!) so that it is passed to SPIRES before being passed to ORVYL, which has a CLEAR ERROR command of its own.

General information

The SHOW ERROR COUNTS command summarizes information about errors that have occurred during a SPIRES or SPIBILD session. The statistics show information about the error codes that would be normally put out in error messages.

Since the statistics are cumulative for the session, it may be useful to reset the error count table under some circumstances, which you do by issuing the CLEAR ERROR COUNTS command.

Example

Here is an example of the SHOW ERROR COUNTS command at the end of a SPIBILD session where records were added to a subfile under INPUT BATCH:

Notice that the errors that have occurred during the session are displayed in descending order by the number of times they occurred. The severity of the error appears in parentheses, "(S)" for Serious, and "(W)" for Warning.

6.3.4  (SHOW FILE ACTIVITY)

The SHOW FILE ACTIVITY command displays information about past activity for each record-type in the file, as well as information about each record-type's base block within a data set.

Syntax

Syntax note: SHOW FILE ACTIVITY may be issued as SHOW SUBFILE ACTIVITY, but because its output relates to the file as a whole, SHOW FILE ACTIVITY is the preferred form.

General information

The SHOW FILE ACTIVITY command summarizes updates that have occurred during SPIBILD processing since the beginning of the attached file's existence. In addition, it tells which data set contains each record-type of the file and tells the number of the base block for each record-type within that data set.

Example

The data in the sample SHOW FILE ACTIVITY display below is described at the end.

    01/07/1987     Activity Record of File GQ.JNK.BLOOD.DONORS
                   Last Compiled: 09/18/1984 at 17:23:24

 Record      Base     Blocks     Adds    Deletes   Updates   Max Rec
  Type       Block   Used -DS                                 Length

 REC01 -Slot     0      1 -1        59         3        26       446
 ZIN02           0      7 -2        54         6         6        62
 ZIN03           1   Comb -2        22         0        25        68
 ZIN04           2   Comb -2         2         0        27       160
 ZIN05           3   Comb -2         8         0        36        49
 ZIN06           4   Comb -2        84         3        16        43
 ZIN07           5   Comb -2        51         4        58       131
 ZIN08           6   Comb -2        12         2        24       129

Residual         2         8        63         3        49       447
Def Queue        1         2

An explanation of the items under each heading follows:

Note: The numbers of adds, deletes and updates maintained by the system may not be precisely accurate since there is no attempt to maintain their integrity in case of system failure -- the expense in doing this is deemed excessive in comparison to their value. However, in all likelihood, the numbers will be accurate.

6.3.4a  (SHOW FILE LIMITS)

The SHOW FILE LIMITS command displays the size of data sets and the depths of trees in the file, marking any close to limits.

Syntax

Syntax Note: SHOW FILE LIMITS may be issued as SHOW SUBFILE LIMITS, but because its output relates to the file as a whole, SHOW FILE LIMITS is the preferred form.

General information

The SHOW FILE LIMITS command shows you data set limits for the attached file, showing you how close the individual data sets are getting to their maximum limits. The limits represent the limits of the specific data set as defined; rebalancing a data set can readjust or redefine the data base so that either the limits are raised or the data sets back away from them.

All data sets are shown, including REC1 through RECF (as they exist for the file), DEFQ, MSTR, RES and RESn (where "n" is an integer, when multiple residuals exist). If a data set is approaching one of the two limits shown (size or tree depth), it is flagged with three asterisks (***).

The columns are:

 - Max Blocks -- maximum number of blocks in the data set
 - Blocks Used -- the number of blocks used (derived from the "Next  Block"  number)
 - Percent Used -- Blocks Used divided by Max Blocks.  If this  value  is  greater  than  94%,
 warning characters (***) are displayed next to the value.
 - Tree Depth -- The number of levels of the deepest tree within the data set.  If this  value
 is 12 or more, warning characters (***) are displayed.

The command also shows the block size for the data sets, which will become more relevant as 4096-byte blocks become available and employed in SPIRES data bases.

6.3.5  (SHOW FILE LOG)

The SHOW FILE LOG command displays the file log entries in the deferred queue.

Syntax

Syntax Note: SHOW FILE LOG may be issued as SHOW SUBFILE LOG, but because its output relates to the file as a whole, SHOW FILE LOG is the preferred form.

General information

The SHOW FILE LOG command shows you data from the deferred queue gathered from sessions where some level of subfile logging is in effect. When the file is processed, the log data may be archived, and then it is discarded. [See 4.3.1.] to learn how to archive it, if desired.

Each line of the log data represents one log record. The type code, in columns 2 and 3, specifies the type of activity that caused the record to be written:

 - C8: SELECT commands
 - C9: COMMANDS issued by the user
 - CA: ASK prompts and responses
 - CB: WITHIN LOG prefixed commands
 - E8: CLEAR SELECT commands that de-select the subfile
 - EA: SELECT CHARGE to the user for elapsed time
 - EB: DISPLAY CHARGE to the user for records seen
 - EC: MINIMUM CHARGE to the user

Details about the log data appear earlier in this manual. [See 4.3.4.]

Example

Here is a sample session showing the use of this command:

-> show file log
 C8   16:25:20  01/12/87  GQ.JNK       $0.00       $0.00       $0.00  DATES
 C9   16:25:22  01/12/87  GQ.JNK  dis 634
 C9   16:26:10  01/12/87  GQ.JNK  add
 C9   16:30:08  01/12/87  GQ.JNK  addupd
 C9   16:30:28  01/12/87  GQ.JNK  dis 312
 C9   16:30:30  01/12/87  GQ.JNK  dis 322
 C9   16:31:00  01/12/87  GQ.JNK  addupd
 C9   16:31:04  01/12/87  GQ.JNK  /display $key
 C9   16:43:42  01/12/87  GQ.JNK  find interest lanolin
 C9   16:43:46  01/12/87  GQ.JNK  type
 E8   16:45:26  01/12/87  GQ.JNK            51  00:20:06 16:25:20 DATES
   (etc.)

6.3.6  (SHOW FILE STATUS)

The command SHOW FILE STATUS displays a summary of information about file growth gathered by the most recent STATUS command.

Syntax

Syntax note: SHOW FILE STATUS may be issued as SHOW SUBFILE STATUS, but because its output refers to the contents of the file as a whole, SHOW FILE STATUS is the "more correct" form.

General information

SHOW FILE STATUS provides a capsule summary of some of the storage and retrieval statistics for the attached file. Most of its contents come from statistics gathered by the most recent STATUS command issued for each data set. If no STATUS command has ever been issued for a data set, most of its statistics will be "0".

The most useful information provided by this command is probably the date a STATUS command was last issued for each record-type. Also useful may be the date and time the file definition for the file was last compiled, shown at the start of the display, and the number of records in each record-type. [The number of records is maintained independently of the STATUS command, so that column will have values whether or not a STATUS command has been issued. The number comes from data in the SHOW FILE ACTIVITY command, specifically the difference between the number of adds and the number of removes for each record-type. Note that its accuracy is not absolutely guaranteed.]

But the SHOW FILE STATUS command isn't as good a source for information to use in making decisions about the file (e.g., rebalancing a record-type, rebuilding the available space chains, etc.) as the data from the STATUS command itself, which is more complete and up-to-date. Still, as a summary of that information, the SHOW FILE STATUS command may be useful to you.

Example

The data in the sample SHOW FILE STATUS display below is explained at the end.

    01/06/1987       Status Record of File GQ.JNK.BLOOD.DONORS
                   Last Compiled: 09/18/1984 at 17:23:24

 Record-    Blocks    Space     Number   Accesses   Tree     Date
  Type       Used    Utilized   Records  / Record   Depth

 REC01 -Slot     1    24.224        56     2.000       0    01/06/87
 ZIN02           1    95.264        48     1.000       1    01/06/87
 ZIN03           1    33.456        22     1.000       1    01/06/87
 ZIN04           1     5.424         2     1.500       1    01/06/87
 ZIN05           1    17.296         8     1.000       1    01/06/87
 ZIN06           1    80.432        81     1.000       1    01/06/87
 ZIN07           1    66.320        47     1.021       1    01/06/87
 ZIN08           1    12.560        10     1.100       1    01/06/87

Residual         8    91.360        60     0.000       0    01/06/87
Def Queue        2

An explanation of the items under each heading follows:

6.3.7  (SHOW FILES)

The ORVYL command SHOW FILES lists the names of the ORVYL data sets stored on your account or the named account.

Syntax

Example

For example, from account GQ.DOC:

You can determine the size of a SPIRES file in blocks by adding the number of blocks shown. [Technically speaking, you must also add one block for each 660 blocks of each data set; these blocks are not part of the data set per se, but are stored in ORVYL's index blocks and are charged to you. Non-IXR data sets do not have these blocks. [EXPLAIN GENERATE NONIXR COMMAND for more information, or see its reference earlier in Part 6.]]

6.3.7a  (SHOW RECORD INFORMATION)

The command SHOW RECORD INFORMATION reports how much storage space is used for a specific record, including information about any large record segments.

Syntax

General information

The SHOW RECORD INFORMATION command will show you statistics about:

 - the length in bytes of the stored record or records;
 - the locator for each large record segment of the record, if it exists; and
 - the length in bytes of each large record segment.

Like SHOW RECORD OVERHEAD, this command shows the record length including the "overhead" data stored with the record, which describes how many occurrences of an element are stored, how long each value is, which optional elements occur, etc.

Example

Here is part of a SPIRES session using the SHOW RECORD INFORMATION command to see information about an index record's large record segments:

For records without large record segments, only the first two lines of the display above are shown.

Users who do not have See access to the file but who would like to see the storage size of a record in the selected subfile can use the $RECINFO function to get that value.

6.3.8  (SHOW RECORD OVERHEAD)

The command SHOW RECORD OVERHEAD reports how much storage space is used for a specific record, a sampling of records or all records in a subfile.

Syntax

General information

The SHOW RECORD OVERHEAD command will show you statistics about:

 - the length in bytes of the stored record or records;
 - the percentage of that space being used to store "overhead" data (see below); and
 - the number of records examined by the command.

The overhead this command refers to is that stored in the record itself that precedes element values, the "header" data, which describes how many occurrences of an element are stored, how long each value is, which optional elements occur, etc. To a large extent, this is overhead prescribed by the file owner in the file definition -- Optional elements in general have slightly more "overhead" than Required ones, and Fixed elements require none. [See the manual "SPIRES File Definition", section B.6.2, for further information on the headers.]

Example

Here is part of a SPIRES session using the SHOW RECORD OVERHEAD command: