*  SPIRES Technical Notes

******************************************************************
*                                                                *
*                     Stanford Data Center                       *
*                     Stanford University                        *
*                     Stanford, Ca.   94305                      *
*                                                                *
*       (c)Copyright 1994 by the Board of Trustees of the        *
*               Leland Stanford Junior University                *
*                      All rights reserved                       *
*            Printed in the United States of America             *
*                                                                *
******************************************************************

        SPIRES (TM) is a trademark of Stanford University.

+1  A Brief Explanation About this Document

This manual is in the process of being written. There are many subjects that fit into the category of data base management that do not and should not belong to other SPIRES manuals. In fact, this manual was created when we began to realize that we did not know what other currently existing manuals should contain certain new material developed and documented in SPIRES.

This manual currently is a catch-all, a place for homeless documentation of SPIRES components. When time becomes available, it will become a structured document, organized like any other SPIRES manual, as a logically constructed reference manual. For the time being, however, it will remain as a "draft" document, with new sections added as needed in order to document the growing number of SPIRES capabilities that cross the boundaries between File Definition, Formats, Protocols and general file management.

1  Sorting Records in SPIRES; SPISORT

SPIRES has several different capabilities for sorting goal records, that is, placing records in order based on the values of one or more goal record elements. Record sorting is often desirable for reports, where you want all the records having a particular value for a given element to be displayed together, all the records having a second value to be together, and so forth.

The four methods of arranging goal records are discussed and compared in the next section. [See 1.1.] The remaining sections of this chapter will cover the most versatile of the methods, a procedure called SPISORT. [See 1.2.]

1.1  An Overview of Record-Sorting Methods

This section will discuss and compare four different ways to sort goal records. The remainder of this chapter will be about one of them, usually called SPISORT, although the actual SPISORT program is just one of its steps. [See 1.2.] The others are discussed in other SPIRES manuals; appropriate references are included here.

The four methods are:

1. the SEQUENCE command
2. SPISORT and record sets
3. the FOR INDEX command
4. the FOR SUBFILE command

Below, each of these techniques is briefly discussed:

The SEQUENCE command

The simplest of the four techniques to use, the SEQUENCE command sorts the records in the current stack or search result. The resulting stack can then be displayed with the TYPE command. It is commonly used in a situation like this:

-> find date.reported after 1980
-Result: 186 REPORTS
-> sequence date.reported, who.reported
-Stack: 186 REPORTS
-> set format discrepancy report
-> in active clear, type
->

The SEQUENCE command:

 - is usually cheaper than SPISORT, but is less versatile;

 - is a single command issued online, with the  sorted  stack  of  records  returned  for  use
 immediately;

 - cannot sort a large number of records (no absolute number can be specified as a limit; as a
 rule of thumb, a practical limit is 3,000 records);

 - orders records only by the first occurrence of an element;

 - requires an existing stack or search result for its input;

 - can sort records by either the internal or external forms of the element values.

 - is documented in the reference manual "SPIRES Searching and Updating".

The SPISORT Procedure

Several steps are involved in using SPISORT. First you create a "set" of records, using the Global FOR commands DEFINE SET and GENERATE SET. Next you issue a SPISORT command to sort the records in the set. Then you may process the sorted records under the Global FOR command FOR SET.

The SPISORT procedure:

 - is usually the most expensive method, but is certainly the  most  powerful  and  versatile,
 handling many situations the other methods cannot;

 - requires that several commands be issued, and a batch job be run (the SPISORT command  runs
 a batch job to execute the SPISORT program);

 - can sort an arbitrarily large number of records;

 - can sort records by multiply occurring elements and structures; the same record may  appear
 several  times in the set, once for each occurrence of the element on which the records are
 being sorted.

 - does not require a pre-existing result or stack to be created;

 - can sort records by either the internal or external form of element values, in ascending or
 descending order;

The FOR INDEX Command

With this technique and FOR SUBFILE below, you are taking advantage of "automatic sorting" done by SPIRES in maintaining the data base. The indexes of the subfile in effect sort goal records by element value. Using the FOR INDEX command with Global FOR processing commands, you can display records in order without having to sort them specially. In addition, using the SEQUENCE option on the FOR INDEX command, you can sort the records retrieved at a given index node (i.e., having the same value for the indexed element) by secondary elements, in a manner similar to the SEQUENCE command. Here is a sample session using the FOR INDEX command:

-> for index name
+> set format $report Name City Donation.Amount
+> in active display all
+>

The report in the active file would display records in alphabetic order by values passed from goal records to the NAME index, which presumably contains the values from the NAME element.

The FOR INDEX command:

 - is cheaper than SPISORT or the SEQUENCE command, but is just slightly more  expensive  than
 the FOR SUBFILE command;

 - requires that the primary element for the sorting be indexed in a simple index;

 - sorts the records by the primary element only in its internal, indexed form, but  secondary
 sorting elements may be used in either their internal or external form;

 - sorts the records only in ascending  order  by  the  primary,  but  sorts  them  in  either
 ascending or descending order for secondary elements;

 - handles only the goal records that are represented in the index;

 - is an easy-to-use "subset" of the capabilities of the FOR PATH command, which has
 more power than FOR INDEX;

 - is documented in the reference manual "Sequential Record Processing in SPIRES:  Global
 FOR".

The FOR SUBFILE Command

If the goal records are to be sorted by the key element of the record, then no special sorting is required at all -- the records are maintained in key order in the tree. Either FOR TREE or FOR SUBFILE (to pick up deferred queue records) processing can be used to retrieve the records in that order. Since the key is unique for each record, there are no secondary elements for sorting. Of the four techniques discussed, this method is the cheapest, since it requires no special sorting.

FOR SUBFILE processing is also discussed in the Global FOR manual mentioned above.

1.2  Sorting Records with SPISORT

SPISORT is a name used to describe a three-step procedure for sorting goal records, though specifically its name refers only to the batch program used in the second step. The three steps you follow are:

1) Create an input data set containing the unsorted data, called a "set" in SPIRES. This ORVYL data set is created in Global FOR mode using the Global FOR commands DEFINE SET, GENERATE SET, and ENDFOR:

-> select my biblio
-> for subfile where fiction yes
+> define set fictin, elem author(tv=all) title
+> generate set
+> endfor
-'ORV.GQ.DOC.FICTIN' has 192 sort entries
-End of Global FOR
->

2) Sort the input data set by issuing a SPISORT command. This command actually runs a batch job that uses the SPISORT program. That job creates an output data set, also called a "set". The SPISORT command for the example begun above might look like this:

-> spisort fictin to fictout
->

3) Process the sorted set to display the records. The FOR SET command lets you process records in a set in Global FOR mode:

-> for set fictout
+> in active, display all
+>

Each of these steps will be described in turn. The next section will discuss the first step, in which a set for sorting is created. [See 1.3.] Following that will be a discussion of the SPISORT command [See 1.4.] followed by more information on the FOR SET command, used to process sets. [See 1.5.]

Discarding the set could be considered a significant fourth step to the procedure. Since the set is stored as an ORVYL data set on your account, it will accrue storage charges, so you should get rid of it when you no longer need it. To discard a set, issue the ORVYL command ERASE:

ERASE setname

Multiple occurrences of an element used as a sort field cause multiple "sort records" to be created in the set. Thus, after the set is sorted, FOR SET processing may cause the same goal record to appear several times, once for each occurrence of the sort element. By default, in fact, when the goal record is displayed under FOR SET, only the occurrence of the sort element that caused the record to appear at that point in the set is displayed; the others are "filtered out" automatically. The filters can be removed, if desired. Filters add a great deal of power and sophistication to SPISORT work, and they are discussed separately later in this chapter. [See 1.7.]

By default, during FOR SET processing, SPIRES retrieves a pointer from the set for each goal record, and then fetches the goal record from the tree (or deferred queue) using that pointer. Thus, each record is, in effect, retrieved twice: once to fetch the sort data, and once to fetch the data for FOR SET processing. In some cases, you can save substantially by creating a "direct" set, in which the set itself contains all the data used for sorting and used for displaying under FOR SET processing. Direct sets and their use are discussed in a later section. [See 1.8.]

1.3  Creating the SPISORT Input File

A set is created with a combination of Global FOR commands: DEFINE SET, GENERATE SET and ENDFOR.

The DEFINE SET command "opens" the set -- that is, it establishes the ORVYL data set that will hold the data.

In addition, it tells SPIRES which elements to put into the set. [See 1.3.1.]

The GENERATE SET command tells SPIRES to do the work of putting data from goal records into the set. [See 1.3.2.]

The ENDFOR command ends Global FOR mode, closing the set.

You must be in Global FOR mode to issue this series of commands. The Global FOR command "FOR class" initiates Global FOR, telling SPIRES which records to process. WHERE clauses and SET SCAN commands may also be incorporated into the procedure as desired. For more information about Global FOR in general (and about the "FOR class" command in particular), see the reference manual "Sequential Record Processing in SPIRES: Global FOR".

The next sections will discuss the DEFINE SET and GENERATE SET commands in detail.

1.3.1  The DEFINE SET Command

A SPISORT input file is created by the DEFINE SET command. Unsorted records are placed in it by a GENERATE SET command. Both of these commands can be issued only when Global FOR is in effect. Any global FOR class, WHERE clause, SET SCAN commands, etc., can be used to specify which records will be placed in the input file. The input file may be created only on your own account.

The form of the DEFINE SET command is

 DEFINE SET setname [REPLACE] [DIRECT [EXTERNAL] [SCAN] [ALL]] [TV=ALL]...
 ... ELEMENTS [=] element-list [+ direct-list] [- direct-list]

or

DEFINE SET setname CONTINUE

"Setname" is the name of an ORVYL file that will hold the sort input data. If this is the name of an existing file, either the CONTINUE, APPEND (the same as CONTINUE) or REPLACE options should be used. (If an existing file is named, and REPLACE is not specified, then the system will ask permission to replace the file. See the examples below for more information on the CONTINUE option.) You may use a fully qualified ORVYL data set name ("ORV.gg.uuu.name" where "gg.uuu" must be your account number) or type only the "name" portion; the "name" portion may not exceed 33 characters in length.

The DIRECT option and its accompanying "direct-list" (a list of elements) are discussed in the section on direct sets. [See 1.8.]

The TV=ALL option specifies that all occurrences of the elements named in the element-list are to be processed, unless the element includes a TV=n option to override the global TV=ALL option. Use the TV=ALL option once before the ELEMENTS keyword as an alternative to including TV=ALL as an option for each element in the element-list.

The ELEMENTS keyword signals that a list of the elements to be used for sorting follows. Here is the syntax of the "element-list" portion of DEFINE SET:

ELEMENTS [=] elemname [(options)] [elemname [(options)] ... ]

For example, to define a set where entries are sorted first by CITY and then by ADD.DATE,

-> define set citysort elements = city(x,tv=all), add.date(d)

Up to sixteen elements may be specified, separated by commas or blanks. The "sort portion" of a sort record, i.e., the elements in the ELEMENTS list, can be about 1000 bytes long per record. (The total length of the sort record, including direct set information, can be 5500 bytes.) When creating sort records under GENERATE SET, SPIRES will start with the value of the first element and continue till the end of the list of elements or the 1000-byte limit is reached, whichever comes first.

The elements may include elements from phantom structures, as well as dynamic elements. With dynamic elements, however, be aware that SPIRES will create the dynamic element's value for the sort file, but will otherwise know nothing about the dynamic element. If you use the set later, to use the dynamic element, you will need to define it again; and since SPIRES will again compute its value when you want to see it, it may differ from its value at the time the set was created. (This is not a problem with direct sets.) [See 1.8.1.]

Use parentheses around groups of elements to specify that several different elements should be used for sorting at a particular level. For example, to specify the primary sorting element as AUTHOR or EDITOR (whichever occurs) and the secondary sorting element as TITLE, use a command such as:

-> define set namelist tv=all elements = (author editor) title

Options for Element-names

The element list is made up of a series of element-name/option pairs. There are several possible options, and all options following an individual element mnemonic must be enclosed in a single pair of parentheses. The possible options are any combination of the following:

D -- Specifies descending sort order for this element. (The default is ascending sort order.)

X -- Specifies that the values to be used for sorting are to be passed through the element's OUTPROC rules before sorting. (The default is to use the internal form of the value.)

I -- Used in conjunction with direct sets and the "X" option above, "I" specifies that the element should be sorted by its external value, but stored in the direct set in its internal value. [See 1.8.1.]

U -- Specifies an "unforced value". When this option is specified, upper-lowercase character strings will not be converted to all uppercase for sorting purposes; note that this only affects the sorting of values, not their display. The two sorted lists of values below demonstrate the difference this option can make:

Forced to upper for sorting:         Unforced:
  Aleph                               alpha
  alpha                               beta
  Beast                               Aleph
  beta                                Beast

Note that lowercase sorts before uppercase if values are not forced to uppercase. (The default is to force character string values to uppercase for sorting.)

The following additional options all have the form

option = n

where "n" is an integer. The "=" is optional, and may be replaced by a blank. The TV option is the one most commonly used; the others are used very infrequently -- their effects can be duplicated with various combinations of the SET FILTER command. [See 1.3.2.] If you are using SPISORT because you are trying to sort more records than the SEQUENCE command can handle, you need not use any of these options.

L=n -- Specifies the maximum length of values for sorting purposes. This option is meaningful only for TYPE=CHAR or TYPE=XEQ elements. Only the first "n" characters of the value will affect sorting. "N" may not exceed 255. (The default is 255; only the first 255 characters of any value can be sorted on, regardless of type.)

TV=n -- Specifies the maximum number of occurrences of the element that are to be processed (after skipping any occurrences as specified by TS; see below) for the entire record. Unlike the SEQUENCE command, the SPISORT program can sort on multiply occurring elements and structures; a single record, when sorted on one of its multiply occurring elements, can appear in several places in the sorted output file. "N" may be a number from 1 to 32,766, or the word ALL. (The default is 1.) A TV=n option in the element-list takes precedence over the global TV=ALL option before the ELEMENTS keyword, if present.

TS=n -- Specifies the number of the occurrence of the element or structure where sort-processing should begin. In other words, SPIRES should begin retrieving occurrences starting with the "nth" occurrence in the record. "N" may be any number from 0 to 32,767. The default is "1", which means to begin with the first occurrence. The sum of TV and TS may not exceed 32,767.

SV=n -- This option is useful only if TV is greater than 1, its default, and if the element is a multiply occurring element within a multiply occurring structure. This option specifies the number of occurrences of the element within each structure to retrieve. Its default is to retrieve all of them. (See examples below.)

SS=n -- Similar to TS=n, but applies to the element at a structural level. SS tells SPIRES to skip to the "nth" occurrence of the element within the structure before fetching sort values. (The default is SS=1, which means skip no values.)

Examples of DEFINE SET Commands

Below are some sample DEFINE SET commands:

-> define set test1 continue
       This command presumes that a DEFINE SET TEST1 has
       been issued before, specifying the sorting options
       desired.  No new options can be specified with the
       CONTINUE option.

-> define set outrec, elements = name zip-code
       A file called OUTREC is to be created to hold sort
       input data.  The records that go into this file will
       be sorted first by NAME, then by ZIP-CODE.

-> define set sortout, elems name(x) zip-code(tv=all)
       A file called SORTOUT is to be created to hold sort
       input data.  The records are to be sorted first by the
       value of the NAME element after it has been passed
       through the OUTPROC for that element in the file
       definition.  The records are then sorted by all
       occurrences of the ZIP-CODE element.  If multiple
       occurrences of ZIP-CODE appear in any one record, that
       record will appear in the set multiple times.

-> define set contacts elements = (home.city work.city) name
       The records in the CONTACTS set will be sorted first
       by city, whether the city is in the HOME.CITY element
       or the WORK.CITY element.  That is, the set will have
       interfiled HOME.CITY and WORK.CITY values.  The records
       are then sorted by NAME within each city.

In the following examples, the element ITEM.DATE is a multiply occurring element within the multiply occurring structure ITEM:

-> define set sortie, element = item.date(tv=all)
       The records in the set SORTIE will be sorted by all
       occurrences of the ITEM.DATE element.  Hence, if any
       one record contains multiple occurrences of ITEM.DATE,
       the record will appear in the set one time for each
       occurrence.  Note that TV=ALL means "all occurrences
       of the element in the record", even if the element is
       within a structure.  SV is not needed here (see below).

-> define set sortof, elem item.date(tv=all,sv=1)
       The records will be sorted by the first occurrence of
       the ITEM.DATE element within all occurrences of the
       ITEM structure.  TV=ALL is modified by SV=1: fetch
       all occurrences of ITEM.DATE that are the first
       occurrence within the ITEM structure.

-> define set sortoe, element item.date(tv=5,ts=3)
       Assuming an ample number of occurrences of ITEM.DATE
       in each record, the records will be sorted by the
       third through seventh (five) occurrences of ITEM.DATE.
       All the occurrences may be within the first occurrence
       of the ITEM structure, or they may be scattered across
       multiple occurrences of it.

In effect, the last three examples show that TV and TS, when used with an element in a structure, ignore structural boundaries, counting absolute occurrences of the element. Using SV and SS can re-establish those boundaries to some degree.

After defining the set, the next step is to put records into it, using the GENERATE SET command, discussed in the next section.

If you want to see the DEFINE SET command issued to create a set, issue the SHOW SET INFORMATION command. [See 1.6.]

1.3.2  The GENERATE SET Command

After the DEFINE SET command has been issued, you must tell SPIRES to place unsorted goal records into the set, which is done with the GENERATE SET command. Remember that the DEFINE SET and GENERATE SET commands can be issued only when Global FOR is in effect; GENERATE SET can be issued only after a DEFINE SET command has been issued.

The GENERATE SET command works in Global FOR like many other commands such as DISPLAY, REMOVE, etc. Its syntax is:

GENERATE SET [ALL|FIRST|*|NEXT|n|REST|LAST] [END='end clause']

The first options indicate which records in the Global FOR class are to be processed by the command. If no option is specified here, ALL is assumed; note that the default, ALL, processes all of the records into the input file.

A typical sequence of commands to create a sort input file might be:

-? select mail file
-? for tree where zip-code prefix 94
+? define set sortout, elements = zip-code name
+? generate set all
+? endfor
-'ORV.GQ.DOC.SORTOUT' has 3900 sort entries
-End of Global FOR
-?

An ENDFOR command (or a command that causes an END OF GLOBAL FOR condition, such as CLEAR SELECT) must occur before the SET can be sorted. Note that the response shown above to the ENDFOR command gives the name of the unsorted file (the set), as well as a count of the number of "entries" in it. The number of entries may or may not be equal to the number of records processed; if you are sorting on multiple occurrences of an element, then a single record may create multiple entries in the set (see Technical Note below).

The GENERATE SET command can be issued more than once while Global FOR is in effect. When Global FOR ends, or when Global FOR begins again, the set named in the DEFINE SET command is closed and the message giving the number of entries in it is output. If more sort input data is to be added to the same set, the DEFINE SET command will have to be issued with the CONTINUE option before any GENERATE SET commands are valid. Note that you cannot specify an element list if the CONTINUE option is used; the element list of the original DEFINE SET command that created the set is used. [See 1.3.1.]

You may set element filters (using the SET FILTERS command) to limit or control the occurrences of values used to create sets. The SET FILTER command(s) you need must be issued before the GENERATE SET command. For example, if you want to sort on the last occurrence of the MOD.DATE element in each record, you could issue these commands:

-> for result
+> define set lastmod, elements = mod.date, id.number
+> set filter for mod.date (last)
+> generate set
+> endfor
-'ORV.GQ.DOC.LASTMOD' has 397 sort entries
-End of Global FOR
->

Using filters as in the example, you can filter out occurrences of an element that you do not want to use for sorting.

The next step is to issue the SPISORT command, the subject of the next section. [See 1.4.]

Note that you can process the generated set under the FOR SET command even before it has been sorted, if desired. [See 1.5.]

Technical Information on the Creation of Sort Entries

How does SPIRES determine the number of "sort entries" to make for a given record? Do sort entries get made for a record having no occurrences of the sort elements?

The answer to the first question depends on several factors. The most important is whether the TV option is specified on any of the elements in the DEFINE SET command. If not, there will be one sort entry for each goal record processed by the Global FOR command GENERATE SET. (For example, the command GENERATE SET 5 would presumably process five goal records and create five sort entries.)

If any of the sort elements do not have a value in a record, SPIRES essentially assigns a null value to the element for sorting purposes. Even if none of the sort elements has a value, SPIRES still creates a sort entry for that record. It is important to realize that the DEFINE SET command does not determine what records deserve sort entries -- each goal record processed by the GENERATE SET command (as determined by the "FOR class" command, the WHERE clause, SET SCAN commands, and the GENERATE SET command itself) generates at least one sort entry. The DEFINE SET and SET FILTER commands may determine how many sort entries are created for each record, but at least one will be created for each goal record processed by GENERATE SET.

Returning abruptly to the question of how many sort entries are created for a given goal record, consider the effect of the TV option for an element on the DEFINE SET command. For example,

-> define set fbifile, element pseudonyms(tv=all)

If a goal record processed by the GENERATE SET command has two occurrences of the PSEUDONYMS element, then two sort entries will be created -- one for each value. But consider this example:

-> define set fbifile, elem pseudonyms(tv=4)

From one to four sort entries will be created for each record, depending on the number of occurrences of PSEUDONYMS. (Remember, if there are no occurrences of PSEUDONYMS in a record, one sort entry will be created.)

If filters are in effect, they are applied first, before the TV, SV, TS and SS options are applied.

The number of sort entries grows substantially when two or more sort elements are multiply occurring. You can use a simple formula to determine that number for any given goal record:

1) For each element in the sort list, determine the number of sort entries created if it were the only element listed. Remember that at least one entry would be created.

2) Multiply those numbers together for the total number of sort entries for that record.

Below is a simple example. Suppose that you want to know how many sort entries will be created for record ABC that has 10 each of the three sort elements named in the DEFINE SET command:

-> stack ABC
-Stack: 1 RECORD
-> for stack
+> define set test, element x(tv=all), y(tv=3), z(tv=2,ts=5)
+> generate set
+> endfor
-'ORV.GQ.DOC.TEST' has 60 sort entries
-End of Global FOR
->

If element X were the only sort element, 10 entries would be created; if element Y, 3 would be created; and if element Z, two would be created. Multiplying them together creates the product 60.

1.4  Sorting the Input File: the SPISORT Command

To sort the input set and create a sorted output file, issue a SPISORT command. The syntax is:

SPISORT infile [TO outfile] [REPLACE] [TIME = n] ...
  ... [ORDER = (sort order list)]

The SPISORT command verb may not be abbreviated.

"Infile" (which is required) is the name of the input file created by your DEFINE SET command.

"Outfile" is the name of the output file to contain the sorted data. If you omit the "TO outfile" parameter, then your input file will be overwritten by the sorted output file.

If you do name an output file, the REPLACE parameter may be used to specify that an existing file of that name may be replaced with the new data. (If you omit the REPLACE parameter and the file already exists, you will be asked whether it's ok to replace the file.)

With the TIME option, you may increase the time allotted for the sort job. The default is 1 minute. (Note that the sort job may not run in CLASS=F with a high time limit. This could cause the sort job to queue, especially if the SPISORT command is being issued from a job (e.g. Batwyl).

The ORDER option allows you to reorder the sort elements specified in your DEFINE SET command. Only elements named in the ELEMENT list (not the "direct" list; see 1.8) may be named here. The parentheses around the order list are optional, but if they are omitted, the ORDER parameter must be last in your SPISORT command.

Each element may be followed by "(A)" or "(D)" to indicate ascending or descending order; whichever was specified for the element in the DEFINE SET command is the default.

The elements may be specified by name (the name given in the DEFINE SET command, though any structural path information should be omitted) or number, where the number is the element's position in the ELEMENT list in the DEFINE SET command (see example below).

By the way, if the element names in the DEFINE SET command are themselves numbers, then SPISORT will assume the numbers in the ORDER parameter are element names, not position numbers. Naturally, however, the possibility of confusion is much greater if you use numbers as element names in the DEFINE SET command, particularly if you might need to use the ORDER parameter. You can see what DEFINE SET was issued to create a set by issuing the command SHOW SET INFORMATION. [See 1.6.]

Examples of SPISORT commands

Here are a few examples of SPISORT commands:

-> spisort books

      In this simple form, the input data in BOOKS will be
      sorted, and the sorted output will overwrite the
      input file.

-> spisort booksin to booksout

      Here, separate input and output files are specified.
      If BOOKSOUT already exists, SPIRES will ask if it is
      OK to replace it.  (You could avoid the question by
      including the REPLACE option.)

-> spisort books time=2 order = price(d),author

      The ORDER option here determines the sorting elements.
      The sort order here overrides the sort order specified
      in the DEFINE SET command.

The SPISORT Command Runs a Batch Job

The SPISORT command actually constructs JCL and runs a batch job on your behalf, to accomplish the sorting of your input file. For this reason, your "current job" number will be affected when you issue a SPISORT command. In addition, SPISORT makes use of the $ASK, $WDST, and $WDSR variables.

1.4.1  SPISORT Error Codes

If a SPISORT command fails, an error code will be supplied. Listed below are the possible codes for SPISORT failures. You may get online explanations, too, with the EXPLAIN command. For example:

-> spisort boooks
-SPISORT terminated, code U119
-Not a legal or complete command
-> explain u119
IN=file does not exist

The error code is also recorded in the $SORTCODE variable, and $NO is set to true when the SPISORT command fails.

SPISORT Error Codes and Explanations

0001  Attempt to define the same parameter twice,
      or a single parameter value exceeds 22 characters.
0002  Invalid parameter option.
0003  Either IN or OUT (or both) have not been specified.
0004  Main block of IN not correct.  Did you specify the
      DEFINE SET file name as the IN=file ?
0005  IN file does not define any data to sort.  Did you
      GENERATE SET after doing the DEFINE SET?
0006  Data blocks of IN not correct.  Did you PUT APPEND
      something to the IN=file ?
0007  OUT=file exists and REP option not specified.
0008  SORT did not output the required number of records.
0009  IN=file did not contain all the records to sort that
      it was supposed to contain.
0010  ORD field in the parm list is invalid.  Are the element
      names or numbers valid?  Did you specify a sort option
      other than "(A)" or "(D)"?
0011  IN=file did not have any sort fields.
0090  ORVYL file system unavailable.
0094  IN=file name illegal, probably invalid characters.
0096  Permanent I/O error on IN=file.
0097  Permanent I/O error on OUT=file.
0099  IN=file not available.  Did you ENDFOR before running?
0105  OUT=file not available.  Did you have it attached?
0107  IN=file access not permitted.  Is the file yours?
0108  IN=file read access prohibited.
0109  OUT=file write access prohibited.
0112  IN=file missing a required block.
0117  OUT=file storage limit exceeded.  Get more ORVYL blocks.
0119  IN=file does not exist.
0121  ORVYL out of block space.  Tell systems.
0194  OUT=file name illegal, probably invalid characters.
0198  OUT=file not available.  Do you have it attached?
0206  OUT=file access not permitted.  Is it yours?
0210  OUT=file storage limit exceeded.  Get more ORVYL blocks.
0214  OUT=file name overflows dictionary.  Tell systems.
0226  OUT=file overflows system tables.  Tell systems.
0230  ORVYL out of block space.  Tell systems.
1000  The SPISORT failed to parse correctly. You may have
      improperly spelled parameters.
1001  SPISORT could not sort your set.

1.4.2  (*) Running a Batch SPISORT Job

The interactive SPISORT command was implemented in March 1989. Prior to that time, the sorting step of the SPISORT procedure had to be accomplished by running a batch job to execute the SPISORT program. (In fact, the SPISORT command runs a batch job, constructing the JCL and submitting the job on your behalf.)

It is still possible, of course, to run the batch job yourself. The paragraphs below describe the JCL to run a batch SPISORT job.

To sort the input set and create a sorted output file, a batch job must be run using the following JCL to invoke the SPISORT program:

//  JOB
//  EXEC SPISORT,PARM='parameters'

The parameters specify the name(s) of the input and output files, and whether an existing file is to be replaced by a new output file. The forms of the parameters are:

PARM='IN=OUT=setname'
  The input file, called "setname", is overwritten by the
  sorted output file.

PARM='IN=setname1,OUT=setname2'
  The input file "setname1" is read, and the output file
  "setname2" is created.  "Setname2" must not exist; if it
  does, use the next example.

PARM='IN=setname1,OUT=setname2,REP'
  A file called "setname2" is to be created.  If it already
  exists, it is replaced.

PARM='IN=setname1,OUT=setname2,REP,ORD=(sort-list)'
  The "sort-list" allows you to reorder the sort elements
  specified in the DEFINE SET command.  Only elements named
  in the ELEMENT list (not the "direct" list; see 1.8) may
  be named here.  The elements may be specified by name (the
  name given in the DEFINE SET command, though any structural
  path information should be omitted) or number, where the
  number is the element's position in the ELEMENT list in the
  DEFINE SET command (see example below).  Each element may
  be followed by "(A)" or "(D)" to indicate ascending or
  descending order; whichever was specified for the element
  in the DEFINE SET command is the default.

Here is an example using the ORD parameter. Suppose your DEFINE SET command was:

-> define set eyes, elements color size weight(D)

You could change the sort order later on the EXEC card:

//  JOB
//  EXEC SPISORT,PARM='IN=OUT=EYES,ORD=(3(A),1,2)'

SPISORT will sort the elements in the order WEIGHT (in ascending order), COLOR and SIZE.

By the way, if the element names in the DEFINE SET command are themselves numbers, then SPISORT will assume the numbers in the ORD parameter are element names, not position numbers. Naturally, however, the possibility of confusion is much greater if you use numbers as element names in the DEFINE SET command, particularly if you might need to use the ORD parameter. You can see what DEFINE SET was issued to create a set by issuing the command SHOW SET INFORMATION. [See 1.6.]

Issue the WYLBUR command RUN to submit the SPISORT job for execution.

If your batch job fails, a SPISORT RETURN code will be supplied. The codes are the same as those listed earlier for the SPISORT command. [See 1.4.1.] The code is prefixed with a "U" in the HASP job log messages.

1.5  Processing the SPISORT Output File: the FOR SET Command

Sets of records created by the DEFINE SET and GENERATE SET commands are processed under the Global FOR command FOR SET. That command has the following form:

FOR SET setname [UNFILTERED|DIRECT] [WHERE clause]

or

FOR source VIA SET setname [UNFILTERED|DIRECT] [WHERE clause]

where "setname" is the name of the ORVYL data set created by the DEFINE SET and GENERATE SET commands or is a stored result or stack. If the set is stored as an ORVYL data set under some account other than your own, then use the form "FOR SET ORV.gg.uuu.setname" where "gg.uuu" is the account number under which the set is stored. For more information about using a stored result or stack with FOR SET, see the "Global FOR" manual; online, EXPLAIN FOR SET COMMAND, IN GLOBAL FOR.

The "source" may be any of the usual Global FOR access classes, e.g. SUBFILE, TREE, UPDATES, etc. In the first form, SUBFILE is assumed to be the source.

The UNFILTERED option tells SPIRES to process the entire goal record, rather than the version whose sort elements are filtered by path occurrence information gathered when the SPISORT sort entries were created. This option is explained in detail in the next section. [See 1.7.]

The DIRECT option is used when you created a direct set (by using the DIRECT option on the DEFINE SET command). The DIRECT option tells SPIRES not to retrieve the goal record that a sort entry in the set represents, but instead to use the data actually in the sort entry. Hence, "FOR source VIA SET setname DIRECT" is exactly the same as "FOR SET setname DIRECT" -- the "source" is always the direct set when DIRECT is specified. Direct sets are discussed in a later section. [See 1.8.]

1.6  The SHOW SET INFORMATION (SHO SET INF) Command

You can get information about generated sets stored on your account by issuing the SHOW SET INFORMATION command:

SHOW SET INFORMATION [setname]

You can issue this command without the "setname" option if you are processing records under a FOR SET command; if not, you must name the set you want information about.

The information displayed includes the complete DEFINE SET command you issued when creating the set, the number of sort entries in the set, the date and time it was created, etc., as you can see from the example below:

-> show set information sortfight

DEF SET SORTFIGHT rep, elem opponent(tv=all), date

Set ORV.GQ.JNK.SORTFIGHT created 06/22/1984
  from subfile FENCING of file ORV.GQ.DOC.FENCING
The set has not been sorted
  SORT order: OPPONENT, DATE
The set has     192 records              5 blocks

->

The first line shows the DEFINE SET command used to create the set. The next line shows the ORVYL data set containing the sort data, and the date, time and account that created it. The next line tells what file and subfile the data comes from. (No subfile is listed if the set was generated after an ATTACH command rather than a SELECT command.)

The next line announces whether the set has been sorted or not, with the next line telling by which elements the set was sorted. (Remember that the ORDER parameter on the SPISORT command allows you to specify a different order for sorting than appears in the DEFINE SET command.) Finally, at least in the example, the display tells you how many sort entry records were generated for the set and how many ORVYL blocks the set uses for storage.

For direct sets, discussed elsewhere, SPIRES tells you the minimum, maximum and average length of the sort entry records at the end of the display. [See 1.8.]

1.7  Filters and SPISORT

When a set is created with the DEFINE SET and GENERATE SET commands, "path information" is created to tell SPIRES how to retrieve appropriate element occurrences involved in the sorting. SPIRES uses the path information when records are processed under the FOR SET command; by default, only the particular occurrence of the element that caused the record to be sorted in that position will be retrieved when the record is displayed. (When multiply occurring elements or structures are named in the DEFINE SET command and TV other than 1 is specified, multiple copies of a single goal record in the set may be created.) This "automatic filtering" applies to formatted or non-formatted output commands.

For example, suppose a set is defined to sort records on the multiply occurring element EMPLOYEE. A record containing two employees appears twice in the set, but each time the record is displayed under the FOR SET command, only a single employee's name appears:

-> for subfile
+> display first
 OFFICE = ACCOUNTING;
 EMPLOYEE = Eight, Sharon;
 EMPLOYEE = Nine, Calvin;
+> define set names element employee
+> generate set *
+> endfor
-'ORV.GQ.DOC.NAMES' has 2 sort entries
-End of Global FOR
-> for set names
+> display all
 OFFICE = ACCOUNTING;
 EMPLOYEE = Eight, Sharon;

 OFFICE = ACCOUNTING;
 EMPLOYEE = Nine, Calvin;

SPIRES automatically filtered out the occurrences of EMPLOYEE that did not cause the record to appear at that position in the set. The automatic filtering occurs regardless of whether the set is sorted or not -- the path information is created when the set is created, not by the SPISORT program.

The UNFILTERED option can be added to the FOR SET command if the automatic filtering described above is not desired:

+> for set names unfiltered
+> display all
 OFFICE = ACCOUNTING;
 EMPLOYEE = Eight, Sharon;
 EMPLOYEE = Nine, Calvin;

 OFFICE = ACCOUNTING;
 EMPLOYEE = Eight, Sharon;
 EMPLOYEE = Nine, Calvin;
+>

Explicit filtering can be done with the SET FILTER command, which may be used to control the sort entries created for a goal record when the set is created. For example, compare these two sets of commands applied to the same subfile:

1) -> for subfile
   +> define set parties, elements guest(tv=all) date
   +> generate set first
   +> endfor
   -'ORV.GQ.DOC.PARTIES' has 25 sort entries
   -End of Global FOR
   ->

2) -> for subfile
   +> define set parties, elements guest(tv=all) date
   +> set filter for guest(1/5)
   +> generate set first
   +> endfor
   -'ORV.GQ.DOC.PARTIES' has 5 sort entries
   -End of Global FOR
   ->

The command sequences are identical, except for the SET FILTER command in the second example, which tells SPIRES to treat the goal records as if only the first five occurrences of the GUEST element exist.

When you use explicit SET FILTER commands on the sort elements to create a set, you should clear those filters when you process the set. (Note that SPIRES will be forgiving if you leave those filters set; that is, the output will be the same whether they are set or not. However, if you were to set other filters on the sort elements when processing the set, the results might be unpredictable. Hence, it is recommended that you not set any filters on the sort elements when you are processing the set under FOR SET.)

Information about filters with direct sets appears in the next section. [See 1.8.] General information about element filters appears later in this manual. [See 21.] The path information may be applied selectively in custom-designed SPIRES formats, if desired. See the manual "SPIRES Formats" for details on the PATH and NPATH options.

1.8  Direct Sets

Under FOR SET processing, SPIRES extracts from each sort entry a pointer, used to retrieve a goal record for processing. A special type of set, called a "direct set", can be created that already contains the goal record data desired for FOR SET processing -- that is, SPIRES does not need to retrieve each goal record for processing, because the desired data for display is already in the set.

Processing a direct set under FOR SET is more efficient than processing a non-direct set. Direct sets are a benefit primarily to large sorting applications, where thousands of sort entries are sorted and processed; and to sorting applications where the same set is sorted multiple times for multiple reports.

The subsections of this section will discuss direct sets in detail. The first subsection will cover the additional options on the DEFINE SET command that are used to create direct sets. [See 1.8.1.] The next will discuss the processing of direct sets, using the DIRECT option on the FOR SET command. [See 1.8.2.] Following that will be a discussion on using filters with direct sets. [See 1.8.3.]

1.8.1  Direct Sets and the DEFINE SET Command

Direct sets are created by adding the DIRECT option to the DEFINE SET command, and used by adding the DIRECT option to the FOR SET command. [See 1.8.2.] Here again is the syntax of the DEFINE SET command:

  DEFINE SET setname [REPLACE] [DIRECT [EXTERNAL] [SCAN] [ALL]] [TV=ALL]...
   ... ELEMENTS [=] element-list [+ direct-list] [- direct-list]

REPLACE, "setname", TV=ALL, and "element-list" were discussed in the section on the DEFINE SET command. [See 1.3.1.]

The DIRECT option tells SPIRES that the set should be a direct set. The other options relating to direct sets, which are discussed in detail below, are:

 - EXTERNAL -- tells SPIRES  to  store  the  external  form  of  all  elements  named  in  the
 "element-list"  and  the  "direct-list" as part of the direct data.  It
 does not affect the "sort" form  used  for  the  elements  in  the  "element
 list",  which  means  that  elements specified in their internal form for sorting will
 have both forms stored in the set: internal for sorting and external  for  displaying.   It
 can  be overridden on an element by element basis in the "direct-list" by putting
 "(I)", for "internal", after  the  element  name;  an  example  appears
 below.

 - SCAN -- Use the WHERE clause in effect on the Global  FOR  command  to  control  what  sort
 entries are created.  If the generated sort entry would not pass the WHERE clause criteria,
 it is not added to the set.

 - ALL -- All the data in the record that passes through the  filtering  in  effect  when  the
 GENERATE SET command is issued will go into the sort entries.

 - + direct-list -- This is a list of the  elements  to  be  included  in  the  set  for  each
 generated  sort  entry.  These "direct-set" elements are tag-alongs; they are not
 used for sorting; all sorting elements must appear in the  "element-list".   Each
 element in the "direct-list" will be added to the set in its internal form unless
 it  is  followed  by  "(X)", or unless the EXTERNAL option is specified.  (If the
 EXTERNAL option is specified, an element can be followed by "(I)" to request  the
 internal form for that element.)

 - - direct-list -- This is a list of elements to be excluded from the set for each  generated
 sort  entry;  it  is  useful  when  you  have specified ALL (see above) but want or need to
 eliminate one or a few elements.

Determining Which Elements Go Into the Direct Set

All the elements in "element-list" -- that is, the sort elements -- will be included in the direct set, so in general, there is no reason to list them also in the "direct-list" (but see the notes below). The other options, ALL and the "direct-lists", let you add other elements from the goal records to the direct set, in a manner similar to the SET ELEMENTS command. For example, if DIRECT ALL is specified with no "direct-lists", then all the data in the record that passes through the filtering in effect when the GENERATE SET command is issued will go into the sort entries generated for the direct set.

Important: DIRECT ALL does not mean that all occurrences of a multiply occurring sort element will go into the set -- only the occurrence of the element that is generating that entry will. (The others are being filtered out, and hence are not included in the sort entry.) If you will need to display all occurrences of a sort element, rather than just the occurrence for that sort entry, do not use direct sets.

The "direct-list" options can be used to add or subtract elements for storage in the direct set. In general, if you use the ALL option, you might subtract elements you did not need to be included in the set; if you do not use the ALL option, you might need to add elements to the direct set. You can, however, add or subtract elements as desired. For instance, you can subtract an entire structure from ALL, and then add back a single element within it:

-> define set x direct all, elems a b c - d + d@e

All the elements in the goal record except structure D but including element E in structure D will go into set X.

Any elements defined in the file definition for the goal record can be named as sort elements or "direct elements"; additionally, elements within phantom structures may be named in a DEFINE SET command for a direct set.

Dynamic elements, including elements from dynamically-defined phantom structures, may also be included. SPIRES will figure out the value for a dynamic value at the time the set is generated, storing that value in the set. Later, under "FOR SET name DIRECT", SPIRES will use the stored value as the element value. [See 1.8.2.] In fact, you don't need to redefine the element when you use the "FOR SET name DIRECT" command; if you do, SPIRES will ignore it (and any other dynamic element definitions) till after the next ENDFOR. (If you SHOW DYNAMIC ELEMENTS under "FOR SET name DIRECT", SPIRES will show it simply as "DEFINE ELEMENT name", with no further information, which indicates that definition is irrelevant under this form of FOR SET processing.)

The SCAN Option: Controlling Which Sort Entries are Created

The SCAN option lets you apply WHERE clause processing as the sort entries are created for the direct set. SCAN causes the GENERATE SET command to apply the current WHERE clause criteria to each sort entry to see if the entry should be included in the direct set.

Here is a very simple example to illustrate the effect. Here is a sample record:

ID = 1;
A = Apple;
A = Orange;
B = Cat;
B = Dog;

And here are commands to select records and create the set, first without the SCAN option:

-> for subfile where a = apple and b = cat
+> define set apples direct tv=all elements=a,b
+> generate set
+> endfor
-'ORV.GQ.JLS.APPLES' has 4 sort entries
-End of global FOR
+> for set apples
+> display all
ID = 1;
A = Apple;
B = Cat;

ID = 1;
A = Apple;
B = Dog;

ID = 1;
A = Orange;
B = Cat;

ID = 1;
A = Orange;
B = Dog;

The WHERE clause was applied to choose records to go in the direct set, but the set has entries for combinations besides the "apple/cat" one.

By adding the SCAN option to the DEFINE SET command, the WHERE clause will be applied as the set entries are created:

-> for subfile where a = apple and b = cat
+> define set apples direct scan tv=all elements=a,b
+> generate set
+> endfor
-'ORV.GQ.JLS.APPLES' has 1 sort entry
-End of global FOR
+> for set apples
+> display all
ID = 1;
A = Apple;
B = Cat;

Note that the same effect could have been achieved by adding a WHERE clause to the FOR SET command in the first example (FOR SET APPLES WHERE A = APPLE AND B = CAT). The difference is that in the second example, only one entry was created at all for the direct set. The WHERE clause processing happened when the set was created (GENERATE SET), not when the set was used (FOR SET).

The SCAN option must immediately follow the word DIRECT in the DEFINE SET command.

External and Internal Forms of Sort and Direct Elements

When generating a set, SPIRES by default places the internal form of both sort elements and direct elements into the set. For sort elements, however, you may want SPIRES to sequence on the external form, in which case you include the "X" parameter in the sort options for the element:

+> define set sortsox, element color(X)

Similarly, you can request that a direct element be put in the set in its external form:

+> define set sortsox direct, element color(X) + size(X)

This technique is primarily useful when you will be processing the direct set multiple times -- the element does not have to be processed through its OUTPROC rules for each report. That can save a great deal of processing, particularly when the OUTPROC rules are complex, e.g., requiring other records to be fetched for table lookups (the $LOOKUP and $SUBF.LOOKUP procs).

You can request that all the elements in the "direct-list" and in the "element-list" be saved in the set in their external forms for output by adding the EXTERNAL option to the DEFINE SET command, following the DIRECT option. Important: That does not affect what form is stored for each element in the "element-list" for sorting purposes. In other words, if EXTERNAL is specified for the direct set and all the elements in the "element-list" for sorting are to be sorted in their internal form, then SPIRES will place both the internal and external form of each in the set.

You can override the EXTERNAL option for elements in the "direct-list" on an individual basis by adding the "(I)" option after its name in the list. For example:

-> define set sortaray direct external elem type + name contact(i)

In that case, SPIRES will generate sort entries where the TYPE element is stored in its internal form for sorting, and the TYPE element is stored in its external form for displays. Also, the NAME element will be stored in its external form; and the CONTACT element will be stored in its internal form. That might be useful in a table-lookup situation where you want the lookup to be done at the time of the display under FOR SET, not at the time the set is generated. (See below.)

Note these implications of storing the external form of a direct element in a set -- the direct element behaves as if it had no processing rules at all, which means:

 - If an element normally does a lookup to another record-type to get its value  on  output,
 that value will be looked up at the time the set is generated, not at the time the set is
 used.   If that is a problem, you can either request the internal form of the element, so
 that the lookup must be done when the set is processed, or process the set under FOR  SET
 without the DIRECT option.

 - both $UVAL and $CVAL will be the same in a format label group containing  a  GETELEM  for
 that element;

 - the functions $GETUVAL, $GETCVAL, $GETIVAL and $GETXVAL, used in a format, will  retrieve
 the same value for that element;

 - if you need the internal form, you will probably have to use the  $PROCSUBG  function  to
 retrieve it.

Be aware that elements defined as "OUTPROC-required" by the file definition can be placed in the set only in their external form, and the above implications will apply to them too.

You can request that a sort element be sorted by its internal form but be stored as a direct element in its external form by explicitly adding it to the direct element list, e.g.,

+> define set sortsox direct, element color + color(X)

Conversely, you can request that the sort element be sorted by its external form but be stored as a direct element in its internal form in one of the following ways:

+> define set sortsox direct, element color(X) + color(I)
 or
+> define set sortsox direct, element color(X,I)

The "I" option indicates that the element should be carried as a direct element in its internal form.

You can specify that SPIRES not include a sort element in the collection of direct elements by using the "- direct-list" option, as in this example:

+> define set sortsox direct, elem=sex,size,color - color

The COLOR element would be used as a sort element but would not be stored as a direct element, and thus would not be accessible under direct set processing.

Virtual Elements, Hidden Elements

Virtual elements can be specified as either sort or direct elements. They become "real" elements when the set is created, in that their current value (at the time the set is created) is stored in the set. Either the internal or external form may be specified for the set. The external form for direct set storage is determined by executing the virtual element's OUTPROC rules; the internal form is determined by executing its OUTPROC rules and then its INPROCs.

You cannot use sets to circumvent security provisions. For example, elements whose values are hidden from your account may not be placed into a direct set. SPIRES may not display an error message when you issue the DEFINE SET command, but the GENERATE SET command will certainly not place the hidden data into the set.

Direct Sets are Larger

The GENERATE SET command works the same for direct sets as for non-direct sets, except that the generated set will be larger. [See 1.3.2.] There is an absolute limit of 5500 bytes per sort entry, but as sort entries become very large, they cause inefficient processing. Hence, do not include non-sort elements in a direct set just to carry them along; be sure you need them in your final product.

The special form in which SPIRES stores direct data in the set requires eight extra bytes of overhead for each element or structure. It may thus be more efficient to place entire structure occurrences in the set (eight bytes for each occurrence of the entire structure) than a few individual elements from the structure (eight bytes for each element value). Of course, for overall storage savings, it is best to erase sets as soon as you no longer need them, since they are duplicating data already stored. [See 1.2.]

1.8.2  Direct Sets and the FOR SET Command

Direct sets are processed like other sets -- under the FOR SET command. However, you must use the DIRECT option to request that the set be processed as a direct set, rather than just as a regular set.

Here again is the syntax of the FOR SET command:

FOR SET setname [DIRECT|UNFILTERED] [WHERE clause]
  or
FOR source VIA SET setname [DIRECT|UNFILTERED] [WHERE clause]

Only the DIRECT option is discussed here. [See 1.5.]

Even if a set is a direct set, it can be processed as if it were a regular set, by omitting the DIRECT option. However, if the DIRECT option is specified, SPIRES will not retrieve the goal record that a sort entry points to, but will instead use the data within the sort entry for any record processing under FOR SET.

Only the following Global FOR record processing commands may be used to process "direct records": DISPLAY, SHOW KEYS and SKIP. Most Global FOR commands will not work when a direct set is being processed, such as TRANSFER, REMOVE, MERGE, DEQUEUE, UNQUEUE and REFERENCE. Note that the STACK command does work, though any stack created will retrieve goal records, not direct records.

You should not create a direct set unless all of the data you will need for reporting is included in the lists of sort elements or direct elements. Otherwise, you will be unable to use the DIRECT option, since you will need other data in the goal records. It is easy, for example, to forget elements that are retrieved by a report format with a $GETxVAL function or by the $GETELEM (A79) processing rule.

Virtual elements are handled in an interesting way in direct sets. They become "real" elements when the set is created, in that their current value (at the time the set is created) is stored in the set. Either the internal or external form may be specified for the set. If you display a direct set record in the SPIRES standard format, any virtual elements in it will be displayed too, without you having to set them explicitly in a SET ELEMENTS command.

Any direct element stored in its internal form is converted to external form by processing it through its OUTPROC rule string. This is true even for virtual elements -- the internal form (which was created under GENERATE SET by executing the element's OUTPROC rules and then its INPROC rules) is run through the OUTPROC rules again to get the external form.

Since you are allowed to issue the GENERATE SET command during direct set processing, you can create a direct set from another direct set. This technique can be considerably more efficient than generating multiple sets from the goal records for several different reports. If you need to sort the goal records several times on different elements, consider creating a direct set for the first one and creating other direct sets from it for the others.

There is no guarantee that using direct sets will save you money. If set entries are very large because you have many direct elements, the overhead to process them could offset the savings from not processing the goal records. If you are concerned about costs and your set will have many direct elements and/or sort elements, you should certainly run some tests using real data before deciding to use direct sets.

1.8.3  Direct Sets and Filters

If you have explicit filters set when you begin processing a direct set, SPIRES will clear those filters:

-> set filter for wearer(first)
-> for set sortsox direct
-Warning: Filters have been cleared until ENDFOR
+> show filters
-No filters found
+> endfor
-Filters have been reestablished
-End of Global FOR
-> show filters
SET FILTER FOR wearer(first)
->

SPIRES clears the explicit filters because when you start processing a direct set, you are in effect working with an entirely different type of record than when you were working with the subfile's goal records, i.e., when the explicit filters were set. Such explicit filters for the goal records could cause problems if they were applied to the direct set. For example, such filters would not work properly with direct elements stored in their external forms.

Although pre-existing explicit filters are cleared when the direct set processing begins, you can set other filters (with the SET FILTER command) after issuing the "FOR SET setname DIRECT" command. Thus, you are allowed to continue filtering the direct set, but only after direct set processing has been initiated. As soon as an ENDFOR condition occurs, however, these filters are discarded, and the explicit filters that were automatically cleared are reestablished by SPIRES, as shown in the example above.

1.9  Display Sets

Display sets are a variation of direct sets. [See 1.8.] Display sets are created with the DEFINE SET and GENERATE SET commands, but unlike direct sets, display sets are not placed in ORVYL data sets. Instead, you specify an output area in which the display set should be placed (e.g. your terminal screen or your active file). The GENERATE SET command sends the set directly to the output area, so you do not need to use the FOR SET command to use the set.

Since display sets do not produce ORVYL data sets, you can't use SPISORT to sort the records in the set. But you may be able to take advantage of the automatic sorting in your record keys (using FOR SUBFILE) or in an index (using FOR INDEX) to control the order in which entries in the set appear. Or, since you can generate a set from a record stack, you could use the SEQUENCE command to arrange records in a particular order before you use the GENERATE SET command.

An advantage of this difference between display sets and normal sets is that display sets do not have the 5500-byte size limit that normal sets have.

Display sets are created by adding a DISPLAY option to the DEFINE SET command. Here is the syntax for display sets:

DEFINE DISPLAY SET [SCAN] [ALL] [TV=ALL] [EXTERNAL] ...
 ... ELEMENTS [=] element-list [+ display-list] [- display-list]

The ALL, TV=ALL, EXTERNAL, and "element-list" options were discussed earlier. [See 1.3.1.] The "display-list" options are similar to the "direct-list" options for direct sets. [See 1.8.1.] They let you add or subtract elements for inclusion in the display set. Note that you do not supply a setname for a display set.

The SCAN option works the same as for direct sets. It lets you apply WHERE clause processing as the sort entries are created for the display set. SCAN causes the GENERATE SET command to apply the current WHERE clause criteria to each sort entry to see if the entry should be included in the display set. [See 1.8.1 for an example.] The SCAN option must immediately follow the words DISPLAY SET.

Use the GENERATE SET command to send the display set to an output area:

[IN ACTIVE|areaname] GENERATE SET [ALL|FIRST|*|NEXT|n|REST|LAST]

If you omit the "IN areaname" prefix, the generated display set will be displayed at your terminal. The second option indicates which records in the Global FOR class are to be processed for the display set. The default is ALL records in the class.

Here is an example showing how to create a display set. Suppose you have a subfile whose goal records consist of a supply item, with structures for each order made for the item. The structures naturally occur in chronological order. You can create a set with an entry for each order this way:

-> display 1
 ID = 1;
 ITEM = Typewriter ribbon;
 ORDER;
   DATE.ORDERED = 03/21/87;
   SUPPLIER = Congden and Crome;
   QUANTITY = 3;
 ORDER;
   DATE.ORDERED = 04/01/87;
   SUPPLIER = Smith Brothers;
   QUANTITY = 5;
 ORDER;
   DATE.ORDERED = 05/04/87;
   SUPPLIER = Congden and Crome;
   QUANTITY = 1;
-> for index item
+> define display set elements = date.ordered(tv=all) + item id qty
+> generate set all
 ID = 2;
 ITEM = Light bulbs;
 ORDER;
   DATE.ORDERED = 03/21/87;
   QUANTITY = 3 boxes;

 ID = 2;
 ITEM = Light bulbs;
 ORDER;
   DATE.ORDERED = 04/13/87;
   QUANTITY = 2 boxes;

 ID = 2;
 ITEM = Light bulbs;
 ORDER;
   DATE.ORDERED = 05/12/87;
   QUANTITY = 5 boxes;

 ID = 4;
 ITEM = Quadrille pads;
 ORDER;
   DATE.ORDERED = 01/05/87;
   QUANTITY = 25;

 ID = 4;
 ITEM = Quadrille pads;
 ORDER;
   DATE.ORDERED = 03/04/87;
   QUANTITY = 13;

 ID = 3;
 ITEM = Rubber stamp;
 ORDER;
   DATE.ORDERED = 06/16/87;
   QUANTITY = 1;

 ID = 1;
 ITEM = Typewriter ribbon;
 ORDER;
   DATE.ORDERED = 03/21/87;
   QUANTITY = 3;

 ID = 1;
 ITEM = Typewriter ribbon;
 ORDER;
   DATE.ORDERED = 04/01/87;
   QUANTITY = 5;

2  An Alternate Form of the Subfile Name

Since file owners have almost complete freedom in choosing subfile names, the possibility always exists that you may have access to two or more subfiles with the same name. If you SELECT one of them, SPIRES will ask you which one you meant, showing you the file names of the subfiles to distinguish between them. Alternatively, when you SELECT one of the subfiles, you can specify all or part of the file name before the subfile name, in order to uniquely identify the desired subfile. This alternate syntax for the subfile name is also available:

 - in SPIBILD for the "ESTABLISH subfile-name" command

 - in the $LOOKSUBF function

 - in system proc $SUBF.LOOKUP (action A65)

 - in the SUBFILE statement in a format definition

 - in the SUBFILE statement in the PHANTOM section of a file definition

The syntax for the subfile name in these commands is:

[&gg.uuu.filename] subfile-name

where the bracketed material is the name of the file as it appears in the first line of the file definition. You may also include a comma after the "filename" if it helps clarify the command syntax for you.

Starting from the left end, only as much of the file name as is needed to distinguish the subfile name from any other is necessary. Thus, for a file named "GA.JNK.MUSIC", you could specify "&GA.JNK.MUSIC" or as little as "&G". The "&" (ampersand) character is required if any part of the file name will be used, since it tells SPIRES that what follows immediately is a file name rather than a subfile name, which is expected.

Here are examples using the file name "GA.JNK.MUSIC" and a "RECORDS" subfile:

SELECT &GA.JNK.MUSIC, RECORDS
ESTABLISH &GA.JNK RECORDS
SELECT &G RECORDS

Though it is not recommended, an even shorter method is available. In some cases, if the file being named is your own, you can replace your account number with an asterisk (*) or a period (.), as in "&*MUSIC" or even "&.". This technique is not recommended when you are writing code that will be executed by other users, since SPIRES may try to substitute their account numbers instead of yours.

3  Sharing Records in SPIRES System Subfiles with Other Users

All of the SPIRES system subfiles, such as FILEDEF and FORMATS, control access to their records by means of the record key, which begins with the account number of the user owning the record. By default, only you, the owner, can display and update your records in those subfiles.

Using a system subfile called METACCT ("met-account"), however, you can allow other users to display and possibly update your records in almost all system subfiles. Hence, you can share your system-subfile records with other users, letting them examine or copy a file definition, for example, or add their own procs to an EXTDEF record of yours. You simply add a record to the METACCT subfile that tells SPIRES what users have what access ("see-only" or "update") to your records in which subfiles. The only exception is METACCT itself, which is not affected by METACCT. A complete list of the affected subfiles appears in the next section.

To handle your records, users must first issue the SET METACCT command, naming the account whose records they want to see or use (yours). Their subsequent commands referring to your records for display and update or even compiling purposes will succeed or fail depending on the level of access you gave them in your METACCT record.

The next section of this chapter describes in detail the METACCT record that you the owner must create in order to give users access to your records. [See 3.1.] The section following that talks about this feature from your user's standpoint, describing the SET METACCT command in particular. [See 3.2.]

Having several people updating your system-subfile records can lead to confusion regarding the "current" version of a record. For example, suppose a user gets a copy of a file definition and makes changes to it; meanwhile, you do the same, updating the record before the other user does. When the other user's update occurs, your changes will be discarded.

To help avoid that type of problem, records in system subfiles may contain a special structure, called VERSION-STR. If used, this structure would in effect block the other user from updating the record until he or she takes your update into account. Details on VERSION-STR appear later in this chapter. [See 3.3.]

3.1  The METACCT Subfile

To give other users access to your records in one or more system subfiles, you must create a record stored under your account in the METACCT subfile. In addition, unless the access is limited to See-Only, the users can add new records to those subfiles for your account, make changes to records and update them, compile them, or even remove them. In a sense, by putting together a METACCT record, you are allowing some user or users to be you, at least in regard to some system subfiles (possibly limited to certain records; see KEY-PREFIX below). [See 3.2 for specific details on what users with METACCT access can do.]

The basic structure of the METACCT goal record-type is this:

ACCT = gg.uuu;             - your account number (record key)
COMMENTS = comments;       - optional comments
ACCESS = level;            - "See-only" or "Update"
  ACCOUNTS = gg.uuu, ...;  - accounts given that access
  [SUBFILE = subfile;]     - specific subfile it pertains to
  [KEY-PREFIX = gg.uuu...] - specific record key prefixes

Since you may have only a single record in the subfile, all special access to your records must be defined in this record. To that end, ACCOUNTS with SUBFILE may repeat under ACCESS, and ACCESS may repeat as well, forming multiple access-structures (see example below).

In case of contradictory statements in regard to a specific account or subfile, be aware that the least restrictive interpretation will always be chosen by SPIRES. For instance, if you give another account the ability to update records in one or all files, no other statements in the record can override that. Examples below will make this point clear.

Here are some other important details about ACCESS, ACCOUNTS and SUBFILE:

ACCESS -- This statement names the access level to be given to the accounts named in the list that follows. The allowed values are See-Only (or S) and Update (or U). A null value for ACCESS (i.e., "ACCESS;") is equivalent to Update. See-Only means that the named users are allowed only to examine the records; they may not add, update or remove your records. Update allows the named users to both examine and update (i.e., add, update and remove) your records in the system subfiles named below. It will also give the user the ability to compile or recompile those records, if the records are compilable. Update is less restrictive than See-Only. Multiple occurrences of ACCESS are allowed, each of which is usually followed by ACCOUNTS and SUBFILE. To summarize, the allowed values are:

 - See-Only -- users may only examine records;

 - Update -- users may add, update, and remove records;

ACCOUNTS -- This statement names the ACCOUNTS affected by the ACCESS statement above. If SUBFILE statements follow, the accounts listed here receive the named access only to those subfiles; hence, METACCT records often have several occurrences of ACCOUNTS, each followed by one or more occurrences of SUBFILE. You provide the same type of values here as you do for the ACCOUNTS statement in a file definition, as listed below. To specify multiple accounts (or groups) in an ACCOUNTS statement, separate the individual values with commas. Allowed values are:

 - PUBLIC -- to give access to all users;

 - gg.uuu -- to give access to the named account;

 - gg.... -- to give access to all users in the named group;

 - g..... -- to give access to all users in the named community.

SUBFILE -- This statement names the specific system subfile whose records you want the users to have access to. If you omit it, access is given to the entire list of subfiles shown below. Multiple ones may be listed for the list of accounts by specifying separate occurrences of the SUBFILE statement beneath the ACCOUNTS statement. Here is the list to choose from:

FILEDEF       RECDEF        BACKFILE       STATCHAR
FORMATS       EXTDEF        BACKRECS       FORCHAR INDEX
VGROUPS       RECHAR        BACKCHAR       SYS PROTO
FORSTAT       STATIC        BACKDEFS       COMP PROTO
FORLOAD       FORCHAR

The only significant omission from the list of system subfiles above is METACCT itself. You cannot give other users access to your own METACCT record; only you can add it or change it.

If you give Update access to the FORCHAR subfile, then you are permitting the other user to remove any compiled formats for any of your files, even if the format's id belongs to neither you nor the other user but to a third user.

Similarly, if you give a user Update access to the COMP PROTO subfile, that user can remove any compiled protocols in any of your protocol files, even if others defined the SYS PROTO records for those protocols.

KEY-PREFIX -- This statement, which may occur multiple times for each subfile (but can occur only if the SUBFILE statement occurs), limits the affected records in the subfile only to those with the specified prefix. The prefix should begin with your account number. For example, if your account is GQ.JNK and your METACCT record has "SUBFILE = FILEDEF; KEY-PREFIX = GQ.JNK.T;" then the access would apply only to file definitions that began with the prefix GQ.JNK.T.

The METACCT record takes effect as soon as you add it to the METACCT subfile.

Here are some sample METACCT records:

ACCT = AB.USE;
ACCESS = Update;
  ACCOUNTS = AB....;

That record gives all accounts in group AB access to user AB.USE's records in all the system subfiles listed above.

The next example is more complicated:

ACCT = AM.USE;
ACCESS = See-Only;
  ACCOUNTS = LO.USE, AM....;
    SUBFILE = FORMATS;
ACCESS = Update;
  ACCOUNTS = AM.UCK;

In this example, account AM.UCK gets Update access to all the system subfiles listed above, including FORMATS. The See-Only limitation for group AM users for the FORMATS subfile doesn't apply to AM.UCK, because the specific account reference to AM.UCK under "ACCESS = Update" is less restrictive than the specific subfile reference for the AM group.

Once the barn door is opened and an access-structure gives Update access to an account for some or all the subfiles, that access will not be revoked by any other access-structure. Below is a bad example, in that the record owner is trying to allow a particular user to update everything but the FORMATS subfile; it will not achieve the desired aim:

ACCT = GA.TES;
ACCESS = Update;
  ACCOUNTS = BY.TES;
ACCESS = See-Only;
  ACCOUNTS = BY.TES;
  SUBFILE = FORMATS;

Since the first access-structure has already given Update access to all subfiles for account BY.TES, the second structure does not override it; hence, BY.TES has Update access to the records in the FORMATS subfile belonging to GA.TES.

If it is really necessary to give an account Update access to all but one or two subfiles to which it should have See-Only access, you must type the name of each of the subfiles for Update access. In practical usage, however, this seems to be an infrequent need. In most cases, you should be able to state the requirements fairly simply.

You could easily concoct more complicated examples, but generally speaking, practical uses are relatively simple, more like the first two examples than the third.

If SPIRES's interpretation of that third example record still strikes you as odd, consider how you would interpret the same record if the See-Only account for the FORMATS subfile were listed as BY....(i.e., group BY) rather than the specific BY.TES. Is the record owner trying to allow group BY to have See access to FORMATS and in addition allow account BY.TES to have Update access to all the subfiles? Or does the record owner really mean to prevent BY.TES from updating those FORMATS records? The former interpretation seems more plausible, which is the way SPIRES would interpret that METACCT record.

3.2  Using METACCT Privileges: The SET METACCT Command

This section describes the privileges that METACCT access provides to you as a user of someone else's records, as well as the procedure you must follow in order to use them.

Generally speaking, you find out that you have been given METACCT access to someone's system-subfile records because he or she tells you about it. Aside from trial and error, you have no way of finding out whether you have access to other people's system-subfile records.

If you have been granted access to someone else's records, you can begin using them by issuing the SET METACCT command in SPIRES:

SET METACCT gg.uuu[, gg.uuu...]

where "gg.uuu" is the other person's account number, i.e., the account of the user whose records are to be retrieved or updated or compiled. By including multiple accounts in the command, you can request METACCT access to the records of several accounts at one time.

SET METACCT is a session command, remaining in effect for the duration of the SPIRES session, or until it is cancelled.

CLEAR METACCT cancels the current METACCT access. Issuing another SET METACCT command also cancels the current METACCT access, and then establishes access for the new list of accounts. SHOW METACCT shows you a list of the accounts currently set, or displays the message: "No accounts defined." (Syntax note: You can also spell out METACCOUNT in these commands if you desire.)

What Access Does METACCT Provide?

Generally, Update access to a system subfile through METACCT means that SPIRES treats you as if you were using both your account and the other user's when you have the subfile selected. Thus, you can see their records with the DISPLAY command, make changes to their records using the MERGE command or the TRANSFER/UPDATE sequence, or discard their records using the REMOVE command. You can even add new records for their account.

If you have been given access to compilable records, such as file, record, vgroup or format definitions, you can compile or recompile those records as well. Note, however, that for file definitions, you must have WRITE access (via ORVYL permits) to the owner's account. Additionally, if the compilation creates new ORVYL data sets, they will not have the normal PUBLIC access permits that are set when files are compiled or recompiled on the owning account. These will need to be set appropriately on the owning account. See section B.12.1 of the manual "SPIRES File Definition"; online, [EXPLAIN ORVYL FILES, PERMITS FOR IMMEDIATE INDEXING.]

See-Only access is a subset of the Update privileges. As its name implies, See-Only access allows you only to see the other person's records (for example, with the DISPLAY or TYPE commands), but not to add new ones, change existing ones or remove old ones.

As described in the previous section, [See 3.1.] the owner may grant this access to his/her records in one, some or all of the following system subfiles:

FILEDEF       RECDEF        BACKFILE       STATCHAR
FORMATS       EXTDEF        BACKRECS       FORCHAR INDEX
VGROUPS       RECHAR        BACKCHAR       SYS PROTO
FORSTAT       STATIC        BACKDEFS       COMP PROTO
FORLOAD       FORCHAR

Other commands that work with source records will probably work through METACCT access. A good example is PERFORM FILEDEF SUMMARY, which summarizes the file definition for the selected subfile. If you have at least See-Only access to FILEDEF for someone else's file definitions, you can issue this command when you have one of their subfiles selected. [See 28.4.] PERFORM PRINT is also allowed when appropriate METACCT access is in effect. [See 28.1.]

3.3  Using VERSION-STR to Track Copies of System File Records

Several system subfiles in SPIRES (e.g., FILEDEF, FORMATS) contain a structure called VERSION-STR. This structure can help you keep track of what "version" (i.e., which copy) of a system-subfile record you are working with. Because VERSION-STR is designed to solve a particular problem, it is necessary to understand the problem in order to understand how VERSION-STR works.

Suppose you and a co-worker are both working on an application. Via SET METACCT [See 3.2.] you transfer the application's file definition from FILEDEF and begin to make some changes to it. Your pal needs to make some changes to it too, so independently of you, she transfers the file definition, makes her changes, and updates it. Then you finish your work and update the record, losing her changes.

The data maintained by the VERSION-STR structure can prevent this problem. When you transfer your copy of the record to work with, the VERSION-STR structure contains a version number, such as "12". Similarly, your co-worker, getting a copy shortly thereafter, would also get version 12 of the record. You both make your changes, leaving the VERSION-STR untouched. When she issues the UPDATE command, SPIRES compares the version number of the stored record with the version number in the input; since they are the same, the record goes back in without an error. At that time, SPIRES assigns the record the next version number, which here is 13.

When you issue the UPDATE command, the command fails, because the version number of the stored record, 13, doesn't match the version number in the input data, 12. Hence you are warned that the record has been updated since you retrieved your copy of it. Presumably, you'll get a copy of the record in its latest form, and merge your changes in with that one. [You could also simply choose to change the version number in your copy of the record to 13 and issue the UPDATE command again. The VERSION-STR feature serves only as a warning; it cannot absolutely prevent records from being updated inappropriately.]

To request that version information be maintained for a system-subfile record, you simply add the VERSION-STR structure to the record:

[VERSION-STR;]
  VERSION-NUMBER = n;
  [VERSION-ACCT = gg.uuu;]

Because VERSION-NUMBER is the key of the VERSION-STR structure, you do not need to include the VERSION-STR statement.

For VERSION-NUMBER, "n" must be a positive integer no greater than 999999; it's most often set to "1". (After 999999, VERSION-NUMBER wraps around to 1 again.)

If you omit VERSION-ACCT, SPIRES will provide it with your account number, i.e., the account of the logged-on user. SPIRES does not verify the account value, so it will accept any value you type, even if it is an invalid account number or form; similarly, if you use an account abbreviation, such as ".", which is recognized as meaning "your account" in most other SPIRES contexts, it will be treated as the typed character and not translated into the account number here. SPIRES will, however, display the account number to you in lowercase, for a reason explained below.

The VERSION-ACCT is significant because SPIRES will ignore the VERSION-STR if the value for VERSION-ACCT doesn't match the account in the key of the record. The example described below will explain why that is useful.

The VERSION-STR structure is available in these system subfiles:

FILEDEF   FORMATS   VGROUPS   RECDEF   BACKFILE   BACKRECS

Practical Uses of VERSION-STR

If you are the only one who updates your system-subfile records, VERSION-STR will be of limited use. You could use it to keep track of copies of a record by their version numbers, just as you might use the MODDATE and MODTIME elements, e.g., to verify that the paper copy you have is the latest version.

When several users may be updating your system-subfile records, VERSION-STR is more useful, as described in the earlier example. Here, in more detail, is how you might use it.

Suppose that one of your applications keeps its production code under account GQ.PRD, and its test code under GQ.TES. Within each GQ.PRD record, you could add the VERSION-STR structure, like this:

FILE = GQ.PRD.ALMANAC;
 ...
VERSION-NUMBER = 1; VERSION-ACCT = gq.prd;

On account GQ.PRD, put the record back into the system subfile (FILEDEF, for this example). SPIRES will immediately change the record's version number to "2" internally.

Sometime later, you get a copy of file definition GQ.PRD.ALMANAC, moving it under your test account, GQ.TES. With the WYLBUR command CHANGE, you change "GQ.PRD" to "GQ.TES". (Since the VERSION-ACCT value is in lowercase, it remains "gq.prd".) You then make other changes you want to make, adding the record to FILEDEF as GQ.TES.ALMANAC. Because the VERSION-ACCT doesn't match the account in the key of the record, the VERSION-STR data is ignored, and isn't updated.

Eventually you are ready to replace the GQ.PRD.ALMANAC file definition with the new version under GQ.TES. Again you issue the CHANGE command to change occurrences of "GQ.TES" to "GQ.PRD" (again leaving the VERSION-ACCT value untouched), and then update the file definition with the new copy.

Because the VERSION-ACCT value now matches the account in the record key, SPIRES will pay attention to the structure. If the input version number ("2") matches the version number in the stored record, then the update can continue. If it doesn't match, then the record has been updated since the time you made the copy of it; SPIRES will reject the new copy, meaning you need to resolve the discrepancies between them.

Technical Notes on VERSION-STR

VERSION-STR and the MERGE command

WARNING: If you use the MERGE command to update a system-subfile record (rarely done under any circumstances, we hope) and the stored record contains the VERSION-STR structure, be sure to include the VERSION-STR in your input as well. If you don't, the VERSION-NUMBER will not get checked/updated appropriately.

The warning also applies to updates done through partial processing; be sure to open and close the VERSION-STR structure during your processing.

Getting rid of the VERSION-STR

Generally speaking, the only way to eliminate the VERSION-STR structure once you have begun using it is to get a copy of the record, remove the record from the subfile, delete the VERSION-STR structure from the copy, and then add the record back into the subfile again.

VERSION-STR and Secure-Switch 4

Users of a system subfile with secure-switch 4 set (only people involved with SPIRES system administration) may also eliminate the VERSION-STR structure of records directly, simply by transferring them, deleting the VERSION-STR lines, and updating them.

4  Object Code Management: The ZAP Subfile-name Commands

In SPIRES, the following kinds of data are or can be compiled:

 - file definitions, stored in the FILEDEF subfile

 - format definitions, stored in the FORMATS subfile

 - file record-type definitions, stored in the RECDEF subfile

 - protocols, whose control information is stored in the SYS PROTO subfile

 - variable group definitions, stored in the VGROUPS subfile

Whenever one of these kinds of data is compiled, SPIRES stores the compiled object code. For file definitions, the compiled characteristics are stored in an ORVYL file, on the file-owner's account, called "filename.MSTR". For all other object code, SPIRES creates a record in another SPIRES system file. In addition, when a user issues a STORE STATIC command, SPIRES creates a record in another SPIRES system file.

To allow you to manage (i.e., remove) records created in these other SPIRES system files by the COMPILE and STORE STATIC commands, a set of ZAP commands is available in SPIRES to delete these records, and optionally to delete the source records from which they were derived.

The general form of these commands is:

ZAP   source-subfile   source-record-key   [SOURCE]

where "source-subfile" is: FORMATS, RECDEF, SYS PROTO or VGROUPS. (Note: the source-subfile names can be abbreviated to three or more characters.) The "source-record-key" is the key of the record given in the COMPILE command; as in these commands, the key need not be fully qualified by the user's account number, since the user can only ZAP object-code records defined by the logged-on account. A ZAP STATIC command is available to remove data records for stored variables.

To use any of these ZAP commands, the source-record must be stored in the appropriate SPIRES system file. That is, to ZAP the compiled code for a format, the definition for that format must be in the public FORMATS subfile. If the source record is not available, then more manual methods of object-code management must be used. Contact the SPIRES consultant for more information.

The following sections describe most of the various ZAP commands available in more detail.

4.1  The ZAP FORMAT Command

To remove object code generated when a format is compiled, the following ZAP command is available in SPIRES:

ZAP FORMAT[S] source-record-key [SOURCE]

The source-record-key is the value of the ID statement in the format, which is the key of the FORMATS subfile record. The source-record-key need not be prefixed by the logged-on user's account number. If the SOURCE option is used, then the source record is removed from the FORMATS subfile.

For example:

ZAP FORMAT XA.G01.DISPLAY
ZAP FORMAT DISPLAY SOURCE
ZAP FORMATS DISPLAY

You can only zap the object-code of a format when you own the format. Whether or not you own the file to which the format applies is immaterial. The file owner can therefore only zap the formats he created for the file.

4.2  The ZAP RECDEF Command

To remove object code generated when a record definition is compiled from the RECDEF subfile, the following ZAP command is available in SPIRES:

ZAP RECDEF source-record-key [SOURCE]

The source-record-key is the value of the ID statement in the record definition, which is the key of the RECDEF subfile record. The source-record-key need not be prefixed by the logged-on user's account number. If the SOURCE option is used, then the source record is removed from the RECDEF subfile.

For example:

ZAP RECDEF GC.JCB.SIMPLE.INDEX
ZAP RECDEF SIMPLE.INDEX SOURCE
ZAP REC SIMPLE.INDEX

4.3  The ZAP SYS PROTO Command

To remove object code generated when a protocol is compiled using the older SYS PROTO method of compilation, the following ZAP command is available in SPIRES:

ZAP {SYS PROTO|SYSPROTO} control-record-key [SOURCE]

The control-record-key is the value of the ID statement in the protocol's compiler-control definition, which is the key of the SYS PROTO subfile record. The control-record-key need not be prefixed by the logged-on user's account number. If the SOURCE option is used, then the control record is removed from the SYS PROTO subfile.

For example:

ZAP SYS PROTO AC.DBA.FULLFACE.PROTOCOLS
ZAP SYSPROTO FULLFACE.PROTOCOLS SOURCE
ZAP PROTOCOL FULLFACE.PROTOCOLS

If the protocol is compiled directly from the source subfile (the newer method of compiling protocols), you should use the ZAP PROTOCOL command:

ZAP PROTOCOL protocol-name [SOURCE] OF protocol-subfile

4.4  The ZAP VGROUP Command

To remove object code generated when a variable group definition is compiled from the VGROUP subfile, the following ZAP command is available in SPIRES:

ZAP VGROUP[S] source-record-key [SOURCE]

The source-record-key is the value of the VGROUP statement in the variable-group definition, which is the key of the VGROUPS subfile record. The source-record-key need not be prefixed by the logged-on user's account number. If the SOURCE option is used, then the source record is removed from the VGROUPS subfile.

For example:

ZAP VGROUP BG.VLJ.GLOBAL.VARIABLES
ZAP VGROUP GLOBAL.VARIABLES SOURCE
ZAP VGROUPS GLOBAL.VARIABLES

4.5  The ZAP STATIC Command

This command removes a particular set of stored static variables created by the STORE STATIC command, or it can remove all sets of stored static variables for a particular variable group.

Three separate command forms are allowed:

ZAP STATIC storage-record-name OF vgroup-name
ZAP STATIC vgroup-name ALL
ZAP STATIC * ALL

The first form removes a particular stored set of static variables; "storage-record-name" is the name given to the stored vgroup when the "STORE STATIC vgroup-name TO storage-record-name" command is issued. The second removes all stored static groups belonging to you for a particular vgroup. The third form eliminates all stored static groups belonging to you for any vgroups.

The ZAP STATIC commands will remove stored sets of static variables only if they belong to you, regardless of whose vgroup they apply to.

5  Multiply Defined Elements

You can have great flexibility with files containing multiply defined element mnemonics -- that is, two or more elements with the same name occurring in a file definition. For example, the element COMMENTS might occur in several different structures in a file definition. Also, "floating structures" [See "SPIRES File Definition", section B.3.6.] cause multiply defined element mnemonics.

The SET ELEMENTS, TYPE, ALSO and SEQUENCE commands allow specification of elements either by a simple element mnemonic or by a "structure@...@element" form (see examples below) which specifies a structural path to the desired element. Note that this second form is needed only if the element mnemonic is not unique in the file definition, i.e., the same name is used for several different elements.

If you use any of these commands with a simple element mnemonic when that element is multiply defined, not all of the occurrences of that element throughout a record are examined or displayed. If we say that record-level elements are at the highest level, and that elements in a record-level structure are at the next lower level, and that the innermost-nested structure elements are at the lowest level, we can say that the simple mnemonic option will cause SPIRES to locate the first appearance of the mnemonic at the highest level it is found in the file. However, using the "structural path" option will locate any specific appearance of the mnemonic.

The SET ELEMENTS and TYPE commands also allow the "@element" form, which means "all elements having the specified mnemonic". Again, this form is only needed if the same element mnemonic occurs more than once in the file definition.

The effects of these features are best explained with examples. You will note that different aliases for the same mnemonic can be used for clarity. The following is a skeletal part of a file definition, followed by three records from the file:

     1.  RECORD-NAME = ID;
     2.  SLOT;
     3.  OPTIONAL;
     4.     ELEM = X;
     5.        ALIAS = XR;
     6.     ELEM = S;
     7.        TYPE = STR;
     8.  STRUCTURE = S;
     9.     REQUIRED;
    10.        ELEM = X;
    11.           ALIAS = XS;

Note that element X appears twice in the definition -- once at the record level (alias "XR") and once within a structure (alias "XS").

ID = 1;          ID = 2;          ID = 3;
X = Afford;      S;               X = Change;
S;                X = Before;
 X = Delete;

The SEQUENCE command

The command "SEQUENCE X" would arrange these records in the order 2,1,3. SPIRES examines values only at the highest level in all records where it occurs in any record -- in this case, the record level. Since null values are listed first, record 2, with no value for X at the record level, comes first, followed by records 1 (X = Afford;) and 3 (X = Change;). The command "SEQUENCE XR" would give the same sequence.

However, you can type "SEQUENCE S@X" and the arrangement becomes 3,2,1. SPIRES looks for occurrences of X specifically in the structure S. Thus, record 3 with no occurrences of the element at that level comes first, then records 2 (X = Before;) and 1 (X = Delete;). "SEQUENCE XS" would also have the same result.

The ALSO Command

If you have a search result containing these three records, and you issue the command "ALSO X STRING FOR", only record 1 will be retained. SPIRES examines only the occurrences of the X element at the highest level the element can appear -- the record level, in this case. Thus, only the "XR" occurrences are examined. ("ALSO XS STRING FOR" would give the same result.)

"ALSO S@X STRING FOR" retains only record 2. ("ALSO XS STRING FOR" would also give the same result.)

The TYPE and SET ELEMENTS Commands

Both of these commands are affected similarly to the ALSO command, though they both have an additional feature, discussed below. For example, "SET ELEMENT X" will cause commands that display the records to display only the "XR" values -- that is, occurrences of the first element X found at the highest possible level, which in this case is the record level.

"SET ELEMENT S@X" will set the element with the "XS" alias -- that is, all occurrences of the element X within the "S" structure.

"SET ELEMENTS @X" is the additional feature: all occurrences of the element X at any level in a record will be "set".

The values for the element list on a TYPE command can also use the "structure@...@element" or "@element" options described here, or they can use neither option, with the same results as described above for the SET ELEMENTS command.

Remember that this information is only relevant to files with multiply defined element mnemonics, which are relatively rare.

5.1  The Throw-away Element: The "-" Element in SPIRES Subfiles

Each SPIRES subfile has a special element, the "-" (hyphen) element, that can be used during data entry for comments. When a record containing occurrences of this element is added to the subfile, the values for the "-" element are thrown away; they are not stored within the record. But why would anyone want to type comments that are thrown away on input?

Some SPIRES users find that they update the same large record or records over and over again. Each time they update the record, however, they may not remember why a particular element or occurrence of an element has a particular value. What these users do is to insert these "throw-away" comments throughout their input data and then save the input data separately from the subfile before adding the record to the subfile. The extra saved copy becomes the master copy of the record; when the record needs to be updated, the master is changed, saved again, and then used to replace the record in the subfile. When the record is added or updated in the subfile, these extraneous comments that are saved in the master copy somewhere else are not stored in the subfile copy.

Here is how some input using this element might look:

NAME = John Klemm;
ADDRESS = 188H Pine Hall;
- This address may change at the end of the year.;
CITY = Stanford;
STATE = CA;
PHONE = 497-4420;
  AREA-CODE = 415;
  - will probably keep this phone number after move.;

As you can see, the "-" element can appear anywhere and any number of times in the input data, when you are using the standard SPIRES format. Of course, the value must end with a semicolon.

6  Storing DECLARE Data for Multipurpose Use

Declare data, the commands and metadata that define declared elements and output control packets, is normally found in protocols. However, in situations where you'd like to use the same declare data either in multiple protocols or repeatedly in command mode, you can store the declare data in a separate subfile and execute the declaration from there. The subfile is possibly one of your own making, but system files are also available in which you may store your declare data.

The first section of this chapter describes how to set up a subfile for declare data. [See 6.1.] The following one describes how to use your stored declare data. [See 6.2.]

6.1  Declare Data Subfiles

You can either create your own subfiles where you can store declare data or else, if your declare data is for declared elements or for output control packets, use two system subfiles.

The DATA MOVE DECLARES and DATA OUTPUT CONTROL Subfiles

Two system subfiles already exist for declare data; anyone can put declare data into them:

DATA MOVE DECLARES  - for declared element data
DATA OUTPUT CONTROL - for output control packets

Additionally, anyone can refer to those records, although only the record owner, plus anyone to whom the record owner gives METACCT access, can update or remove them. If privacy of the declare data itself is an issue for you, you should create your own subfile(s) to hold it, as described below.

For each declared element or each output control package you want to store in one of the above subfiles, you just add an ID statement at the top and add it into the appropriate subfile:

DATA MOVE DECLARES subfile     DATA OUTPUT CONTROL subfile
  ID = gg.uuu.name;              ID = gg.uuu.name;
    [declare element             COMMENTS = optional comments;
     statements]                  PACKET = name;
    ...                            [output control statements]
                                   ...
                                  PACKET = name;
                                   [output control statements]
                                   ...
</