MOOT Intent: Db and Use Cases

Introduction and Terminology

Inspired by Eric Charles' analysis for ACD LATC configuration, I will make a(nother) stab at getting the concept of intent into MOOT. Eric has broken up the specification of ACD intent into 6 pieces: Mode, Bias, Timing, Hld, PHA and Veto. These pieces are known as precincts. Ultimately, intent boils down to LATC register specification (and other things; e.g., what belongs in an lci script, but for now we consider only LATC input, the most complicated part of the instrument configuration). The registers managed by a single precinct need not end up in the same LATC component. For example, Mode manages registers from AEM, ARC and AFE.

Eric had the idea of using xml files to describe intent and wrote a dtd (Document Type Definition) description of an XML format) for this purpose. The format has evolved somewhat to meet the needs of both the operator interface (reads and writes the files) and MOOT (reads only). For each precinct the registers and register fields it controls are listed and other inputs, if any, needed to generate settings are described. An XML document satisfying the dtd provides a full description of ACD intent by, for each precinct

• identifying the method to be used to generate outputs
• including register settings inline if appropriate
• referring to ancillary files, if any, needed to generate the settings
• setting values for relevant constants (e.g., "counts above threshold"), if any

It also specifies outputs: parameter files (in the MOOT sense; that is, sources used to create configuration binaries) and offline calibrations. See this old example which describes intent for the six ACD precincts (new design only allows an inline intent description for one precinct at a time). It is valid according to this XML schema. The most recent version of the schema can be found in the Offline CVS repository, package mootCore. A document describing intent for one of the precincts is called a vote; that is, a vote is a sort of instance of a precinct.

Aliases

DB Structure

Relationships

Recall that the source files used to generate a configuration (that is, the files which are input to FSW-supplied utilities like LATC) are known as Parameters. The goal is to generate the parameter files associated with a particular intent and also, given a configuration, to recover what the intent was when the configuration was generated. To do this, MOOT has to keep track of some additional (relatively) static relationships, such as the parameter classes associated with a particular precinct. It also has to keep track of relationships among instances of objects, such as the ancillary file(s) used in the generation of a particular parameter file.

Primary Table definitions

The following can be thought of as primarily describing "things" or "classes of things" rather than relationships between things.

Ancillary, Ancillary_class

Precincts

Ideally this is a relatively static table. For each delegate, experts should come up with a suitable description of its precincts.

Votes

It might be helpful to extract important quantities from a vote file and store them in additional fields (e.g. method) in the row to make it easier for an operator to decide which votes to use by browsing the table.

Ancillary file aliases

The pair (name, aclass_fk) is constrained to be unique. Vote files may (in fact, in current implementation, must) specify ancillary files by an alias.

Vote file aliases

The pair (name, precinct_fk) is constrained to be unique.

Parameters

Configs

Relation tables

Parameters_to_Ancillary

Keeps track of ancillary files used in generation of a particular parameter file. This table already exists in MOOD but is unused.

Precincts_to_AClass

One or more kinds of ancillary file may be associated with a precinct [probably MOOT doesn't need to keep track of this]

Several other relation tables will be added for various purposes: Vote_PClass_AClass, Container_Precinct, LATC_Backup.

Other relations

Some relationships are more conveniently expressed with additional foreign keys.

• Add vote_fk, a foreign key referencing Votes, to Parameters
• Add precinct_fk, a foreign key referencing Precincts, to Parameter_class

Use cases

Registering ancillary files

Managing precincts and votes

The set of precincts is expected to be relatively static. They will be defined by subsystem or other experts once and for all (or nearly) and corresponding rows will be added to Precincts by the MOOT maintainer.

The collection of votes is likely to continue to increase, albeit slowly. A MOOT function will be provided so that experts can register them on their own. If all subsystems use Eric's scheme (and they will since Eric has done the work of defining precincts for them), a vote corresponds to an XML element, including its attributes and child elements, as serialized in a file. This file will be archived (that is, copied to MOOT's archive in SLAC afs space) upon registration just as parameter and ancillary files are archived.

Creating and registering parameter files

Parameter files may be derived from any number (including zero) of ancillary files. In the new scheme of things, they will always be related to a vote. Someone (subsystem expert to begin with; ultimately perhaps an operator) will determine which vote should be used to generate a parameter file or files. The vote should, directly or indirectly, specify the procedure to be run and the inputs: ancillary files and/or constants. At the close of the procedure whoever (or whatever) invoked it should have all information necessary to register the new parameter file(s) and indicate which ancillary files they depend upon. MOOT functions will be available to

• Register a parameter file
• Make new entries in Parameters_to_Ancillary. (Will provide a single function to do both if the call interface is not too horrendous.)
• Determine whether it is necessary to rebuild using the specified vote file. In particular, have any of the referents of aliases it uses changed since the last rebuild (if any) using it? In order to determine this, MOOT will have to be able to parse the vote file at least to the extent of finding ancillary file aliases and determining which ancillary class they are referring to.

Building a config

Note that the complete process, all the way from generating ancillary files to coming out with a new config, takes place in a sequence of steps which can be spaced out in time. Steps or parts of steps which are done via MOOT services are indicated like this.

1. (Optional) Create entry or entries in Votes defining intent. This step is optional because most of the time the operator will want to apply an old definition of intent to new data.
2. (Optional) Create new ancillary files from charge injection or other data. Register the ancillary files. This step normally will occur, but it could conceivably be omitted if the same data are to be used with a new vote (intent). Even when this step does occur, it need not be executed for all delegates or all precincts controlled by a delegate.
3. Run (delegate-owned) procedures to create new parameter files based on intent. This usually involves invoking a MOOT function to resolve ancillary file aliases. Register the new parameter files and relations between them and ancillary files. This step need not occur for all delegates, nor for all precincts belonging to a delegate.
4. By specifying a collection of votes and perhaps other criteria, select parameters for a new config and build it. This may involve creating new FSW binaries.

Recovering intent

• get a list of parameter files supplied as input when the config was built
• get a list of the parameter files actually used to generate the binaries constituting the config

The essential content (e.g., register settings) of the two sets of files will be identical, but other features, such as the votes associated with the parameter files when they were registered, may differ.

A new MOOT query will be added to return the key of the vote associated with a parameter file, making it possible to recover the complete set of votes (= intent) associated with a config.

New or enhanced MOOT user functions

Create/insert/build/update

Register a vote file written, debugged

MootBuild::registerVote(const std::string& precinct, 
                        const std::string& source, 
                        const std::string& descrip, 
                        const std::vector<std::pair<std::string, std::string>>& parmAnc)
                        const std::string& schemaVersion="*",
                        const std::string& schema="intent") 

The third-to-last argument is used to describe which ancillary file classes a parameter file produced by the Vote depends on. The service will add this information to the vote file before copying it to the archive. The last two arguments specify the schema file to be used when validating the vote file. By default the latest version of the file with core name "intent" will be used, e.g. something like intent_v2r0p0.xsd.

 

Register an ancillary file written, debugged

unsigned registerAncillary(const std::string& class, 
                           const std::string& source, 
                           const std::string& descrip, 
                           const std::string& instrument,
                           const std::string& GMT_creationTime, 
                           PARMQUAL quality=PARMQUAL_test,
                           const std::string& source_fmt="xml")

Add or update an (ancillary file) alias written, debugged

MootBuild::setAncAlias(const std::string& aliasName, 
                       unsigned ancKey, 
                       const std::string& ancClass)

Add or update a vote alias written

MootBuild::setVoteAlias(const std::string& aliasName, 
                        unsigned voteKey, 
                        const std::string& precinct)

Register a parameter file written, debugged

unsigned registerParameter(const std::string& class,
                           const std::string& source, 
                           const std::string& descrip,
                           const std::vector<unsigned>& ancKeys,
                           unsigned voteKey=1, 
                           PARMQUAL quality=PARMQUAL_test,
                           const std::string& sourceFmt="xml")

[Do we want to return oldKey as older register-parameter routines did? Don't think so.]

The default value of 1 for voteKey refers to a dummy vote file; its use here implies the parameter file was generated in some non-standard manner.

If ancKeys is non-empty, MOOT will make entries in Ancillary_to_parameters relating the new parameter file to the ancillaries. Another possibility would be to use the vote key to determine what the ancillary files were, however we might get the wrong answer if the "new" parameter file was in fact generated a while ago, and since then the resolution of an alias name for some ancillary file had changed.

Still to do: check whether ancillary files are of proper classes as specified by vote.

Modify buildConfigFromParams to accept optional vote key argument

The vote key should resolve to the global intent file used to generate parameter files making up the config.

Query

Resolve an ancillary file alias written, debugged

unsigned resolveAncAlias(const std::string& alias, const std::string& ancClass)

Returns key of file registered in Ancillary if there is one; else 0.

 

Resolve all aliases associated with a particular vote file written, debugged

int resolveAncAliases(std::vector<unsigned>& ancKeys, unsigned voteKey)

Return number of ancillary file keys found or negative value for error (e.g., no such voteKey) — or should it throw an exception in this case?

 

Resolve a vote file alias written, debugged

unsigned resolveVoteAlias(const std::string& alias, const std::string& precint)

Returns key of file registered in Votes if there is one; else 0.

 

Check if vote is "up to date" written, debugged

bool voteIsUpToDate(unsigned voteKey)

Returns true if associated parameter files have been created for the ancillary files which are the resolutions of aliases mentioned in the file. The same function will also work for a global intent file whose contents are a list of references to aliases for "little" vote files. In this case the function will resolve the references, then check if the individual vote files are up to date.

 

Check if config is "up to date"

bool configIsUpToDate(unsigned configKey)

Only applies to configs for which optional global-intent file key has been supplied. This function checks that the global-intent file is up to date, then compares set of parameter files generated by use of global-intent file with current value of all relevant aliases with the set of parameter files specified when the config was built.

 

Return key of specified parameter file class which is most recent with respect to a particular vote file

Decided not to implement. Status of individual parameter files is not so interesting as the whole collection. See next function.

 

Return keys of all parameter files associated with vote file currently written, debugged

unsigned getVoteParameters(unsigned voteKey, std::vector& parmKeys,
                           bool& isUpToDate);

..if vote is up to date; otherwise just set isUpToDate to false. May also be used with container vote file.

 

Get parameter file information by key

Exactly what that information should be is still to be determined

 

Get ancillary file information by key

Exactly what that information should be is still to be determined

Introduction

DB structure

Use cases

MOOT functions

Glossary

Glossary and examples

Delegates, precincts and votes

A Delegate is a sphere of influence or control within the universe of configurable quantities. The most typical example is a subsystem, such as the ACD. Delegates don't show up much in the implementation of MOOT because the concept is too coarse.

The quantities controlled by a particular delegate may be further partitioned into precincts. A precinct is defined by the quantities it determines and by the kinds of inputs (e.g., data like ACD pedestal calibrations or settings like "number of counts above pedestal") used to determine them. Register values or other quantities controlled by a precinct would normally be established all at the same time by means of a single procedure.

A vote describes the particular inputs and (perhaps) procedure to be used (e.g., "run program X, using the most recent good pedestal calibration, establishing thresholds 17 counts above pedestal"). A vote file corresponds to the intent for the items (e.g. registers; flight software parameters) it controls. Roughly speaking, precinct/vote have a class/instance relationship. A global intent file is another xml file whose only content is a list of references to vote aliases. At any given time these may be resolved using MOOT's vote_alias table.

Ancillary file, ancillary class

Ancillary files are typically generated as a result of analysis of one or more data sets from physics running or special charge-injection runs. They are in turn input to procedures producing configuration source files (aka parameter files, q.v.), such as those for LATC, and offline calibrations (e.g., ACD pedestals). Each ancillary file belongs to a particular ancillary class. Examples of ancillary classes are ACD_BiasDacCalibration, CAL_trg_alignment, TKR_Mask.

FSW input file, class

An FSW input file is a binary which, if all goes well, will be registered with fmx and ultimately uploaded to the instrument. Each such file belongs to one of the known FSW input classes. Examples of such classes are latc_SPT, lci_ACD.

Parameter file, class

A parameter file is source used to produce an FSW (binary) input, typically by running a flight software program like LATC_parser. Several parameter files may be used to produce a single FSW input (but not the other way around: that is, the association of {parameter files} to {FSW inputs} may be many-to-one, but never one-to-many). Like ancillary files or FSW inputs, parameter files always belong to one of the known set of types, called parameter classes. Examples are latc_CFE_CAL_FLE, latc_DFT_GNL_Timing, lci_TKR.

---

Created: 19 Jan 2007 Last revised: Wednesday, 07-Nov-2007 17:21:34 PST

J. Bogart