Channel Archiver Changes

New "versions" are marked by some additional functionality. The "patches" are then used to debug those new things.

• 10/16/2001
  Patch #1:
  Replaced the copy option with rename in the ArchiveManager because copy had a fault and won't work anyway if the archive is added to.
• 10/15/2001
  Version 1.9:
  Basically includes all the 1.8.5 patches which weren't available to non-SNS colleagues because I couldn't access our web server. Introduces the CAManager tool.
• 10/02/2001
  Patch #5:
  
  • Thomas Birke improved the CGI interface: List of channels is now shown in the "Names" box, the online plot will by default show reduced data which looks identical to the full data but runs quicker (see below).
  • Introducing the BucketingValueIteratorI. It performs data-reduction by returning at most 4 (first, min, max and last) values per bucket (a timeslice).
    This is especially useful for producing plots of the retrieved data. If you specify the bucket-width as the (time-)width of one pixel of the output image, the amount of data to plot can be dramatically reduced.
  • ArchiveEngine should now store one sample after a re-connect. That sample might have an adjusted time stamp, not the original IOC time stamp, because the preceding "disconnect" status is written with a time stamp from the host that runs the ArchiveEngine.
  • The ArchiveEngine has now a command line option to switch off the online-configuration web-interface.
  • GNUPlotExporter had problems with non-numeric data. While it'll never be able to plot strings, it should now handle enumerated values by plotting the numbers.
  • ArchiveManager can copy a channel inside the directory file, creating a new name for the same data. It can also delete a channel. Note: not the data but the channel. The data is still in the data file but you can no longer get to it. To remove the data, xport the archive into a new one, then the deleted channel's data will also vanish. No, there is no undo for the delete.
    Useful: copy + remove = rename.
  • Added a first version of CAManager - a Tcl/Tk based frontend to ArchiveEngine configuration (see doc/camanager/camanager.html for details).
• 07/25/2001
  Patch #4:
  Working on scripting support for Matlab, see casi/matlab/Readme.txt. You can already create Matlab command files with most of the retrieval tools. But those command/data files are quite big if you look at a lot of data. Scripting support provides direct access to the data from within Matlab. You can e.g. loop over data for a whole year and assemble rolling statistics without having the whole year worth of data in memory - which would be too much. Instead, you look at one value at a time and then go on to the next value. casi/matlab starts to work: You can list channel names, find a channel and value-after-some-time and loop over value. But there are no additional examples, the error handling might be pretty awful, and the API can still change a lot.
• 07/24/2001
  Patch #3:
  
  • Redid the documentation: The original docs concentrated on compilation and installation. Now that has been moved down and more emphasis it put on usage, closer to a "user manual". Ben Kruger, a LANL summer student, provided some of the new pages and paragraphs.
  • GNUPlotExporter fixes, second round: Given a start and end time stamp, the GNUPlot exporter will try to come up with values at those exact dates so that online plots look nice. The value preceding 'start' is copied onto the 'start': OK since we archive changes. The previous value is still valid. Similarly, the last value before 'end' is duplicated with 'end' as a time stamp. Or 'now', if that's earlier.
  • CGIExport: will swap start/end time if given in wrong order.
  • Use of "precision": The ArchiveEngine archives the precision. The original retrieval tools used that precision when formatting values. As it turned out, few of the IOC databases are configured properly, so users complained about the truncated numbers (default is PREC=0, so you saw no fractional digits). The early fix was to ignore PREC. But then our BESSY colleagues actually depend on the PREC-driven formatting and had to put the PREC formatting back in.
    As a compromise, a PREC of zero is now ignored, PREC > 0 is acknowledged.
• 07/18/2001
  Patch #2:
  
  • GNUPlotExporter fixes, suggested by Thomas Birke. Correct problems with gaps caused by subsequent status values (Archiver off, disconnected, ...). Will now try to show one value at given start and end time stamp by mapping the last value before start onto start and copying the last value before end onto the end.
  • SpreadsheetExporter: Will now show one value before and after the given time range.
• 06/21/2001
  Patch #1:
  
  • ArchiveEngine: Exploring possible problems when writing to a disk that is NFS-mounted. Fixed EngineServer, the engine's HTTP server, to run on even if the write thread hangs because of a NFS disconnect. Top-level engine status page shows if the write thread is writing right now.
• 06/07/2001
  Version 1.8:
  
  • MultiArchive:
    
    • Didn't really handle multiple archives until now. Hopefully not too many bugs left.
    • No longer investigates all subarchives unless necessary.
      For find-channel and get-value-after-timestamp this can be skipped -> much faster!
      When a list of channels and/or times is requested, the sub archives are still queried.
  • GNUPlotExporter:
    
    • Use two Y-axes if exporting excactly two channels
    • Can call GNUPlot via pipe, skipping the command-file generation
    • Use 'steps' instead of 'linespoints'
    • Show units in legend
  • CGIExport:
    
    • Benefits from new GNUPlotExporter (cannot use pipe on Win32).
    • Image is now a HREF link to GNUPlot data. That's different from the spreadsheet format, but might still be useful.
  • Engine:
    don't write buffers to disk at shutdown because this can take too long. After receiving a signal (e.g. on computer shutdown), the engine will now stop ASAP. The archive contains data up to the previous write period.
    New log messages, Web interface shows total number of channels and connected channels (suggested by Bob Hall)
  • casi:
    Added Bob Hall's python-based archiveBrowser to the example scripts.
• 04/30/2001
  Patch #6:
  
  • MatlabExporter: channel names which are invalid as Matlab variable names (because they include ':') are patched with underscores.
• 04/17/2001
  Patch #5:
  
  • The "Exporter" had a bug which made the "fill" option fail with some small archives (didn't find proper first value, displayed nothing as a result).
  • Added getDoubleTime() as suggested by Bob Hall (SLAC) to casi.
• 04/16/2001
  Patch #4:
  The "ExpandingValueIterator" had a bug which made the result of ArchiveExport and CGIExport appear to go back-in-time, even though the actual archived data was fine.
• 04/11/2001
  Patch #3:
  These apply to the Exporter classes in LibIO as well as ArchiveExport, WinBrowser and CGIExport:
  • Removed the "rounding" option from the Exporter as well as CGIExport etc. because noone (including me) really understood this. For Matlab, raw data export is best because you can do interpolation in there, for the rest filling (staircase) and linear is what you get.
  • Fixed the exporter's 'fill' option to remove the N/A at the start and end: When beginning at 1pm and there is no value at 1pm but a valid one from 12:55, that value will be filled/repeated into the 1pm time slot.
    At the end, channels which are not disconnected but also have no new value are repeated to show that the last entry is still valid.
• 04/10/2001
  Patch #2:
  
  • Cleaned up the HTML TABLE for CGIExport
  • shell/file "globs" are now case-independent and the CGIExport tool supports them since most users don't understand regular expressions.
• 04/05/2001
  Patch #1:
  
  • Matlab support for CGIExport, redid the CGI interface to make it better looking (by definition) and allow space for the Matlab button. Also new: An Excel download was added, similar to the Spreadsheet download but launching Excel on Win32.
• 03/02/2001
  Version 1.7:
  
  • Moved into the SNS CVS repository:

               Example checkout:
               cvs -d :ext:kasemir@ics-srv01.sns.ornl.gov:/sns/ADE/cvsroot \
                  get -d ChannelArchiver epics/supTop/extensions/1.1/src/ChannelArchiver        
               

  • New MatlabExporter class in LibIO
  • ArchiveExport and Winbrowser have Matlab option
• 03/15/2001
  Patch #3:
  
  • Export class has "fill" option. CGIExport used this, now ArchiveExport does so, too.
• 02/21/2001
  Patch #2:
  
  • Changed Engine/ScanList to use a list of pointers instead of elements to help solve compile problems.
• 02/14/2001
  Patch #1:
  
  • Completely removed atac, the tcl-only scripting.
  • Engine adds "Archive_Off" on shutdown instead of trying to hack this in on restart. Though different from the original prototype, this makes more sense to me.
  • New config parameter !ignored_future <hours>
  • On restart, the current value is archived, if necessary the time stamp is adjusted. Previously, the "current" time stamp of operator input channels was likely to be older than the "Archive_Off" mark. As a consequence, it wasn't archived. This is correct in one way but also confusing for the retrieval because you see an old value, then archive off and then nothing. The new behaviour matches what's happening on disable/enable: After a re-enable, the current value is written with a time stamp that matches the "enabling" event.
  • Added MatLab examples to the casi/perl directory.
• 01/29/2001
  Version 1.6:
  
  • Removed atac, the old tcl-only scripting interface.
  • Removed namespaces. Though a good idea, Microsoft C++ is for now the only compiler that implements them and actually uses namespaces in the supplied library. I should burn the books that told me to use namespaces.
  • Engine now ignores stamps which are "in the future", compared to the clock of the machine where the engine is running. This avoids problems since once a channel has logged a (bogus) value from the "future", you cannot go back-in-time, so you cannot add correct values. For now the definition of "future" is hardcoded in getIgnoredFutureSecs(), Engine/Engine.h.
  • Removed ASSERTs in Engine/ChannelInfo, replaced by error messages.
  • Replaced some usage of std::strbuf with sprintf because strbuf's operators caused unexplainable lockup. Should remove all strbuf calls.
  • The scripting interface CASI now works with perl on Linux and Win32.
• 11/20/2000
  Patch #1:
  
  • Added "FileSize" switch to ArchiveManager
  • Bug (which needs more attention): The LOG_MSG utility uses a lock. It's possible to deadlock if a LOG_MSG call uses another call which in turn (because of an error) tries to call LOG_MSG. As a quick fix, the calls in CtrlInfoI.cpp are removed because they caused problems with channels that have invalid enum. values.
    If the engine hangs, try to find out where like this:
    Find the PID of the engine. On Linux, you'll see 3 PIDs, pick the lowest one. Launch "gdb ArchiveEngine" in Engine/O.Linux or wherever the executable is. Then type "attach (PID)". Now you can switch threads with "thread 1", "thread 2" etc. and use "bt" to see where the thread currently is.
• 07/28/2000
  Version 1.5:
  With the helpful work of Nick Pattengale, a summer student at LANSCE-8, LANL,
  • the SWIG-based scripting interface has undergone more testing
  • simple write support (copy really) has been added
  • CArDMiner, a python GUI tool for archive management has been added (Nick Pattengale).
    Later additions might make CArDMiner live up to its name and turn it into a real "data mining" tool. For now it can be used as a replacement for the command-line ArchiveManager tool:
    extract subarchives, repair/remove invalid entries.
    While certainly slower than a compiled program, it is much easier to extend, which is useful since "broken" archives are usually broken in a way not anticipated, hence not considered in a compiled program. CArDMiner allows to apply "snippets" of python code to the channels/values being copied, which are much quicker to write/modify than a complete code/recompile/install cycle for the ArchiveManager would be.
    
• 07/24/2000
  Patch #3:
  Investigated performace differences on Win32 and Linux, allow LowLevelIO to use either fd or FILE pointer.
• 07/20/2000
  Patch #2:
  CHAR data type is now supported, treated like a number.
• 07/3/2000
  Patch #1:
  
  • New doc/libio/Layout.htm
  • Fixes for SGI, cleaning up signal handling in Engine/main.cpp
• 06/27/2000
  Version 1.4:
  
  • Scripting support moved to SWIG, replacing the tcl-only ATAC API.
  • Scripting allows simple write-access to archive, we are working on a python-GUI for archive management
• 06/26/2000
  Patch# 6:
  Engine: used 1 month as file default, not 24h
  Engine: more thread safe signal catching
  HTTPServer: Empty client would linger open and cause Engine to hang. Those could be cause by keeping the "Reload" button down in the web browser. How those connections are closed. Hammering the HTTPServer does still consume CPU up to the point were the WriteThread cannot run, so overwrites can occur. But things should be fine afterwards. The best solution might be a seperate, lower-priority thread for the HTTPServer. This is put off for now.
  LibIO/Bin*: replace stdio by low level IO (open/read/write) to track down thread-related problems.
  
• 06/14/2000
  Patch# 5:
  Testing: SWIG for scripting interface,
  replacing ATAC with CASI - "Channel Archiver Scripting Interface"
  Currently: ported tcl scripts to slightly different API, creating python support.
  
• 06/02/2000
  Patches:
  More careful checking for "MultiArchive" master file.
  Other fixes to help Chris Larrieu compile with some HPUX compiler: Overlooked namespace std issues, rdbuf()->freeze().
• 05/25/2000
  Patches:
  Optional namespace std,
  removed nested class definition for LibIO/CtrlInfoI.
• 05/04/2000
  Version 1.3:
  
  • "MultiArchive" can read more than one archive, easing the previous housekeeping problems: ArchiveEngine can now be restarted e.g. monthly, creating easier to manage subarchives which can still be viewed as a whole via the MultiArchive.
  • Central ArchiverConfig.h file for sources under ChannelArchiver directory, allowing compile-time configuration of Engine, Manager, CGIExport, ... in one place.
  • WinBrowser (and Exporter class) can export linear interpolated spreadsheet data.
  • Adjustments for hp700 at BESSY
  • TCL examples: better response to window resize.
• 03/23/2000
  Version 1.2:
  Optional password for stopping the ArchiveEngine over the Web, to be configured via Engine/Config.h.
  TCL example: PlotBrowser.tk
• 03/01/2000
  Version 1.1:
  More documentation about Engine, Manager, ATAC.
  ArchiveManager can read/write ASCII files.
  (for repairing archives the hard way)
  atac allows to read arrays, more example tcl/tk scripts to list channels, dump values, browse values in a list box or do X-Y-Plots.
• 02/09/2000
  Intruduced version/release scheme to better keep track of changes since the software is no longer only at LANL.
  First Release, Version 1.0.
  Fixed memory leak on RedHat 6.x (should appear on all systems that use that version of egcs) by replacing std::string with self-written stdString, maintaining API.
  Fixed CtrlInfo size problem for compatibility with wrong size calculation of original engine.
• 01/28/2000
  CGIExport: support for retrieving status info and "filled" export where unchanged values are repeated; channel list and channel information are now sorted.
  ATAC: Scalar access to data; allows to create e.g. beam statistics with plot.
  WinBrowser: Fixed axis plotting code.
• 12/18/1999
  I/O Library: Split pure interface classes from the actual implementation for the original, binary data file format. This will now allow the independent developement of other data file formats.
  ATAC: Started TCL interface to the I/O library.
• 11/08/1999
  The Archive Engine is now multithreaded.
  Fixes as a result of Test run on LEDA at LANL.
  In a test setup the Engine ran fine until the disk got full (PC, original configuration files from LEDA, ca. 1200 channels every 30 secs, free disk space of 200MB used after 5-7 days). Switched to/added standard EPICS make system Makefiles.
• 10/18/1999
  Profiled & enhanced engine:
  Reduced #of disk accesses.
• 9/24/1999
  No warnings left with MS VC++ or g++ (egcs-2.91.66) except for Tools/gnu_regex.c (only needed for Win32).
  Eliminated calls to string::copy to make it work on RedHat 5.2.
  Some changes to the Engine to improve performance at higher rate of incoming values.
• 9/15/1999
  The new archive engine can read the config files that we use on LEDA and even does some of what's written in there. It does handle disabling channels and creates new data files, e.g. one per day. The web interface is still simple and on a fixed port (4812), but already shows status information for the engine, groups, channels, and allows adding channels to groups (not permanent, though).
  The IO-library has only a few warnings left when compiled with egcs (no warnings with MS VC). Several bugs have been eliminated. I'm pretty sure that the sequential readout part can stay as it is, buffered read might also be OK. The write portion is likely to change while I'm still working on the engine. As soon as the engine is believed to be stable enough so that I can run it in parallel to the old engine I'll 'freeze' the IO-library by actually building it as a library. I might also need a more sophisticated version number scheme then.
• 6/28/1999
  Compiles on Solaris with egcs
• 6/25/1999
  The Archive class now takes a full pathname (with '/' or '\' on Win32, Unix paths with '/' only) and opens data files in that directory without calling chdir but generating the appropriate names. Thus you can now open more than one Archive.

---

-Kay (kasemir@lanl.gov)