CMLOG WISH LIST
Ideas for Browser and Server improvements for a release beyond cmlog V2.0
0.0 Save messages over server crash
In case of server crash, all messages from logging clients that are still logging messages will be lost. In future, CMLOG will provide a way to save these messages which then can be merged into the centralized database by a simple utility. (this is stated in the cmlog document.
1.0 There are interesting issues that didn't get included in 2.0.***
They are listed here: 2.0 Wish List and test status That page is for the last major release 2.0. Look on that page for items not included in Version 2.0 that might be implemented in the future. One that comes to mind is the issues with the -hu browser option.
1.1 Provide a way to pump SLC errorlog data into the cmlog database ***.
RonC brought up this idea.
I'd like to see a program which puts, say,
6 months of old ERRLOG data into CMLOG, and adds new stuff on top.
That way people wouldn't have to wait for it to be useful for bigger searches.
We could also give thought to some interesing compression and hashing, whereby
the repeated strings (even just parts of messages) could be stored just once,
reducing the file size greatly.
1.2 Limited Storage support. ***
If cmlogServer is running on a gateway with limited storage, then how could we support queries over long periods of time? One solution lies in NFS exporting the cmlogDatabase data. Then a cmlogServer running on another machine could browse that database and older database files (not on gateway). But, because of the buffering of data in cmlogServer, a query would not come up to current time.
Section 2.0 and other sections of 1.0 talk about allowing the browser to connect to multiple servers. Maybe the solution lies in there instead.
1.3 Server timestamping of log messages. ***
On the server, add timestamps to the messages in the diagnostic log file. See the previous wish list for more details on this idea.
1.4 Run a script when a condition occurs. ***
Steph. brought up this interesting idea.
The ability to run a script when a given condition occurs (like
severity>2).
The ability to send email if a given condition occurs.
A way to indicate a beep or other sound if a condition occurs.
The script itself could do the beeping and emailing and whatever else based on arguments passed to it - so cmlogrc would have script_name and script_args (our dm execute-scripts do this type of thing).
One approach would be to provide a field in codeConversion (cmlogrc) in addition to color.
4 Fatal Red Flash Script_name Beep email_address
But a different approach would be needed for the ablility to do something given a search string. A new verb (or addition to updateMessage) might do the trick.
1.5 Browser auto-reconnect to dead server. ***
Provide a way for the browser to automatically try reconnecting to the server after the server has gone away. This could be controlled through the cmlogrc file, giving an option like: auto_reconnect_time = 60. The 60 is the time between retries. If nothing is specified for this option, then it works like it does now (with no retries).
1.6 Browser expert mode. ***
Provide an expert mode for the browser where there would no conformation popup dialogs. For example, there would not be a dialog box that comes up when the user disconnects from the server saying "Are you sure?". That could be implemented by having a setting in the cmlogrc file like this: conformation_dialog_boxes = yes/no. If nothing is spcified, then the default is "yes".
1.7 Browser -icon flag.
Add support for -icon flag which would cause the browser to come up iconized.
1.8 Browser -background flag.
Add support for a -background flag which causes the browser to come up behind x-applications that are already on the display. By default it comes up in front of them.
1.9 Browser dump all tags command.
Provide a way, from the browser to dump all tags associated a given message. This would be useful for debugging. This could be invoked from the pull down menu and possibly added to the API. For example, a client is sending messages and we don't know what tag they used (set). We needed this when we connected ALH. It was sending messages but we don't know what all the tags were that it was setting. So, from the browser, under query/help, we could have a button called 'dump entire message' which would give us all the contents of the message.
1.10 Browser cut and paste operations. ***
The browser does not support mouse cut and paste operations like most other x-applications do.
Greg adds: We're aiming for is intuitive use, so
ctrl-c ctrl-v will work, plus be available from the edit menu.
1.11 Allow client timestamping of cmlog messages. ***
Fix a 'feature' in the timestamping of messages
There's a bug in cmlog time processing. We pass the
vms timestamp from the error message in the "time"
cdev tag but don't ever use it in the browser. That's
because the cmlog client code timestamps error messages
in the cmlogTime cdev tag and we can't use that one because
cmlog client will just write over anything we put in there
with it's own time stamp. Jie said he will fix that in
the next release where we can pass a value ourselves.
Epics messaes have the the timestamp in cmlogTime, so
we want to use that tag too so that the time shows
up on the browser using one tag for slc and epics.
So for now, we let cmlog client code timestamp the vms mesage
instead of using the timestamp that came from vms. But,
we do pass the vms timestamp in the 'time' cdev tag in case
anyone wants to look at it.
Actually, we want two time stamps (according to Matthais Clausen). We want "client time stamp", and we want "time the data was logged by cmlogServer". The former is necessary if we're going to correlate events that occur on the ioc with messages that are in cmlog.
1.12 Increase speed of queries.
Queries (searches) for historical data can take quite a long time. String searches take the longest. Various things could be done, like indexing the data.
1.13 Provide compressed storage.
It would be nice to provide compressed database storage. Disk usage can be quite large since so many unix applications like EPICS just send strings. Compression could be configurable. For example, cmlogServer could be told to only compress data that is older than N days. There could be various levels of compression. I assume that data that is "less compressed" could be retrieved faster.
1.14.Multiple servers
For a large system like NLC, it would be necessary to store data with multiple servers. That is, some clients would write to one server and other clients other servers. That, of course is already possible. The tricky part is, that it would be desired to use one browser interface to browse the data which was storred across all servers at once. One way to accomplish this would be to have all the servers passing the data to some common process for storage and then store in one common database from that process. Providing a reliable system to do this could be tricky. Failover should be considered at every failure point so messages are not lost. Additionally, the amount of data that each server buffers in memory for some amount of time before writing to disk is an issue because one server could flush the data before another and it might be hard to keep the storred data in time order since the first flush may be older data than is held in the other server's memory.
1.15 EPICS integration issue.
There are issues with Epics. I forget the details, I seem to remember that we needed to request that Epics code be modified to do a better job of settting values for some tags when messages are logged from the ioc. (from ronm)
Jie adds this comment:
Find a way to parse the epics logging string so that
cmlog can display the string under an appropriate tag. The simple way to
do it to put each field inside a pair of double quote. Now,for the epics
string, there is no way to parse it.
1.16 Ronm's favorite ideas from the 2.0 Wish list that were not implemented. (see that page for details on each item. Only item number and title is listed here).
1.5 cmlog browser..... Query buttons bump next field value.
1.6 cmlog browser..... Sticky query buttons.
1.11 cmlog browser..... Support up arrow for update/query selection string history.
1.13 cmlog browser..... save complete context in .cmlogrc
2.2 cmlog client..... Client failover.
2.7 cmlog client..... Add meetering summary messages.
3.8 cmlog Server..... Remove server context messages
1.17 Messages logged with cmlog_logmsg() should go to vxWorks console too.
Stephanie requests this:
On the IOC, when I log a message using cmlog directly (not going
through epics via epicsPrintf), the message does not appear
on the IOC console. We use cmlog_logmsg to log messages from a sequence
and these messages do NOT appear on the console.
Is there some sort of flag I can set to
see the messages both through the cmlog browser AND on the
IOC console? Please provide a flag so that we can turn on and off console logging
at the prompt like with EPICS.
1.18 On the browser query box, either get rid of the "search all messages" button or reverse it so it's set by default.
We don't need the slider if we don't have the box of course.
Another option is to have the button say "Search a subset number of messages" and keep the slider.
1.19 The browser query search syntax for strings doesn't work well for strings
Searching for a string (like Host has "MCC"), works for "has".
But, equals "==" doesn't work well. It's funny for < and > too.
Can it tell if the target is a string and not have "==" and "< etc"?
Or, maybe a whole new syntax or interface would be better.
We could the "help" button and not allow user to enter a C-like line and only use what is now in Help.
1.20 The browser backspace key doesn't work right. It gobbles characters forward
1.21 Provide the ability to plug in different databases (like oracle).
1.22 Provide slac like metering, or at least something better than what is there now
1.23 On the browser Query menu (and update menu?), provide a new option in addition
to what is already there which allows the user to ask if any tag has a given value. If
the value is in one or more fields, then, the line is displayed. This is like vms errlog
where you can say "string=" and it searches all fields for the string. Thanks to Mike for
mentioning this.
2.0 Ideas from Jie Chen. ***
1) Allow a browser to connect to multiple servers. The browser can be
operated in two modes. One in automatic mode which means the browser uses
broadcast mechanism to find a cmlogServer. The other mode is to allow
operator or a configuration file to select multiple servers for a
particular display window. (RonM wonders if it would be possible to be connected to multiple servers at the same time and browse across all those servers).
2) Allow multiple client daemons running on single host. This will
allow applications to send messages to different servers. RONM: I saw
this on a SOLARIS host, when I was trying to run two applications
(alarm handler and archiver) and have them log to different
cmlogServers (prod and dev). I set CMLOG_HOST and CMLOG_PORT differently
for each applicaiton. But, they ended up logging to the same cmlogServer.
That's because there can be only one cmlogClinetD (now) on solaris
and both applications connect to it. It probably determines which
cmlogServer to connect to based on how it was started (if it was
started on it's own). Or else, the first application that starts sends
it the HOST and PORT to which to connect to.
3) Java based simple browser to enable viewing cmlog messages inside
a netscape.
4) 64 bit support. (We have to do it sooner or later, 64 bit is
coming :-).
5) Add new APIs to allow message not logged (These messages are just
displayed).
6) Allow searching to go backward.
7) Remove ACE from cmlogClientD code so that it can be ported easily
to other platforms (VMS, NT).
8) This is may not be necessary. Change DB+ library to new version.
9) Add an option (either command line or configuration file) to cmlogServer so that a server can become read only server which can be useful for querying purpose or web.
The above features actually start addressing problems of salability
of cmlog to the size of NLC. More ideas are welcome.
In addition, I have some other things in mind. In particular I am
interested in distributed object or agent based systems integrated with
latest security technology. Those systems or technology can be applied
to software systems such as large data archiving system (to be developed
:-) ), etc.
SLAC
Last modified: 07-Jul-2005 by Ron
MacKenzie