* Copyright (c) 2002 The University of Chicago, as Operator of Argonne
* National Laboratory.
* Copyright (c) 2002 The Regents of the University of California, as
* Operator of Los Alamos National Laboratory.
* EPICS BASE Versions 3.13.7
* and higher are distributed subject to a Software License Agreement found
* in file LICENSE that is included with this distribution.
Synchronized Time Stamp Support
Synchronized Time Stamp Support
Author: Jim Kowalkowski
MODIFICATION: 04AUG1998 (Marty Kraimer)
The routine TSsetClockFromUnix was made global. Calling it forces a resynchronization
with a host time server. This is useful to force a master timing ioc to
New software has been added to IOC core to maintain time stamps. The new
software has the ability to maintain time stamps over all IOCs on a network.
The purpose of this paper is to explain how EPICS will synchronize the
time stamps. In addition, this paper will explain how to configure and
use the new EPICS time stamp support software.
Time Stamp: Two long words representing the time in seconds/nanoseconds
past an epoch. The first long word represents the seconds past the epoch,
the second long word represents the nanoseconds within the second. The
EPICS epoch is currently January 1, 1990. A commonly used Unix epoch in
January 1,1900 or in vxWorks case, January 1,1970.
Event System: A hardware based subsystem for delivering events to all
IOCs on a network. Typical events are the "tick" event and the "reset counter"
Event System Synchronized Time: Time stamps maintained using an event
system. The clock pulses which increment the time stamps are provided by
a single source, all IOCs increment time stamps using the single source
Soft Time: Each IOC increments the time stamps using it's own internal
Master Timing IOC: The IOC which knows the actual time and is the source
of the actual time to all those who inquire.
Slave IOC: An IOC which relies on a master IOC to provide the actual
time. This IOC will keep it's time stamp in sync with a master.
Soft Slave: An IOC which uses Soft Time and synchronizes it's time stamp
with a master.
Event Slave: An IOC which uses Event System Synchronized Time to maintain
it's time stamp. This type of IOC uses the master to verify that it's time
stamp is correct.
Soft Master: A Master Timing IOC that maintains it's time stamp using
a private clock.
Event Master: An IOC which uses Event System Synchronized Time and is
a Master Timing IOC. In addition, this IOC is the source of the clock pulses.
Time stamps are maintained across IOCs using UDP with a master/slave relationship.
A master timing IOC is responsible for knowing the actual time. A slave
IOC uses the master to verify it's own time stamps. Time stamps are maintained
in two fashions: using an event system or using IOC tick counters. Both
differ radically and require some explanation.
Event System Support
The time stamp software has special requirements in order to provide event
system synchronized time. The time stamp software assumes that hardware
is present which has finite counters for counting pulses or ticks delivered
to it from some master tick generator. The software further assumes that
every so other, before the finite counters overflow, a signal or event
will occur to reset the counters to zero. The software requires access
to these counters at any time and notification when the "counter reset"
arrives. The combination of hardware tick counters and reset event notification
gives the software the ability to maintain time. If the event system supports
other events, it is required that the software be notified of the occurrence
of each event. The event time will be recorded in a table which holds one
time stamp for each possible event so that a user can inquire as to when
the event occurred.
Soft Time Support
If no event system is present, the time stamp software can operate in a
software timing mode. In this mode, no events are available, only the current
time can be retrieved. An IOC using soft time will increment a time stamp
using the system clock (usually updated at 60Hz). The master/slave relationship
allows a slave to periodically ask a master for the correct time. At this
point the slave's time stamp can be adjusted to match the masters.
Role of a Master Timing IOC
A master timing IOC has difference responsibilities depending on if it
is an event master or a soft master. An event master will be an event generator
(create "tick" and "counter reset" events). When the event master detects
a "counter reset", it broadcasts the time the event occurred to all event
slaves on the network. The slaves can use the time stamp data to verify
their own time. A soft timing master does not have any events, so no broadcasting
is done. Both soft and event masters have the ability to be queried at
any time by slaves for the current time.
Role of a Slave Timing IOC
A slave timing IOC also has difference responsibilities depending on if
it is an event slave or a soft slave. An event slave contains hardware
to manage "tick" and "counter reset" events. The time stamp support software
uses this information to maintain time. An event slave will listen for
broadcasts from the event master timing IOC and use the information to
verify it's time. A soft slave periodically queries a master timing IOC
to get the current time.
Design Specifications Summary
Event System Synchronized Time
All IOCs will have identical event times. In addition, all IOCs will maintain
the same current time.
The minimum events which must be supported is two. One must be used
as a "tick" event and one must be used as a "reset hardware tick counter"
event. The second will be used to update the time stamp representing the
An optional event can be used as a "heartbeat" event. This event can
be used to signal errors.
All IOCs of this type will maintain time stamps which are within two clocks
ticks or 1/30th of a second of a master. The master may be a designated
IOC or the Unix boot server. A Unix boot server master or the server pointed
to by the EPICS environment variable EPICS_TS_NTP_INET must have NTP available
User Level Interface
General Use Functions
Three functions exist in the synchronous time stamp support software for
the user to retrieve time stamps:
To retrieve the time stamp which represents the time that an event occurred,
use TSgetTimeStamp(). TScurrentTimeStamp() can be used to
retrieve the time stamp which represents the time of day. TsLocalTime()
is the function for returning the current time stamp in the old time stamp
driver, the routine can still be used to retrieve this information. The
function TSgetFirstOfYear() attempts to give the caller a time stamp
representing January 1 of the current year relative to the vxWorks epoch
TSgetTimeStamp(event_number, struct timespec* time_stamp)
TScurrentTimeStamp(struct timespec* time_stamp)
TSgetFirstOfYearVx(struct timespec* time_stamp)
An EPICS enironment variable named EPICS_TS_NTP_INET exists which can
be set to point to an NTP server. The default NTP server is the IOC boot
The following functions can be run from the vxWorks shell to give information
about the time stamp driver.
TSreport(): Display all the information about the current configuration
and status of this time stamp driver.
TSprintRealTime(): Call the ErGetTime() function described below
(if present) and print the time stamp returned from it. Also print the
current EPICS time stamp.
TSprintTimeStamp(event_number): Print the EPICS time stamp for the
TSprintCurrentTime(): Print the EPICS time stamp representing the
TSprintUnixTime(): Send a time stamp query transaction to the boot
server or NTP server and print the time stamp returned.
TSprintMasterTime(): Send a time stamp query transaction to the
master time server and print the time stamp returned.
A global variable exists named TSdriverDebug. Setting this variable
to a positive value in the vxWorks start up script will inform the time
stamp driver to print information about what it is doing. The greater the
value, the more information the driver will print. The number can be set
and adjusted to any value any time while the IOC is running.
Record support will have the ability to tie the record processing time
to an event. This means that a user can specify that processing of a record
is due to an event (from the event system). When the record gets processed,
the time in the TIME field of the record will be the time when the event
occurred. In order to support the event times from record support, two
new fields will be added to dbCommon. The fields will be Time Stamp Event
(TSE), and Time Stamp Event Link (TSEL). The TSE field will be the actual
event time the user is interested in. The TSEL field will be a link used
to populate the TSE.
To facilitate the use of the time stamp support software, a new record
support function will be added:
This routine uses TSgetTimeStamp(prec->tse, &prec->time) to set the
processing time of the record. The new recGblGetTimeStamp() will replace
the existing call to tsLocalTime(). If the TSE field is zero (the default),
then TSgetTimeStamp() will report the current time. It is important to
remember that if a TSE field is set, then the processing time (in field
TIME) will always reflect the last time the event occurred.
The synchronous time stamp support software is configured by calling TSconfigure()
from the "startup.cmd" file. The parameters to this routine are:
This routine must be run before iocInit(). The synchronous time stamp support
software is initialized as part of iocInit. Running TSreport() after
iocInit() will produce a report which shows the current state of the driver.
master_indicator: 1=master timing IOC, 0=slave timing IOC, default
sync_rate_seconds: The clock sync rate in seconds. This rate tells
how often the synchronous time stamp support software will confirm that
an IOC clock is synchronized. The default is 10 seconds.
clock_rate_hz: The frequency in hertz of the clock, the default
is 1000Hz for the event system. The value will be set to the IOC's internal
clock rate when soft timing is used.
master_port: The UDP port which a master timing IOC will use to
receive time stamp requests. The default is 18233.
slave_port: The UDP port which a slave will use to receive time
stamp information from a master.
time_out: UDP information request time out in milliseconds, if zero
is entered here, the default will be used which is 250ms.
type: 0=normal operation, 1=force soft timing type.
Event System interface
The event system interface consists of seven function which can be provided
by an event system. The synchronous time stamp support software uses a
card number of zero on all functions that require a card number. The functions
are as follows:
The definition are as follows:
long ErHaveReceiver(int event_card_number)
long ErGetTicks(int event_card_number, unsigned long* ticks)
long ErRegisterEventHandler(int event_card_num,EVENT_FUNC event_func)
long ErRegisterErrorHandler(int event_card_num,ERROR_FUNC error_func)
long ErForceSync(int event_card_number)
long ErGetTime(struct timespec* time_stamp)
All of these routines are checked to exist when the time stamp support
code initializes. All have some kind of default routine provided if they
are not found, most of which just return an error condition. The functions
which the time stamp support software registers (event and error) have
the following format:
ErHaveReceiver(): Returns -1 if no event (timing) hardware present,
else returns the number of supported events.
ErGetTicks(): Returns the number of ticks since the last hardware
tick counter reset.
ErRegisterEventHandler(): Informs the event system of a function
to call when an event occurs, the format of the function will be defined
ErRegisterErrorHandler(): Informs the event system of a function
to call when an error occurs, the format of the function will be defined
ErForceSync(): This function will force an event generator to generate
a tick reset event and send it.
ErGetTime(): This function returns the actual time. The intention
here is that this function will retrieve the actual time from GPS system
or equivalent and return is in time stamp format.
ErSyncEvent(): Return the event number for the tick reset event.
ErDirectTime(): Return 0 for normal operation, return value not
= 0 for systems that has direct time access, such as a GPS.
ErDriverInit(): The time stamp driver initialization function will
call this user supplied function before it returns and after it sets up
the vxWorks clock and attempts to set up the TIMEZONE variable. This can
be useful to initialize a system such as a GPS (the year can be determined
using the vxWorks ansiTime library).
Here the Card is the event system board of interest (always zero), the
EventNum is the event that occurred, and the Ticks is the number of ticks
since the last board tick counter reset.
TSeventHandler(int Card,int EventNum,unsigned long Ticks)
TSerrorHandler(int Card, int ErrorNum)
In addition to the above functions, a global variable exists on the
IOC which can be used to indicate that direct time is available. Setting
the variable TSdirectTimeVar to a nonzero value has the same effect
as providing the ErDirectTime() function.
Creating Direct Time Support
Most of the above interface functions apply only when special event hardware
is present. A much simpler configuration is when a GPS is present and time
stamps are distributed using IRIG-B to all or most of the IOCs. The easiest
way to implement this scenario is to define ErDirectTime() to return the
value one and define ErGetTime(). The job of ErGetTime() will be to actually
be to generate and return the time stamp representing the current time
(from the EPICS epoch). At system initialization, the actual time is retrieved
from a Unix server through NTP or the Unix time protocol, so that year
will be valid in the vxWorks time clock. At the GPS driver initialization,
the vxWorks function clock_gettime() can be used to calculate the year.
Record support will internally eventually call ErGetTime() each time the
record is processed. The GPS driver should be included as part of drvSup.ascii,
so it initialized in the proper order.
An IOC can be configured to run in one of four ways: synchronous-master,
soft-master, synchronous-slave, soft-slave. Each mode maintains time differently.
When the time stamp support code initializes, it determines the mode which
it will operate in based on configuration parameters and event system function
Synchronous timing is determined by the presence of event system hardware.
The function ErHaveReceiver() gives this information. All IOCs using synchronous
time will have the exact same time stamps. The "tick" event increments
all IOC's event system tick counters, the "reset tick counter" resets the
counter to zero and posts the event to the time stamp support software.
The time stamp support software system knows the current time by looking
at the last "tick reset event" time and querying the event system with
ErGetTicks() for the number of ticks which have elapsed since the last
reset. Since the event handler function gets called with the "ticks since
last reset", an event time is the last "tick reset event" time plus the
ticks count in the call.
A synchronous master is determined by the first parameter to TSconfigure()
and the presence of event hardware. A master must have an event generator
and an event receiver. Since the event system is configured from EPICS
database records, the database must have records in it to initialize the
event system. The master is responsible for providing the "tick" event
and the "reset tick counter" event.
When the time stamp support software's event handler function on a master
timing IOC receives a "reset tick counter" event, a time stamp message
is broadcast out on the slave_port (configured by TSconfigure()). The master
timing IOC is also listening on the master_port (also configured by TSconfigure())
for incoming requests for time stamp information.
At boot time, a master will set the vxWorks clock from the Unix boot
server time. The time is retrieved from the boot server using NTP or the
time protocol, or the server pointed to by the EPICS environment variable
EPICS_TS_NTP_INET using NTP. The GPS module takes an unknown period of
time to sync to the correct time, so it can not be used until it's time
is valid. The event system is not up and running until record support initialization
is complete, therefore the event system time stamps are not ticking until
the event system initializes. At the time of the first sync, the event
system is known to be up, and the "reset tick counter" or "sync" event
is set to the current time (vxWorks clock).
A synchronous slave is determined by the first parameter to TSconfigure()
and the presence of event hardware. This mode is automatically selected
if no TSconfigure() call is used in the "startup.cmd" file. This type of
slave must have an event receiver. The EPICS database must be configured
to initialize the event system.
When a synchronous slave configuration is determined, the IOC broadcasts
a request for master on the master_port port. If a master is found, then
all current time, sync rate, and clock are extracted from the master's
response (the sync rate and clock rate from the TSconfigure() are overwritten
to match the master's configuration). If a master is not found, the slave
IOC goes into a polling mode to try to find a master every two minutes.
While a slave has no master, The IOC's clock is initialized from the Unix
boot server and TSgetTimeStamp() returns the vxWorks time clock value.
The slave experiences the same problems as the master upon boot.
Soft time IOCs use the 60 hertz clock available from vxWorks to maintain
a time stamp. The IOC will increment a time stamp at a 60 hertz rate. Soft
time is determined by the absence of event system hardware. All soft timing
IOC's will not have the same time stamps.
A soft master is determined by the first parameter of TSconfigure() and
the absence of event hardware. Upon boot, the master soft timing IOC retrieves
the current time from the Unix boot server. The IOC sets up a soft time-stamp
counter using a one tick watch dog and sets the vxWorks clock. From this
point on, the master runs using the soft time-stamp counter. The master
listens on the master_port port for requests for the master's current time.
A soft slave is determined by the first parameter of TSconfigure() and
the absence of the event hardware. This mode is automatically selected
if no TSconfigure() is present in the "startup.cmd" file. The basic operation
of a soft slave is to synchronize the time stamp with a master when possible
or to the Unix boot server if no master is available. Upon initialization
the time stamp support code determines if a master is present on the network,
and if NTP support is available from the Unix boot server or the server
pointed to by the EPICS environment variable EPICS_TS_NTP_INET. The slave
will request time stamps from a master or unix server at sync_rate_in_seconds
rate from TSconfigure(). If the time stamp on the slave is found to be
off, the clock will be incrementally adjusted so that by the next sync,
the clock will be corrected to match the master's time stamp. A soft slave
will automatically switch between a master and the unix boot server depending
on if the master is available or not. In order to sync with the unix boot
server, NTP must be available there for query only.
Home page for Jim
firstname.lastname@example.org (Jim Kowalkowski)
Argonne National Laboratory Copyright