README_devIocStats Last Updated 08/11/2011 ------------------ I - General Notes on devIocStats - All Operating Systems -------------------------------------------------------- Include either devIocStats.dbd OR devVxStats.dbd in the IOC application, not both. devIocStats.dbd (preferred) provides DTYPs that start with "IOC" and devVxStats.dbd provides DTYPs that start with "VX" (included for backwards compatibility). String (stringin) Records (DTYP = "IOC stats"), INP = @one of the following: ============================================================================ startup_script_[12] -path of startup script (2 records) Default - uses STARTUP and ST_CMD environment variables or startup and st_cmd symbols. If STARTUP is not available, uses IOCSH_STARTUP_SCRIPT. bootline_[1-6] -CPU bootline (6 records) (set to " for all except RTEMS and vxWorks) bsp_rev -CPU board support package revision (set to " for all except RTEMS and vxWorks) kernel_ver -OS kernel build version epics_ver -EPICS base version engineer -IOC Engineer Default - uses ENGINEER env var or engineer symbol. If ENGINEER is not available, uses LOGNAME or USER env var. location -IOC Location Default - uses LOCATION env var or location symbol. hostname -IOC Host Name from gethostname pwd[1-2] -IOC Current Working Directory (2 records) from getcwd up_time -IOC Up Time Except for up_time, the support is static. The record only needs to be processed once, for which SCAN=Passive and PINI=YES is recommended. When a string cannot be determined, the PV will show "" or "" or some other variation. String (stringin) Records (DTYP = "IOC epics var"), INP = @ ============================================================================ The record will report an error if INP is not a valid EPICS variable listed in envDefs.h. Gets the value of the EPICS environment variable in the INP field using envGetConfigParam. The record only needs to be processed once, for which SCAN=Passive and PINI=YES is recommended. String (stringin) Records (DTYP = "IOC env var"), INP = @ ============================================================================ Gets the value of the environment variable in the INP field. Unless the environment variable changes on-the-fly, the record only needs to be processed once, for which SCAN=Passive and PINI=YES is recommended. If the environment variable does not exist, the PV will show "". Analog In (ai) Records (DTYP = "IOC stats"), INP = @one of the following: ========================================================================= records - number of records ca_clients - number of current CA clients ca_connections - number of current CA connections suspended_tasks - number of suspended tasks The following are implemented for RTEMS, vxWorks, and some soft IOCs (see notes below for exceptions and more detail): free_bytes - number of bytes not allocated on this machine allocated_bytes - number of bytes allocated (for soft IOCs, physical memory used at the moment (RSS)) total_bytes - number of bytes physically installed on this machine sys_cpuload - estimated percent CPU load on the system no_of_cpus - number of CPU cores on the system ioc_cpuload - estimated percent CPU utilization by this IOC fd - number of file descriptors currently in use max_fd - max number of file descriptors Note - free_bytes, total_bytes, sys_cpuload, no_of_cpus can be instantiated once per system instead of per IOC if desired (ie, when multiple IOCs run on the same system). The following are implemented for RTEMS and vxWorks IOCs only and set to 0 for other types of IOCs: max_free - size of largest free block min_sys_mbuf - minimum percent free system MBUFs sys_mbuf - number of system MBUFs inp_errs - number of IF input errors out_errs - number of IF output errors The following are implemented for vxWorks IOCs only and set to 0 for other types of IOCs: free_blocks - number of blocks in IOC not allocated allocated_blocks - number of blocks allocated min_data_mbuf - minimum percent free data MBUFs data_mbuf - number of data MBUFs The following are implemented for RTEMS IOCs only and set to 0 for other types of IOCs: workspace_alloc_bytes - number of RAM workspace allocated bytes workspace_free_bytes - number of RAM workspace free bytes workspace_total_bytes - number of RAM workspace total bytes Analog In (ai) Records for Cluster Statistics (RTEMS and vxWorks IOCs only) (DTYP = "IOC stats clusts"), INP = @clust_info where: ========================================================================== pool - 0 (data) or 1 (system) (the data pool is not implemented for RTEMS and all data pool values are set to 0) index - index into cluster array type - 0=size, 1=clusters, 2=free, 3=usage Analog Out (ao) Records (DTYP = "IOC stats"), OUT = @one of the following: ========================================================================== memoryScanRate - max period (sec) at which new memory stats can be read, default = 10 sec fdScanRate - max period (sec) at which file descriptors can be counted, default = 20 sec cpuScanRate - max period (sec) at which cpu load can be calculated, default = 10 sec caConnScanRate - max period (sec) at which CA connections can be counted, default = 15 sec Subroutine (sub) Records, SNAM = one of the following: ====================================================== rebootProc - System reset or epics exit. scanMon - Calculate time between record updates. Set INAM to scanMonInit. II - Notes on Soft IOC Implementation of devIocStats ---------------------------------------------------- (1) The suspended task count will only include tasks that call taskwdInsert on startup, including all of iocCore tasks and all sequence tasks. Some (many?) support modules that create tasks neglect to call taskwdInsert. (2) The reboot subroutine record will exit the IOC process. It is assumed some soft IOC managing process will automatically restart the soft IOC. (3) None of the available methods returns a true value for CPU utilization for multi-threaded applications on multi-core systems. (4) Implementation of system CPU utilization on Linux and Solaris show the same values as "top". A posix implementation that uses getrusage() is provided for MacOS where everything else seems to be unavailable. (5) Implementation of IOC CPU utilization return 0%-100% of CPU load with respect to the system (divided by the number of CPUs where applicable). On Solaris, this matches the "top" results. On Linux, "top" normalizes the process values per core (so that top_value = ioc_cpuload*no_of_cpus). To allow showing or changing the normalization, support for the number of CPUs is provided so that a calc record can be used to get the desired value. (6) Memory usage is available for Linux only. (7) The following records can be instantied per system instead of per IOC if desired (value is the same for all IOCs on the same system): free_bytes, total_bytes, sys_cpuload, no_of_cpus IV - Notes on RTEMS Implementation of devIocStats by Till Straumann -------------------------------------------------------------------- Memory / Heap Statistics ======================== - gathering heap statistics could be expensive; I wouldn't want to run this too often w/o knowing how it is implemented and how it could impact the running system. Furthermore, vxStats reports 'free', 'used' and 'total' memory. Unfortunately, it just assumes 'total = free + used' instead of the correct total number (difference could give a hint about fragmentation). However, if you know the true total amount of memory you can estimate fragmentation in your head... - RTEMS also uses a so-called 'workspace' memory area which is independent of the malloc heap. Some system-internal data structures are allocated from the workspace area. CPU Load Estimate from the IDLE Task (from Phil Sorensen at CHESS) ==================================== - The cpu usage is done the same way as in cpukit/libmisc/cpuuse from the RTEMS source. - Comment from Till: The way they determine CPU usage is more appropriate but only so if the 'billing' algorithm has access to a high resolution clock. This is the case for powerpc under 4.9. The code would have to be modified to make use of the high resolution clock however. If no high resolution clock is available the 'billing' is always in ticks even if no full tick period is used. This tends to show too high usage numbers. CPU Load Estimate from miscu_cpu_load_percentage in ssrlApps ============================================================ #include which defines double miscu_cpu_load_percentage( struct timespec *lst_uptime, struct timespec *lst_idletime ); and void miscu_cpu_load_percentage_init( struct timespec *lst_uptime, struct timespec *lst_idletime ); The header also documents the semantics. In particular, be prepared for NAN which is also returned under pre-4.9 RTEMS versions so your iocAdmin code could try the new routine during initialization and fall back to the vxStats method if NAN is returned. One last thing: if you want to try this routine from the CEXP shell then keep in mind that CEXP has no way to know the type of the return value of any function; it just assumes 'long'. You can, however force CEXP to recast a symbol (in this case to a function-returning-double pointer) so you do (note the exclamation mark which deviates from the C syntax. It tells Cexp that you really want to redefine an already known symbol.) : Cexp@ioclab1> double (*miscu_cpu_load_percentage)() ! Cexp@ioclab1> miscu_cpu_load_percentage() 3.603457 Cexp@ioclab1> CPU Load Estimate using the cpuBurn Method (no longer available for RTEMS) ========================================== - The CPU load estimate works as follows: 1) during initialization (it is assumed the CPU is otherwise idle at this time) a busy loop is executed for a fixed amount of time and the code uses the number of iterations done as a measure for a 'unloaded' CPU. 2) when EPICS is up, a very low priority task is scheduled periodically and executes the same busy loop counting iterations during a fixed time interval. The idea is that any higher priority work will preempt the spinning task and thus reduce the number of iterations it is able to perform. The ratio of iterations under load to iterations while idle is an estimate for CPU load. This algorithm has a few gotchas: 1) implementation of the busy-loop was inappropriate for a modern compiler (which could notice that the loop doesn't do anything and therefore optimize it away). I replaced the code by something that should not be removed by the compiler. 2) CPU load is only estimated during intervals the measuring task is burning cycles. The algorithm is blind while the task is sleeping. 3) Some modern CPUs would rather go into low-power sleep when idle instead of burning cycles -- (unfortunately a chip-errata of the mpc7450 prevents us from using sleep on the MVME6100) Summary: IMO it would be better to use info from the OS scheduler about CPU usage. Unfortunately, this info is not very meaningful unless the scheduler uses a high-resolution clock for billing [available with RTEMS 4.9 on select architectures]. Networking (Mbufs) ================== The RTEMS/BSD stack has only one pool of mbufs and only uses two sizes: MSIZE (128b) for 'ordinary' mbufs, and MCLBYTES (2048b) for 'mbuf clusters'. Therefore, the 'data' pool is empty. However, the calculation of MinDataMBuf always shows usage info of 100% free (but 100% of 0 is still 0). Suspended Tasks =============== Note that e.g., the rtems GDB agent uses a helper task which is suspended most of the time. Hence on systems where the agent is active a count of 1 suspended task is normal. Reboot ====== In RTEMS there is no general 'reset/reboot' system call. The idea is that the RTEMS application is terminated using 'exit()' which should allow a firmware monitor to resume. However, on some boards this is not possible (and 'exit' would not by itself reset the hardware). For rtems 4.9.1, the bsp_reset function is available on most BSP's which performs a hard reset of the CPU (but not necessarily a VME reset). Misc IOC Info ============= The startup script, engineer and ioc_location information is simply read from environment variables "INIT", "ENGINEER" and "LOCATION", respectively. Of course, these variables (in particular "INIT") must somehow be set e.g., (when using GeSys) via the BOOTP commandline. This could have been implemented in a more 'vxworks-alike' way by looking up symbols via cexpsh but using env-vars seems more generic/portable to me. BSP Revision is currently not implemented (since this info is IIRC not present in RTEMS proper; for BSPs distributed with RTEMS the revision matches the kernel revision, however).