What is Cexp? ------------- -- Brought to you by Till Straumann -- -- README,v 1.14 2004/05/21 21:09:21 till Exp -- The Cexp utility is a C-expression interpreter which gives its user access to all symbols present in the currently executing program. Its primary target is RTEMS, an open source RTOS, but Cexp should compile and run on virtually any platform supported by the BFD library. If Cexp is linked to an executable, it is possible to invoke arbitrary functions and to read/write variables (and hence virtually any memory location) by interpreting C-style expressions. Cexp can access object files using either libelf or libbfd. The latter is preferred as it is more powerful: - not limited to the ELF object format. - cexp implements a dynamic/runtime loader using libbfd - using BFD / libopcodes, a disassembler comes for free. However, linking against the BFD library is subject to the GPL (whereas libelf is under LGPL). See LICENSING below for more information. What is New? ------------ - for later versions, consult the ChangeLog file. - 2003/02/14: CEXP_1.2.beta ("Valentine") released. It supports G++ prioritized execution of static constructors/destructors. - 2003/02/13: CEXP_1.1.beta is released. It fixes a bug of the exception handler registration. Makefile.syms has been added and modularized copies of the RTEMS 'cdtest' sample program. Description ----------- CEXP is a simple utility featuring - symbol table - C expression parser/interpreter - type "engine" - user definable variables - recursive invocation / simple scripts - dynamic/runtime 'module' loader - disassembler - CEXP is reentrant Cexp knows about the basic C types (no aggregates), i.e. char, char* short, short* long, long* double, double* long (*)() double (*)() and interprets C-style expressions, e.g. Cexp> printf("Hello world") Cexp> some_variable = 0xdeadbeef Cexp> some_double_variable = *(double*)&some_variable Cexp> a=printf, a && a("The square root of 2 is %g\n",sqrt(2.0)) Symbol Table ------------ A) Using a Symbol Table File - - - - - - - - - - - - - - - On startup, Cexp reads the 'system' symbol table either from the executable itself, or from a stripped-down '.sym' file - such a stripped-down symbol file can be created using the 'xsyms' utility. B) Builtin Symbol Table File - - - - - - - - - - - - - - - The 'xsyms' tool (BFD version only) can now be used to generate a symbol table in C-source form for subsequent compilation and linkage into the final executable. This usually involves linking the executable twice: 1) compile and link all sources, libraries etc.; build an executable 2) xsyms -C executable mysymtab.c 3) compile mysymtab.c (set -I to Cexp source directory, since cexpsyms.h is included) 4) link application again, but this time add 'mysymtab.c' to the list of sources. NOTE: Step 1) linking succeeds without a symtab.o because there's a weak NULL-ptr alias for the builtin symtab. A similar source file can also be generated using the 'ldep' utility (discussed elsewhere, it is part of RTEMS-GeSys). Without access to a symbol table file which also provides Cexp with information about the architecture it is executing on, Cexp might fail to detect the correct BFD CPU architecture which is needed for the disassembler. In such cases, the '-a' command line option can be used to specify the architecture (arch string can be obtained by running 'objdump -f' on the ELF executable). Symbol Type - - - - - - Once the symbol table is read, Cexp guesses the symbol/variable types from the symbol sizes. (Guess is only supported on ELF targets.) In some cases, this guess is obviously wrong, but you can always cast the symbol to the correct type. Arrays are normally 'void' like every symbol for which's type Cexp cannot make a guess. It is still possible, however, to use such a symbol, simply by taking its address and casting to a different pointer type: Cexp> *(((long*)&some_array)+25) evaluates to the unsigned long array element some_array[25]. Alternatively, you can redeclare its type: Cexp> long *some_array ! Cexp> *(some_array + 25) Note the exclamation mark: terminating a variable declaration with an exclamation mark allows you to redefine its type. Only new variables may be declared without the '!'. This protects you from erroneously use existing symbols. Cexp provides a series of lookup functions for symbols. Most noteworthy: lkup(char *regexp) lkaddr(unsigned long addr) These are just normal C functions which can be invoked by Cexp. Note that lkup() takes a regular expression argument allowing for powerful searching. C-Expression Parser / Interpreter --------------------------------- The C expression parser has some restrictions: - no '?' ':' expression - no [], '->' - the '.' operator has a special meaning - see below - no multilevel pointer dereferencing (since e.g. char** is not a valid type, Cexp cannot dereference **addr. You must explicitly cast this: Cexp> *(char*)*(long*)some_char_pointer_pointer which assumes that sizeof(long) == sizeof(char*) for the particular machine. However, most of the other operators are available, including the 'comma' operator and the logical '&&' and '||' with the correct semantics, i.e. conditional evaluation of expression parts e.g. Cexp> (dd=deviceDetect(_some_device_address)) && driverInit(dd) will call the driverInit() function only if deviceDetect() returns a nonzero value. The '.' operator (structure field access) has a special meaning to Cexp: symbol "member" (in an OOP sense) access. Currently, the only member function defined is 'help'. Consult the 'Help' section about details. Function calling: The user is responsible for feeding properly typed arguments; unused arguments will be filled with integral/scalar 0, which on most ABIs is safe. Since the symbol table (on ELF, Cexp is not using .stab but .symtab) provides no info about a function's return type, Cexp assumes all functions to return long. Floating point functions must be cast appropriately: Cexp> ((double(*)())sqrt)(2.345) Or Cexp> double (*sqrt)() Cexp> sqrt(2.345) A lot of type casting can be avoided by re-declaring or using user defined variables, see below. Type Engine ----------- Cexp knows about (and only about) the primitive types listed above. NOTE: Cexp treats ALL integral types as UNSIGNED although no 'unsigned' keyword is used or recognized. The main reason for this is to save typing. Hence char is in fact unsigned char etc. An 'int' type is missing (but could easily added) which means that on some machines (e.g. an alpha) no 32bit type is currently available... User Variables -------------- Cexp supports (fully typed) user variables which are visible in a global namespace (i.e. visible to other instances of Cexp). The name of user variables must not collide with symbols present in the symbol table. (When loading object files [AKA 'modules'], name clashes are ignored. A symbol in a newly loaded module that conflicts with an already defined user variable is silently ignored [object file symbols have a higher priority than user variables]. However, after unloading the conflicting module, the user variable becomes again 'visible'.) A user variable is created simply by assigning it a value which also automatically defines its type. hallo="hallo" creates a 'char*' and assigns it the address of a string constant. The string is stored in 'malloc()ated' memory and 'lives' forever. Subsequent use of the same string (strcmp(a,b)==0) results in re-using the already stored instance of the string. String constants must not be written-to! The type of user variables can be modifed, simply by re-declaring them, in a C-style manner: long hallo results in 'hallo' maintaining its value (the address of the string constant) but interpreting it as a long. Hence chppt = &hallo is an 'disguised' char** and *(char*)*chppt yields 'h' User variables can also hold function pointers and hence can be handy abbreviations for long names and casts. E.g. a convenient variable can be set for 'sqrt()' (which returns a double value): s=((double(*)())sqrt) It is then possible to automatically get the correct return type: printf("Let's print a double: %g\n",s(44.34)) Help Facility ------------- The '.' operator (structure field access) has a special meaning to Cexp: symbol "member" (in an OOP sense) access. Currently, the only member function defined is 'help'. I.e. by typing Cexp> some_symbol.help() You get help information about that symbol. If no help is defined, or if you request 'verbose' help: Cexp> some_symbol.help(1) the symbol address, value and type information are printed (same format as the 'lkup' output). You can add/modify help information to a symbol (e.g. a user defined variable) simply by providing a string argument to the symbol's help member: Cexp> long myvar Cexp> myvar.help("I just created a 'long' variable") Note that help is stored 'per symbol', i.e. when creating an 'alias' as in the example above: Cexp> s=(double (*)())sqrt the help information is not propagated to 's'. If 'sqrt' had help information, it would have to be copied: Cexp> s.help(sqrt.help()) This example shows that the return value of the 'help() member' is the address of the (static) help text string. How to Start/Invoke Cexp - Simple Scripts ----------------------------------------- Cexp has two entry points, cexp_main(int argc, char **argv) and cexp(char *arg1,...) i.e. a 'main' style and a 'vararg' style. The 'vararg' version ends up building an argument list and calling cexp_main(). It is mainly intended for recursively invoking 'cexp()' e.g. for reading a series of lines from a file. Note that the string argument list submitted to cexp() must be NULL terminated. When calling cexp() from cexp() (e.g. to interpret a script), this is not necessary, however, because any unused arguments (up to the internal maximum of 10 args) are filled with zeroes... The calling syntax of cexp_main/cexp is as follows: cexp [-s ] [-h] [-d] [-q] [] -h and -d have the obvious effect of printing usage info and enabling debugging information (only available if configured with --enable-YYDEBUG). The -q ('quiet') flag instructs cexp not to print any normal output (errors are still reported to stderr) on stdout. This behavior can be desirable when evaluating scripts. Note that the -s option MUST be used the first time 'Cexp' is started and it must be provided with an appropriate symbol file. This can be the executable itself of a stripped version (use the 'xsyms' utility) to reduce memory usage and loading time on RTEMS systems. A basic check is made to protect against version mismatch between the symbol file and the executable. Once Cexp has loaded the system's symbol table, further instances will simply use the global system table if the -s argument is missing. cexp then reads commands from stdin (using the TECLA library) or alternatively from a script file. Cexp ignores any characters present on a line after it scans the comment tokens '#' or '//'. Example for invoking Cexp from Cexp to evaluate a script: Cexp> cexp("script_file") While the former syntax involves recursive execution of Cexp, it is now also possible to let the current instance divert its input to read from a script using the '<' operator: Cexp> < script_file There are slight syntactic differences when using this syntax, though. Evaluation of the file name is not performed by the parser but by a simple preprocessor. The filename doesn't have to be embedded in double quotes (but it can be). Note that no escape sequences are recognized. It is possible, however, to use filenames with embedded whitespace by enclosing the script name in single or double quotes. In this case, the entire string between matching quotes is used. E.g., (RHS string delimiter is '"' -- remember that there are no escape sequences...): < a b c --> read from file "a" < 'a b' "c" --> read from file "a b" < "a'b" --> read from file "a'b" < "a\"b" --> read from file "a\\" (a followed by a single backslash) Loadable Modules / Runtime Loader --------------------------------- When built aginst the BFD library, cexp is capable of dynamically loading object files into a running program. Cexp keeps track of module dependencies. Note that this only covers symbol table dependencies. Cexp rejects unloading a module 'A' if there is still another module, 'B' loaded which had undefined symbols resolved against 'A's symbols. Obviously, more subtle dependencies, such as threads using a modules text or data cannot be tracked easily and are ignored. An object file (AKA 'module') can be loaded invoking the command: Cexp> someModule=cexpModuleLoad("someModule.o") The loader returns a 'module ID', which in this example is stored in the user variable 'someModule'. If errors occur during the load (such as undefined references or multiple symbol definitions), they are reported and a NULL module ID is returned. After loading, C++ static constructors are executed and exception handler frames are registered. Note that these features are probably only supported on ELF and, especially the exception handling, might only work for gcc compiled code. Unfortunately, even different versions of gcc and/or target architectures involve varying implementations of exception handling - YMMV... An (unused) module can be unloaded by passing its ID to cexpModuleUnload() (prior to unloading, the C++ static destructors are executed): Cexp> cexpModuleUnload(someModule) Two more routines are useful in this context: cexpModuleInfo([ID]) prints info about a specific module (ID) or all currently loaded modules if passed NULL ID. cexpModuleFindByName("regexp") searches the list of modules for a regular expression and reports the IDs of all matches. It returns the first ID found (or NULL if there was no match). Search Path for Loadable Modules - - - - - - - - - - - - - - - - - If set, the PATH environment variable is consulted by the loader when it tries to locate object files by subsequently prepending colon-separated search paths as listed in PATH to the file name. The search starts with the first component and stops as soon as the file is found. An empty search path (two colons in a row or a leading or trailing colon) is equivalent to the current directory ('.'). Note that the current directory is not searched if PATH is set but does not contain the CWD. PATH is ignored if the object name contains a (relative or absolute) directory path already. Example: search '/tmp', '/TFTP/BOOTP_HOST/blah/bin' and finally the CWD: setenv("PATH","/tmp:/TFTP/BOOTP_HOST/blah/bin:",1) Building loadable modules - - - - - - - - - - - - - Note the important difference between a dymamically loaded object and a _shared_ object (such as a shared library). The latter, although also 'dynamically loaded' is different from the former. A shared object is shared among several entities using disjunct address spaces (e.g. different UNIX processes). Supporting shared objects involves PIC and GOTs. *********************************************************** CEXP does _not_ support shared libraries! An attempt to run a loaded module that was compiled with '-fpic', '-fPIC' and/or linked with the '-shared' options will crash hard. *********************************************************** To illustrate the difference, consider a LINUX system running two instances of the 'cexp' demo. 'cexp' is linked against glibc which is a shared library - both instances of 'cexp' use the _same_ copy of libc residing in physical memory. Let's now assume the both of the demo programs issue cexpModuleLoad("someObjectFile.o") Cexp's dynamic loader doesn't support shared objects - hence both instances of 'cexp' will end up with their own copy of 'someObjectFile' which will get loaded twice to physical memory. The main target of Cexp is RTEMS, a real-time OS which has a global address space shared by all threads. Hence, there is no need for shared object support but dynamically loading code is still very desirable. The Cexp dynamic loader simply accepts _any_ relocatable object file. The simplest modules are just object files: cross-gcc -c -O some_object.c Multiple objects can be combined using the '-r' linker option cross-ld -o some_object.o -r some1.o some2.o some3.o It is also possible to convert entire (static) libraries into loadable objects: cross-ld -o lib_object.o -r --whole-archive libSome.a The 'system' symbol table is itself an object file from where Cexp loads the initial symbol table at startup. The executable itself may be used for that purpose (unless it is a pure binary as it is the case on some embedded systems). Alternatively (for saving memory and time), a stripped-down object file containing only the symbol table may be generated using the 'xsyms' tool (available on ELF targets only), e.g: xsyms cexp cexp.sym cexp -s cexp.sym Loadable Modules 'Magic' - - - - - - - - - - - - - When loading modules, 'cexp' does some magic operations on special symbols it recognizes. Note that the all-capital names given here are macros - the actual names can be found in the header. CEXPMOD_INITIALIZER_SYM (defined in cexpmodP.h), when present in a loaded module, a routine with this name is invoked by cexpModuleLoad() just after calling C++ constructors. This 'module-constructor' routine [for C++ information see below] can be used to initialize a (non-C++) module. CEXPMOD_FINALIZER_SYM, when present in a loaded module, a routine with this name is invoked just prior to calling C++ destructors and unloading the module. The FINALIZER may reject the unloading attempt by returning a nonzero value. CEXP_HELP_TAB (defined in cexpHelp.h). A module may define (multiple) 'help' tables (their name must begin with the magic CEXP_HELP_TAB string) for providing help information about specific symbols contained in a module. See cexpHelp.h for more information about the help information. cexpModuleLoad() automatically registers this data. Building the 'Main' Application - - - - - - - - - - - - - - - - When linking a traditional application, only objects referenced by the application will be linked into the executable. E.g. an application which does not use 'printf()' will not have that routine available. In the normal case, this is fine. When using a runtime loader like CEXP, there arises a problem: imagine you want to load a piece of code which _does_ use 'printf()': - Cexp resolves the module's undefined symbols, encounters 'printf' but doesn't find it in its symbol table - it rejects loading your module. - It is not possible either to simply link your module against libc! The module might reference other libc objects which _are_ present in the application already - Cexp would complain that their symbols are already defined. Therefore, the 'primary' or 'system' application should be built to include ALL parts of the core libraries (such as the C-library, the RTEMS executive managers etc.) which will possibly be used by modules loaded into the running system. Hence, there is a new task for the system designer which (for a statically linked application) would be automatically performed by the linker: You must now tailor/configure the core parts of your system. This is essentially a memory/functionality tradeoff (Sidenote: vxWorks is configured in such a way, too). This 'tailoring' essentially means that you have to tell the linker what parts of the basic system libraries (RTEMS managers, CPU/BSP support, libc, networking etc.) should forcibly be included into the link. Note that it is only necessary to take into account the libraries needed by CEXP and the OS itself during this process. Any library CEXP does _not_ depend on, directly or indirectly, can of course be dynamically loaded any time later. A new tool 'ldep' is available greatly alleviating the task of analyzing link file interdependency and generating symbol lists etc. It is available as part of the 'GeSys' package (www.slac.stanford.edu/~strauman/rtems/gesys) and comes with more documentation. Some info is available here: http://www/~strauman/rtems/epics/README.config Using the Parser from a Program - - - - - - - - - - - - - - - - In some cases, you might want to access/use symbols in loaded modules from a program rather than through the cexp shell or a shell script. If the module providing the 'caller' is loaded after the module defining the 'callee' this is transparent. E.g., let a module 'A' define a function 'a()' and module 'B' define a routine 'b()' which shall call 'a()'. If module 'A' was loaded before 'B' then cexp's linker will resolve the reference and no special treatment is required. If, however, module 'B' must for some reason be loaded before 'A' then 'B' cannot simply call 'a()' since that would result in a linker error. Here's how you can load a module from a program and use symbols exported by that (or any other) module (no error checking shown): void (*p_a)(); /* declare function pointer to 'a()' */ cexpModuleLoad("A",0); /* use file name as module name; PATH is searched */ /* lookup the symbol */ p_a = cexpSymValue(cexpSymLookup("a",0)); /* call function */ if ( !p_a ) /* error; symbol not found */ else p_a(); Another method is using the interpreter (slower, of course). Since the interpreter is re-entrant, this involves dragging around context information: CexpParserCtx c = cexpCreateParserCtx(0); /* these two calls could/should be wrapped into 'cexpParseLine'... */ cexpResetParserCtx(c, "a()"); cexpparse(c); Note that more documentation about these calls can be found in 'cexp.h'. Note also that some of the cexp library calls require more arguments than apparent from the examples in other sections of this document where the shell is explained! Most routines are geared for shell use and declare the most important arguments first, followed by optional arguments and they use defaults when these optional arguments are NULL. The shell implicitely sets extra arguments to zero so that when you type e.g., 'cexpModuleLoad("file.o")' this effectively is expanded to 'cexpModuleLoad("file.o",NULL)' since cexpModuleLoad() expects two parameters. OTOH, if you code cexpModuleLoad("file.o") in a program then the compiler will warn you and the second argument will be undefined. Therefore, if you want to use library calls from a program you should consult 'cexp.h' for reference. Before you can use the Cexp library, it must be properly initialized: CEXP Liberary Initialization - - - - - - - - - - - - - - - When using the 'cexp_main()' AKA shell entry point, the library is initialized automatically (you might still want to use cexpInit() if you intend to use your own signal handler...). If you want to use cexp from a program as shown in the examples above, the library must be initialized before using it: cexpInit(0); /* initialize internal data; use default signal handling */ cexpModuleLoad(0,0); /* attach built-in symbol table. If you dont have a built- */ /* in table, you must pass a filename argument (see cexp.h)*/ C++ Information - - - - - - - - Dynamically loading C++ modules requires additional support. 1) static constructors and destructors 2) multiply defined instances of code (templates, multiple inclusion of headers) 3) exception handling Unfortunately, all of these items are highly implementation dependent and it is therefore unlikely that C++ support works for compilers other than gcc. Alas, there are even significant differences between different versions of gcc and target ABI's. IMHO, C++ is unreadable, unportable, bloat-prone and should be avoided if at all possible. That said, let's dive into the details: 1) static constructors and destructors take care of initializing/finalizing static objects such as: BlahClass blahObj(a,b,c); int test=testInitialize(x,y); gcc creates code for initializing/finalizing these kinds of objects and tags this code with symbols similar to '__GLOBAL__.I.xxx' / '__GLOBAL__.D.xxx'. If Cexp encounters such symbols, it builds constructor/destructor lists when loading the module and executes the respective code after loading / prior to unloading. Later versions of gcc use the '__cxa_atexit' callback mechanism which requires CEXP to call '__cxa_finalize()' when a module is unloaded. This seems to be a C++ ABI standard and is supported by Cexp. There are other possible implementations (such as creating special ctor/dtor sections) which are not supported - as I said, there is no general way for handling C++ :-( 2) C++ allows for multiple definitions/instantiations of code (a header included by more than one 'xxx.cc' file may define class members) plus there is the 'template' ''feature'' of C++. These let C++ code size _explode_ keeping hardware manufacturers happy. Gcc deals with this problem by putting every piece and bit of (possibly redundant code and/or data) into a separate '.gnu.linkonce.t.xyzkljsbjj783c' SECTION. The idea is that the linker (which is in our case 'Cexp') eliminates redundant sections. The sad result are bloated object files and Cexp symbol tables (Unfortunately, Cexp must keep _all_ of the zillions of '.gnu.linkonce.x.yzu' symbols around since a redundant copy might be loaded/linked at any time :-0 ) You can help CEXP, however by using proper linker scripts when linking large C++ applications (such as EPICS). If you are sure that your large application is the only (C++) object (sharing common code), you may provide the linker with a script which instructs it to integrate _all_ the "gnu.linkonce.t.xxx" sections into ".text" etc. If you don't believe me - just let CEXP load an EPICS IOC and list all the 'linkonce' symbols: lkup("linkonce") you will be surprised! 3) exception handling (while nice at the abstracion level) is another field that is highly architecture/compiler/version dependent and a seed to bloat. Even reentrancy might not be taken for granted. Gcc uses two flavors of exception handling 'longjmp' and 'eh_frame' style which are (to some extent) both supported by Cexp... As is said: C++ IS THE DEVIL in the detail. While linking C (and even fortran etc.) code compiled with different compilers (e.g. libraries) is seamless under a common ABI, it is very complex if not impossible with C++. Important PowerPC ABI Information --------------------------------- Cexp currently does not support the short data areas according to the SYSV/EABI specifications. Short data sections are merged by Cexp into the normal data segment and hence cannot be accessed via R13/R2. Hence, when compiling loadable modules, the compiler's -msdata flag must not be set to either of 'eabi', 'sysv' or 'use'. If you try to load code compiled with an improper -msdata setting, you will get a 'Relocation of unsupported size/type requested' error for the relocation type 'R_PPC_EMB_SDA21' or 'R_PPC_SDAREL16' or similar. LICENSING & DISCLAIMERS ----------------------- CEXP is released under the terms of the EPICS open license (consult the LICENSE file for details) Copyright 2002, Stanford University and Till Straumann Stanford Notice *************** Acknowledgement of sponsorship * * * * * * * * * * * * * * * * This software was produced by the Stanford Linear Accelerator Center, Stanford University, under Contract DE-AC03-76SFO0515 with the Department of Energy. Government disclaimer of liability - - - - - - - - - - - - - - - - - Neither the United States nor the United States Department of Energy, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any data, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Stanford disclaimer of liability - - - - - - - - - - - - - - - - - Stanford University makes no representations or warranties, express or implied, nor assumes any liability for the use of this software. This product is subject to the EPICS open license - - - - - - - - - - - - - - - - - - - - - - - - - Consult the LICENSE file or http://www.aps.anl.gov/epics/license/open.php for more information. Maintenance of notice - - - - - - - - - - - In the interest of clarity regarding the origin and status of this software, Stanford University requests that any recipient of it maintain this notice affixed to any distribution by the recipient that contains a copy or derivative of this software.