2002 Free Software Foundation, Inc."> ]> The GNU Testing Framework 2002 Sept 04 &version New release Rob Savoye Free Software Foundation

rob@welcomehome.org

&dejagnu-copyright; 0.6.2 2002-7-16 rob@welcomehome.org Add new tutorial as a new chapter. 0.6.1 2001-2-16 rob@welcomehome.org Add info on the new dejagnu.h file. 0.6 2001-2-16 rob@welcomehome.org Updated for new release. 0.5 2000-1-24 rob@welcomehome.org Initial version after conversion to DocBook. This document attempts to describe the functionality of DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in Expect, which uses Tcl as a command language. Expect serves as a very programmable shell; you can run any program, as with the usual Unix command shells---but once the program is started, your test script has fully programmable control of its input and output. This does not just apply to the programs under test; expect can also run any auxiliary program, such as diff or sh, with full control over its input and output. DejaGnu itself is merely a framework for creation of a test suites. Test suites are distributed separately for each GNU tool. DejaGnu is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A Test Harness is the testing infrastructure that is created to support a specific program or tool. Each program can have multiple test suites, all supported by a single test harness. DejaGnu is written in Expect, which in turn uses Tcl -- Tool command language. There is more information on Tcl at the Scriptics web site, and the Expect web site is at NIST. DejaGnu offers several advantages for testing: The flexibility and consistency of the DejaGnu framework make it easy to write tests for any program, with either batch oriented, or interactive programs. DejaGnu provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test for GDB can run (from any Unix based host) on any target architecture that DejaGnu supports. Currently DejaGnu runs tests on many single board computers, whose operating software ranges from just a boot monitor to a full-fledged, Unix-like realtime OS. All tests have the same output format. This makes it easy to integrate testing into other software development processes. DejaGnu's output is designed to be parsed by other filtering script, and it is also human readable. Using Tcl and expect, it's easy to create wrappers for existing test suites. By incorporating existing tests under DejaGnu, it's easier to have a single set of report analyse programs.. Running tests requires two things: the testing framework, and the test suites themselves. Tests are usually written in Expect using Tcl, but you can also use a Tcl script to run a test suite that is not based on Expect. (expect script filenames conventionally use .exp as a suffix; for example, the main implementation of the DejaGnu test driver is in the file runtest.exp.) Julia Menapace first coined the term ``Deja Gnu'' to describe an earlier testing framework at Cygnus Support she had written for GDB. When we replaced it with the Expect-based framework, it was like DejaGnu all over again... But more importantly, it was also named after my daughter,Deja Snow Savoye (now 12 years old in Sept of 2002), who was a toddler during DejaGnu's creation. This release has a number of substantial changes over version 1.3. The most visible change is that the version of Expect and Tcl included in the release are up-to-date with the current stable net releases. The biggest change is years of modifications to the target configuration system, used for cross testing. While this greatly improved cross testing, is has made that subsystem very complicated. The goal is to have this entirely rewritten using iTcl by the next release. Other changes are: More builtin support for building target binaries with the correct linker flags. Currently this only works with GCC as the cross compiler, preferably with a target supported by . Lots of little bug fixes from years of heavy use at Cygnus Solutions. DejaGnu now uses Automake for Makefile configuration. Updated documentation, now in SGML (using the free GNU DocBook tools) format. NT support. There is beta level support for NT that is still a work in progress. This requires the Cygwin POSIX system for NT. To use DejaGnu on NT, you need to first install the Cygwin release. This works as of the B20.1 release. Cygwin is a POSIX system for NT. This covers both utility programs, and a libray that adds POSIX system calls to NT. Among them is pseudo tty support for NT that emulates the POSIX pty standard. The latest Cygwin is always available from this location. This works well enough to run "make check" of the GNU development tree on NT after a native build. But the nature of pty's on NT is still evolving. Your mileage may vary... DejaGnu grew out of the internal needs of Cygnus Solutions. (then Cygnus Support). Cygnus maintained and enhanced a variety of free programs in many different environments, and we needed a testing tool that: was useful to developers while fixing bugs. automated running many tests during a software release process. was portable among a variety of host computers. supported cross-development testing. permitted testing interactive programs, like GDB; and permitted testing batch oriented programs, like GCC. Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing. But all the requirements are important: for instance, it is imperative to make sure that GDB works as well when cross-debugging as it does in a native configuration. Probably the greatest challenge was testing in a cross-development environment (which can be a real nightmare). Most cross-development environments are customized by each developer. Even when buying packaged boards from vendors there are many differences. The communication interfaces vary from a serial line to ethernet. DejaGnu was designed with a modular communication setup, so that each kind of communication can be added as required, and supported thereafter. Once a communication procedure is coded, any test can use it. Currently DejaGnu can use rsh, rlogin, telnet, tip, kermit, and mondfe for remote communications. DejaGnu conforms to the POSIX 1003.3 standard for test frameworks. I was also a member of that committe. The POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET. TET was created by Unisoft for a consortium comprised of X/Open, Unix International, and the Open Software Foundation. The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says ``The sun shall shine'', a corresponding assertion might be ``The sun is shining.'' A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written. As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. DejaGnu supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case: PASS A test has succeeded. That is, it demonstrated that the assertion is true. XFAIL POSIX 1003.3 does not incorporate the notion of expected failures, so PASS, instead of XPASS, must also be returned for test cases which were expected to fail and did not. This means that PASS is in some sense more ambiguous than if XPASS is also used. FAIL A test has produced the bug it was intended to capture. That is, it has demonstrated that the assertion is false. The FAIL message is based on the test case only. Other messages are used to indicate a failure of the framework. As with PASS, POSIX tests must return FAIL rather than XFAIL even if a failure was expected. UNRESOLVED A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires that a human being go over results, to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to PASS or FAIL before a test run can be considered finished. Note that for POSIX, each assertion must produce a test result code. If the test isn't actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. This means that you have to be careful when writing tests, to not carelessly use tcl statements like return---if you alter the flow of control of the tcl code you must insure that every test still produces some result code. Here are some of the ways a test may wind up UNRESOLVED: A test's execution is interrupted. A test does not produce a clear result. This is usually because there was an ERROR from DejaGnu while processing the test, or because there were three or more WARNING messages. Any WARNING or ERROR messages can invalidate the output of the test. This usually requires a human being to examine the output to determine what really happened---and to improve the test case. A test depends on a previous test, which fails. The test was set up incorrectly. UNTESTED A test was not run. This is a placeholder, used when there is no real test case yet. The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional: UNSUPPORTED There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented. DejaGnu also uses this message when a testing environment (often a ``bare board'' target) lacks basic support for compiling or running the test case. For example, a test for the system subroutine gethostname would never work on a target board running only a boot monitor. DejaGnu uses the same output procedures to produce these messages for all test suites, and these procedures are already known to conform to POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, you must avoid the setupxfail} procedure as described in the PASS section above, and you must be careful to return UNRESOLVED where appropriate, as described in the UNRESOLVED section above. &user; &ref;