<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V3.1//EN" [
<!-- Begin Document Specific Declarations -->
<?Fm: Validation Off>
<!ENTITY version "1.4.3">
<!ENTITY dj "DejaGnu">
<!ENTITY dejagnu-copyright "
<YEAR>2002</YEAR>
<HOLDER>Free Software Foundation, Inc.</HOLDER>">
<!ENTITY dejagnu-code-copyright "
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
This file documents the GNU Testing Framework ``DejaGnu''
Copyright (C) 92 - 2001, 2002 Free Software
Foundation, Inc.
This text may be freely distributed under the terms of the GNU
General Public License.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
">
<!ENTITY dejagnu-copyright "
Copyright 92 - 2001, 2002 Free Software
Foundation, Inc.
Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.
Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.
Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions.
">
<!-- The reference material -->
<!entity ref SYSTEM "ref.sgml">
<!-- The user manual -->
<!entity user SYSTEM "user.sgml">
<!-- End Document Specific Declarations -->
]>
<book>
<bookinfo>
<title>&dj;</title>
<subtitle>The GNU Testing Framework</subtitle>
<date>2002 Sept 04</date>
<edition> &version</edition>
<releaseinfo> New release</releaseinfo>
<authorgroup>
<author>
<firstname>Rob Savoye</firstname>
<affiliation>
<orgname>Free Software Foundation</orgname></affiliation>
<!-- <authorblurb>
<title>Rob Savoye</title>
<para>
His home page is at <ulink>
URL="http://www.welcomehome.org/rob.html">this
location</ulink>
</para>
</authorblurb>
-->
</author>
</authorgroup>
<address>
<email>rob@welcomehome.org</email>
</address>
<!-- &cygnus-street-address; -->
<copyright>&dejagnu-copyright;</copyright>
<!-- <legalnotice>
<para> -->
<!-- [FIXME: must put legal notice here] -->
<!-- </para> -->
<!-- &cygnus-legal-notice; -->
<!-- </legalnotice> -->
<revhistory>
<revision>
<revnumber>0.6.2</revnumber>
<date>2002-7-16</date>
<authorinitials>rob@welcomehome.org</authorinitials>
<revremark>Add new tutorial as a new chapter.</revremark>
</revision>
<revision>
<revnumber>0.6.1</revnumber>
<date>2001-2-16</date>
<authorinitials>rob@welcomehome.org</authorinitials>
<revremark>Add info on the new dejagnu.h file.</revremark>
</revision>
<revision>
<revnumber>0.6</revnumber>
<date>2001-2-16</date>
<authorinitials>rob@welcomehome.org</authorinitials>
<revremark>Updated for new release.</revremark>
</revision>
<revision>
<revnumber>0.5</revnumber>
<date>2000-1-24</date>
<authorinitials>rob@welcomehome.org</authorinitials>
<revremark>Initial version after conversion to DocBook.</revremark>
</revision>
</revhistory>
</bookinfo>
<toc></toc>
<preface id=preface>
<title>Abstract</title>
<para>This document attempts to describe the functionality of
DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in
<productname>Expect</productname>, which uses
<productname>Tcl</productname> as a command
language. <productname>Expect</productname> serves as a very
programmable shell; you can run any program, as with the usual
Unix command shells---but once the program is started, your
test script has fully programmable control of
its input and output. This does not just apply to the programs
under test; <command>expect</command> can also run any auxiliary
program, such as <command>diff</command> or <command>sh</command>,
with full control over its input and output.</para>
<para>DejaGnu itself is merely a framework for creation of a test
suites. Test suites are distributed separately for each GNU
tool.</para>
</preface>
<chapter id=overview xreflabel=Overview>
<title>Overview</title>
<sect1 id=whatis xreflabel="What is &dj; ?">
<title>What is &dj; ?</title>
<para><productname>DejaGnu</productname> is a framework for
testing other programs. Its purpose is to provide a single
front end for all tests. Think of it as a custom library of
Tcl procedures crafted to support writing a test harness. A
<emphasis>Test Harness</emphasis> is the testing
infrastructure that is created to support a specific program
or tool. Each program can have multiple test suites, all
supported by a single test harness. DejaGnu is written in
<productname>Expect</productname>, which in turn uses
<productname>Tcl</productname> -- Tool command
language. There is more information on Tcl at the <ulink
URL="http://www.scriptics.com">Scriptics</ulink> web site, and the
Expect web site is at <ulink
URL="http://expect.nist.gov">NIST</ulink>.</para>
<para>DejaGnu offers several advantages for testing:</para>
<itemizedlist mark="bullet" spacing="compact">
<listitem><para>The flexibility and consistency of the DejaGnu
framework make it easy to write tests for any program, with
either batch oriented, or interactive programs.</para>
</listitem>
<listitem><para>DejaGnu provides a layer of abstraction which
allows you to write tests that are portable to any host or
target where a program must be tested. For instance, a test
for <command>GDB</command> can run (from any Unix
based host) on any target architecture that DejaGnu
supports. Currently DejaGnu runs tests on many single board
computers, whose operating software ranges from just a boot
monitor to a full-fledged, Unix-like realtime OS.</para>
</listitem>
<listitem><para>All tests have the same output format. This
makes it easy to integrate testing into other software
development processes. DejaGnu's output is designed to be
parsed by other filtering script, and it is also human
readable.</para>
</listitem>
<listitem><para>Using Tcl and expect, it's easy to create wrappers
for existing test suites. By incorporating existing tests under
DejaGnu, it's easier to have a single set of report analyse
programs..</para>
</listitem>
</itemizedlist>
<para>Running tests requires two things: the testing framework, and
the test suites themselves. Tests are usually written in
<productname>Expect</productname> using Tcl, but you can also use a
Tcl script to run a test suite that is not based on
<productname>Expect</productname>.
(<productname>expect</productname> script filenames conventionally
use <emphasis>.exp</emphasis> as a suffix; for example, the main
implementation of the DejaGnu test driver is in the file
<productname>runtest.exp</productname>.)</para>
<para>Julia Menapace first coined the term ``Deja Gnu'' to describe an
earlier testing framework at Cygnus Support she had written for
<command>GDB</command>. When we replaced it with the Expect-based
framework, it was like DejaGnu all over again... But more importantly, it
was also named after my daughter,<ulink
URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink> (now 12
years old in Sept of 2002), who was a toddler during DejaGnu's
creation.</para>
</sect1>
<sect1 id=new xreflabel="Release Notes">
<title>What's New In This Release</title>
<para>This release has a number of substantial changes over version
1.3. The most visible change is that the version of Expect and Tcl
included in the release are up-to-date with the current stable net
releases. The biggest change is years of modifications to the
target configuration system, used for cross testing. While this
greatly improved cross testing, is has made that subsystem very
complicated. The goal is to have this entirely rewritten using
<productname>iTcl</productname> by the next release. Other changes
are:</para>
<itemizedlist>
<listitem><para>More builtin support for building target binaries
with the correct linker flags. Currently this only works with
<productname>GCC</productname> as the cross compiler,
preferably with a target supported by
<xref linkend=libgloss>.</para></listitem>
<listitem><para>Lots of little bug fixes from years of heavy
use at Cygnus Solutions.</para></listitem>
<listitem><para>DejaGnu now uses
<productname>Automake</productname> for Makefile
configuration.</para></listitem>
<listitem><para>Updated documentation, now in SGML
(using the <ulink
URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">free
GNU DocBook tools</ulink>) format.</para></listitem>
<listitem><para>NT support. There is beta level support for NT
that is still a work in progress. This requires the <ulink
URL="http://sources.redhat.com">Cygwin</ulink> POSIX system
for NT.</para></listitem>
</itemizedlist>
<sect2 id=cygwin xreflabel="NT Support">
<title>NT Support</title>
<para>To use DejaGnu on NT, you need to first install the
<ulink URL="http://sources.redhat.com/cygwin">Cygwin</ulink>
release. This works as of the B20.1 release. Cygwin is a POSIX
system for NT. This covers both utility programs, and a libray
that adds POSIX system calls to NT. Among them is pseudo tty
support for NT that emulates the POSIX pty standard. The
latest Cygwin is always available from <ulink
URL="http://sources.redhat.com/cygwin">this location</ulink>. This
works well enough to run <emphasis>"make check"</emphasis> of
the GNU development tree on NT after a native build. But the
nature of pty's on NT is still evolving. Your mileage may
vary...</para>
</sect2>
</sect1>
<sect1 id=designgoals xreflabel="Design Goals">
<title>Design Goals</title>
<para>DejaGnu grew out of the internal needs of Cygnus Solutions. (then
Cygnus Support). Cygnus maintained and enhanced a variety of free
programs in many different environments, and we needed a testing
tool that:</para>
<itemizedlist mark="bullet">
<listitem><para>was useful to developers while fixing
bugs.</para></listitem>
<listitem><para>automated running many tests during a software
release process.</para></listitem>
<listitem><para>was portable among a variety of host
computers.</para></listitem>
<listitem><para>supported cross-development
testing.</para></listitem>
<listitem><para>permitted testing interactive programs, like
<command>GDB</command>; and </para></listitem>
<listitem><para>permitted testing batch oriented programs, like
<command>GCC</command>.</para></listitem>
</itemizedlist>
<para>Some of the requirements proved challenging. For example,
interactive programs do not lend themselves very well to automated testing.
But all the requirements are important: for instance, it is imperative to
make sure that <command>GDB</command> works as well when cross-debugging
as it does in a native configuration. </para>
<para>Probably the greatest challenge was testing in a cross-development
environment (which can be a real nightmare). Most cross-development
environments are customized by each developer. Even when buying packaged
boards from vendors there are many differences. The communication
interfaces vary from a serial line to ethernet. DejaGnu was designed with
a modular communication setup, so that each kind of communication can be
added as required, and supported thereafter. Once a communication
procedure is coded, any test can use it. Currently DejaGnu can use
<command>rsh</command>, <command>rlogin</command>,
<command>telnet</command>, <command>tip</command>,
<command>kermit</command>, and <command>mondfe</command> for remote
communications.</para>
</sect1>
<sect1 id=posix xreflabel="A POSIX Conforming Test Framework">
<title>A POSIX conforming test framework</title>
<para>DejaGnu conforms to the POSIX 1003.3 standard for test
frameworks. I was also a member of that committe.</para>
<para>The POSIX standard 1003.3 defines what a testing framework needs to
provide, in order to permit the creation of POSIX conformance test
suites. This standard is primarily oriented to running POSIX conformance
tests, but its requirements also support testing of features not related
to POSIX conformance. POSIX 1003.3 does not specify a particular testing
framework, but at this time there is only one other POSIX conforming test
framework: TET. TET was created by Unisoft for a consortium comprised of
X/Open, Unix International, and the Open Software Foundation.</para>
<para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
An assertion is a description of behavior. For example, if a standard
says ``The sun shall shine'', a corresponding assertion might be ``The
sun is shining.'' A test based on this assertion would pass or fail
depending on whether it is daytime or nighttime. It is important to note
that the standard being tested is never 1003.3; the standard being tested
is some other standard, for which the assertions were written.</para>
<para>As there is no test suite to test testing frameworks for POSIX
1003.3 conformance, verifying conformance to this standard is done by
repeatedly reading the standard and experimenting. One of the main
things 1003.3 does specify is the set of allowed output messages, and
their definitions. Four messages are supported for a required feature of
POSIX conforming systems, and a fifth for a conditional feature. DejaGnu
supports the use of all five output messages; in this sense a test suite
that uses exactly these messages can be considered POSIX conforming.
These definitions specify the output of a test
case:</para>
<variablelist>
<varlistentry>
<term>PASS</term>
<listitem><para>A test has succeeded. That is, it demonstrated that
the assertion is true.</para></listitem>
</varlistentry>
<varlistentry>
<term>XFAIL</term>
<listitem><para>POSIX 1003.3 does not incorporate the notion of
expected failures, so <emphasis>PASS</emphasis>, instead of
<emphasis>XPASS</emphasis>, must also be returned for test cases
which were expected to fail and did not. This means that
<emphasis>PASS</emphasis> is in some sense more ambiguous than if
<emphasis>XPASS</emphasis> is also used.</para></listitem>
</varlistentry>
<varlistentry>
<term>FAIL</term>
<listitem><para>A test has produced the bug it was intended to
capture. That is, it has demonstrated that the assertion is false.
The <emphasis>FAIL</emphasis> message is based on the test case only.
Other messages are used to indicate a failure of the framework. As
with <emphasis>PASS</emphasis>, POSIX tests must return
<emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
if a failure was expected.</para></listitem>
</varlistentry>
<varlistentry>
<term>UNRESOLVED</term>
<listitem><para>A test produced indeterminate results. Usually, this
means the test executed in an unexpected fashion; this outcome
requires that a human being go over results, to determine if the test
should have passed or failed. This message is also used for any test
that requires human intervention because it is beyond the abilities
of the testing framework. Any unresolved test should resolved to
<emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
run can be considered finished.</para>
<para>Note that for POSIX, each assertion must produce a test result
code. If the test isn't actually run, it must produce
<emphasis>UNRESOLVED</emphasis> rather than just leaving that test
out of the output. This means that you have to be careful when
writing tests, to not carelessly use tcl statements like
<emphasis>return</emphasis>---if you alter the flow of control of the
tcl code you must insure that every test still produces some result
code.</para>
<para>Here are some of the ways a test may wind up
<emphasis>UNRESOLVED</emphasis>:</para></listitem>
</varlistentry>
</variablelist>
<itemizedlist mark=bullet>
<listitem><para>A test's execution is
interrupted.</para></listitem>
<listitem><para>A test does not produce a clear
result. This is usually because there was an
<emphasis>ERROR</emphasis> from DejaGnu while processing
the test, or because there were three or more
<emphasis>WARNING</emphasis> messages. Any
<emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
messages can invalidate the output of the test. This
usually requires a human being to examine the output to
determine what really happened---and to improve the test
case.</para></listitem>
<listitem><para>A test depends on a previous test, which
fails.</para></listitem>
<listitem><para>The test was set up
incorrectly.</para></listitem>
</itemizedlist>
<variablelist>
<varlistentry>
<term>UNTESTED</term>
<listitem><para>A test was not run. This is a placeholder, used
when there is no real test case yet.</para></listitem>
</varlistentry>
</variablelist>
<para>The only remaining output message left is intended to test
features that are specified by the applicable POSIX standard as
conditional:</para>
<variablelist>
<varlistentry>
<term>UNSUPPORTED</term>
<listitem><para>There is no support for the tested case. This may
mean that a conditional feature of an operating system, or of a
compiler, is not implemented. DejaGnu also uses this message when
a testing environment (often a ``bare board'' target) lacks basic
support for compiling or running the test case. For example, a
test for the system subroutine <emphasis>gethostname</emphasis>
would never work on a target board running only a boot
monitor.</para></listitem>
</varlistentry>
</variablelist>
<para>DejaGnu uses the same output procedures to produce these messages
for all test suites, and these procedures are already known to conform
to POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3,
you must avoid the <emphasis>setup</emphasis>xfail} procedure as
described in the <emphasis>PASS</emphasis> section above, and you must
be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
as described in the <emphasis>UNRESOLVED</emphasis> section
above.</para>
</sect1>
</chapter>
<!-- include the user manual -->
&user;
<!-- include the reference manual -->
&ref;
</book>
<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-omittag:t
sgml-shorttag:t
sgml-namecase-general:t
sgml-general-insert-case:lower
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:nil
sgml-parent-document:nil
sgml-exposed-tags:nil
sgml-local-catalogs:nil
sgml-local-ecat-files:nil
End:
-->