Skip to main content.

DUTS Documentation

Introduction

This document describes the basics of DUTS, the DENX Universal Test System, a framework for testing U-Boot and Linux kernel in the embedded environment.

The test system (also called the framework) is geared towards command line interface testing: it is driven by console input/output, and has a form of composition of custom scripts written in expect/Tcl.

Basic concepts

The test framework is designed around a set of entities, described below. In general the reader is assumed proficient with U-Boot and Linux kernel development and use, as well as embedded environments, the ELDK in particular.

Virtual laboratory (VL)

VL is the basic environment that groups:

By default, the installation supports the DENX laboratory setup, operating within its specific layout and using own tools to control target devices. In order to use DUTS with a different laboratory, the appropriate methods for low-level operations on devices need to be implemented. As an example, a VL-less configuration is also provided - see 'Introducing suppport for a new VL' in the Advanced Topics section for details.

Testsystem

A container grouping tests that logically fall into a particular topic or area, and which should be kept and managed together. A testsystem typically comprises a number of related test cases. At a particular test run, DUTS is aware of the currently selected testsystem and only test cases grouped within this testsystem are available for execution.

Test case (TC)

A sequential execution flow of commands issued at host, firmware or kernel prompt. Test case is a central entity within DUTS framework and has the following characteristics:

Test cases can be simple and complex. Simple TCs are defined in a standard way by a series of commands and their corresponding outputs. Non-standard (custom user-developed) test cases are external scripts written in expect and can perform advanced operations when the simple approach is not sufficient.

Note that test cases are not proper (stand-alone) executable scripts, and are meant to be invoked from within DUTS framework only.

Device (target board)

Entity representing a physical target device. Every board supported by DUTS is required to have a description used by the framework to properly operate the device. The key characteristics of the device abstraction are the following:

Using DUTS

From the end user perspective there is one central point for all DUTS operations, a highest level script tool 'duts', which provides a user interface for accessing the framework. Some of the most important actions it allows to perform cover the following:

To support DUTS usage the following resources are available besides this manual: on-line help summary is built-in the tool, 'docs' directory contains examples with rich comments on low-level details (test case construction, configuration files etc.)

[duts]$ ./duts 
Need to specify a board to test

usage: ./duts [options] <board>

  runs selected testcase on <board>. Specifying ? lists available choices.

  possible [options] are listed below, defaults are in <>.

duts options:
 -c value             use supplied configuration, ? lists choices <>
 -maxcost value       exclude tests more expensive than argument <>
 -mincost value       exclude tests less expensive than argument <>
 -td value            testsystem directory (defaults to dulg) <>
 -date value          use supplied date <>
 -ts value            narrow to supplied list of test sets <>
 -tc value            use supplied list of test cases, ? lists choices <>
 -var value           override supplied variables, ? lists defined vars <>
 -continue            continue previously interrupted run
 -n                   dry-run
 -showtc              show details about selected testcases
 -showconfig          show details about selected configuration
 -v                   be verbose
 -help                Print this message
 -?                   Print this message


For more information and examples see http://www.denx.de/wiki/DUTS/DUTSDocs.

Directory layout

The DUTS framework directory contents are briefly described below:

config
devices
docs
duts
include
testsystems
tools

A particularly important aspect of managing potentially many testsystems under DUTS control is a working directory. This concept specifies which testsystem is actually selected for the current DUTS execution run - all DUTS commands/operations (like test case execution etc.) are executed within the scope of a given testsystem. The framework is aware only about test cases grouped within one particular testsystem currently selected, and is not able to perform operations on test cases from different testsystems in a single run.

The working directory is specified with '-d' option, or by setting DUTS_DIR environment variable at the host that DUTS is installed. By default (no specification) a DULG testsystem is selected.

Mechanisms overview

Once the 'duts' tool is executed, it takes the following main steps:

For each test case:

Devices management

Devices description files

In order for a target device to be recognised and supported by DUTS, a description entry is required for the device. Per convention device descriptions are kept in .tgt files in the 'devices' DUTS directory.

Please refer to docs/example.tgt for additional explanation of individual fields, and to devices/*.tgt files for working examples of supported target boards.

Of particular interest from the user perspective is the 'Vars' section in the device description structure: it allows to assign certain values to variables for a given device, which then can be referenced by name in test cases' bodies.

Displaying the list of supported target devices

[duts]$ ./duts  -showconfig ?
Defined configurations for the devices:
  bamboo
  bubinga
  luan
  ebony
  ocotea
  sycamore
  walnut
  yellowstone
  yosemite
  yucca
  mpc8349emds
  lite5200b
  v38b
  tqm8272
  tqm834x
  tqm8540
  tqm8541
  tqm8555
  tqm8560

Detailed info on selected target board

[duts]$ ./duts -showconfig v38b
Configuration for board: v38b
  vars set: CFG_RAM_WS_BASE CFG_ROOTPATH CFG_KERNEL_START CFG_KERNEL_END
CFG_RAMDISK_START CFG_RAMDISK_END CFG_MTD_ERASE_DEVICE CFG_FLASH_PROT1_START
CFG_FLASH_PROT1_END CFG_FLASH_TST_START CFG_FLASH_TST_END
CFG_FLASH_ERASE_START CFG_FLASH_ERASE_END CFG_FLASH_TST_SIZE
CFG_FLASH_ERASE_SECT CFG_FLASH_ERASE_BANK CFG_FLASH_TST_SECT

  make params: target 'v38b' arch 'ppc' ccompile '/opt/eldk-4.1-2007-01-19/usr/bin/'

Test cases organisation and management

Test case class (context)

As mentioned earlier, among test case attributes is its context i.e. a state of the environment that the given test case expects to find. For test cases supposed to run on the target device the context can be:

Internally, DUTS state machine detects current context and switches to the desired one, so the test case that is scheduled to run next is fully set to go.

Test cases designed to run on the VL host machine have only 'host' context and do not require switching.

Test case description files

Test cases realisation has the form of text files that are kept in testsystems//testcases directory and by naming convention must end with a '.tc'.

DUTS processes the .tc files in alpha order of their name, which allows for elementary control over the processing sequence.

While simple test cases' bodies are wholly buried within the .tc file, for custom test cases a description in the .tc file only points to an external script that does the user-specified job. In the latter case the assumed convention is that these external scripts have user*.exp naming pattern.

For further advise and elaborate comments on the .tc file structure and detailed description of all fields of the duts_tc structure please refer to docs/example.tc.

Displaying the list of defined testcases

List test cases in the default testsystem:

[duts]$ ./duts -tc ? v38b
Testcases directory: ./testsystems/dulg
List of testcases currently defined:
 
  InstallKernelTftp, u-boot
  UBootInstallRamdisk, u-boot
  LinuxMtdDump, linux
  LinuxMtdErase, linux
  LinuxMtdWrite, linux
  LinuxProcMtd, linux
  LinuxBootNfs, u-boot
  ...
  ...
  UBootDocHelp, u-boot
  UBootDocbootHelp, u-boot

List test cases in the explicitly specified testsystem:

[duts]$ ./duts -td testsystems/ltp -tc ? v38b | grep linux
Testcases directory: testsystems/ltp/
List of testcases currently defined:
 
  LTPRunAllTests, linux

Format of the above list is very simple and for each test case provides the following info:

  <test case name>, <context label>

Since all DUTS interfaces are organised around textual representation, it is therefore very easy (and typical of UNIX) to freely manipulate output for user purposes. For example, to get a list of DULG test cases related to Linux somehow (either by the name or context) one typically issues:

[duts]$./duts  -tc ? v38b | grep inux
  LinuxMtdDump, linux
  LinuxMtdErase, linux
  LinuxMtdWrite, linux
  LinuxProcMtd, linux
  LinuxBootNfs, u-boot
  LinuxBootNetNfs, u-boot
  LinuxBootSelf, u-boot
  LinuxConfig, host

Detailed info on selected test case

Simple test case details

[duts]$ ./duts  -tc UBootCpFlash v38b
Testcases directory: ./testsystems/dulg
Testcase 'UBootCpFlash' details:
  Type:         u-boot
  Pre:
                "prot off $CFG_FLASH_TST_START $CFG_FLASH_TST_END" ".*"
                "era $CFG_FLASH_TST_START $CFG_FLASH_TST_END" ".*"

  Commands:
                "cp.b $CFG_RAM_WS_BASE $CFG_FLASH_TST_START $CFG_FLASH_TST_SIZE" ".*"

  Filename:     ./testsystems/dulg/testcases/03_UBootCmdGroupFlash.tc

User-defined (custom) test case details

[duts]$ ./duts  -tc LinuxBootNetNfs v38b
Testcases directory: ./testsystems/dulg
Testcase 'LinuxBootNetNfs' details:
  Type:         u-boot
  Commands:     !userLinuxBootNetNfs.exp
  Filename:     ./testsystems/dulg/testcases/02_non-59.tc

Notes on the above examples:

Executing test cases

Run all available test cases in the testsystem:

[duts]$ ./duts -c _default v38b
Board name: v38b
Testcases directory: ./testsystems/dulg
Selected config: _default
List of selected test cases:
InstallKernelTftp UBootInstallRamdisk LinuxMtdDump LinuxMtdErase LinuxMtdWrite
LinuxProcMtd LinuxBootNfs LinuxBootNetNfs LinuxBootSelf UBootBoot UBootHelp
UBootHelpAbbrev UBootHelpLong UBootSetSerial UBootPrintenvBoardID BoardId
BDI2000Config LinuxConfig UBootConfig UBootCpHelp UBootEraseHelp
UBootProtectHelp UBootCpFlash UBootCpFlashErrors UBootEraseStartEnd
UBootEraseSectors UBootEraseBank UBootEraseAll UBootProtect UBootBdinfoHelp
UBootBdinfo UBootConinfoHelp UBootConinfo UBootFlinfoHelp UBootFlinfo
UBootIminfoHelp UBootIminfo UBootHelpHelp UBootProtectHelp UBootBootpHelp
UBootDhcpHelp UBootLoadbHelp UBootLoadsHelp UBootRarpHelp UBootTftpHelp
UBootPrintenvHelp UBootPrintenvWithArgs UBootPrintenvNoArgs UBootSaveenvHelp
UBootSaveenv UBootSetenvHelp UBootSetenvDelete UBootSetenvSet UBootSetenvQuote
UBootRunHelp UBootRun UBootRunSequence UBootBootdHelp UBootAutoscrHelp
UBootAutoscrScript UBootAutoscrMkimage UBootAutoscr UBootBootmHelp UBootGoHelp
UBootBaseHelp UBootBase UBootCrc UBootCrcStore UBootCmpHelp UBootCmp
UBootCmpExt UBootCpHelp UBootCp UBootCpExt UBootMdHelp UBootMd UBootMdExt
UBootMdLength UBootMmHelp UBootMm UBootMmW UBootMmB UBootMtestHelp UBootMtest
UBootMwHelp UBootMw UBootMwExt UBootNmHelp UBootNm UBootLoopHelp UBootDateHelp
UBootDate UBootEchoHelp UBootEcho UBootResetHelp UBootReset UBootSleepHelp
UBootSleep UBootVersionHelp UBootVersion UBootI2cHelp UBootIdeHelp
UBootDiskbootHelp UBootDocHelp UBootDocbootHelp

confirm to start execution? [y] 

Selected sub-set of test cases

[duts]$  ./duts -c _default -tc UBootIminfoHelp -tc UBootMtestHelp v38b
UBootInstallRamdisk
Board name: v38b
Testcases directory: ./testsystems/dulg
Selected config: _default
List of selected test cases:
UBootIminfoHelp UBootMtestHelp UBootInstallRamdisk

confirm to start execution? [y] 

All test cases in the specified .tc file

[duts]$ ./duts t v38b 03_UBootCmdGroupFlash.tc
Board name: v38b
Selected config: _default
List of selected test cases:
UBootCpHelp UBootEraseHelp UBootProtectHelp UBootCpFlash UBootCpFlashErrors
UBootEraseStartEnd UBootEraseSectors UBootEraseBank UBootEraseAll UBootProtect

confirm to start execution? [y] 

Similar to the above, but with absolute path to the .tc file

[duts]$ ./duts t v38b testsystems/ltp/testcases/ltp.tc 
Board name: v38b
Selected config: _default
List of selected test cases:
LTPRunAllTests

confirm to start execution? [y]

Logging test case output

The flow of each test case (inputs sent and outputs received during its execution) is captured in a log file, by default named after the test case's name (this can be changed by a Logfile section in the test case description structure): each test case is thus accompanied by a log file with its execution record.

A summary with results of all test cases executed in the given DUTS run is put into {$boardname}_results.log.

Additional examples

Build Linux kernel and run LTP against it

1. List test cases in the 'platform' testsystem

Build test cases are grouped under 'platform':

[duts]$ ./duts -td testsystems/ltp -tc ? v38b
Testcases directory: testsystems/platform/
List of testcases currently defined:

  Kernel24Build, host
  Kernel26Build, host
  KernelBuild, host

Kernel24Build and Kernel26Build test cases are deprecated and will be removed from DUTS in the future. KernelBuild is the recommended testcase for building Linux kernel, both 2.4 and 2.6. In case of 2.6 kernel it also supports building into a separate build directory, thus elminating the need for a writable source directory.

2a. Build 2.4 kernel

To build a 2.4 kernel, a writable directory with source tree is needed. It is expected to be called linuxppc_2_4_devel and be present in the directory where duts is executed from. The following invocation of duts will:

[duts]$ ./duts -d testsystems/platform/ t v38b -t KernelBuild
Board name: v38b
Testcases directory: testsystems/platform/
Selected config: _default
List of selected test cases:
KernelBuild

confirm to start execution? [y] 

###################################
# running test case: KernelBuild
###################################
WARNING: destination kernel image already exists and will be overwritten: '/tftpboot/v38b/uImage-duts'

******************************************************************
* Building kernel image, please wait - this may take a while...
******************************************************************
 
***********************************
* Finished building kernel image
***********************************


###################################
# finished processing test cases
###################################

2b. Build 2.6 kernel

Building 2.6 kernel requires a writable build directory called obj to be present it the directory where duts is executed from. Source tree is assumed to be located in /git/linux-2.6-denx and can be read-only. The following invocation of duts will:

[duts]$ ./duts -d testsystems/platform/ t rainier -t KernelBuild
Board name: rainier
Testcases directory: testsystems/platform/
Selected config: _default
List of selected test cases:
KernelBuild

confirm to start execution? [y] 

###################################
# running test case: KernelBuild
###################################

******************************************************************
* Building kernel image, please wait - this may take a while...
******************************************************************

***********************************
* Finished building kernel image
***********************************


###################################
# finished processing test cases
###################################

3. Execute LTP test case:

[duts]$ ./duts -d testsystems/ltp/ t v38b
Board name: v38b
Testcases directory: testsystems/ltp/
Selected config: _default
List of selected test cases:
LTPRunAllTests

confirm to start execution? [y]

Configuration

Configuration view concept

The configuration view is a labeled set of firmware, kernel and host contexts definitions. By using this label the user can switch between different pre-defined configurations.

There's one special config view (_default), which covers the default settings, but each of the context attributes can be overridden by user-defined entries.

A context is comprised of attributes

Configuration view description files are kept in DUTS 'config' directory and must end with '.cfg' in order to be recognised and read by the framework.

For further advise and reach comments on the .cfg files structure and composition refer to the docs/example.cfg file.

Listing defined config views

[duts]$ ./duts c
Defined configuration views:
  _default
  config_L26

Selecting a particular config view

[duts]$ ./duts t v38b -c config_L26
Board name: v38b
Selected config: config_L26
List of selected test cases:

Advanced topics

Notes on developing test cases

Sections

Each test case it described by a series of sections, with a mandatory set required in every test case. Below are only highlights of the most commonly used - for full anatomy of a test case refer to docs/example.tc and to real-life examples in the available testsystems.

The most important sections from the perspective of a testsystem developer are the following:

Commands (mandatory) Pre (optional) Post (optional)

There can only be one of each of these per test case and each contains a series of commands and expected outputs, or specifies an external executable to be invoked by the framework. Actions specified in Pre and Post sections are performed before and after actions in Commands section, respectively. Additional feature of Pre/Post is their flows are not logged to the test case log file. They are meant to allow for preparation before the proper actions and cleaning up afterwards.

Simple vs. custom test cases

Instead of specifying test case flow by a sequence of inputs and outputs (a so called 'simple' test case approach), the user can provide a pointer to external script with custom design contents that is invoked by the framework ('custom' test case approach). This allows for non-standard behaviour and a generic way to extend th framework. There can only be one external script specified in the a given Pre, Commands or Post sections.

Each test case is assigned to a context as mentioned in the overview. The Type field is the connection point between a test case and a particular context implementation. The context that a given test case relies on must be specified in a duts_config structure and an associated configuration view definition must be selected (-c) when running the test case.

Board specific test cases, overloading

When the main 'duts' tool is ordered to run test cases, they are executed against a certain device (board), and while typically they are supposed to be uniform across different boards and CPU architectures, some may require board-specific handling (and hence proprietary implementation).

Upon startup 'duts' searches the default test cases directory and reads in all definitions found, then builds internal representation for further processing; when a sub-directory is found with the name of the current board target it is also searched for board-specific test cases, and if exist they are also included.

Note DUTS does not check for test cases' names (definitions) conflicts: if multiple definitions of a test case exist with the same name, the last one prevails. This allows for some form of overloading, e.g. there can be a generic test case used for most of the boards, with a custom implementation for some, all with just one name the operator can use for all boards.

Introducing support for a new board

Support for a new target board begins with producing a device description and putting it in a .tgt file, first creating the .tgt file itself if the board does not belong to some already existing.

There is a special '_common' device definition that is always pulled in first by the framework, but its settings can be overridden by a specific device's description.

Given the current default DUTS settings the only requirement for a new board to be recognised by the framework is to provide a new duts_device structure representing the board. For proper usage of the test cases it is usually required to provide the Vars section.

IMPORTANT NOTICE for DULG users: be very careful when defining variables used in the flash-related test cases, grouped in the testsystems/dulg/01_flash.tc and testsystems/dulg/03_UBootCmdGroupFlash.tc files.

For further reading refer to docs/example.tgt, which gives elaborate comments on the .tgt files structure and composition, devices/NOTES provide additional hints and finally existing devices/*.tgt descriptions can be of substantial help when adding support for a new target device.

Introducing suppport for a new VL

The default DUTS configuration is oriented around the local DENX virtual laboratory set-up. When the test framework is required to run in some other environment the low-level layer operations need to be provided:

Self-hosted environment

For demonstration and guidance purposes a VL-less configuration view is provided for developers adapting DUTS to other laboratory set-ups. The self-hosted mode of operation assumes a simplistic scenario when there is only a single target device considered, and its console is directly connected to the host machine over a serial interface.

This is intended to be example only, no full management of the device can be secured in such set-up (like switching on and off the device etc.) so certain operations require manual intervention of the operator. Another limitation is that connection details are buried in the _device_connect_target() method; the default uses 'rlogin' connection, but commented out is also variant using the standard 'cu' command. For details please refer to config/self-hosted_ops.tcl.

The self-hosted mode of operation is selected with 'self-hosted' configuration view like in the following example:

[duts]$ ./duts t v38b -c self-hosted -t UBootVersion
Board name: v38b
Testcases directory: ./testsystems/dulg
Selected config: self-hosted
List of selected test cases:
UBootVersion

confirm to start execution? [y]

####################################
# running test case: UBootVersion
####################################

* * * * * * * * * * * * * * * * * * * * * *
*   Please perform the following tasks:
* * * * * * * * * * * * * * * * * * * * * *

 1. power on the target device
 2. leave firmware with the ready prompt (this may require breaking autoboot etc.)
 3. disconnect from the console port (so DUTS can take over)

Please confirm the above steps were completed? [y]

Known limitations