Testing with Renode

Renode’s testing capabilities

Renode is very well suited to be a part of an automated tests scenario, e.g. run in the background on a CI server.

Renode is integrated with the Robot Framework testing suite and provides user-friendly scripts for running tests.

It comes with a variety of prepared test scripts but it also allows you to extend them or add new ones.

Running the robot test script

Running a robot test script in Renode is as simple as executing a single command:

$ renode-test my_test.robot

Note

The renode-test command is available if Renode is installed in the system from package. For a version built from the source code, use test.sh instead.

The above command will:

  • start a Renode instance in the background,
  • enable Renode’s built-in Robot Framework server (providing an interface between Robot Framework and Renode) on port 9999 (the port number can be changed by the user),
  • start the Robot Framework test engine and connect to Renode,
  • run the provided my_test.robot test case,
  • print the progress status on the console,
  • generate the log and the summary after finishing the test.

Below, you can see an example output:

Preparing suites
Started Renode instance on port 9999; pid 2293056
Starting suites
Running tests/platforms/LiteX-VexRiscv.robot
+++++ Starting test 'LiteX-VexRiscv.Timer Test'
+++++ Finished test 'LiteX-VexRiscv.Timer Test' in 46.79 seconds with status OK
+++++ Starting test 'LiteX-VexRiscv.I2C Test'
+++++ Finished test 'LiteX-VexRiscv.I2C Test' in 8.10 seconds with status OK
Cleaning up suites
Closing Renode pid 2293056
Aggregating all robot results
Output:  /home/antmicro/renode/output/tests/robot_output.xml
Log:     /home/antmicro/renode/output/tests/log.html
Report:  /home/antmicro/renode/output/tests/report.html
Tests finished successfully :)

Note that two entries in the output associated with a single test case appear - one when the test starts and another when it finishes (useful when running tests in parallel). The second message contains information about the test’s duration and status.

The details of the run can be found in:

  • the robot_output.xml report (suitable for automatic parsing),
  • the log.html and report.html documents (suitable for an interactive inspection).

Creating the test file

Robot Framework uses text files based on a custom syntax to express the test cases. The details of the grammar can be found in the official documentation. In order for the robot files to work with Renode, proper configuration (explained below) is needed.

Here is an example of a simple robot test file that works with Renode:

*** Settings ***
Suite Setup     Setup
Suite Teardown  Teardown
Test Teardown   Test Teardown
Resource        ${RENODEKEYWORDS}

*** Test Cases ***
Should Print Help
    ${x}=  Execute Command     help
           Should Contain      ${x}    Available commands:

The Should Print Help test case executes the help command in Renode’s monitor and verifies the result.

Integration with Renode is achieved by adding entries to the settings section. The RENODEKEYWORDS variable (initialized by the renode-test script) contains the path to the renode-keywords.robot script responsible for setting up the connection with Renode. The rest of the settings configure the suite/test setup and teardowns.

It is recommented to copy the above Settings section to each new robot test file.

Adding new test cases

Each robot test file might contain many test cases. For general instructions on how to define tests cases, please refer to the Robot Framework documentation. In this section we will focus on using the Robot Framework-Renode integration.

The Robot Framework-Renode integration layer provides keywords allowing the user to control and inspect the state of the simulation directly from the Robot Framework test file in a similar manner to the built-in keywords. The basic keywords allow the user to:

  • start the emulation (Start Emulation),
  • clear the emulation (Reset Emulation),
  • execute a command in the Monitor (Execute Command),
  • allocate a file in the Renode temporary folder (Allocate Temporary File),
  • download a file to the Renode temporary folder (Download File),

and more (see the source code for details).

Additionally, Renode provides a set of keywords for interacting with UART devices. They allow to:

  • write a text to UART (Send Key To Uart, Write Char On Uart, Write Line To Uart),
  • wait for a specific line to appear on UART (Wait For Line On Uart, Wait For Prompt On Uart),
  • wait for any output on UART (Wait For Next Line On Uart),
  • wait for the lack of output on UART (Test If Uart Is Idle).

There is also a set of keywords for interacting with network devices that allows to:

  • wait for a next outgoing network packet (Wait For Outgoing Packet),
  • wait for a specific outgoing network packet (Wait For Outgoing Packet With Bytes At Index).

It is possible to extend the Renode-Robot Frameworkd interface by implementing more keywords in C# if necessary.

For reference on how to use the keywords mentioned in this section, see the robot test files that Renode comes with.

Advanced usage

Running many test files with a single command

The example in the previous section presented how to run a single test file (which might still contain many test cases). It is possible to run many test files and aggregate the results into a single report. In order to do that, you need to pass many test files as an argument to renode-test command:

$ renode-test my_tests.robot additional_tests.robot extra_tests.robot

The tests will be executed in the same order as the arguments were provided in.

An alternative way is to prepare a yaml file with the list of tests to execute, e.g.:

- my_tests.robot
- additional_tests.robot
- extra_tests.robot

and to call renode-test with a special switch:

$ renote-test -t my_tests.yaml

Note

The yaml notation allows the user to include other yaml files and to group entries that should not be executed in parallel (see the next section).

Running tests in parallel

Test cases from a single file will always be executed in serial (in the order they are defined in the file), but it’s possible to run test from different files in parallel. In order to do that, execute the renote-test command with a special switch:

$ renode-test -j12 my_tests.yaml

This will allow to run up to 12 Renode instances, each one running test cases from a different file. Using the yaml file allows grouping entries that should not be executed in parallel (because, e.g., they use a shared resource like a port number):

- my_tests.robot
- my_group:
    - my_test2.robot
    - my_test3.robot

In the example above, my_test2.robot will be executed before my_test3.robot but in parallel with my_tests.robot.

You can also pass many test files as arguments (i.e., without the yaml file), but this won’t allow you to do the grouping:

$ renode-test -j3 my_tests.robot my_tests2.robot my_tests3.robot

Stopping on error

By default renode-test will run all the provided test cases. It is possible, however, to stop the execution on the first encountered error. In order to do that, run the renode-test script with:

$ renode-test --stop-on-error my_tests.robot

Running multiple instance of renode-test at the same time

Renode communicates with the Robot Framework executor over a network socket. This means that running two test-renode instances at the same time will result in a network port conflict.

In order to avoid that, you can explicitly specify the port number to be used for the communication between Robot Framework and Renode:

$ renode-test -P 9997 my_test.robot &
$ renode-test -P 9998 my_test2.robot &

Repeating tests

It is possible to run the specified tests multiple times using:

$ renode-test -n 10 my_test.robot

This will repeat all the test cases from my_tests.robot 10 times.

Running selected fixtures

It is possible to run only selected test cases from the file using:

$ renode-test -f "*GDB*" my_tests.robot

In the example above only test cases with GDB in their name will be run.

Running tests interactively

By default, renode-test command will run tests in the background and just report results on the console. It is, however, possible to enable printing log messages to the console just like when running the renode command:

$ renode-test --show-log my_tests.robot

Note

Note that this will cause the test progress report messages to be mixed with the log messages.

What’s more, it is also possible to show the Monitor and analyzers windows and interact with them:

$ renode-test --enable-xwt my_tests.robot

Note

Note that interacting with the running test may influence the results.