• QaWiki
• AutomatedTests
• WritingTests
• t_functional

Writing QA tests for t_functional

Introduction

This page provides an overview on writing test scripts for the "t_functional" CentOS QA process.

Quick overview

Writing tests for t_functional is 1 part web interface, 1 part git command line, 2 parts text editor and 2 parts virtual machine testbed. In essence you:

1. Create your own clone of the CentOS github t_functional repository using the github web interface. This gives you something to push your changes to.

2. Create a "working copy" of that clone in a local environment (something that contains an install of git, a text editor, a CentOS VM environment etc).
3. Make changes to this working copy - add new tests, improve existing tests, update documentation etc.
4. Commit your changes, adding suitable comments to describe the nature of the change(s).
5. Push your changes to your github-based clone.
6. Submit a "request merge" using the github web interface.

When you make a request to merge, one of the CentOS QA team will review your request(s) and assuming everything is OK, merge your tests into the master branch.

Getting Started

You'll need a GitHub account and access to an environment that contains a working install of git, a text editor of some kind for writing tests and (ideally) a virtual machine environment with which to exercise your scripts prior to submission.

All of the necessary commands required to get your local copy up and running are explained in the GitHub interface once you create your own clone.

If you're new to git, try this Getting Started with Gitorious and Git.

What are the QA scripts?

Small, self-contained test scripts that provide "component testing" of CentOS RPMs. These scripts verify packages install correctly via yum and ensure that whatever the package installs, works as expected. If you change settings for your script (i.e. change the httpd configuration), make shure you revert it back to its original state after the test completes, so it doesn't interfere with other tests that might need the same package.

What are they used for?

Quality assurance - making sure that each package installs and functions correctly on a given architecture. The CentOS QA process directly benefits from having a set of repeatable, automated tests to run against each distinct build and package as and when it's created.

When do they get used?

As part of the QA process for every CentOS build prior to its testing/release, every time a CentOS supplied package is updated and on a nightly basis.

Where are they stored?

In a publically available repository hosted on gitorious.org

What's in the repository?

Here's a breakdown:

What language are tests written in?

Most of the test scripts are written in Bash. You're free to write test scripts in any language that's installable via yum - Python/Perl/Ruby etc. The only proviso is that, as a first step, you make a call (using a simple Bash script) to:

• t_InstallPackage <package_name> 

to install whatever package(s) need to be available for your subsequent, non-Bash test scripts to execute against. In short, at least some part of your test scripts will need to be in Bash.

What's t_installPackage?

To promote a manageable level of consistency across the test suite, a handful of useful functions and variables have been consolidated into a small (but expanding) Bash library. Standard test script tasks such as logging, service control, package installation etc should (ideally) all be performed via calls to functions provided by the helper library. Unless there's a sound (and documented) reason for not doing so, use of the library should be preferred at all times.

Is the Bash library documented anywhere?

All of the functions available in the Bash library are fully documented using comments contained within the library itself, there's nothing particularly complicated or cryptic in the implementations, so you should be able to work out what's going on fairly easily. If the usage of anything in the library isn't obviously self-evident, please let us know. The library itself is by no means comprehensive and simply serves as the basis for writing consistent test scripts.

How do I use the library?

You should include the following statement in your test script:

• source ../0_lib/functions.sh

If you run the tests with the runtests.sh script, that is allready done in there.

What return value/exit status should I return?

Your scripts should exit with a status code of 0 to indicate success, and 1 to indicate failure.

What environment is best for writing tests?

Something with a working copy of git, a text editor to write tests in, and (preferably) a virtual machine environment so you can run tests/roll back/run tests/roll back etc etc. As all tests are run against C5/C6 for both architectures, please be certain your tests work in all available cases (as there might be different paths, other package names or other differences between C5 and C6).

What should I be testing for?

First and foremost, that the packages you've chosen to test, install correctly. This should be the first thing your script does via a call to t_InstallPackage. t_InstallPackage will (as the name suggests) attempt to install the requested packages from the local QA repository (or from the web if you run a local test VM) via a call to yum. If the yum install process fails, t_InstallPackage will exit with a fail status code, and the test harness halts.

Assuming the package(s) install correctly, your scripts should then exercise the packages binaries in however way you see fit - start with something simple; perhaps just calling the binary with the --help switch and checking the exit status is correct (or grep'ing for expected words). Once you're comfortable with how the test scripts work, try something more inventive/advanced

Is there an execution order?

Tests are executed in alphabetical order. Any files named 'readme' (case insensitive) or starting with '_' are ignored. On that basis, if you have any shared variables or config values that need a home, you could put them in a file named (for example) '_config' and refer to it from within your test scripts. You are of course free to keep everything inside a single file, but if it's a common value shared amongst your test scripts for a given package, it might make sense to separate things into a stand-alone file; whatever you believe is the most managable arrangement.

How much debugging output should I provide?

You're free to produce as much debugging output as you feel is necessary to convey the actions your script is performing. If your script returns an exit status indicating failure, it's (obviously) a lot easier to decipher what went wrong if your script is emitting clear and concise messages. As a first step, each of your test scripts should make a call to t_Log (or similar, if you're not using Bash), including the name of your script and a short description of what you're testing for. For example

• t_Log "Running $0 - checking procinfo runs and returns non-zero exit status."

What should I name my tests?

Scripts are processed in alphabetical order and grouped together into folders on a per-package basis. Package test folders should be named p_XXX where XXX matches the output of:

• rpm -q --qf "%{name}\n" <package_name>

Following the same approach, files within each package test folder are processed in alphabetical order. So (for example), tests that start with '0_' are processed before those starting with '5_', which are processed before those starting with '10_' etc. You should install any packages that your test requires in low-numbered scripts and then test against that package in incrementally higher scripts. If that makes no sense, see the "Hello World" example at the end of this document for a practical example.

How do I test my tests?

In order to test your scripts in "stand-alone" mode, you'll need to perform the following command (assuming you're in the t_functional directory):

• source tests/0_lib/functions.sh

You can try executing the runtests.sh script found in t_functional, but some of the tests in 0_common will fail owing to the repo.centos.qa hosts being unreachable outside of the CentOS QA environment (which is not publically accessable and itself has no public access). If you intend to run the entire test suite - or you create scripts which might fail inside the QA environment due to lacking internet access - you can run the suite using:

• SKIP_QA_HARNESS=1 ./runtests.sh

from within the t_functional directory to skip QA-specific settings.

If you intent to just run your p_xxx tests please use

• SKIP_QA_HARNESS=1 ./runtests.sh p_xxx

from within the t_functional directory.

What comments should I include in my tests?

Start your tests with a comment block which includes your name and e-mail address. After that, make a call to t_Log, passing in $0 (or the non-Bash equivalent for your script's file name). Something like:

• t_Log "Running $0 - Postfix SMTP socket connect + 220 banner response test."

Make the comment relevant to what you are really testing. Some good examples are :

• php: testing mysql connection
• mysql: create database, load sample schema, drop database
• sshd: local connections over ssh work
• sshd: testing key based access is functional

Examples of not so good comments:

• php test
• does this work
• another test

If you are writing a test to satisfy or check for a regression or issue reported at bugs.centos.org, make sure you include reference to that issue number. One way to achieve that is in the test comment above. eg: to test if the issue reported in bugs.centos.org/view.php?id=4955 is being satisfied :

•  t_Log "Running $0 - NTP should use the CentOS pool (#4955)" 

Anything else to watch for when writing tests?

Yes, two things:

• tests should only use CentOS infrastructure. Meaning i.e. a wget test should fetch its page from a site within centos.org.

• The QA infrastructure runs in a closed network. If you write tests which fetch something from 'outside' (once again i.e. a wget test) please include a possibility to check the SKIP_QA_HARNESS variable (take a look at this test) so the difference between QA- and non-QA-infrastructure can be taken into account.

A Practical Example

We'll now assemble all of the above information into a practical example, to help get you started. For the purposes of this example, we're going to stick to Bash - adapt as required based on your language of choice.

Firstly, get yourself a copy of the current testing repository. This is available via

• https://github.com/CentOS/sig-core-t_functional

If you're not familiar with how git works, spend some time searching around the web for a couple of git tutorials to help you get comfortable with the concepts, terminology and execution.

Once you've got a working tree, it's time to pick a package. For the purposes of this example, we'll use AIDE - the Advanced Intrustion Detection Engine.

First thing to do is create a folder for your package. Using the standard Linux `mkdir':

        cd t_functional/tests
        mkdir p_aide

Now that we have a home for our tests, we need to set about getting the package installed. Repeating the advice provided earlier, all test scripts are executed in alphabetical order so we'll put a call to t_InstallPackage in a file named '0-install-aide.sh'. Using your preferred editor:

        #!/bin/bash

        t_Log "$0 - installing AIDE"
        t_InstallPackage aide

That's all we need. Breaking this down, we start our script with a logging statement via t_Log. Nothing particularly special/complex going on there. Following on, we get our package installed via a call to a library provided function - t_InstallPackage. You don't need to check the return values from either of these functions. The t_InstallPackage function evaluates the exit status from yum and if there's a problem, will abort the test run.

Now to write a (very) simple test script to exercise the AIDE binary. Back to your editor, create a new file called 5-aide-basic-test.sh

        #!/bin/bash

        t_Log "$0 - basic AIDE initialisation test"

        AIDE=`which aide`
        [ -x "$AIDE" ] || { t_Log "FAIL: AIDE binary doesn't exist or isn't executable"; exit $FAIL; }

        # Perform an initialisation of the AIDE database
        $AIDE --init

        # Check for a 0 exit status
        t_CheckExitStatus $?

Again, nothing particularly complex here. The only thing probably worth explaining is the call to `t_CheckExitStatus', which is just a convenience wrapper around an evaluation of $? with 0. Using t_CheckExitStatus is the preferred means of evaluating exit codes from previously called functions. If the exit status isn't 0, a failure message is logged and the test harness halts.

More information

Any questions should be directed at either the centos-devel mailing list or the #centos-devel channel on Freenode IRC.