Building a unit testing environment with PHPUnit

As the software grows up, the need for proper testing becomes more and more important. Huge number of dependencies between the components requires something more than manual checking if everything works after implementing a new feature. The solution is automated testing. In case of PHP, the best tool for testing the scripts is PHPUnit by Sebastian Bergmann. The package is based on JUnit in Java, so everyone who programmed in that language, should get into it very quickly. In this post, I would like to describe, how to build a testing environment for our application with PHPUnit.

As the software grows up, the need for proper testing becomes more and more important. Huge number of dependencies between the components requires something more than manual checking if everything works after implementing a new feature. The solution is automated testing. In case of PHP, the best tool for testing the scripts is PHPUnit by Sebastian Bergmann. The package is based on JUnit in Java, so everyone who programmed in that language, should get into it very quickly. In this post, I would like to describe, how to build a testing environment for our application with PHPUnit.

PHPUnit

PHPUnit runs so-called unit tests. A single unit test verifies the correctness of one particular project item. An item could be a class method or any other self-containing element that can be rationally tested. A set of unit tests allows to verify a single component using the following assumptions: if the single tests pass, it means that the method implementations do not have visible bugs and we can assume that the whole component implementation works. PHPUnit executes tests one after another and compares the actual results with the expected ones, defined by the programmer. If they are equal, the test passes. At the end, we get a report with the number of passed tests and detailed information about failures or errors.

A big advantage of the package is a wide variety of options for building even very complex test environments for huge applications, support for mock objects or generating code coverage reports, when connected with XDebug.

Installing PHPUnit

The installation procedure will be described briefly. I hope that this post is read by programmers who actually heard something about PHPUnit or did something with it and now are looking for more detailed information. Anyway, it is worth mentioning that the author recommends using PEAR installer to install the package:

pear channel-discover pear.phpunit.de
pear install phpunit/PHPUnit

The application works in the command line mode. Let's type phpunit in the console. If we get a list of available options, it means that the package has been installed properly.

Step 1 - directory structure

We start our journey with building a directory structure of our testing environment. PHPUnit allows to create complex environments, where we can precisely select the tests we would like to run at the moment. For example, if we want to test everything before releasing a new version, we run all the tests, and when we add a new feature to a component, we would like to use only the tests that explicitely cover it. However, this requires a piece of magic and planning from us. The directory structure should reflect the structure of our project and its own directories. Consider the following application:

/MyApplication
  /Pertrozer
    /Standard.php
    /Agrippy.php
  /Sprutifier
    /Jawback.php
    /Clonetrick.php
  /Gepertor.php
  /Ummobogorotitor.php

The structure I propose looks like this:

/tests
  /Extra
  /Package
    /Pertrozer
      /StandardTest.php
      /AgrippyTest.php
    /Sprutifier
      /JawbackTest.php
      /ClonetrickTest.php
    /Stuff
      /Stuff1Test.php
    /GepertorTest.php
    /UmmobogorotitorTest.php
    /AllTests.php
  /AllTests.php
  /Bootstrap.php
  /config.xml

Below, I present the assumptions I have used:

1. The tests are located in the /tests/Package directory whose structure is similar to our project one.
2. The tests for a sample file Foo.php should be located in FooTest.php file in the proper folder of /tests/Package.
3. The structure /tests/Package may contain the test files and directories that do not have their counterparts in the original project structure. They can contain unit tests for some other units than classes.
4. In the /tests/Extra directory we can place extra implementations required by the testing environment. A sample use is provided later.
5. The rest of the files are the elements of the testing environment and we are going to describe them now.

Setting up a test suite

Our first tas is writing the implementations of AllTests.php files that represent test suites. Tests suites run all the test cases at the given level. This is the first file: /tests/AllTests.php which runs the whole test environment:

<?php
/**
 * A test suite that runs everything.
 *
 * @author Tomasz "Zyx" Jędrzejewski
 */

require_once 'PHPUnit/Framework.php';
require_once './Package/AllTests.php';

class AllTests
{
    /**
     * Configures the unit testing suite
     *
     * @return PHPUnit_Framework_TestSuite
     */
    public static function suite()
    {
        $suite = new PHPUnit_Framework_TestSuite('Package');
        $suite->addTest(Package_AllTests::suite());

        return $suite;
    } // end suite();
} // end AllTests;

The main reason for creating it is our own comfort, so that we do not have to go too deeply into the directory structure. We have here just one static method called suite(). It creates and returns the PHPUnit_Framework_TestSuite object which collects the objects to run. Here, we only add another, proper test suite returned by Package_AllTests::suite().

Now we can create the file /tests/Package/AllTests.php with very similar, but quire more complex structure:

<?php
/**
 * Runs the tests from the /Package directory
 *
 * @author Tomasz "Zyx" Jędrzejewski
 */

require_once('GepertorTest.php');
require_once('UmmobogorotitorTest.php');

class Package_AllTests extends PHPUnit_Framework_TestSuite
{
    /**
     * Configures the test suite.
     * @return Package_AllTests
     */
    public static function suite()
    {
        $suite = new Package_AllTests('Package');
        $suite->addTestSuite('Package_GepertorTest');
        $suite->addTestSuite('Package_UmmobogorotitorTest');

        return $suite;
    } // end suite();

    /**
     * Executed before the test suite.
     */
    protected function setUp()
    {
        // some setup code
    } // end setUp();

    /**
     * Finishes the testing suite
     */
    protected function tearDown()
    {
        // some finalization code
    } // end tearDown();

} // end Package_AllTests;

This time instead of creating an empty class, we have used the inheritance. Thus, we can use setUp() and tearDown() methods executed before starting and after finishing the test suite. They have several purposes. We may use it to initialize the environment before the testing or cleaning up. In the suite() method we link the tests and test suites that we want to become a part of the current suite. Note that the classes with the test implementations are included at the beginning of the file with simple require_once.

Step 2 - writing tests

The actual unit tests are placed in the class extending PHPUnit_Framework_TestCase. A single unit test is represented by a method testSomething() that does not take any arguments by default. A test case contains the setUp() and tearDown() methods, too. They are fired before and after every executed unit tests. Below, I show the code of a sample file GepertorTest.php:

<?php
/**
 * Tests for Gepertor component
 *
 * @author Tomasz "Zyx" Jędrzejewski
 */

class Package_GepertorTest extends PHPUnit_Framework_TestCase
{
    private $_gepertor;

    protected function setUp()
    {
        $this->_gepertor = new MyApplication_Gepertor;
    } // end setUp();

    protected function tearDown()
    {
        $this->_gepertor->dispose();
        unset($this->_gepertor);
    } // end tearDown();

    public function testMethod1()
    {
        $this->assertEquals('Foo', $this->_gepertor->method1());
    } // end testMethod1();

    public function testMethod2()
    {
        $this->assertEquals('Bar', $this->_gepertor->method2());
    } // end testMethod2();
} // end Package_GepertorTest;

The PHPUnit_Framework_TestCase class provides several useful methods for comparing the given result with the expected values. Their names start with assert and one of them is assertEquals(). In the first argument, we specify the expected value, and provide the actual result to the second one. The method passes, if they are equal. Another useful methods are assertContains() which checks if the first argument is an element of the array or iterator from the second argument. assertSame() is pretty similar to assertEquals(), but also checks the types of the values, similarly to == and === operators. You may use it to compare objects and arrays. For testing boolean values, we have assertFalse() and assertTrue() methods. Finally, we can use fail() which always fails with the specified message. To get the full list of the available assertions, please refer to the PHPUnit manual. Note also that a single unit test may contain more than one assertion, i.e.

list($foo, $bar) = testedFunction(1, 2, 3);
$this->assertEquals(7, $foo);
$this->assertEquals(8, $bar);

In this case, all the assertions must pass to complete the test successfully.

Sometimes we want to pass the test, if the code returned an exception. This is pretty useful, because we should also check, how our code behaves in unexpected and invalid situations. PHPUnit offers several ways to test the returned exception. The first way is to use annotations:

/**
 * @expectedException fooException
 */
public function testSomething()
{
    testedFunction();
} // end testSomething();

We may also set the expected exception manually:

public function testSomething()
{
    $this->setExpectedException('fooException');
    testedFunction();
} // end testSomething();

Or do it manually:

public function testSomething()
{
    try
    {
        testedFunction();
    }
    catch(fooException $exception)
    {
        return;
    }
    $this->fail('Exception fooException not returned');
} // end testSomething();

Step 3 - configuration and project loading

One could expect that we have already written everything we need, but this is not enough. Above all, we need a way to load the code of the tested components into our testing environment and we would not rather use require_once here. As you can see, I do not even try to load the tested classes in the presented code. Currently, nearly all frameworks are equipped with autoloaders and it could be nice to make use of them. The question is: where to initialize them. We cannot do that in the test files, because we will throw away the possibility to run groups of tests at once. It cannot be also placed in the test suites, because we will not be able to run single test cases.

A solution is creating an extra PHP file, /tests/Bootstrap.php. We put the configuration of our autoloader there, without packing it into functions or classes. We will inform PHPUnit about it with an extra option passed to the command line tool or in the configuration file. Personally, I recommend the second way, because we can configure much more useful stuff there. Such a configuration file (/tests/config.xml) is presented below:

<phpunit bootstrap="Bootstrap.php"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         stopOnFailure="false">
  <!-- configure other elements -->
</phpunit>

As the attributes of the main tag <phpunit> we specify different parameters, such as how to treat PHP errors or the location of the bootstrap file. For now, we do not need anything more.

Step 4 - run the tests

This is it - our test suite is completed and we can start testing. We run the command line and go to the /tests directory with the cd command. Here we can call the following commands:

1. Run all the tests:

   phpunit --configuration ./config.xml AllTests
   

2. Run a single test:

   phpunit --configuration ./config.xml Package_GepertorTest
   

After completing the tests, we should get a brief report:

PHPUnit 3.3.9 by Sebastian Bergmann.

.......

Time: 0 seconds

OK (7 tests, 7 assertions)

The dots symbolize passed tests. If a test fails, we can see there a letter F (failure - assertion not passed) or E (PHP error during execution). The failure or error details are presented in an extra report.

Generating code coverage reports

While writing tests, it is useful to know, what lines of code are actually executed during the test execution, and what are not (possibly we would need to write extra tests for them). This problem is known as code coverage and the idea is to show the application source code covered with colors: green, if the line was executed at least once in at least one of the tests or red/orange, if not. PHPUnit allows to generate such reports, if we have XDebug installed. Generating code coverage reports requires some extra configuration in our test suite.

Open the /tests/config.xml file and add the following lines as a body of <phpunit> tag:

<logging>
    <log type="coverage-html" target="coverage-html/" charset="UTF-8"
        yui="true" highlight="true"
        lowUpperBound="35" highLowerBound="70"/>
</logging>
<filter>
      <blacklist>
            <directory suffix=".php">./</directory>
      </blacklist>
      <whitelist>
            <directory suffix=".php">../MyApplication/</directory>
      </whitelist>
</filter>

The first tag, <logging> specifies the directory, where the code coverage report in the HTML format will be placed. PHPUnit supports more formats - their descriptions can be found in the manual, but here we will focus on this one particular situation. Moreover, we need to define some filters. By default, the code coverage report contains the source code of all the loaded files, including our test environment. I know that analyzing the code coverage report of the testing environment may be a fun, but it has no practical meaning for us. This is why we add the current ./ directory to the blacklist and define the application directory in the white list.

Please note that in this case, PHPUnit will always generate us code reports. If we would like to have a control over this process (collecting the information is a memory-consuming process, in certain situations it may lead to crash the testing environment due to the running out of free memory), we must create separate configuration files.

Sometimes, to launch a unit test, we must run some extra application code for initialization purposes. The code may belong to a different component and give us a false feeling that we have covered it with unit tests. PHPUnit offers some tools to define precisely which methods and/or classes are covered by a particular unit tests. If the test runs some code from a different component, it will not be counted to the report. Let's get back to our test case and add the code coverage definitions to it:

<?php
/**
 * Tests for Gepertor component
 *
 * @author Tomasz "Zyx" Jędrzejewski
 */

/**
 * @covers MyApplication_Gepertor
 */
class Package_GepertorTest extends PHPUnit_Framework_TestCase
{
    private $_gepertor;

    protected function setUp()
    {
        // @codeCoverageIgnoreStart
        $this->_gepertor = new MyApplication_Gepertor;
        // @codeCoverageIgnoreEnd
    } // end setUp();

    protected function tearDown()
    {
        // @codeCoverageIgnoreStart
        $this->_gepertor->dispose();
        // @codeCoverageIgnoreEnd
        unset($this->_gepertor);
    } // end tearDown();

    /**
     * @covers MyApplication_Gepertor::method1
     */
    public function testMethod1()
    {
        $this->assertEquals('Foo', $this->_gepertor->method1());
    } // end testMethod1();

    /**
     * @covers MyApplication_Gepertor::method2
     */
    public function testMethod2()
    {
        $this->assertEquals('Bar', $this->_gepertor->method2());
    } // end testMethod2();
} // end Package_GepertorTest;

We have specified explicitely, what methods are intended to be covered by each test case. The method comment may contain several @covers annotations, if the test covers more than one method. Note also that we enclosed some initialization and destruction code within @codeCoverageIgnoreStart and @codeCoverageIgnoreEnd block, so that it would not be counted as testing the constructors or destructors.

Tips and tricks

Different application components require singinficantly different approaches, often not-so-easy to invent and/or implement. A couple of months ago, I read a post written by one of the PEAR 2 Pyrus installer authors where he talked about the unit testing suite for it. The unit tests required connecting with external servers, downloading and installing fictional packages. The time needed to take the full testing procedure was about half an hour. This shows that writing unit tests is not a piece of cake and sometimes requires more patience and time than writing the actual application.

When I started writing unit tests for Open Power Template a year ago, my biggest problem was testing the syntax of the template language. I realized that creating a separate method for each test, then putting templates into separate directories and so on would take too much time and make me get lost in the overall procedure. Then I noticed that I can extend PHPUnit with something like virtual file system. The complete unit test, including templates, PHP initialization code and expected results is saved as a single text file on the hard drive. It looks like this:

The test checks the attribute loops.

>>>>templates/test.tpl
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<opt:root>
<foo>
<opt:attribute name="$attributes.name" value="$attributes.value" opt:section="attributes" />
bar
</foo>
</opt:root>

>>>>data.php
$view->attributes = array(0 =>
    array('name' => 'abc', 'value' => 'def'),
    array('name' => 'ghi', 'value' => 'jkm')
);

>>>>expected.txt
OUTPUT

>>>>result.txt
<foo abc="def" ghi="jkm">
bar
</foo>

The beginning of a file is a comment that is ignored by the parser. Each new file is followed by a block >>>>filename that defines its name. I used some conventions. templates/test.tpl contains the main test template. data.php can be used to initialize the view and/or main class with some necessary definitions. The expected result is defined in the expected.txt file. If it contains the word OUTPUT, the test case compares the results to the contents of result.txt file. In any other case, it expects the exception defined in expected.txt. To map the file to a filesystem I used PHP streams functionality and wrote a special wrapper, so everything acts like normal files:

file_get_contents('test://expected.txt');
include('test://data.php');

Another trick involves using the data providers. We can easily notice that the code that launches the textual unit tests is pretty the same, and the only difference is the name of the file to load. I do not create 150 methods like testSomething() that do exactly the same, but rather define a single one with one argument - the file to run:

/**
 * @dataProvider instructionProvider
 */
public function testInstruction($filename)
{
    // the code that loads and tests the file
} // end testInstruction();

You can notice an annotation. @dataProvider informs about a method that generates the data for the unit test and serves them in the arguments. It looks like this and is placed in the same test case class, as the unit test:

public function instructionProvider()
{
    return array(0 =>
        array('attribute_1.txt'),
        array('attribute_2.txt'),
        array('attribute_3.txt'),
        array('block_1.txt'),
        array('block_2.txt'),
        // etc...
    );
} // end instructionProvider();

In this way, I can easily test the language syntax.

Summary

PHPUnit provides lots of different features and I was not able to describe them in just one post. However, I hope that it will help you building your own automated testing environment for your applications.