How we do automated regression testing with Selenium and Hudson

When developing a piece of software that has a lifecycle that spans over several years and periodically is released you have to do regression testing, i.e. making sure that previous features doesn't break because of new stuff.

And after a while when your software grows and you add new features, the amount of regression tests increases. To avoid drowning yourself with testing you need to automate as much as possible.

The Sculptor team is a bunch of guys that are driven by interest and are developing the software on there spare time during late nights. We doesn't have the time to do deep manual testing, hence automation is very attractive for us. And since we are geographically distributed and don't have a central CI environment we have to solve some practical problems locally.

What we do (amongst other things), involves having a local Hudson server running on our developing environment (i.e. my iMac) that (of course) builds all projects and runs unit tests.

But we also use Selenium to run automated functional tests to make sure that our example application works. I though I should show some more details about how we use Selenium.

As I said, we have an example application called Library. If you look in the source code for the library-web module you will find the directory:

src/main/webapp/selenium

In there you will find a bunch of tests. The root is the suite file:

___test-suite.xhtml

Selenium test can be written in various programming languages. We have chosen to keep it simple and implement the tests in html.

Having these tests enables us to run them as soon as the code base changes thanks to Hudson and Maven. In Hudson we just creates a new job that is triggered to run as soon as the Library projects is compiled and the unit tests passes. In the pom-file for the Library web project we have a profile that we can use to start a local instance of the Jetty server. Deploy the the application. Run our selenium tests. And finally stop the server. The Maven configuration for this is:

<profile>
<id>regression</id>
<build>
<plugins>
 <plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.11</version>
   <configuration>
     <scanIntervalSeconds>10</scanIntervalSeconds>
     <stopKey>foo</stopKey>
     <stopPort>9999</stopPort>
   </configuration>
   <executions>
     <execution>
       <id>start-jetty</id>
       <phase>pre-integration-test</phase>
       <goals>
         <goal>run</goal>
       </goals>
       <configuration>
         <scanIntervalSeconds>0</scanIntervalSeconds>
         <daemon>true</daemon>
       </configuration>
     </execution>
     <execution>
       <id>stop-jetty</id>
       <phase>post-integration-test</phase>
       <goals>
         <goal>stop</goal>
       </goals>
     </execution>
   </executions>
 </plugin>
 <plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>selenium-maven-plugin</artifactId>
   <version>1.0</version>
   <configuration>
     <suite>src/main/webapp/selenium/___test-suite.xhtml</suite>
     <browser>*firefox</browser>
     <results>${project.build.directory}/target/selenium.html</results>
     <startURL>http://localhost:8080/${artifactId}</startURL>
   </configuration>
   <executions>
     <execution>
       <id>run-tests</id>
       <phase>integration-test</phase>
       <goals>
         <goal>selenese</goal>
       </goals>
     </execution>
   </executions>
 </plugin>
</plugins>
</build>
</profile>

But, since the amount of tests keeps growing and the time it takes to run them all also grows, there is a need to be able to run just a single test or a few of them. For instance, it is very convenient to have that possibility when developing new features, fixing bugs, or just doing a refactoring. To enable this we use the very competent Firefox plugin called SeleniumIDE. Beside recording test case, you can also load already defined test cases (or suites) and run them.