TestNG is a testing framework for the Java programming language inspired by JUnit and NUnit but introducing some new functionalities that purport to make it more powerful and easier to use. TestNG is designed to cover all categories of tests, including unit, functional, and integration tests. You may be thinking that we already have JUnit, which evolved a lot on versions 4.0 and 4.5 and it's being used in lots of project, so why do we need TestNG? Well, after checking the net for an answer to this I read that JUnit has always been a unit-testing framework, meaning that it was built to facilitate testing single objects, and it does so quite effectively. TestNG, on the other hand, was built to address testing at higher levels, and consequently, has some features not available in JUnit. One of these features is the flexibility of TestNG. While the code looks quite similar while comparing a JUnit tests and a TestNG one, TestNG doesn't require you to have public or static methods for certain steps of the lifecycle, while JUnit does. This gives the TestNG users more flexibility when developing tests based in POJOs. Another important difference is the existence of dependencies. With JUnit you can't mark a test as depending in another, thus makiung impossible to specify a test order unless you use some tricks like namig of tests. With TestNG you can declare dependencies, so one test won't run until the dependency has been tested and succeeded. TestNG also generates an XML file with a list of failed tests, so you can run only these tests once you fix the errors, something quite handy when you application has some thousands of test units and that JUnit doesn't provide (although IDE's plugins remove this difference). And last but not least TestNG offers parametric testing. This means you can define groups of parameters for a test in an xml file and use it for the test, allowing you to test the result with different values without having to write several test cases. There are some other minor differences but as you can see TestNG is an extension of the capabilities of JUnit. It's true you can do the same (or something quite similar) with JUnit and some plugins, but TestNG provides these features by default. TestNG is integrated with Maven by default and can be called as an Ant task. It has pluguins for all the popular IDE (Eclipse, Netbeans, IDEA).

TestNG

You can find the documentation of TestNG in the official page, and a book is available that describes the framework in more detail. Before starting, if you have any existing JUnit test you can run JUnitConverter to migrate them automatically to TestNG, so you can see the differences. Writing a test with TestNG is typically a three-step process:

• Write the business logic of your test and insert TestNG annotations in your code.
• Add the information about your test (e.g. the class name, the groups you wish to run, etc...) in a testng.xml file or in build.xml.
• Run TestNG.

In TestNG a suite is represented by one XML file. It can contain one or more tests and is defined by the <suite> tag. A test is represented by <test> and can contain one or more TestNG classes. A TestNG class is a Java class that contains at least one TestNG annotation. It is represented by the <class> tag and can contain one or more test methods. A test method is a Java method annotated by @Test in your source. A TestNG test can be configured by @BeforeXXX and @AfterXXX annotations which allows to perform some Java logic before and after a certain point. The list of valid annotations is:

• @BeforeSuite: The annotated method will be run before all tests in this suite have run.
• @AfterSuite: The annotated method will be run after all tests in this suite have run.
• @BeforeTest: The annotated method will be run before the test.
• @AfterTest: The annotated method will be run after the test.
• @BeforeGroups: The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked.
• @AfterGroups: The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked.
• @BeforeClass: The annotated method will be run before the first test method in the current class is invoked.
• @AfterClass: The annotated method will be run after all the test methods in the current class have been run.
• @BeforeMethod: The annotated method will be run before each test method.
• @AfterMethod: The annotated method will be run after each test method.
• Parameters of these annotations:
  • alwaysRun: For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to. For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.
  • dependsOnGroups: The list of groups this method depends on.
  • dependsOnMethods: The list of methods this method depends on.
  • enabled: Whether methods on this class/method are enabled.
  • groups: The list of groups this class/method belongs to.
  • inheritGroups: If true, this method will belong to groups specified in the @Test annotation at the class level.

• @DataProvider: Marks a method as supplying data for a test method. The annotated method must return an Object[][] where each Object[] can be assigned the parameter list of the test method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equals to the name of this annotation.
  • name: The name of this DataProvider.
• @Factory: Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return Object[].
• @Parameters: Describes how to pass parameters to a @Test method.
  • value: The list of variables used to fill the parameters of this method.
• @Test: Marks a class or a method as part of the test.
  • alwaysRun: If set to true, this test method will always be run even if it depends on a method that failed.
  • dataProvider: The name of the data provider for this test method.
  • dataProviderClass: The class where to look for the data provider. If not specified, the data provider will be looked on the class of the current test method or one of its base classes. If this attribute is specified, the data provider method needs to be static on the specified class.
  • dependsOnGroups: The list of groups this method depends on.
  • dependsOnMethods: The list of methods this method depends on.
  • description: The description for this method.
  • enabled: Whether methods on this class/method are enabled.
  • expectedExceptions: The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
  • groups: The list of groups this class/method belongs to.
  • invocationCount: The number of times this method should be invoked.
  • invocationTimeOut: The maximum number of milliseconds this test should take for the cumulated time of all the invocationcounts. This attribute will be ignored if invocationCount is not specified.
  • successPercentage: The percentage of success expected from this method
  • sequential: If set to true, all the methods on this test class are guaranteed to run sequentially, even if the tests are currently being run with parallel="true". This attribute can only be used at the class level and it will be ignored if used at the method level.
  • timeOut: The maximum number of milliseconds this test should take.
  • threadPoolSize: The size of the thread pool for this method. The method will be invoked from multiple threads as specified by invocationCount. Note: this attribute is ignored if invocationCount is not specified

A test is considered successful if it completed without throwing any exception or if  it threw an exception that was expected (see the documentation for the expectedExceptions attribute found on the @Test annotation). Your test methods will typically be made of calls that can throw an exception, or of various assertions (using the Java "assert" keyword).  An "assert" failing will trigger an AssertionErrorException, which in turn will mark the method as failed (remember to use -ea on the JVM if you are not seeing the assertion errors). TestNG also include JUnit's Assert class, which lets you perform assertions on complex objects (check JUnit page).

Examples

Next some examples of TestNG in use. First a piece of code that shows how to declare some groups for our Tests, so we run one group or another depending on our needs:

public class Test1 {
  @Test(groups = { "functest", "checkintest" })
  public void testMethod1() {
  }
  @Test(groups = {"functest", "checkintest"} )
  public void testMethod2() {
  }
  @Test(groups = { "functest" })
  public void testMethod3() {
  }
}

You can define the groups at class level adding any method-specific group if needed:

@Test(groups = { "checkin-test" })
public class All {
  @Test(groups = { "func-test" )
  public void method1() { ... }
  public void method2() { ... }
}

Parameters can be defined in the TestNG XML file, and be called like:

@Parameters({ "first-name" })
@Test
public void testSingleString(String firstName) {
  System.out.println("Invoked testString " + firstName);
  assert "Cedric".equals(firstName);
}
@Parameters({ "datasource", "jdbcDriver" })
@BeforeMethod
public void beforeTest(String ds, String driver) {
  m_dataSource = ...;                              // look up the value of datasource
  m_jdbcDriver = driver;
}

You can even define default values in case the parameter is not defined in the XML file:

@Parameters("db")
@Test
public void testNonExistentParameter(@Optional("mysql") String db) {
    //code
}

Data Providers can be used instead of parameters to run the test using a set of values:

//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
 return new Object[][] {
   { "Cedric", new Integer(36) },
   { "Anne", new Integer(37)},
 };
}
//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
 System.out.println(n1 + " " + n2);
}

Or use some class as provider if it can be used for several test classes:

public static class StaticProvider {
  @DataProvider(name = "create")
  public static Object[][] createData() {
    return new Object[][] {
      new Object[] { new Integer(42) }
    }
  }
}
public class MyTest {
  @Test(dataProvider = "create", dataProviderClass = StaticProvider.class)
  public void test(Integer n) {
    // ...
  }
}

If you declare your @DataProvider as taking a java.lang.reflect.Method as first parameter, TestNG will pass the current test method for this first parameter. This is particularly useful when several test methods use the same @DataProvider and you want it to return different values depending on which test method it is supplying data for:

@DataProvider(name = "dp")
public Object[][] createData(Method m) {
  System.out.println(m.getName());  // print test method name
  return new Object[][] { new Object[] { "Cedric" }};
}
@Test(dataProvider = "dp")
public void test1(String s) {
}
@Test(dataProvider = "dp")
public void test2(String s) {
}

You can specify dependencies for your test, both on methods:

@Test
public void serverStartedOk() {}
@Test(dependsOnMethods = { "serverStartedOk" })
public void method1() {}

or on groups:

@Test(groups = { "init" })
public void serverStartedOk() {}
@Test(groups = { "init" })
public void initEnvironment() {}
//dependency on any group that starts with "init."
@Test(dependsOnGroups = { "init.* })
public void method1() {}

You can use Factories to create test objects dynamically:

public class WebTestFactory {
  @Factory
  public Object[] createInstances() {
   Object[] result = new Object[10];
   for (int i = 0; i < 10; i++) {
      result[i] = new WebTest(i * 10);
    return result;
  }
}
public class WebTest {
  private int m_numberOfTimes;
  public WebTest(int numberOfTimes) {
    m_numberOfTimes = numberOfTimes;
  }
  @Test
  public void testServer() {
   for (int i = 0; i < m_numberOfTimes; i++) {
     // access the web page
    }
  }
}

There are other advanced functionalities like Dependency Injection or reporting available, but they are out of scope here. As you can see TestNG is, by default, much more versatile (and powerful) than JUnit.

TestNG and EJB3

With JUnit we have an extension available to test EJB 3.0. TestNG can make use of its capabilities ot test EJB without needing any external library. There's an example on how to use JBoss for this, and in the end of the post the author also mentions an structure used to test JSF + EJB. The key is to build a class that will setup the environment for you and use it in your test. Thanks to annotations we can tell TestNG how to manage that helper class, making our life easier and giuving us absolute control on what can it do.

Using JUnit extensions in TestNG

That's probably the killer functionality of TestNG. Thanks to its versability it allows you to use JUnit extensions, although with some restrictions (so you can't use 100% of them).

TestNG and JMockit

Any test framework needs a proper mock objects' framework to be able to test the application properly. JUnit has the well known EasyMock, but this framework is quite coupled with JUnit so it doesn't work for us. For TestNG the recommended framework is JMockit. You can check the official tutorial and this two parts tutorials for information on the framework.