Java Unit Test with TestNG

Terminology

• CUT (Class Under Test) – a class in the production code which is being tested
• MUT (Method Under Test) – a method in the production code which is being tested
• Fixture – the set of fields defined in a test class and initialized in setUp() for use in multiple tests

Frameworks

While you will not be arrested by the Unit Testing Police for not following the policies below, you will be helping the team by ensuring uniformity and consistency:

• Use TestNG
  • This is the prevailing framework and easily supports test groups to facilitate integration and other kinds of tests

  • Although TestNG allows you to use JUnit style assertions, it is less confusing to simply stick with TestNG style everywhere

• Use Mockito

  • Although some existing tests were written with EasyMock, you should find that Mockito can easily do everything that EasyMock does but with much less code

• Avoid PowerMock
  • While PowerMock can coexist with TestNG and Mockito, the features it provides enable developers to work around untestable code, rather than making the code more testable

• Use Hamcrest

  • This is primarily relevant to special assertions using assertThat()

Workflow

If you follow a TDD methodology, write the tests first which express the features you wish to implement.  Otherwise, make a first pass at writing the code under test, then proceed to the list below.

1. You can use your IDE to generate at least one test per method in the CUT
   • In IntelliJ, select the class name and press ALT-Enter, then select "Create Test"
   • It is usually useful to have a setUp() method, so check this box
   • Select all the methods
2. Mark each test class with groups = "unit", like so:
   • @Test(groups = "unit")
     publicclassMyClassTest
     {
         ...
     }
3. Name your test "methodUnderTest_Conditions" or "methodUnderTestConditions" or "testMethodUnderTestConditions"
   • Prefer the names from left to right, depending on the PMD/Checkstyle rules in effect
   • This scheme makes it easy to see what test and method failed during test failures
   • Sometimes it is convenient to also append the expected output value to the name, like: equals_WrongClass_False()
4. Write a test for every code path through each method
   • For each method, there should be at least as many tests as the cyclomatic complexity of the MUT
   • If you know that an expression inside the method under test throws, then you should write a test for that scenario
5. Every nullable parameter should have a test for both the null and not-null conditions
   • The expected behavior of the method under a null parameter should be clearly documented in the Javadoc for the method
   • If the above is not true, then add the appropriate Javadoc
   • /**
      * Returns the list of hotel summaries for the IDs provided.  Does not return null.
      *
      * @param hotelIds    the list of hotel IDs for the summaries to fetch--must not be null
      */
     publicList<Summary> getHotelSummaries(List<String> hotelIds);
     // Or possibly:
     /**
      * Returns the list of hotel summaries for the IDs provided, or null if hotelIds is null or no data is available.
      *
      * @param hotelIds    the list of hotel IDs for the summaries to fetch
      */
     publicList<Summary> getHotelSummaries(List<String> hotelIds);
6. If the expected behavior of a test is for the MUT to throw an exception, then use the expected annotation for TestNG:
   • @Test(expected = InvalidArgumenException.class)
     publicvoidtestGetHotelSummariesNullHotelIDsThrows()
     {
         this.retriever.getHotelSummaries(null);
     }
7. If an object is constructed the same way in several tests, hoist it to the test fixture
   • Move the local variable to a field of the test class
   • If the value is immutable, make it static final and initialize it directly
   • Otherwise, initialize it in @BeforeMethod public void setUp()
     • @Test(groups = "unit")
       publicclassHotelRetrieverTest
       {
           privateLodgingContentRetriever contentRetriever;
           privateReviewRetriever reviewRetriever;
           privateHotelRetriever hotelRetriever;
           @BeforeMethod
           publicvoidsetUp()
           {
               this.contentRetriever = mock(LodgingContentRetriever.class);
               when(this.contentRetriever.getContentSummary(any(), any(), any()))
                   .thenReturn(newContentSummary(1234, "Hotel Name", ...);
               this.reviewRetriever = mock(ReviewRetriever.class);
               when(this.reviewRetriever.getReviewSummary
               this.hotelRetriever = newHotelRetriever(this.contentRetriever, this.reviewRetriever);
           }
       }
8. Minimize mocking noise
   • The mocks become much more readable if you use static imports
   • when/then clauses are easier to read when the then clause is wrapped:
     • when(retriever.getHotelSummaries(any()))
           .thenReturn(...);
9. Use good assertions
   • assertEquals() is usually the best assertion, because it is the strongest (most precise)
   • assertSame() is appropriate for testing fluent APIs and normalizing/interning caches
   • Avoid assertNotNull(), which is often the weakest assertion–try to find a reliable property of the result which can be asserted instead
   • If there are no good built-in matchers for your assertion, use Hamcrest's assertThat() and write the appropriate custom matcher
10. Ensure the test can fail
    • Confirm this by first writing the test in a way that fails
      
      • Try to fail the test by only modifying the test input(s) or expected output(s), not the assertions
      • It is the assertion which must fail, not the whole test, per se
    • If an assertion can never fail, then it is a useless NOP, and the test itself provides false coverage
      • // HotelRetriever.java
        publicList<Summary> getHotelSummaries(List<String> hotelIds)
        {
            finalList<Summary> result = newArrayList<>();
            // result is never reassigned
            ...
            returnresult;
        }
        // HotelRetrieverTest.java
        @Test
        publicvoidtestGetHotelSummaries()
        {
            // Useless assert, because it can never fail
            assertNotNull(retriever.getHotelSummaries(HOTEL_IDS));
        }
11. Do not use reflection to violate privacy
    • A common pattern uses Spring's ReflectionTestUtils to make a private field accessible for override in a test– DO NOT DO THIS!!!
    • Instead, make the field an injected dependency using c'tor injection:
      • // Bad
        publicclassHotelRetriever
        {
            privatestaticfinalLogger LOGGER = LoggerFactory.getLogger(HotelRetriever.class);
            ...
        }
        publicvoidtestGetHotelSummaries()
        {
            finalLogger logger = mock(Logger.class);
            ReflectionTestUtils.setField(retriever, "LOGGER", logger);
            ...
            verify(logger).info(containsString("message"));
        }
        // Good
        publicclassHotelRetriever
        {
            privatefinalLogger logger;
            ...
            publicHotelRetriever()
            {
                this(LoggerFactory.getLogger(HotelRetriever.class));
            }
            // Package private for testing
            HotelRetriever(Logger logger)
            {
                this.logger = notNull(logger);
            }
        }
        publicvoidtestGetHotelSummaries()
        {
            finalLogger logger = mock(Logger.class);
            finalHotelRetriever retriever = newHotelRetriever(logger);
            ...
            verify(logger).info(containsString("message"));
        }
    • Or, add a [package private] getter if necessary
12. Test methods which throw only need to declare throws Exception
    • More precise throws clauses are useless because nobody except the test runner should be calling the test method
    • The thrown exceptions should be exercised in separate tests via @Test(expected = ThrownException.class)
13. Use code coverage metrics to show you where tests are missing
    • Code coverage only tells you where you are not done – it does not tell you when you are done !!!
    • It does not guarantee that your tests make strong assertions – only manual inspection can verify this!
    • It does not know whether you have tested boundary conditions – only careful analysis can verify this!

Guidelines

1. Tests must be parallelizable with no synchronization
   • This also implies that the test runner must be able to run them sequentially in any order and succeed
   • Both of the above conditions means that tests may only share immutable state!
   • All mutable state must be constructed at test initialization time (mostly in setUp())
   • Code which relies on 3rd-party singletons should mock the singletons to satisfy this requirement
   • If the CUT is a singleton, it should be implemented in a way which allows multiple instances for testing (usually by providing a package private c'tor)
2. Tests must be repeatable
   • Ideally, all code executed in a test should be observationally deterministic–each step should produce the same output, as far as the caller is concerned
   • This means that non-deterministic elements must be mocked, if possible (this includes the clock/calendar!)
3. Each test should only verify one effect
   • All method calls except the MUT should serve to initialize the test
   • All assertions in the test should only verify the behavior of the single MUT
   • A test may have multiple assertions, but only to determine the shape of a single output
4. Verify state, not behavior
   • Strongly prefer to assertEquals() on the result rather than verify(mock).methodWasCalled()
     • assert when you can, verify when you must
     • Seeing both assert and verify in a test is a smell–you should usually only need one or the other
   • You may need multiple assertions to fully qualify the final state
   • You rarely need more than one verify to qualify the behavior of the MUT
   • By only checking the state, which is the public contract, you allow the implementation to arrive at that state by various means (behaviors)
   • Don't defeat this by verifying the exact path through the MUT–doing so makes the test brittle without adding value
5. Prefer real objects over mocks
   • Strongly prefer to assertEquals() on the result rather than verify(mock).methodWasCalled()
     • assert when you can, verify when you must
     • Seeing both assert and verify in a test is a smell–you should usually only need one or the other
   • You may need multiple assertions to fully qualify the final state
   • You rarely need more than one verify to qualify the behavior of the MUT
   • By only checking the state, which is the public contract, you allow the implementation to arrive at that state by various means (behaviors)
   • Don't defeat this by verifying the exact path through the MUT–doing so makes the test brittle without adding value
6. Prefer real objects over mocks
   • If a class goes out of process (calls a service/database, touches the filesystem, etc.), use a mock
   • If a class has non-deterministic behavior (calls random(), uses the clock, etc.), use a mock
   • If a class has complex construction requirements (you need to build a dozen objects to call the c'tor), use a mock
   • Otherwise, use a real instance of the class
7. Tests should drive the conversion of field/setter injection to c'tor injection
   • C'tor injection is superior because it forces the developer to eliminate circular dependencies
   • C'tor injection allows dependencies to be final
   • Setters provide more public interface which needs to be tested for full code coverage
   • Field injection requires reflection to override in tests
8. Tests must be fast
   • The entire suite of unit tests should take no more than a few seconds to execute per library/module
   • Fast tests facilitate a tight write-test-fix loop; slow tests discourage the execution of tests
   • Do not let a test wait for some elapsed time to occur–the clock should be mocked in the first place, so it should be independently settable