Negative Testing, Notably Injected Streams

09 December 2011

In this post, I'll cover negative testing, which is to make sure we know what error conditions can occur and how to make sure our code can handle them appropriately. Let's say that I want to track our plebs in a registry as just a list of Plebeian POJOs consisting of just a first and last name. I can load my registry from an InputStream of newline-delimited String objects, each of which is itself a comma-delimited last and first names. Constructing the registry should throw either an exception if something went wrong or complete and able to return an unmodifiable list of Plebeian objects.

But first, I'll be up front about this: the Apache Commons projects are freakin' awesome, especially Commons Lang and Commons IO. The reason they're so freakin' awesome is that they drastically reduce the amount of boilerplate code you need to write and add tons of one-liner convenience methods. I highly recommend their usage in your code when allowed not only for ease of use but to reduce the amount of code that needs to be tested. For this post I'll be using the following pieces of Commons functionality:

My Plebeian object is just a POJO with two fields: lastName and firstName. Here's my PlebeianRegistry object:

public class PlebeianRegistry {
    private final List<Plebeian> plebeians = new ArrayList<Plebeian>();
    
    public PlebeianRegistry(final InputStream inputStream)
    throws IOException {
        Validate.notNull(inputStream, "InputStream parameter cannot be null");
        final List<String> lines = IOUtils.readLines(inputStream);
        
        for (final String line : lines) {
            // get max of 2 delimited fields
            final String[] fields = StringUtils.splitPreserveAllTokens(
                    line, ",", 2);
            
            // only honor entries with two fields
            if (fields.length == 2) {
                final String lastName = fields[0];
                final String firstName = fields[1];
                
                plebeians.add(new Plebeian(lastName, firstName));
            }
            
        }

    }
    public List<Plebeian> getPlebeians() {
        return Collections.unmodifiableList(plebeians);
    }
    
}

There are a number of ways that constructing our registry of plebs could go wrong:

  1. The InputStream parameter could be null
  2. Reading from the InputStream could throw an exception
  3. A line of input could have fewer or more than 2 fields

All of these behaviors signal errors that could occur when constructing the registry and because I want to know how I'm handling error conditions, I write negative tests that simulate cases where things do go wrong.

First, what happens when the caller passes null to the PlebeianRegistry constructor? I could just wait til the constructor tries to read from the InputStream and then track down the line of code but if the reading is part of a big ol' line of complex code, it can be tricky to track down exactly what threw the NullPointerException. And what if a bunch of other things happened before the read (such as opening, writing to, and closing a file) that can't easily be rolled back? It's usually best to exit as soon as we know something's wrong. To achieve this, the first thing the constructor does is verify that the InputStream parameter is not null:

Validate.notNull(inputStream, "InputStream parameter cannot be null");

I get a helpful error message stating that the supplied parameter can't be null. This behavior is easily verified with the following test:

@Test(expected=NullPointerException.class)
public void nullInputStreamParameterShouldThrowException()
throws IOException {
    InputStream inputStream = null;
    new PlebeianRegistry(inputStream);   
}

The @Test annotation takes an optional parameter named expected that tells jUnit that I expect a NullPointerException to be thrown when passing a null InputStream to the PlebeianRegistry constructor. Alternatively, if I wanted to check the text of exception message, I could write the test as:

@Test
public void nullInputStreamParameterShouldThrowExceptionWithExpectedMessage()
throws IOException {
    InputStream inputStream = null;
    
    try {
        new PlebeianRegistry(inputStream);
    }
    catch (NullPointerException e) {
        assertThat(e.getMessage(), is("InputStream parameter cannot be null"));
    }
    
}

Second, what happens when reading from the InputStream throws an IOException? This can occur for a variety of reasons, but how I get an InputStream to throw an Exception? By mocking one out via simple extension, of course! In this test, I give myself a little helper class that extends InputStream and throws an IOException whenever the read() method is called:

class ExceptionThrowingInputStream extends InputStream {
    @Override
    public int read() throws IOException {
        throw new IOException("This is a purposeful " +
                "exception for negative testing");
    }
    
}

Now I can just pass in an instance of my exception-throwing InputStream to my PlebeianRegistry constructor and test away:

@Test(expected=IOException.class)
public void badStreamShouldThrowException()
throws IOException {
    InputStream inputStream = new ExceptionThrowingInputStream();
    
    new PlebeianRegistry(inputStream);
    
}

All this test says is that when my InputStream throws an IOException, I expect to see it bubble up to the caller.

Third, what happens when a line from the InputStream contains fewer or more than two fields? The data in the input could have been corrupted for some reason so only the last name appears or perhaps the plebs first name (the second field) contains a comma. To accommodate both conditions, I use the Commons Lang method StringUtils.splitPreserveAllTokens passing the max number of tokens I want, followed up with a check on how many fields I actually got. I only want lines of input containing two fields and ignoring input lines containing one field and can do so with this test:

@Test
public void linesWithLessThan2FieldsShouldBeIgnored()
throws IOException {
    String lineSeparator = System.getProperty("line.separator");
    
    StringBuilder sb = new StringBuilder();
    sb.append("plebs last name").append(lineSeparator);
    
    // use IOUtils to convert the string to an InputStream
    InputStream inputStream = IOUtils.toInputStream(sb.toString());
    
    PlebeianRegistry registry = new PlebeianRegistry(inputStream);
    
    // the registry should be empty
    assertThat(registry.getPlebeians().isEmpty(), is(true));
    
}

The registry was successfully constructed by is empty since the only line of input contained a single field and was ignored. Conversely, the plebs first name may actually contain a comma that we want to preserve:

@Test
public void linesWithMoreThan1CommaShouldTreatSecondAsPartOfFieldText()
throws IOException {
    String lineSeparator = System.getProperty("line.separator");
    
    StringBuilder sb = new StringBuilder();
    sb.append("harris,bill,schmoe").append(lineSeparator);
    
    // use IOUtils to convert the string to an InputStream
    InputStream inputStream = IOUtils.toInputStream(sb.toString());
    
    PlebeianRegistry registry = new PlebeianRegistry(inputStream);
    
    // there should be 1 Plebeian object in the registry
    assertThat(registry.getPlebeians().size(), is(1));
    
    Plebeian plebeian = registry.getPlebeians().get(0);
    assertThat(plebeian.getLastName(), is("harris"));
    assertThat(plebeian.getFirstName(), is("bill,schmoe"));
    
}

In the last line the test verifies that constructor preserved the second comma as part of the plebs first name.

For good measure, I threw in some extra tests that aren't necessarily negative tests but verify certain behaviors that I haven't covered in other posts to date:

Testing the first is simply a matter of noting that in the Java API for read(), -1 should be returned when the end of stream is reached:

@Test
public void linesWith2FieldsShouldBeHonoredAndInputFullyRead()
throws IOException {
    String lineSeparator = System.getProperty("line.separator");
    
    StringBuilder sb = new StringBuilder();
    sb.append("plebovitz,joe").append(lineSeparator);
    sb.append("plebington,sam").append(lineSeparator);
    
    // use IOUtils to convert the string to an InputStream
    InputStream inputStream = IOUtils.toInputStream(sb.toString());
    
    PlebeianRegistry registry = new PlebeianRegistry(inputStream);
    
    // first assert that the entire stream has been read
    // -1 is returned if there's nothing left to read
    assertThat(inputStream.read(), is(-1));
    
    // there should be 2 Plebeian objects in the registry
    assertThat(registry.getPlebeians().size(), is(2));
    
}

The second behavior is that I don't want a caller constructing a PlebeianRegistry to be able to modify the list, such as adding or removing Plebeian objects or even clearing the entire list. To do so, note the getter for the list:

public List<Plebeian> getPlebeians() {
    return Collections.unmodifiableList(plebeians);
}

Collections.unmodifiableList is a simple wrapped provided by Java itself that throws UnsupportedOperationExceptions whenever an attempt is made to modify the list. Note that it doesn't stop someone from modifying the objects within the list, only that the list itself cannot be added to or removed from. Testing that the list returned is not modifiable is easy with this test:

@Test(expected=UnsupportedOperationException.class)
public void listOfPlebeiansShouldNotBeModifiable()
throws IOException {
    String lineSeparator = System.getProperty("line.separator");
    
    StringBuilder sb = new StringBuilder();
    sb.append("pleb,joe").append(lineSeparator);
    
    // use IOUtils to convert the string to an InputStream
    InputStream inputStream = IOUtils.toInputStream(sb.toString());
    
    PlebeianRegistry registry = new PlebeianRegistry(inputStream);
    
    registry.getPlebeians().clear();
    
}

An effect of the expected annotation is that the test only says that an UnsupportedOperationException is thrown somewhere in the code, not exactly which line threw it. If I wanted to be absolutely certain of the line that the UnsupportedOperationException was thrown, I'd have to write the test using fail():

@Test
public void listOfPlebeiansShouldNotBeModifiable2()
throws IOException {
    String lineSeparator = System.getProperty("line.separator");
    
    StringBuilder sb = new StringBuilder();
    sb.append("pleb,joe").append(lineSeparator);
    
    // use IOUtils to convert the string to an InputStream
    InputStream inputStream = IOUtils.toInputStream(sb.toString());
    
    PlebeianRegistry registry = new PlebeianRegistry(inputStream);
    
    try {
        registry.getPlebeians().clear();
        fail("An UnsupportedOperationException should've been thrown");
    }
    catch (UnsupportedOperationException e) {}
    
}


Surrounding the call to clear() in a try-catch block allows the UnsupportedOperationException to be caught and handled before the fail() has a chance to fail the test.

Negative testing is a very aspect part of test-driven development and can get dramatically more involved than these simple examples but doing so can get you closer to 100% coverage and very important to verifying business requirements.