Behaviour driven testing with Cucumber-JVM and Stubbed Cassandra

So you're tasked with building a service that facades your company's legacy user management system. You need to do this as your part of the company needs to handle the legacy system going down.

To add resiliency you are required to save a user's information in case the legacy system is down. You need to be able to handle restarts so this cache will need to be persistent.

You're going to use Cassandra as your persistence, so how do test this? You sit down your your analyst and QA and come up with the following feature:

	Feature: Retrieve User Information
	Scenario: User information is retrieved from the company wide repository
	Given The company wide user repository is available
	When User information is requested
	Then The user information is returned
	Scenario: Company wide information store is down
	Given The company wide user repository is unavailable
	When User information is requested
	Then Unable to retrieve user information is returned
	Scenario: Company wide information store is down but the information has been saved previously
	Given The user had been retrieved previously
	And The company wide user repository is unavailable
	When User information is requested
	Then The user information is returned

view raw retrieve_service.feature hosted with ❤ by GitHub

How would you implement these features? Assuming that the legacy system is a HTTP service you can use Wiremock to mock it being up and down.

For example here is how to mock the legacy system being up with Wiremock:

	@Given("^The company wide user repository is available$")
	public void setupUserRepository() throws Throwable {
	givenThat(get(urlEqualTo("/user/" + id)).willReturn(
	aResponse().withBody("{\"firstName\":\"Chris\",\"lastName\":\"Batey\"}")
	));
	}

view raw Stepdefs.java hosted with ❤ by GitHub

And an example of it being down:

	@Given("^The company wide user repository is unavailable$")
	public void userServiceDown() throws Throwable {
	givenThat(get(urlEqualTo("/user/" + id)).willReturn(
	aResponse().withStatus(500)
	));
	}

view raw Stepdefs.java hosted with ❤ by GitHub

So the next requirement is that Cassandra being down doesn't make your service fail i.e.

	Scenario: Company wide information store is up but data store down
	Given The data store is down
	When User information is requested
	Then The user information is returned
	Scenario: Data store being slow should not cause slow down user transactions
	Given The data store has problems writing
	When User information is requested
	Then The user information is returned

view raw retrieve_service.feature hosted with ❤ by GitHub

The first one you could stop Cassandra, perhaps using the great tool CCM. However this is slow, and you need to write code to make sure it is back up/down, all of this in a different process. And how about the next test? How do we make Cassandra return the result slowly? Or produce a Write Timeout Exception?

That is where Stubbed Cassandra comes in handy. To get is started you can add some code like this to start it before the tests and close it after tests:

	private final static Scassandra SCASSANDRA = ScassandraFactory.createServer();
	...
	@Before
	public void beforeScenario() {
	SCASSANDRA.start();
	}
	...
	@After
	public void after() {
	SCASSANDRA.stop();
	}

view raw Stepdefs.java hosted with ❤ by GitHub

Now implementing the step definition to mimic Cassandra being down is as easy as:

	@Given("^The data store is down$")
	public void cassandraDown() throws Throwable {
	SCASSANDRA.stop();
	}

view raw Stepdefs.java hosted with ❤ by GitHub

Which is a lot quicker than turning off a real Cassandra, and starting it back up in the @Before is also very quick.

Now to mimic Write time outs in Cassandra:

	@Given("^The data store has problems writing$")
	public void cassandraWriteTimeouts() throws Throwable {
	PrimingRequest existingUser = PrimingRequest.queryBuilder()
	.withQuery("insert into users(id, firstName, lastName) values ('chbatey','Chris','Batey')")
	.withResult(PrimingRequest.Result.write_request_timeout)
	.build();
	SCASSANDRA.primingClient().primeQuery(existingUser);
	}

view raw Stepdefs.java hosted with ❤ by GitHub

Very similar things could be done for read timeouts and unavailable exceptions.

This article gave you an insight to how you can behaviour drive features relating to Cassandra being down. The full code and running tests are here. Full information on how to use Stubbed Cassandra is here, you'll probably want to documentation for the Java client for Scassandra which is here.

The modern developer: Understanding TCP

A couple of years ago I am not sure when interviewing candidates that I would have delved into their knowledge of the TCP protocol.

However I think I've changed my mind, why?

Reason one: The Cloud. The marketer tells us that the cloud means that we don't need to care where our servers are or where our applications are deployed. But for the developer I think the opposite is true. Suddenly rather than internal high speed network links our applications are being deployed onto commodity hardware with shoddy network links that regularly go down.

Reason two: Micro-service architecture. As we split up our applications into small components we're also adding to the number of integration points, most of those integration points will be over TCP.
So what should a well rounded developer know about their system and its dependencies:

1. For any calls outside of their application's process:
1. What is the underlying protocol?
2. What is the connection timeout for that protocol?
3. What is the read timeout for that protocol?
4. What is the write timeout for that protocol?
2. Are there any firewalls between your application and its dependencies?
3. Does the traffic sent between your applications, or your application and its dependencies go over the Internet?
4. What happens if a dependency responds slower than usual at the application level?
5. What happens if a dependency responds slower, or not at all, at the protocol level?

And I think the best way to know the answer to these question is to test these scenarios. And to do that they need a good understanding of how TCP works and when it can go wrong. They need to be able to use tools like tcpdump, wireshark and netcat. How many "Java developers" do you think fall into this category and would test these scenarios? 

How many would say: Well I call a Java method that does the connection, what do I care?

As soon as you remember that the people writing these libraries are just as human as you, you might have second thoughts about not testing the "too edge-case". Most people are surprised to realise that a lot of libraries just use the operating system's timeout value for TCP connections, which is usually measured in minutes, not seconds. How do you explain to your users why your application hung for 10 minutes?

I no longer see any of the above scenarios as edge cases, it just might take a few weeks in production for each of them to happen.

Overriding Spring Boot HTTP status codes for health pages

One thing I found a little baffling when I started using spring boot was that the health pages returned 200 regardless of your 'Status'. So 'DOWN' would return a 200 with the actual information in a JSON payload. This means any monitoring system needs to parse the JSON rather than just know based on the HTTP status code.

Finally for Spring Boot 1.1 you can override this! For example if you want DOWN to be a 500, then add the following to your application properties file.

endpoints.health.mapping[DOWN]=INTERNAL_SERVER_ERROR

Using Stubbed Cassandra's new JUnit rule

In my first post about unit testing Java applications that use Cassandra you had to start Scassandra before your test, reset it between tests and finally stop it before it after your tests. Since then I've accepted a pull request which hides all this away in a JUnit rule.

So the old code in your tests looked like this for starting Scassandra:

	private static PrimingClient primingClient;
	private static ActivityClient activityClient;
	private static Scassandra scassandra;
	@BeforeClass
	public static void startScassandraServer() throws Exception {
	scassandra = ScassandraFactory.createServer();
	scassandra.start();
	primingClient = scassandra.primingClient();
	activityClient = scassandra.activityClient();
	}

view raw ExampleDaoTest.java hosted with ❤ by GitHub

This for stopping Scassandra:

	@AfterClass
	public static void shutdown() {
	scassandra.stop();
	}

view raw ExampleDaoTest.java hosted with ❤ by GitHub

And this for resetting Scassandra between tests:

	@Before
	public void setup() {
	activityClient.clearAllRecordedActivity();
	primingClient.clearAllPrimes();
	}

view raw PersonDaoTest.java hosted with ❤ by GitHub

Now you can simply do this:

	@ClassRule
	public static final ScassandraServerRule scassandra = new ScassandraServerRule();
	@Rule
	public final ScassandraServerRule resetScassandra = scassandra;
	private static PrimingClient primingClient = scassandra.primingClient();
	private static ActivityClient activityClient = scassandra.activityClient();

view raw PersonDaoTest.java hosted with ❤ by GitHub

Assigning it to a ClassRule means that it will start Scassandra before all your tests and stop it after all your tests. Then assigning it to a regular Rule means that the Activity client and Priming client will be reset between tests.

Unit testing Java Cassandra applications

I often get asked what is the best method for unit and and integration testing Java code that communicates with Cassandra.

Here are what I think the options are:

1. Mocking libraries: Use mocking libraries such as Mockito to mock out the driver you are using
2. Cassandra Unit
3. Integration test the class in question with against a real Cassandra instance running locally
4. Stubbed Cassandra (disclaimer: I made this)

Lets take these one by one and look at their respective advantages and disadvantages. The things to consider are:

• Speed of the tests
• Able to run the tests concurrently and how easy this is to achieve
• Able to test everything? Even failures?
• Are the tests brittle?
• Readability of the tests
• Requirements on environment e.g do you need a Cassandra instance installed?
• Confidence it will work against a real Cassandra
• Subjective nonsense by writer of this blog

1 Mock the library

You can use a combination of factories and mocking to stop the driver actually interacting with Cassandra. Then verify your code's interactions with these mocks.

Advantages:

• Fast - no I/O
• Execute tests concurrently
• No Cassandra instance required on machine your code compile runs
• Can test everything including the various failures as you can mock the library to throw ReadTimeout exceptions etc

Disadvantages:

• You may mock the driver to behave in a different way to how it will behave 
• Very hard to understand tests due to large amount of boiler plate mocking code
• Very brittle tests. Change your driver, change your tests! Change from a query to a prepared statement, change your tests!
• A lot of boiler plate. Take the Datstax driver for example, it returns a ResultSet, which is iterable, fancy writing the code that mocks it returning many results?

Conclusion:

• Don't do it if you wish to remain sane

2 Cassandra Unit

Cassandra Unit is a tool for starting an embedded Cassandra in the JVM your tests are running. It also has a great API for ingesting data into Cassandra for your tests.

Advantages:

• Pretty fast. It is all in process.
• Can run tests concurrently if they use different keyspaces and none of the tests turn it off etc
• Can use CQL to load data as it is a real Cassandra. I think this leads to readable tests
• No Cassandra required on machine. Can use the it via a Maven dependency
• High confidence your code will work against a real Cassandra 

Disadvantages:

• Unable to test failure scenarios. What is a read time out when there is only one node running in the same JVM as your test?
• No way to verify consistency of queries

Conclusion:

• Very useful tool for in process happy path tests

3 Integration style tests using a real Cassandra

This is something I've done a lot of recently. Every dev machine at my current work place has a Cassandra instance running (using the awesome tool ccm). Then we test our DAOs by assuming it is there and doing testing in a dynamic keyspace.

Advantages:

• Can run tests concurrently if they use different keyspaces and none of the tests turn it off etc
• Tests aren't brittle, can change from queries to prepared statements, or the queries involved without changing tests
• Readable tests - all data setup is done in CQL
• Very high confidence it will work against a real Cassandra as the test is against a  real Cassandra :)

Disadvantages:

• Probably the slowest option. But it is still millisecond quick.
• Hard to test scenarios other than turning the node off. This then makes the tests slow.
• Cassandra required to build and run tests

Conclusion:

• Slightly slower but very useful for testing happy paths
• Very similar to using Cassandra unit

4 Stubbed Cassandra

Stubbed Cassandra is a new open source tool that pretends to be Cassandra and can be primed to returns rows, read timeouts, write timeouts and unavailable errors. It can be used via a maven dependency or as a standalone server.

Advantages:

• Very fast
• Can test all types of errors and be confident in what the driver does as the driver thinks it is a real Cassandra
• Can run many instances inside the same JVM listening on different binary ports. So can run tests concurrently with no extra effort e.g no requirement to use different keyspaces
• Tests less brittle than mocking the driver. Can change driver without changing test but if you change queries you need to update your priming
• No requirement to have a real Cassandra. Just brought in by a maven dependency

Disadvantages:

• Slightly more brittle than a real Cassandra/Cassandra unit. Requires priming on the query, priming for each prepared statement
• Slightly less confidence it will work against a real Cassandra as it isn't a real Cassandra. But more confidence than mocking
• It is new and does not support all of Cassandra's features, so if you use a feature that Scassandra doesn't support you are stuck!

Conclusion:

• Best solution for all error case testing
• Best solution if you need to execute tests concurrently