The modern developer: Understanding TCP

A couple of years ago I am not sure when interviewing candidates that I would have delved into their knowledge of the TCP protocol.

However I think I've changed my mind, why?

Reason one: The Cloud. The marketer tells us that the cloud means that we don't need to care where our servers are or where our applications are deployed. But for the developer I think the opposite is true. Suddenly rather than internal high speed network links our applications are being deployed onto commodity hardware with shoddy network links that regularly go down.

Reason two: Micro-service architecture. As we split up our applications into small components we're also adding to the number of integration points, most of those integration points will be over TCP.
So what should a well rounded developer know about their system and its dependencies:

1. For any calls outside of their application's process:
1. What is the underlying protocol?
2. What is the connection timeout for that protocol?
3. What is the read timeout for that protocol?
4. What is the write timeout for that protocol?
2. Are there any firewalls between your application and its dependencies?
3. Does the traffic sent between your applications, or your application and its dependencies go over the Internet?
4. What happens if a dependency responds slower than usual at the application level?
5. What happens if a dependency responds slower, or not at all, at the protocol level?

And I think the best way to know the answer to these question is to test these scenarios. And to do that they need a good understanding of how TCP works and when it can go wrong. They need to be able to use tools like tcpdump, wireshark and netcat. How many "Java developers" do you think fall into this category and would test these scenarios? 

How many would say: Well I call a Java method that does the connection, what do I care?

As soon as you remember that the people writing these libraries are just as human as you, you might have second thoughts about not testing the "too edge-case". Most people are surprised to realise that a lot of libraries just use the operating system's timeout value for TCP connections, which is usually measured in minutes, not seconds. How do you explain to your users why your application hung for 10 minutes?

I no longer see any of the above scenarios as edge cases, it just might take a few weeks in production for each of them to happen.

Overriding Spring Boot HTTP status codes for health pages

One thing I found a little baffling when I started using spring boot was that the health pages returned 200 regardless of your 'Status'. So 'DOWN' would return a 200 with the actual information in a JSON payload. This means any monitoring system needs to parse the JSON rather than just know based on the HTTP status code.

Finally for Spring Boot 1.1 you can override this! For example if you want DOWN to be a 500, then add the following to your application properties file.

endpoints.health.mapping[DOWN]=INTERNAL_SERVER_ERROR

Using Stubbed Cassandra's new JUnit rule

In my first post about unit testing Java applications that use Cassandra you had to start Scassandra before your test, reset it between tests and finally stop it before it after your tests. Since then I've accepted a pull request which hides all this away in a JUnit rule.

So the old code in your tests looked like this for starting Scassandra:


This for stopping Scassandra:


And this for resetting Scassandra between tests:


Now you can simply do this:


Assigning it to a ClassRule means that it will start Scassandra before all your tests and stop it after all your tests. Then assigning it to a regular Rule means that the Activity client and Priming client will be reset between tests.

Unit testing Java Cassandra applications

I often get asked what is the best method for unit and and integration testing Java code that communicates with Cassandra.

Here are what I think the options are:

1. Mocking libraries: Use mocking libraries such as Mockito to mock out the driver you are using
2. Cassandra Unit
3. Integration test the class in question with against a real Cassandra instance running locally
4. Stubbed Cassandra (disclaimer: I made this)

Lets take these one by one and look at their respective advantages and disadvantages. The things to consider are:

• Speed of the tests
• Able to run the tests concurrently and how easy this is to achieve
• Able to test everything? Even failures?
• Are the tests brittle?
• Readability of the tests
• Requirements on environment e.g do you need a Cassandra instance installed?
• Confidence it will work against a real Cassandra
• Subjective nonsense by writer of this blog

1 Mock the library

You can use a combination of factories and mocking to stop the driver actually interacting with Cassandra. Then verify your code's interactions with these mocks.

Advantages:

• Fast - no I/O
• Execute tests concurrently
• No Cassandra instance required on machine your code compile runs
• Can test everything including the various failures as you can mock the library to throw ReadTimeout exceptions etc

Disadvantages:

• You may mock the driver to behave in a different way to how it will behave 
• Very hard to understand tests due to large amount of boiler plate mocking code
• Very brittle tests. Change your driver, change your tests! Change from a query to a prepared statement, change your tests!
• A lot of boiler plate. Take the Datstax driver for example, it returns a ResultSet, which is iterable, fancy writing the code that mocks it returning many results?

Conclusion:

• Don't do it if you wish to remain sane

2 Cassandra Unit

Cassandra Unit is a tool for starting an embedded Cassandra in the JVM your tests are running. It also has a great API for ingesting data into Cassandra for your tests.

Advantages:

• Pretty fast. It is all in process.
• Can run tests concurrently if they use different keyspaces and none of the tests turn it off etc
• Can use CQL to load data as it is a real Cassandra. I think this leads to readable tests
• No Cassandra required on machine. Can use the it via a Maven dependency
• High confidence your code will work against a real Cassandra 

Disadvantages:

• Unable to test failure scenarios. What is a read time out when there is only one node running in the same JVM as your test?
• No way to verify consistency of queries

Conclusion:

• Very useful tool for in process happy path tests

3 Integration style tests using a real Cassandra

This is something I've done a lot of recently. Every dev machine at my current work place has a Cassandra instance running (using the awesome tool ccm). Then we test our DAOs by assuming it is there and doing testing in a dynamic keyspace.

Advantages:

• Can run tests concurrently if they use different keyspaces and none of the tests turn it off etc
• Tests aren't brittle, can change from queries to prepared statements, or the queries involved without changing tests
• Readable tests - all data setup is done in CQL
• Very high confidence it will work against a real Cassandra as the test is against a  real Cassandra :)

Disadvantages:

• Probably the slowest option. But it is still millisecond quick.
• Hard to test scenarios other than turning the node off. This then makes the tests slow.
• Cassandra required to build and run tests

Conclusion:

• Slightly slower but very useful for testing happy paths
• Very similar to using Cassandra unit

4 Stubbed Cassandra

Stubbed Cassandra is a new open source tool that pretends to be Cassandra and can be primed to returns rows, read timeouts, write timeouts and unavailable errors. It can be used via a maven dependency or as a standalone server.

Advantages:

• Very fast
• Can test all types of errors and be confident in what the driver does as the driver thinks it is a real Cassandra
• Can run many instances inside the same JVM listening on different binary ports. So can run tests concurrently with no extra effort e.g no requirement to use different keyspaces
• Tests less brittle than mocking the driver. Can change driver without changing test but if you change queries you need to update your priming
• No requirement to have a real Cassandra. Just brought in by a maven dependency

Disadvantages:

• Slightly more brittle than a real Cassandra/Cassandra unit. Requires priming on the query, priming for each prepared statement
• Slightly less confidence it will work against a real Cassandra as it isn't a real Cassandra. But more confidence than mocking
• It is new and does not support all of Cassandra's features, so if you use a feature that Scassandra doesn't support you are stuck!

Conclusion:

• Best solution for all error case testing
• Best solution if you need to execute tests concurrently

Introducing Scassandra: Testing Cassandra with ease

Scassandra (Stubbed Cassandra) is a new open source tool for testing applications that interact with Cassandra.

It is intended to be used for:

• Unit testing DAOs that interact with Cassandra
• Acceptance testing applications that interact with Cassandra

The first release is primarily aimed at Java developers. Subsequent releases will be aimed at people writing black box acceptance tests.

It works by implementing the server side of the CQL binary protocol, meaning that Cassandra drivers, such as the the Datastax Java Driver, believe it to be a real Cassandra.

The original motivation for developing Scassandra was to test edge case / failure scenarios. However it quickly became apparently it is just as useful with regular happy path unit tests.

So how does it work?

When you start Scassandra it opens two ports:

• Binary port: this is the port you'll configure your application with rather than the binary or a real Cassandra. 
• Admin port: this is for Scassandra's REST API that provides priming and retrieving a list of executed queries. 

By default, when your application connects to Scassandra and executes queries Scassandra will respond with a result with zero rows.

Then via Scassandra Java Client, and later the REST API, you can prime Scassandra to return rows (where you can specify the column types and values), read request time outs, write request time outs and unavailable exceptions.

After you've run your tests you can verify the queries and prepared statements that your application has executed. Including the consistency the queries have been executed with. So if you have requirements to execute certain writes at a high consistency but other queries can be at a lower consistency this can be tested via black box acceptance tests.

The benefits of Scassandra over testing against a real Cassandra instance:

• Test failures deterministically: where previously you would need to have a multi node cluster with the ability to bring down nodes. 
• Test the consistency of queries. This has come up at my workplace where a requirement was that for most queries we can downgrade consistency when there are failures but for certain important writes they had to be executed at QUORUM.
• Have fast running DAO tests that don't require mocking out driver classes or a real Cassandra running.

So how do I use Scassandra?

The first release of Scassandra is aimed at Java developers. Scassandra comes in two parts:

• Scassandra server. This is a Scala application that has been put in Maven central with a pom that will bring in its transitive dependencies.
• Scassandra Java client. A thin wrapper written in Java to make using Scassandra from Java tests easy. This has methods to start/stop Scassandra and classes that prime / retrieve the list of executed queries.

For the first release it is expected that Scassandra will only be used via the Java client and no one will use it as a standalone executeable or interact with the REST API directly.

To get started with the Scassandra Java client then go here. Or checkout the example project here.

If you aren't using Java or a language that can easily use the Java client then the next release will be for you where we'll build a standalone executable and from that release on we'll make the REST API backward compatible as we'll expect people to use it directly.

It is all open source on github and you can find Scasandra server here.

The Java client is here.

And all the details of the REST API e.g how to prime are on the Scassandra sever website here.

Using Stubbed Cassandra: Unit testing Java applications

My first article on Scassandra introduced what it is and why I've made it.

This article describes how to use Scassanda to help unit test a Java class that stores and retrieves data from Cassandra.

It assumes you're using a tool that can download dependencies from maven central e.g Maven, Gradle or SBT.

First add Scassandra as a dependency. It is in maven central so you can add it to your pom with the following xml:

<dependency>
  <groupId>org.scassandra</groupId>
  <artifactId>java-client</artifactId>
  <version>0.2.1</version>
</dependency>

Or the following entry in your build.gradle:

dependencies {
    compile('org.scassandra:java-client:0.2.1')
}

There are four important classes you'll deal with from Java:

• ScassandraFactory - used to create instances of Scassandra
• Scassandra - interface for starting/stopping Scassandra and getting hold of a PrimingClient and an ActivityClient
• PrimingClient - sends priming requests to Scassandra RESTful admin interface
• ActivityClient - retrieves all the recorded queries and prepared statements from the Scassandra RESTful admin interface

The PrimingClient and ActivityClient have been created to ease integration for Java developers. Otherwise you would need to construct JSON and send it over HTTP to Scassandra.

You can start a Scassandra instance per unit test and clear all primes and recorded activity between tests.

To start Scassandra before your test starts add a BeforeClass e.g:


You can also add a AfterClass to close Scassandra down:

Now that you have Scassandra running lets write a test. Perhaps you want to test a simple Java DAO that connects to Cassandra and executes a query.

And you have a backing table like:

CREATE TABLE person (
  id int,
  first_name text,
  PRIMARY KEY (id)
)

Lets TDD the DAO using Scassandra starting with our connect method:

Lets look at what this code is doing:

• Line 4: Informs the activity client to clear all recorded connections. This is to stop other tests that have caused connections interfering with this one. 
• Line 6: We call on connect on our PersonDao.
• Line 8: We call retrieveConnections on the activity client and expect there to be at least one. The Java Datastax driver makes multiple connections on startup so you can't assert for this to be 1.

This fails with the following message:

java.lang.AssertionError: Expected at least one connection to Cassandra on connect

at org.junit.Assert.fail(Assert.java:88)

at org.junit.Assert.assertTrue(Assert.java:41)

at com.batey.examples.scassandra.PersonDaoTest.shouldConnectToCassandraWhenConnectCalled(PersonDaoTest.java:94)

Now lets write some code to make it pass:

Now lets test the retrieveNames function gets all the first_names out of the person table.

This will prime Scassandra to return a single row with the column first_name with the value Chris. We expect our DAO to turn that into a List of strings containing Chris. To make this pass we need to execute a query and convert the ResultSet, something like this:

Next lets say you have the requirement that you really must not get an out of date list of names. So you want to test that the consistency you do the query is QUORUM. You can test this like this:

Lets look at what each line is doing:

• Line 4 builds the expected query, note the consistency is also set. If you build a Query without a consistency it defaults to ONE.
• Line 7 clears all the recorded activity so that another test does not interfere with this one. It also clears the queries that were executed as part of connect (the Datastax Java driver issues quite a few queries on the system keyspace on startup)
• Line 11 retrieves all the queries your application has execited
• Line 12 verifies the expected query that was built on Line 4 has been executed

This will fail with an error message like this:

java.lang.AssertionError: Expected query with consistency QUORUM, found following queries: [ {Query{query='select * from people', consistency='ONE'}]

We can make this pass by adding the consistency to our query:

And we're done!

This has been a brief instruction to Scassandra but hopefully the above gives you an idea of how Scassandra can be used to test your Java applications that use Cassandra. We've covered:

• Priming basic queries
• Verifying queries
• Verifying connections

Future blog posts will show you how to:

• Prime prepared statements
• Prime different column types in responses
• Prime error cases

Scassandra has only just been released. The future road map includes:

• JUnit rule so you don't need to handle starting/stopping and clearing recorded activity
• More generic priming e.g any query on this table
• Support for more drivers

All the code for this example can be found in full here.

Testing your RESTful services with Groovy, Spock and HTTPBuilder

I like to test the HTTP portion of the RESTful web services I build in isolation. This is because, in the languages I use (Java, Scala), most of the code responsible for taking in HTTP requests and parsing them is very framework specific and simply delegates the actual logic to another part of my application. Ideally I can launch this part of my application in the same way as it is launched in production but with the rest of the application stubbed out.

Recently I've been using Groovy/Spock for these integration tests, here's how for a really simple web service. In this example it's been built with Spring Boot.

Let's not go into the details of how Spring boot work but it is essentially the same application as in Spring's guide.

The web application has a single path: /exampleendpoint which takes in a single query param: input and returns a JSON object with a single field payload with the value: Something really important: with the input appended.

I want to test that the web service takes in HTTP, takes the query parameter out and builds the JSON correctly. With Groovy/Spock/HTTPBuilder here's how it looks:

Let's look at what is going on here:

• In the setup phase we create a RESTClient
• In the exercising phase we make a call out to the web application.
• In the verification phase we check that the response code is 200, the content type is "application/json" and the body is JSON with a single payload field that contains "Something really important: Get a hair cut"

Benefits over writing this in Groovy/Spock over say just using Java/JUnit:

• The HTTP libraries are easier to use with less code
• The assertions are more expressive
• More magic, for instance the second with() block works on a Groovy map that has automagically created

The above example doesn't deal with how to start and stop the web service under test, I left that out as it is very web framework specific. 

What do you need in your build to get this setup? I used Gradle for this example so it is probably easier to just check out the build.gradle, a lot of it can be ignored as it is related to Spring Boot rather than Spock. Here are the important bits:

The plugins:

And the dependencies:

The entire project is here. Happy testing.