Cassandra vs MongoDB: The basics

Cassandra and MongoDB are two of the more popular NoSQL databases. I've been using Cassandra extensively over the past 6 months and I've recently started using MongoDB. Here is a brief description of the two, I'll follow up this post with a deeper comparison of the more advanced features.

Cassandra

Cassandra was originally created by Facebook and is written in Java, however it is now a Apache project. Traditionally Cassandra can be thought of as a column orientated database or a row orientated database depending on how you use columns. Each row is uniquely identified by a row key, like a primary key in a relational database. Unlike a relational database each row can have a different set of columns and it is common to use both the column name and the column value to store data. Rows are contained bya  column family which can be thought of as a table.

Client's use the thrift transport protocol and queries look like:

set Person['chbatey']['fname'] = 'Chris Batey';

Where Person is the column family, chbatey is the row key, fname is the column name and "Chris Batey" is the column value. Column names are dynamic so a client can store any key/value pairs. In this sense Cassandra is quite schemaless.

Then came Cassandra 1.* and CQL 3. Cassandra Query Language (CQL) is a SQL like language for Cassandra. Suddenly Cassandra, from a client's perspective, become much more like a relational database. Queries now look like this:

insert into Person(fname) values ('chbatey')

Using CQL3 there are no more dynamic column names and you create tables rather than column families (however the map type basically gives the same functionality). It's all still column families under the covers, CQL3 is just a very nice abstraction (a simplification). 

Cassandra appears to be moving away from a thrift protocol and moving to a proprietary protocol referred to as a native protocol. 

Overall Cassandra is quite a "rough around the edges" database to use (less so with CQL3) from a client perspective. It's real power comes from its horizontal scalability and tuneable eventual consistency. More on this in a future post.

MongoDB

MongoDB is a document database written in C++. Document databases are very intuitive as you simply store and retrieve documents! No crazy data model to learn, for MongoDB you simply store and retrieve JSON (BSON) objects.

Storing looks like this:

db.people.save({_id: 'chbatey', fname:'Chris Batey'})

Retrieving looks like this:

db.people.find({_id: 'chbatey'})

Simple!

MongoDB has a very rich JSON based querying language and a fantastic aggregation framework. From a client's perspective MongoDB is a vastly more featured database with support for ad-hoc querying (Cassandra you must index everything you want to search by). 

Conclusion

This post was a very brief description of Cassandra and MongoDB. In future posts I will compare:

• Fault tolerance - replication
• Read and write consistency
• Clients
• Hadoop support

Particularly for Cassandra it is very important how your data centre and Cassandra cluster are laid out as to which read and write consistency levels you need to get the desired behaviour. 

21st July 2013: What's happening?

I am always looking to improve as a Software Engineer. To keep track of what I'm working on I've broken it down to the following categories:

• Languages: My day job is primarily Java so I like to use other languages for everything else.
• Frameworks: Usually tightly coupled to a language but becoming less so - especially for JVM based languages.
• Databases: The world is changing. No longer can you get away with relational database / SQL knowledge
• Craftsmanship: How do I go about producing better, more maintainable software as well as helping those around me to do the same.
• General knowledge: Keeping up with technological world takes some doing. I try to read a few articles a day and listen to podcasts.

I won't work on each category every week. Here's what I've been doing the last week:

Languages

At work I'm a complete Java head. Over the past few years I've primarily developed standalone multi-threaded server applications for financial companies. More recently I've been developing cloud based applications so been doing a lot more Java development where it is deployed to a container e.g tomcat.

For this reason, when not at work, I am completely avoiding Java. This week I've been learning to test Java applications using Groovy (ok ok so I didn't leave Java complete behind!) and been learning to unit test the logic in Gradle scripts using GroovyTest.

In addition to Groovy I've been working on Python this week. If you live in London you might be aware you can get Transport for London to send you your travel statements in CSV format. I've been writing a Python application to parse these and work out how much money I spend commuting to work. Blog posts coming about this but initial version on github: https://github.com/chbatey/oystergrep

Databases

Having worked with Cassandra a lot over the last six months I'm now exploring MongoDB. Leaving the relational world for the NoSQL world has been a great learning experience this year. I'll put up a comparison for Cassandra vs MongoDB soon. Cassandra is such a low level, developer must understand everything database so MongoDB is quite refreshing!

Craftsmanship

I've started doing katas again the last few weeks. I've started with sorting algorithms. I'm doing this quite quickly and in Python to further solidify my knowledge of the language. Here's merge sort: https://github.com/chbatey/kata_mergesort quicksort coming!

General Knowledge

Started going through the backlog of programming throwdown last few weeks: http://programmingthrowdown.blogspot.co.uk/. Not bad listening for the train, though I wish they spoke about games less!

Mergesort in Python

My train was cancelled today and as I am trying to cement my knowledge of python  I decided to do mergesort in python from memory. I find when adding new languages to my toolkit It is easy to forget how to setup a quick project with unit tests so I find it useful to do regular little projects from scratch Here's the code:


 And hear's the unit tests (of course the unit tests came first!):

I really like python and its unit testing framework. So simple to get going and for doing TDD.

Uncle Bob: Automated acceptance tests

Yesterday I went to see Uncle Bob at the Skills Matter Exchange in London. Having read and enjoyed Clean Code and Clean Coder it was great to see Uncle Bob in the flesh.

The talk was on automated acceptance tests. Such a simple topic - we all automate our acceptance tests don't we?

A few points I took away from the talk:

• Can we get our stakeholders to write our acceptance tests? If not is it at least business analysts or QAs? If it is developers you're in trouble!
• Developers are brilliant at rationalising about concepts such as "is it done?". Don't trust a developer to tell you something is done!
• Acceptance tests should be automated at the latest half way through an iteration if your QAs are going to have time to do exploratory testing
• The QAs should be the smartest people in your team. They should be specifying the system with the business analyst not verifying it after it has been developed
• Your functional tests are just another component of the system. If that part of brittle it means your project is badly designed! Go back to the drawing board.

A final point that stuck with me is that acceptance tests don't need to be black box tests. The language they are written in should be high level (it was your stake holder who wrote it right??). But the implementation could interact with a version of your system that has the database or HTTP layer mocked out. Think of it this way:

• How many times do you need to test the login functionality of your application? Once!
• How many times will you test it if all your tests go through the GUI/web front end? Hundreds!

Hearing Uncle Bob speak reminds me that even when I am working on a project I think is being developed in a fantastic way, with fantastic testing - I can still try and make it better.

Mocking Iterable objects generically with Mockito

I often find my self having to mock iterable objects. At times I've created a test stub that overrides the iterator method and stubs the iterator, other times I have used mockito to mock the iterator. 

Most recently it was the Cassandra Datastax Java Driver's ResultSet object (which doesn't implement an interface and has a private constructor so you can't extend it and override or create an alternative implementation) which motivated me to create a generic method. 

So basically I want this code to work but with aSetOfStrings is a mock. You probably won't be mocking collections but it'll give you the idea.

         Set<String> aSetOfStrings = // passed in some how

       for (String s : aSetOfStrings) {

            results.add(s);

       }

And the method for creating the mock iterable object needs to be generic. E.g:

       public static <T> void mockIterable(Iterable<T> iterable, T... values)

Where the first parameter is the mock iterable object and the var arg is the objects that it should return. 

The foreach loop internally uses two methods on the iterator returned by the set's iterator() method: hasNext() and next().

So to get it to work three methods need to be mocked:

1. Mock the iterable's (the set) iterator() method to return a mock iterator
2. Mock the iterator's hasNext() method to return true N times followed by false where N is the number of values that you want the iterator to return.
3. Mock the iterator's next() method to return the N elements in order.

Using mockito number one is easy:

        Iterator<T> mockIterator = mock(Iterator.class);

       when(iterable.iterator()).thenReturn(mockIterator);

Number two and three are slightly more complicated. Mockito lets you pass in a vararg for what to return. The slight complication is that the signature is:

        thenReturn(T, T...)

This is to enforce that at least one element is passed in. This means that for the hasNext() we need to pass in true N times followed by a false but the first true needs to be passed in separately rather then in the vararg. The same applies for next() - we can't simply use the vararg passed into our mockIterable(..) method we need to build a new array with N-1 elements in. This can be done as follows:

1. If no values are passed in all we need to do is mock hasNext() to false.
2. If a single value is passed in we don't need to build an array to pass into thenReturn.
3. Finally, for more than one value we need to build the correct boolean array and values array to pass into thenReturn. For example:

MockIterator.mockIterable(mockIterable, "one", "two", "three");

We'd need the following mocking calls:

        when(mockIterable.hasNext()).thenReturn(true, [true, true, false]) 

and

        when(mockIterable.next()).thenReturn("one", ["two", "three"])

And here is the code to do it:

And the full code along with unit tests and a gradle script to build + pull in the dependencies is here.

Time series based queries in Cassandra 1.2+ and CQL3

Cassandra is fantastic for storing large amounts of data, and as of 1.2/CQL3 working with time series data just got a lot easier. Here is a basic example that stores some kind of posts (e.g blog) that can be queried by username and a time period.

Everything is assuming you have Cassandra 1.2+ installed and are using CQL3 with the cqlsh python client. Here are the exact versions I used for the below example:

[cqlsh 3.0.2 | Cassandra 1.2.5 | CQL spec 3.0.0 | Thrift protocol 19.36.0]

Creating a keyspace:
create keyspace cassandraspike WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor': 1};

The syntax has changed for creating keyspaces in CQL for cassandra 1.2 so if this fails check you aren't running against a pre-1.2 version of cassandra.
use cassandraspike;

Then create a posts table:
create table posts(username varchar, time timeuuid, post_text varchar, primary key(username, time))

We've used a compound primary key that is the username and the time. The time is of type timeuuid which is a new data type in CQL3 that us a type 1 UUID that contains a timestamp so it both stores the time of the post and makes our rows unique.

If you're familier with column families then it might interest you to know that the first column in your primary key becomes the column family (CF) row key. Every subsequent part of the primary key becomes part of the CF column names in that CF row. This has two implications:

• There will only be as many CF rows as there are variations of the first element in your primary key. This can be a problem if this element has a very low cardinality as you can end up with very wide CF rows.
• CF columns are stored in order so the fact that each CF column is prefixed with the time means that your data is stored in the order it happened. Making it very efficient to be queried by time.

Hopefully you are sufficiently convinced that your data is going to be stored in a way that is efficient to query by time. So let's store some data in and query it by user name and time.

To insert we'll make use if the now() function which gives you a timeuuid for the current time:

insert into posts(username, time, post_text) values ('chbatey', now(), 'i am writing something about cassandra');

Selecting this back works but gives you a time which isn't too human read able:
select * from posts;

 username | time                                 | post_text
----------+--------------------------------------+----------------------------------------
  chbatey | 59ad61d0-c540-11e2-881e-b9e6057626c4 | i am writing something about cassandra


That's where the dateOf function comes in handy which converts a timeuuid into a date:
select username, dateOf(time), post_text from posts;

 username | dateOf(time)             | post_text
----------+--------------------------+----------------------------------------
  chbatey | 2013-05-25 14:38:14+0100 | i am writing something about cassandra

I've now inserted another post at a different time:
select username, dateOf(time), post_text from posts;


username | dateOf(time)             | post_text
----------+--------------------------+----------------------------------------
  chbatey | 2013-05-25 14:38:14+0100 | i am writing something about cassandra
  chbatey | 2013-05-25 14:40:35+0100 | i am writing something about cassandra


Now lets say you are only interested in the posts chbatey made at 14:38:
select username, dateOf(time), post_text from posts where time >= minTimeuuid('2013-05-25 14:38') and time < minTimeuuid('2013-05-25 14:39') and username = 'chbatey';

 username | dateOf(time)             | post_text
----------+--------------------------+----------------------------------------
  chbatey | 2013-05-25 14:38:14+0100 | i am writing something about cassandra

This query is more complicated. It makes use of another cql function: minTimeuuid. The minTimeuuid function gives a fake (as it isn't unique) timeuuid that is the smallest possible timeuuid for the given time. This is very hand when you want to do less than/greater than queries on timeuuid fields.

In the above query we are getting everything that is greater than the minimum timeuuid for the time 2013-05-25 14:38 but less than the minimum timeuuid of the next minute 2013-05-25 14:39.

Rather than that we could have used the very similar function maxTimeuuid function with 2013-05-24 14:39:59. However I find original one easier to understand as I read it as greater than or equal to 14:38:00 and less than 14:39:00.

Installing Cassandra on Mac OS X

I've recently posted some more tips on using Cassandra on Mac OSX: Cassandra on Mac

If you don't already have homebrew then install it from here.

Then it as simple as:

brew install cassandra

This doesn't install the python driver for the cqlsh command line tool. To do this install it first install python if you haven't got it already:

brew install python

This should have also installed pip - the python package manager - so you can then install the cql python module:

pip install cql

Now try and start cqlsh

You might get this:


Python CQL driver not installed, or not on PYTHONPATH.
You might try "easy_install cql".

One second didn't I just install the cql module?

This could be because the Python in your path is the Mac OS X version. Not the version you installed with home brew that has cql. I fixed this by adding /usr/local/bin to the start of my PATH as that is where the brew Python executable lives:

export PATH=/usr/local/bin/:$PATH

Unless you've started cassandra the next time you try cqlsh you'll get:


cqlsh
Connection error: Could not connect to localhost:9160

Now if you do a brew info on cassandra:

brew info cassandra


To have launchd start cassandra at login:
    ln -sfv /usr/local/opt/cassandra/*.plist ~/Library/LaunchAgents
Then to load cassandra now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.cassandra.plist

Unless you are going to use cassandra a lot I wouldn't set it to load on startup as it does use a reasonable amount of memory. Instead to just start it off:

launchctl load /usr/local/opt/cassandra/homebrew.mxcl.cassandra.plist 

Finally cqlsh should connect to cassandra:



cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 3.0.2 | Cassandra 1.2.5 | CQL spec 3.0.0 | Thrift protocol 19.36.0]
Use HELP for help.
cqlsh> 

Or if you prefer the older cassandra-cli interface to cassandra:


cassandra-cli
Connected to: "Test Cluster" on 127.0.0.1/9160
Welcome to Cassandra CLI version 1.2.5

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown


All done.