tag:blogger.com,1999:blog-41613156447224069952024-03-18T02:48:23.060-07:00Christopher Batey's Blog has moved to www.batey.info@chbatey
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.comBlogger77125tag:blogger.com,1999:blog-4161315644722406995.post-19520455020230239672016-03-12T08:56:00.002-08:002016-03-12T08:56:40.195-08:00Blog has moved to batey.infoBeing able to work offline in Markdown has finally convinced me to leave Blogger. You can find my latest posts as well as all of my of my old ones at <a href="http://batey.info/">batey.info</a>chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com25tag:blogger.com,1999:blog-4161315644722406995.post-22095238442573816852015-08-05T02:11:00.000-07:002015-08-05T02:14:01.677-07:00A collection of tech talks I think we should all watchAn incomplete list of talks that I enjoyed. I'll add more and summaries at some point!<br />
<h3>
</h3>
<div>
<br /></div>
<h3>
Distributed systems</h3>
<div>
<br /></div>
<div>
These are all by Aphyr about Jepson:</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/mxdpqr-loyA/0.jpg" src="https://www.youtube.com/embed/mxdpqr-loyA?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/NsI51Mo6r3o/0.jpg" src="https://www.youtube.com/embed/NsI51Mo6r3o?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/QdkS6ZjeR7Q/0.jpg" src="https://www.youtube.com/embed/QdkS6ZjeR7Q?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br />
<br /></div>
<h3>
Perfromance </h3>
<div>
<br /></div>
<div>
This is also known as the Martin Thompson and Gil Tene section.</div>
<div>
<br /></div>
<div>
<b>Responding in a timely manner</b> by Martin Thompson:<br />
<a href="https://www.youtube.com/watch?v=4dfk3ucthN8">https://www.youtube.com/watch?v=4dfk3ucthN8</a></div>
<div>
<b>Understanding latency</b> by Gil Tene:<br />
<a href="https://www.youtube.com/watch?v=9MKY4KypBzg">https://www.youtube.com/watch?v=9MKY4KypBzg</a></div>
<div>
<br /></div>
<h3>
Functional/Events/Message driven</h3>
<div>
<br /></div>
<div>
This is also known as the Greg Young section.</div>
<div>
<br /></div>
<div>
<b>Querying event streams</b> by Greg Young:<br />
<a href="https://www.youtube.com/watch?v=DWhQggR13u8">https://www.youtube.com/watch?v=DWhQggR13u8</a></div>
<div>
<b>To be message driven</b> by Todd Montgomery:<br />
<a href="https://www.youtube.com/watch?v=DL_-ENSpcAg&list=PLSD48HvrE7-bo9rWaCLjxAocrxREREHnt&index=5">https://www.youtube.com/watch?v=DL_-ENSpcAg&list=PLSD48HvrE7-bo9rWaCLjxAocrxREREHnt&index=5</a></div>
<div>
<br /></div>
<h3>
Technology talks</h3>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com13tag:blogger.com,1999:blog-4161315644722406995.post-66461858732751030732015-07-31T03:56:00.000-07:002015-07-31T09:33:25.934-07:00Stubbed Cassnadra 0.9.1: 2.2 support, query variables and verifification of batches and PS preparationsVersion 0.9.1 has a few nice features and has reduced a lot of technical debt. The first features aren't that noticeable from a users point of view:<br />
<br />
<b>Java 1.6 support</b>! This was contributed by Andrew Tolbert as he wanted to test against the 1.* branch of the C* Java driver which still supports Java 1.6. I hadn't intentionally dropped 1.6 support but I had brought in a jar that was compiled against 1.7 and thus wouldn't run in a 1.6 JVM. The offending jar was an internal C* jar that could help with serialisation so it was quite a lot of work to replace the use of this jar with custom serialisation code.<br />
<br />
Another non-feature feature: Moving to Gradle and having a single build for the Server, Java client and Integration tests against all the versions of the C* Java driver. The aim of this was to make it MUCH easier for other people to contribute as before you had to install the server to a maven repo, then build the client and install it, then run the tests. Now you just run: ./gradlew clean check, Simples.<br />
<br />
<h3>
Real features</h3>
<div>
<br /></div>
<div>
Verification of batch statements containing queries. Priming and prepared statements in batches will be in the next release.</div>
<div>
<br /></div>
<div>
Support for version 2.2 of the C* driver.</div>
<div>
<br /></div>
<div>
Verification of the prepare of prepared statements. This will allow you to test you only prepare the statements once and at the right time i.e. application start up. </div>
<div>
<br /></div>
<div>
Queries that contain variables are now captured properly. As of C* 2.0 you can use a prepared statement like syntax for queries and pass in variables. These are now captured as long as you primed the types of the variables (so Stubbed Cassandra knows how to parse them).</div>
<div>
<br class="Apple-interchange-newline" />
A farJar task so you build a standalone executable. Useful if you aren't using Stubbed Cassandra from a build tool like maven.<br />
<br /></div>
<div>
As always please raise issues or reach out to be on twitter if you have any questions or feedback.</div>
<div>
<br /></div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com18tag:blogger.com,1999:blog-4161315644722406995.post-15746106445166157222015-07-07T05:51:00.000-07:002015-09-10T07:24:20.156-07:00Cassandra 3.0 materialised views in action (pre-release)Disclaimer: C* 3.0 is not released yet and all these examples are from a <a href="https://github.com/carlyeks/cassandra/tree/ticket/6477-8099">branch</a> that hasn't even made it to trunk yet.<br />
<br />
So this feature started off as "<a href="https://issues.apache.org/jira/browse/CASSANDRA-6477">Global indexes</a>", the final result is not a global index and I don't trust any claim of distributed indexes anyhow. If your data is spread across 200 machines, ad-hoc queries aren't a good idea reagardless of how you implement them as you will often end up going to all 200 machines.<br />
<br />
Instead materialised views make a copy of your data partitioned a different way, which is basically what we've been doing manually in C* for years, this feature aims to remove the heavy lifting.<br />
<br />
I'll use the same data model as the last article which is from the KillrWeather application. I will attempt to show use cases which we'd have previously used Spark or duplicated our data manually.<br />
<br />
Recall the main data table:<br />
<br />
<script src="https://gist.github.com/chbatey/f24608e26085fe4f08fe.js"></script>
This table assumes our queries are all going to be isolated to a weather station id (wsid) as it is the partition key. The KillrWeather application also has a table with information about each weather station:<br />
<br />
<script src="https://gist.github.com/chbatey/326a1170bd365a28302c.js"></script>
I am going to denormalise by adding the columns from the weather_station table directly in the raw_weather_data table ending up with:<br />
<br />
<script src="https://gist.github.com/chbatey/bd36ef3ffc71031513e0.js"></script>
Now can we do some awesome things with materialised views? Of course we can!<br />
<br />
So imagine you need to read the data not by weather station ID but by state_code. We'd normally have to write more code to duplicate the data manually. Not any more.<br />
<div>
<br /></div>
<div>
First let's insert some data, I've only inserted the primary key columns and one_hour_precip and I may have used UK county names rather than states :)</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/a218f6d916f52476428d.js"></script></div>
<div>
<br /></div>
<div>
We can of course query by weather id and time e.g</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKYSeNgYIvMMjtywGPSxQ85WcyOXmk8EsLDiOf1q7icYg8KYpC0IU0Do7Y2O6Q0YqDbdHZ6Zxq0AemYgjyB12UnkCbsyeeID3Zqpe4AG2T-qZ8f-Cl3mLAriR91z_4thEayi7PzzXfpQh1/s1600/Screenshot+2015-07-07+09.34.29.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="96" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKYSeNgYIvMMjtywGPSxQ85WcyOXmk8EsLDiOf1q7icYg8KYpC0IU0Do7Y2O6Q0YqDbdHZ6Zxq0AemYgjyB12UnkCbsyeeID3Zqpe4AG2T-qZ8f-Cl3mLAriR91z_4thEayi7PzzXfpQh1/s640/Screenshot+2015-07-07+09.34.29.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
We can then create a view:<br />
<br />
<script src="https://gist.github.com/chbatey/378bb5481739fb7ee475.js"></script></div>
<div>
<br /></div>
<div>
We've asked C* is to materialise <span style="font-family: Courier New, Courier, monospace;">select country_code from raw_weather_data</span> but with a different partition key, how awesome is that?? All of the original primary key columns and any columns in your new primary key are automatically added and I've added country_code for no good reason.<br />
<br />
With that we can query by state and year as well. I included year as I assumed that partitioning by state would lead to very large partitions in the view table.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2IktZb684JZySm9Ru_SSrk1V9KeIrutQjYK6KT07fPpbQPvaSwrn8R6PrB16LjLMnrxLsI2L4mRlXDge6_gP4VTGyeaxKJAq2VCy7I1yAhLNge_00jvPiNEhdbOLWgkyFzI4l8HuFb3WA/s1600/Screenshot+2015-07-07+13.18.08.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="122" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2IktZb684JZySm9Ru_SSrk1V9KeIrutQjYK6KT07fPpbQPvaSwrn8R6PrB16LjLMnrxLsI2L4mRlXDge6_gP4VTGyeaxKJAq2VCy7I1yAhLNge_00jvPiNEhdbOLWgkyFzI4l8HuFb3WA/s640/Screenshot+2015-07-07+13.18.08.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br /></div>
<div>
Where as secondary indexes go to the original table, which will often result in a multi partition query (a C* anti-pattern), a materialised view is copy of your data partitioned in a new way. This query will be as quick as if you'd duplicated the data your self.<br />
<br />
The big take away here is that YOU, the developer, decide the partitioning of the materialised view. This is an important point. There was talk of you only needing to specify a cardinality, e.g low, medium, high or unique and leave C* to decide the partitioning. Where as that would have appeared more user friendly it would be a new concept and a layer of abstraction when IMO it is critical all C* developers/ops understand the importance of partitioning and we already do it every day for tables. You can now use all that knowledge you already have to design good primary keys for views.<br />
<br />
<h3>
The fine print</h3>
<br />
I'll use the term "original primary key" to refer to the table we're creating a materialised view on and MV primary key for our new view.<br />
<ol>
<li>You can include any part of the original primary key in your MV primary key and a single column that was not part of your original primary key</li>
<li>Any part of the original primary key you don't use will be added to the end of your clustering columns to keep it a one to one mapping</li>
<li>If the part of your primary key is NULL then it won't appear in the materialised view</li>
<li>There is overhead added to the write path for each materialised view</li>
</ol>
Confused? Example time!</div>
<div>
<br /></div>
<div>
<b>Original primary key</b>: <span style="font-family: Courier New, Courier, monospace;">PRIMARY KEY ((wsid), year, month, day, hour)</span></div>
<div>
<b>MV primary key</b>: <span style="font-family: Courier New, Courier, monospace;">PRIMARY KEY ((state_code, year), one_hour_precip)</span></div>
<div>
<b>Conclusion:</b> No, this is actually the example above and it does not work as we tried to include two columns that weren't part of the original primary key. I updated the original primary key to: <span style="font-family: Courier New, Courier, monospace;">PRIMARY KEY ((wsid), year, month, day, hour, state_code) </span>and then it worked.</div>
<div>
<br /></div>
<div>
<div>
<b>Original primary key</b>: <span style="font-family: Courier New, Courier, monospace;">PRIMARY KEY ((wsid), year, month, day, hour)</span></div>
<div>
<b>MV primary key</b>: <span style="font-family: Courier New, Courier, monospace;">PRIMARY KEY ((state_code, year))</span></div>
<div>
<b>Conclusion</b>: Yes - only one new column in the primary key: state_code</div>
</div>
<div>
<br /></div>
<div>
Here are some of the other questions that came up:</div>
<div>
<ol>
<li>Is historic data put into the view on creation? Yes</li>
<li>Can the number of fields be limited in the new view? Yes - in the select clause</li>
<li>Is the view written to synchronously or asynchronously on the write path? <b>Very subject to change!</b> It's complicated! The view mutations are put in the batch log and sent out before the write, the write can succeed before all the view replicas have acknowledged the update but the batch log won't be cleared until a majority of them have responded. See the diagram in this <a href="http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views">article</a>.</li>
<li>Are deletes reflected? Yes, yes they are!</li>
<li>Are updated reflected? Yes, yes they are!</li>
<li>What happens if I delete a table? The views are deleted</li>
<li>Can I update data via the view? No</li>
<li>What is the overhead? TBD, though it will be similar to using a logged batch if you had duplicated manually.</li>
<li>Will Patrick Mcfadin have to change all his data modelling talks? Well at least some of them</li>
</ol>
<h3>
Combining aggregates and MVs? Oh yes</h3>
</div>
<div>
<br /></div>
<div>
You can't use aggregates in the select clause for creating the materialised view but you can use them when querying the materialised view. So we can now answer questions like what is the total precipitation for a given year for a given state:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirs8rSklnV8KI__LWvpjWGIQB5lL6ZIZdEflbTMv1-q1VS5HjmUxemtGhqanq-wwqNt4fiPgOq9SrTrMvuoSs-YSwsyIPEffrugGwhxFr4HxmbkKVufOqiBkBoyHHO6i1r_21jSxFYlQyx/s1600/Screenshot+2015-07-07+13.26.40.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="104" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirs8rSklnV8KI__LWvpjWGIQB5lL6ZIZdEflbTMv1-q1VS5HjmUxemtGhqanq-wwqNt4fiPgOq9SrTrMvuoSs-YSwsyIPEffrugGwhxFr4HxmbkKVufOqiBkBoyHHO6i1r_21jSxFYlQyx/s640/Screenshot+2015-07-07+13.26.40.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
We can change our view to include the month in its key and do the same for monthly:<br />
<br />
<script src="https://gist.github.com/chbatey/0e39fd9601e3fd50916f.js"></script>
And then we can do:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuAAuuDjCIaHoj3hRN2HdEraeH-dYj5bguupRQl8MCe9zNAw4a3X1-KOEMlNb-8JIsn9fISH7_EYpxXBnhqbhDUX6QE89halrDvgyN7wqniHN0P1HZ0J4SdiimdlEhfHoXHJEzfnjQHXmT/s1600/Screenshot+2015-07-07+13.32.20.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuAAuuDjCIaHoj3hRN2HdEraeH-dYj5bguupRQl8MCe9zNAw4a3X1-KOEMlNb-8JIsn9fISH7_EYpxXBnhqbhDUX6QE89halrDvgyN7wqniHN0P1HZ0J4SdiimdlEhfHoXHJEzfnjQHXmT/s640/Screenshot+2015-07-07+13.32.20.png" width="640" /></a></div>
<br />
<br />
Though my data had all the rain in one month :)<br />
<br />
<h3>
Conclusion</h3>
</div>
<div>
<br /></div>
<div>
This feature changes no key concepts for C* data modelling it simply makes the implementation of the intended data model vastly easier. If you have data set that doesn't fit on a single server you're still denormalising and duplicating for performance and scalability, C* will just do a huge amount of it for you in 3.0.</div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com8tag:blogger.com,1999:blog-4161315644722406995.post-15722064674871637762015-07-03T06:36:00.001-07:002015-07-03T07:31:17.729-07:00A few more Cassandra aggregates...My last post was on UDAs in C* 2.2 beta. C*2.2 is now at RC1 so again everything in this post is subject to change. I'm running off 3.0 trunk so it is even more hairy. Anyway there are more built in UDAs now so let's take a look...<br />
<br />
I am going to be using the schema from KillrWeather to illustrate the new functionality. KillrWeather is a cool project that uses C* for its storage and a combination of Spark batch and Spark Streaming to provide analytics on weather data.<br />
<br />
Now C* hasn't previously supported aggregates but 2.2 changes all that, so let's see which parts of KillrWeather we can ditch the Spark and go pure C*.<br />
<br />
The raw weather data schema:<br />
<br />
<script src="https://gist.github.com/chbatey/f24608e26085fe4f08fe.js"></script>
Spark batch is used to populate the high low "materialised view" table:<br />
<br />
<script src="https://gist.github.com/chbatey/aedc95e61934ddd8cddd.js"></script>
The code from KillrWeather Spark batch:<br />
<br />
<script src="https://gist.github.com/chbatey/c0ada6d4c3d512999102.js"></script>
There's a lot going on here as this code is from a fully fledged Akka based system. But essentially it is running a Spark batch job against a C* partition and then using the Spark StatsCounter to work out the max/min temperature etc. This is all done against the raw table, (not shown) the result is passed back to the requester and asynchronously saved to the C* daily_aggregate table.<br />
<br />
Stand alone this would look something like:<br />
<br />
<script src="https://gist.github.com/chbatey/abddd26af0599a8e94bf.js"></script>
Now let's do something crazy and see if we can do away with this extra table and use C* aggregates directly against the raw data table:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVyLZSpq07TbEhXc8b6VykD_dyUk6VpgI0Z8jac0ptAKCZPbOQKsxTzraxMbKEjKw_RIYOkX1tf33fwAh0Wmox_oF45eEipmG49aA4xj7ihIufCdo21hkhj_4Hj27YUCIqW6ky1ewneU0N/s1600/Screenshot+2015-07-03+09.59.59.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="185" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVyLZSpq07TbEhXc8b6VykD_dyUk6VpgI0Z8jac0ptAKCZPbOQKsxTzraxMbKEjKw_RIYOkX1tf33fwAh0Wmox_oF45eEipmG49aA4xj7ihIufCdo21hkhj_4Hj27YUCIqW6ky1ewneU0N/s400/Screenshot+2015-07-03+09.59.59.png" width="400" /></a></div>
<br />
<br />
Because we have the year, month, as clusterting columns we can get the max/min/avg all from the raw table. This will perform nicely as it is within a C* partition, don't do this across partitions! We haven't even had to define our own UDFs/UDAs as max and mean are built in. I wanted to analyse how long this UDA was taking but it currently isn't in trace so I raised a <a href="https://issues.apache.org/jira/browse/CASSANDRA-9723">jira</a>.<br />
<br />
The next thing KillrWeather does is keep this table up to date with Spark streaming:<br />
<br />
<script src="https://gist.github.com/chbatey/0d5c076c8c3d864cd51e.js"></script>
Can we do that with built in UDAs? Uh huh!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtVJBr-5WM8tQWLF7p6LnJWrCh_gvOJD-12Rss9le3akkfpvhD1z-yTL900763F6IBuN4XN-qAEZmwFRF5CfpaJ7qlmZLuNXTjMCkSz4gynmqAK7yKxaugH4nmdzyFxqSzY-ix3Uj-aosM/s1600/Screenshot+2015-07-03+11.18.54.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="127" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtVJBr-5WM8tQWLF7p6LnJWrCh_gvOJD-12Rss9le3akkfpvhD1z-yTL900763F6IBuN4XN-qAEZmwFRF5CfpaJ7qlmZLuNXTjMCkSz4gynmqAK7yKxaugH4nmdzyFxqSzY-ix3Uj-aosM/s400/Screenshot+2015-07-03+11.18.54.png" width="400" /></a></div>
<br />
The data is a little weird as for one_hour_precip there are negative values, hence why it appears that we have less rain in a month than we do in a single day in that month.<br />
<br />
We can also do things that don't include a partition key like get the max for all weather stations, but this will be slow / could cause OOM errors if you have a large table:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTzus9NIRU13qXhgbFLciSnaelzA5Z_ARhV_A1nBjBCDAI0mKIyEpHasXa1IamHhCv3Wxd5R6c4eXT9fz-CIWJAnA4vQoKZpk_tHtj2jgF6xddkWTFCcw_UNWWXJKQUMNWW5ze6lu1WuOm/s1600/Screenshot+2015-07-03+13.50.24.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="46" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTzus9NIRU13qXhgbFLciSnaelzA5Z_ARhV_A1nBjBCDAI0mKIyEpHasXa1IamHhCv3Wxd5R6c4eXT9fz-CIWJAnA4vQoKZpk_tHtj2jgF6xddkWTFCcw_UNWWXJKQUMNWW5ze6lu1WuOm/s320/Screenshot+2015-07-03+13.50.24.png" width="320" /></a></div>
<br />
<br />
All the raw text for the queries are on my <a href="https://github.com/chbatey/cassandra-examples">GitHub</a>.<br />
<br />
<br />
<br />
<br />chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com40tag:blogger.com,1999:blog-4161315644722406995.post-35369997023097451842015-05-28T02:12:00.002-07:002016-04-08T12:47:04.501-07:00Cassandra Aggregates - min, max, avg, group by<a href="http://localhost:4000/cassandra-aggregates-min-max-avg-group.html">This blog has moved to batey.info and won't be updated here.</a><br />
<br />
Disclaimer: all this was against 2.2-beta so the syntax may have changed.<br />
<br />
Cassandra 2.2 introduced user defined functions and user defined aggregates. We can now do things like min, max and average on the server rather than having to bring all the data back to your application. Max and min are built in but we'll see how you could have implemented them your self.<br />
<br />
<h3>
Max/Min</h3>
<br />
Here's an example table for us to try and implement max/min against.<br />
<br />
<script src="https://gist.github.com/chbatey/2a6edaba343a05462443.js"></script>
<br />
User defined aggregates work by calling your user defined function on every row returned from your query, they differ from a function because the first value to the function is state that is passed between rows, much like a fold.<br />
<br />
Creating an aggregate is a two or three step process:
<br />
<ol>
<li>Create a function that takes in state (any Cassandra type including collections) as the first parameter and any number of additional parameters</li>
<li>(Optionally) Create a final function that is called after the state function has been called on every row</li>
<li>Refer to these in an aggregate</li>
</ol>
<div>
For max we don't need a final function but we will for average later.<br />
<br /></div>
<script src="https://gist.github.com/chbatey/c40482646ab899d8117e.js"></script>
<br />
<div>
Here we're using Java for the language (you can also use JavaScript) and just using Math.max. For our aggregate definition we start with (INITCOND) null (so it will return null for an empty table) and then set the state to be the max of the current state and the value passed in. We can our new aggregate like:</div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaPK6Yclb437v_9Qtn5_Hg0qA2H_MwKFJSGh-BRBSJP_C0EuAg2m_opcBpz_kvdoq3TJZualaEr4wn5-NyhUw4blQCYaTWcXN2YXIoR0uBGaAGNLACUpnq5k5gsRRAw2groui9qGnXVbMs/s1600/Screenshot+2015-05-28+09.26.58.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaPK6Yclb437v_9Qtn5_Hg0qA2H_MwKFJSGh-BRBSJP_C0EuAg2m_opcBpz_kvdoq3TJZualaEr4wn5-NyhUw4blQCYaTWcXN2YXIoR0uBGaAGNLACUpnq5k5gsRRAw2groui9qGnXVbMs/s320/Screenshot+2015-05-28+09.26.58.png" width="289" /></a></div>
<br />
<h3>
GroupBy</h3>
<div>
<br />
So there's no group by keyword in Cassandra but you can get similar behaviour with a custom user defined aggregate. Imagine you had a table that kept track of everything your customers did e.g</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHwy0Jjx_baDw5EjNlSMNuXixhll1KgutEUa1QG-SMOFvOdKh3J8S4HBC0QJ6CWsQ764M-cHiouEUEbUA7G0v6qzJtn5wSD8_r_6UZLNqcsZGefqlvaoL4WvSm5q5GJAbAWNIWjEJgwQ_-/s1600/Screenshot+2015-05-28+09.31.41.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="115" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHwy0Jjx_baDw5EjNlSMNuXixhll1KgutEUa1QG-SMOFvOdKh3J8S4HBC0QJ6CWsQ764M-cHiouEUEbUA7G0v6qzJtn5wSD8_r_6UZLNqcsZGefqlvaoL4WvSm5q5GJAbAWNIWjEJgwQ_-/s400/Screenshot+2015-05-28+09.31.41.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
We can write a UDA to get a count of a particular column:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/55e559408f6016bd72db.js"></script></div>
<div>
<br /></div>
<div>
And we keep track of the counts in a map. Example use for counting both the event_type and the origin of the event:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg28oetODZ5bpeLHCilS_95_o4RhOYzPjIa5YNUc6gy1ZiHoNm91TLie8oJp-HaZahSWTxTnd2mYonEufHcnIjQ3REmM6jlZQNykNIU4-Pi_I_oenyTTRGYsUn2mBzK1rMvFXDMzBZEuTH0/s1600/Screenshot+2015-05-28+09.57.34.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg28oetODZ5bpeLHCilS_95_o4RhOYzPjIa5YNUc6gy1ZiHoNm91TLie8oJp-HaZahSWTxTnd2mYonEufHcnIjQ3REmM6jlZQNykNIU4-Pi_I_oenyTTRGYsUn2mBzK1rMvFXDMzBZEuTH0/s400/Screenshot+2015-05-28+09.57.34.png" width="400" /></a></div>
<br />
More often than not when you use group by in other databases you are totalling another field. For example imagine we were keeping track of customer purchases and wanted a total amount each customer has spent:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoF6IhNyjvafT4YiKR7y5px2s9ZgjvF8DXV7h-4VKZPkPr7KpJpG0NfUHY601CHDScq_txE7lV_Qc5mry0fPpkU9WFV6mwtmpcpcZU8lzRGCw-rK1CD761SuyDtvICk8xRMFsNmXnEKAKc/s1600/Screenshot+2015-05-28+10.00.20.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="157" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoF6IhNyjvafT4YiKR7y5px2s9ZgjvF8DXV7h-4VKZPkPr7KpJpG0NfUHY601CHDScq_txE7lV_Qc5mry0fPpkU9WFV6mwtmpcpcZU8lzRGCw-rK1CD761SuyDtvICk8xRMFsNmXnEKAKc/s400/Screenshot+2015-05-28+10.00.20.png" width="400" /></a></div>
<br />
<br />
We can create a generate aggregate for that called group_and_total:<br />
<br />
<script src="https://gist.github.com/chbatey/709ce9dcebd12a0f8820.js"></script></div>
<br />
And an example usage:
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbLy_DyGfR8aNKaw1aA-6x5IzBynhcthGJxxBoQJoyczL0IYwxbcf_YhhPqIT0n_k65X9JTD26dBgo5b6kS47yXZpPR_GLWxf9t5x6lzVmINVQX9ILZ8mh_Y23aLeqttH32ZUcWTjDxAiZ/s1600/Screenshot+2015-05-28+10.02.34.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="221" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbLy_DyGfR8aNKaw1aA-6x5IzBynhcthGJxxBoQJoyczL0IYwxbcf_YhhPqIT0n_k65X9JTD26dBgo5b6kS47yXZpPR_GLWxf9t5x6lzVmINVQX9ILZ8mh_Y23aLeqttH32ZUcWTjDxAiZ/s400/Screenshot+2015-05-28+10.02.34.png" width="400" /></a></div>
<br />
<h3>
</h3>
As you can see Haddad spends way too much.<br />
<br />
<h3>
Average</h3>
<div>
<br />
The Cassandra docs have an example of how to use a user defined aggregate to calculate aggregates: http://cassandra.apache.org/doc/cql3/CQL-2.2.html#udas<br />
<br /></div>
</div>
<div>
<h3>
Small print</h3>
</div>
<div>
<br />
If you've ever heard me rant about distributed databases you've probably heard me talk about scalable queries, ones that work on a 3 node cluster as well as a 1000 node cluster. User defined functions and aggregates are executed on the coordinator. So if you don't include a partition key in your query all the results are brought back to the coordinator for your function to be executed, if you do a full table scan for your UDF/A don't expect it to be fast if your table is huge.</div>
<div>
<br /></div>
<div>
This functionality is in beta for a very good reason, it is user defined code running in your database! Proper sandboxing e.g with a SecurityManager will be added before this goes mainstream.</div>
<div>
<br /></div>
<br />
<br />
<br />chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com490tag:blogger.com,1999:blog-4161315644722406995.post-77922462483675214472015-05-10T00:59:00.000-07:002015-05-10T00:59:21.789-07:00Building well tested applications with SpringBoot / Gradle / Cucumber / GatlingI am a huge test first advocate. Since seeing Martin Thompson speak I am now trying to include performance testing with the same approach. I am going to call this approach Performance, Acceptance and Unit Test Driven Development, or PAUTDD :)<br />
<br />
Tools/frameworks/library come and go so I'll start with the theory then show how I set this up using Junit and Mockio for unit testing, Cucumber for acceptance tests and Gatling for performance tests. Well I won't show JUnit and Mockito because that is boring!<br />
<br />
So here's how I develop a feature:<br />
<ol>
<li>Write a high level end to end acceptance test. There will be times where I'll want acceptance tests not to be end to end, like if there was an embedded rules engine.</li>
<li>Write a basic performance test.</li>
<li>TDD with unit testing framework until the above two pass.</li>
<li>Consider scenario based performance test.</li>
</ol>
<div>
I hate to work without test first at the acceptance level, even for a small feature (half a day dev?) I find them invaluable for keeping me on track and letting me know when functionally I am done. Especially if I end up doing a bit too much Unit testing/Mocking (bad Chris!) as when head down in code it is easy to forget the big picture: what functionality are you developing for a user. </div>
<div>
<br /></div>
<div>
Next is a newer part of my development process. Here I want a performance test for the new feature, this may not be applicable but it usually is. Which ever framework I am using here I want to be able to run it locally and as part of my continuous integration for trend analysis. However I want more for my effort than that, I really want to be able to use the same code for running against various environments running different hardware.<br />
<br />
I hate performance tests that aren't in source control and versioned. Tools that use a GUI are no use to me, I constantly found while working at my last company that the "performance testers" would constantly change their scripts and this would change the trend way more than any application changes. I want to be able to track both.</div>
<div>
<br /></div>
<h3>
So how to set all this up for a Spring Boot application using Cucumber and Gatling?</h3>
<div>
<br />
This post is on build setup, not the actual writing of the tests. My aim is to enable easy acceptance/performance testing.</div>
<div>
<br /></div>
<div>
Here's the layout of my latest project:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHmd1C6WMJOnx2VTawr975iITPY-GqDb4erBBa23_29Zr85uJ0YpcUvKNcqrudchVb46-B9xPG3eS0bJokUR3-BsqJ_JRrazRp1MZEhEKC12KobN41msNtoT2GnQBdcVi38neqmHwCQM6Z/s1600/Screenshot+2015-02-23+17.52.17.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHmd1C6WMJOnx2VTawr975iITPY-GqDb4erBBa23_29Zr85uJ0YpcUvKNcqrudchVb46-B9xPG3eS0bJokUR3-BsqJ_JRrazRp1MZEhEKC12KobN41msNtoT2GnQBdcVi38neqmHwCQM6Z/s1600/Screenshot+2015-02-23+17.52.17.png" width="189" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Main is the source, test is the unit tests and e2e is both the Cucumber and the Gatling tests. I could have had separate source sets for the Cucumber and Gatling tests but that would have confused IntelliJ's Gradle support too much (and they are nicely split by Cucumber being Java and Gatling being Scala).</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<h4 style="clear: both; text-align: left;">
Cucumber JVM</h4>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
There are various articles on Cucumber-JVM, here's the steps I used to get this running nicely in the IDE and via Gradle.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
First the new source set:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<script src="https://gist.github.com/chbatey/6da41beb0d620618d180.js"></script></div>
<div class="separator" style="clear: both; text-align: left;">
Nothing exciting here, we are using the same classapth as test, we could have had a separate one.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Next is dependencies, this is actually the Gatling and HTTP client (for hitting our application) as well.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<script src="https://gist.github.com/chbatey/69fd3687103d0fc36811.js"></script></div>
<div>
We cucumber JUnit and Spring dependencies.<br />
<br />
Next is the source code for the acceptance tests. The Features are in the resource folder and the source is in the Java folder. To allow running via Gradle you also create a JUnit test to run all the features. Intellij should work find without this by just running the feature files.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHojyyI7z-FmO2PbfgsDbSP6oMZYcjsWCGe6TOWoQZVl-g9owtbaoFn4eIoLC4qkf-0Ha1_HwFRhkpPKNWaHI87NzekiYpmNrtVCzFJZupxRhWOjZb7zc3-sbNpZh_0EPa1hW8mCwfYv5j/s1600/Screenshot+2015-02-23+18.14.28.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHojyyI7z-FmO2PbfgsDbSP6oMZYcjsWCGe6TOWoQZVl-g9owtbaoFn4eIoLC4qkf-0Ha1_HwFRhkpPKNWaHI87NzekiYpmNrtVCzFJZupxRhWOjZb7zc3-sbNpZh_0EPa1hW8mCwfYv5j/s1600/Screenshot+2015-02-23+18.14.28.png" width="320" /></a></div>
<br />
<br />
Here I have Features separated by type, here's an example Feature:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYtKRvw8zLsUoH5pPB6hofJR8RkJEg6-JrkpWJ65RFTWYpKBP7QfUOy9SLScsVV3jQvuuiQEgba6RgHM_2BHxA_lgzbLYL0E_x00uZ7LMH5C_M4ZEbEfm4slsoD1sYNCWwr5pYFR0lRRT6/s1600/Screenshot+2015-02-23+18.16.56.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="62" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYtKRvw8zLsUoH5pPB6hofJR8RkJEg6-JrkpWJ65RFTWYpKBP7QfUOy9SLScsVV3jQvuuiQEgba6RgHM_2BHxA_lgzbLYL0E_x00uZ7LMH5C_M4ZEbEfm4slsoD1sYNCWwr5pYFR0lRRT6/s1600/Screenshot+2015-02-23+18.16.56.png" width="400" /></a></div>
<br /></div>
<div>
I like to keep the language at a high level so a non-techy can write these. The JUnit test RunEndToEndTests looks like this:<br />
<br />
<script src="https://gist.github.com/chbatey/5027dca5c71034353a8a.js"></script><br />
This is what Gradle will pick up when we run this from the command line. You could separate this out into multiple tests if you wanted.<br />
<br />
For running inside IntelliJ you might need to edit the run configuration to include a different Glue as by default it will be the same as the package your Feature file is in, for this project this wouldn't pick up the GlobalSteps as it is outside of the security/users folder. This is what my configuration looks like, I set this as the default:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjb1BT_Ao3bjEWlGm0kfG7GKaB4xmU-z8VmWrVStkCuBKt6IyCdpjDEXdOzjNPe6W3X3ScYiAAkZNwvOT-oyfhJQ1UpPXTDa8PH6gC35ruRZhwFT64aBqdGKtLxphTrDspP-qs7wFmcRkCb/s1600/Screenshot+2015-02-23+18.21.18.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="71" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjb1BT_Ao3bjEWlGm0kfG7GKaB4xmU-z8VmWrVStkCuBKt6IyCdpjDEXdOzjNPe6W3X3ScYiAAkZNwvOT-oyfhJQ1UpPXTDa8PH6gC35ruRZhwFT64aBqdGKtLxphTrDspP-qs7wFmcRkCb/s1600/Screenshot+2015-02-23+18.21.18.png" width="400" /></a></div>
<br />
Now our features will run if you want to see what the implementation of the Steps look like, checkout the whole project from Github.<br />
<br />
<h4>
Gatling</h4>
</div>
<div>
<br />
This is my first project using Gatling, I wanted my scenarios in code that I could have in version control. Previously I've used JMeter. Where as you can checkin the XML it really isn't nice to look at in diff tools. I've also been forced *weep* to use more GUI based tools like SOASTA and HP Load runner. One thing I haven't looked at is Gatling's support for running many agents. For me to continue using Gatling beyond developer trend analysis this needs to be well supported.</div>
<div>
<br /></div>
<div>
We already have the dependencies, see the dependencies section above, and we're going to use the same source set. The only difference is we're going to be writing these tests in Scala.</div>
<div>
<br /></div>
<div>
My first requirement was not to have to have Gatling installed manually on developer and CI boxes. Here's how to do this in Gradle:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/fc3fb0d15a3b9d925ccc.js"></script></div>
<div>
Where BasicSimulation is the fully qualified name of my Gatling load test. All we do here is define a JavaExec task with the Gatling main class, tell Gatling where our source is and which simulation to run.<br />
<br />
To tie all this together so it runs every time we fun Gradle check we add the following at the bottom of our Gradle build file:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL0qSFqnle3x0X9BZcpzP71DD1HJTBJsR5l8UIQR42Rcp-0H7RThPk1YYeGuNaass0A2iZORVKbhn4jhE5emWLatI5ZZItI4rWD9Rq1zsY0J1e9r1ugmBoq4uRg12_wNbEf9Gi6aTdjryc/s1600/Screenshot+2015-02-23+18.29.29.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="41" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL0qSFqnle3x0X9BZcpzP71DD1HJTBJsR5l8UIQR42Rcp-0H7RThPk1YYeGuNaass0A2iZORVKbhn4jhE5emWLatI5ZZItI4rWD9Rq1zsY0J1e9r1ugmBoq4uRg12_wNbEf9Gi6aTdjryc/s1600/Screenshot+2015-02-23+18.29.29.png" width="200" /></a></div>
<br /></div>
<div>
This will produce reports that can be published + consumed by the Jenkins Gatling plugin.<br />
<br />
To run the same load test from Intellij we do exactly the same in a run configuration:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCK-9_DgH56IMya3kZcJWirmYcXOFRfgPyAJP-HY8n5l99zUj_BWqxBTivg9_QKntgOVjUYnJKpkcS8nxBCAM5zn8_acGYRlNN7_jLb1NRjxIDBSj4Xiwm9pmMAI0i0sIPvHldBZzNSTw3/s1600/Screenshot+2015-02-23+18.31.45.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="104" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCK-9_DgH56IMya3kZcJWirmYcXOFRfgPyAJP-HY8n5l99zUj_BWqxBTivg9_QKntgOVjUYnJKpkcS8nxBCAM5zn8_acGYRlNN7_jLb1NRjxIDBSj4Xiwm9pmMAI0i0sIPvHldBZzNSTw3/s1600/Screenshot+2015-02-23+18.31.45.png" width="320" /></a></div>
<br /></div>
<div>
<br /></div>
<div>
A basic Gatling tests looks like this:<br />
<br />
<script src="https://gist.github.com/chbatey/9c5ee580cbae648dd30f.js"></script><br />
This test will run a very simple test where a single virtual user hits the /api/auction URL 100 times with a pause of 10 milliseconds. The top couple of lines start the Spring boot application and register a shut down hook to stop it.<br />
<br />
We'll then end up with a report that looks like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLlCFnZKVVR88I4Mif50wCZX3legO5sdsqSnkobzxaWM9YcDrf7XRoX3wPdjOwZeB7oob9JWCRhGFwakuNHOMYnSzpVr8wCVla0FxIc2SeNHj4TI176fMv5A-YYNdAvarAZyjhVcCQ5Ffz/s1600/Screenshot+2015-02-23+18.34.48.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="251" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLlCFnZKVVR88I4Mif50wCZX3legO5sdsqSnkobzxaWM9YcDrf7XRoX3wPdjOwZeB7oob9JWCRhGFwakuNHOMYnSzpVr8wCVla0FxIc2SeNHj4TI176fMv5A-YYNdAvarAZyjhVcCQ5Ffz/s1600/Screenshot+2015-02-23+18.34.48.png" width="400" /></a></div>
<br /></div>
<div>
This is a pretty terrible load test as it runs for 2 seconds and has a single user. But the point of this post is to setup everything so when adding new functionality it is trivial to add new performance and acceptance tests.<br />
<br />
That's it, happy testing! If you want to see the whole project is on <a href="https://github.com/chbatey/killrauction">Github</a>. It us under active development so you'll know how I got on with Gatling based if it is still there and there are lots of tests that use it!</div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com9tag:blogger.com,1999:blog-4161315644722406995.post-71538387288008752822015-05-04T09:19:00.002-07:002015-05-04T12:25:17.749-07:00Strata workshop: Getting started with Cassandra<div class="p1">
<span class="s1">Downloading and installing Cassandra:</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Linux/Mac:</span></div>
<div class="p1">
<span class="s1">curl -L http://downloads.datastax.com/community/dsc-cassandra-2.1.4-bin.tar.gz | tar xz</span></div>
<br />
<div class="p1">
<span class="s1">(or use home brew)</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Then run:</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">./bin/cassandra</span><br />
<span class="s1"><br /></span>
<span class="s1">To start Cqlsh (may need to install Python)</span><br />
<span class="s1"><br /></span>
<span class="s1">./bin/cqlsh</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Windows:</span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s2"><a href="http://planetcassandra.org/cassandra/">http://planetcassandra.org/cassandra/</a> or grab a USB key from me.</span></div>
<div class="p1">
<br /></div>
<div class="p1">
<span class="s1">Workshop code (we may not get to this):</span></div>
<div class="p1">
</div>
<div class="p1">
<span class="s1">git clone <span class="s2"><a href="https://github.com/chbatey/strata-cassandra-workshop">https://github.com/chbatey/strata-cassandra-workshop</a></span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Cql docs:</span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1"><b><a href="http://docs.datastax.com/en/cql/3.1/cql/cql_intro_c.html">http://docs.datastax.com/en/cql/3.1/cql/cql_intro_c.html</a></b></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Cassandra docs:</span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1"><b><a href="http://docs.datastax.com/en/cassandra/2.1/cassandra/gettingStartedCassandraIntro.html">http://docs.datastax.com/en/cassandra/2.1/cassandra/gettingStartedCassandraIntro.html</a></b></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Java Driver Docs:</span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1"><b><a href="http://docs.datastax.com/en/developer/java-driver/2.1/java-driver/whatsNew2.html">http://docs.datastax.com/en/developer/java-driver/2.1/java-driver/whatsNew2.html</a></b></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Data modelling exercises:</span><br />
<span class="s1"><br /></span>
<span class="s1">First create the keysapce:</span><br />
<span class="s1"><br /></span>
<span class="s1">
</span><br />
<div class="p1">
<span class="s1">CREATE KEYSPACE killrauction WITH replication = {'class': 'SimpleStrategy' , 'replication_factor': 1 };</span></div>
</div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">1) Get into CQLSH and create a table for users</span></div>
<div class="p2">
<span class="s1">- username</span></div>
<div class="p2">
<span class="s1">- firstname</span></div>
<div class="p2">
<span class="s1">- lastname</span></div>
<div class="p2">
<span class="s1">- emails</span></div>
<div class="p2">
<span class="s1">- password</span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p3">
<span class="s1">- salt for password</span></div>
<div class="p3">
<span class="s1"><br /></span></div>
<div class="p3">
<span class="s1">2) Auction item table (no bids)</span></div>
<div class="p1">
<span class="s1">- name</span></div>
<div class="p1">
<span class="s1">- identifier?</span></div>
<div class="p1">
<span class="s1">- owner</span></div>
<div class="p3">
<span class="s1">
</span></div>
<div class="p2">
<span class="s1">- expiration</span></div>
<div class="p2">
<span class="s1"><br /></span></div>
<div class="p2">
<span class="s1">3) The bids</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Data:</span></div>
<div class="p2">
<span class="s1"> - item identifier</span></div>
<div class="p2">
<span class="s1"> - bid time</span></div>
<div class="p2">
<span class="s1"> - bid user</span></div>
<div class="p2">
<span class="s1"> - bid amount</span></div>
<div class="p2">
<span class="s1"><br /></span></div>
<div class="p2">
<span class="s1">Considerations</span></div>
<div class="p2">
<span class="s1"> - Avoid sorting in the application</span></div>
<div class="p2">
<span class="s1"> - Two bids the same price?</span></div>
<div class="p2">
<span class="s1"> - Really fast sequential access</span></div>
<div class="p2">
</div>
<div class="p3">
<span class="s1"> - Current winner?</span></div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com0tag:blogger.com,1999:blog-4161315644722406995.post-26041942806986017202015-03-27T10:20:00.001-07:002015-03-27T10:24:01.078-07:00Cassandra anti-patterns webinar: Video and Q&ALast week I gave a webinar on avoiding anti-patterns in Cassandra. It was good fun to do and prepare and if you look through my blog most of the sections have a dedicated post.<br />
<br />
Here is the recording:<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/eqOPn5EtR7Q" width="420"></iframe>
<br />
<br />
We got a lot of questions and didn't get to them in the recording so catching up now. If I have missed yours or you think of more then ping me on twitter: @chbatey<br />
<br />
<br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: When is DSE going to support UDTs? </span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
DSE 4.7 will include a certified version of Cassandra 2.1, sometime in the next few months.<br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Can you alter a UDT? </span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
<span style="font-family: inherit;"><span style="background-color: white; white-space: pre-wrap;">Yes see here: </span><span style="white-space: pre-wrap;">http://www.datastax.com/dev/blog/cql-in-2-1</span></span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: with denormalized data, how do you handle a store name change or staff name change? </span><br />
<span style="background-color: white; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; white-space: pre-wrap;">First make sure you need the update, when modelling data immutably that is not often the case. If you need to change a small number of rows I'd do it with a small script/program, large number of rows Apache Spark. </span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: I had the idea that C* 2.x has vector clock, am I wrong? </span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; white-space: pre-wrap;">No Vector clocks in Cassandra, see </span><span style="white-space: pre-wrap;">http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks</span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">
</span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Using the event source model with frequent rollups, would that not generate a 'queueing' style anti-pattern if data from previous rollup period then gets deleted? </span><br />
<span style="background-color: white; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; white-space: pre-wrap;">If you used the same partition and did range queries, yes. But I would use a partition say per day (or what ever the period is that you didn't have rolled up), thus avoiding ever reading over the deleted data.</span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: How would you do the "roll ups" in the account balance calculation example?
</span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
<span style="background-color: white; white-space: pre-wrap;">Most cases I'd do it in application for the first query that required it. It doesn't matter if two threads get to it first as they can both calculate it and the write to the roll up table would be idempotent. If the rollup calculation takes too long and you don't want to slow down a user request with it then you can schedule it in your app or by a different process.</span><br />
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Why would you not use counters for balance?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;">Cassandra counters are more for things like statistics, page views etc. You can't update them atomically and they are slower to update then a pure write.</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: C = Quorum?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="white-space: pre-wrap;">Coordinator</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: How might you go about modeling the "versioning" of time series data so as to avoid updates? I mean where you write a measurement for a particular timestamp and then later on you need to write a new measurement for the same timestamp.</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Use a TimeUUID rather than a Timestamp. Then you can have millions per millisecond.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: If I perform an "if not exist" write and it fails to reach enough replicas, what state can I expect the data to be in? In other words, can I expect the data to not be written to the cluster? </span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">So assuming it for past the if not exists part (for that you'll get applied = false in the response. Then it is like any write. Cassandra will return how many replicas acked the write. You can't be sure that the rest didn't get it as they may have just not have responded.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: I'm wondering if Cassandra could be used to implement distributed locks (Like Redis, Zookeeper)?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">You can with LWTs, here are the details: </span></span><span style="white-space: pre-wrap;">http://www.datastax.com/dev/blog/consensus-on-cassandra</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: In order to emulate a queue without falling on this anti-pattern, can I use the new Date Time Compaction Strategy and TTL?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Answered at the end of the recording</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: And we have 24 table per date. After day we create one table on date and drop table per hour. Is it anti patern.
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Moving table is like moving partition, it does avoid the anti-pattern but it is a lot of work.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Why not change the tombstome grace period to delete quickly?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">You can, but then you need to keep up with repairs which may not be possible.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: What would the use case for using Cassandra in a queueing pattern vs. a traditional message oriented middleware?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">People typically try and use Cassandra as as queue when they already have it in their infrastructure and they need to get messages from one DC to another. This is when they fall into the anti-pattern.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: For the Queue anti-pattern, the > timeuid clause will help on fetch, what about compaction/jvm issues; any recommendations or comments? </span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="font-family: inherit;"><span style="background-color: white; white-space: pre-wrap;">Nothing specifically, the best discussion of Cassandra JVM tuning for GC that I have read is here: </span><span style="white-space: pre-wrap;">https://issues.apache.org/jira/browse/CASSANDRA-8150</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: There are times where data simply cannot be written simultaneously and therefore must be joined at a later time. What do you recommend for joining needs? An external tool such as Spark SQL or ?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Answered at the end of the recording.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Probably one of the best Webinars. Example, were really great. Appreciate DataStax arranging for this. Thanks.
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Okay okay this wasn't a question :)</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Will quorum reads of a partially-successful counter update get the latest info?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Depends on the number of replicas the write for to and at what consistency. You'll get back in the WriteTimeoutException how many acked the write. If it is a QURORUM (e.g 2 if RF = 3) then it will read it, otherwise you don't know.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Can you point to a good read for retry, no rollback?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">On failure modes: http://www.datastax.com/dev/blog/cassandra-error-handling-done-right</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: How would I go about solving limit offset queries, without having to skip rows programmatically, for example taking a simple page 2 customer table?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Just make sure you have a clustering column and start the next limit query from the last result from the previous query.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: You said Cassandra does not do a rollback. Is that true for all cases -- are there any instance where Cassandra would do a rollback?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Not as far as I know.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: I missed the beginning. Are UNLOGGED batches OK to use to speed up writes?
</span><span style="font-family: inherit;"><span style="background-color: white; white-space: pre-wrap;">See: </span><span style="background-color: white; white-space: pre-wrap;">http://christopher-batey.blogspot.com/2015/02/cassandra-anti-pattern-misuse-of.html </span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Great presentation. Regarding the secondary index question, the second one should be much more faster, as it hits the primary key, yes?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Yes, so it only needs to go to a small section of the secondary index table as it knows which node the partition is on.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: which is the best pattern for timeseries
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">This depends on the type of time series, quantity/frequency. What you basically want is partitions that don't grow too large, so in the millions, not hundreds of millions and the use of a TimeUUID as the clustering column.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Are the batch execution started in separate threads when using the the batch optimization?
</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">They will be sent off in parallel, I don't know the threading model here but I imagine they are split on one thread and sent aync. A good question for the cassandra devs who hangout in #cassandra on freenode.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: What approach can be taken with dse, which is C* 2.0 and doesn't have UDT's?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">You can just have a lot of columns! The next DSE version will be 2.1</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Using a time bucket is a way to also prevent the rows from growing too wide (I.e. many millions of columns). Any guidance for the recommended tradeoffs between wide rows with slice queries and more narrow rows and some multi-partition queries?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">There is rarely a general rule for Cassandra, it is all about your data set and read/write frequency. However in general I do my best to keep all reads from a single partition and go out of my way to keep it at most 2. If you have a very high ingest rate and you read for long periods this can get hard and you may need to go to more partitions.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: </span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Do the same rules apply to batch loading when using SSTableLoader and/or the BulkOutputFormat with Hadoop?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">I've never used the BulkOutputFormat with Hadoop. For the SSTableLoader. For the sstableloader command, once you have generated the SS tables then it handles the importing.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: </span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">is BatchType.LOGGED the default for a BatchStatement?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Yes</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: </span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">do we have any ORM framworks for datastax cassandra</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">The DataStax Java/C# driver now have it built in, there is also the less popular SpringData</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: </span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">What if you have constraint to write data in table only if it is different (by different I meant different by all properties which can be 5-10)?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">If you want to write this at a high throughput then I would resolve it at read time as otherwise you'll be doing a read then write which has a lot of race conditions and it a lot slower. IF you include a TimeUUID and write all updates you can then work it out at read time.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: </span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Do tombstones get created with data inserted with a TTL and automatically deleted when expired?</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Yes it generated a tombstone. For immutable timeseries data the new DateTieredCompaction strategy makes deleting this data a lot more efficient.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: </span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Can you go explain a bit more about the de-normalization solution to secondary indexes.</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Write the same data but with a partition key as staff ID and the time as the clustering column. This means you can go to a single partition and do a range query. Even a secondary index with a partition key in the query is worse than this as it has to go to the secondary index table and then do a multi partition query in the original table keyed by customer id.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: </span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Does the removal of a secondary index cause a performance hit during the delete? Assuming you aren't using the index for any queries</span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">Don't know about this one, I've asked around and will update once i get an answer.</span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: </span><span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Question about secondary indexes vs inverted indexes...is inverted superior to secondary? Will global indexes replace inverted indexes?</span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;"><br /></span></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">By inverted I am assuming you mean manually inserting data twice with a different primary key. This will always out perform secondary index as you're storing all the customer events for a staff member on one node and sequentially on disk. For global indexes we'll have to wait and see but that is the idea. The only concern I have is you can specialise the double write to exactly what you want (e.g bucket up staff members or not) where as global indexes will have to be a more general solution.</span></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;"><br /></span></span></div>
<div>
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;">Q: Using the default token split on adding a node in 1.2.x, what issues/symptoms will I experience if I continue to use this method with low numbers of nodes? </span><br />
<span style="background-color: white; font-family: 'Courier New', Courier, monospace, arial, sans-serif; font-size: 14px; white-space: pre-wrap;"><br /></span>
<span style="font-family: inherit;"><span style="background-color: white; white-space: pre-wrap;">I assume you're talking about vnodes as without them you pick the token split. The allocation of tokens with vnodes is well discussed here: </span><span style="background-color: white; white-space: pre-wrap;">https://issues.apache.org/jira/browse/CASSANDRA-7032</span></span></div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com6tag:blogger.com,1999:blog-4161315644722406995.post-59107485098786539352015-03-17T09:38:00.003-07:002015-03-17T09:38:43.445-07:00Using Gradle as a poor man's Cassandra schema management toolI work across a desktop and two laptops so reproducible builds mean a lot to me! I often slate Gradle for being buggy and not doing the simple things well (e.g. dependency management for local development).<br />
<br />
However it is awesome when you want a quick bit of build logic. I wanted to build my schema for a Cassandra application I am working on to keep my various machines up to date.<br />
<br />
So easy in an extensible system like Gradle. I already had my schema creation commands in src/main/resources/schema/tables.cql<br />
<br />
I then added a built script dependency to my build.gradle:<br />
<br />
<script src="https://gist.github.com/chbatey/ce468dc108091e0e056b.js"></script><br />
Then added a few imports and a couple of nifty tasks:<br />
<br />
<script src="https://gist.github.com/chbatey/f3fcfb7bbd928c1ece11.js"></script><br />
Of course this relies on one CQL command per line and isn't exactly liquabase but not bad for 10 minutes hacking.<br />
<br />
Lots of these hacks can lead to very ugly build scripts so be careful :)chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com1tag:blogger.com,1999:blog-4161315644722406995.post-76560760020267315032015-03-16T00:45:00.000-07:002015-03-16T00:46:20.867-07:00Pushing metrics to Graphite from a Spring Boot Cassandra applicationIf you're going down the microservice rabbit whole using frameworks like Spring Boot and Dropwizard it is imperative you can monitor what is going on, part of that is pushing metrics to some type of metrics system.<br />
<br />
The last set of applications I built used Graphite for this purpose, and fortunately the DataStax Java driver stores lots of interesting metrics using the brilliant dropwizard metrics library.<br />
<br />
Here's what it takes to get the Cassandra metrics and your custom metrics from a Spring boot application into Graphite.<br />
<br />
This article assumes you know how to use the DataStax Cassandra driver and the Dropwizard metrics library and you're familiar with tools like Maven and Gradle. If you don't go read up on those first.<br />
<br />
First let's get the Cassandra driver and metrics libraries on our classapth, here is my example using Gradle:<br />
<br />
<script src="https://gist.github.com/chbatey/79cd5945c1fb6c9a1c58.js"></script><br />
I've included the Actuator from Spring boot as well.<br />
<br />
Assuming you have a bean that is your Cassandra Session add a bean to expose the MetricRegistry and to create a GraphiteReporter:<br />
<br />
<script src="https://gist.github.com/chbatey/ed5ae8f51131abc75cb3.js"></script>
Here I have a graphite server running on 192.168.10.120. If you don't want to install Graphite to try this out I have a Vagrant VM on my <a href="https://github.com/chbatey/graphite-vm">GitHub</a> which launches Graphtie + Graphana.<br />
<br />
If we had the Cluster as a bean rather than the Session we'd have injected that. We've now set it up so that all the metrics the DataStax Java driver records will be published to Graphite every 30 seconds.<br />
<br />
Now we can plot all kinds of graphs:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOfGAKzOuGdn5VpefaBsJtxzUY0pKc02ZHCJ7al64WKKwyivZ7Wd5PTOmiuMrx65chzgmkon84jBVarZgH6jMA09dk4qjyvYtwPqI8w9ysT9mYX9JD5I6bW75qKQnnxQ8ruxRLjSIjtB-s/s1600/Screenshot+2015-02-23+21.50.29.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOfGAKzOuGdn5VpefaBsJtxzUY0pKc02ZHCJ7al64WKKwyivZ7Wd5PTOmiuMrx65chzgmkon84jBVarZgH6jMA09dk4qjyvYtwPqI8w9ysT9mYX9JD5I6bW75qKQnnxQ8ruxRLjSIjtB-s/s1600/Screenshot+2015-02-23+21.50.29.png" height="268" width="400" /></a></div>
<br />
<br />
For instance we can plot request times, number of errors, number of requests, etc. This becomes even more powerful when you are deploying multiple versions of your application and you pre-fix each instance with a identifier such as its IP.<br />
<br />
<h3>
Adding our own metrics with annotations</h3>
<div>
<br /></div>
<div>
The next step is to add more metrics, as the ones in the DataStax library aren't very fine grained, for example we might want to time particular queries, or look at our response times.</div>
<div>
<br /></div>
<div>
You can do this manually but it is easier with annotations. We can do this with the <a href="http://www.ryantenney.com/metrics-spring/">Metric-Spring</a> project. This project integrates Spring AOP with drop wizard metrics.</div>
<div>
<br /></div>
<div>
However it is quite fiddly to get working as we now have three libraries that want to create a MetricRegistry: SpringBoot, Cassandra Driver and Metric-Spring.</div>
<div>
<br /></div>
<div>
To get everyone to use the Cassandra driver's MetricRegistry we need to create a MetricsConfigurerAdapter:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/079d88a1577736ef1a30.js"></script></div>
<div>
The reason we're injecting the Session is we can no longer register a bean for the MetricRegistry as Spring-Metric does this and we don't want to end up with two. To get this to work we have to remove the metricRegistry bean from the code above. The other thing we do is add the EnableMetric annotation to our Application class:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxszpH6ZdRwELpIs3rdTup7N_Wruwuoc5yjt_Suge0yaeVfqxFnWPDsufLNu1G7Nzn7VQg36eZ4sFoz6pN_s4nqKSmVmQHHhMycnQ_1iEGpbVHttOUpsQNeaffSgT99bLPvGUbdpY4WUUn/s1600/Screenshot+2015-02-23+21.59.21.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxszpH6ZdRwELpIs3rdTup7N_Wruwuoc5yjt_Suge0yaeVfqxFnWPDsufLNu1G7Nzn7VQg36eZ4sFoz6pN_s4nqKSmVmQHHhMycnQ_1iEGpbVHttOUpsQNeaffSgT99bLPvGUbdpY4WUUn/s1600/Screenshot+2015-02-23+21.59.21.png" height="110" width="400" /></a></div>
<br />
Once all this is done we can annotate our public methods with @Timed like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE8Bvx-Lg_e2Ju32mvyHNEfWSjm2WxmmCKFrQPros4IstQSVLQLWePwycSZP2xpH6-ymdlQvJXu7TeuTcYw7PUoXmBPDgaLdI0WERPRHe2_ZgBlFoVBkxnq7SUIR-mAXyA8IdM68-X_OLw/s1600/Screenshot+2015-02-23+22.00.15.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE8Bvx-Lg_e2Ju32mvyHNEfWSjm2WxmmCKFrQPros4IstQSVLQLWePwycSZP2xpH6-ymdlQvJXu7TeuTcYw7PUoXmBPDgaLdI0WERPRHe2_ZgBlFoVBkxnq7SUIR-mAXyA8IdM68-X_OLw/s1600/Screenshot+2015-02-23+22.00.15.png" height="91" width="400" /></a></div>
<br /></div>
Then in Graphite we can see them, their name is derived from the fully qualified method name.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheIXKGA6CQGOodpzJQjcvtbDT5GwPvytVovB1dGWfdP3gDET5sozBEVUEPv7R9IuA4VsTBcrFmlrTfRxkCmVSTw6jdIu0uc5j7JMAqgvikGY57rO1JK2oscEX8SMv8S6KRN2tSMJ-KqoWP/s1600/Screenshot+2015-02-23+22.01.33.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheIXKGA6CQGOodpzJQjcvtbDT5GwPvytVovB1dGWfdP3gDET5sozBEVUEPv7R9IuA4VsTBcrFmlrTfRxkCmVSTw6jdIu0uc5j7JMAqgvikGY57rO1JK2oscEX8SMv8S6KRN2tSMJ-KqoWP/s1600/Screenshot+2015-02-23+22.01.33.png" height="198" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
So now our Spring Boot application has Cassandra metrics and our own custom application metrics all pushing to Graphite!</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The whole application is on <a href="https://github.com/chbatey/killrauction">GitHub</a> if you want the full Spring config and dependencies.</div>
<br />chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com1tag:blogger.com,1999:blog-4161315644722406995.post-59297347827291771302015-03-12T23:30:00.000-07:002015-03-12T23:30:12.565-07:00Cassandra schema migrations made easy with Apache SparkBy far the most common question I get asked when talking about Cassandra is once you've denormalised based on your queries what happens if you were wrong or a new requirement comes in that requires a new type of query.<br />
<br />
First I always check that it is a real requirement to be able to have this new functionality on old data. If that's not the case, and often it isn't, then you can just start double/triple writing into the new table.<br />
<br />
However if you truly need to have the new functionality on old data then Spark can come to the rescue. The first step is to still double write. We can then backfill using Spark. The awesome thing is that nearly all writes in Cassandra are idempotent, so when we backfill we don't need to worry about inserting data that was already inserted via the new write process.<br />
<br />
Let's see an example. Suppose you were storing customer events so you know what they are up to. At first you want to query by customer/time so you end up with following table:<br />
<br />
<script src="https://gist.github.com/chbatey/15486be113cc2ed06f16.js"></script><br />
Then the requirement comes in to be able to look for events by staff member. My reaction a couple of years ago would have been something like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh50OdO955txRIwRcmBrP16qhz-OAA21uh6kuvRz3Zc8CLxH9OivvKlC_MG944bwLQdmgEVHxpk3ChFJvddk8l62QrExyzg72nEWNntZXnMadhc4WLxm5wGqSIZHoh3SKjMmb6muyb1A1Hy/s1600/oh-noes-everybody-panic.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh50OdO955txRIwRcmBrP16qhz-OAA21uh6kuvRz3Zc8CLxH9OivvKlC_MG944bwLQdmgEVHxpk3ChFJvddk8l62QrExyzg72nEWNntZXnMadhc4WLxm5wGqSIZHoh3SKjMmb6muyb1A1Hy/s1600/oh-noes-everybody-panic.gif" /></a></div>
<br />
However if you have Spark workers on each of your Cassandra nodes then this is not an issue.<br />
<br />
Assuming you want to a new table keyed by staff_id and have modified your application to double write you do the back fill with Spark. Here's the new table:<br />
<br />
<script src="https://gist.github.com/chbatey/dff6f9e5a4363a2672ef.js"></script><br />
Then open up a Spark-shell (or submit a job) with the Spark-Cassandra connector on the classpath and all you'll need is something like this:<br />
<br />
<script src="https://gist.github.com/chbatey/1286acb7dd2bf70873ec.js"></script><br />
How can a few lines do so much! If you're in a shell obviously you don't even need to create a SparkContext. What will happen here is the Spark workers will process the partitions on a Cassandra node that owns the data for the customer table (original table) and insert it back into Cassandra locally. Cassandra will then handle the replication to the correct nodes for the staff table.<br />
<br />
This is the least network traffic you could hope to achieve. Any solution that you write your self with Java/Python/Shell will involve pulling the data back to your application and pushing it to a new node, which will then need to replicate it for the new table.<br />
<br />
You won't want to do this at a peak time as this will HAMMER you Cassandra cluster as Spark is going to do this quickly. If you have a small DC for just running the Spark jobs and let it asynchronously replicate to your operational DC this is less of a concern.chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com4tag:blogger.com,1999:blog-4161315644722406995.post-50656370468167309372015-03-11T23:43:00.000-07:002015-03-11T23:43:20.309-07:00Cassandra anti-pattern: Logged batchesI've previously blogged about other anti-patterns:<br />
<ol>
<li>Distributed joins</li>
<li>Unlogged batches</li>
</ol>
<div>
This post is similar to the unlogged batches post but is instead about logged batches.</div>
<div>
<br /></div>
<div>
We'll again go through an example Java application.<br />
<br />
The good news is that the common misuse is virtually the same as the last article on unlogged batches, so you know what not to do. The bad news is if you do happen to misuse them it is even worse!<br />
<br />
Let's see why. Logged batches are used to ensure that all the statements will eventually succeed. Cassandra achieves this by first writing all the statements to a batch log. That batch log is replicated to two other nodes in case the coordinator fails. If the coordinator fails then another replica for the batch log will take over.<br />
<br />
Now that sounds like a lot of work. So if you try to use logged batches as a performance improvement then you'll be very disappointed! For a logged batch with 8 insert statements (equally distributed) in a 8 node cluster it will look something like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaW9d771fGONX4nlhBWOnbjbLtNn9wvYf5pinU0dW-a8jSF7UEaidQSmN9H98lI3PCS9v2H5oyUsJ0w5d4NDrrLIo556KrMxbnHm5JwSD8RAPfqx1OOfv0IHhWEyvbXzmTEKB7ofwSAt-R/s1600/Screenshot+2015-02-10+13.51.56.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaW9d771fGONX4nlhBWOnbjbLtNn9wvYf5pinU0dW-a8jSF7UEaidQSmN9H98lI3PCS9v2H5oyUsJ0w5d4NDrrLIo556KrMxbnHm5JwSD8RAPfqx1OOfv0IHhWEyvbXzmTEKB7ofwSAt-R/s1600/Screenshot+2015-02-10+13.51.56.png" height="247" width="320" /></a></div>
<br />
The coordinator has to do a lot more work than any other node in the cluster. Where if we were to just do them as regular inserts we'd be looking like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVZnjIp2DmnKiiC4p9CDImhLVBwLAT8gn65A3pShreVlyaBOl8pOWagJuxZC3nnB9RHMK6OVggXmWaR9W1DQVYnzrx4KWM593EvM5wuNc3-D5-jcw1btUNcPHl0AdN1lKG2_rqPRFrPokX/s1600/Screenshot+2015-02-09+14.03.16.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVZnjIp2DmnKiiC4p9CDImhLVBwLAT8gn65A3pShreVlyaBOl8pOWagJuxZC3nnB9RHMK6OVggXmWaR9W1DQVYnzrx4KWM593EvM5wuNc3-D5-jcw1btUNcPHl0AdN1lKG2_rqPRFrPokX/s1600/Screenshot+2015-02-09+14.03.16.png" height="257" width="320" /></a></div>
<br />
A nice even workload.<br />
<br />
<h4>
So when would you want to use logged batches?</h4>
</div>
<div>
<br />
Short answer: consistent denormalisation. In most cases you won't want to use them, they are a performance hit. However for some tables where you have denormalised you can decide to make sure that both statements succeed. Lets go back to our customer event table from the previous post but also add a customer events by staff id table:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/31cd5b1e3406d1d2a2cb.js"></script></div>
<div>
<br /></div>
<div>
We could insert into this table in a logged batch to ensure that we don't end up with events in one table and not the other. The code for this would look like this:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/c0d92981f92cd6feef5a.js"></script></div>
<div>
<br /></div>
<div>
This would mean both inserts would end up in the batch log and be guaranteed to eventually succeed.</div>
<div>
<br /></div>
<div>
The downside is this adds more work and complexity to our write operations. Logged batches have two opportunities to fail:</div>
<div>
<ol>
<li>When writing to the batch log</li>
<li>When applying the actual statements</li>
</ol>
<div>
Let's forget about reads as they aren't destructive and concentrate on writes. If the first phase fails Cassandra returns a WriteTimeoutException with write type of BATCH_LOG. This you'll need to retry if you want your inserts to take place. </div>
</div>
<div>
<br /></div>
<div>
If the second phase fails you'll get a WriteTimeoutException with the write type of BATCH. This means it made it to the batch log so that they will get replayed eventually. If you definitely need to read the writes you would read at SERIAL, meaning any committed batches would be replayed first.<br />
<br />
<h4>
Conclusion</h4>
</div>
<div>
<br />
Logged batches are rarely to be used, they add complexity if you try to read at SERIAL after failure and they are a performance hit. If you are going to use them it is in the odd situation where you can't handle inconsistencies between tables. They allow you to guarantee the updates will eventually happen, they do not however offer isolation i.e a client can see part of the batch before it is finished. </div>
<div>
<br /></div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com0tag:blogger.com,1999:blog-4161315644722406995.post-24238938252663878302015-02-23T09:36:00.001-08:002015-02-23T09:36:27.896-08:00Spring Security + Basic Auth + MD5Password encoding with salt all stored in CassandraI've just put together a simple Spring boot application that has REST endpoints secured by basic auth with the users stored in Cassandra. I want the application to be completely stateless and will assume access is over HTTPS.<br />
<br />
I found it surprisingly difficult to plug all this together with Java config, there are very few complete examples so I ended up spending more time looking at the Spring source than I expected. Ah well that just confirms my love of using open source libraries and frameworks.<br />
<br />
Essentially you need an extension of the <span style="background-color: white; color: #a9b7c6; font-family: Menlo;"><span style="font-size: x-small;">WebSecurityConfigurerAdapter</span></span><span style="background-color: white; color: #795da3; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 12px; line-height: 16.7999992370605px; white-space: pre;"> </span>class where you can programatically add your own <span style="background-color: white; color: #a9b7c6; font-family: Menlo;"><span style="font-size: x-small;">UserDetailsService</span></span>.<br />
<br />
Here's my example, I'll explain it below.<br />
<br />
<script src="https://gist.github.com/chbatey/3a6e3a13ee2bfc02ccba.js"></script><br />
Line 11: I've injected the MD5PasswordEncoder as I also use in the code that handles the creation of users in the database.<br />
<br />
Line 14-22: Here is where we configure our custom <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">UserDetailsService </span>which I'll show later. We don't want to store user's passwords directly so we use the built in <span style="color: #a9b7c6; font-family: Menlo; font-size: x-small;"><span style="background-color: white;">MD5PasswordEncoder</span></span>. Just using a one way hash isn't good enough as people can break this with reverse lookup tables so we also want to sprinkle in some salt. Our implementation of the <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">UserDetailsService </span>will have a field called Salt and we use the <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">ReflectiveSaltSource </span>to pick it up. Given how common salting passwords is I was surprised there wasn't a separate interface where this was explicit, but ah well.<br />
<br />
Line 25-34: Here we define what type of security we want, we tell Spring security to be stateless so it doesn't try and store anything in the container's session store. Then we enable BasicAuth and define the URLs we want to be authorised. The API for creating users is not authorised for obvious reasons.<br />
<br />
Next we want to build an implementation of the <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">UserDetailsService</span> interface that checks Cassandra.<br />
<br />
I won't go through the Cassandra code in the blog but just assume we have a DAO with the following interface:<br />
<br />
<script src="https://gist.github.com/chbatey/d11a4a27d729c5241814.js"></script><br />
If you're interested in the Cassandra code then checkout the while project from GitHub.<br />
<br />
With that interface our <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">UserDetailsService </span>looks like this:<br />
<br />
<script src="https://gist.github.com/chbatey/e288d7431dc1a2f84ebb.js"></script><br />
Here we use the awesome Optional + Lambda to throw if the user doesn't exist. Our DAO interface doesn't use Runtime exceptions as I like type systems, but this is a nice pattern to convert between a Optional and a library expecting exceptions.<br />
<br />
The UserWithSalt is an extension of the Spring's User, with one extra field that the <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">ReflectiveSaltSource </span>will pick up for salting passwords.<br />
<br />
<script src="https://gist.github.com/chbatey/1d74914ee345e51a6163.js"></script><br />
That's pretty much it, when a request comes in Spring security will check if the path is authorised, if it is it will get the user details from our <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">UserDetailsService </span>and check the password my using the <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">ReflectiveSaltSource </span>and <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">MD5PasswordEncoder</span>. So our database only has the MD5 password and the salt used to generate it. The salt is self is generated using the Java <span style="background-color: white; color: #a9b7c6; font-family: Menlo; font-size: x-small;">SecureRandom</span> when users are created.<br />
<br />
Full source code is at <a href="https://github.com/chbatey/killrauction">GitHub</a> and I've created the branch blog-spring-security in case you're reading this in the future and it has all changed!<br />
<br />
<br />
<br />chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com7tag:blogger.com,1999:blog-4161315644722406995.post-17285722693168422862015-02-18T07:13:00.000-08:002015-02-18T07:13:18.213-08:00A simple MySql to Cassandra migration with SparkI previously blogged about a Cassandra anti-pattern: <a href="http://christopher-batey.blogspot.co.uk/2015/02/cassandra-anti-pattern-distributed.html" target="_blank">Distributed joins</a>. This commonly happens when people move from a relational database to a Cassandra. I'm going to use the same example to show how to use Spark to migrate data that previously required joins into a denormalised model in Cassandra.<br />
<br />
So let's start with a simple set of tables in MySQL that store customer event information that references staff members and a store from a different table.<br />
<br />
<script src="https://gist.github.com/chbatey/b1e04f55f8c5f2133426.js"></script><br />
Insert a few rows (or a few million)<br />
<br />
<script src="https://gist.github.com/chbatey/eef906a61f52eca0c511.js"></script><br />
<div>
Okay so we only have a few rows but imagine we had many millions of customer events and in the order of hundreds of staff members and stores.<br />
<br />
Now let's see how we can migrate it to Cassandra with a few lines of Spark code :)<br />
<br />
Spark has built in support for databases that have a JDBC driver via the <a href="https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/rdd/JdbcRDD.html" target="_blank">JdbcRDD</a>. Cassandra has great support for Spark via <a href="https://github.com/datastax/spark-cassandra-connector" target="_blank">DataStax's open source connector</a>. We'll be using the two together to migrate data from MySQL to Cassandra. Prepare to be shocked how easy this is...<br />
<br />
Assuming you have Spark and the connector on your classpath you'll need these imports:<br />
<br />
<script src="https://gist.github.com/chbatey/0ca630e2c4a0ea191578.js"></script><br />
Then we can create our SparkContext and it also adds the Cassandra methods to the context and to RDDs.<br />
<br />
<script src="https://gist.github.com/chbatey/b224ef4d1bb00a03d1d6.js"></script>
</div>
My MySQL server is running on IP 192.168.10.11 and I am connecting very securely with with user root and password password.<br />
<br />
Next we'll create the new Cassandra table, if yours already exists skip this part.<br />
<br />
<script src="https://gist.github.com/chbatey/8b8465a7a33665e55d33.js"></script><br />
Then it is time for the migration!
<br />
<br />
<script src="https://gist.github.com/chbatey/792f1e2885e70f4d376d.js"></script><br />
We first create an JdbcRDD allowing MySQL to do the join. You need to give Spark a way to partition the MySql table, so you give it a statement with variables in and a starting index and a final index. You also tell Spark how many partitions to split it into, you want this to be greater than the number of cores in your Spark cluster so these can happen concurrently.<br />
<br />
Finally we save it to Cassandra. The chances are this migration will be bottle necked by the queries to MySQL. If the Store and Staff table are relatively small it would be worth bringing them completely in to memory, either as an RDD or as an actual map so that MySQL doesn't have to join for every partition.<br />
<br />
Assuming your Spark workers are running on the same servers as your Cassandra nodes the partitions will be spread out and inserted locally to every node in your cluster.<br />
<br />
This will obviously hammer the MySQL server so beware :)<br />
<br />
The full source file is on <a href="https://github.com/chbatey/spark-sandbox/blob/master/src/main/scala/RdmsToCassandra.scala" target="_blank">Github</a>.chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com7tag:blogger.com,1999:blog-4161315644722406995.post-38517542327127723542015-02-09T06:41:00.000-08:002015-02-09T06:48:21.181-08:00Cassandra anti-pattern: Misuse of unlogged batches<div>
This is my second post in a series about Cassandra anti-patterns,<a href="http://christopher-batey.blogspot.co.uk/2015/02/cassandra-anti-pattern-distributed.html" target="_blank"> here's the first on distributed joins</a>. This post will be on unlogged batches and the next one on logged batches.</div>
<div>
<br /></div>
Batches are often misunderstood in Cassandra. They will rarely increase performance, that is not their purpose. That can come as quite the shock to someone coming from a relational database.<br />
<div>
<br /></div>
<div>
Let's understand why this is the case with some examples. In my last post on Cassandra anti-patterns I gave all the examples inside CQLSH, however let's write some Java code this time.</div>
<div>
<br /></div>
<div>
We're going to store and retrieve some customer events. Here is the schema: </div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/f5cfdd1627652a790906.js"></script></div>
<div>
<br /></div>
<div>
Here's a simple bit of Java to persist a simple value object representing a customer event, it also creates the schema and logs the query trace.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/7423c409066441b65161.js"></script></div>
<div>
<br /></div>
<div>
We're using a prepared statement to store one customer event at a time. Now let's offer a new interface to batch insert as we could be taking these of a message queue in bulk.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/00c6e32e8a9bcab4564a.js"></script></div>
<div>
<br /></div>
<div>
It might appear naive to just implement this with a loop:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/f28df3175dc43028d13b.js"></script></div>
<div>
<br /></div>
<div>
However, apart from the fact we'd be doing this synchronously, this is actually a great idea! Once we made this async then this would spread our inserts across the whole cluster. If you have a large cluster, this will be what you want.</div>
<div>
<br /></div>
<div>
However, a lot of people are used to databases where explicit batching is a performance improvement. If you did this in Cassandra you're very likely to see the performance reduce. You'd end up with some code like this (the code to build a single bound statement has been extracted out to a helper method):</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/e256ceca0815215c31b9.js"></script></div>
<div>
<br /></div>
<div>
Looks good right? Surely this means we get to send all our inserts in one go and the database can handle them in one storage action? Well, put simply, no. Cassandra is a distributed database, no single node can handle this type of insert even if you had a single replica per partition.<br />
<br />
What this is actually doing is putting a huge amount of pressure on a single coordinator. This is because the coordinator needs to forward each individual insert to the correct replicas. You're losing all the benefit of token aware load balancing policy as you're inserting different partitions in a single round trip to the database.<br />
<br />
If you were inserting 8 records in a 8 node cluster, assuming even distribution, it would look a bit like this:</div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKovuvavBTOpaNjgd6mSgCmRPO54FTeL5vr-b4W6nbqbRf8MTmuOp50OKS5Sz9-NjLTEnbzu1N6rvCsKG0JCKpKwQB9S-1nLW8ZkkhMgsG_84cf6HVW_Cukvbm5RljR_Ggip-j_N0T19oZ/s1600/Screenshot+2015-02-09+13.59.08.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKovuvavBTOpaNjgd6mSgCmRPO54FTeL5vr-b4W6nbqbRf8MTmuOp50OKS5Sz9-NjLTEnbzu1N6rvCsKG0JCKpKwQB9S-1nLW8ZkkhMgsG_84cf6HVW_Cukvbm5RljR_Ggip-j_N0T19oZ/s1600/Screenshot+2015-02-09+13.59.08.png" height="186" width="400" /></a></div>
</div>
<div>
<br /></div>
<div>
Each node will have roughly the same work to do at the storage layer but the Coordinator is overwhelmed. I didn't include all the responses or the replication in the picture as I was getting sick of drawing arrows! If you need more convincing you can also see this in the trace. The code is checked into Github so you can run it your self. It only requires a locally running Cassandra cluster.<br />
<br />
<h4>
Back to individual inserts</h4>
</div>
<div>
<br />
If we were to keep them as normal insert statements and execute them asynchronously we'd get something more like this:</div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4gqDGVoqQGyhGDB1bbpuriW5hrwoxat7UROh1rwfY2jFZV3uCt5B_EAiueWtKYzWvLFG_PQtxRtgVvgHli0zXf2DKhs5K8tA-M39bZPMeWXpMFoWDPzpI4kwqSjye_LcH9v9kgsLoKhKI/s1600/Screenshot+2015-02-09+14.03.16.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4gqDGVoqQGyhGDB1bbpuriW5hrwoxat7UROh1rwfY2jFZV3uCt5B_EAiueWtKYzWvLFG_PQtxRtgVvgHli0zXf2DKhs5K8tA-M39bZPMeWXpMFoWDPzpI4kwqSjye_LcH9v9kgsLoKhKI/s1600/Screenshot+2015-02-09+14.03.16.png" height="257" width="320" /></a></div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Perfect! Each node has roughly the same work to do. Not so naive after all :)</div>
<div>
<h4>
<br /></h4>
<h4>
So when should you use unlogged batches?</h4>
</div>
<div>
<br />
How about if we wanted to implement the following method:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/chbatey/aca06fd4daa8d999681d.js"></script></div>
<div>
<br /></div>
<div>
Looks similar - what's the difference? Well customer id is the partition key, so this will be no more coordination work than a single insert and it can be done with a single operation at the storage layer. What does this look like with orange circles and black arrows?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIXUW0NuJixKLUEBdoZH-HrJysN7icZ5vW_zdWYx-sgGFLhrT4HP77jMXZvGl7lVMpnC9hQZmKq_xKJPkZ9prTg8MDuP-dLweJk0bo_mUUq4WMaqxrauNTf__h-idfbdy-Ob307yZ4mpjk/s1600/Screenshot+2015-02-09+14.35.37.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIXUW0NuJixKLUEBdoZH-HrJysN7icZ5vW_zdWYx-sgGFLhrT4HP77jMXZvGl7lVMpnC9hQZmKq_xKJPkZ9prTg8MDuP-dLweJk0bo_mUUq4WMaqxrauNTf__h-idfbdy-Ob307yZ4mpjk/s1600/Screenshot+2015-02-09+14.35.37.png" height="314" width="320" /></a></div>
<br /></div>
<div>
Simple! Again I've left out replication to make it comparable to the previous diagrams.<br />
<br />
<h4>
Conclusion</h4>
</div>
<div>
<br /></div>
<div>
Most of the time you don't want to use unlogged batches with Cassandra. The time you should consider it is when you have multiple inserts/updates for the same partition key. This allows the driver to send the request in a single message and the server to handle it with a single storage action. If batches contain updates/inserts for multiple partitions you eventually just overload coordinators and have a higher likelihood of failure.<br />
<br />
The code examples are on github <a href="https://github.com/chbatey/cassandra-anti-patterns/tree/master" target="_blank">here</a>.</div>
<div>
<br /></div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com15tag:blogger.com,1999:blog-4161315644722406995.post-75916230970180610542015-02-03T07:29:00.000-08:002015-02-03T07:42:30.694-08:00Testing Cassandra applications: Stubbed Cassandra 0.6.0 released<a href="http://scassandra.org/" target="_blank">Stubbed Cassandra</a> (Scassandra) is an open source test double for Cassandra. Martin Fowler has a<a href="http://www.martinfowler.com/bliki/TestDouble.html" target="_blank"> very general definition</a> of what a test double actually is.<br />
<br />
When I refer to a test double I mean stubbing out at the protocol level. So if your application makes calls over HTTP, your test double acts as an HTTP server where you can control the responses, and most importantly: inject faults. <a href="http://wiremock.org/" target="_blank">Wiremock</a> is a great example of a test double for HTTP.<br />
<br />
I like this kind of stubbing out as it allows me to really test drivers / network issues etc. Deploying to cloud environments where network/servers going down happens more frequently makes this even more important. If you're using a JVM language and all this happens in the same JVM it is also quick.<br />
<br />
<h4>
Why is this release important?</h4>
<br />
This is an important release for Scassandra as it now supports all types, previously it only supported the subset of CQL that my old employer, BSkyB, used. Now's a good time to mention that this tool was developed completely in my own time and not while working there :)<br />
<br />
It still has lots of limitations (no user defined types, no batch statements, no LWT) but as it is designed to test individual classes it is still usable for all your code that doesn't use these features even if they are used some where in your application.<br />
<br />
I had previously used it for full integration tests, in that case it had to support your entire schema. I have stopped doing that as I intend to build a different type of Cassandra testing tool for that using <a href="https://issues.apache.org/jira/browse/CASSANDRA-6659" target="_blank">CASSANDRA-6659</a>. This JIRA extracted an interface for handling queries, which I want to use to inject faults/delays etc. If you haven't used Scassandra before it is important to know it doesn't embed a real Cassandra, it just implements there server side of the native protocol and "pretends" to be Cassandra.<br />
<br />
Version 0.6.0 has a view of Cassandra 3.0, where embedded collections are likely to be supported. Previously you used an enum to inform Scassnadra what your column types are, or the variables in prepared statements. For example:<br />
<br />
<script src="https://gist.github.com/chbatey/c8379544ab4d7ceee7f0.js"></script><br />
Here the <b>withColumnTypes</b> method on the builder informs Scassandra how to serialise the rows passed into <b>withRows</b>.<br />
<br />
This worked for primitive types e.g Varchar, Text. But what about collections? Sets were supported first so I went with VarcharSet etc, bad idea! What about Maps? That is a lot of combinations, and even worse List<Map<String, Int>>?<br />
<br />
An enum was a bad idea, so in 0.6.0 I've introduced the CqlType, this has sub classes for Primitive/Collections and there is a set of static methods and constants to make it nearly as convenient as an enum for the simple types. The advantage of this is I can now embed types inside each other e.g<br />
<br />
<script src="https://gist.github.com/chbatey/d097b1f25ac5b48324d6.js"></script><br />
And then when Cassandra 3.0 comes we can have things like map(TEXT, map(TEXT, TEXT)) for a multi map.<br />
<br />
The end goal is actually for you to give your schema to Scassandra and it will just work this out. This is some way off as it requires being able to parse CQL and at the moment Scassandra just pattern matches against your queries.<br />
<br />
Happy testing and as always any feature requests/feedback just ping me on twitter <a href="https://twitter.com/chbatey" target="_blank">@chbatey</a>chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com17tag:blogger.com,1999:blog-4161315644722406995.post-33836025675232867082015-02-03T01:55:00.002-08:002015-02-03T07:44:40.082-08:00Unit testing Kafka applicationsI recently started working with Kafka. The first thing I do when start with a tech is work out how I am going to write tests as I am a TDD/XP nut.<br />
<br />
For HTTP I use <a href="http://wiremock.org/" target="_blank">Wiremock</a>, for Cassandra I wrote a test double called <a href="http://scassandra.org/" target="_blank">Stubbed Cassandra.</a> The term test double comes from the awesome book <a href="http://www.amazon.co.uk/Release-It-Production-Ready-Pragmatic-Programmers/dp/0978739213" target="_blank">Release It!</a> where it recommends for each technology you integrate with having a test double that you can prime to fail in every way possible.<br />
<br />
I couldn't find anything for Kafka but I did find a couple of<a href="http://pannoniancoder.blogspot.co.uk/2014/08/embedded-kafka-and-zookeeper-for-unit.html" target="_blank"> blogs and gists</a> for people running Kafka/Zookeeper in the same JVM as tests.<br />
<br />
That's a start, I took it one step further and wrote a version that will hide away all the details, including a JUnit rule so you don't even need to start/stop it for tests as well as convenient methods to send and receive messages. Here's an example of an integration test for the KafkaUnit class:<br />
<br />
<script src="https://gist.github.com/chbatey/ab8a0bb4ee3eb8345c3f.js"></script>
<br />
Let's say you have some code that sends a message to Kafka, like this:<br />
<br />
<script src="https://gist.github.com/chbatey/09e4b2837feaffd5c6eb.js"></script>
A unit test would look something like this:<br />
<br />
<script src="https://gist.github.com/chbatey/d8c0ac2fc49d399cd16f.js"></script>
It is in Maven Central, so if you want to use it just add the following dependency:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><dependency></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> <groupId>info.batey.kafka</groupId></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> <artifactId>kafka-unit</artifactId></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> <version>0.1.1</version></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"></dependency></span><br />
<br />
If you want to contribute check it out on <a href="https://github.com/chbatey/kafka-unit" target="_blank">github</a>.<br />
<br />
It is pretty limited so far, assumed String messages etc. If I keep working with Kafka I'll extend it and add support for injecting faults etc. Also for the next version I'll come up with a versioning mechanism that includes the Kafka version.chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com10tag:blogger.com,1999:blog-4161315644722406995.post-38493942048687309452015-02-02T04:24:00.000-08:002015-02-03T07:44:55.718-08:00Cassandra anti-pattern: Distributed joins / multi-partition queries<div class="p1">
<span class="s1">There’s a reason when you shard a relational databases you are then prevented from doing joins. Not only will they be slow and fraught with consistency issues but they are also terrible for availability. For that reason Cassandra doesn’t even let you join as every join would be a distributed join in Cassandra (you have more that one node right?).</span></div>
<div class="p2">
<span class="s1"></span><br /></div>
<div class="p1">
<span class="s1">This often leads developers to do the join client side in code. Most of the time this is a bad idea, but let’s understand just how bad it can be.</span></div>
<div class="p2">
<span class="s1"></span><br /></div>
<div class="p1">
<span class="s1">Let’s take an example where we want to store what our customers are up to, here’s what we want to store:</span></div>
<div class="p1">
</div>
<ul>
<li>Customer event</li>
<ul>
<li> customer_id e.g ChrisBatey</li>
<li> staff_id e.g Charlie</li>
<li> event_type e.g login, logout, add_to_basket, remove_from_basket</li>
<li> time</li>
</ul>
<li>Store</li>
<ul>
<li>name</li>
<li>store_type e.g Website, PhoneApp, Phone, Retail</li>
<li>location</li>
</ul>
</ul>
<div class="p1">
<span class="s1">We want to be able to do retrieve the last N events, time slices and later we’ll do analytics on the whole table. Let’s get modelling! We start off with this:</span></div>
<br />
<script src="https://gist.github.com/chbatey/38258aca848a1bae15ff.js"></script>
<br />
<div class="p1">
<span class="s1">This leads us to query the customer events table, then if we want to retrieve the store or staff information we need to do another query. This can be visualised as following (query issued at QUORUM with a RF of 3):</span><br />
<span class="s1"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuzuxNcqmalb7BJb3SLoum_HajeagYMD0TLxMq5Y7KYVPnGiEdlxfNWoeXHXyHBegiSljylgiJ5vdcuYPJCpef9col6CwYUpya3h2UZcjym6-hdXvVDtRT3Ic3VJvVlO7UEfdAXUSBKHg1/s1600/ClusterWithTwoQueries.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuzuxNcqmalb7BJb3SLoum_HajeagYMD0TLxMq5Y7KYVPnGiEdlxfNWoeXHXyHBegiSljylgiJ5vdcuYPJCpef9col6CwYUpya3h2UZcjym6-hdXvVDtRT3Ic3VJvVlO7UEfdAXUSBKHg1/s1600/ClusterWithTwoQueries.png" height="248" width="400" /></a></div>
<span class="s1"><br /></span>
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">For the second query we’ve used a different coordinator and have gone to different nodes to retrieve the data as it is in a different partition.</span></div>
<div class="p2">
<span class="s1"></span><br /></div>
<div class="p1">
<span class="s1">This is what you’d call a one to one relationship for a single query but in reality it is a many to one as no doubt many customer events reference the same store. By doing a client side join we are relying on a lot of nodes being up for both our queries to succeed.</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1">We’d be doing a similar thing for staff information. But Let’s make things worse by changing the staff relationship so that we can associate multiple staff members with a single customer event.</span></div>
<br />
<script src="https://gist.github.com/chbatey/d7611071cc50f0640f24.js"></script>
The subtle difference here is that the staff column is now a set. This will lead to query patterns like:<br />
<br />
<script src="https://gist.github.com/chbatey/5da761c93032af6189aa.js"></script>
This looks good right? We’re querying by partition id in the staff table. However it isn’t as innocent as it looks. What we’re asking the coordinator do do now is query for multiple partitions, meaning it will only succeed if there are enough replica up for them all. Let’s use trace to see how this would work in a 6 node cluster:<br />
<br />
<script src="https://gist.github.com/chbatey/1bf953b5c873153482aa.js"></script>
Here I've span up a 6 node cluster on my machine (I have a lot of RAM) with the IPs 127.0.0.(1-6).
<br />
We'll now insert a few rows in the staff table:
<br />
<br />
<script src="https://gist.github.com/chbatey/8379cd0a175f3fadcad2.js"></script>
Now lets run a query with consistency level ONE with tracing on:<br />
<br />
<script src="https://gist.github.com/chbatey/530c5d970a7de694923a.js"></script>
The coordinator has had to go to replicas for all the partitions. For this query 127.0.0.1 acted as coordinator and the data was retrieved from 127.0.0.3, 127.0.0.5, 127.0.0.6. So 4 out of 6 nodes needed to be behaving for our query to succeed. If we add more partitions you can see how quickly we’d end up in a situation where all nodes in the cluster need to be up!<br />
<br />
Let’s make things even worse by upping the consistency to QUORUM:<br />
<br />
<script src="https://gist.github.com/chbatey/2fc2ab2bb8782da80757.js"></script>
Here 127.0.0.1 was the coordinator again, and this time 127.0.0.2, 127.0.0.3, 127.0.0.4, 127.0.0.5 were all required, we’re now at 5/6 nodes required to satisfy what looks like a single query.
<br />
This makes the query vastly more likely to ReadTimeout.<br />
<br />
It also gives the coordinator much more work to do as it is waiting for responses from many nodes for a longer time.
<br />
<br />
So how do we fix it? We denormalise of course!<br />
<br />
<script src="https://gist.github.com/chbatey/c924abf863740013499d.js"></script>
Essentially we've replaced tables with user defined types.<br />
<br />
Now when we query for a customer event we already have all the information. We’re giving coordinators less work to do and each query we do only requires the consistency’s worth of nodes to be available.<br />
<br />
<h4>
Can I ever break this rule?
</h4>
<br />
In my experience there are two times you could consider breaking the no-join rule.<br />
<ol>
<li>The data you’re denormalising is so large that it costs too much </li>
<li>The table like store or staff is so small it is okay to keep it in memory </li>
</ol>
So lets take the first one. Let’s say each event has a larger blob/JSON/XML associated with it that you needed to keep verbatim for later reporting and you need to query it in multiple ways so you end up with a table per query. If the raw data is many TBs then denormalising may require a much larger cluster. At this point you could consider trading off availability/speed for the cost of the larger cluster. This doesn’t mean once you have the IDs from the lookup table you should have large IN queries, alternatively you can still issue the queries to the verbatim data table independently using the native driver’s async functionality.<br />
<br />
The other time you may want to avoid denormalisation is when a table like staff or store is so small it is feasible to keep a copy of it in memory in all your application nodes. You then have the problem about how often to refresh it from Cassandra etc, but this isn't any worse than denormalised data where you typically won’t go back and update information like the store location.<br />
<br />
<h4>
Conclusion </h4>
<br />
To get the most out of Cassandra you need to retrieve all of the data you want for a particular query from a single partition. Anytime you don’t you are essentially doing a distributed join, this could be explicitly in your application of asking Cassandra to go to multiple partitions with an IN query. These types of queries should be avoided as often as possible. Like with all good rules there are exceptions but most users of Cassandra should never have to use them.
<br />
<br />
Any questions feel free to ping me on twitter @chbateychbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com13tag:blogger.com,1999:blog-4161315644722406995.post-37340622420261662302015-01-26T09:24:00.000-08:002015-01-26T09:24:34.932-08:00Spark + Cassandra: The basics + connecting to Cassandra from spark-shellA lot of people are getting excited about Apache Spark. The release of the open source <a href="https://github.com/datastax/spark-cassandra-connector" target="_blank">Cassandra connector</a> makes a technology like Spark even more accessible. Previously to get going you'd need a Hadoop infrastructure, now you can do away with all that and start using Spark directly against Cassandra, no HDFS required.<br />
<br />
<a href="http://christopher-batey.blogspot.co.uk/2015/01/spark-12-with-cassandra-21-setting-up.html" target="_blank">My last two posts on the topic were all about setting up a vagrant VM with Cassandra and Spark </a>installed. That's all well and good if you already working in the JVM ecosystem, you know what vagrant and Ansible are and you love your self a bit of SBT, but now it is time to take a step back. This post is aimed at getting you started with Spark and Cassandra without the assumption you know what sbt assembly means! By the end of this one the goal is to be able to execute (and understand what is going on) Spark/Cassandra jobs in the Spark REPL, and the next article will be submitting a standalone job.<br />
<br />
I'll assume you have Cassandra installed and that you have downloaded a <a href="https://spark.apache.org/downloads.html" target="_blank">Spark bundle from their website</a>. It doesn't matter which version of Hadoop it has been built against as we're not going to use Hadoop. If you don't have a locally running Cassandra instance I suggest you just use the VM from the previous article, <a href="http://christopher-batey.blogspot.co.uk/2013/10/installing-cassandra-20-on-ubuntu.html" target="_blank">follow this article for Ubuntu</a>, use homebrew if you are on Mac OSX or if all else fails just<a href="http://cassandra.apache.org/download/" target="_blank"> download the zip from the Apache website</a>.<br />
<br />
So first things first... why should you be excited about his?<br />
<br />
<ul>
<li>If you're already using Cassandra your data is already distributed and replicated, the Cassandra connector for Spark is aware of this distribution and can bring the computation to the data, this means it is going to be FAST</li>
<li>Scala and the JVM might seem scary at first but Scala is an awesome language for writing data transformations</li>
<li>The Spark-Shell: this is a REPL we can use for testing out code, simply put: it is awesome</li>
<li>Spark can also connect with other data sources: files, RDMSs etc, which means you can do analytics combining data in Cassandra and systems like MySQL</li>
<li>Spark also supports streaming, meaning we can combine new data in semi-real time with out batch processing</li>
<li>Most importantly, you don't need to extract-transform-load your data from your operational database and put it in your batch processing system e.g. Hadoop</li>
</ul>
<h2>
Lets get going</h2>
<div>
So what do we need to get all this magic working?</div>
<div>
<ul>
<li>Java - Java 7 or 8</li>
<li><a href="http://www.scala-sbt.org/" target="_blank">SBT</a> - any 0.13.* will work. This is build tool used by the majority of Scala projects (Spark be Scala)</li>
<li><a href="http://www.scala-lang.org/download/" target="_blank">Scala</a> - Spark doesn't officially support 2.11 yet so get 2.10</li>
<li>A Cassandra cluster</li>
<li>A Spark installation (we're going simple this time so all on one computer)</li>
<li>The Cassandra Spark connector with all of its dependencies bundled on the the classpath of the spark-shell for interactive use</li>
<li>A fat jar with all our dependencies if we want to submit a job to a cluster (for the next post)</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCAZodJtOyezfAFsfim-TsZV0RePfYKW4tir7zyLvXG9SBfqP7gHOCvemfSVskkTY5yw8a2aUu4o3ZUwvAQcKuhiUT7Mc2YbN0N3uFcZABPds6pbWZ6711fZRawQ5D2Y6RU_eNOcvgoggJ/s1600/download.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCAZodJtOyezfAFsfim-TsZV0RePfYKW4tir7zyLvXG9SBfqP7gHOCvemfSVskkTY5yw8a2aUu4o3ZUwvAQcKuhiUT7Mc2YbN0N3uFcZABPds6pbWZ6711fZRawQ5D2Y6RU_eNOcvgoggJ/s1600/download.jpg" height="92" width="320" /></a></div>
<div>
<br /></div>
<div>
Hold up: jargon alert. Bundled dependencies, classpath? Fatman?</div>
</div>
<div>
<br /></div>
<div>
Both Cassandra and Spark run on the JVM, we don't really care about Cassandra and we're not submitting code to run inside Cassandra, but that is exactly what we're going to do with Spark.</div>
<div>
<br /></div>
<div>
That means all the code and libraries that we use are going to have to go everywhere our computation goes. This is because Spark distributes your computation across a cluster of computers. So we have to be kind and bundle all our code + all the dependencies we use (other jar files e.g for logging). The JVM classpath is just how you tell the JVM where all your jars are. </div>
<div>
<br /></div>
<h4>
Getting the Spark-Cassandra connector on the classpath</h4>
<div>
<br />
If you're from JVM land you probably are used to doing things like "just build a fat jar and put it on the classpath" if you're not then that is just a lot of funny words. So the connector is not part of core Spark, so you can't use it by default in the spark-shell. To do that you need to put the connector <b>and all its dependencies</b> on the classpath for the spark-shell. This sounds tedious right? You'd have to go and look at the build system of the connector and work out what it depends on. Welcome to JVM dependency hell. </div>
<div>
<br /></div>
<div>
SBT, Maven, Gradle to the rescue (sort of). Virtually all JVM languages have a build system that allow you to declare dependencies, then it is the build system's responsibility to go get them from magic online locations (maven central) when you build your project. In Scala land this is SBT + Ivy.</div>
<div>
<br /></div>
<div>
When you come to distribute a JVM based application it is very kind to your users to build a far jat, or an "executable jar". This contains your code + all your dependencies so that it runs by its self, well apart from depending on a Java Runtime. </div>
<div>
<br /></div>
<div>
So what we need to do is take the connector and use SBT + the assembly plugin to build our selves a fat jar. The Spark-Cassandra connector already has all the necesary config in its build scripts so we're just going to check it out and run "sbt assembly".</div>
<br />
<br />
<script src="https://gist.github.com/chbatey/8d091963204c8aeb2db1.js"></script>
<br />
Lets take this line by line:<br />
<ol>
<li>Line 1: Clone the Spark-Connector repo</li>
<li>Line 11: Run the SBT assembly command</li>
<li>Wait for ages</li>
<li>Line 14: Tells us where SBT has put the fat jar</li>
</ol>
<div>
Now it is time to use this jar in the Spark Shell:</div>
<div>
<br /></div>
<script src="https://gist.github.com/chbatey/017321c58e234d920d79.js"></script>
<br />
<div>
Nothing fancy here, just gone into the bin directory of where I unzipped Spark and ran spark-shell --help. The option we're looking for is <b>--jars</b>. This is how we add our magical fat jar onto the classpath of the spark-shell. If we hadn't built a fat jar we'd be adding 10s of jars here!</div>
<br />
However before we launch spark-shell we're going to add some properties to tell spark where Cassandra is, in the file (you'll need to create it): {Spark Install}/conf/spark-defaults.conf add:<br />
<br />
<b> spark.cassandra.connection.host=192.168.10.11</b><br />
<b><br /></b>
Replace the IP with localhost if your Cassandra cluster is running locally. Then start up Spark-shell with the --jars option:<br />
<br />
<script src="https://gist.github.com/chbatey/2f19f493b7d653a31415.js"></script>
Now lets look at the important bits:<br />
<ol>
<li>Line 1: Starting spark-shell with --jars pointing to the fat jar we built</li>
<li>Line 10: Spark confirming that it has picked up the connector fat jar</li>
<li>Line 11: Spark confirming that it has created us a SparkContext</li>
<li>Line 13: Import the connector classes, Scala has the ability to extend existing classes. The effect of this import is that we now have cassandra methods on our SparkConext</li>
<li>Line 16: Create a Spark RDD from a Cassandra table "kv" in the "test" keyspace</li>
<li>Line 19: Turn the RDD into an array (forcing it to complete the execution) and print the rows</li>
</ol>
<div>
Well that's all folks, next post will be about submitting jobs rather than using the spark-shell.</div>
<br />
<br />
<br />chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com10tag:blogger.com,1999:blog-4161315644722406995.post-74778262888354063292015-01-21T08:27:00.002-08:002015-01-21T08:34:31.063-08:00Spark 1.2 with Cassandra 2.1: Setting up a SparkSQL + Cassandra environmentIn my <a href="http://christopher-batey.blogspot.co.uk/2014/12/getting-started-cassandra-spark-with.html" target="_blank">previous post on Cassandra and Spark</a> I showed how to get a development environment setup with Vagrant/Ansible/VirtualBox without installing Cassandra/Spark on your dev station.<br />
<br />
This update will get us to a point where we can run SQL (yes, SQL, not CQL) on Cassandra. It is just a trivial example to show the setup working.<br />
<br />
The previous article was back in the days of Spark 1.0. With Spark 1.1+ we can now run SparkSQL directly against Cassandra.<br />
<br />
I've updated the <a href="https://github.com/chbatey/vagrant-cassandra-spark" target="_blank">Vagrant/Ansible provisioning</a> to install Spark 1.2 and Cassandra 2.1, and I've added a new "fatjar" with the latest Cassandra Spark connector so that we can use it in the Spark shell and show this magic working. The 1.2 connector isn't released yet so I have built against the Alpha, see <a href="https://github.com/datastax/spark-cassandra-connector" target="_blank">here for details</a>. We're just that cutting edge here...<br />
<br />
So, once you have ran vagrant up (this will take a while as it downloads + install all of the above) you'll need to SSH in we can get into Spark shell.<br />
<br />
I've setup the following alias so no worrying about classpaths:<br />
<br />
<b>alias spark-shell='spark-shell --jars /vagrant/spark-connector-1.2.0-alpha1-driver-2.1.4-1.0.0-SNAPSHOT-fat.jar</b><br />
<br />
First lets jump into cqlsh and create a keyspace and table to play with:<br />
<br />
<script src="https://gist.github.com/chbatey/a58d40b801e587a053c2.js"></script>
<br />
Not the most exciting schema you'll ever see, but this is all about getting the Hello World of SparkSQL on Cassandra working!<br />
<br />
Now we have some data to play with lets access it from Spark shell.<br />
<br />
<script src="https://gist.github.com/chbatey/be40894a5f8a6fed582d.js"></script>
<br />
Lets go through what has happened here:<br />
<br />
<ul>
<li>Lines 3-6: Mandatory Spark ASCII art</li>
<li>Line 12: Import the connector so we can access Cassandra</li>
<li>Line 15: Create a CassandraSQLContext</li>
<li>Line 18: Set it to our test Keyspace we created above</li>
<li>Line 20: Select the whole table (very exciting I know!)</li>
<li>Line 21: Get Spark to execute an action so all the magic happens</li>
</ul>
<br />
That's all for now, tune in later for a more complicated example :)<br />
<br />
Here's the link to all the Vagrant/Ansible <a href="https://github.com/chbatey/vagrant-cassandra-spark" target="_blank">code</a>.<br />
<br />
<br />
<br />chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com3tag:blogger.com,1999:blog-4161315644722406995.post-65373145773801292482015-01-06T11:30:00.000-08:002015-01-06T11:33:25.166-08:00Wiremock: Now with extension points (open source == awesome)I have been using Wiremock as my preferred HTTP test double for some time now. I think it is a fantastic tool and I mentioned it quite a lot at a talk I gave at <a href="https://skillsmatter.com/skillscasts/5810-building-fault-tolerant-microservices" target="_blank">Skills matter</a> and it turned out the author, <a href="https://github.com/tomakehurst" target="_blank">Tom Akehurst</a>, was in the audience.<br />
<br />
Shamefully I had a private fork of Wiremock at the company I worked for, we'd hacked away at it and added support for copying our platform headers, adding our HMAC signatures to responses etc. We'd also used it for load testing and made a bunch of the Jetty tuning options configurable. Some of this, HMAC, was confidential, 90% not so much :)<br />
<br />
So over the Christmas holidays, with the help of Tom, I've been hacking away with Wiremock, and the new release now contains:<br />
<ul>
<li>Configurable number of container threads</li>
<li>Exposed Jetty tuning options: Acceptor threads & Accept queue size</li>
<li>Extension points</li>
</ul>
<div>
The first two were my PRs, the latter was by Tom, who (rightly) rejected my PR as it added too much latency to start up as it was reflection based. But kindly Tom hashed out an alternative documented here: https://github.com/tomakehurst/wiremock/issues/214</div>
<div>
<br /></div>
<div>
If you've used Wiremock before you'll know you run/interact it in two modes: via its Java API and as a standalone process. This means you can use it for unit/integration testing and black box acceptance testing. Let's look with the Java API, how to use this feature in standalone mode is documented on the Wiremock site:</div>
<div>
<br /></div>
<script src="https://gist.github.com/chbatey/a66c433c119b76537442.js"></script>
<br />
<div>
</div>
This is the class you extend to extend Wiremock and here is a simple implementation that copies over headers that begin with Batey, this example is inspired by a platform requirement to copy over all platform headers when dealing with requests.
<br />
<br />
<script src="https://gist.github.com/chbatey/afac5af88af51d779909.js"></script>
Simple! Now to use it from the Java API you add the following to your stubbing:<br />
<br />
<script src="https://gist.github.com/chbatey/f6eced96e06132cfc29f.js"></script>
The name, CopiesBateyHeaders, in your implementation needs to match the stubbing. We can now test a piece of code that looks like this:<br />
<br />
<script src="https://gist.github.com/chbatey/0a821c9d21d784e4c8a8.js"></script>
For both cases: When the dependency does copy the header over and when it doesn't. Here is the test for does:
<br />
<br />
<script src="https://gist.github.com/chbatey/bc5afd4fdf04e6f2494b.js"></script>
And doesn't:
<br />
<br />
<script src="https://gist.github.com/chbatey/56d732a2bf0880fd72db.js"></script>
Now you're probably thinking we could have just primed this right?<br />
<br />
Well I hate noise in tests, and we want a single test making sure we throw an error if the header isn't copied but for all the rest of the behaviour (obviously there isn't any in this example) we can now forget about the fact our dependency should copy the headers, thus reducing noise in the priming of all our other tests.<br />
<br />
I find this particularly important in black box acceptance tests, which often get very noisy.<br />
<br />
I love open source :) All the code for this example is on my github <a href="https://github.com/chbatey/wiremock-extension-example" target="_blank">here</a>.chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com1tag:blogger.com,1999:blog-4161315644722406995.post-34431029243015298182014-12-22T06:22:00.000-08:002014-12-22T06:22:49.624-08:00Getting started: Cassandra + Spark with VagrantI play with a lot of different technologies and I like to keep my work stations clean. I do this by having a lot of vagrant VMs. My latest is Apache Spark with Apache Cassandra. We're going to install a working setup of Cassandra/Spark using Vagrant and Ansible. The Vagrant/Ansible is on Github <a href="https://github.com/chbatey/vagrant-cassandra-spark" target="_blank">here</a>.<br />
<br />
To get going you'll need:<br />
<ul>
<li><a href="https://www.vagrantup.com/" target="_blank">Vagrant</a></li>
<li><a href="https://www.virtualbox.org/" target="_blank">Virtual Box</a></li>
<li><a href="https://github.com/ansible/ansible" target="_blank">Ansible</a> (used for provisioning)</li>
<li>Git</li>
</ul>
<div>
If you haven't used Ansible before ignore all the paid for Ansible Tower and install it with your favourite package manager e.g homebrew or apt. </div>
<div>
<br /></div>
<div>
Once that's installed checkout the <a href="https://github.com/chbatey/vagrant-cassandra-spark" target="_blank">Vagrant file</a>.</div>
<div>
<br /></div>
<div>
Then launch the VM with vagrant up. This can take some time as it actually installs:<br />
<ul>
<li>Java</li>
<li>Cassandra</li>
<li>Spark</li>
<li>Spark Cassandra connector</li>
</ul>
<div>
I could have baked a virtual box with all this in but the Ansible also documents you install all of these (and me once I've forgotten). As well as being slow it has the disadvantage that if downloads Cassandra/Spark so if their repositories are down it won't work.</div>
</div>
<div>
<br /></div>
<div>
The VM runs on port 192.168.10.10. Your Spark master should be up and running on <a href="http://192.168.10.10:8080/">http://192.168.10.10:8080/</a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc1hgz79eGsBuZiaYiQnMU0KO0Cp1vajBK_KpdbB8llCXAQbzN4heZTut5n_jC9_bjuhgJF7rpNJzs4HFKAvaST1MQxxWEkkbVnc2mHMVaa_UQeK0u0f-mxrZcIDvt33MCc0eoUN_oYTGw/s1600/Screenshot+2014-10-18+22.09.06.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc1hgz79eGsBuZiaYiQnMU0KO0Cp1vajBK_KpdbB8llCXAQbzN4heZTut5n_jC9_bjuhgJF7rpNJzs4HFKAvaST1MQxxWEkkbVnc2mHMVaa_UQeK0u0f-mxrZcIDvt33MCc0eoUN_oYTGw/s1600/Screenshot+2014-10-18+22.09.06.png" height="180" width="320" /></a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<div>
You'll also have ops centre installed at: <a href="http://192.168.10.10:8080/">http://192.168.10.10:8888/</a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglDrOn2ob5-jAE9vjGcPQv6igJI5lrfWFKaQ93dFeUGtUlA55fSuJjSryy_rXZEKtuUlnhu3jWgXXVdL2GlgEENPZhIuaC6zZL0jRsJo57fFWIipezQR93DJUyi7b0BNZMxvE7eZU860Cx/s1600/Screenshot+2014-10-18+22.11.18.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglDrOn2ob5-jAE9vjGcPQv6igJI5lrfWFKaQ93dFeUGtUlA55fSuJjSryy_rXZEKtuUlnhu3jWgXXVdL2GlgEENPZhIuaC6zZL0jRsJo57fFWIipezQR93DJUyi7b0BNZMxvE7eZU860Cx/s1600/Screenshot+2014-10-18+22.11.18.png" height="255" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
To add the cluster simply click "Add existing cluster.." then enter the IP 192.168.10.10</div>
<div>
<br /></div>
<div>
If you want to use cqlsh then simply "vagrant ssh" in and then run "cqlsh 192.168.10.10"</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI1iJXGaOpRUdZVVL8LC1OZpUJRKYTdA3tZ9WhAJLyaQ_4dnKexcffk70X9jQEGotvgoMZeH1eK9sMesA6LfRLIXBJjGPskPJW6r23zsxvk8Y34k-RkpzCWOeOqX-6FMnpxVsgsJYqHzOI/s1600/Screenshot+2014-10-18+22.14.32.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI1iJXGaOpRUdZVVL8LC1OZpUJRKYTdA3tZ9WhAJLyaQ_4dnKexcffk70X9jQEGotvgoMZeH1eK9sMesA6LfRLIXBJjGPskPJW6r23zsxvk8Y34k-RkpzCWOeOqX-6FMnpxVsgsJYqHzOI/s1600/Screenshot+2014-10-18+22.14.32.png" height="252" width="320" /></a></div>
<div>
<br /></div>
<div>
To get spark shell up and running just "vagrant ssh" in and then run the spark-shell command:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBjmOtzZxg2n5GkBGyAPtPqt0kx-V3yrEZyHAKQC4tHdXPaqtVTm2HeMYOEwIFSqXana0AnWwWFMfaSEVcIspa4VSoiP1OyGat0qqj27HjX9IfPjFZSdv1m9kCPaFaoGhzQqM2WfSMzKRb/s1600/Screenshot+2014-10-18+22.17.26.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBjmOtzZxg2n5GkBGyAPtPqt0kx-V3yrEZyHAKQC4tHdXPaqtVTm2HeMYOEwIFSqXana0AnWwWFMfaSEVcIspa4VSoiP1OyGat0qqj27HjX9IfPjFZSdv1m9kCPaFaoGhzQqM2WfSMzKRb/s1600/Screenshot+2014-10-18+22.17.26.png" height="207" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Spark shell has been aliased to include the Cassandra spark connector so you can start using Cassandra backed RDDs right away!<br />
<br />
Any questions or problems just ping me on twitter: @chbatey</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com2tag:blogger.com,1999:blog-4161315644722406995.post-30302532639598416812014-12-08T07:51:00.001-08:002014-12-08T08:09:57.045-08:00Streaming large payloads over HTTP from Cassandra with a small Heap: Java Driver + RX-JavaCassandra's normal use case is vast number of small read and write operations distributed equally across the cluster.<br />
<br />
However every so often you need to extract a large quantity of data in a single query. For example, say you are storing customer events that you normally query in small slices, however once a day you want to extract all of them for a particular customer. For customers with a lot of events than this could be many hundreds of megabytes of data.<br />
<br />
If you want to write a Java application that executes this query against a version of Cassandra prior to 2.0 then you may run into some issues. Let us look at the first one..<br />
<br />
<h4>
Coordinator out of memory:</h4>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiF9Q6OCaLHAQhtVc8datPW9yuWEAN7UDysF9gM_iv57duJ7biYbqpkqPbxnWL15mGpyn7KYAfEK6HQoH6ENkjt1V5uaUTf-peizWr_rCpZRAJc6gCKFofvDmQwGZ3bzxQJCU0rH46j1s0P/s1600/CoordinatorOutOfMemory.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiF9Q6OCaLHAQhtVc8datPW9yuWEAN7UDysF9gM_iv57duJ7biYbqpkqPbxnWL15mGpyn7KYAfEK6HQoH6ENkjt1V5uaUTf-peizWr_rCpZRAJc6gCKFofvDmQwGZ3bzxQJCU0rH46j1s0P/s1600/CoordinatorOutOfMemory.png" height="167" width="400" /></a></div>
<br />
Previous versions of Cassandra used to bring all of the rows back to the coordinator before sending them to your application, so if the result is too large for the coordinator's heap it would run out of memory.<br />
<br />
Let's say you had just enough memory in the coordinator for the result, then you ran the risk of...<br />
<br />
<h4>
Application out of memory:</h4>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHNqRFdtZ62gkxsw2eoU9ANkm1gsZF2BaTLQDlyvKbtN351BlugGjySLboW9Xr0A-4b-5luO7z96DoexVLOWoDMmoBlL8MuCx4XcNyWW3NpCThn5X1Y-0nCfJy4Qd8jP5bmjk51-afpJlV/s1600/AppOutOfMemory.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHNqRFdtZ62gkxsw2eoU9ANkm1gsZF2BaTLQDlyvKbtN351BlugGjySLboW9Xr0A-4b-5luO7z96DoexVLOWoDMmoBlL8MuCx4XcNyWW3NpCThn5X1Y-0nCfJy4Qd8jP5bmjk51-afpJlV/s1600/AppOutOfMemory.png" height="158" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
To get around this you had to implement your own paging, where you split the query into many small queries and processed them in batches. This can be achieved by limiting the results and issuing the next query after the last result of the previous query.</div>
<div>
<br /></div>
<div>
If your application was streaming the results over HTTP then the architecture could look something like this:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3E9VmDXLH5nPdkKzt-oNwtrMsXflYy4fPWZik8STobgeq9cAEvh1pYKhb4e_cv5UuQdUxTw2wQ4JSuftCxUcaSVKGEWsx3QnnvgHXNCVH33IOFrkT9hL7LpXcKIMOtHc85HSzHBF4XIEP/s1600/ArchitectureForExtract.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3E9VmDXLH5nPdkKzt-oNwtrMsXflYy4fPWZik8STobgeq9cAEvh1pYKhb4e_cv5UuQdUxTw2wQ4JSuftCxUcaSVKGEWsx3QnnvgHXNCVH33IOFrkT9hL7LpXcKIMOtHc85HSzHBF4XIEP/s1600/ArchitectureForExtract.png" height="166" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Here we place some kind of queue, say an ArrayBlockingQueue if using Java, between the thread executing the queries and the thread streaming it out over HTTP. If the queue fills up the DAO thread is blocked, meaning that it won't bring any more rows from Cassandra. If the DAO gets behind the WEB thread (perhaps a tomcat thread) blocks waiting to get more rows out of the queue. This works very nicely with the JAX-RS StreamingOutput.<br />
<br />
This all sounds like a lot of hard work...</div>
<div>
<br /></div>
<div>
<h4>
The 2.0+ solution</h4>
<div>
<br /></div>
From version 2.0, Cassandra would no longer suffer from the coordinator out of memory. This is because the coordinator pages the response to the driver and doesn't bring the whole result into memory. However if your application reads the whole ResultSet into memory then your application running out of memory is still an issue.<br />
<br />
However the DataStax driver's ResultSet pages as well, which works really nicely with Rx-Java and JAX-RS StreamingOutput. Time go get real, let's take the following schema:<br />
<br />
<script src="https://gist.github.com/chbatey/902a5f2434013c899757.js"></script>
</div>
<div>
<br /></div>
And you want to get all the events for a particular customer_id (the partition key). First let's write the DAO:<br />
<br />
<script src="https://gist.github.com/chbatey/288166148746ff815b10.js"></script>
Let's go through this line by line:<br />
<b><br /></b>
<b>2: </b>Async Execute of the query that will bring back more rows that will fit in memory.<br />
<b>4: </b>Convert the ListenableFuture to an RxJava Observable. The Observable has a really nice callback interface / way to do transformation.<br />
<b>5: </b>As ResultSet implements iterable we can flatMap it to Row!<br />
<b>6: </b>Finally map the Row object to CustomerEvent object to prevent driver knowledge escaping the DAO.<br />
<br />
And then let's see the JAX-RS resource class:
<br />
<br />
<script src="https://gist.github.com/chbatey/5b4ce80e1655741322d5.js"></script>
Looks complicated but it really isn't, first a little about JAX-RS streaming.
<br />
<br />
The way JAX-RS works is we are given a StreamingOutput interface which we implement to get a hold of the raw OutputStream. The container e.g Tomcat or Jetty, will call the write method. It is our responsibility to keep the container's thread in that method until we have finished streaming. With that knowledge let's go through the code:<br />
<br />
<b>5:</b> Get the Observable<CustomerEvent> from the DAO.<br />
<b>6:</b> Create a CountDownLatch which we'll use to block the container thread.<br />
<b>7: </b>Register a callback to consume all the rows and write them to the output stream,<br />
<b>12: </b>When the rows are finished, close the OutputStream.<br />
<b>16: </b>Countdown the latch to release the container thread on line <b>33</b>.<br />
<b>26:</b> Each time we get a CustomerEvent, write it to the OutputStream.<br />
<b>33: </b>Await on the latch to keep the container thread blocked.<br />
<b>39: </b>Return the StreamingOutput instance to the container so it can call write.<br />
<br />
Given that we're dealing with the rows from Cassandra asynchronously you didn't expect the code to be in order did you? ;)<br />
<br />
The full working example is on my <a href="https://github.com/chbatey/cassandra-customer-events-dropwizard" target="_blank">GitHub</a>. To test it all I put around 600mb of data in a Cassandra cluster for the same partition. There is a sample class in the test directory to do this.<br />
<br />
I then started the application with a MaxHeapSize of 256mb, then used curl to hit the events/stream endpoint:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYglNNtvvndgRmo_dPjzzmKagdIX2e4k56Xn5nPaMbWhT23RL9i4jiyz_T00kWMWqxWVQMUSJjvVVbvoMdS78L8zYigmLtxUkAvhTSy2X_fZImEkPQhnRtHcgDCbGrzEdsyU3op7yHesC_/s1600/Screenshot+2014-12-08+15.39.57.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYglNNtvvndgRmo_dPjzzmKagdIX2e4k56Xn5nPaMbWhT23RL9i4jiyz_T00kWMWqxWVQMUSJjvVVbvoMdS78L8zYigmLtxUkAvhTSy2X_fZImEkPQhnRtHcgDCbGrzEdsyU3op7yHesC_/s1600/Screenshot+2014-12-08+15.39.57.png" height="180" width="400" /></a></div>
<br />
As you can see 610M came back in 7 minutes. The whole time I had VisuamVM attached to the application and the coordinator and monitored the memory usage.<br />
<br />
Here's the graph from the application:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ2KhkKY3eIcWno0hyphenhyphenJGpQWxFpc66FpQTn11DtCDFWiiD0_ZNKsOnFXJd4NLZQC7hqwCXKjR2OfxYqO_bs7Gba5-0CIBJvHUEUOnoa8LQjjoKRJ_hqdePyJfrjkB0xL0h-pnVywMKlgY-e/s1600/AppMemory.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ2KhkKY3eIcWno0hyphenhyphenJGpQWxFpc66FpQTn11DtCDFWiiD0_ZNKsOnFXJd4NLZQC7hqwCXKjR2OfxYqO_bs7Gba5-0CIBJvHUEUOnoa8LQjjoKRJ_hqdePyJfrjkB0xL0h-pnVywMKlgY-e/s1600/AppMemory.png" height="235" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The test was ran from 14:22 to 14:37. Even though we were pumping through 610M of data through the application the heap was gittering between 50m and 200m, easily able to reclaim the memory of the data we have streamed out.</div>
<br />
For those new to Cassandra and other distributed databases this might not seem that spectacular, but I once wrote a rather large project to do what we can manage here in a few lines. Awesome work by the Apache Cassandra commitors and the DataStax Java driver team.<br />
<br />chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com1tag:blogger.com,1999:blog-4161315644722406995.post-33596783690316002182014-12-05T03:56:00.000-08:002014-12-05T03:56:48.002-08:00Cassandra summit EU - British Gas, i20 water, testing Cassandra and paging!Yesterday was the EU Cassandra Summit in London, 1000 crazy Cassandra lovers. I've only just recovered from what was a hectic day.<br />
<br />
Over the course of the day I got to do chat with two awesome companies, Michael Williams from i20 water and Josep Casals from British Gas Connected Homes. Both of these companies are using Cassandra to store time series data from devices, dare I use the ever popular buzz phrase Internet of Things?<br />
<br />
But really they are, i20 water enable water companies to place sensors all around their network and gather the data to detect leaks, saving them 100s of millions of litres of water a day.<br />
<br />
British Gas Connected homes are enabling their customers to turn their central heating on and off via their mobile, and are expanding into monitoring boilers and predicting when they'll fail/need a service.<br />
<br />
In addition to speaking with Cassandra users I also snuck in a talk and a lightning talk. The talk was on how to test Cassandra applications and the lightning talk on server side paging.<br />
<br />
Here are the slides for the talk, the video will no doubt be online soon:<br />
<br />
<iframe frameborder="0" height="400" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/42388739" width="476"></iframe><br />
<br />
And for the lightning talk:<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="355" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/42388804" style="border-width: 1px; border: 1px solid #CCC; margin-bottom: 5px; max-width: 100%;" width="425"> </iframe> <br />
<div style="margin-bottom: 5px;">
<strong> <a href="https://www.slideshare.net/chbatey/cassandra-summiteu2014-pagingnoanimation" target="_blank" title="Cassandra Summit EU 2014 Lightning talk - Paging (no animation)">Cassandra Summit EU 2014 Lightning talk - Paging (no animation)</a> </strong> from <strong><a href="https://www.slideshare.net/chbatey" target="_blank">Christopher Batey</a></strong> </div>
chbateyhttp://www.blogger.com/profile/13384294386607277964noreply@blogger.com0