Getting started with Hystrix and Tenacity to build fault tolerant applications

Applications are becoming increasingly distributed. Micro service architecture is the new rage. This means that each application you develop has more and more "integration points".

Any time you make a call to another service or database, or use any third party library that is a black box to you, it can be thought of as an integration point.

Netflix's architecture gives a great example of how to deal with integration points. They have a popular open-source library called Hystrix which allows you to isolate integration points by executing all calls in its own worker queue and thread pool.

Yammer have integrated Hystrix with Dropwizard, enabling enhancement of applications to publish metrics and accept configuration updates.

Here is an example application that calls out to three HTTP services and collects the results together into a single response.

Rather then calling into a HTTP library on the thread provided by Jetty this application uses Yammer's Hystrix wrapper, Tenacity.

Let's look at one of the integration points:

	public class PinCheckDependency extends TenacityCommand<Boolean> {
	private HttpClient httpClient;
	public PinCheckDependency(HttpClient httpClient) {
	super(IntegrationPoints.PIN_CHECK);
	this.httpClient = httpClient;
	}
	@Override
	protected Boolean run() throws Exception {
	HttpGet pinCheck = new HttpGet("http://localhost:9090/pincheck");
	HttpResponse pinCheckResponse = httpClient.execute(pinCheck);
	int statusCode = pinCheckResponse.getStatusLine().getStatusCode();
	if (statusCode != 200) {
	throw new RuntimeException("Oh dear no device information, status code " + statusCode);
	}
	String pinCheckInfo = EntityUtils.toString(pinCheckResponse.getEntity());
	return Boolean.valueOf(pinCheckInfo);
	}
	@Override
	public Boolean getFallback() {
	return true;
	}
	}

view raw PinCheckDependency.java hosted with ❤ by GitHub

Here we extend the TenacityCommand class and call the dependency in the run() method. Typically all this code would be in another class with the TenacityCommand just being a wrapper, but this is a self-contained example. Let's explain what it is doing:

• Making an HTTP call using the Apache HTTP client
• If it fails throw a Runtime exception

By instantiating this TenacityCommand and calling execute(), your code is automagically executed on its very own thread pool, and requests are queued on its very own work queue. What benefits do you get?

• You get a guaranteed timeout, so no more relying on library's read timeout that never seems to work in production
• You get a circuit breaker that opens if a configured % of calls fail, meaning you can fail fast and throttle calls to failing dependencies
• You get endpoints that show the configuration and whether a circuit breaker is open

If the call to run() fails, times out or the circuit breaker is open Tenacity will call the optional getFallback() method, so you can provide a fallback strategy rather than failing completely. 

Another hidden benefit is how easy it is to move to a more asynchronous style of programming. Let's look at the resource class that pulls together the three dependencies:

	@GET
	@Timed
	public String integrate() {
	String user = new UserServiceDependency(httpClient).execute();
	String device = new DeviceServiceDependency(httpClient).execute();
	Boolean pinCheck = new PinCheckDependency(httpClient).execute();
	return String.format("[User info: %s] \n[Device info: %s] \n[Pin check: %s] \n", user, device, pinCheck);
	}

view raw IntegrationResource.java hosted with ❤ by GitHub

Let's ignore the fact we are calling out to other HTTP services from a resource layer. The above code shows how to use Tenacity synchronously. Apart from the advantages you gain regarding failures, all the calls are still happening one by one as we call execute() which blocks so we don't call the second dependency until the first one has finished.

However, this doesn't have to be the case. Now you've snuck Tenacity into your code base you can change the code to something like this:

	@GET
	@Timed
	@Path("async")
	public String integrateAsync() throws Exception {
	Future<String> user = new UserServiceDependency(httpClient).queue();
	Future<String> device = new DeviceServiceDependency(httpClient).queue();
	Future<Boolean> pinCheck = new PinCheckDependency(httpClient).queue();
	return String.format("[User info: %s] \n[Device info: %s] \n[Pin check: %s] \n", user.get(), device.get(), pinCheck.get());
	}

view raw IntegrationResource.java hosted with ❤ by GitHub

And without your colleagues realising you've made all of your calls to your dependencies execute asynchronously and (possibly) at the same time, then you block to bring them all together at the end.

We've barely scratched the surface of Hystrix and Tenacity but hopefully you can already see the benefits. All the code for this example a long with instructions on how to use wiremock to mock the dependencies is here.