I just finished a great project with modern tech, gave the client more than they originally asked for, and got to learn something new.
In this post I’ll cover details about the project and how things worked. Along the way I’ll also give my personal opinions and I won’t back them up with pesky facts either, because, you know, that’s what you get to do in a project overview!
I ♥ REST
“Web services for a major entertainment industry player” sounds pretty good, but it’s even better. The client opted for REST web services with a JSON response; a very modern choice.
We’ve heard horror stories of making SOAP services work cross-platform, data type mismatches, etc. With REST, if your environment can make an http request and parse text, you’re good to go.
JSON is also super easy. You don’t have to write a DTD or XSD schema; instead you just create your objects and use a JSON serializer. For this project we used Jackson.
The other parts of our tech stack were: Java, Spring, Hibernate, Tomcat, Maven, JUnit, Joda-Time, and Eclipse. All of these technologies are open source. In fact, the only part that wasn’t open source was the Oracle JDBC driver.
The Force is Strong With This One
One of our requirements involved subtracting dates to get the number of days in between. Leap years and daylight savings time (and the different dates for daylight savings time over the years) combine to make this a hairy task.
The built-in Java date and time API couldn’t do this easily, but we were pleasantly surprised to see that Joda-Time could. Using the Joda-Time date object, it was just one line to get the number of days between two dates.
If your service runs in your own private network and the clients are also written by you, it’s tempting not to worry about security. Additionally, with REST it’s nice to avoid https when you can as responses can then be cached by a caching proxy server.
On the other hand, it’s nice to know your requests can only come from ‘approved’ clients. To do that, we could use BASIC AUTH (username/password), or we could go the https route and use client certificates. BASIC AUTH sends the password over clear text, which seems dangerous, and client certificates can be a pain to setup (plus you have SSL overhead and don’t get any benefit out of a caching proxy server anymore).
Our client chose to go another route and use a relatively new standard: OAuth. OAuth has been in the news recently because of Twitter and Facebook, but for locking down simple web services, we actually use a variant called “2-legged OAuth”.
The gist: clients have an ID and a secret. For each request several parameters are combined and run through a one-way hash. Those parameters include: the URL, the query string, the HTTP method, some OAuth values, the current time, the ID, and the secret. That hash, and the parameters used to create it (but NOT the secret), are sent plain text in the request. Since the server knows the secret, it can recompute the hash and make sure your hash matches its hash. It also verifies the request wasn’t made too long ago.
This adds very little overhead to the request: just computing a hash and sending the OAuth values. Plus, it ensures the request URL and query string weren’t modified (if they were, the hashes won’t match), and it works with or without https. Finally, the server can also use it for authorization, for example: one ID may only have GET permission while another has GET, PUT, and POST permissions.
We used JUnit to test our business logic and Hibernate queries. We also had tests ensure our objects serialized to JSON correctly. For example: if a developer changed something in the code that also changed our JSON output, we would break our client’s code. If this happened, we wanted a test to fail, but that posed a problem as JSON fields can be in any order. If we compare two JSON strings and the order differs, the data may be equivalent, but our test would fail.
To work around this issue we wrote a custom assert method to serialize an object to JSON and then deserialize the JSON to a Java Map (in Python, a dictionary; in Ruby, a hash). Now we can compare our maps with each rather than comparing two strings, and the test works just fine since maps don’t care about order!
We needed to meet a few metrics, one of which was code coverage. Code coverage measures how much of your code is executed by the tests. If your tests finish running and only exercise 53% of the lines, then you have 53% code coverage.
This shows you holes in your unit testing, but be careful. Just as a passing test doesn’t guarantee bug-free code, 100% test coverage doesn’t mean all your logic is tested. Case in point: our data access objects had something like 65% – 75% code coverage even though we hadn’t written a single test for them! Code at a higher level was calling the getters and setters, ratcheting up the code coverage. So code coverage has some value, but you still must analyze what’s going on.
The code coverage tool we used (Cobertura) also provided another metric called branch coverage. Branch coverage measures the conditions in an if-statement or a loop. Take this example:
if (isWeekday && hasDiscount)
The above line must be executed three times to get 100% branch coverage: once when isWeekday is false, once when isWeekday is true but hasDiscount is false, and once when both are true.
Both code coverage and branch coverage helped us find holes in our testing, but we particularly liked branch coverage. There are occasional false positives with code coverage, but branch coverage always seemed to flag something worth looking at.
One of the best things about a web service is how testable they are. You have very predictable output based on the input, so you can automate your testing.
In addition to our unit tests we also used SoapUI. Despite its name, it can test REST services too. SoapUI is a standalone GUI for testing web services, and we had it run 164 different tests.
We also used SoapUI to run our load tests (many users at once) and soak tests (few users for many hours).
However, getting SoapUI to provide the OAuth values for each test was challenging. Luckily, SoapUI has a scripting API to customize each request.
In the Groove
SoapUI’s scripting library lets you use Groovy, a dynamically typed language for the Java virtual machine. Just about any line of Java will work in Groovy, but Groovy also has its own syntax which was extremely nice to work with.
Groovy adds methods to built-in Java classes, and supports closures, which can make code more readable and terser than Java. For example: converting a List to a Map is 3 – 4 lines of Java, but it can be done in 1 line of Groovy.
Maven was our build tool, and once you understand it you can get a build up and running very quickly.
Additionally, our client imposed a few restrictions such as minimum code coverage, maximum cyclomatic complexity per method, and they also required a particular header for each source file. We enforced all of these rules as part of our Maven build process.
The maven Cobertura plugin can be configured with a minimum line coverage and minimum branch coverage (we set both to 90%).
The McCabe Cyclomatic Complexity Number (CCN) determines how complex a method is, and the JavaNCSS maven plugin can determine the CCN for each of your methods. We set the CCN to a maximum of 10.
Finally, the Checkstyle maven plugin and some rules from our client ensured each source file had the appropriate header documentation.
If any of these metrics failed, or if the code didn’t compile, or the tests failed, then the entire build would fail. This meant we couldn’t deploy a new build to our test server until we fixed the issues. This nicely ensures you don’t ship some bad code on delivery day.
Overall the project went extremely well. Everyone on the team learned something new, we delivered on time, and we even met some additional client requests. It can be fun to use the knowledge you already have, but more fun to learn something new and cool. On this project I got to do both.