I was pleased to find recently that Roy Osherove’s Art of Unit Testing was available on Safari.  I have been following Roy’s blog for a while now, and was quite excited at the prospect of him writing a book on Unit Testing.  It was only my personal cheapness that kept me from shelling out the $25 to get the E-Book version from Manning ahead of time.  I have to say, now that I have read it, that it would have been well worth the money.  Before I get too deep I want to provide some context for what I am about to say.

I consider myself an experienced TDD practitioner and Unit Test Writer
So that means that I was reading this book hoping to gain some insight.  I wanted to find out how to write better, more readable, more maintainable tests.  I was also hoping for a little bit of “atta-boy” affirmation that the way I do things is the “right” way.  The astute reader may be able to tell that in order for my first hope to be true, the second may have to get some points taken away.  This was in fact the case, and to be honest, coming out of it I feel like I’ve gotten more value from the things I’ve learned than I received from whatever ego stroking may have occurred with what I am currently doing right.

So lets get started….
I was expecting the book to start out essentially as it did, some brief history about the author and an introduction to Unit Testing for those who may not be familiar with it.  I have to say I was expecting the book to be a little more TDD-centric than it was, but I think most of that was my own bias for TDD as “The Only Way To Write Software”.  Roy actually explained what TDD was, and also why he wasn’t going to harp too much on it throughout the book.  I have to say, I can see why he made the decision that he did.  I can also say that it seemed perfectly clear to me that TDD is a technique that he feels has a lot a value, which made me happy.  Since this is supposed to be a review from the perspective of an experienced practitioner of TDD and Unit Testing, I’m not going to go into anything that was touched on in the early chapters, apart from noting that they contained a general introduction to the tools, technique and philosophy of unit testing.  I can also say that, though I was already familiar with the material, I didn’t mind reading through it at all.  Overall, Roy’s writing style was light and quite pleasant, even for a technical book.

And now into the meat of the book…
For me, things started getting interesting in Part 3 of the book.  This is where issues of test design and organization are addressed.  This is one of those areas that I feel like I need some guidance on, mostly because I developed my testing idioms mostly through habit, and trial and error.  I look back on tests I have written in the past (which could be as little as two days ago) and I wonder how I could have come up with such a brittle, unmaintainable nightmare.  I feel like I need guidance from the experts on what I can do better when writing my tests.  Roy delivered on these items in chapter 7 “The pillars of good tests”.  One of the lessons I took away from this was the value in testing one concept per test.  I had heard this as “one assert per test” in the past, and scoffed at the idea.  But Roy presents a very compelling argument for why this is a good idea, if you are testing multiple concepts, you don’t know the extent of the problem when your test fails.  And lets face it, the failing test is the reason we’re doing this whole thing.  I’ve seen personally the failing test that just keeps failing.  You tackle the issue from one failed assert only to rebuild, and find one right after it which fails as well.  One of the issues I’ve had with this is the redundant setup and configuration that could be required for exercising this concept, but this issue is also addressed by the straight forward recommendation of creating clear and understandable configuration methods.  In the past I have generally not been really good about applying DRY to my test setup, which, I know, is another case of treating tests differently from regular code.  Having someone in a position of authority (like Roy) say, “put your setup in separate methods so you can re-use them and make your tests more readable” made it okay to do the thing that I knew I should be doing anyway.  The key concepts covered are making tests readable, maintainable, and an accurate portrayal of the authors intent.

Even more in depth….
Section 4 goes even further and talks about how to integrate unit testing into an organization which is not already doing it.  This is an interesting subject to me as I have recently moved to a company which has not been doing unit testing and TDD as part of their regular development process.  Roy draws on his experiences as a consultant to provide some really good advice for how to go about enacting this sort of change in an organization.  I particularly pleased with his candor when he describes his failed attempts at integrating unit testing.  It would have been quite easy to simply say “Based on my considerable expertise, these are the things you need to do”, but he chooses instead to share some real-world experience in a straight forward way that only adds to my respect for him as a professional.  In addition to this, he touches on techniques for integrating testing into “legacy” code (i.e. code which is not tested).  He does a good job at introducing some techniques for testing what is essentially untestable code, which a very large nod at Michael Feathers’ “Working Effectively with Legacy Code”.

The book ends with three appendices, one discussing the importance of testability in the design process, one listing vairous testing tools (both Java and .Net), and the last listing guidelines for conducting test reviews.  This last one is nice, because it presents a concise view of all of the guidelines presented throughout the book, and provides page references where you can get the “why” behind each. 

All in all…
This is a really good book, which should be part of any agile development library.  It doesn’t matter if you are writing your first unit tests, or you’re a seasoned pro, there is going to be something here for you.  I think it is great that Roy has chosen to share his experience with the developer community in this way.  I came into this book with some rather high expectations and I think they were met.

A note on TypeMock….
I remember seeing some criticism floating around on twitter suggesting the book was rather pro TypeMock.  There was also the comment that Roy’s affiliation with TypeMock was not made clear early on.  I can’t say I saw either of these things when I was reading it.  For starters, I already knew Roy worked for TypeMock, so perhaps that skewed my ability to objectively judge if the disclosure was done in a timely manner or not.  I can say that the places in the book which there seemed to be a preference for TypeMock were places where he stated things like “I feel TypeMock has a better syntax in this case”, or “TypeMock is the only tool with provides these capabilities”.  For starters, the first is a statement of preference.  Sure Roy helped design the API for TypeMock, so it seems only natural that he would prefer it to other frameworks, but having used it I would have to agree with the statement.  It is a great API, and example if a fluent interface done well.  The second comment is also plain fact.  Of the mocking libraries available in the .Net space, TypeMock is the only one that allows you to swap instances of objects in place, without making changes to the classes using them.  You can argue over whether or not this is a good or a bad thing, but the fact remains that it is a feature specific to TypeMock.  Maybe I was expecting something more blatant and obvious, but I just didn’t see it.

The folks at TypeMock have released a new UnitTesting tool aimed specifically at catching deadlocks in multithreaded code called TypeMock Racer, and what’s more they are offering free licenses to folks willing to review it during the 21 day free trial period.  As anyone who knows me can testify to, I am a whore for free-bees, so I decided to take them up on this.

For the impatient, here is the executive summary:
This is a good tool.  Period.  It is, however, also a very specific tool that is intended to help find very specific problems.  If you are not doing any multithreaded code, there is no need to have it in your toolbox.  The cost ($890 US) is high enough that it doesn’t really seem worth it to get it “just in case” (unless, of course, you right a nice review on your blog and get a free license).  If you do work with multithreaded code, and you are concerned about deadlocks, this tool will save you lots of time, which ultimately means money.

Now, for the details

First of all, this is a tool from TypeMock, so I expected some pretty incredible things, even at the 1.0 release.  After all, their flagship product TypeMock Isolator is phenomenally powerful, so much so that the free license I was given for posting an advertisement for the release of their ASP.Net product has gone largely unused.  I’m just scared of it.  It’s like having access to a 50 horse-power tablesaw with no blade guard.  I may be careful enough to use it correctly most of the time, but the fact that it can take off my forearm without pausing to ask my permission makes me reluctant to get too close.  I do know that there are some very real problems that the tool can solve that just can’t be done with any other tool, so I have every intent of getting up my courage, grabbing a first aid kit, and jumping in to see what I can do…eventually.

But Racer is different.  It is a very powerful tool with a very specific purpose.  It makes it easier to run tests in multiple threads, and detect deadlocks.  As far as I can tell, it just detects deadlocks, not race conditions, as it’s name seems to suggest.  Not that this is bad, just that it only covers half of the rather shaky ground that is traveled while working with multithreaded code.  It is arguable, however, that dealing with deadlocks is the more difficult of the two problems, so getting this one tacked first is not a bad thing.

So how does it work?

Fairly straight forward really.  Start with a regular unit test, that is exercising some code that is utilizing locks in an attempt to be thread safe.  It looks like Racer supports just about every form of Synchronization supported by .Net, so you can feel free to use whichever mechanism you are more comfortable with to get the work done.  Now that you have a test, the simplest way to make use of Racer, is to add the Typemock.Racer library reference to your test project, and then add a ParrellelInspection attribute to the test.  This causes Racer to do it’s thing, which by default means running the test once with a single thread, and then run it again using multiple threads (2 by default).  If there are no problems, nothing much new happens.  You see some additional information about the test being run first with one thread, and then multiple, and something about scenarios executed…nothing that exciting.  The coolness happens when you actually get a deadlock.  For starters you test output includes a bunch of new information, the most noticeable of which is a big “DEADLOCK discovered when running the following” message.  Also is a description of the scenario that caused the deadlock.  Something like “Thread 1 acquired lock A, Thread 1 released lock A, Thread 2 attempted to acquire lock B, etc”.  Cooler yet is a message that says “To reproduce or debug this, copy the following attribute, paste it on top of your test and rerun the test:”, followed by something that looks like

[SpecificScenario("2", 1, 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2)]

Which, while being completely incomprehensible to humans, causes your test to run in the specific configuration which caused the deadlock to occur.  This means that you can accurately recreate the exact situation that lead to the problem.  If you have ever had to try to track down a deadlock in a live system, you will realize that this information just saved you countless hours of trying to recreate the production environment, and lots of trial and error getting things into the state that caused the problem.  One word: Brilliant!

So what is the down side?

I have to say that I’ve not yet been able to figure out how Racer determines what scenarios to run, or what the bits in the “SpecificScenario” attribute mean(well, the first string parameter is the number of threads, and the other numbers refer to the specific threads, and match the summary of the scenario, but beyond that, not a clue).  It would be interesting to know these things, but not really critical, as long as you are confident all appropriate scenarios are being executed.

There is also an interesting feature that I can’t quite get my head around.  When you run a test with a deadlock, an image is generated, which is supposed to be a visual representation of the scenario that created the deadlock.  Here is an example:

 

Now, I see the three objects I was locking against (sync1, sync2, and sync3), and I guess the odd rhombus shaped objects represent the threads, but I’m not really sure what the diagram is trying to tell me.  This is, no doubt, something which is still fairly raw in this early version.  I think it could be very useful if it were clearer what the shapes and arrows represent, but at this point it is simply a cleaver bit of kit that you can show somebody so that they can be confused too.

The last issue I can see with it currently is the price.  At $890 US for a single user license, it isn’t an impulse buy.  Granted, I think it can pay for itself easily after finding a few deadlocks in some production code (the earlier they are found the more that find is worth), but I don’t see it being a terribly easy sell for management, at least if you are not actively trying to correct a threading issue.  I feel pretty fortunate that I work for a company that understands the value of good tools, and is willing to provide them if there is a need, but I think it would take some convincing to get management to agree to purchase Racer simply because it is a good tool, and could really pay off “one day”.  If we were facing a threading issue, and I could demonstrate that racer would allow us to find it, and accurately reproduce it, it would be a fairly easy sell. 

So overall

I think this is an excellent tool.  Based on the fact that this is an early release, I can only see it getting better over time.  It is a rather pricey tool, however, so you may have to do some convincing to get the boss-man to get you a license.  There is a 21-day trial, however, so if you find yourself in a situation where you either have, or you could conceivably have, a risk of deadlock, then grab the trial, and use the first detected deadlock as justification to get a full license.