Wednesday, 20 May 2009

Agile versus the traditionalists - Part 1

I have called this Part 1 because I suspect know that this won't be the last entry in this blog on the topic. It's just too big.

Let me make my position clear. I am firmly in the "traditionalist" camp. But, having said that, I think that there is a lot of value in agile approaches. I'm certainly not going to dismiss it out of hand. Of course, some practices associated with agile (like pair-programming) have proved to be less effective than the original proponents claim, but there is a lot to be said, in many environments, for ideas such as test-first design, use of backlogs, user stories, etc. But there is also a lot to be said, in many environments, for having documented requirements specifications.

For an excellent description of what I am talking about, see David Parnas's classic paper "A Rational Design Process, How and Why to Fake It" in (D.M. Hoffman and D.M. Weiss, Software Fundamentals: Collected Papers by David L.Parnas, Addison Wesley, 2001). Even though you didn't really develop your product that way, you should make it look as if you had a chronological flow of development artefacts (for example, stakeholder requirements, system requirements, architecture, etc.). Why is this important? Well, firstly, the development team won't be there for ever. So it won't be possible to just walk over and ask Joe. Joe is now working for your competitor. But someone still has to maintain the system, so you need some way of recording the common knowledge. But we have all our user stories, you say. Unfortunately, a list of user stories does not always give a sufficiently clear overall picture. Sometimes, you need to be able to understand the whole system at various levels and user stories just don't do that.

So it may still be necessary to document requirements. Note that "document requirements" does not necessarily mean the same as "requirements document". But a requirements document does allow an overview of all requirements in a form that is familiar to people.

But isn't there an overhead in doing this? Sure, there is. The question, though, is: does the investment pay off? Well, if you are delivering a "small" system, say up to a couple of hundred requirements, it may well not. Such a system probably only has at most a few tens of user stories, and experienced people can hold these in their heads. This sort of development is ideal for agile. But when the numbers of requirements starts to creep up it becomes essential to be able to see the big picture. Another situation when the need for a requirements document occurs, even with a realtively small requirements set, is where the requirement specification forms part of a contract. Software has the great advantage here of being incredibly malleable, and also usable even if not complete. Hardware systems generally have neither of these characteristics. Even something as simple as a garden shed to hold tools would be useless without one of its major components, without walls the weather and theft protection is lacking, without a door you can't put things in or take them out, without a roof you lose weather protection, without a lock you lose theft protection, and so on.

So where is the gain in creating a requirements document? Simply, it presents the requirements in a consistent, navigable and understandable way. So, if something needs to change in the future, it is easy to see where that change needs to be made. This means that it is possible for people who were not involved on the original project to make changes in the full understanding of what these changes mean.

And more: if you don't just create requirements documents, but use a requirements tool, you can also see what information is linked. So when a stakeholder says a requirement needs to change, you can see what effect that has right down to implementation level. Simply having a set of user stories, without the linking, does not allow this. And naïve use of user stories will make it hard to use common components (I have seen this happen in practice, leading to a system that we estimated at being around three times as large as it needed to be because people re-implemented).

So, what do I recommend? Surprisingly, perhaps, I would say "use agile". But use it with your eyes open. Use it where appropriate. And don't be browbeaten into not documenting your requirements just because that is the fashionable thing to do.