To paraphrase Winston Churchill "Documentation is the worst form of communication available to you... except for all the others that have been tried".
One of the agile principles is "The most efficient and effective method of conveying information to and within a development team is face-to-face conversation". Conversation and close collaboration are fine on small projects with little novelty and little staff turnover. Let's assume that you use conversation and close collaboration. What do you do when you need to remind yourself of a decision that was taken a few weeks (or months, or even sometimes years) ago? How does conversation help that, unless you have recorded the conversation (and guess what, if it is written, that's a document, just one that is particularly hard to use). What if the best answer to "why are we doing that?" is "go ask Joe", with whom you had the conversation? Well, I have news for you, Joe left six months ago and now works for a competitor, so good luck with that, you had better hope that he documented something.
Communication through conversation is suitable for ephemeral information. If the information has a life, it needs to be recorded. If this is not done as a group activity, then each person on the team will make their own record, which will likely be incomplete and almost certainly inconsistent with the record of other people. Try this experiment next time you are in a meeting - get two people to take minutes (independently of each other) and compare afterwards what they have written.
So you need to record things, and the natural form of record is some form of document. The main problem with documents is not the documents per se, it is their use as if they were the product. They are not, they are just a (form of) repository. Ideally, as someone else has pointed out, the documents would just be reports. We use documents because they have structure, and that brings benefits, like making it easier to find things and also giving guidance about what needs to be considered by having sections calling out specific topics (what, it had to be secure? we never had a conversation about that!). By all means converse and collaborate closely, but record the results in a form that you can easily find them when you need them in the future (which you will, except for the most trivial systems).
The hatred of "big" requirements documents seems to me to largely stem from a post hoc ergo propter hoc logical fallacy - these failing projects had big requirements documents therefore big requirements documents caused the failure. Of course it is essential to document requirements, it is stunningly naive (but amazingly successful for methodologists on a personal basis) to think otherwise, though it does pander to the prejudices of developers. As I have said before, documenting requirements does not always or necessarily mean a requirements document. However, let's not forget the advantages of a document: amongst others, it gives us one place to find the requirements for the whole system, it gives us a great way of locating related (overlapping, duplicate or inconsistent) requirements, it gives us a basis for configuration management, it is very often needed for contractual reasons.
There is absolutely no reason why documents cannot be developed incrementally, filling in information as it is obtained. There is also absolutely no reason why sections (even line items) cannot be approved and acted upon independently. There is also absolutely no reason why every requirement has to be in one document, so long as people can find the information. In particular, it is extremely bad practice to place requirements from different levels (e.g. stakeholders and architects) in the same document. Likewise, why document the detail of all (sub-)systems in one document?
One final question - if documents are so bad, how come the form that most people choose to use to argue against them is ... documents? Surely they should just converse and closely collaborate with everyone else to get their points across?
