Thanks for all this wonderful discussion.
I am very interested the Domain Driven Design approach of Eric Evans, for the main reason that it recognises (to me) the importance of human culture in some projects. The car analogy of Ihe is perfectly valid, but that is an engineering persective and more careful thought up front is capable of making the right specification, that is what structural engineers do all the time and very successfully.
The extra problem with some software is that it comes to be an extension of ourselves, particularly that which is used in everyday work. That is why free and open-source has not taken over completely the desktop environment, the economics of it suggest that it should, that word-processing and spreadsheet software is now a commodity. People get familiar with doing things a particular way and if one person doesn't know how to achieve a goal another person does, this is the view of culture as a shared mental map of the world and how it works. People find ways of working around problems in existing systems, then once those practices are repeated, they become ingrained, effectively at both a personal and organisational level
So I agree with Kurt about systems thinking, but the scope of the system being analysed includes aspects of basic human nature. This is where DDD seems to offer some assistance as it places all the analysis focus on the main feature of humans 'language'. By teasing out the meaning of words used by Domain Experts two things happen; firstly the developer/designer (the difference seems rather meaningless to me) comes to understand the domain intimately and this flows through into the domain model built into the software; secondly the assumptions, and possible inefficiences, that the Domain Experts themselves have made, to get their jobs done, come into sharp focus.
As a result of this second factor there seems to be real opportunities for a redesign of the whole system, software and ways of working (culture), one that is non-threatening to the actual workers as they are part of the team. Having to explain something to someone is a very effective way to analyse the fine detail of a domain.
My ideal is now one where a prototype system is built in real-time, with a business analyst, domain experts and a designer (typing code madly) all working together on it. This seems possible via a Naked Objects framework.
If you contrast this to the creation of a requirements request-for-tender document, in-house people are trying to redesign a system, but largely ignoring this cultural aspect. They are using blind-faith, hoping that what is provided will be accepted as an improvement. If there are problems at the end for any reason, such acceptance, once disappointed is very hard to re-establish. Agile is capable of handling culture as it involves users, but DDD goes further, or is useful in more complex scenarios perhaps.
One aspect of good leaders IMO is that they are capable almost intuitively of handling this human aspect of systems, not many technologists are, that is why they become technologists quite possibly, they gravitate to things that are more predictable, or, that they can tell to do something (figuratively) rather than having to persuade or convine that the old 'tried-and-true' methods can be replaced by something better.
In the analogy of redesigning a car, the thing that would stop such mistakes is market-research. In the case of custom software the 'human aspect' has to be incorporated in the process in some more effective way. This tendency for human nature to result in a 'herd mentality' is very strong in the IT industry itself i believe, so its not strange that it should be such an issue in the design of complex software.
The design of physical products is more amenable to an up-front requirements gathering and RFT type approach, as its more a case of defining the boundaries within which a solution must be provided. From that point the solution providers assume most of the risk I believe, that is reason tenders are so legalistic, to try to prevent loopholes. This does not work with software.
If anyone is interested in this I also have to recommend, not an IT text, but one by a primatologist, called 'The Age of Empathy' by Frans de Waal, where he draws parallels between discoveries from observations of primate society/culture and the human. We have complex language and use of symbols and tools, but underneath this visible layer are more basic universal behaviours/drives, that essentially make it possible to live together, largely via empathy. I am also about to read 'The Social Life of Information' by Brown and Duguid, which I think will have more specific but similar insights.
So, perhaps good IT project managers have a good ability, perhaps a natural ability, to take these human factors into consideration. When telling people what they should do, in a top down way, is just not going to work.
