Hi Mike,
Creativity is the word, and there is a book on this subject 'Software Creativity' by Robert Glass, that is a compendium of articles actually.
All that you say I agree with fully, another reason metaphors in software are problematic. My interest is mainly why large projects fail and why tendering doesn't work that well for software. I suggest when it does work, that the cost is far greater than is necessary, at least double in my experience, in order to guarantee a result, irrespective of it being a good result. There is a tender for a 'health information exchange' locally at the moment, a perfect thing for open-source in my view, complex yes but essentially commodity. I know that this will not be the result.
In such "human-facing software" projects there is a gradient of sorts from the unfamiliar where creativity is needed to the familiar. In the film metaphor (see, its hard not want to use analogies), we can think of multiple 'takes' to get a scene done perfectly at one end, to the cutting-room where the range of options is much limited. So, I guess my main interest is the issue of 'creativity at large scale'. Maybe leadership is what is needed, someone to inspire and marshall everyone to work as one. Great and complex things are built by a few individuals with a shared vision too, but over longer time frames than is usually required for 'procurements'.
