Empirical observation from the Project Management field of knowledge tells us that planning ahead of time saves a lot of resources down the road (entire PMBoK thing is basically just about this and the common dictionary thing).
As the level of complexity increases, projects require more planning in order to be efficient and succeed. Large-scale construction projects is one of the most vivid examples of that.
So, why do not people use similar principles in the development widely? Then, we could improve our success rate and save the resources by implementing certain functionality just with the component interactions (vertical solution architecture) instead of hard coding it. And I'm not even talking about constant quality, efficiency and risk management.
Reasons like "you just do not have the crystal ball to know about all the requirements" or "development is an art, and you can neither plan or control this" just do not seem to be valid.
- A. You can know about all the requirements that matter or that could matter (you just need to concentrate on these). All the information is available. You just have to work hard with it.
- B. Development is purely about logics (although efficient logical compositions are beautiful, just like it is with the math equations). If the same unit-test being run three times could fail once due to some unpredictable magic, then this would be art to develop something stable in such environment.
PS: the answer to the "Why" question is simple - the development looks too easy to start these days, while planning requires some effort. However, as the project goes - they swap places.
PPS: There actually are development situations where non-deterministic test results are possible (i.e.: sampling performance of neural network "learning" speed or the capability of evolution algorithms to avoid local extrema). Dealing with these could be considered an art. Fortunately, every-day development projects rarely have to deal with these.