The article concerns the problem of excess of program creation terms as a result of a prejudice that programming is simple and even simpler.
Failure to finish development of software in time is a common thing in practice of program product creation. It is bad for all the participants from the ordinary developers who have to work in hectic rush to the managers and clients who have to put up with lags and extra costs.
There is a great amount of reasons for this phenomenon but in this article I would like to suggest that the main cause is a wide-spread belief that software development is simple. And I would like to emphasize an interesting thing here. On the whole, everybody understands that program creation is rather a laborious process but as it comes to discussion or presentations everyone pursuing his own interests starts saying that all the difficulties are left behind and it is now when everything becomes simple and easy when we can use a new technology / programming language / methodology. This happens again and again for many years. But on the whole there are no improvements and terms are exceeded as before. It is the causes and consequences of these phenomena that I would like to touch upon in this article.
I will rely not on abstract speculations but on my own experience. I am involved in creation of static code analyzers Viva64 and VivaMP to check programs written for 64-bit systems and created with the use of OpenMP technology. In such programs there are some error patterns which are, however, rather easy to detect with the help of source program-code analysis. But it is not the main point. The matter is that while promoting these tools I faced great resistance caused by a wrong idea that 64-bit technologies and OpenMP are very simple things. The result of this confidence is the necessity to overcome immense resistance of people who consider that there are no problems and that there cannot be any. So to say, I turned out to be the enemy of their peace. Of course, it is the same with other technologies but in this article I will speak upon OpenMP technologies and 64-bit programs as the most familiar to me.
The belief that everything is simple is not unusual. You are constantly being conceived that you can easily make your program a 64-bit one and get the increase of performance and immense memory size at once. It is described how you can easily turn your code into a parallel one by using OpenMP. We are surrounded by different articles containing theses of "it is enough just to recompile an application", "by a simple arrangement of OpenMP directions" kind. And I am walking around at this wonderful feast with posters with large titles of articles "20 issues of porting C++ code on the 64-bit platform", "32 OpenMP traps for C++ developers". And I feel just like a mean little old man who is dissatisfied with and is constantly grumbling at everything. It is this strange feeling and resistance to "simplicity adherents" that made me ponder.
A thought appeared that perhaps it is me who is wrong. Together with others I must say that everything is simple. And that with our tools it is even simpler. As simple as it just can be. But something is wrong. There is a contradiction. I am familiar with the actual state of things and I am sure that certain difficulties exist that await programmers while mastering parallel technologies and 64-bit systems with large memory size. There is no simplicity. It is deceit. And deceit is harmful. It is harmful to me as a person who is interested in promoting his tools. It is harmful to a manager who cannot forecast the terms correctly. And of course it is harmful to a programmer who will experience disappointment in new technologies and will face overtime work. That's why I decided to adhere to my opinion and I will try to change your viewpoint too.
One can still try to convince me that creation of 64-bit programs is not so difficult, that some programs can be recompiled without corrections for 64-bit systems and that errors can be automatically found and corrected when different methodologies are used. Although it would be not so easy because I possess experience proving that these processes are complicated and that one can easily make a great mistake forecasting the terms.
But when parallel programming is viewed in the same way it is sabotage. It is a complex and too difficult task which cannot be solved by a simple arrangement of OpenMP directions around loops and use of some library. But it is considered to be so. A lot of developers who haven't studied the questions of creating parallel algorithms yet are convinced that one shouldn't think about it and the problem will be solved by OpenMP or any other technology. It seems easy and clear to parallelize any program according to these articles. But popularization of what the reader cannot do and check himself creates only an illusion of understanding. Perhaps it enlarges his vocabulary and mind but it also brings this useless illusion of understanding where there is actually no understanding and knowledge.
This doesn't mean that OpenMP or any other technology is bad. They are all perfect. But you should understand their possibilities and impossibilities. Don't hope to parallelize your existing programs easily. Most likely, you will have to create new algorithms and convert data structures and mechanisms working with them. If you want to get sure try yourself to parallelize the algorithm of array sorting which everybody wrote at school and university quickly. If you fail, look at Batcher's parallel sorting. Don't find it simple? Of course, it is not. This example is given to show what kind of difficulties you are to face when dealing even with common algorithms used in your programs. But it is not enough. Perhaps you use iterators in loops in your C++ programs, don't you? But do you know that when giving examples of for parallelizing popular articles concerning OpenMP do not mention that iterators cannot be used in it? Indexes should be simple data types (for example, int). That is when you calculate time for reconstructing algorithms keep in mind that you may have to change data structures and methods of working with them as well. And this all demands additional time and effort.
There are no bad people who want to deceive you and convince you that everything is simple. But there are different interests leading to mutual deceit. Those who promote 64-bit hardware platforms and operating systems are surely interested in convincing you that it is very simple to port your applications on them. It is understandable. Porting on 64-bit systems is already delayed and if one says that there are difficulties it can prolong it for several years.
Developers of parallel systems advertise different program technologies and libraries saying that everything will become parallel nearly by itself. Those who create these libraries/technologies try to convince programmers that their library/technology is the best because it is the simplest. And a sad confusion occurs. The fact that a library is simple and convenient (for example, in my opinion, OpenMP is simple and convenient) doesn't necessarily mean that creation of parallel programs is simple too!
Programmers who have read a lot of articles telling them that everything is wonderful in 64-bit and parallel worlds but who lack enough practice deceive their managers and themselves when it comes to terms. Managers deceive clients. However managers themselves prefer to consider everything simple as it seems to them that it will help to save on developers. As the result there are no guilty but everybody deceives each other.
While writing this text I came across a wonderful article by Dijkstra "Two views of programming" which is very close to this article in sense. I would like to quote a short passage which sums up everything said before in a very good way: "
It is not so much the computer manufacturers, that want to do as if they sell an easy product; it is not so much the managers of software projects, that would like to view the programming activity as a simple and predictable one; it is not so much our educational institutes, that would like to train their students with predictable success.
It is the comfortable illusion of Man as elaborate automata that, like a drug, seems to have freed its victims from the burden of responsibility. Accepting programming as a hard intellectual challenge would place the full weight of that burden back upon their shoulders.".
But theweight of this burden mustn't become the reason to go on keeping our eyes shut. It is harmful to everyone. We must accept the problem as it is.
The conclusion of this all must be a more critical view of the possibilities of new technologies. New means and methods are useful but they have never been able - and perhaps never will be - to replace the very process of software development which consists of many steps and items. First of all, it is necessary to study and improve the development process and only after that choose new technologies for implementing a project.
Wonderful books by Steve McConnell "How much a program project costs" and "Code Complete" are indispensable to breaking the circular deceit of simplicity. Besides, one should pay attention to a cult book by Frederick P. Brooks "The Mythical Man-Month: Essays on Software Engineering".
I wish you soon recovery, success in forecasting difficulty of program projects and their completion in time!