Software Development Still Sucks
from the it's-not-pretty dept
We've discussed the Mythical Man-Month concept, where adding more developers to an already late software project tends to slow it down even more. It's an important concept that's often quoted, but not often followed by management. It appears that Scott Rosenberg has written up a new book that picks up where the Mythical Man-Month left off, looking at how software development is still a complete mess. Specifically, he focuses on Mitch Kapor's Chandler Project, which has taken years to not get very far. However, the point is that almost all software development seems to be an incredibly messy and unorganized process -- leading to all sorts of problems both in planning and in implementation of any particular software product. Basically, as much as it's called "computer science," it really isn't much of a science. It's hardly even a process. I recently had a chat with someone who had switched jobs from one big company with a reputation for extremely complex and buggy software to one that had a reputation for very clean software. He talked about how he thought he'd learn so much at the new company, only to realize that the development process there was just as messy (in some cases, even messier). Rosenberg makes a few interesting points, about how hardware improvements have followed Moore's Law, while it seems like software has almost been dragged along by accident -- though, perhaps a more accurate way of looking at things is that the hardware improvements of Moore's Law have actually allowed us to hide the problems of software development by simply throwing more horsepower at things. No matter what, it sounds like an interesting book that will probably ring true to many software developers.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Reader Comments
Subscribe: RSS
View by: Time | Thread
Processes are ignored
I love that quote, and am going to use it in my class next week.
There are processes in place - the SDLC, etc... but a large part of the problem it too many people are willing to take shortcuts - th eprocess is ignored, we can fix it later.
It is s sad state of affairs.
[ link to this | view in chronology ]
science vs engineering
Software engineering has a huge body of great literature behind it now, as a process that is almost 40 years old. The problem is with software engineering management since valuable texts like Sommerville are thrown aside as "adademic" and each management team thinks it knows best about how to reinvent wheels.
"Basically, as much as it's called "computer science," it really isn't much of a science."
Confusing CS with SE is a common mistake. "Computer science has as much to do with computers as astronomy has to do with telescopes" - I forget the source. Computer science is about fundamental principles, software engineering is about practical solutions to concrete problems.
"..perhaps a more accurate way of looking at things is that the hardware improvements of Moore's Law have actually allowed us to hide the problems of software development by simply throwing more horsepower at things."
Absolutely correct. The vast majority of software is bloated beyond comprehension. It starts with the seductive hypothesis that programmers time is expensive but cycles are cheap. With modern development environments in just a few thoughtless keystrokes a naive programmer can include gigabytes of unneeded libraries and create a circus of invisible suboptimal code. Then the complexity explodes. 75% of cost is in the last 25% of the lifecycle, test, debug and maintainance.
[ link to this | view in chronology ]
While experienced ones..
Which is the point where an experienced developer [sic] will throw out all of those libraries and roll his own tightly coded, highly optimized, undocumented routines, creating a new circus of buggy suboptimal code.
Then the software, the server, and the company explodes...
[ link to this | view in chronology ]
Re: While experienced ones..
Meanwhile a 13year old kiddy in Kazakhstan has written a much better implementation of the original idea in Brainfuck which runs on a TI-83 calculator in 2k of RAM, that nobody will ever hear about ouside the demo scene.
Lions led by donkeys.
[ link to this | view in chronology ]
It's not the code, it's the people
As long as there are people with personalities empowered to make decisions and be creative software dev will always be a mess.
There are different (often better and worse) ways to implement everything, from the language itself to the talent with which it is employed.
Want mess-free software dev? Standardize everything, lobotomies all around, and love the total crap you end with ;-)
[ link to this | view in chronology ]
preemptive deadlines
[ link to this | view in chronology ]
'Software Development' is not 'Computer Science'
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Ummm... wrong.
[ link to this | view in chronology ]
Re: Ummm... wrong.
It's kinda funny how you do it yourself :)
[ link to this | view in chronology ]
Re: Re: Ummm... wrong.
[ link to this | view in chronology ]
Re: Re: Ummm... wrong.
[ link to this | view in chronology ]
The true problem....
Look at how medicine does things. If you develop a new procedure or technique for doing something, often you publish your techniques and your peers critique and refine your technique. This technique is then taught to new doctors along with others before it, and the body of tested, proven knowledge grows.
In Software Engineering, however, seems like everyone has their own home brew for doing every thing, and we are too quick to take the "let's start from a blank page and make everything from scratch" approach. I guess part of this can be attributed to the fact that SE grew up in the world at a time when copyright and patents were king, making it very difficult to do external review. However, has anyone been to an internal code review lately? Seems to me that most of the ones I've been in degenerate into this "your member variable needs to be named this, or your comment needs to say this". I think we would be better served to concentrate on the fundamental algorithms implemented (at code reviews) and also attempting to enforce re-use of code more often as a 1st line of development.
Basically, what I'm saying is that SE needs to develop an "institutional memory" of proven techniques to address problems that have been critically reviewed and accepted as standard. These techniques need to be taught by students and as more improved ones come along [and are peer reviewed] incorporated into the body of knowledge.
[ link to this | view in chronology ]
Re: The true problem....
[ link to this | view in chronology ]
The true problem....
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
A House of Clay
1) The software world has not really come up with the functional equivalent of a "blue print," a straightforward diagram speaking to the customer (whoever determines what the software should do), the construction workers, and the architects. So, a customer might say they want the software equivalent of a "2000 sq.ft. house with 14 windows and 2 doors" but they won't be able to really envision it until its built. And, if the development team is bigger than 3 engineers or has to "prefabricate the frame" before it is assembled, then the "support beams" might not actually connect properly.
2) Code is maleable like clay. It is easier and, occassionally, less time consuming to simply build the shapes of the code and then press it together than it is to create an accurate, detailed design.
Hence we build a house of clay. The word science is not really a question...its more the word engineer that I question.
[ link to this | view in chronology ]
The problem comes down to...
The ego of management, coupled with the ego of the 'star' developers.
[ link to this | view in chronology ]
Software Development & Moore's Law
It is misleading to claim that hardware is improving according to Moore's Law while software is not. Look at how most of the transistors are used on a typical present-day CPU chip: cache memory consists of a single cache line circuit replicated thousands of times; the circuit was only designed once, yet every one of those replications counts towards Moore's Law. Similarly multiple function units for doing addition, multiplication, division etc--modern CPUs tend to have several of these, which are all identical, yet again they add to the total transistor count.
In software, we are not allowed to count such multiple repetitions of the same code. But if we did, you could easily come to the conclusion that software productivity is way ahead of hardware productivity.
[ link to this | view in chronology ]
Re: Software Development & Moore's Law
[ link to this | view in chronology ]
My Opinion is:
misanthropic humanist, "Lions led by donkeys." is one of my favorite qoutes too. It really is a perfect description for this topic. :P
[ link to this | view in chronology ]
Chandler
- establish a benchmark for how great software can be created
- create software that can be used and appreciated by ordinary folks, not just corporations
- provide a showcase and platform for open source and free software
- keep up with the latest fads, and start new ones, so the project and its principals would have buzz and be in constant demand at parties
- set Microsoft back on its heels and maybe even lower its stock price
All this is nice (especially the last), but it has little to do with making and keeping customers and growing a business. Kapor did an outstanding job launching Lotus, but after that his record wasn't so good, in spite of all his intelligence, user interface savvy and people smarts. IIRC ON Technology was originally launched as some sort of generalized services platform on top of the desktop operating system; they hired a bunch of "rock star" types to flesh out the vision. After operating 2-3 years in stealth mode and having little to show, the company changed direction and decided to commercialize a calendar tool they'd developed called Meeting Maker. Even that took a lot longer than expected; I remember a long WSJ article written by a reporter who was given access, that probably was a short version of Rosenberg's book.
Then there was GO Corporation, the pen computing startup that Kapor founded with Jerry Kaplan. In fairness, that idea was simply ahead of its time.
[ link to this | view in chronology ]
Re: The true problem....
Within a limited domain application factories are a good idea. If you can solve any problem just once and do it correctly the rewards are huge. The only downsides are:
1) Absolute correctness. You need to invest at least 5 times the development time in making sure your thing factory works, otherwise every single application you generate is flawed, a position far worse than cocking up only one clients code. Given that ratio you probably want to reuse your code generator on at least 10 or 20 projects.
2) Flexibility/brittleness. Can you knock up a medical spectrographic analysis tool that uses discrete wavelet transforms using your framework? I doubt it very much if you primarily designed it to create e-commerce solutions.
You definitely read those other programmers wrong. All programmers are lazy, it is a singular virtue, and they would welcome any enabling tool that reduces the number of lines of code written.
[ link to this | view in chronology ]
The true problem....
Thank you for your thoughts. M.H. Wrote: 1) Absolute correctness. You need to invest at least 5 times the development time in making sure your thing factory works, otherwise every single application you generate is flawed, a position far worse than cocking up only one clients code. John's Answer: Once you get used to the idea of never writing applications but rather the generators instead, development is just as fast. And if you do find a mistake, all you need to do is fix it in the generator and regenerate an app for all your customers rather than digging through the code of each customer. Also, there are rarely mistakes because the generator does all the typing. What I do is capture the technical specifications of the customer in an expert system rather than on paper. This allows me to do all kinds of "what if we do it this way" experiments with the customer. I can let the customer play and discover what he really wants rather than trying to lock him down with a static technical specification that he must sign off on. So the customer gets zero typing errors, fast development, and he can change his mind about what he wants or need all at no extra cost to me.
M.H wrote: 2) Flexibility/brittleness. Can you knock up a medical spectrographic analysis tool that uses discrete wavelet transforms using your framework? I doubt it very much if you primarily designed it to create e-commerce solutions. John's answer: Of course the math routines would have to be coded by hand but presentation, data access, security and so on is quite common and doing all that by hand and then chasing down the bugs and then changing it all by hand because your customer really wants a different look and feel takes so much time away from the important fun and creative work that developers should be doing. I have found that software development doesn’t have to suck. The key for me was chosing to see what's common in about what we do and passing those jobs off to a computer.
[ link to this | view in chronology ]
THANK YOU MIKE!!!
*THIS* is the kind of stuff that makes me still subscribe to Techdirt. Insightful commentary -- in this case, a book review -- based on information and analysis, and not drama, or BPM (bitching, pissing and moaning). I've commented before that Techdirt's signal-to-noise ratio has too much NOISE. THANK YOU THANK YOU THANK YOU for proving that you guys still have signal strength.
(Sorry, this doesn't contribute much to the discussion, but I think while it's useful to try and point out when things are going wrong, it's equally important to cheer loudly when things go right.)
[ link to this | view in chronology ]
First, most people actually writing code have little or no training in how to do software development. Sure they have had a few, or even a lot of programming classes, but these classes almost never teach anything other then language syntax. This includes many CS majors I have meet who have no clue about gathering requirements, revision control, test plans, etc.
Second, speed is more important then quality. Getting things out the door as fast as possible ALWAYS trumps requirements gathering, complete testing, robust error handling, proper use of revision control, etc. In 17 years I have yet to work on a project where this was not the case. Because these quality controls are not done along the way something ALWAYS gets screwed up, thus you get more bugs and never finish in time.
Third, lost cost is more important then quality. The bottom line controls everything. No matter how much something actually costs management ALWAYS wants to spend less. This is why most projects end up overbudget. Penny pinching leads to cost overruns in the long term.
Finally, management is self serving and has no backbone. Managers main goal seems to be to make themselves look good in the eyes of the higher ups. There goal is NOT quality software. Quality does not make them look good. It actually makes them look bad. Low projected budgets, aggressive timetables, etc are what make them look good. SO even if all the expert developers under a manager say something will take X amount of time and Y amount of money the manager will always back down when those above them say it needs to be done faster and cheaper.
None of the above is unique to just software, I've seen it on every hardware project as well. The real core problem is not really managing projects the way they should be.
[ link to this | view in chronology ]
Problems No Unique to Software
I feel that writing generators solves problems 1-3 that you have presented, which were developer training, speed of development, and cost.
I feel your comment that the problem crosses over to other disciplines is very insightful. I disagree with your assessment of what the actual problem with managers is. In my opinion, managers respond to incentive just like everyone else. They have a room full of programmers and they get a certain amount of money for each hour of development that each one of them does. Yes, they can reduce their costs if they can do the job with less programmers and in less time, but they can't bill as much either. Since the customer pays the cost, incentive causes managers and programmers alike to believe and sell the idea that only people can program. So as long as industry spokesmen (all of us) can convince ourselves and the public that this is true, then programming will continue to suck and we will all continue to profit from that notion. No disrespect intended; it is human nature to honestly believe what it is profitable believe in.
[ link to this | view in chronology ]
So what?
So what? My truck is not very efficient at getting me to work every day either, and the engineering required to build it was hundreds of years in the making.
Software (and its development) has revolutionized faster than anything in recorded history. Is the process messy? yeah. so what?
[ link to this | view in chronology ]
Schrodingers truck
Interesting attitude BOF, I kinda like it and I could follow that reasoning up to a point. I mean computers are only going to get faster and faster right. If it achieves its goals who cares?
Here's where it breaks down. The problem with the truck analogy is that there is no strange hidden condition where if you pull out the throttle, with the gear stick in reverse, but the steering wheel at exactly 15 degrees while there is exactly 33% fuel in the tank...the entire fucking thing just explodes and kills everyone.
For all its inefficiencies it wont fail spectacularly and catastrophically for some arcane reason.
That's exactly what software does once its complexity goes above a certain point.
[ link to this | view in chronology ]
WE NEED A COMMITTEE!
[ link to this | view in chronology ]
Software Engineering
With 700 page books being commonplace in the industry, that go out of date in a matter of months, it is no wonder that things in development get a little chaotic.
There is nothing like software development, to compare it to anything else doesn't make sense. For example, above it was compared with "hardware" that keeps getting faster. Never mind that the hardware really hasn't changed that much, just gotten faster. I don't think many people would be lined up to buy the latest copy of Word if it was just "faster." No they want it "simpler," and paradoxically "with more features."
There are literally thousands of software development methodologies. Most companies try and follow one. When you hear someone say there company doesn't have one, chances are they do, but that particular programmer doesn't like they one they have.
So give software engineers a break. If there were a better way, they would take it.
[ link to this | view in chronology ]
The Real Reason
coding is fun! Planning isn't. I agree "programmers are lazy".
After years of fun I learned that the easiest way to get a job done is to do it right the first time. Of course that requires a
decent set of firm specs ( not a trivial requirement ).
I have been involved in several large projects ( 15 or more programmers ) that were very successful ( no reported bugs
after years in thousands of installations ). That was accomplished by months of planning before any code was written. The code was written in about 5% of the total time. The only problems found in Quality testing were due to coding typos. Wasn't fun, but fixing bugs for irate customers isn't fun either!
[ link to this | view in chronology ]
Wowee
I also didn't know that IT Managers or Project Managers were ever hired without so much as writing one line of Basic. I've looked over the requirements for these positions, being a project coordinator myself, and what they want would require the years of experience that you guys have!
(I do know about managerial agendas.)
Thanks for the eye-opener.
[ link to this | view in chronology ]
Software Development Still Sucks
[ link to this | view in chronology ]
This!
First, most people actually writing code have little or no training in how to do software development. Sure they have had a few, or even a lot of programming classes, but these classes almost never teach anything other then language syntax. This includes many CS majors I have meet who have no clue about gathering requirements, revision control, test plans, etc.
Second, speed is more important then quality. Getting things out the door as fast as possible ALWAYS trumps requirements gathering, complete testing, robust error handling, proper use of revision control, etc. In 17 years I have yet to work on a project where this was not the case. Because these quality controls are not done along the way something ALWAYS gets screwed up, thus you get more bugs and never finish in time.
Third, lost cost is more important then quality. The bottom line controls everything. No matter how much something actually costs management ALWAYS wants to spend less. This is why most projects end up overbudget. Penny pinching leads to cost overruns in the long term.
Finally, management is self serving and has no backbone. Managers main goal seems to be to make themselves look good in the eyes of the higher ups. There goal is NOT quality software. Quality does not make them look good. It actually makes them look bad. Low projected budgets, aggressive timetables, etc are what make them look good. SO even if all the expert developers under a manager say something will take X amount of time and Y amount of money the manager will always back down when those above them say it needs to be done faster and cheaper.
None of the above is unique to just software, I've seen it on every hardware project as well. The real core problem is not really managing projects the way they should be."
This, this, and this! Did I say this?!
Nailed it! You Sir, just stated cleanly and succinctly why I am leaving this good-for-nothing God-forsaken so-called 'profession'.
"Make those good-for-nothing code apes work faster so we can invoice, or else we'll outsource them all": your bosses' boss. He has an MBA, don't-you-know.
[ link to this | view in chronology ]
It is very exciting to read this article now. You are right, even in today's reality, creating software is not such an easy task. Although most developer companies such as https://www.timesolv.com/ I have long optimized the process of creating high-quality programs, there are still difficulties. The process of creating a new program is very complicated. This is a whole business process to which you need to approach very seriously and consistently
[ link to this | view in chronology ]