Does Software Development Have A Culture Of Rewarding Failure

FailureHave you ever worked on a really successful project, one of the rare on time and on budget ones? Did you get a reward and a lot of praise in the end? Yeah I didn’t think so. Now think back to a project where everything went wrong but through super-human efforts you were still able to get the beast into production. Any accolades for this one? My guess is you probably got a few and if not you personally, than someone on or around your team did. This seems to be a really common scenario with software projects; people who are associated with highly successful projects are forgotten while people on “death marches” (that are too expensive to cancel) are praised and hailed as heroes.

The Well-Run Project

Really successful projects are few and far between but when they do happen they are a joy to work on. Not only that, but the software delivered is higher quality and meets expectations. Budgets and deadlines are adhered to and everyone associated with the project is happy (more or less :)). Most of us dream of working on such a project. So when a project like that finishes, how does the business show it’s appreciation and express it’s desire to see more projects of the same caliber (i.e. what kind of positive reinforcement is used)? Well, everyone gets a pat on the back and then moves on to the next thing. A little anti-climactic wouldn’t you agree? You see, everyone expected the project to go well and when it did, noone was surprised, everything went according to plan, why would anyone reward or even acknowledge it when things go according to plan. That my friends is what I call negative reinforcement.

The Crappy Project

Crappy, death-march projects are routine in software development. Projects begin with unrealistic expectations, concrete commitments are made based on estimates and information that is far from concrete. And when things inevitably go pear-shaped developers cop the brunt of the fallout. We are forced to shortcut our process and compromise on quality and when that is not enough, well we just work longer and longer to make up the shortfall. At the end everybody is stressed and tired, but sometimes the gargantuan efforts are enough to get the project (or what’s left of it) into production. When that happens, we crack open the champagne and hail the project as a miracle of modern software development. Everyone shakes their heads and marvels at how the team was able to do the impossible. Awards and praises are handed out and everyone is too happy to notice that the project was actually a year late and millions of dollars over budget. I lie, everybody is well aware of the fact that the project was a complete debacle, but with so many people patting each other on the back, it just doesn’t seem politic to say anything. You would agree that this is the ultimate in positive reinforcement.

It Is A Bizarro Universe

The above scenarios strike me as being completely cuckoo. We acknowledge and praise people who try (and sometimes succeed) to pull projects out of the jaws of failure but don’t do the same for people who push project into the ‘jaws of success’.

Have you ever tried to train a dog? Next time you do, try the following. Every time a dog does something you like (such as fetching a stick or rolling over), a positive behavior – just ignore it. But every time the dog does something ‘bad’ (like peeing on the carpet), a negative behavior – give it a treat. My guess is you will quickly end up with one confused and badly behaved dog, and yet we do this on an industry-wide level. It is truly bizarre.

This Culture Needs To Change

In an industry where project going pear-shaped is the norm, it really behooves us to look more closely at the ones that don’t. I for one would love to see a little bit more appreciation from everyone for projects where things go according to plan and especially for the people on those projects. It might sound crazy but I do believe that delivering a year-long project on time while having all developers work at a sustainable pace (oh I dunno, let’s say 40 hours a week :)) deserves at least an honorable mention for the people involved.

The other side of the coin is rather than celebrating the belated delivery of the latest death-march, how about digging into it and trying to figure out why it was 6 months late and why people had to work 80 hour weeks to keep it from complete disaster. And when we do figure out what the problems were it may be worth it actually learning something and trying not to repeat the same mistakes rather than just writing it all up in a report and putting it on the wiki or in the document repository. Because wikis and repositories are just some of the places where lessons go to die.

For more tips and opinions on software development, teams, process and people subscribe to today.

Image by salimfadhley

  • I always get jealous of someone not only being able to clearly see a problem like this, but then also being able to articulate it so well. Top post, keep them coming!

    Two weeks ago I had a cross discipline meeting (a rare occasion) where them peeps who get to see and speak with actual customers were speaking of, surprise surprise, content and even impressed clients. Why on earth this rarely ever tickles down to the developer work force I fail to understand.

    You’d think a sense of achievement would help motivation and thus productivity. This week we had a celebrated project, guess what; lots of overtime, causing problems with other projects because of claiming their resources etc etc. Have some pie!

    • Yeah, whenever I hear of people on a particular project going ‘the extra mile’ (i.e. working obscene hours) and then getting acclaimed and rewarded for it, I always think of how the developers on projects that don’t require this kind of commitment feel.

      It creates the perception (a true one) that if you want to be recognized and rewarded for you skill you must be on a a project with issues. Because of this developers sometimes consciously want to work on death-march style projects just so they have a chance of being seen as valuable by their company.

      • Jeremy

        It makes me wonder if those who are “recognized” should spend their 5 minutes explaining how if the project had been properly planned they wouldn’t have had to kill themselves.

        If one person did it it’d be suicide, if we all did it…

  • Ewan

    I think you have hit the nail _very_ firmly and squarely on the head with this post. The dog (mis)training analogy is perfect.

    I have seen many projects that were nothing other than complete and total unmitigated failures but the screwups where so endemic and started so high up in the organisation that to admit that the project was anything other than a complete success is total heresy.

    I have often noticed that once a company has gone through a couple of these situations, you start to see a cult of hero worship grow around a few individuals who are deemed to have “saved” a failed project (“saved” is obviously a relative term…. ). In reality they often just got lucky with a few decisions. Very quickly their opinions and views become gospel. They build up immense, unofficial power and any attempt to try and actually rectify the fundamental issues are blocked and subverted for fear that they lose their power base.

    We _are_ destined to keep repeating the same mistakes.

    • I know exactly what you mean, the real danger here as that those individuals actually start to believe that they have some magic powers to “save” projects, at which point they don’t really want to hear what anyone else has to say.

  • Ville Laurikari

    Great post! This reminds me of the book Death March by Edward Yourdon. Highly recommended. He categorizes death march projects into four categories (ugly, suicide, kamikaze, mission impossible). One of these categories is perhaps even desirable. The book also has some good tips on how to survive a death march project without losing your sanity.

    • I read that book years ago and would definitely add my recommendation to yours. It is certainly one of the books you should read as soon as you can when you become a developer along with “Peopleware” and “Mythical Man Month”.

  • Dale Schumacher

    This article is right on, and I agree with other comments that it is very well articulated. However, I do train dogs, and would like to suggest an improvement to your metaphor. Instead of rewarding the negative behavior directly, yell at the dog and when he stops, reward the “successful” termination of the negative behavior. This is more like the way we reward heroism on software projects.

    • I like it, you’re likely to get a dog that is even more confused :)

  • David

    But I think the bigger question that needs to be answered is: How can I (a dedicated developer) get this type of problem corrected?

    I have recently been pushed into a situation that is a death march project due to the fact that the team I am on is the best in the company. If we do manage to succeed, then yes, it will be great; but already we are being told to not fix the problems that are already there; just get the job done. Three weeks into the project, and I already feel dirty. I, for one, will not feel “great” if we get the required work completed to the client’s “satisfaction”.

    On the other hand, the project that we were working on, which has a long history of on-time and above required functionality shipping, we seldom see any thing past an internal good feeling.

    So how, can I make the company see the point – that what we were working on is the good job, not the debacle that we are attempting to save?

    • That is the million dollar question isn’t it. Unfortunately there is no simple an easy answer, at least I don’t know of any (if someone else does than I would love to hear it). We are talking about creating a mental shift within the whole organization.

      The first thing to do would be to get your teammates on board and make sure they share the same view as yourself. After that it is time to work on your immediate management. You would be surprised how much power a team can have to create change around them if they present a united front. Unfortunately, what actually ends up happening is that everyone is brave over drinks after work, but when push actually comes to shove it is really tough to express you disapproval in the face of your superiors. So only one person (the spokesman) ends up looking like a bad guy and things remain the same.

      I can go on and on about the various ways you can use to shift opinion and get people behind you, but none of them provide any guarantees. It is unfortunately not an exact science. I should really put my thoughts in a separate blog post :)

  • RyanM

    Amen brother! Software development has a lot of problems, but this is among the greatest imho. A primary reason I think that software engineering is rarely accepted into the family by other engineering disciplines.

    • Enzo

      Really? Which engineering disciplines? Those that can’t build a road right? Those that can’t make levees? Those can’t make a decent car? Those that can’t a fuckin’ coke machine take a dollar bill?

      • That is actually a really interesting point, most other engineering disciplines really do have their own share of problems. The house my parents live in, the driveway is on one side and the backyard is on the other, but you can very clearly tell that it was meant to be the other way around, someone had to do some quick thinking when they built it initially :).

  • With the great variability in project successes (and failures), and the various ways both are accomplished that, when a project goes according to expectations with no “heroics” needed, people may end up feeling that’s the way it ought to be most of the time. So they don’t feel doing what is expected should get special notice. Heroics have always been looked upon as something to be rewarded — all those “win one for the Gipper” sports-like phrases business is so fond of using.

    So while it is regrettable, it isn’t going to change until project success rates change (for the better) and maintain that rate reasonably consistently. Human culture, in general, rewards “getting things done,” often regardless of how. So until getting things done in an as expected manner becomes the norm, rather than exception, expectations and rewards aren’t likely to change much as a cultural phenomenon. Individuals can recognize the more reasonable success, of course, long before that and that will help change “the culture.” But I think a higher success rate using “sustainable” behaviors will be what makes any change most likely.

    Then, the overall satisfaction and tone of the culture will rise and the crisis behaviors, whether they lead to successes or not, will be viewed more critically and considered a less desirable way to work.

    • I agree with you but unfortunately it becomes a chicken and egg scenario. Should business start getting more realistic about software projects to improve success rates or should success rates improve before the business starts getting more realistic.

      We are now talking about fundamental, industry-wide change, and that is extremely tough.

  • This is excellent:

    I am so jaded with job adds “requiring” individuals (Project Managers, Business Analysts, Software Architects, Developers of all types, etc…) to have a “consistent record of success” in their projects (meaning: in time, within budget, high quality,…) when this is actually a rare occurence, especially when ALL these factors must apply.

    What this post points to is the “Elephant-in-the-room” of Software Engineering that is still largely ignored by the establishment (i.e. senior corporate management and anyone who needs or wants their support).

    A couple comments:

    About the “dog parable”, I would refine it as follows: Ok, so you ask for but ignore good behavior, BUT YOU GET VERY UPSET ABOUT BAD BEHAVIOR, and you provide the treat when the dog comes begging for mercy… I would actually LOVE to try this on a dog to observe the long-term effect of this technique, but I love animals (esp. dogs) too much to seriously consider that. That said, if you or anyone elser did try something like that, I am all ears…

    About the (more serious) state of affairs in Software Engineering, I think it would be a wonderful idea to follow up on this post by creating a support mini-organization to provide practical help to software engineering professionals faced with the above conundrum. the result could be could provide practical tips to achieve the following:
    – Since credit is only provided to project that come close to failure, how can a Project lead create that perception as quickly as possible
    – Since heroics are the easiest way to get rewarded, how can a project create that impression while working (on the average) 40hrs/week or less
    – ETC… This could be called Practical Insight (for) Success (in the) Software Engineering Debacle (a.k.a. PISSED)

    A bon entendeur, salut! (Excuse my French)

    • I’d never really do something like this to a poor animal :), Pavlov performed experiments similar to this and they were beyond cruel.

      Yeah create course – “perception management for IT professionals”. How to look like you’re working your a** off and still have a life :).

  • Ah yes, the Death March, a frighteningly common phenomenon. We are rewarded when we fail because we have compromised all the values we hold dear to follow the dictates of lousy management. Being a “good soldier”, loyal, obedient is what counts. It remains of great value to the corporate leadership.
    I like your posts, they resonate!

    • Thanks Rainer, I try to call it the way I see it :).

  • liviu

    But what if it isn’t developer’s fault. What if the project managers and top management promise the client obscene short terms just to hook them. Of course the developers must work extra hours because the project will never finish in time. And if they didn’t work extra their commitment will be questioned because the project “failed” to be on schedule. They have to do everything that is humanly possible to cover their a**es first..

    • The ass covering culture ultimately gets us nowhere, everyone is worse off if they engage in this kind of behavior. What developers need is to have the courage to stand up to these kinds of pressures in a united fashion. You need to remember that developers are not easily replaceable. A developer probably is (although even that is arguable), but not a whole team. Until that happens, management will continue to have unrealistic expectations and reward behavior that is ultimately detrimental to the industry at large.

  • Jack9

    “You need to remember that developers are not easily replaceable.” – Huh? I don’t see that happening anywhere but the lowest rungs of the industry where decent developers are a rare commodity. If that’s what “needs” to happen, then this is an idea that does not mesh with reality. In reality, developers only want and expect things to go as planned.

    Developers create systems. They are paid to think, estimate and implement, not to strike out randomly. There’s no need for positive/negative reinforcement as no third-party validation can compete with their innate desire to be right. It’s what feeds their ego and allows them to say “this is the way it should be done because I say so.”

    • I completely disagree, people are people, it doesn’t matter if they are developers or not. And people are complicated beings, being a developer does not make you an automaton with no feelings and no conception of what goes on in the outside world.

      A genuine word of praise or a show of appreciation really can go a long way. This is why so many developers work for inferior pay but don’t quit because the love the people they work with etc.

  • Pingback: Does Software Development Have A Culture Of Rewarding Failure Software Rss()

  • Colossal

    I’ve been in IT for 20 years, and I usually find articles about IT pretty off-the-mark. Yours hit a bullseye, however. Accurately describes what goes on in corporate IT every day.

  • Doug Nelson

    It is this way because this is the way the educational system is set up. Students who study on their own, don’t disrupt class, and answer all their test and homework questions are either ignored or are punished by giving them much extra tedious busy work to complete.

    Students who beat up other students, don’t complete assignments, and disrupt class receive guidance counseling, attention, ADHD pills that they can sell at a profit, and other rewards.

  • Dan

    Working many hours to achieve something, regardless of the efficiency, is praised by the baby boomers. Working with high efficiency and flexible lifestyle is praised by generation Y. It’s a generational misunderstanding.

    • That is actually a really interesting way of looking at it, although ultimately we pick up our work habits from the older generations so the gen Y’s eventually turn into copies of the baby boomers – you just watch, it will happen :).

  • Han Solo

    Not just software development…but I have noticed over the last 2 decades that using crappy IT solutions in general is rewarded.

    Just like the Windows guys are usually much more visible and popular in the eyes of management than the UNIX guys…

    why? Because they are always in front of management talking about how they solved this big problem or put out this big fire, while the Unix guys are never having to explain anything because their shit just works and works and works.

    Hence the windows guys are rewarded for choosing a crappy os, and crappy IT solutions, while the UNIX guys are punished for making a very reliable and stable environment.

  • This is not a software development problem. It is a problem of human nature, “squeaky wheel gets the grease,” is a proverb for a reason. Even further back, it is mentioned a few thousand years ago in the “Art of War.”

    “Hence to fight and conquer in all your battles is not supreme excellence; supreme excellence consists in breaking the enemy’s resistance without fighting.”

  • Sundar Viswam

    Well said! But isn’t there something more fundamental to it?

    1) IT people are paid by the hour. If a project is delayed, then everybody gets more hours and more ‘job security’ and as a lesson learnt, more hours for the next project! That applies right from the top of the food chain to the bottom. Everybody’s happy and so everybody pats everybody else on the back.

    Instead, deliver projects early or on time and people get only their fair share and might even get laid off because you’ve stupidly completed all the work there was. You’re literally snatching food from everybody’s dinner table. So why do you deserve praise?

    This is with all jobs that get paid by the hour, whether it is a plumber, electrician, mason et al. And the IT industry ‘professionals’ get the highest pay by the hour. Delay is a natural corollary and is directly proportionate to the rate of pay.

    2) The IT industry is perhaps the only one where managers don’t need to have worked in the business that the project is working on. In fact, they don’t even need to have worked in the IT industry. Will you allow a non-accountant to be chief accountants? No. Will you allow a non-doctor to lead a team of doctors and nurses? No. But an IT project manager need neither know software nor hardware nor the business. Everybody’s able to take him for a ride, confuse problems, complicate solutions and demand more hours, more time. Or else. As long as a manager in the IT industry or any industry for that matter is not capable of challenging why something takes so much time, no project will ever get done on time and therefore never on budget.

    3) In work as in life, there are people who will consume more that they produce. In the IT industry, they will consume more hours than they need and everybody else will suffer. In every phase of the SDLC, there are the ‘stretchers’ who can stretch any piece of work. The other phases will suffer the consquences. It’s not only the developer whose time get sqeezed; BAs have the same complaint, Testers complain, Designers complain. And all are as capable of over-consumption as the next.

    The honest, hardworking and efficient ones will keep complaining and suffering and failing and losing and wondering why, while the inept and the clever will continue to succeed and take it all to the bank. We can stand on our heads and devise all kinds of tools and techniques and methodologies, but any system is only as good as the people who run it.

    Another point too:

    The work done by plumbers, masons and electricians etc cannot be outsourced whatever the cost. We’ve had a recession in this country for a couple of years. But have you seen the cost of these services coming down? No, not yet, not until somebody can repair my water line and faucet from India.

    But most IT work can be outsourced and will continue to be outsourced. As long as we, individually, consume more than we produce, our work will keep moving to places where consumption equals production. It is moving to India and China now, but Indians in IT have already hiked up their own pay so much, now the Indian industry is outsourcing IT work to Sri Lanka and Bangladesh!

    Business is like water, it will find its lowest level.

    • Hi Sundar,

      You make several good points, but you can counter all of them. If you can complete work on time and on budget then the business is more competitive which means it can get more projects done which means it is ultimately more profitable as a direct result of well-run IT projects. If you use some of those profits to reward the people responsible they will be more likely to strive to succeed in future. It is a win/win for everyone. Not to mention the fact that many IT professionals are on a salary and don’t get paid by the hour.

    • James

      I think one’s perspective is colored by the type of work you have done the most.

      Software development is what my company does. It’s our core competency. We develop products.

      We don’t bill customers by the hour, we work on the next version, and hope it provides our customers with a reason to upgrade, and new customers to buy the software. Our survival depends on us providing these reasons.

      So working “dumber” as you mention, by taking longer to complete a project, doesn’t benefit us at all, in fact it hurts us, as our competition gains on us.

      That doesn’t change the fact that we have a well entrenched Death March celebrating culture here though :)

  • Pingback: pequt's me2DAY()

  • Pingback: Paul M. Jones » Blog Archive » Does Software Development Have A Culture Of Rewarding Failure()

  • I’ve seen this time and again. The manager at my previous job was an extreme example of this (and he’s one of the folks who gets asked to mentor people in “agile” methods). He rejected a simple, scalable and elegant design in favor of a far more complicated and user-hostile approach, one that required a LOT more work, and then topped it off by waiting until the last possible minute to put the highest risk parts of the project into the current task list. The result was a crunch, continually adding people to the team (which of course meant more overhead from ramping newbies, therefore further delays), and in the end a product that didn’t work. (It did in fact “work” as designed, but since it wasn’t designed to work the way that a sane user would operate, the users weren’t able to use it. AFAIK they still can’t, and it’s been nearly 4 months since it “launched.”)

  • Dan

    Hey Alan, there is actually a great book on exactly this topic: “A Hero Behind Every Tree”

    • Looks like an interesting read, I might check it out when I get a chance.

  • Rajesh Chairmakani

    Interesting read. you have touched upon a topic where most managers will not dare to talk about.

  • Pingback: Webs Developer » Learning A Software Development Lesson From A Children’s Poem()

  • Pingback: Did Your Boss Thank You For Coding Yourself to Death?()

  • Pingback: Did Your Boss Thank You For Coding Yourself to Death? | Binhs Blog()

  • NilaB

    I know a gal who creates no-drama, non-death-march projects, but as you stated, she’s been overlooked in all aspects of career growth, besides looking and sounding like a ritzy lady-golfer, no drama. Others with drama-laden messes get the kudos for dramatically dragging it out of the toilet, but everybody ignores that they let it get into the toilet in the first place. Maybe management gets a vicarious thrill, like having your own private alligator-wrestler, never mind whether that’s good for business. Management “says” they want to hire the very best, but too often they get swept-off-their-feet by thrilling and dramatic GI-Jo Action Figures with a good line of BS, and then they burn-&-churn their good little worker bees.