Before There Was Lean, Agile Or Waterfall There Was Theory X, Y And Z

xyzWhen we talk about Lean or Agile or even Waterfall we talk about software development processes, but software development is a relatively young industry so we have been able to piggyback upon the work done in other industries (such as manufacturing) to create processes and management frameworks. This is both a good thing and a bad thing, we did not have to start from scratch, but we have also inherited some stuff that clearly doesn’t fit and we now have to weed it out while retaining and innovating practices that work for our industry (we want to throw out the proverbial bath water, but want to keep the proverbial baby :)).

The whole idea of introducing processes is all about seeking efficiency this is just as true in software development as it is in any other industry, and leading/managing people is central to this. If we want to effectively prune the processes and practices that don’t fit while retaining ones that do, we need to be aware of what has worked (and didn’t work) previously, after all we don’t want to make the same mistakes that others have made before us. That is to say, it behooves us to be aware of some historical facts regarding where management and process theory comes from. I am a firm believer in learning from the successes and failures of others (not just from your own) and so it is to this end I’d like to share with you a little bit of process and scientific management history. Most of this largely predates software development, but you will undoubtedly see the seeds of the processes you use today in it, as well as seeing where certain management attitudes come from.

The First Foray Into Process – Frederick Taylor

Frederick W. Taylor is considered by many to be the “father of scientific management”. Taylor developed his scientific management theories in the late 1800s and early 1900s. Taylor studied, measured and documented the behavior of steel workers. He attempted to study the most efficient way to perform a task by breaking each task down into smaller component tasks (he called this process job fractionalization). He performed a great many studies (called time and motion studies) using a stop watch to find the “one right way” of doing particular tasks. The idea was that by successfully combining the most efficient elements the best production methods could be adopted.

Taylor developed four principles of scientific management:

  1. Replace rule-of-thumb work methods with methods based on a scientific study of the tasks
  2. Rather than letting employees gain experience themselves, scientifically select train and develop each employee
  3. Provide “Detailed instruction and supervision of each worker in the performance of that worker’s discrete task”
  4. Divide work nearly equally between managers and workers, so that the managers apply scientific management principles to planning the work and the workers actually perform the tasks.

In Taylor’s opinion it was a worker’s nature to slack off and furthermore a worker wasn’t really capable of understanding what they were doing in the first place. It was therefore the job of management to control and force the lazy workers to be productive and efficient by giving them clear and unambiguous direction (does that sound like some managers you might have known :)).

It is interesting to note that Taylor had several disciples who pushed his theories and were reasonably successful in getting them implemented by industry. One of the more famous of his disciples was Henry Gantt – yep you guessed it, the guy who invented the Gantt chart :).

A More Humane Approach – Elton Mayo And The Hawthorne Effect

Elton Mayo is believed to have pioneered the human relations movement of production management. Mayo believed that the emotional state of workers is just as important as finding the best combination of movements when it comes to achieving maximum productivity. Mayo believed that workers form social groups at work and therefore cannot be treated in isolation but must be seen as members of a group. He believed that financial incentives are less important to a worker than the need to belong to a group.

Mayo is best known for a series of experiments he conducted in 1927 at the Hawthorne Works of the Western Electrical Company.

The Hawthorne Experiment

During the experiment Mayo varied the intensity of light on the shop floor in order to find the optimum level of lighting that would result in maximum productivity. He found that regardless of the degree of light, worker productivity increased. Knowing that they were the subjects of a study made the workers change their behavior. This phenomenon became known as the Hawthorne Effect. Interviewing the workers after the experiment Mayo found that they performed better because they were treated better by their supervisors during the experiment. The workers were also more motivated as their tasks acquired greater meaning as part of an experiment.

Theory X, Y And Z

In 1960 Douglas McGregor, a Management professor at the MIT Sloan School of Management repackaged and renamed Taylor’s and Mayo’s theories, he called them respectively Theory X and Theory Y. At the time the majority of managers were proponents of Theory X, i.e. they took a pessimistic view of human behavior and believed that people were inherently lazy and needed to be pushed to achieve better productivity with rewards and punishments.

McGregor was believed to be a proponent of Theory Y, he thought that his work to repackage Taylor’s and Mayo’s theories would prompt the managers of that time to question the ideas that underpin both theories, to achieve greater understanding.

In the 1980s William Ouchi took Theory Y a step further. He studied the benevolent version of Theory Y used by Japanese management and called it Theory Z. At the time Theory Z was thought by many to be the secret of the Japanese competitive advantage. Using Theory Z the Japanese were able to bring together management and workers in cohesive work groups. Everyone was part of the decision making process, workers and management worked together in quality circles. Everyone was involved in kaizen – a continuous effort to improve all aspects of the company and of self. Of course we know that another thing that stems directly from this is the idea of Lean software development.

It is interesting to note that the ideas employed by Japanese manufacturing were not actually invented in Japan but instead stem from the work done by people such as Mayo and William Deming.

I hope this has given everyone a high level overview of where the idea of process comes from and what shaped (and continues to shape) the attitudes of the managers that we have today. Of course as software developers we are even more interested in process as applied to our own field (agile, lean, waterfall etc.). And since Lean seems to be the flavor of the month at the moment I will try to explore what I believe to be the origins of the Lean movement (not the software one, but the real one i.e. Toyota Way etc.) in a subsequent post. It is sometimes better to go back to the roots of a process (or movement) and built on top of the basic principles rather than trying to retrofit and existing process from one industry to another (e.g. manufacturing to software), but you will have to judge for yourself. Don’t forget to grab my RSS feed so you don’t miss out on that discussion.

Image by Beadmobile

  • John “Z-Bo” Zabroski

    You forgot Dr. W. Edwards Deming vs. Peter Drucker/Michael Porter. Porter’s value-chain stuff has obviously influenced “software product line” thinking (in a negative fashion, I might add).

    • http://www.skorks.com Alan Skorkin

      I am still planning to go into more detail regarding Deming and his influences and how this is/can be applicable to software. I will definitely also have a look at Drucker/Porter how that you mention it, thanks.

  • Isaac Gouy

    > Hawthorne Effect

    “Was there Really a Hawthorne Effect at the Hawthorne Plant?”
    http://www.nber.org/papers/w15016.pdf

    “We find that existing descriptions of supposedly remarkable data patterns prove to be entirely fictional. “

    • http://www.skorks.com Alan Skorkin

      And yet they also say that there are still more subtle manifestations of the Hawthorne effect in the data.

      Regardless from personal experience I have found that the important findings (the ones that related to people and how to treat them etc.) of the Hawthorne experiments are definitely true.

  • Justin Hunter

    Alan,

    Very good post. I found it particularly interesting since I just wrote a similar blog post myself, mentioning Deming, Lean, Agile, etc.

    It’s interesting how two people can address the same general theme in different ways. My blog post also examined a key element of the pre-cursers to Lean but highlighted different aspects of what is “being left out of the discussion” today that remain relevant to Software Design and Software Testing.

    Please see: What Else Can Software Development and Testing Learn from Manufacturing?

    http://hexawise.wordpress.com/2009/08/25/what-else-can-software-development-and-testing-learn-from-manufacturing-dont-forget-design-of-experiments/

    Thanks.

    Justin Hunter
    Founder of Hexawise http://www.hexawise.com

    - Justin

    • http://www.skorks.com Alan Skorkin

      Your post is a great post as well, I am still planning to go into more detail about Deming’s work and my thoughts on it, so your post is very pertinent :).

  • Pingback: Before There Was Lean, Agile Or Waterfall There Was Theory X, Y And Z [ SKORKS ]

  • Pingback: Lean Six Sigma Kaizen. Produzione snella e quality manufacturing.