Developers under thirty may not know the name, Tom DeMarco, but if you ever drew a paycheck from a large organization, you’ve felt his influence. When your boss said “You can’t control what you can’t measure”, he was channeling Mr. DeMarco.
I can think of no single individual who has had a more pervasive and decisive say in how we manage software development. Corporate CIOs, IT directors, and senior architects have listened to him prescribe “best practices” for more than 30 years. So when he says “I was wrong all along” … it’s like hearing Robert McNamara confessing his tragic mistakes. You feel you knew it all along … but it hits hard to hear him say it.
In “Software Engineering: An Idea Whose Time Has Come and Gone?”, Mr. DeMarco confronts his life’s legacy -- his insistence that precise planning and intense monitoring are essential to project success -- and condemns it.
Let’s not go overboard. He isn’t against measurement (and neither am I). If you can measure it … and your measure bears a well-understood relationship to the good or evil you want to assess … and the cost of measurement is reasonable … then measure it.
But putting control and measurement first utterly obscures what really matters: the potential value of the project and the forces that can determine the timing and success of the project. Indeed, planning and measurement can … and often do … actively impede success.
Re-blogging is an etiquette violation. I’m doing it anyway because I fear readers unfamiliar with Mr. DeMarco will miss the many gems in his short and sweet mea culpa. Some juicy quotes to entice you.
“Strict control is something that matters a lot on relatively useless projects and much less on useful projects”
“The more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.”
“First we need to select projects where precise control won’t matter so much. Then we need to reduce our expectations for exactly how much we’re going to be able to control them, no matter how assiduously we apply ourselves to control.”
“Consistency and predictability are still desirable, but they haven’t ever been the most important things. The more important goal is transformation.”
“You say to your team leads, for example, “I have a finish date in mind, and I’m not even going to share it with you. When I come in one day and tell you the project will end in one week, you have to be ready to package up and deliver what you’ve got as the final product.””
Before we get all smug about agile and the disaster that is “Waterfall Design” … I want you to think a solid few minutes about whether you are susceptible to putting engineering practices in front of something more important.
His critique seems to target only an overweening attention to planning and control. That just happens to be his personal road to perdition. He aims at a wider target.
Look at the title again. He asks is “Software Engineering” an idea whose time has gone? If you find that TDD or DDD or BDD … are what you talk about first, are you making the same category of mistake? How can these practices – whatever their merits - be more important than the value of the project and whether it is delivering on its promise? If, for example, you believe that there isn’t great software without unit tests … have you fallen into the same trap?
Imagine the courage it takes to come to such conclusions about your life’s work? I salute Mr. DeMarco. And I hasten to add that his writing was always more nuanced and more useful than his own harsh self-criticism suggests. He remains well worth reading … more so in the light he shines from this vantage point.
Aside: I learned about this article from a Kent Beck tweet. You’ll find the endlessly fascinating Kent Beck blog here.
11 comments:
This reminded me of a book I read a while back, "Priceless: On Knowing the Price of Everything and the Value of Nothing "; one of the points is that economists spend so much time trying to "price" things that they miss the bigger picture. I guess you could paraphrase this into "software engineering: on controlling everything and delivering nothing"...
It's also another illustration of the fascinating process by which a smart idea is expressed by a smart individual, but as the idea gains wider adoption it also gets simplified to the absurd, embraced by zealots with no taste for nuance!
That being said, I have to tip my hat to someone going against "software engineering" as a whole. In the realm of provocative statements, this is top material, and great food for thought!
Saying software engineering's time is gone is an overstatement. A maturing movement will always pass from the "X is always good" stage to "properly done X is good" stage.
Hopefully De Marco's article will help to dispell the zeolotry that Mathias mentioned in his comment.
> If you find that TDD or DDD or BDD
> … are what you talk about first,
> are you making the same category of
> mistake?
Yes... if you're talking about them as local optima, which, unfortunately, many programmers often do.
"Saying software engineering's time is gone is an overstatement."
Is it? I don't think so.
Look at it this way: Software development is a new discipline, relatively speaking. It doesn't have the time under its belt that other disciplines have. It doesn't have the legacy of ship building, or bridge engineering, or painting, or music writing.
Some disciplines are clearly more science and engineering than craft or art. Software development? No one knew how to classify it or manage it or anything when it came on the scene. So they thought it was more of an engineering discipline. Maybe they were wrong? I've always *felt* like it was more of a craft than a science... And that's the way I read Mr. DeMarco's paper.
@ Mathias - good reference.
@Scott .. right. And I think you make this point at length in your post The Myth of Developer Productivity".
If I recall, the essence of your argument is that something truly improves productivity if it improves the entire production chain in a sustainable fashion.
Mr. DeMarco's more far reaching point is that we have to keep the value of the project in mind at all times. That "value" is constantly on the move as what had priority 'x' today has "u" or "z" tomorrow. Hence his advice, "be ready to ship it any time because we may say 'enough' at any time."
This would be cliche without his twist ... his challenge: your project has value in inverse proportion to your concern for proper s/w engineering. Ouch!
@ctford and @chris ... I think you are in violent agreement. To say "substance trumps process" still leaves room for process to matter.
Ward,
> his challenge: your project has
> value in inverse proportion to your
> concern for proper s/w engineering.
In the context of the article that we're talking about, and in context of Tom DeMarco's work, software engineering has a specific meaning, and that meaning arguably has nothing to say directly about TDD or BDD for example.
This might be dumb, but I don't understand the following statement made by Demarco in the article:
"What’s immediately apparent is that control is re-
ally important for Project A but almost not at all
important for Project B. This leads us to the odd
conclusion that strict control is something that
matters a lot on relatively useless projects and
much less on useful projects."
Why would A need more control than B? What did I miss!
@Scott - no doubt. That was exactly what he meant. And that alone makes it fascinating.
But why squander the opportunity to hold the same mirror up to ourselves? What is this century's cant?
I'm convinced we really agree on this one. I am not knocking TDD or BDD one bit. I am knocking comments I have heard (not from you) such as "if there aren't tests, I won't even look at it." Or the project is doomed because it lacks such-and-such.
Maybe it would be better with such-and-such. Maybe lack of such-and-such makes it a poor example. But is it good stuff? Would it have seen the light of day or would it have been crippled had we insisted that those inspired to write it follow our practices? There are some amazing Excell, Access, and FoxPro apps out there. They live on because they do good service.
I can't bring myself to recommend these technologies. But I must confess: too many of my "beautifully crafted" apps have slipped beneath the dark waters of irrelevance.
I still wish I had invented the pet rock.
Cheers, W
Tobin,
Project A has a much smaller return on investment. Think of ROI as the project's profit (it's not organizational profit per se, but I'm representing it this way to create a temporary, synthetic context learning context).
If the project's profit margin is so small, then the profit can be disrupted by smaller problems than if the profit margin were large.
Because project A's profit is a narrow sliver, Project A is arguably well-served by rigorous controls that see that little-to-no amount of disruption enters the context.
Those stricter controls are in place on project A to keep it from even minimal deviations because it has minimal return.
Implicit in all of this is the recognition of the force of disruptions. Some kinds of disruptions might have minimal deviation, some might have maximal. Some disruptions are very small, but happen repeatedly or constantly so that they are incredible risk.
Ultimately, we're talking about statistical process control here, which means measurement, metrics, and their use to manage work to a goal. Statistical process control isn't a substitute for ability, it's a support mechanism for learning about certain kinds of disruption and for shaping improvements.
Ward,
Well, I suppose you can hear it from me then: If there aren't tests, I won't even look at it.
But I'll go further: If there aren't the right kinds of tests, I won't even look at it.
This is a personal choice that comes from experiences of working on with high-performance projects. It's simply far too heart-breaking to drop into a project that hasn't understood the purpose of testing and how testing is done effectively. And further; a project whose management doesn't understand productivity beyond mere tool-driven efficiencies.
Frankly, I haven't often heard many people in my own professional circle lay out a reasonable business case for TDD or BDD that would even convince me to use these things today, and I've been using TDD since 2000!
It's really important that we understand the organizational and cultural effects and the organizational and cultural pre-conditions for these kinds of approaches. They're not tools. They're ways. They don't fit into every context, but I have yet to see a context that when steered toward the organizational and cultural qualities where TDD and BDD make sense that hasn't benefited.
This is why I'm making a differentiation around local optima. Without broad experience in the ways as well as experience with the organizational and cultural aspects, then the perspective is bound to be distorted, and likely suffering from typical programmer distraction-by-tool-stimulation syndrome.
I prefer to work on projects that are willing to embrace much higher performance than what can be delivered by mere tools, but I'm willing to use any tool that supports the organization as it changes its culture.
I used to build systems with FoxPro. I used a primitive, ad hoc form of TDD. Those systems still work and are still in operation. They don't need to be changed, and have had few calls to be changed over the last 15 years. When they are changed, the productivity costs to the business borders on negligent. If I were to retrofit tests to those systems without changing their design, I might make the productivity problem even worse.
Fit into a framework of organizational learning where a learning culture is leveraged methodically to raise expectations for productivity to a level above and beyond that which can be achieved by the mere tools that supported the journey, those tools are foundational.
In the right context, TDD and BDD are foundational. I personally don't want to work anymore in the contexts where expectations for business productivity are lower than what can be achieved in a culture that understands TDD and BDD (for example) beyond the mere mechanics. It's like watching your best friend throw his life away on a video game addiction.
There's no doubt that abruptly forcing new paradigms onto software development efforts that are facing the cumulative constraints of decades of primitive, sub-optimal productivity is no way to get some necessary software to see the light of day. But that's no reason to not begin to change culture and organization and tools and ways and means so that these conditions aren't perpetuated to the detriment of the businesses and the people they serve.
The barriers to understanding things like TDD and BDD are rarely intellectual. They're typically organizational, and typically within organizations that continue to reinforce primitive ideas about management and culture.
So, I don't want to look at something if it doesn't have tests, and the right kind of tests that evidence the kinds of culture and organization that won't continue to break my heart. Call it a personal weakness, but I can't erase my memory and experience in cultural and organizational change as easily as I can erase a file.
neverindoubtnet.blogspot.com; You saved my day again.
Post a Comment