Monday, January 28, 2008

On the effectiveness of TDD

There's a fascinating exchange between Phil Haack and Jacob Proffit on the implications of a National Research Council of Canada paper titled "The Effectiveness of Test-first Approach to Programming".

The experimenters divided 24 third-year CS students into two groups, one practicing Test-First development and one practicing Test-Last development. Each implemented the same functionality. The Test-First group always wrote unit tests before writing each feature. The Test-Last group wrote unit tests after writing all of the features. It wasn't a long running experiment so I would be tempted to describe the "Test-Last" group as the "Test-Soon" group; but that's quibbling. The researchers arrive at conclusions favorable to Test-First ... which you should gather from reading the report (and the commentary) on your own.

Phil discusses this study in his post "Research Supports the Effectiveness of TDD". While he doesn't say that the study actually proves that TDD is effective, he clearly thinks highly of it.

Jacob responds in a post on his own blog, "TDD Proven Effective. Or is it?", with a devastating (IMHO) critique of the study and obliquely criticizes Phil for succumbing to "Confirmation Bias".

The essence of Jacob's argument (if I may) is that (a) the study data do not confirm the thesis that Test-First is more "effective" than Test-Soon, (b) there are disturbing data suggesting that the Test-Soon control group produced better code, (c) there is substantial cost to changing one's programming practice to Test-First and (d) he is reluctant to make such a switch until there is decent evidence for it.

I think the study, Phil's post, and Jacob's critique make superb reading so you shouldn't rely on my commentary for anything other than inspiration to check it out for yourself.

I think Jacob has the best of it here (and it appears that Phil eventually accepts Jacob's critique while keeping faith with TDD). I especially appreciate the manner in which the two of them move the debate along. Their exchange is full of passion and intelligence but never strays from civility.

In the end they agree (without evidence) that, whether you prefer "Test-First" or "Test-Soon", the outcome is substantially better than "Test-Never".

Sadly, the NCR experimenters didn’t include a "Test-Never" group but I think there may be good evidence to support the notion that some testing (whether "first" or "soon") beats no testing. I intend to read one of the referenced papers on this subject: "A Longitudinal Study of the Use of a TestDriven Development Practice in Industry"

Don't look to me for conclusions. I am generally persuaded that we should have more social research before we start telling everyone what they have to do. That said, I am persuaded of the merits of testing and have a good feeling about TDD ... when I take time to practice it.

Tuesday, January 22, 2008

Prism Camp: Reflections on the "Composite WPF Review"

I'm just back from Redmond where a group of us devoted a well-spent week to opinionating on the future of a framework for composite applications.

That would be the "Composite WPF" framework (code named "Prism") that Patterns and Practices will build to fill the gap between the Composite UI Application Block (CAB) of 2005 and whatever someday blooms among the bleached bones of Acropolis.

What is Prism?

"Prism" is the code name for forthcoming guidance and code aimed at builders of composite applications with WPF client UIs.

It's tempting to call it a framework because Prism will ship with many collaborating components that collectively constitute a foundation upon which to build a WPF application.

Like any framework, it will promote a certain way of doing things by making some paths easy and other paths difficult. A framework makes choices so you don't have to. That is its virtue and its curse.

Prism aspires to be the better CAB. CAB is both a reference point and the point of departure. If you know what CAB is, even approximately, then you understand the perspective and proclivities of the Prism gang.

Acropolis was supposed to be the better CAB. Now that that project is gone - its team dispersed - Prism holds the "better CAB" baton. The aspirations and budget are far less grand; for many the reduced expectations offer a better promise of success.

Prism would address the same concerns as CAB and hope to do no worse. Yet, at least within this group, we believe we can do better than CAB and that, in order to do better, Prism must start over, unencumbered by CAB code or APIs.

What is "better"? The PnP people had a few months to kick that around. Then they called all of us in to kick it around some more.

The Campers

We were a good mix of experiences and prejudices.

The Patterns and Practices side was well represented of course. The indomitable Glenn Block, product planner for Prism, presided as camp director. Blaine Wastell, the "Client UX" program manager, maintained a sharp focus on gathering our priorities. Frances Cheung, the dev lead , helped keep the proceedings in line and Ezequiel Jadib of Southworks captured the artifacts. Later in the week I was reintroduced to Bob Brumfield in his newly appointed role as chief architect.

Other Microsofties made cameo appearances throughout the week. Some came from related wings of Patterns and Practices (e.g., Peter Provost, Shaun Hayes, Ade Miller, Chris Tavares); others dropped in from outside teams (e.g., David Hill, Brad Abrams, Rob Relyia, Jaime Rodriguez ). It was comforting to have their interest and support.

The main body of us were a mix of consultants, corporate developers, and third party product managers who have been watching, teaching, or building with CAB for a considerable time.

[Please accept my apologies for omissions, misspellings, and missed assignments. Out of respect for the companies and individuals involved, I'm comfortable identifying only the few who I am certain would agree to be named. I'm striving to convey my sense of the room rather than take accurate inventory. ]

A Great Event

I thought this was a wonderful mini-conference, the kind of event Microsoft should be proud of. Let me try to explain why.

But before I do, a few word to those who did not attend. You were missed … and you didn't really miss out. There was nothing finally decided. It was simply a gathering of reasonable size trying to advance the cause. Think of it as a focus group. No one says "man, how could they not invite me to that focus group." Your input will be needed over the coming year and you should contribute your ideas and enthusiasm. PnP needs you. Trust me.

The Right Timing

It's early days for Prism. PnP has had enough time to set goals and give it some shape. But PnP is not so far along that they are set in their ways. They came to listen while there is still time to listen … while it is still possible to say "that's a terrible idea … a complete waste of time" without wounding the poor sod who gave months of his soul to it.

The Right People

Most of us are experienced CAB developers. Many have placed big bets on CAB. We have a stake. We know what we like and what we dislike. And we all feel passionately about having a good foundation for building applications by composition.

That doesn't make us right. It just means that we have common ground … the appropriate experience and mindset for this phase of Prism evolution. It was fantastic comparing notes with people who knew what they were talking about and could speak with conviction about what is right for their employers, their customers, and their colleagues.

I hasten to add that these were not CAB bigots. Of course most of us are unconscious prisoners of the CAB-way to some degree. But everything is on the table. We lobbied for capabilities, not implementations.

We were fortunate to be joined by Jeremy Miller. Jeremy is a long time critic of CAB and while his ignorance of CAB is profound, so are his ideas for alternative approaches to application development. Jeremy is not some kid waving a little red book of Agile practices; he's grounded in real world experience and he listens. He earned the respect of everyone in the room and I am hopeful that he'll keep listening, criticizing, and contributing as Prism evolves.

We kind of blew it by not probing David Hill for more on Acropolis. As I understand it, he drove the original CAB effort and its Acropolis successor. Going through this composite framework development process not once but twice - and with WPF no less - makes him an invaluable source for do's and don'ts. David offered several insightful comments throughout the proceedings. But in his one shot at the podium he had barely enough time to present his 30,000 foot perspective before he was wisked off stage in favor of the next topic. No one's fault really. Still, he has a host of experience and we should find a way to tap it over the coming months.

The Right Focus

I think we covered what could reasonably be covered at this point. I'll probably be drilling deeper into this in some of my next posts but in brief we covered

What PnP thinks are the essential patterns of a composite UI.

We got to agree and disagree. Too much to say in this post; will follow up soon.

Why we (or our customers) are moving to WPF and what we will do with it

PnP has been flogging the notion that we have some compulsion to build "Differentiated UIs" (read: introducing novel effects into business applications). I'm sure there's a post or two coming on this one. My sense is that differentiated UIs are low on the wish list for most of us outside PnP. There were more than a few sneers at pointless eye candy. So it surprised me when we were able to dream up a few WPF effects that could, in David Platt's memorable phrase, "help the user shovel dung faster." There's more to this differentiated UI angle than I cynically thought.

What we value and de-value in CAB

There was vigorous discussion as expected; there probably should have been more.

I have the strong impression that PnP wants to burn it all back while many of us are saying "not so fast". Some of the bigger customers with substantial investments in CAB are properly insistent that there be some road from CAB to Prism. It needn't be automated. But we can't just ignore the CAB goodness either.

In this regard, there was talk of a CAB emulator, written on top of Prism, that could help bridge the gap. It's doubtful that PnP would build it but Ohad Israeli and I are keen on the idea and, at minimum, it's an essential thought experiment for Prism developers: "we can get rid of CAB feature 'X' because, if you really need it, you can emulate it in Prism by doing 'Y'".

There are a few of us who worry about the too hasty demise of the WorkItem. It's evident that PnP would like to kill it and Jeremy has been beating the poor animal savagely. But at least a few of us think the WorkItem has great value and is simply misunderstood. Actually, I'll bet it's more than a few of us. Look for my defense of the WorkItem coming soon.

But I have to concede that it's a darn good thing to be pressed so hard to defend it. If it survives in some form, it will be because it deserves to live and has been adequately justified.

The Prism Quality Attributes

We talked at length about what qualities Prism must exhibit. These became known as the "ilities" as in "flexibility, extensibility, scalability, debugability, testability, subsetability (sic), learnability (sic) …" Of course we considered many important qualities that don't fit the form (simplicity, performance, etc.).

We stack ranked them. I don't remember how it came out but I think extensibility or subsetability came first and learnability came second. I find these preferences fascinating. There was clearly a high premium on the notion that even core features should be removable and replaceable. Note also that learnability trumped simplicity by a wide margin. Evidently people will put up with something that is not simple as long as it can be justified, understood, and explained (and replaced if it doesn't seem useful).

Just because a quality didn't score well didn't mean we don't care about it. There were vigorous discussions about the ability to test and debug Prism applications. Debugging WPF is still a dark art and the room was hungry for tools and guidance. Those are not PnP's responsibility; but we look to them anyway.

I must observe that the audience love for testing was tepid at best. This had to drive Jeremy nuts - as it should. The honest truth is that almost none of us are close to following agile practices. I'd bet we'd all be staring at our shoes if he asked us point blank "do you unit test your MVP triads?" "what's your code coverage percentage?" or "are you using automated builds?" This is our reality folks. Jeremy can argue forever that sound practices both simply the code and remove the need for most of the crutches we cling to. It just isn't happening.

[Note to Chris Holmes and Kent Boogaart: yes, I know you guys are the worthy exceptions.]

I think Prism can show some leadership here by making Prism apps easy to test and providing real guidance on testing. There's a good reason why virtually no one tests their CAB UIs: it is a total bitch to do it and there are few (if any) examples.

Delivering Prism

Schedule and deliverables were two elephants in the room. I'm hoping for a complete package at the end of 2008. Anything later implies overreaching. In other words, if Prism can't be delivered by end of 2008 than its too big and too late.

We all heard something about four months to an alpha. Now that's encouraging.

What's in that alpha? Don't know. It's too early for that. We should expect some clues soon.

We do have a feel for what's out of scope.

Prism targets WPF and there will be no special effort to support alternative client platforms. The Acropolis team already broke their pick on the dream of a single framework for Web, Windows Forms, Mobile and WPF.

On the other hand, I expect PnP to be diligent in relegating WPF references to separate assemblies and to avoid patently unnecessary reliance on WPF constructs. I begged for sensitivity to Silverlight whose XAML-based presentation layer adopts the WPF paradigm, if not its code base.

Prism will be friendly to the graphically gifted designers who can wield Expression and XAML to gorgeous effect. But Prism will not offer a drag-and-drop developer experience in the manner of Acropolis.

Prism will be a coder's framework and it will rely on patterns and practices which, while proven, are not as widely known as they should be. DependencyInjection, for example, is a given. Remember, we stressed the importance of "learnability" … and that implies a willingness to learn.

We settled on a variant of the Woodgrove sample application ( as the reference implementation (aka, "RI") that demonstrates the core capabilities. Yeah, the eye candy in the center is a bit much but, supplement it with an editable grid and some pop-up data entry and the there's proof enough.

PnP stressed that the primary objective of the first RI is proof of Prism capability. There is concern (borne of sad experience) that some folks will treat it as a framework in its own right and try to shoe horn their own applications into it. We may expect the RI to demonstrate good practices and offer early recommendations. But to ask for more is to ask for too much too soon. Caveat Scriptor!

What's Next?

A few of us were able to hang out for a few days to help identify the core capabilities and commitments. We got off to a pretty good start, having identified a manageable list of roughly twelve key concepts. The PnP team should be fleshing these out and prioritizing them in the next week or so. Then they'll probably spike on them and blog about them.

PnP showed a couple of their spikes at the meeting and promised to publish them soon. I expect we'll see an active cycle of spikes and blog posts at first. Then I'm sure they'll come up with a better way to get this kind of material out there.

What You Can Do

There is something very important that you can do … and that I intend to do: send the team the use cases that you believe really matter. We shouldn't leave them to develop Prism in a vacuum. We have to tell them what we would like to see.

Here's an example use case.

I can add a view to a region without holding a direct reference to that region.

Example: I want to add my view to the "WorkingPane" region in the Shell. I don't hold a reference to the Shell and I don't know exactly what kind of thing the "WorkingPane" is. I write


You CAB fans would say "a SmartPart can be added to a Workspace by knowing the name of the Workspace".

The syntax is wholly fictional and besides the point. I don't know if there is a RegionManager class and I don't care. I'm just trying to be understood.

There is no fancy form to fill out. It's just a matter of expressing your thoughts as simply and clearly as you can. If you can't say it in a sentence or two, break it up into smaller cases.

It's probably smart to supply some brief justification too.

Building by composition implies the ability to stuff a view into some container control in another layout view that I know nothing about. The layout could have been written by someone else and delivered to the application in a module unknown to me. All that matters is that I believe my view should be put in that container control, whatever it happens to be.

That's the kind of statement that supports many cases so economize accordingly.

Wrap Up

I think we should commend Microsoft - PnP in particular - for reaching out. Like all of us, they have a budget and a deadline. They spent a chunk of money and time to do what we've been begging them to do: talk to us. They could have thrown a PowerPoint and a one hour LiveMeeting our way. Instead they hosted a three day event. Let's show them they did the right thing by rewarding their openness with our constructive feedback.