Monday, April 9, 2012

DevForce and Second Level Caching

We are asked occasionally whether DevForce supports second level caching, that is, does DevForce have some means on the server of remembering previously queried entities between client requests.

This is rarely a real problem in a DevForce application because (a) DevForce applications are usually rich client applications and (b) the DevForce EntityManager cache typically delivers the performance benefits folks seek from a server side cache.

My response, although grounded in long experience, is not always persuasive. Second level caching should improve scalability in theory and theory often trumps reality.

In this post I discuss how to tell if you would benefit from server-side caching and how you might be able to use an Entity Framework second level cache to achieve it.

I haven't tried to install an Entity Framework second level cache. In this post I provide links to information about how to implement second level caching. The links come from reliable sources and it looks like this kind of caching should work. If you try it, please let me know how it goes. I haven’t tried it myself because I find that, for almost all of our customers, this is a solution looking for a problem. But if it makes sense for your application and you give it a try, I'm counting on you to get back to me with your results… and maybe contribute some guidance and code to help others.

Perceived Problem

You have a great many users who repeatedly query for the same entities. Those entities hardly ever change but for some reason they keep asking for them and for some reason you can’t cache them on the client.

You’ve measured and these queries account for a significant percentage of database hits. Moreover, they’re really bogging the database down. You’ve determined conclusively, after careful study of production traffic, that these repetitive database queries are choking your database. You’re pretty sure that server-side caching would provide significant relief.

Are you sure?

Honestly, I don’t think this happens often … which is why you should have the measurements that prove poor performance is traceable to this cause. Don’t guess that this is the problem. Don’t forecast that it is going to be a problem. You need proof.

The interest in second level caches arises most often among people who are evaluating DevForce and haven’t yet built an application with it. Such inquiries are typically speculative. Trust me, you can waste a lot of time investigating something that isn’t going to make any real difference in your application. It might make matters worse.

But suppose you’ve demonstrated that this is a real problem in your working application. You’ve established that the client app can’t cache these entities locally for some reason (perhaps it’s a web client) … which may be why you’re looking into caching on the server.

Maybe it really is time to consider EF “Second Level Caching

You could try caching query results in a Query Interceptor. But that will require code you must write and if an EntityServer (aka, BOS) is involved you’ll have to make it thread safe. Consider EF “Second Level Caching” before rolling your own.

If there’s a 2nd, there must be a 1st

Time for some definitions. The “first level cache” is the local cache of entities retrieved by some persistence manager. The DevForce EntityManager is a first level cache on the client. EF’s ObjectContext is a first level cache on the server.

When writing with EF Code First, you create a DbContext which is wrapper around an ObjectContext. You can think of your DbContext as a first level cache if you wish.

On the EntityServer (aka, the BOS) DevForce creates a new ObjectContext for each client request.

This EntityServer is in-process in a 2-tier deployment.

These first level caches do a great job of holding frequently requested entities. But they (and their entities) disappear when the EntityManager or ObjectContext disappear. The EntityManager on the client can live a long time, the life of the user session perhaps. The ObjectContext, on the other hand, evaporates after each client request.

If you had a “second level cache” it would sit outside of the EF ObjectContext. It would outlive the ObjectContext and would be shared by multiple instances of ObjectContexts. When your query reaches a new, empty ObjectContext, EF makes a request to the database. A second level cache could intercept that database request and satisfy it with previously retrieved results.

That sounds like the perfect resolution to your problem. If you had a second level cache, it could hold query results for the entities that clients are clamoring for … and the database pressure would be reduced.

Unfortunately EF doesn’t have an out-of-the-box second level cache.

From time to time you’ll hear someone argue that NHibernate is better than EF because NHibernate does have a second level cache. Often the person making this argument (a) is unaware of the limitations of a second level cache, (b) doesn’t realize that DevForce client-side caching usually eliminates the need for a second level cache and (c) has no evidence that the application would benefit from a second level cache. There’s the whiff of FUD in the air.

Let’s deodorize. As it happens, there is an open source second level cache for EF!

In brief, it’s a plug-in that intercepts EF requests to the EF “Store Provider” (the component that turns EF store queries into SQL queries on a database, be it SQL Server, Oracle, or something else).

clip_image001

The second level cache checks if it’s holding results for that query; if so, it returns them from its cache, short circuiting the call to the database; if not, the query passes through to the Store Provider. Then the second level cache intercepts and caches the returned results for next time. I’m simplifying of course; you’ll want to dig into the resources mentioned below for full details.

Notice that the component includes a tracing interceptor as well.

You don’t have to design your application for second level caching up front. You can add the second level cache component later … when you know you need it. It’s presence (or absence) is largely transparent to DevForce and EF. You’re just “wrapping” the Store Provider in this caching component; it looks like a normal Store Provider to EF.

Learn about Second Level Caching in EF

If this approach sounds like it would help, you can learn more about it from these sources:

Second-Level Caching in the Entity Framework and Windows Azure (Julie Lerman, Sept 2011)

EF Caching with Jarek Kowalski's Provider (EF Team, Sept 2010)

Using tracing and caching provider wrappers with Code First

Thursday, March 22, 2012

Squash Entity Framework startup time with pre-compiled views

In brief

Your application can stall for several minutes while Entity Framework gathers the information it needs to perform queries and saves, a lengthy process it performs twice: before the first query and before the first save. Those minutes pile up, wasting developer time and angering your customers. You can drastically reduce these delays by pre-compiling the Entity Framework’s “views” of your model … as I explain in this post and demonstrate in its 14 minute accompanying video.



Costly EF startup

If you’ve used Entity Framework for a line-of-business application model, you’ve suffered a lengthy delay before the first query completes and a similar delay before the first save completes. Subsequent queries and saves finish are much quicker, completing in an amount of time commensurate with the request.

The delay is a non-linear function of the number of entities in the model. It often feels exponential. You probably won’t notice it in a toy model (every demo you’ll ever see) because the delay is lost in the wash of everything else that you’re thinking and learning about. But when the model grows to normal size – 100 or more entities – the delay mushrooms to a minute, two minutes or more. And you suffer this delay every time run the application … which you do all day, every day during development. Multiply that by the number of developers on the project and you’re wasting a lot of time … and money.

The cost is far worse than the time lost. Make a developer wait two or three minutes per iteration and she’s bound to forget why she ran the app in the first place. Two minutes is a long time. The mind wanders. The mind turns to email, Twitter, and Facebook. Productivity is shot.

Now I don’t think you should be going near a database during normal development iterations. I recommend that you toggle the app to run against an in-memory representation of your data layer such as the DevForce “Fake Backing Store”. But maybe you’ll disregard my suggestion. And everyone has to hit the database occasionally just to confirm that the app works end-to-end.

So the development cost is terrible no matter what you do … unless your developers’ time is free; perhaps you price them at zero dollars and you’re response to every productivity decline is to hire more developers. Your second instinct is to outsource.

What about your customers and internal end users? If the app runs 2-tier, they suffer the delay every time they launch the app. Does their time matter to you? I’ll bet someone will make sure it matters to you.

You won’t field customer complaints if your application runs n-tier (e.g., in a Silverlight application) because the Entity Framework runs on the server. The startup penalty is paid only by the first user to query and save. If you run n-tier and you don’t care about developer productivity, turn the page and move along.

Pre-compiled Views to the rescue

I’ve been wondering what to do about this for a long time. I’d heard that “Entity Framework Pre-compiled Views” might help. I also had heard that it was troublesome and might not work. It seemed like one more thing to get around to someday.

Then one of our professional services customers called and complained. His project had started fine but hit the wall at around 200 entity types. The first query and first save each took about 50 seconds on most machines. Team productivity had sunk, morale was sinking, and he was catching serious political flak internally. Our own staff confirmed that the problem was real. Since we (IdeaBlade) had recommended EF Code First, we had to do something.

My colleague, Steven Schmitt, did the leg work that proved EF pre-compiled views (a) work for Code First models, (b) were easy to create, and (c) improved performance dramatically: the 50 second first query dropped to seven seconds; the 50 second first save dropped to less than one second.

He deserves the credit … I’m taking the glory by blogging about it.

The accompanying 14 minute video shows EF’s slow launch times for a 200+ entity model, demonstrates how to create pre-compiled Views, and explains a bit about how they work.

I produced the video to spare you a parade of screen shots. I think it also conveys the seriousness of the problem and the practical benefit of pre-compiled Views more effectively than I can in spare prose.

The EF view generation tool does not work with EF 4.3 yet. Microsoft sources report that an update is in the works.

For those of you who want just the facts, here they are:
  1. Ensure that SQL Server Express is installed. You can get around it with a DefaultConnectionFactory but it’s such a pain. Save your energy for better things and just install the thing.

  2. In Visual Studio 2010, open the Extension Manager (Tools | Extension Manager).

  3. Search for “Entity Framework Power Tools”. The version as I write is “Entity Framework Power Tools CTP1 0.5.0.0”.

  4. [optional] Review the online information about it. These tools do more than pre-compile EF views.

  5. Locate your custom DbContext class in Solution Explorer [note: we’re describing how to pre-compile views for an EF Code First model. You follow a similar approach for an EDMX-based model although I haven’t tried it personally.]

  6. Make sure that your DbContext class has a public parameterless constructor … or the tool will fail in a mysterious way.

  7. Select your DbContext class, right-click, and select “Entity Framework”

  8. Select the “Optimize Entity Data Model” sub-item

  9. Wait … the tool takes a while to compile the "views”.

  10. When it’s done, your DbContext has a companion DbContext.Views class file.

  11. Build and run.
You should notice an immediate improvement in start time. There is still a delay before the first query completes. But it should be a fraction of the former delay … around 1/7th of the time. The delay for the first save should be gone; it takes no longer than the second save.

DevForce Developer Notes

Your DevForce application benefits from EF Pre-compiled views when you follow these steps. A DevForce Code First model doesn’t have to have a custom DbContext class … but you will have to create one to use this tool.

DevForce developers typically don’t define a parameterless constructor because DevForce wants a constructor that takes a connection string. Add the parameterless constructor anyway. Don’t worry, we will pickup the appropriate constructor at runtime.

When the model changes

Entity Framework detects if your entity model classes have changed since you compiled the EF views class. When you attempt your first query, you’ll get a clear runtime exception telling you to re-compile the views class.

Only database-related changes to persisted data and navigation properties matter. You can add UI hint attributes (e.g., [Display…]) and non-persisted custom properties (e.g., FullName) without triggering an exception. Any change that would affect the mapping between your entity classes and the database will trigger the exception.

How does EF know that the model has changed? I’m not certain but I have a pretty good guess. Ignore the views class filename and look at the name of the views class itself. It will be something like “ViewsForBaseEntitySets72E6108A34B7DB042DBA3C465F35B967B4E3C76051DFBAB958B69CB0D23EA8B7”.

The hex suffix at the end looks like a hash. I’m guessing it is a hash of your entity model classes and that Entity Framework spends the initial seconds before the first query reflecting over and hashing your entity model classes before comparing that hash to this views class suffix. Inside the class itself are a couple more hash values. Maybe it's using those too or instead. Someday I'll find out. It's evident that its doing some kind of comparison between the entity model classes and this views class to ascertain if there is a disconnect.

Anyway, at runtime, if EF detects a difference, it throws an exception which should terminate your app. You'll encounter the exception quickly and unmistakeably when your app first requests data. I presume that will be before you push to production :). Just re-run the tool and you should be back in business.

At IdeaBlade we’re looking into a way to detect the views/model incompatibility at build time and regenerate the pre-compiled views automatically.

Meanwhile, it’s good to know that EF fails fast when the pre-compiled views and your model are out of sync … and the remedy is as simple as re-running the tool.

Hope this helps real-world EF developers everywhere.

Update - March 23

My buddy Steve Schmitt reminds me of a few more points.
  • Rowan Miller and the EF team deserve credit for developing the EF Power Tools; we just downloaded it.
  • There’s a bit more info about the tool online.
  • If you don’t want to regenerate the views for whatever reason, you can just delete the views file and you’re back to “normal”.

Update - April 6

The EF team published this month an important white paper on performance in EF 4 and 5 that bears on pre-compiled views and other tactics that could make a significant difference for your project.

Thursday, March 8, 2012

Synchronous tasks with Task<T>

I extracted this thought from an email by Microsoft’s Brad Wilson and circulated within my company. Why not share it with you?

Brad starts with an important piece of advice: don’t make a synchronous activity async!

Ok, but how do you construct a Task<T> that you’ll consume within the context of a bundle of tasks? Brad shows how. Hey … thanks Brad!

-----
The following is an anti-pattern with tasks on a server:
return Task.Factory.StartNew(
    () => model.Deserialize(stream, null, type));
This will run your code on a new thread, forcing a context switch, which is unnecessary because your code is fundamentally synchronous. If you’re going to run synchronously, you should just run synchronously, and return a TaskCompletionSource that’s populated with your result. For example:
object result = model.Deserialize(stream, null, type);
var tcs = new TaskCompletionSource<object>();
tcs.SetResult(result);
return tcs.Task;
If Deserialize might throw, then a version with try/catch would be a better implementation of the Task contract:
 var tcs = new TaskCompletionSource<object>();

 try
 {
     object result = model.Deserialize(stream, null, type);
     tcs.SetResult(result);
 }
 catch(Exception ex)
 {
     tcs.SetException(ex);
 }

 return tcs.Task;

What do you mean by "fundamentally synchronous"?


I can assure you that the Deserialize method in question is synchronous.
model.Deserialize(stream, null, type));
That expression blocks until it returns the deserialized object. You see the stream parameter and think "this should be an asynchronous method". Maybe it should be, smarty pants; come back when you have written a deserializer that can reliably produce an object graph without reading the entire stream first.

While we're waiting for your DeserializeAsync implementation, let's push on the proposition that we should not move the execution of Deserialize to another thread.

Clearly, if this method is reading a stream, it could take "a long time" to complete. If we are running on the client, we'll freeze the UI until the deserialization completes. That can't be good. And it isn't. If we're running on the client, you should consider moving the execution of this method to another thread.

In this case, we're running on the server. There is no user waiting for the method to return so we don't care about speed on any particular thread. I'm sure the client cares about a fast response but the response isn't coming until the work is done ... on one thread or another.

We do care about total server throughput. We gain nothing by moving execution to another thread; in fact, we lose because of the thread context switching cost. This is why Brad says that spawning a new task with "Task.Factory.StartNew" is "an anti-pattern ... on a server". It's cool on the client; not cool on the server.

I'm ready with DeserializeAsync; now what?

Should we invoke the async method within a delegate passed to "Task.Factory.StartNew"? No, we should not!

This surprised me too ... until someone walked me through it ... until someone asked me "what do you think will happen on the thread you spawn?" I realized that all I would do on that new thread is dream up some way to wait for DeserializeAsync to finish. Of course DeserializeAsync spawns its own thread so I've got an original thread waiting for my task thread which is waiting for the DeserializeAsync thread. That's a complete waste of time ... and a pointless, resource-wasting context switch.

What's the point of TaskCompletionSource?

We're in this situation because for some (good) reason we want to expose a method - synchronous or asynchronous - as a Task. We don't want or need to spawn a new thread to run that method. We just want to consume it as a Task. The TaskCompletionSource is the wrapper we need for this purpose. It lets us return a Task object with the Task API that we, like puppeteers, can manipulate while staying on the current thread.

For another, perhaps better, and certainly excellent explanation of this, I recommend the following Phil Pennington video on TaskCompletionSource. Happy coding!

Friday, November 11, 2011

Build business apps in .NET - not HTML & JavaScript

A lot of people are freaking out about the apparent decline of .NET client technologies such as WPF and Silverlight. I’m one of those people. How can one, in good conscience, advocate new development in WPF or Silverlight (or Windows Forms)?

Well I can advocate it, I do advocate it, and I’m sleeping well at night.

A recent comment to one of my post on WinRT provoked an outsized reaction that has become this post. Here’s that comment:

Given that most people who are skilled in HTML5+JS write applications that are cross-browser and cross-OS every day; why would they ever consider writing for a single OS?

This fellow managed to capture in a few words the current mood and madness. My rant follows:

Who are you kidding?

First, hardly anyone is skilled in HTML 5. Period. That’s just wrong on the facts.

Secondly, most people “who are skilled in HTML + JavaScript” – cross-browser or otherwise - do not have a clue how to write a business application in JavaScript.

Most people who write apps that emit HTML do not write much JavaScript; they exploit controls that rely on JavaScript and they may write a little decorative JavaScript but that's the extent of their experience.

How do I know that? I've been hanging out where the experts in JavaScript hang ... and the recurring verities are (a) most people write horrible JavaScript (hence the need for Crockford's "JavaScript - The Good Parts"), the apps are almost unmaintainable when they do, and (c) the appropriate design and implementation patterns for JavaScript are in flux.

This is a fascinating and creative time in the evolution of JavaScript programming. New frameworks are popping up left and right. Lots of lectures and discussions about how to do it right. That's all healthy.

But the sheer wildness and variety of competing and overlapping toolkits/frameworks/guidance reveals the immaturity of the JavaScript environment as a business application development platform.

Get real about business application development

Let me be crystal clear about what I mean when I talk about a "business application" … because I want to confine my remarks to just this one kind of app. I take a completely different position on the appropriate development and delivery platform for other kinds of applications. This post is about business applications only.

[Warning: I will delete every comment that attempts to refute my argument by reference to any other kind of application. I will be especially dismissive of counter-claims which only make sense with regard to consumer facing, social networking, content broadcasting apps. You are simply not paying attention if you respond in that way and I will not waste my time or my reader(s) time with your inappropriate reply.]

Here is what I mean by a “business application”:

  1. It is a "sovereign app". That one app will dominate a user's attention for hours at a time, day after day.

  2. The user is paid to use the app. The person who pays him/her requires that user to get work done with the app as quickly and efficiently as possible.

  3. It is data-centric. The user consumes a large volume and variety of data. More importantly, the user routinely adds and updates that data in all of its variety.

  4. Therefore, the app must be highly responsive and immersive in order to maximize user productivity. It must present data and accept user input at top speed. Every moment spent waiting or wondering or searching around or flipping pages is wasted user time and wasted business owner money.

In my experience, the only apps that hit the required standard of user efficiency are apps that execute predominantly on the client. I don't care if its written in Windows Forms, WPF, Silverlight, Java, or in JavaScript ... it has to execute on the client and make only the minimally necessary trips to the server.

This is the context for answering your question. Why would I write for one OS?

If I'm writing a business app, in all likelihood there is only one OS that matters: Windows. Windows just happens to own 95% of my targeted user base (my estimate) ... today and for the near term.

[Note: The OS-on-a-PC figures are imperfect but the consensus is that Windows has no less than 88% share of all PCs which includes home PCs. Home PCs don’t matter for this analysis.]

Don't bother telling me about mobile. Don't bother giving me statistics on web traffic by OS (where Windows gets 86% of 2011 web traffic, btw). Don't tell me your vision for the future. This discussion is about business apps today and they run on PCs ... period.

If the time & cost of developing and maintaining business apps in HTML/JS were anywhere close to the marks of a .NET client platform I'd say "go HTML/JS".

But they aren't. Not close. Not soon to be close. And that fact is nowhere in dispute, not even among the most ardent HTML/JS fans.

So, as a business person, I've got a decision to make. The economics of today tell me "build it in .NET" because I can do it cheaply, effectively, with great quality using readily available resources, mature tools and experienced developers.

Of course, as a business man, I'll be concerned about the future and the long-term viability of my application. I’m perturbed that the app I am building today will be seen in 3 years as pinned to a legacy technology. That's a tough pill ... and it's why there is so much consternation. No one wants the “legacy” sign hanging around his neck.

But it’s not like everyone is going to forget how to write in .NET. Support won’t vanish. I will be able to find technology and people to keep the application alive in ten or fifteen years. I’ve got time to amortize my development costs and prepare for the inevitable transition(s) down the road.

The alternative - writing a business app in HTML/JS today – makes no economic sense. It doesn’t give me more reach because there are virtually no targeted users that I can’t reach today and for years to come. They are running Windows.

For sure the application will take 2 to 3 times as long to build in HTML/JS. I don’t have – nor can I find - the developers with the skills and experience for building an app like mine. The tools and frameworks they need don’t quite exist yet; the current prototypes are re-released with breaking changes every couple of months. My developers will be following half-baked implementation regimes, borrowed from other platforms (… oh, like .NET), that are as yet unproven for JavaScript business apps.

Yes … I could trust my critical business application development to HTML+JavaScript … if I’m a nutter.

Summary

Let me net it out for you:

  • A tiny % of today's developers are capable of writing an HTML/JS business application.

  • There are almost no examples of such apps today and almost no place to go to learn how to make them.

  • For most business apps there is only one OS that matter: Windows

  • The economics of developing a business app that runs on Windows are vastly superior to the economics of developing a business app in HTML/JS.

All that can change. All that will change. But not this year, not next year, and maybe not in 2013 either.

If you have the luxury of screwing around, have fun in HTML/JS. I know that I have that luxury and I am, indeed, having fun.

If you must write a business app for delivery in the next 12 months, do it in a .NET client technology.

Update 11/12

A discussion of this post has emerged on a G+ thread that you may find entertaining or even illuminating.

Thursday, November 3, 2011

DevForce Code First with AOP

We recently released DevForce 6.1.3 whose signature feature is Code First development of a distributed entity model, implemented with Aspect Oriented Programming (AOP). Let me unpack that for you.

Code First – You write the entity classes and only the entity classes. You don’t use a visual designer. You let the Entity Framework map your classes by convention to database objects and clarify in code those mappings that can’t be inferred.

Distributed Entity Model – “Distributed” is the key word: in a DevForce application, the client may have to rely on an intermediary server to reach the database. In DevForce, you program with a single set of entity classes, the classes you wrote, on both the server and client. DevForce handles the coordination and movement of data across tiers and over the network. The client could be Windows Forms, WPF, or Silverlight. The server could run as a Windows Service, in IIS, or in Azure. The network could be a LAN or the web.

Aspect Oriented Programming – Who needs change notification, property validation, change tracking, lazy loading, client-side caching? You do … if you want to query, save, and bind a WinForms or XAML UI directly to your interrelated collections of entities. That takes infrastructure (INotifyPropertyChanged, INotifyDataErrorInfo, etc) that you should not write yourself. Where is that infrastructure hiding when your source code only says: public string Name {get; set;}?  It’s inside your entity model assembly, after we rewrote it with PostSharp. The infrastructure you need is ready at runtime.

You can read all about DevForce Code First here. There’s a six part walkthrough of a WPF sample here. Right now, I want to talk about why I am personally jazzed about the DevForce Code First approach.

Why Code First?


DevForce has long been friendly to Entity Framework. Heretofore, we integrated closely with the Entity Data Model Designer in Visual Studio and relied upon the resultant EDMX file as the basis for generating entity classes with our own T4 template (which you can customize). That approach is still enormously popular and we remain enthusiastic about it.

But I am smitten by our new Code First option. I like Code First because I think it can reduce the friction of developing and maintaining my entity model. Let me show you a Product class I wrote in Code First:
[ProvideEntityAspect]
public class Product
{
    public int ProductId { get; set; }
    public string ProductName { get; set; }

    public int CategoryId { get; set; } // Foreign key for "Category"
    public Category Category { get; set; }

    public Guid SupplierKey { get; set; } // Foreign key for "Supplier"
    public Supplier Supplier { get; set; }
}

Product has six properties: the key (ProductId), a name, two foreign keys (CategoryId, SupplierId) and two “navigation properties” that return related parent entities (Category, Supplier). There’s no base class … unless I want one. The only evidence of infrastructure is that peculiar ProvideEntityAspect attribute … to which we will return.

This code is obvious and easy to write. The class conforms to Entity Framework naming conventions so there is no need for explicit mapping with attributes or the fluent API at this point. You’re welcome to add explicit mapping as you choose or need to do so. I didn’t need to … and I prefer to stick with the conventions.

I hammered out this class as it occurred to me. I didn’t fire up the EDM Designer in Visual Studio. I don’t have an EDMX file to manage. No partial classes separate the generated code from my custom entity business logic (of which there is none at the moment but it’s coming). There are no companion metadata classes for manipulating attributes of generated properties. There is no code generation for Silverlight clients as there is in RIA Services. The entity source code you see is the same source code living in all .NET server and client environments.

I can evolve this class as I go … with or without a pre-existing database. I can add validation attributes when and where I want them. Or I can add Product validation rules separately if I don’t want them buried in the entity. I can write DevForce property interceptors to inject behavior when a Product consumer gets or set any of these properties. What begins as little more than a property bag can grow to have as much (or as little) business logic as I think the entity requires … without touching the properties; they remain exactly as you see them.

I can write tests for my added business logic - including tests that exercise the navigations from Product to Category and Supplier – without ever contacting a server or a database.

I’m focused on the Product class and just what it needs to function properly in my business domain. The infrastructure is invisible … although accessible when I want it.

It’s not zero friction. But it’s as frictionless an approach to entity model development as I’ve seen. Which means I should be able to move my development forward faster, with higher quality, and a better chance of conveying my intentions clearly to the developer who picks up after I’ve left the project.

Beyond Entity Framework


At first blush, what I’ve described … the Product class I’ve shown … could come straight out of an Entity Framework tutorial. If you add .NET validation attributes to the properties, Entity Framework will perform object validation for you and reject attempts to save invalid entities. These DevForce entities actually are ready-to-go as Entity Framework entities. What value is DevForce adding that’s not already in Entity Framework?

DevForce brings two huge advantages:
  1. DevForce entities are queried and saved in n-tier deployments … such as Silverlight and cloud applications.
     
  2. DevForce entities are equipped to participate in Windows Forms, WPF, and Silverlight UIs.
Native Entity Framework is a strictly 2-tier technology. The client must have line-of-sight to the database. You can’t serialize and de-serialize entity graphs across the web. You can’t compose LINQ queries on a remote client. You can’t query asynchronously and manage exceptions remotely. You can’t prepare change-sets remotely and save them remotely. Such capabilities become possible only if you write a ton of difficult infrastructure … or use DevForce.

You are welcome to march into the wilderness on your own. Go write and maintain the myriad services that shuffle data into and out of DTOs (Data Transfer Objects). Try that on today’s line-of-business application with its hundreds of interrelated entities and see how much time you waste on those subterranean layers while the UI languishes and your customer taps his toes.

If you stay 2-tier and you’re pushing entity data into an ASP Web Form, the native Entity Framework entity may be good enough. Your UI controls can translate the validation attributes to JavaScript. You have little use for property change notification. You’re web server has line-of-sight to the database and can run EF on its full .NET stack.

That is not good enough for Windows Forms, or WPF, or Silverlight … the preferred client environments for line of business applications. These client technologies favor bi-directional data binding. Their controls listen for the PropertyChanged event to fire when something sets the property of a bound entity object. Many grid controls respond to hints they discover in attributes decorating the entity properties. Many controls can cancel and roll back user changes automatically if the entity supports IEditableObject. They can light up with validation error information when the entity supports IDataErrorInfo or INotifyDataErrorInfo.

Entity Validation in WPF

In this WPF sample screen shot, the “Supplier” text box is bound to the Product.Supplier.CompanyName property. DevForce navigates from the Product to the Supplier instance in cache. The user entered a value in mixed case (“All Fine Foods has a name that is far too long”). A “get-interceptor” converts the value to uppercase before returning it to the text box. The property validation mechanism immediately applies the property length validation rule and records the rule violation. The binding, which detects that Product implements IDataErrorInfo, displays the validation message.

Developers want their entities to provide this kind of support automatically. They don’t want to write it. They don’t want to see it. Developers are really tired of writing … or wading through … pages of property definitions such as

public string ProductName
{
    get { return _productName; } 
    set
    {
        if (_productName != value)
        {
            if (Validate(value))
            {
                _productName = value;
                RaisePropertyChanged("ProductName");
            }
        }
    }
}
private string _productName;

That’s all noise and no business value. It should be quieter and simpler:
public string ProductName { get; set; }

Let’s push it a bit farther and think about what must be going on inside a property that returns a related entity. Here is how you write the Product’s Category property in DevForce Code First:

public Category Category { get; set; }

It is as plain, expressive, and simple as ProductName. We have the same need for validation and notification. We have two additional requirements, neither of them met by Entity Framework: (a) we must have the machinery for retrieving, caching and saving the related Category entity on a remote client and (b) we must plug that machinery into the property’s get and set methods at runtime. 

The apparent innocence of the ProductName and Category properties is possible thanks to some nifty work behind the curtain.

Aspect Oriented Programming (AOP)


DevForce makes writing properties as simple as shown while endowing them with the capabilities necessary for participation in the .NET client UI technologies. Clearly the “property as shown” cannot be the actual implementation at runtime. Somehow the infrastructure code has to insinuate itself into the property.

DevForce uses PostSharp and Aspect Oriented Programming (AOP) to inject the infrastructure into your classes. It re-writes the compiled model assembly at build time. That ProvideEntityAspect attribute adorning the top of the Product class tells DevForce that Product is a kind of entity and should implement the behaviors that all entities exhibit. DevForce and PostSharp replace the get and set methods of entity properties with code that ties into DevForce infrastructure. The revised Product class inherits from and implements the interfaces that light up the UI controls. It also acquires support for querying, caching, change-tracking, and saving product data. The Product class, after re-write, is an enriched version of the original Product class you wrote. The Product objects that you “new” at runtime are instances of the enriched Product class. When a binding reflects into an instance of Product, it finds the interfaces it’s looking for.

Many frameworks, including Entity Framework and NHibernate, use an alternative approach called “dynamic proxies.” They leave the original compiled classes alone. Instead, they rely on an external component (e.g., a “Context” or “Session”) to encase new and queried entities in a wrapper class that derives from your entity class. This wrapper overrides your entity’s properties with implementations that delegate (“proxy”) to the framework’s infrastructure.

That infrastructure, if expanded appropriately, could do all of the interception, routing, and notification that DevForce does. You’d have to adapt your entity and workflow to play along. All of your entity properties would have to be virtual. You couldn’t construct them directly with “new”; in fact, you’d have to be careful always to reference the proxied object. You’d also have to get used to debugging with strange wrapper types, sporting bizarre long names.

We like the AOP approach better. We think most people will find it more natural to work with a class that is fully baked at build time rather than maneuver to consume proxies that are emitted dynamically at runtime.

Could this be magic?


I don’t trust all that magic!'” That’s a common early reaction to Code First, AOP (and dynamic proxies). When I rely on Entity Framework naming conventions to map class and property names to tables and columns, how do I know what it did or did not map? When AOP is injecting logic into my classes that I can’t see, how do I know what it is doing?

I push the button on my coffee maker and good, hot brown stuff pours into my cup. That’s magic to me. The compiler turns my auto-property into two get and set methods with a backing field. That’s magic. Then it becomes IL code which becomes machine code which makes the hardware jump. That’s magic.

The difference is that coding by convention and AOP are new magic for most of us. We’re used to the old magic. We probably never truly understood it; we just stopped worrying about it. Before, when the property code was visible and the entity-data map was in an EDMX file, we felt the visceral comfort of touching the code and displaying the XML if we wanted to. But we rarely wanted to. When we did look, we were as often confused by what we saw as not.

What matters most is good diagnostics and easy debugging. I need a clear explanation when things go wrong and I need to know what to do about it. Fortunately, the Entity Framework mapping error messages are pretty good. I think the AOP debugging experience will feel familiar to you. You breakpoint your custom code exactly as you did before. You step through your code as you did before. You test your code as you did before.

I am keen to know what you think after you’ve tried it. We’re eager to hear your suggestions. IdeaBlade issues DevForce updates every six-to-eight weeks so we can respond quickly.

What about existing databases?


I often hear that Code First is only for “greenfield” development and small database schemas. Perhaps half of our customers (re)build existing applications for existing databases with hundreds of tables some of which have hundreds of columns. Heaven help them.

Such “brownfield” development presents at least three challenges:
  1. Getting started … because no one wants to write all those entities by hand.
  2.  
  3. Evolving the entity model and a production database as requirements change.
  4.  
  5. Keeping the model and the database in sync without the assistance of the EDM Designer.
We meet the first challenge with an approach we call “Code Second”. Speaking schematically, you aim the EDM Designer at the database and generate DevForce “Code First” classes. Then you throw away the EDMX file and proceed in Code First style.

I’m optimistic but less confident in my response to the second and third challenges. I know developers in other communities (NHibernate in particular) have confronted these demons and wrestled them to the ground. They’ve survived without Database First tooling as far as I can tell. I’m betting they’ve discovered successful practices we can learn from and adapt.

Code First style entails evolving the model and existing database in small doses, supported by tests. Runtime mapping exceptions are sufficiently informative in the context of small changes; repairing small breaks should be easy. You shouldn’t wake up in shocked surprise one morning to find that the model or the database have been massively transformed. That’s a disaster for Database First as much as Code First; it indicates far more fundamental flaws in your development process.

A new module may require many new entity types to support its scenarios. The new entity types can be related to other types, new and existing. The new model ultimately translates to new tables and foreign key relationships to new and existing tables. I think I would develop a new module as I would a “greenfield” project with stubbed stand-ins for the existing entities and tables.

It may be wise to preserve this separate model and separate database … forever. But if that’s not in the cards, when the new project matures, I’d fold my Code First model into the existing model. I’d script database changes from the generated Code First design database and apply them to the existing database.

I readily admit that I do not have the concrete experience to back up these suggestions. I expect to have more to say on this subject in future posts. One of our customers hired us to help them rebuild their 500 table application in DevForce Code First. I promise to report back on do’s and don’ts.

What next?


I’d love to know what you think. Is Code First attractive to you? Do you have pros and cons to share? Would you explore our walkthrough and give me your feedback? Is there something that bothers you about what I’ve said? Anything you want me to expand upon?

I’m here for you.

Monday, October 10, 2011

Windows 8 Metro apps in JavaScript

I’m having a blast exploring Windows 8 Metro app development and poking around in the preview bits shipped at the 2011 BUILD conference.  Last weekend, I presented on Metro app development using JavaScript at the Silicon Valley Code Camp.

I’ve posted the sample code, my presentation script, and my slide deck on the IdeaBlade wiki.

I have not gone crazy and transferred my affections from .NET to JavaScript. I’m just curious about all the hub-bub. It’s also the only fair way to confront the assertions that JavaScript has grown up and that we’ll all soon be JavaScript programmers.

What is all this noise about “programming in standards-based HTML5 and JavaScript” … which is technically possible in Windows 8 … although in practice you’ll be so imbricated in Windows that you might as well have written it in COM.

I exaggerate … slightly. I’m not being critical though. You want to program for the iPad? You’ll use Objective C. For the Android? You’ll write in Java. At least for a Windows 8 device you can write your app in Javascript … or C++ … or C#/VB. The languages are (mostly) open; the application code is locked in.

I had tons of fun following along with Chris Tavares and Chris Sells as they built a Metro RSS Reader in JavaScript. The resulting demo app looks like this:

WinJsDemo-Part 5

I stole borrowed almost everything I showed from Chris’ talk … unless I pinched it from Luke Hoban’s talk.

It’s always a kick to take a new technology out for a spin. The Microsoft folks are doing wonders with JavaScript on Windows but it’s still a dog walking on hind legs and always will be. I get … and approve of … what they’re doing: embracing the fresh faces coming out of school who “know” (be very wary of that verb!) HTML and JavaScript … not C#. That’s smart business.

I expect to keep playing with JavaScript and writing about my experiences. I’m sure that it’s the perfect tool for many jobs. HTML and JavaScript (of a more general variety) may turn out to be the universal solvent for cross-vendor mobility apps.

But not now and not soon. When it’s time to get serious about writing business applications, I’m sticking with the XAML technologies, Silverlight especially. Far more productive, less painful, and less risky in the hands of muggles … which is to say, most of us.

Thursday, September 22, 2011

“Desktop” could be Win 8’s killer feature with consumers

Crazy? Or just counter intuitive? Give me a moment to explain.

The story everyone tells goes something like this: Microsoft is strongly entrenched on the PC and in the enterprise. But (XBox and Kinect aside) it’s lost touch with the consumer. Lose the consumer and you will lose the enterprise. It’s only a matter of time before iPads displace PCs just as PCs displaced IBM 3270 terminals. Microsoft must respond quickly with consumer-friendly mobile products.

I agree for the most part. And I feel terrific about the Windows 8 preview and tablet that Microsoft unveiled at //build/. Microsoft definitely got some of its mojo back. I’m not just on a “sugar high” as some would have it. Windows 8 is a giant step in the right direction. Regular folks really do like that Metro look. The “developers’ preview” is obviously not as polished as the iPad but there is time before release to perfect the fit and finish.

Unfortunately, it won’t be released until December 2012 (give or take). Fifteen months! In a few weeks, after we’ve sobered up, that’s going to feel like forever. Sure, when the grand day finally arrives, we’ll be cheering for the underdog, the version 1.0 Win 8 newbie against the heavyweight 3.0 iPad market leader.

Hmmm. Should we be worried about a replay of the Windows Phone adoption debacle?

Lessons from Windows Phone 7

There is nothing seriously wrong with the Windows Phone itself. It’s a joy to use, in some respects superior to the iPhone and in no material respects is it worse. It’s where Metro was born … the same Metro that’s going to wow the tablet market in 2012. Critics loved it … as they love the Windows 8 preview.

We can’t fault Microsoft for losing sight of the consumer. The Windows Phone is totally consumer-focused; two years in and we still lack an enterprise story. To make sure we get the point and avoid all possible confusion, Microsoft killed off WP7’s enterprisey predecessor.

How is that working out? Microsoft’s share of the phone market has declined steadily and is headed for 1%. Great phone, unwavering consumer focus, pitiful results.

But wait,” they say, “it will all get better with Mango!” Really? Mango is indeed a huge technical improvement. But I don’t understand why that will make a difference in the marketplace. It is not sufficiently different from the iPhone. Seriously, what will I be able to do with a Mango phone that I can’t do with the iPhone? I’m sure you have a list; will anything on that list be enough to change consumer behavior? Will any of it change the foot-dragging behavior of the sales channel? I don’t get it.

Look, I completely buy the consumer friendliness bit. The traditional desktop experience doesn’t fly on the phone and it won’t fly on the tablet. The way we write business apps today is totally alienating. So I’m down with the design vision: “Metro good, touch good, square windows bad, icon overload bad, mouse-and-keyboard bad.”

The Metro-ish design changes are absolutely necessary … but they aren’t sufficient … because the iPad (like the iPhone) has a commanding lead in these respects.

Windows 8 can be super great … and still bomb at the box office. “Great” isn’t good enough. If it were, Apple might have displaced Microsoft on the desktop long ago. The Mac was always prettier and easier to use. Didn’t matter; share never broke 10%. Twenty years and the Mac couldn’t knock Microsoft out of the ring. It was Microsoft’s ring.

If “great design” isn’t good enough, what is good enough? What does Microsoft have that Apple lacks? What would make someone choose a Microsoft tablet over an iPad tablet … maybe even put down the iPad and take up a Win 8 device.

Enterprise software

I argue that consumers – not all, but millions of them – would put down their iPads and buy Windows 8 tablets if they had the ability … nay, the ease … of running their existing enterprise applications on that same fun tablet.

We consumers aren’t just sitting around watching the Price is Right while stuffing Twinkies into our mouths. Most of us have jobs. Many of us use computers on the job. And while we may season our days with guilty pleasures (“hello, Twitter”), we occasionally get work done.

News flash: people take their work home! Americans especially. The work we take home is often written for and running on a Microsoft Windows machine.

Why does Microsoft appear to be running away from this obvious fact? Microsoft could … and should make work life – the life bound up in Microsoft-based business applications – a high-profile component of the Win 8 marketing plan.

Unfortunately, conventional wisdom says they shouldn’t do that. Conventional wisdom says that mobile devices must cater exclusively to an infantilized consumer.  Little baby consumer want pretty applications that are fast and fluid. Little baby consumer want an RSS reader with big pictures … like this one straight from the Jensen Harris Metro-style keynote:

image

Say what? Geez, who doesn’t need a little pizzaz in his life but call me different: my RSS reader is for reading. Jensen, dude, you’re showing me eight pointless photographs when I want a list of twenty titles and abstracts … you know, something I can use.

But I digress …

Where did we get this limiting caricature of the consumer? I think we bought it from Apple.

Where did Apple get it? Market research? I don’t believe it. I believe Apple made it up. It’s too convenient.

Apple has no enterprise play. They couldn’t promote enterprise applications on the iPad because they have no enterprise applications to promote. So they follow the Apple playbook: if we can’t do it, you shouldn’t want to do it. They pretend that mobile is some kind of other worldly, purely personal experience where work does not belong.

The sad thing is that Microsoft is falling for it. They fell for it with the phone. They’re poised to fall for it again on the tablet. By slavishly chanting Apple’s mantra, Microsoft effectively disarms itself.

Let me be crystal clear.  The tablet is not a PC. Mobile is not the office. Traditional business application UX is tired, clumsy, and unfriendly; todays business apps are as ill-suited for work as they are for play. WIndows 8 Metro is cool beans and a necessary way forward.

But its not all about fun and games. Apple owns that market anyway. Microsoft must wrest it away from them. Microsoft can’t win with great design. It can’t win share with advertising.

Let me channel today’s consumer for you. I …. don’t … care! Microsoft, I think you’re irrelevant. You’re going to have to rip that iPad out of my hands and compel me to use Windows 8.

Convince me, Microsoft, that I can play while I work on that one hot device that covers both my personal life and my job life. Convince my boss that the Windows 8 tablet is both a match for the iPad and a productivity platform for enterprise applications … at no additional cost.

Perfect execution of the perfect Windows 8 design won’t move the needle in 2013. By then it will be “me too, too late”. 

But enterprise software is the differentiator that Apple can’t match. Exploit that. Don’t let Apple fake you out. Leverage your strength. The enterprise angle must be a strategic element of your consumer play.

That’s why I think the Windows 8 “Desktop” stack is the killer feature in Windows 8. Ignore the voices that tell you otherwise. Is “Desktop” too confusing to some consumers? Work on making it less confusing. Don’t kill it; invigorate it.

Energize your enterprise developers – hundreds of thousands strong – who want to Metro-ize their business apps. Stop telling them what they can’t do in Metro. Inspire them to build great Metro business apps. Get moving on the roadmap for the enterprise store; get a proto-type out there. Reassure your corporate customers and developers that you mean business by investing meaningfully in the evolution and marketing of the mature technologies they use today. Praise Silverlight 5 and get hopping on Silverlight 6. We aren’t writing business applications in HTML and JavaScript today. We’re not going to have production ready HTML/JS apps in 2012. That’s just jerking us around. If you really don’t think Metro is the answer for all desktop application needs, then paint us a Windows future that is an answer and build us a bridge from here to there.

I realize I’m proposing an “and” strategy – consumer apps and business apps - not an “or” strategy. “And” strategies are dangerous because they tend to defocus. You have limited resources to be sure. But you can’t just go to market with a carbon copy of Apple’s strategy. They’ll kill you like they’re killing you on the phone. How many billions will you spend for a few points of share? Redirect a fraction of those billions to make enterprise software a first class, integrated pillar of your tablet … on day one. That’s a damn cheap hedge against a repeat of the Windows Phone launch if you ask me.