In my previous blog post on Real Native WinRT development, I wrote some native C++ code that created an instance of the Windows.UI.Xaml.Application class, and called its Application.Run method. Because I was using pure native C++ instead of the compiler’s WinRT language projection that sane developers will use, this very simple task took a short helper function and 15 lines of code. (If you read that article, you may recall that there was an unresolved bug—the Run method was failing. I’ve since updated the post to fix that. It turned out that I was using the wrong threading model. Surprisingly, you have to initialize the UI from a multithreaded context.)

While the code was verbose compared to the equivalent when using the WinRT language projections—a mere two lines—it’s about to get even more complicated. Although my previous example did technically run, it turns out that to do anything useful in a WinRT Xaml application, we need to derive a type from the built-in Application class. (That’s because you have to override at least one method to be able to show a non-empty UI.) With the WinRT language projections, inheritance is straightforward:

ref class App :  public Windows::UI::Xaml::Application
{
    ...
};

But I’m choosing to do this natively because I like to understand what’s going on under the covers. In the raw native world, inheritance looks rather more complex.

Just to be clear I’m doing this to learn how WinRT really works. I would NOT normally write a C++ WinRT application this way.

WinRT uses COM for its underpinnings, and classic COM never supported implementation inheritance. So the first challenge was to work out how WinRT maps the concept of inheritance into a COM-like world. I’ve not yet found any clear documentation for this, but there’s a clue in the Application class documentation: the Attributes section shows that the class has a ComposableAttribute with a first argument of Windows.UI.Xaml.IApplicationFactory. I’ve not found any documentation for that interface, but its definition lives in Windows.UI.Xaml-coretypes.h, and from that we can see that this interface defines a single member:

HRESULT CreateInstance(
  /* [in] */ IInspectable *outer,
  /* [out] */ IInspectable **inner,
  /* [out][retval] */ Windows::UI::Xaml::IApplication **instance)

But where do I get an object that implements this IApplicationFactory interface? Last time, I called RoActivateInstance to create an instance of the Application class. But if I want to get hold of the factory for that class I need a different API: RoGetActivationFactory. All WinRT classes provide an activation factory, which is what RoActivateInstance uses to create new objects. (It’s very similar in concept to a classic COM class factory.) But when doing something fancier, such as inheritance, we need to talk directly to that factory, rather than letting RoActivateInstance do that for us.

Inheritance and COM

Here’s how WinRT makes inheritance work in a COM-like world. It appears to involve three objects:

• The ‘inner’ object, which provides the base implementation

• An ‘outer’ object, which defines overrides for virtual methods

• A wrapper object which combines the ‘inner’ and ‘outer’ objects into what looks like a single object

We provide the ‘outer’ object, while the ‘inner’ and the wrapper objects are provided by the WinRT type from which we are deriving. These three objects correspond to the three arguments to IApplicationFactory.CreateInstance. (I’ve modified the order, because to me, this order makes more sense: we start with the base type, we override virtual methods, and the result is a new type that is the combination of the base object and the overrides.)

Although this system provides a way to model inheritance, it’s actually quite different from how inheritance looks in ordinary C++, C#, or even in the C++ language projection for WinRT. First of all, it’s based on object instances, rather than types. But also, unlike with C++ (or C#) inheritance, we have to define overrides for all virtual methods. That’s because our ‘outer’ object defines overrides by implementing a special ‘overrides’ interface specific to the base class. If you look at a non-sealed WinRT class, you’ll usually see that it implements two interfaces. For example, the Application class implements IApplication and IApplicationOverrides. That second interface defines the methods which inheriting objects can override. (Remember, in COM everything looks like a method, including properties, so virtual properties use this same mechanism.)

Since COM interface implementation is an all-or-nothing prospect, we have to implement every overridable method. If we don’t really want to override a method, we just call back to the base implementation, using the ‘inner’ object returned by the CreateInstance method.

If you use the C++ WinRT language projection, you can see the compiler generating exactly this sort of code. If you pass the /d1ZWtokens switch, the compiler will display the full token stream of the code it’s compiling, including any generated code. (Beware: this will slow down compilation considerably, as it dumps several megabytes of code into Visual Studio’s mysteriously slow Output panel.) Actually it generates a bit more than a plain old call to the base class. Here’s the code it produces in an ordinary WinRT project (with the WinRT projection enabled) for a method that I’m not overriding:

inline long __stdcall ::HighLevelWinRTClient::App::
  __cli_Windows_UI_Xaml_IApplicationOverrides____cli_OnInitialize()
{
    struct Windows::UI::Xaml::IApplicationOverrides^ __tempValue;
    long __hr = __cli_baseclass->__cli_QueryInterface(
        const_cast<class Platform::Guid%>(
            reinterpret_cast<const class Platform::Guid%>(
                __uuidof(struct Windows::UI::Xaml::IApplicationOverrides^))),
        reinterpret_cast<void**>(&__tempValue));

    if (!__hr) 
    {
        __hr = __tempValue->__cli_OnInitialize();
    }

    return __hr ; 
}

This uses a class member called __cli_baseclass, which is where the compiler stores that ‘inner’ IInspectable that comes back from CreateInstance. It calls QueryInterface to ask for the IApplicationOverrides interface, and uses that to call the base class’s original implementation of the method. This seems pretty inefficient—it’s clearly going to be quicker to ask for that interface just once and store that, rather than doing a QueryInterface every time the method runs. I’ll take that more efficient approach when I write my derived class by hand—the joy of real native code is that you can control this sort of thing.

And the horror of real native code is that you have to write this sort of thing, whether you want control over it or not. We now have some work to do: we must implement this IApplicationOverrides interface on an object which we pass into the activation factory’s CreateInstance method as the ‘outer’ object. Since this is a WinRT interface it derives from IInspectable. (You can see that the first argument of IApplicationFactory.CreateInstance takes an IInspectable* argument.) Remember that IInspectable is the new interface from which all WinRT interfaces derive, and it derives from the classic COM IUnknown interface. So we’ll need to supply a working COM object with a viable implementation of IInspectable. While I could write that by hand, it’s not much fun. Fortunately, it turns out that I don’t have to write it completely from scratch, even in this fully-native world. I can turn to a new library, called WRL.

WRL (Windows Runtime Library?)

I’m not entirely sure what WRL stands for because I’ve not found the documentation for it yet, but I’m guessing it might be the Window Runtime Library. Whatever it stands for, it appears to be the successor to ATL, which was a widely used library for writing COM components in C++. If you’re familiar with ATL, you’ll feel at home with WRL, but of course WRL knows about WinRT things like IInspectable, which is why we’re using it. Here’s the class definition for my Application-derived class:

namespace LowLevelWinRTClient
{
  using namespace Windows::ApplicationModel::Activation;

  class DerivedApp :
      public Microsoft::WRL::RuntimeClass<Windows::UI::Xaml::IApplicationOverrides>
  {
    InspectableClass(L"LowLevelWinRTClient.DerivedApp", TrustLevel::BaseTrust)

    Microsoft::WRL::ComPtr<Windows::UI::Xaml::IApplicationOverrides> _spBaseImplementation;

  public:
    DerivedApp(void);
    ~DerivedApp(void);

    void SetBase(Windows::UI::Xaml::IApplicationOverrides* pBaseImplementation)
    {
      _spBaseImplementation = pBaseImplementation;
    }

    HRESULT STDMETHODCALLTYPE OnInitialize();
    HRESULT STDMETHODCALLTYPE OnActivated(IActivatedEventArgs *args);
    HRESULT STDMETHODCALLTYPE OnLaunched(ILaunchActivatedEventArgs *args);
    HRESULT STDMETHODCALLTYPE OnFileActivated(IFileActivatedEventArgs *args);
    HRESULT STDMETHODCALLTYPE OnSearchActivated(ISearchActivatedEventArgs *args);
    HRESULT STDMETHODCALLTYPE OnSharingTargetActivated(IShareTargetActivatedEventArgs *args);
    HRESULT STDMETHODCALLTYPE OnFilePickerActivated(IFilePickerActivatedEventArgs *args);
  };
}

There are a few things to notice here. The first is that this derives from WRL’s RuntimeClass base type. This implements the methods of the IUnknown and IInspectable base interfaces that all WinRT interfaces derive from. So all we have to do is implement the methods specific to whichever interfaces we want to offer.

I’ve passed the IApplicationOverrides interface as a template argument to RuntimeClass—WRL needs to know which interface(s) we want to implement to implement the IUnknown::QueryInterface and IInspectable::GetIids methods correctly. WLR defines several RuntimeClass types, each taking a different number of interfaces as template arguments, but since I only need to implement one interface on this particular object, I’m using the one-argument version.

The next point of interest is the line beginning InspectableClass. That’s a macro which provides information necessary to implement IInspectable. As you may recall, IInspectable offers methods to discover an object’s type name and its trust level, so as you’d expect, we need to provide these to the macro, so that RuntimeClass can implement IInspectable correctly.

I’ve also declared a field to hold the ‘outer’ object—the base implementation that we use for methods that we don’t really want to override. (We can also use it when our overrides need to call the base class as part of their implementation.) I’ve written a public helper method to set that base reference because I’ve chosen to make this IApplicationOverrides object standalone—it won’t actually create its own base instance. You probably wouldn’t do it this way for real, but for this example, I wanted to keep things that are separate COM objects as separate classes, just to make it easier to see all the moving parts required for WinRT inheritance.

Finally, we implement all the methods defined by IApplicationOverrides. (Again, I couldn’t find documentation for this class, so I got this method list by looking at the relevant header file.)

Derived Implementation

Most of my method implementations look something like this:

HRESULT DerivedApp::OnInitialize()
{
    return _spBaseImplementation->OnInitialize();
}

I’m just calling the corresponding method on the base class—this is how we choose not to override a method. If there are arguments, we just pass them straight through. Six of my seven methods defer to the base class like this.

The only method I’m truly overriding is OnLaunched. In the world of the C++ WinRT language projection, the C++ Metro template contains code which provides the application’s window with its initial content, and then activates it, like this:

void App::OnLaunched(LaunchActivatedEventArgs^ pArgs)
{
  Window::Current->Content = ref new MainPage();
  Window::Current->Activate();
}

Here’s raw native code that does roughly the same thing.

HRESULT DerivedApp::OnLaunched(ILaunchActivatedEventArgs *args)
{
  using namespace Windows::UI::Xaml;

  ComPtr<IWindowStatics> spWindowStatics;
  HRESULT hr = Windows::Foundation::GetActivationFactory(
    ComHstring::Make(RuntimeClass_Windows_UI_Xaml_Window),
    &spWindowStatics);
  ComPtr<IWindow> spCurrentWindow;
  if (SUCCEEDED(hr))
  {
    hr = spWindowStatics->get_Current(&spCurrentWindow);
  }

  ComPtr<Windows::UI::Xaml::Markup::IXamlReaderStatics> spXamlReaderStatics;
  if (SUCCEEDED(hr))
  {
    hr = Windows::Foundation::GetActivationFactory(
      ComHstring::Make(RuntimeClass_Windows_UI_Xaml_Markup_XamlReader),
      &spXamlReaderStatics);
  }
  ComPtr<IUIElement> spContent;
  if (SUCCEEDED(hr))
  {
    wchar_t const* xamlContent =
      L"<Grid xmlns=\"http://schemas.microsoft.com/winfx/2006/xaml/presentation\">"
      L"  <TextBlock Text=\"Hello, world\" />"
      L"</Grid>";
    ComPtr<IInspectable> spContentAsInspectable;
    hr = spXamlReaderStatics->Load(
      ComHstring::Make(xamlContent),
      &spContentAsInspectable);
    if (SUCCEEDED(hr))
    {
      hr = spContentAsInspectable.As(&spContent);
    }
  }
  if (SUCCEEDED(hr))
  {
    hr = spCurrentWindow->put_Content(spContent.Get());
  }
  if (SUCCEEDED(hr))
  {
      hr = spCurrentWindow->Activate();
  }

  return hr;
}

Again, roughly 10 times as much code!

OK, that’s not quite true. The original code relies on the project defining a UserControl-derived class which it just instantiates and uses as content. I didn’t want to tackle writing a UserControl in raw native code just yet, so instead, I used the WinRT XamlReader class to load some Xaml from a string constant. About half the code you see here is related to that.

On the other hand, it’s actually slightly more complex than it looks. I got bored of writing code to create and destroy HSTRINGs, so I wrote a little helper, ComHstring, which I’m using in a few places in that code. It provides implicit conversions to HSTRING, along with a destructor that automatically deletes the HSTRING once the temporary returned by its Make method is done with. (As far as I can tell, WRL doesn’t have an HSTRING wrapper, which is slightly surprising.)

Static Methods

In that last example, you’ll notice a couple of calls to GetActivationFactory, which is a WRL wrapper around RoGetActivationFactory. Earlier I mentioned that as a way of getting access to the mechanism for inheritance, but it’s also the way to get access to the static methods defined by a class. Classic COM doesn’t have a concept of static methods—methods are always presented through an interface pointer, so there’s always an instance involved. When WinRT classes define static methods, they appear on the activation factory.

Instantiating the Derived Type

Now that we have an implementation of IApplicationOverrides, we’re ready to create an instance of our Application-derived type. Here’s the code:

ComPtr<Windows::UI::Xaml::IApplicationFactory> spApplicationFactory;
hr = Windows::Foundation::GetActivationFactory(
  ComHstring::Make(L"Windows.UI.Xaml.Application"),
  &spApplicationFactory);
CheckHresult(hr, L"GetActivationFactory");
    
ComPtr<DerivedApp> spDerivedApp = Make<DerivedApp>();
ComPtr<Windows::UI::Xaml::IApplicationOverrides> spDerivedAppOverrides = spDerivedApp;

ComPtr<Windows::UI::Xaml::IApplication> spApplication;
ComPtr<IInspectable> spInner;
hr = spApplicationFactory->CreateInstance(
  spDerivedApp.Get(),
  &spInner,
  &spApplication);
CheckHresult(hr, L"Application CreateInstance");

ComPtr<Windows::UI::Xaml::IApplicationOverrides> spBaseImplementation;
hr = spInner.As(&spBaseImplementation);
CheckHresult(hr, L"QI for base IApplicationOverrides");
spDerivedApp->SetBase(spBaseImplementation.Get());

(ComPtr is a WRL smart pointer. It automates some aspects of COM that otherwise get tedious fast. It’s similar to ATL’s CComPtr, although some operations that CComPtr performed implicitly now require explicit code, largely because those implicit operations were responsible for a lot of bugs in code written by people who didn’t really understand how CComPtr works. With the new ComPtr, a failure to understand how it works is more likely to lead to a compiler error than a runtime bug.)

We get the Application class’s activation factory (using GetActivationFactory which is a wrapper for RoGetActivationFactory). We create an instance of our IApplicationOverrides class—the Make<T> method I’m using here is a WRL helper that creates and initializes RuntimeClass-based object. We then call the factory’s CreateInstance method, passing in our overrides implementation. CreateInstance passes back two interface pointers. One is the ‘inner’ object, which we then pass into our overrides object—remember, it needs that to be able to call the base class for methods that it doesn’t wish to override. (Notice I’m doing the QueryInterface for IApplicationOverrides during this construction phase, instead of doing it in every single method invocation as the C++ compiler seems to in its projection.)

At the end of all this, the spApplication variable points to our finished object—it is an interface pointer to the wrapper generated by the Application class which combines our overrides with the base implementation. So we can now run that to start the application:

hr = spApplication->Run();

When I run my application, I see the “Hello, world” TextBox that I created in my OnLaunched method appearing, verifying that I have successfully overridden that method in my derived class.

What could be simpler?

I’ve been at Microsoft’s //build/ conference this week, where they announced WinRT. WinRT is a new API for building Windows applications that use the new ‘Metro’ style. Under the covers this uses a lot of COM-based technology. However, we’ve seen very little of that COM layer.

The various languages that support WinRT (C#, VB, JavaScript, and C++) all define “projections” to make the new WinRT-based objects easier to work with. After all, who really wants to use COM if they can help it? Even the vast majority of the C++ examples we've been shown have used the C++ projection. Slightly confusingly, this hasn’t always been made very clear—some presenters have shown C++ examples and talked about native code and COM while showing us things that are actually the C++ projection.

If you want to crow about how C++ lets you do native development (and apparently some people seem very excited by this) you don’t get full bragging rights if you’re using the projection. The C++ Metro project templates in the Visual Studio 11 preview are all working at a level removed from the true native experience.

That’s a good thing, by the way (for much the same reasons that managed code is a good thing). Without the various language projections, WinRT would be a clear reminder of why so many of us walked away from COM a decade ago. There will rarely be any good reason to work directly with WinRT at the native level. (Adopting a boastful “more native than thou” attitude may be a reason of sorts, but it’s certainly not a good one.) But I like to know how things work, so I thought it’d be fun to try out some real native WinRT development. This means eschewing the projections, and getting down to the level where you will see an actual vtbl call. (We didn’t see any such thing in the ‘big picture’ talks on the first day of the conference, despite what some presenters claimed.)

Raw Native WinRT Client Projects

The first challenge is to create a C++ project capable of working in the WinRT environment, but without opting into the projected world. None of the templates supplied with the Visual Studio preview seem to do this—the Metro projects all turn on the projection features, but the classic C++ templates produce programs which won’t run in the new WinRT world. (If you saw any of the //build/ presentations on WinRT, you may remember the green and blue boxes—the green boxes are the new Metroid world, while the blue boxes represent the classic pre-Win8 world.)

You need your project file to contain a combination of settings which, as far as I can tell, can’t actually be configured in Visual Studio’s UI, so some manual project file editing is required. In particular, you need an <Immersive>true/</Immersive> element in a property group to enable the deploy step that builds a new appx package for your app—if I’ve understood correctly, you won’t get into the green box without this. (This property is present in all the new Metro templates, but as far as I can see, there’s no UI for turning this on in an existing C++ project.) However, that setting also turns on all the projection stuff, so you need to turn that back off again. You need a <CompileAsWinRT>false</CompileAsWinRT> inside the <ClCompile> section for each configuration, and also <GenerateWindowsMetadata>false</GenerateWindowsMetadata> and <WindowsMetadataFile /> inside each configuration’s <Link> section.

If you don’t turn those off, you’ll get a load of errors when you try to include the header files that define the native versions of all the WinRT types and interfaces—the C++ compiler will complain that it has already seen all these types (because it imports them automatically when you’re using the C++ WinRT projection), and that they all look different (because the C++ projection represents many these types differently from the native versions).

So, we now have a project that builds genuine native C++ code with none of the projection features switched on, but which puts the result into the new appx packaging format. This will also use the appx-aware debugger. (You can turn that on with <DebuggerFlavour>ImmersiveLocalDebugger</DebuggerFlavor>, but that setting will be in effect as a result of turning on the immersive setting for the build system.)

Next, we need some code.

Ruh Roh!

First, let’s initialize the runtime:

HRESULT hr = ::RoInitialize(RO_INIT_SINGLETHREADED);

[Update (2011/09/20): it turns out that this should be RO_INIT_MULTITHREADED if you want it to work.]

That should look hauntingly familiar to anyone who’s done any COM development. It’s not quite the same—the classic COM versions of this API start with ‘C’ rather than ‘R’, so that’d be CoInitialize, and if you called the version that accepted a threading mode (CoInitializeEx) that’d be COINIT_APARTMENTTHREADED.

But at this stage, WinRT definitely sounds like native COM, albeit spoken in a silly voice.

Old-School Error Handling

You’ll have noticed the HRESULT in that first line of code. If you’ve done any COM, you’ll be all too familiar with this. It’s just a 32-bit integer, and all COM operations (APIs and also method calls on objects) return these. If the top bit is set, that indicates an error, and the number may or may not tell you something about what failed. You have to check these every time you do anything. So I wrote a little helper:

inline void CheckHresult(HRESULT hr, LPCWSTR message)
{
  if (FAILED(hr))
  {
    wcout << L"Error (0x" << hex << hr << L") during: " << message << endl;
    exit(1);
  }
}

That’s a bit brutal, but it’ll do for now. We should call it after RoInitialize:

CheckHresult(hr, L"RoInitialize");

And we’ll be seeing more of that sort of thing.

Creating an Object

Now that we’ve initialized WinRT, we need an object. The first thing most immersive applications do is create an Application object. For example, in the high-level world of the C++ projection for WinRT, here’s the generated program entry point in App.cs.g from an ordinary metro app:

int main(lang::array<Platform::String^>^ args)
{
  auto app = ref new App();

  app->Application::Run();
}

That first line constructs the App object, which derives from the Application class. For now, I’m not going to get into derivation—for this blog post, I’m just going to construct the base Application class directly. As you can see above, in the world of the C++ projection for WinRT creating an object is one line of code. Here’s the native version of that one line of code:

const wchar_t *appClassName = L"Windows.UI.Xaml.Application";
HSTRING hstring;
hr = ::WindowsCreateString(appClassName,
       static_cast<UINT32>(::wcslen(appClassName)), &hstring);
CheckHresult(hr, L"WindowsCreateString");
IInspectable* pInspApp;
hr = ::RoActivateInstance(hstring, &pInspApp);
CheckHresult(hr, L"RoActivateInstance");
::WindowsDeleteString(hstring);
CheckHresult(hr, L"WindowsDeleteString");

What a lot of code. Actually the interesting part is just two lines in the middle:

IInspectable* pInspApp;
hr = ::RoActivateInstance(hstring, &pInspApp);

The rest is all string handling or error handling, which is exactly the sort of low-level cruft that the language projections save you from. All this sort of thing will be going on, it’s just that if you choose to use the high-level projections, they hide all that for you.

HSTRING is WinRT’s native representation of strings. COM veterans will know that this is new—in old-school COM, we represented strings in numerous ways, but this wasn’t one of them. (In the VB/scripting/IDispatch/dual side of the COM house, strings were typically BSTRs, while in the non-C++-languages-need-not-apply side of COM, they were often just plain old LPWSTRs, although there were a bunch of other more unusual options.) I’ll leave the what and why of HSTRING for another time, so for now, just know that this is how WinRT expects its strings to look, so we had to create one from our C-style string constant to be able to call RoActivateInstance.

So that’s two departures from classic COM: there’s a new string type, but there’s also the fact that we’re using a string at all. I’m passing a string containing the name of the class I want to instantiate (“Windows.UI.Xaml.Application”) to RoActivateInstance. The nearest equivalent classic COM API was CoCreateInstance, and that used GUIDs to identify types. But in WinRT, you’ll see strings in a lot of places you used to see GUIDs.

In fact that call to RoActivateInstance introduces a third new thing: IInspectable.

IInspectable

In classic COM, all interfaces derived from a base interface called IUnknown, which offered two services: reference counting-based lifetime management (through the AddRef and Release methods), and the ability to request other interfaces that the object might offer (through the QueryInterface method). IUnknown is still there, but there’s a new interface which everything in WinRT seems to derive from: IInspectable.

IInspectable derives from IUnknown, as we can see from the native C++ definition (in the SDK’s inspectable.h file, which looks like it was generated from IDL):

MIDL_INTERFACE("AF86E2E0-B12D-4c6a-9C5A-D7AA65101E90")
IInspectable : public IUnknown
{
public:
  virtual HRESULT STDMETHODCALLTYPE GetIids( 
    __RPC__out ULONG *iidCount,
    __RPC__deref_out_ecount_full_opt(*iidCount) IID **iids) = 0;
        
  virtual HRESULT STDMETHODCALLTYPE GetRuntimeClassName( 
    __RPC__deref_out_opt HSTRING *className) = 0;
        
  virtual HRESULT STDMETHODCALLTYPE GetTrustLevel( 
    __RPC__out TrustLevel *trustLevel) = 0;
        
};

So IUnknown is still there—this still looks and feels like COM. But every WinRT interface gets three new features on top of IUnknown’s services. GetIids lets us ask an object for a list of the interfaces it offers, and whatever it returns in this list, it is committed to making available through QueryInterface. (The old-school COM model is that you had to know what to ask the object for. Now you can ask it what it offers.)

You can also ask an object its type—with classic COM there wasn’t a standard ubiquitous way that you could ask any old interface pointer “what kind of object do you point to?” The assumption was that since you wrote the program, you should know what the object is. (And if you don’t know what the object is, what makes you think you can do anything useful with it?) But some of the languages WinRT projects into just assume it’s possible to ask an object its type, so this method makes that possible.

Finally, there’s GetTrustLevel. As the documentation (http://msdn.microsoft.com/library/br205824(v=VS.85) ) helpfully says, this “Gets the trust level of the current Windows Runtime object.” Well that clears things up. I’m assuming this has something to do with the security sandboxing model, but I haven’t yet had time to look at that in any detail.

As it happens, we’re not going to use any of the IInspectable features here. I want to call a method on the WinRT Application class, so I already know precisely which interface I want. I can therefore use the classic COM mechanism for getting hold of it:

Windows::UI::Xaml::IApplication* pApplication;
hr = pInspApp->QueryInterface(__uuidof(pApplication), (void**) &pApplication);
CheckHresult(hr, L"QI for IApplication");

That’s pretty ordinary COM. (Beautiful, isn’t it? *shudder*)

I’ve now got two references to the object—one typed as IInspectable, and one as IApplication. IApplication is a WinRT interface so it derives from IInspectable, making that first reference is superfluous—my pApplication pointer gives me all I could need, so I’ll let the other one go:

pInspApp->Release();

Remember, Release (along with AddRef) is one of the very few COM interface methods that doesn’t return an HRESULT, so we don’t need to check that this worked. It’s not allowed to fail.

So, we have finally written the native equivalent of that first line of code in the high-level projected C++ program. It was a while ago, so in case you’ve forgotten it, here’s the high level entry point code again:

int main(lang::array<Platform::String^>^ args)
{
    auto app = ref new App();

    app->Application::Run();
}

C++ may produce fewer machine code instructions per line of source than the other WinRT languages, but sometimes it achieves this by making you do all the work yourself… (I can hear all the people who used VB back in the 1990s when I was a C++ COM developer saying “I told you so!”)

We’re now ready to move onto the second line—the method call.

Calling Methods

Now that we’re up and running with our object, you may be relieved to hear that invoking methods is pretty straightforward:

hr = pApplication->Run();
CheckHresult(hr, L"Application.Run");

OK, it’s still twice as much code as you’d expect to write in VB or C# thanks to the return code based error handling, but it’s much less effort than it took to get to this point. COM makes fairly light work of invoking methods. Which reminds me…

That’s Not a Vtbl—THIS is a Vtbl

In the ‘big picture’ talks on the first afternoon of //build/ we were shown a C++ Metro app, and a big deal was made of showing the disassembly for a WinRT method call. It looked something like this:

	myTextBlock->SelectAll();
00F55F4F  mov         ecx,dword ptr [ebx+0C4h]  
00F55F55  call        Windows::UI::Xaml::Controls::TextBlock::Windows::UI::Xaml::Controls[::ITextBlock]::SelectAll (0F56160h)  

That’s a call to a TextBlock’s SelectAll method, which they chose because it’s a simple no-parameters method, which makes the resulting compiled code really simple. There are two instructions. The first instruction (mov) loads the implicit ‘this’ argument, passing it via the ECX register (which is one of the numerous ways of passing arguments in assembly language, although most go on the stack). The second instruction (call) is what simple method calls look like in assembly language.

This was proudly announced as a vtbl call, which was offered as evidence of the innate efficiency of C++ compared to some other languages. But as anyone who’s done much C++ COM development will have noticed, that wasn’t true—that’s not what vtbl calls look like. And if you step into that call in the debugger, the claims for efficiency look a whole lot more doubtful, because you end up in a compiler-generated thunk that’s 55 instructions long! (There is a real vtbl call, but it’s buried somewhere in the middle of those 55 instructions.)

The C++ projection for WinRT looks pretty expensive compared to classic COM. Going back to my truly native example, here’s what our native call to the Application object’s Run method looks like in the disassembler:

	hr = pApplication->Run();
00F211E9  mov         eax,dword ptr [esp+14h]  
00F211ED  push        eax  
00F211EE  mov         ecx,dword ptr [eax]  
00F211F0  call        dword ptr [ecx+40h]  

That’s compiled in Release mode by the way—you’ll get more verbose code in Debug. This is about as efficient as a vtbl call gets. The first instruction (mov) is loading the pApplication variable from memory into a CPU register. The second instruction (push) is passing that as the implicit ‘this’ argument—here we’re using the stack, which is how it’s done in COM. (COM doesn’t ever put ‘this’ in ECX, which was one giveaway that the earlier code wasn’t native COM. Everything goes on the stack in COM.) And then the final two instructions are what a vtbl call looks like. COM interface pointers point into a part of the object that contains another pointer, which points to the vtbl for whichever interface you’re working with. The vtbl is an array of function pointers, one for each method in the interface.

The vtbl here is for IApplication. Looking at the function pointers in the vtbl, the methods defined by base interfaces come first, so the first three slots are for IUnknown’s QueryInterface, AddRef, and Release, and the next three are for IInspectable’s methods, so the IApplication methods actually start at slot 6 (or in 0-based counting, at offset 5 which, given 32-bit function pointers, is actually offset 0x16, i.e. 20). The call instruction is looking up the slot at offset 0x40, and since this is a 32-bit process, turning that byte offset back into a slot offset we get 0x10, i.e. 16, which is the interface’s 17th method. So taking out the 6 methods of IUnknown and IInspectable, that means this code is invoking the 11th method in IApplication. And if you go and look at the interface definition for IApplication (which appears to be in Windows.UI.Xaml-coretypes.h), you’ll see that this is indeed the Run method, as you’d expect.

COM vtbl calls always involve this multiple indirection. We always use COM objects via interface pointers. To do anything useful with an interface pointer, we dereference it to get the vtbl. And then we retrieve nth entry in the vtbl. And that points to a method, so we perform an indirected call through that vtbl entry.

Modern CPUs are not very good at dealing with calls through pointers by the way—they like to look ahead, but this sort of indirection prevent them from knowing where they’re going until they get there. So vtbl calls are relatively expensive—much more expensive than, say, an inlined method call such as the CLR might perform. Native C++ can offer some performance benefits over managed code, but it’s not as clear cuts as a lot of people seem to think, particularly with COM in the picture. (Anything crossing a COM boundary defeats a lot of optimizations, something that's not true when using a .NET library.)

Reimagining Success

Incidentally, this call to Run fails. I know I’m right back in the world of COM because I get a return HRESULT of E_FAIL (0x80004005), which means roughly “Something went wrong, and the developer who wrote this method couldn’t [be bothered to] find a more informative error code.” If you pass that to the Windows API for formatting error messages you just get “Unspecified error”.

At the time of writing this, I have some theories as to why this might have failed, but I’ve not tested them yet. But since the //BUILD/ conference is apparently about reimagining everything, I’m going to reimagine what success means. My goal here was to show some of the very basic operations—creating new objects and invoking methods—done in real native code, and I’ve done that, even though my application does nothing useful as yet. To build something meaningful, I need to get into more complex mechanisms like inheritance, and event handling. But it’s nearly time for breakfast, and I want to get this posted before this morning’s talks, so I’ll leave things broken for now, and will follow up with an application that actually does something in a future post.

[Update (2011/09/20): as mentioned in the update above, this turned out to be a threading model issue. WinRT wants this part of the application initialization to happen from a multithreaded context, rather than a singlethreaded one. Also, I put back a missing p tag that had caused a paragraph to vanish in the original post.]

Obviously, nobody in their right mind is going to write applications this way. I just like to know what’s really going on under the covers, so I think it’s interesting to explore the details at this level—if you want to stand a chance of realizing any of the benefits that native code hypothetically offers, I think it’s important to understand this level of detail even if you ultimately choose to work at a higher level. If you care about performance, you need to know what the language projections are really doing for you.

I recently recorded a podcast with Jesse Liberty, in which we talked about various C# things, and in particular, about the new async features Microsoft recently previewed for C# 5.

I recently had an email from a Pluralsight On-Demand! customer about a problem with WPF’s RadioButton. (Yes, the italics and the exclamation mark are part of the brand, apparently…) I thought I’d post my response here, because it might be useful to a wider audience.

Here’s the problem: if you use the GroupName property on some radio buttons in a user control, and you use that user control more than once in the same window, the radio buttons form a single group across the whole window—they are not scoped to the user control as you might expect. For example, suppose your user control contains this rather contrived Xaml:

...
<GroupBox Header="Normal GroupName">
    <StackPanel>
        <RadioButton GroupName="MyGroup" Content="One" IsChecked="True" />
        <RadioButton GroupName="MyGroup" Content="Two" />
        <RadioButton GroupName="MyGroup" Content="Three" />
    </StackPanel>
</GroupBox>
...

(It’s contrived because in practice, you probably wouldn’t set a GroupName at all here. I’m just showing it as a simple example that demonstrates the issue.) If you try to use this user control several times in the same window, e.g.:

<StackPanel>
    <my:WithRadios />
    <my:WithRadios />
    <my:WithRadios />
</StackPanel>

you will find that the radio buttons don’t group by user control. They form one group that spans all the user controls. So you’ll only be able to select one item at a time, even though the layout implies three separate groups:

From WPF’s perspective, this behaviour is by design—the whole point of GroupName is to allow you to define radio buttons that act as a group without having to be contained by the same parent panel. But from our application’s perspective, it’s not likely to be what we want—if a user control contains some radio buttons, the grouping should probably remain within the user control.

In this particular example, it’s easily solved: don’t specify a GroupName. The buttons are all children of the same StackPanel, so they will automatically form a group that’s independent of any other radio button groups. However, that won’t always work in real examples. The GroupName property exists for a reason: sometimes it’s useful to split logically related radio buttons across multiple panels for layout purposes. If we want to do that inside a user control, we’ll have a problem if we try to use multiple instances of that user control inside a single window.

This is a somewhat obscure combination of circumstances—I’ve been working with WPF since its first preview went public around 7 years ago, and I’ve not run into this myself, nor do I recall ever having been asked about it before. However, it seems like a reasonable thing to want to do, unusual though it may be.

Unfortunately, there’s nothing built into WPF to help. Ideally, WPF would provide something like an attachable RadioButton.IsGroupNameScope property that we could set to limit the scope of our names, just like we can with grid size groups. Unfortunately, group names are always scoped to the containing root visual, which is usually your window, and there’s no way to change that. So we need to build some extra stuff to make this work. I’ll start by showing you how the finished result looks in use:

<GroupBox Header="Unique GroupName" Grid.Column="2">
    <StackPanel nn:LocalName.BaseName="{nn:UniqueName}">
        <RadioButton GroupName="{nn:LocalName}"
                     Content="One" IsChecked="True" />
        <RadioButton GroupName="{nn:LocalName}"
                     Content="Two" />
        <RadioButton GroupName="{nn:LocalName}"
                     Content="Three" />

        <RadioButton GroupName="{nn:LocalName Letter}"
                     Content="A" IsChecked="True" />
        <RadioButton GroupName="{nn:LocalName Letter}"
                     Content="B" />

        <RadioButton GroupName="{nn:LocalName Colour}"
                     Content="Red" IsChecked="True" />
        <RadioButton GroupName="{nn:LocalName Colour}"
                     Content="Green" />
        <RadioButton GroupName="{nn:LocalName Colour}"
                     Content="Blue" />
    </StackPanel>
</GroupBox>

That actually defines three different name groups, so this single StackPanel contains three distinct sets of radio buttons.

This example relies on some code that defines some custom features I’ve added to my Xaml. (The nn: prefix in front of all of them is just an XML namespace prefix mapped to my example project’s default namespace by the way.) The first feature is an attachable, inherited property called LocalName.BaseName:

<StackPanel nn:LocalName.BaseName="{nn:UniqueName}">

This property holds a text string that can be used as a name, or as the basis for a name. The property doesn’t do anything itself—its value will be picked up by elements inside the panel, as we’ll see shortly.

This same line of Xaml also uses the second feature, a custom markup extension called UniqueName. This generates a unique string at runtime. (It relies on the Guid type.) The string is generated in the extension’s constructor, so every instance of this extension will return a different string. If you use the extension multiple times in a single Xaml file, each usage will return a different name. But more importantly, a single use of this markup extension will return a different name each time the Xaml is loaded. Since each instance of a user control loads its own copy of its Xaml, that means that the value returned by this UniqueName markup extension will be different for each instance of any user control that uses it.

That solves the problem of getting a different group name in each user control. However, we still need to use the same name across multiple radio buttons within the control—if we just use this UniqueName markup extension on every radio button, each button will end up in its own group, which would be useless.

To deal with that, the final feature is a custom markup extension that uses the inherited LocalName.BaseName. You can see it here:

<RadioButton GroupName="{nn:LocalName}" Content="Two" />

This sets the radio button’s GroupName property to be whatever the value of LocalName.BaseName is on this element. And since my custom attachable property is inherited, the value will come from the containing StackPanel in this case. Or you could put it higher up—you could set LocalName.BaseName on, say, the Grid at the root of your user control’s content, and then any time you use the LocalName markup extension anywhere in the control, it’ll pick up that name.

This gives us what we need: we get a common name within our user control, meaning radio buttons function as a group within the user control, but the name will be different for each instance of that user control. So the upshot is that each user control gets its own distinct group.

This markup extension supports an optional argument: a string that is simply appended to the BaseName. (That’s why I called it base name—it’s not necessarily the name that actually gets used.) Here’s a radio button that uses the argument:

<RadioButton GroupName="{nn:LocalName Letter}" Content="B" />

For example, I ran the program just now, and looking in the debugger, on a particular instance of the user control the base name ended up as “a6b9345b660a4f248b5b5232a48ab653” and so we’d get a name of “a6b9345b660a4f248b5b5232a48ab653Letter” on this particular radio button. This is how the example above gets three distinct groups of radio buttons.

The code for all three features is pretty simple. Here’s the attachable property:

public class LocalName
{
    public static string GetBaseName(FrameworkElement obj)
    {
        return (string) obj.GetValue(BaseNameProperty);
    }
 
    public static void SetBaseName(FrameworkElement obj, string value)
    {
        obj.SetValue(BaseNameProperty, value);
    }
 
    public static readonly DependencyProperty BaseNameProperty =
        DependencyProperty.RegisterAttached("BaseName", typeof(string),
        typeof(LocalName),
        new FrameworkPropertyMetadata(null,
            FrameworkPropertyMetadataOptions.Inherits));
}

That’s about as simple as a dependency property gets—a pair of accessors, and the property registration. The only thing I’ve changed here from the code that Visual Studio’s propa snippet spits out is that I’ve turned on property inheritance.

Here’s the markup extension that generates a unique name:

public class UniqueNameExtension : MarkupExtension
{
    private string _name;
    public UniqueNameExtension()
    {
        _name = Guid.NewGuid().ToString("N");
    }
 
    public override object ProvideValue(IServiceProvider serviceProvider)
    {
        return _name;
    }
}

Finally, there’s the LocalName markup extension. Remember, this uses the inherited value of the LocalName.BaseName property, optionally appending an extra bit of text. It looks like this:

public class LocalNameExtension : MarkupExtension
{
    private string _qualifier;
 
    public LocalNameExtension()
    {
    }
    public LocalNameExtension(string qualifier)
    {
        _qualifier = qualifier;
    }
 
    public override object ProvideValue(IServiceProvider serviceProvider)
    {
        var targetProvider = (IProvideValueTarget)
            serviceProvider.GetService(typeof(IProvideValueTarget));
        var target = (FrameworkElement) targetProvider.TargetObject;
 
        string name = LocalName.GetBaseName(target);
        if (_qualifier != null) { name += _qualifier; }
        return name;
    }
}

The empty default constructor isn’t redundant. It’s there because if you provide a non-default constructor, C# stops generating the default, empty one it usually provides. (I mention this because I frequently encounter developers who don’t know this, and experience has taught me that people like to email me corrections for things in my blog that aren’t actually wrong. This feels like the sort of thing I’d get emails for, so I’m just trying to pre-empt that.)

You can download an example project from here: RadioGroupScope.zip.

There may be simpler ways to achieve this by the way—this is a solution I just made up to answer a question that a customer asked me. (You could solve this by implementing radio button exclusion in the viewmodel. That might not actually be simpler, though.) Three custom Xaml features may seem like an extravagance, but in the end, the individual parts seemed simpler this way than any of the solutions I came up with that used fewer parts. But please email me if you have suggestions for improvements.

At the PDC 2010 conference, Microsoft showed their proposed asynchronous features for C# 5. There’s one particular feature I want to examine: error handling. Anders Hejlsberg showed this, but he did so fairly quickly. The feature is so elegant that it’s easy to overlook what it does for you, so I thought it would be worth a closer look.

The broad theme of the new async features is that you can write code that looks just like normal code, but which works asynchronously. For example:

private async void FetchData()
{
    using (var wc = new WebClient())
    {
        try
        {
            string content = await wc.DownloadStringTaskAsync(
                new Uri("http://www.interact-sw.co.uk/oops/"));
 
            textBox1.Text = content;
        }
        catch (WebException x)
        {
            textBox1.Text = "Error: " + x.Message;
        }
    }
}

Most of this seems fairly ordinary—we’re using the WebClient class to download some text via HTTP, putting the result in a TextBox. (I tested this out in a WPF application, but it would also work in Windows Forms.) We have some exception handling, most obviously the catch block to handle a WebException. There’s no web page with the specified URL, so the server will respond with a 404 HTTP error, meaning we’ll definitely get an exception. But that’s not the only exception-related code: more subtly, there’s a using block to ensure that we Dispose the WebClient even in the face of exceptions. (As far as I can tell, WebClient doesn’t actually do anything in its Dispose, and only implements IDisposable as a result of deriving from Component. But in the absence of any documentation telling us that disposal is optional, we should always Dispose, hence the using block.)

That’s simple enough. But what’s not obvious is that this code works asynchronously. This FetchData method won’t make callers wait for the download to complete. To prove it, the button click handler that calls this method in my test application looks like this:

private void button1_Click(object sender, RoutedEventArgs e)
{
    Debug.WriteLine("Calling FetchData");
    FetchData();
    Debug.WriteLine("FetchData returned");
}

Single-stepping through the code in the debugger, we get slightly surprising behaviour: as we step through FetchData, we get as far as the call to DownloadStringTaskAsync, but then FetchData returns, and we find ourselves back in the click handler—the second Debug.WriteLine runs, showing a message in the debug output window confirming that FetchData has completed, and yet single stepping revealed that it only got half way through.

There are three reasons for this. The most important reason FetchData didn’t run to completion is that it hasn’t finished fetching the data. But there are two more details that explain what it did instead. First, notice that the FetchData method declaration has the async keyword in it. This enables the new C# 5 asynchronous functionality—it doesn’t do anything in itself, but it needs to be present for us to use the new language features, such as the await keyword that appears before the call to DownloadStringTaskAsync. And that keyword is the final piece that causes this code to execute in the unusual way we saw, returning before it had really finished.

The await keyword tells the C# compiler that we need some operation to complete before the code that follows can continue. Of course, with normal synchronous methods, this isn’t something you need to ask for—it’s the default. Normally, a method will not return until it has finished its work, so any code that follows a call to that method will be made to wait. However, with asynchronous operations, it’s different—we typically make a method call to start some work, and the method returns without waiting for that work to complete.

In the example above, the WebClient.DownloadStringTaskAsync method is asynchronous—it returns without waiting for the download operation to complete. It returns a Task<string> object, a class from the Task Parallel Library (TPL) added in .NET 4. A Task<T> represents an operation which may take some time to complete, and which, if it eventually succeeds, will produce a result of type T.

The new await keyword is designed to be used with exactly this sort of asynchronous method—it lets us tell the compiler: this method is going to return more or less immediately, but we need to wait until the work it started has finished before running the remaining code.

Normally, after a call to WebClient.DownloadStringTaskAsync (or any other method) returns, you’d expect the program to continue straight on to the next line—if you wanted to wait for the task to complete before proceeding, you’d need to write extra code. You could call the task’s Wait method, but that would block, and since we’re running on a UI thread here, that would be unacceptable.

This new C# keyword, await, does something different. It doesn’t proceed to the next line, but it doesn’t block either—as I already said, our FetchData method just returns. Actually, to be a little more precise, the C# compiler generates code that checks whether the task has already completed—asynchronous APIs sometimes complete certain requests immediately. If the task is already complete by the time the method returns, the program will carry on and execute the next line as normal. But if not, it returns, but it does one more thing before returning: it sets up a callback that will be invoked once the asynchronous operation finally completes. The code that follows the await statement will only run when this callback is invoked.

To summarize: the rest of the method executes eventually, but it only does so once the operation we’re waiting for completes.

To appreciate what the compiler has done for us, let’s see what it takes to do it ourselves. Looking at just the code inside the try block for now, it’s as though the compiler has expanded that out to:

var wc = new WebClient();
Task<string> contentTask = wc.DownloadStringTaskAsync(
    new Uri("http://www.interact-sw.co.uk/oops/"));
 
Action completionHandler = delegate
{
    textBox1.Text = contentTask.Result;
};
if (contentTask.IsCompleted)
{
    completionHandler();
}
else
{
    contentTask.ContinueWith(t => completionHandler(),
        TaskScheduler.FromCurrentSynchronizationContext());
}

Messy, huh? In fact I’ve simplified this a little—this isn’t what the C# compiler really generates. I’m sticking with the usual idioms for Task<T> because it illustrates the intent more clearly. But even this simplified version is pretty complex. A fair amount of the complication is down to the code that detects when the supposedly asynchronous operation actually completed synchronously. In situations where immediate completion is unlikely, we could just do this:

var wc = new WebClient();
Task<string> contentTask = wc.DownloadStringTaskAsync(
    new Uri("http://www.interact-sw.co.uk/oops/"));
 
contentTask.ContinueWith(t =>
    {
        textBox1.Text = contentTask.Result;
    },
    TaskScheduler.FromCurrentSynchronizationContext());

That’s safe even if the task does happen to complete immediately—Task<T> is smart enough that if you call its ContinueWith method after the task has completed, it’ll still run the completion code. But it’s still significantly messier than the first example—I’ve now had to add an explicit anonymous callback method, and I’ve also had to tell the TPL that I want that handler called via the current synchronization context. (I need this because the callback updates a TextBox, so it needs to run on the UI thread.)

So far, I’ve only really dealt with three lines of the original code. I’ve left out something very important: error handling.

The TPL helps us out with errors—Task<T> keeps hold of any exceptions that occurred while the operation was executing. Its exception handling is pretty good—even if you need to do complex, multi-step operations asynchronously, it can handle errors gracefully. But despite the amount of work the TPL does for us, back in the world of C# 4 we still need to make some changes to our code to exploit this:

contentTask.ContinueWith(t =>
    {
        wc.Dispose();
        if (t.IsFaulted)
        {
            textBox1.Text = "Error: " +
                t.Exception.InnerExceptions[0].Message;
        }
        else
        {
            textBox1.Text = contentTask.Result;
        }
    },
    TaskScheduler.FromCurrentSynchronizationContext());

There are a couple of unsatisfactory things about this. First, it no longer looks like normal idiomatic C# error handling—I’ve lost my try and catch blocks. Second, I’ve had to add an explicit call to Dispose here to clean up the WebClient—I can no longer rely on a using statement. Both problems are caused by the fact that I had to split up my code to accommodate the callback required for asynchronous operation.

By using an anonymous method, I have at least managed to write code that runs in the order in which it reads, but there’s a lot of boilerplate in there now, making it more effort to write, and more importantly, harder to read. And that’s the sort of code we have to write today with C#/.NET 4 if we want to perform asynchronous work with proper error handling. But with its new asynchronous features, C# 5 will let us write the code I showed in the original example, using ordinary-looking exception handling, and it will all just work.

Here’s what I see in Visual Studio when I run the original example on a machine with the async CTP installed (and with VS configured to stop even when handled exceptions are thrown):

If we weren’t paying close attention in this debugging session, this would look like normal synchronous code that happened to throw an exception, but if we look closer, we can see that it’s stranger than that. Earlier, we saw that the FetchData method returned without waiting for the web operation to complete. A quick look at the Output panel in Visual Studio confirms this—I can see the “FetchData returned” message—and yet we’re somehow back in FetchData again.

The web operation happens asynchronously, so it only discovers problems some time after kicking off the operation. So it makes sense that the exception gets thrown some time after FetchData returns to its caller. The weird thing is that the exception was somehow thrown back into FetchData, even though that method has already returned.

This works because the C# compiler has split my code up to enable asynchronous execution, and that’s why I can find myself back inside a method that I only invoked once and which has already returned. And this reveals that the compiler has performed more complex surgery than the manual version I wrote earlier—my version just looked for the exception in the Task<string>, but here, the exception has been rethrown back into my code. (I’m not going to go into the full details, but if you’re interested, there’s a helper class in the System.Runtime.CompilerServices namespace called TaskAwaiter, and the exception is rethrown by its EndAwait method in this particular example.)

We can get a bit more insight into the exception handling by looking at the IntelliTrace panel. (This requires a sufficiently empowered copy of Visual Studio 2010—I think it’s only available on the Ultimate edition.) IntelliTrace shows us a log of interesting stuff that has happened in the debugging session:

I’ve cropped this to show just the events relevant to this discussion. You can see that it began with a button being clicked—IntelliTrace automatically adds entries for various user input events. And you can see that the last entry is the exception currently being debugged—the same one as in the earlier screenshot. But you can see three more Exception entries: a throw, a catch, and another throw. Let’s look at the first one:

This is the original error—the point at which the exception was first thrown. Notice that the thread ID here is different. In the previous screenshot we were on thread 1724, which happens to be the UI thread in this debugging session, but in this first trace entry, the thread ID was 3764. This shows that we weren’t on the UI thread when things went wrong.

And that’s what you’d expect—under the covers WebClient depends the HttpWebRequest class, which in turn depends on sockets, which use thread pool threads for asynchronous completion notifications. So the HTTP 404 status is first detected on a thread pool thread, and that’s where the WebException is first thrown. But as you can see in the IntelliTrace log, it’s caught, and then rethrown back on the UI thread. That happens because we used the DownloadStringTaskAsync method—it makes the exception available through the Task<string> object, and the code generated by the C# compiler for the await keyword calls a helper method that detects that the task is in a ‘faulted’ state. This helper retrieves the exception and rethrows it.

The full details of this are all quite involved. The transformation that the compiler applies to your code is non-trivial. There are further complexities caused by the fact that the await keyword knows nothing about Task<T>—it goes through extension methods in this example. (The C# compiler requires the await argument to provide a helper method, called GetAwaiter, which must return an object that providers two more helpers, BeginAwait, and EndAwait. It doesn’t care whether these are instance methods or extension methods, but in this case, the class library provides extension methods that add these to Task<T>.) This makes the whole asynchronous mechanism extensible—you can use these language features in conjunction with your own classes if, for some reason, you don’t want to use the TPL. See Jon Skeet’s blog on this for more details.

But the upshot is that exception handling (not just try/catch blocks, but also using statements) work in a way that seems perfectly straightforward, despite the hoops the compiler has to jump through to make them work in asynchronous scenarios. If you’ve written much code to work with asynchronous services, and to handle errors that emerge from them, you can see how these new C# 5 language features make consuming those services far easier. I can’t wait to be able to use them.

I’ve just been watching the recording of the excellent language panel talk from the PDC 2010 conference. The player uses Silverlight Smooth Streaming to adapt the video quality to the available bandwidth. The source material was all in high definition, and in situations where you can get a sustained 3Mb between Microsoft’s servers and your computer, you get a rather good image. But if you have less bandwidth, you’re not out of luck—you still get to watch the stream. Smooth Streaming can drop down to lower quality video to adapt to slower bitrates.

Now I’ve noticed that not everyone has the same obsession with detail in video and graphics as me, so trying to demonstrate smooth streaming in action can be frustrating. There will be a massively obvious change in quality when it switches bitrate that I would have thought anyone watching couldn’t possibly miss, but it turns out that most people find it hard to tell the difference. So I was delighted by Anders Hejlsberg’s choice of shirt for this panel. He made it far easier to see the changes in bitrate.

Here is his shirt, in glorious HD. (If this doesn’t look big enough to you to be an HD image, bear in mind that I’ve cropped that out of a wide shot that was showing all four panel members.)

Figure 1: Anders tries to decide just how much he likes his shirt

The video player frequently decided that there was no possible way a 3Mb stream could ever fit down my 8Mb internet connection. (Congestion elsewhere in the network I assume—I don’t usually have trouble getting all 8Mb when necessary, thanks to a 5:1 contention ratio.) The drop in resolution brought with it an interesting visual artefact: Anders’ shirt suddenly turned into a mass of stripes:

Figure 2: Anders is amused by his shirt suddenly going all stripy

This is aliasing in action, by the way. (It’s the same fundamental phenomenon that antialiasing tries to eliminate in fonts.) Aliasing occurs whenever you have some sort of discrete sampling process (e.g., taking a photo with a digital camera or converting font outlines into pixels) in which the sampling resolution is too low for the level of detail in the source. It so happens that with any regular sampling system (e.g. a grid of pixels), if you try to represent a repeating pattern whose scale is too close to the sampling pattern, it ends up looking like a much coarser pattern.

Here, switching to this medium resolution has created a situation where the spacing of the pattern on Anders’ shirt is too close to the spacing of the pixels. The limitations of the sampling resolution end up causing what should be fine detail to show up as a coarser pattern—the horizontal stripes you see here. In signal processing terms, we’d call that an alias—a low-frequency manifestation of a high-frequency component in the original signal.

This looks to me like a shortcoming in the encoding process—whenever you down-scale an image, you should always run it through a low-pass filter first. This removes the detail that you’re unable to represent, in order to stop that detail causing low-frequency aliases. In other words, the original high-definition image should have had the detail in the shirt blurred away before being scaled down, because that would have prevented those stripes from appearing.

Sometimes you can’t avoid it—it might be that the aliasing occurs at the point of acquisition. (E.g., if your camera is perfectly in focus, pointing at an object with too much detail—at that point, the stripes would be there in the original image acquired by the camera. The only way to avoid that is to deliberately defocus the camera slightly, which sort of acts like a low-pass filter.)

But in this case, there’s no aliasing on the original high-definition picture, so that aliasing has crept in as part of the downscaling. That’s usually entirely avoidable, so it’s a bit disappointing that it has happened, although it’s convenient if your goal is to show when things switch between different quality levels.

I was curious to see how the lower bitrates looked, so I deliberately started overloading my internet connection with other work to push Smooth Streaming down to some lower bitrates. (Incidentally, this experiment verified that I definitely had all 8Mb of bandwidth available to me, so the streaming bottleneck definitely wasn’t at my end.) Here’s the next notch down:

Figure 3: Anders is saddened by the increasingly fuzzy image

It’s noticeably more pixelated in the high contrast boundaries, e.g. the top of his head, the edges of his trousers. The aliasing is pretty much the same as in the previous image.

And here’s the lowest bitrate available:

Figure 4: Anders cheers up when he realises that the decreased resolution means many more people will be able to see the video

The aliasing hasn’t got any worse at lower resolutions, but it hasn’t gone away either. You’d typically expect it to look different for different scales, but it’s difficult to compare images here—the camera was at different levels of zoom, and Anders was leaning at different angles in each shot, so it’s hard to draw any further conclusions from these last two images. Anyway, if you’ve ever been watching a smooth streaming video source, and have noticed an ordinary looking pattern suddenly go stripy, this is why.

One of the many things I do for a living is producing content for Pluralsight’s online .NET training product, Pluralsight On-Demand! I’ve always been keen to try out techniques that go beyond the basic format of slides and demos with a voiceover. I had a lot of fun putting together one video in particular: the printing module in our Silverlight material. So I’m excited that it’s temporarily available for free—up until 26th September 2010, you can watch the Silverlight Printing module here without needing a subscription.

Printing is often a frustrating topic to talk about in person, because more often than not, I’ll teach it in a classroom where there’s no printer. The moment that should be the denouement—getting to see the content emerge from the printer—ends up being a bit of an anti-climax. I have to tell the class “and at this point, something would emerge from the printer, if we actually had one.” Obviously, you can print to XPS or PDF, but it’s just not the same.

So I incorporated some video camera footage to avoid this problem in the online course. It doesn’t make a whole lot of difference technically, but I think it makes it more satisfying to watch.

But the printing module also includes a sequence that goes a bit further, providing a clear technical benefit, rather than merely making it a bit more fun to watch. It uses a video technique I’ve not tried before to demonstrate an issue with the bitmap-based printing Silverlight uses. I could describe what I’ve done, but it’ll be easer just to watch it. (It’s in the “Print Quality” segment, if you want to go straight there without watching the whole thing. It’s approximately three and a half minutes in if you just want to see the bit I’m talking about, but it’ll probably make more sense if you watch that whole section.)

Interestingly, this particular topic was actually easier to demonstrate in a video than it is in person.

It’s always pleasing when the medium enhances something. If you try to use one medium as though it were really another—like how some early television was more or less just radio with a moving picture of a person speaking into the microphone—you often end up fighting the medium, which compromises the content. But if you adapt how you work to fit the medium, it can make things better. I hope I’ve achieved that with this particular demo, but it’s something that I think we’ve already been doing in other ways for a while.

If you’re a Pluralsight On-Demand! subscriber (yes, the italics and exclamation mark are officially part of the brand…) you’ll have noticed that we have brought a lot of new content online recently. You may also have noticed that it’s much less closely related to the classroom courses than some of the material that’s been around for longer. Rather than simply treating online content as an alternative delivery mechanism, we’ve been producing new material designed specifically for this system. For example, we wanted online content to be easy to search and browse, and we found that an important way to achieve this was to structure the content with a rather different granularity—we have more and smaller modules than a typical classroom class.

I’ve been enjoying learning how to do things differently for this medium. I’m looking forward to producing more online courses.

This is the final part of a series of six blogs:

• Simple but Too Specialized
• Moving Away from Anonymous Types
• LINQ to LINQ
• The Missing LINQ: IEnumerable<T> and Scalars
• Other Languages
• Closing Thoughts (this entry)

Part 6 – Closing Thoughts

Back at the start of this series, I started with a simple, specialized Cartesian product where I knew the number of sets in advance. Anonymous types made this easy, but I had to move away from those to get a more general-purpose solution. This seems to happen quite a lot with anonymous types—they’re handy for very well-contained scenarios, but I often find myself getting rid of them as my code grows more complicated.

I think this is an example of a broader issue. LINQ works really well in monolithic applications—if you can get everything you need in a single method, or better yet a single query, it does a great deal of very useful work for you. But as soon as you want to start splitting things up, life gets harder. You might want to reuse parts of a query, or perhaps you just want to decompose your code into smaller chunks to improve readability, testability, and maintainability. Either way, dismantling a LINQ query into its component parts is possible, but rarely straightforward. I saw a similar step up in complexity here—I started with something that was easy, using small, specialized LINQ queries, but generalizing it was a lot more work.

However, although the general solution was significantly less convenient than the specialized one, it was still reasonably compact. I had been expecting to see a bigger difference between the C# code and the examples from geekier languages such as Clojure, F# and Haskell. But the results are surprisingly similar, I think. The biggest difference I can see is that Haskell’s type system makes it easier to define functions capable of operating over any monadic type—the examples for all other languages only work on lists.

The next biggest difference is that the C# code has more explicit typing—I’ve had to say a lot about the argument and return types of the CartesianProduct method. You’re allowed to state this stuff explicitly in Haskell if you want, but you’re not required to. This brings me onto my final point: ceremony.

An Unfashionable Defence of Ceremony

It has recently become fashionable to attack some languages for requiring more “ceremony” than others. Apparently, “ceremony” is anything you have to write because it's formally required, but which doesn’t directly say anything about what your program does. The need for explicit type declarations in C# is one example. With Haskell I was able to write just the essence, whereas with C#, I had to say more about what I was doing before I could do it. (Haskell is very much a statically typed language, by the way. It’s just that you can leave its type inference system to do a lot of the work.)

I hold an unfashionable view: I’m not convinced that absence of ceremony is intrinsically a good thing—in this particular case I suspect it makes it easier for someone reading the code. I don’t want to have to perform complicated type inference in my head when reading source code—I’d rather the code was explicit about the types it’s using because a) that saves time when I’m reading it and b) removes doubt over whether the author wrote what he or she meant to write.

In what little Haskell I’ve done, I’ve found it much easier to write explicit type declarations even in cases where the compiler doesn’t need them. The ceremony’s not for the compiler—it’s for my benefit. Just to be clear: I don't think that ceremony is intrinsically good either. Pointless verbosity helps nobody. I’m just saying that sometimes, making things explicit can improve readability.

This is part five in a series of blogs:

• Simple but Too Specialized
• Moving Away from Anonymous Types
• LINQ to LINQ
• The Missing LINQ: IEnumerable<T> and Scalars
• Other Languages (this entry)
• Closing Thoughts

Part 5 – Other Languages

In the previous blogs in this series, I’ve shown how to generate Cartesian products in C#. I started with the simple, elegant, anonymous type solution that works when you know at compile time exactly what your inputs are. But because my original scenario involved varying inputs, I built a more generalized solution, and then honed it to the point where I was reasonably happy with its size and readability.

In this part, I thought it’d be interesting to show solutions in a handful of other languages. In language advocacy fist fights^H^H^H^H^H^H^H^H^H^H^H debates, it’s common to argue that a language’s power is inversely proportional to the amount of code required to express a particular concept. In other words less is more—the language that can solve a particular problem in the smallest space wins. (I don’t completely agree with that—I think readability matters, and terseness is sometimes the enemy of readability. But size has the benefit of being objective.)

Haskell offers the shortest solution I’ve seen to this problem:

sequence [['1', '2'], ['A', 'B']]

This produces:

["1A","1B","2A","2B"]

That initially struck me as a little wierd until I realised that a string in Haskell is just a list of characters. I think it makes slightly more sense to think of that output as this:

[['1','A'],['1','B'],['2', 'A'],['2','B']]

That makes it more obvious that sequence has just produced every possible concatenation. But Haskell prefers to print lists of characters using string literal syntax. In fact if you type in the second representation at a Haskell prompt, it'll print out the first one.

You might think that this is cheating because I appear to be using a built-in function to do the job. But that would be a bad argument on two counts. First, the expressiveness of the available library functions is a critically important aspect of the practical usefulness of any language. Second, this built-in sequence function is not just for Cartesian products—it’s more general-purpose. What you’re looking at in that example is a very general mechanism being applied in this particular case to produce an n-ary Cartesian product.

The sequence function works with any monadic type. For example, I can use Haskell’s Maybe monad, which is roughly analogous to the idea of a nullable type:

sequence [Just 1, Just 2, Just 3]

Here I’m passing a list of maybes where every item does in fact have a value, and it produces:

Just [1,2,3]

But if I include Nothing, which is (sort of) analogous to null, then the usual behaviour of the Maybe monad kicks in, which is to say I get Nothing out. So this:

sequence [Just 1, Just 2, Nothing, Just 3]

produces this:

Nothing

Haskell also uses monads to handle IO, so I could use sequence to execute a series of IO actions:

get2chars = sequence [getChar, getChar]

The getChar function returns an IO Char, so passing a list of these into sequence produces something of type IO [Char], i.e., it produces an IO monad which can feed a list of Char values into a subsequent step:

do { result <- ioseq; putStrLn result }

That reads two characters, and then prints them to the output. Not a particularly useful example perhaps, but it illustrates that the way in which the sequence function combines the elements is defined not by sequence itself, but by the monad you’re using.

If you’re not entirely familiar with monads, and want to understand this in terms more familiar to a C# developer, you could think of Haskell’s sequence function as being the generalized version of the CartesianProduct function I wrote earlier, but rather than being specific to IEnumerable<T>:

public static IEnumerable<IEnumerable<T>> CartesianProduct<T>(
    IEnumerable<IEnumerable<T>> inputs)

it works for any type that provides the LINQ SelectMany operator. (So a hypothetical C# equivalent of sequence would work against LINQ to XML or LINQ to Entities, and not just LINQ to Objects, which is what our current CartesianProduct implementation uses.) SelectMany is just LINQ’s name for one of the standard monadic operators. (It’s sometimes called “bind”, and Haskell denotes it with the >>= operator.) So you could imagine this sort of thing:

public static IEnumerable<IMonad<T>> Sequence<T>(IEnumerable<IMonad<T>> inputs)

That’s not real because there is no IMonad<T> in .NET—I’m just trying to illustrate the broad idea behind Haskell’s sequence function. So the reason we can use sequence to build an n-ary Cartesian product is the same reason we can build them with C# LINQ queries with multiple from clauses—both C# and Haskell represent lists as monadic types whose binding behaviour happens to be a Cartesian product. C# doesn’t explicitly call these things monads, but it’s the same idea in practice.

(Note that sequence has one limitation that my C# code doesn't: Haskell has no direct equivalent of System.Object. The C# method can combine any old mixture of types, whereas Haskell's a bit more picky. As it happens, I don't care for my original problem: I only needed to cope with the number of input sets being unknown at compile time. The set members all happened to be of the same type; the ability to cope with a mixture of types was an accidental bonus, and not even one I wanted to keep. As you will recall, I eventually moved from object to a type argument. But if you really wanted to handle a mixture of types not known until runtime, Haskell's actually at a disadvantage; or perhaps the idiomatic Haskell response would be to wave my hand in the manner of a Jedi and tell you that this is not the type system you're looking for. In practice, I suspect that if you're using Haskell, you'd probably be unlikely to find yourself in a position where you were trying to handle that unknown mixture of types anyway.)

So Haskell wins because it has a built-in function, sequence, which combines a list of monadic types. I had to write my own function to do that in C#. But if Haskell didn’t have sequence, I could build it in much the same way I did with C#. In C# I used the Aggregate operator, and in Haskell I could use the similar foldr method.

mySequence = foldr
  (\ input soFar ->
    soFar >>= (\prevProductItem ->
      input >>= (\item -> [item:prevProductItem])))
  [[]]

Although that works here, it’s a little too specialized—I’m constructing lists with the [] syntax here, and the original sequence function works with any monad. Fortunately, as mentioned earlier, Haskell requires monadic types to define an operation to construct an instance of the monad from a value. (The gap I filled in C# with my EnumerableFrom method in the previous part. And as I noted in a recent update to the previous article, Rx provides something similar through its EnumerableEx.return method.) This is provided by the return operation:

mySequence = foldr
  (\ input soFar ->
    soFar >>= (\prevProductItem ->
      input >>= (\item -> return (item:prevProductItem))))
  (return [])

It might not be obvious if you’re unfamiliar with Haskell, but the structure’s pretty similar to the C# version. The from clauses have turned into >>= operators, the call to Append has been replaced with the colon operator. And in fact that’s a prepend, rather than an append; everything’s turned inside out because foldr works from back to front… (I could have used foldl, but foldr seems to be the more common idiom in Haskell.)

Haskell offers a specialized syntax for chaining monads together: the do notation. The following example is equivalent to the preceding one, but it might be more idiomatically correct:

mySequence = foldr
  (\ input soFar ->
      do prevProductItem <- soFar
         item <- input
         return (item:prevProductItem))
  (return [])

Hopefully this makes the structural similarity between this Haskell example and the C# CartesianProduct slightly more obvious—compare the <- clauses in this do notation with the from clauses in the C#. (Compared to foldr, LINQ’s Aggregate happens to use the reverse order for its arguments, and also for the arguments for the lambda, but otherwise, this is pretty similar.)

This still relies on monadic operators, so just as C# from clauses work with any type that offers the relevant LINQ operators, all the Haskell examples shown so far will work with any monad. And when I feed in a list of lists, it’s the Haskell List monad that’s doing the actual work of producing the cross product here, just as the C# code ultimately relies on the LINQ to Objects code in the Enumerable class. If you prefer your code more explicit, here’s one that generates the products without relying on monads:

explicitCartes = foldr
  (\ input soFar ->
    concatMap (\prevProductItem ->
      concatMap (\item -> [item:prevProductItem]) input)
      soFar)
  [[]]

Here’s a completely different take on the same problem:

compCartes = foldr
  (\ input soFar ->
    [item:prevProductItem | prevProductItem <- soFar, item <- input])
  [[]]

That’s using a list comprehension. Here I’m essentially exploiting the fact that the language will generate a 2-set Cartesian product if I define a list in terms of two input sets, and once again I’m using foldr to generalise that to an n-ary cross product. (This last example only works with lists—unlike the built-in sequence, and the last two mySequence examples above, this won’t work with other monads.)

There are probably other viable approaches. These may not be particularly idiomatically good examples of Haskell by the way—not only do I not know much Haskell, I’ve also attempted to keep some kind of connection between my C# examples and the Haskell code through my choice of variable names, to make it easier to explore the similarities and differences.

Craig Andera helpfully supplied me with a Clojure version:

(defn cross-prod [& colls]
  (->> colls
       (reduce #(for [x %1 y %2] [x y]))
       (map flatten)))

I’m pretty sure that’s doing much the same thing—reduce is Lisp’s name for what Haskell calls foldr, and what LINQ calls Aggregate.

I also took a shot at an F# version. I don’t really know much F#, but F# MVP Richard Minerich (aka rickasaurus) generously looked at my example, and was kind enough to describe it as “quite good”. He also came up with something more clever, but it would actually have required more code, so I'm sticking with my original:

let cartes inputs =
  List.foldBack
    (fun input soFar -> seq { for y in soFar do for x in input do yield x::y })
    inputs
    (Seq.singleton List.Empty)

let result = cartes [[1..5];[42;50];[99;100]] |> Seq.toArray

The List.foldBack operation is similar to Haskell’s foldr, and again, the overall structure is much the same as the C# example. Don’t be thrown by the name seq—this is different from Haskell’s sequence function. In F#, the seq keyword is used to introduce a block of code that will produce a sequence of items as its output. The block that follows here is conceptually pretty similar to the C# example, but with a pair of for … in … do constructs in place of a pair of from clauses.

(Note that as with the Haskell version, this one’s also picky about types. But again it doesn’t matter to me, because in my original problem, the input sets all contained members of the same type. It was only the number of sets I didn't know at compile time.)

The final line of this F# example is just for test purposes, and the reason I’m feeding the result through the Seq.toArray function is that the debugger shows more useful information for arrays than it does for generic sequences. (I guess that’s because a sequence might be infinite, so attempting to display the whole thing might be undesirable. But in converting it to an array, I’ve already chosen to force the sequence to be fully evaluated, so the debugger can show it safely.) Here’s the output from evaluating that last line in the F# Interactive window:

val result : int list array =
  [|[1; 42; 99]; [2; 42; 99]; [3; 42; 99]; [4; 42; 99]; [5; 42; 99];
    [1; 50; 99]; [2; 50; 99]; [3; 50; 99]; [4; 50; 99]; [5; 50; 99];
    [1; 42; 100]; [2; 42; 100]; [3; 42; 100]; [4; 42; 100]; [5; 42; 100];
    [1; 50; 100]; [2; 50; 100]; [3; 50; 100]; [4; 50; 100]; [5; 50; 100]|]

So that seems to work.

Old School

I asked a friend who works mainly in Java for a Java version of this code. I’m not going to offer any commentary beyond observing that a) this is probably roughly what it would have look like in C# before C# 3.0 turned up and b) this is why I like C# 3.0 so much:

import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;


public class Cartesian {
    public static void main(final String... args) {

         final Object[] letters = { 'A', 'B', 'C' };
         final Object[] numbers = { 1, 2, 3, 4 };
         final Object[] colours = { "Red", "Blue" };

         final List<List<Object>> params = new ArrayList<List<Object>>();
         params.add(Arrays.asList(letters));
         params.add(Arrays.asList(numbers));
         params.add(Arrays.asList(colours));

         for( final List<Object> item : cartesian(params)) {
             System.out.println(item);
        }
    }

     public static <T> List<List<T>> cartesian(final List<List<T>> params) {
         if( !(params.size() > 0) ) return new LinkedList<List<T>>();
         return cartesian(new LinkedList<List<T>>(params),
             new ArrayList<List<T>>());
     }

    private static <T> List<List<T>> cartesian(
      final LinkedList<List<T>> params, final List<List<T>> results) {
         if( !(params.size() > 0) ) return results;
         aggregate(params.removeLast(),results);
         return cartesian(params,results);
     }

    private static <T> void aggregate(final List<T> items,
                                      final List<List<T>> target) {
        if( target.size() == 0 ) {
            for(final T item : items) {
                final List<T> targetItems = new ArrayList<T>();
                targetItems.add(item);
                target.add(targetItems);
            }
        } else {
            final List<List<T>> oldTarget = new LinkedList<List<T>>(target);
            target.clear();
            for(final T item : items) {
                for(final List<T> oldTargetItems : oldTarget) {
                    final List<T> targetItems = new ArrayList<T>();
                    targetItems.add(item);
                    targetItems.addAll(oldTargetItems);
                    target.add(targetItems);
                }
            }
        }
    }
}

Next time, some final thoughts.

This is part four in a series of blogs:

• Simple but Too Specialized
• Moving Away from Anonymous Types
• LINQ to LINQ
• The Missing LINQ: IEnumerable<T> and Scalars (this entry)
• Other Languages
• Closing Thoughts

Part 4 – The Missing LINQ: IEnumerable<T> and Scalars

Last time, I turned LINQ on itself: I used a LINQ query to build a LINQ query. This worked, but the result wasn’t quite as neat as it could have been. I can improve matters by adding a couple of helpers to the world of LINQ. I’ve often found it slightly odd that these don’t seem to be built into LINQ, but they’re easily added.

LINQ offers no built in clean way to take a single scalar item and convert it into a sequence with that single item in it. The usual approach is just to build a single-element array. This works, but it's rather clunky, and I often run into situations where this curious omission makes code more awkward than it needs to be. It has led to a rather ugly first argument to Aggregate:

        (IEnumerable<IEnumerable<T>>) new T[][] { new T[0] },

What I’m actually saying here is that I want to take an single element—an empty sequence of T (new T[0])—and turn it into a sequence that contains just that one element. (I’m building a sequence of sequences here, remember.) The Enumerable class does actually provide a helper for building an empty sequence, so I can make this code say what it does slightly better by writing this:

(IEnumerable<IEnumerable<T>>) new IEnumerable<T>[] { Enumerable.Empty<T>() },

I think it’s now slightly more obvious that an empty sequence is involved here. But I’ve ended up building an array to get my one-element sequence, and I think it’s ugly. What really I’d like to write is this:

Enumerable.From(Enumerable.Empty<T>()),

I’m imagining a From method here that returns a single-element sequence containing whatever element you pass as the argument. Sadly, this method doesn’t appear to exist. That seems weird to me—we have the Empty<T> method, so it would seem natural also to have a method that wraps a single item as an enumerable. (Moreover, LINQ draws heavily from the concepts behind monadic types such as those offered in Haskell. One of the fundamental operations of a monad is the ability to construct one from any underlying type. There are only two fundamental operations, so this is a particularly striking omission.) It’s easy enough to write our own though:

private static IEnumerable<T> EnumerableFrom<T>(T item)
{
    return new T[] { item };
}

Other implementations are possible of course—this might be a more expressive idiom:

private static IEnumerable<T> EnumerableFrom<T>(T item)
{
    yield return item;
}

That’s prettier, and appeals much more to my language geek tendencies, but it generates a lot more code under the covers. So I’ll stick with the first one.

A similar problem exists with concatenation. On this line here:

select prevProductItem.Concat(new T[] { item }));

I have an IEnumerable<T> (prevProductItem), and a single value of type T (item), and I want to append that to the original enumeration. The Concat operator is only able to combine two enumerations, so I’m obliged to turn that single item into a sequence by wrapping it in an array. This is essentially another instance of the previous problem, so I could just use my new EnumerableFrom method:

select prevProductItem.Concat(EnumerableFrom(item)));

But that feels a bit heavy handed. I’d love it if there were an overload of Concat that accepted singular values, in which case I could write:

select prevProductItem.Concat(item); // Won't compile

Although that might cause ambiguity, so perhaps it’d be better to have a distinct operator for appending just one item:

select prevProductItem.Append(item)); // Won't compile without help

As far as I can tell, that doesn’t exist. (Although it’s always possible that it does exist under a less obvious name, and I’ve simply missed it. Wouldn’t be the first time…) But I can write an extension method to provide it:

static class AppendExtension
{
    public static IEnumerable<T> Append<T>(this IEnumerable<T> that, T item)
    {
        IEnumerable<T> itemAsSequence = new T[] { item };
        return that.Concat(itemAsSequence);
    }
}

With this and my EnumerableFrom helper method in place, I can simplify our CartesianProduct method a little:

public static IEnumerable<IEnumerable<T>> CartesianProduct<T>(
    IEnumerable<IEnumerable<T>> inputs)
{
    return inputs.Aggregate(
        EnumerableFrom(Enumerable.Empty<T>()),
        (soFar, input) =>
            from prevProductItem in soFar
            from item in input
            select prevProductItem.Append(item));
}

That’s looking a lot calmer to me.

I fear it may be write-only code—I’m not convinced that if I were to come across that finished product 6 months after writing it that I’d be able to work out how it works without first referring back to this blog series… However, because I know what an n-ary Cartesian product is, I’ll find it pretty easy to read code that uses this method.

Next time we’ll look at some solutions to this problem in other programming languages, to see how C# compares.

[Update, 5th August 2010: A few people have emailed me to mention that the Reactive Extensions for .NET (Rx) offer the missing features I add. So if these ever make it out of the labs into the full .NET Framework, I'll get my wish. The monad influence is clear - they've given their version of what I called EnumerableFrom the name return (yes, that's a blank web page - there doesn't seem to be much online documentation, and that's the only reference I could find - you'll need to download Rx to see the actual docs). Haskell also calls it return.]