PhatBoyG.com

Last weekend, Dru Sellers and I (along with 100 or so others) attended the Continuous Improvement in Software Development (referred to as KaizenConf, likely due to the URL) conference in Austin, TX. We left Tulsa late Friday night after taking my girls trick-or-treating for Halloween and drove all night arriving in Austin around 5:30 AM. After a quick nap and a shower, we went to the first day of the conference.

The night before at the opening, those who were there put up a series of topics that they wanted to discuss. One of those talks was on Enterprise Service Bus (ESB) patterns and implementations using MassTransit. I think it was pretty obvious where Dru and I were going to spend our time right after lunch. There were a number of people that we have had conversations with that also attended and they were excited to learn more about distributed application designs and how to implement them using open-source tools like MassTransit.

The First Session: Alternative Architectures

After the morning announcements, Dru and I joined Ben Scheirman and Ayende Rahien in a conversations about alternatives to the RDBMS. The conversation started a bit rough, but quickly opened up into other ways to store data in our applications. Ayende brought up CouchDB and the things he learned about it. There was also some in depth discussions about proper use of databases and separating transactional data from reporting to avoid transaction blocks. Some concepts on how to achieve the appropriate separation, including asynchronous ETL (extract-transform-load) were discussed. The Map/Reduce algorithm was also covered and some examples were given in the discussion of how to map data into dimensions for reporting. The proceedings from this talk can be found on the wiki.

The Second Session: Lean Architecture

My second session of the day was on Lean Architecture. This discussion was forward looking and related to how development teams could branch out and work on features with the intent of releasing individual features as they are complete (instead of waiting for a big release with many unrelated features). There was a lot of talk about how the teams work, how version control and build processes would need to adapt to handle the complexity of dealing with multiple development versions of a single code base. I think one of my biggest take aways from this session was the need for an integration branch that is created from the trunk before each merge. It is then possible to integrate a branch into the trunk without immediately impacting the trunk. Once the simulation branch merge is complete and tested, the simulation is merged into the trunk (which should be easy, since it was created from the trunk originally). This alone would help to ensure a solid trunk that can be delivered on demand. The proceedings from this talk are also on the wiki.

Over lunch we enjoyed some amazing Austin weather while talking about projects and recent events. We then went to setup for the discussion on ESB/MassTransit. Upon arrival, we were surprised at the number of people in attendance. There was a crowd of at least 20 people and the video crew was setup with remote microphones and the works. Once we had gotten the projector setup and demos loaded, we started into the discussion.

The Third Session: ESB Patterns and MassTransit

The conversation started with a general discussion about messaging patterns. A reading list was presented, along with the major patterns that are used in a publish/subscribe system like an ESB. Applications for this type of system including everything from a command/query interactive system to migrating a batch processing application to a more real-time asynchronous process. We touched on where MassTransit was in the development lifecycle and some of the things we learned over the last year of development. We showed some of the sample applications, including the new web service bridge sample for connecting external customer web services to an internal domain.

Needless to say, we were amazed at the response we got related to MT and ESB patterns in general. It was great to have Ayende and Jeremy Miller bouncing ideas around regarding the project and integration with other existing systems. The notes from the discussion are located on the KaizenConf Wiki.

The Last Session: A Mixed Bag

My last session of the day was split between two completely different sessions. One was on Advanced IOC usage, and the other was on moving from Project to Product. I bounced between these two sessions, picking up some interesting bits from each one. Both were late in the day, so I was fading pretty fast considering I only had 90 minutes of sleep the previous night. Good concepts were captured in the notes, so be sure to check those out (I’ll have to just to remember them).

After the sessions for the day, we went to the Hyatt with Ben (B#) and provided input as he started writing a new sample for MassTransit. The context for this new sample was the Gregor Hohpe article about how Starbucks Does Not Use Two-Phase Commit. This was mostly a learning exercise for Ben, however, we gained some insight watching him build it out. I think Dru refactored the host quite a bit to remove some extraneous ceremony that was just unnecessary. Once we had the first bits working, it was time to find some dinner and the rest of the gang. We had some great Tex-Mex at Trudy’s and then went back to the hotel bar to watch Texas Tech put the smack down on U-Texas Longhorns.

The next morning we met up at Starbucks (go figure) to complete the new sample Ben was building (yes, the irony of working on a sample about Starbucks @ Starbucks didn’t get past us). We ended up converting from using regular message consumers to a saga-based approach given the requirements and it took another 30 minutes to get it all working. Oren had some great ideas on how we could tweak little things here and there to eliminate some friction as well. Once we were done, we headed on over to the conference to see what was in store for the morning sessions.

This year, the focus of the conference was on continuous improvement. Therefore, instead of just having additional sessions on Sunday morning the attendees voted on sessions that they wanted to help improve. At least, that’s how I understood it, we were working on the Starbucks sample, remember? Anyway, when we arrived we found that the ESB Patterns session was one of the topics. The room was full with something like 22 votes on the page. I was surprised to see the response of the people, truly surprised.

Continuous Improvement: ESB Patterns and MassTransit

The actions are summarized on the wiki, and included things like documentation, more sample applications, and help on how to add messaging to an existing application. We also dug into the things needed to help people build and understanding messaging components in their applications. Things like diagnostics, conventions to avoid common missteps, hands-on labs to walk through building a message-based component, and a few other getting started items were brought up. From a technical side, we talked about additional patterns such as an overall conversation id to correlate multiple sagas (like a saga of sagas), as well as built-in support for compensation.

A large part of the discussion was on how functional programming features could be used to enhance the system. An example in Erlang of a message exchange was drawn up, along with how it might be written in C#. Matthew Podwysocki was consulted on how this example might be written in F#, and there was discussion about how the threading model of F# doesn’t really support the style used by Erlang. Some work is still needed there I suppose but we’ll continue to be open about the possibilities offered by functional languages. Glenn Block also talked about getting in touch with the connected systems group to see what they could offer.

County Line BBQ

With that, the conference was closed. Once again, the open space technology used for the event was awesome. The rules of open space dictate that what happened is the only thing that could have happened, and I agree with that completely. It was great to be a part of yet another excellent event and I look forward to future installments in other locations. Be sure to keep up with the wiki, as videos from most of the sessions will be made available online soon. One of our improvements was to better document our proceedings, and I think we’ve managed to succeed on that one for sure.

This past Thursday and Friday, the Tulsa TechFest was held at OSU Tulsa in Tulsa (could I write Tulsa one more time, I knew I could). Attendance was high and most of the sessions I attended were in rooms full of people. The day started off early Thursday morning, but that’s not the start of the story.

The night before, Dru and Rob came by the house for a little pre-conference warmup (and by warmup, I don’t mean this). Rob went through his presentation on continuous integration one last time while Dru and I worked through our presentation on messaging (and how to do it with MassTransit, of course). The night ended early for me, but Rob and Dru met up with Ben at the hotel and closed the bar (and by closed, I mean walked in at last call and got one beer).

The next morning we all met up and caught up on things since the last gathering. Since the last time we saw Ben, he’d been through a hurricane and granted the MVP Award from Microsoft. We then planned out our day of sessions based on the information currently available to us.

The speaker for the CSS talk was unable to attend, so the four of us convened an open-space session on CSS. The discussion in the fishbowl was good with a lot of interesting topics. Ben gave an on-screen demonstration of CSS from the ground up for those in the room that were new to it, providing context for the audience. CSS is extremely important considering it is the best (only?) way to layout and style websites consistently across browsers. I think everyone brought up how much of a turd IE 6 is when it comes to CSS compatibility.

After lunch, it was time for Dru and I to present our session on message-driven architecture (using MassTransit). You can see the first hour of the session on video here. The crowd really got into it, asked a lot of questions, and hopefully came away with an understanding of asynchronous application design and messaging.

After that session, we sat down with a guy that works for Sun and talked about enterprise application architecture. It was interesting comparing the mature open-source nature of Java to the budding open-source landspace in .NET. After the closing session and prize giveaway, there was a speakers dinner (Rib Crib, good stuff). Once we had eaten, we went to the hotel and did a little code sharing and Dru and Ben went through ASP.NET MVC some more. Then we went over to Dirty’s Tavern for some post-day fun. I was worn out, so I went back to my car and called it a night.

The next day was full of interesting stuff. A nice introduction to ASP.NET MVC by Ben, some extensive coverage of log4net by Dru, and I gave a presentation on iPhone development. Outside of the actual sessions there were a lot of great conversations about development and tools in general. We also recorded Ray Lewallen’s session on Behavior Driven Development, which can be viewed here.

My iPhone development session was purely introductory to show the tools and how they are used to build and deliver applications for the iPhone. The room was absolutely packed and hopefully everyone walked away with some good information. I know at least one guy did, he left two seconds after I said that building iPhone applications requires a Mac!

To wrap it up, the event was a huge success. There were a ton of people there, the vendor room was always alive with activity (likely due to Chris Koenig and his Rock Band setup giving some much needed ADHD relief between sessions). Chris also had a couple of great sessions on Silverlight and the new features in 2.0 that should really improve the use of Silverlight for Rich Internet Applications (RIAs).

I’m happy to say that we’ve just tied a bow around the latest release of MassTransit. Release 0.4 includes a number of new features and some tweaks to the internals as well. I’m going to describe a few of those features below, but you can grab the latest from the trunk or download the 0.4 release.

Building MassTransit

Since Visual Studio 2008 has been out for almost a year, it is now required to open the updated solution for MassTransit. In the main folder, the MassTransit-2008.sln is the one to use to build and run the unit tests. Many of the samples solutions are also 2008 solutions. The assemblies, however, are still targeting the .NET 2.0 framework, making them usable on both 2005 and 2008 projects. With only the .NET 3.5 framework installed, you should be able to run the build.bat to build the project without Visual Studio (our CI server does this).

Timeout Service

To enable automated support for timeouts in sagas, a new timeout service is available. This is a general service that can be used to schedule timeouts for whatever purpose may be needed. To schedule a timeout, the application should publish a ScheduleTimeout message with the duration or time when a response should be sent. The application/service can then consume the TimeoutExpired message, which will be published by the timeout service when the timeout period expires.

Message Deferral Service

One of the scenarios I often find in our systems is the need to poll a remote resource to determine if an operation has completed. To support this behavior without custom code in each instance, a new message deferral service has been added. This can be used to defer the delivery of a message until a period of time expires. The deferral services leverages the timeout service for scheduling and we republish a message after that timeout expires.

For example, we have a CheckRemoteResponseStatus message in one of our systems. This is initially published after a request is submitted to a remote system and a remote transaction id is returned. The first time the consumer gets the message, it checks the remote system for a response. In most cases, the response is immediately available and the saga continues. However, sometimes the remote system is too busy to respond and returns a pending status. In this case, the same CheckRemoteResponseStatus message is published within a DeferMessage. The deferred message service handles that message and will republish the original CheckRemoteResponseStatus message when the timeout expires. The saga will then handle the message to see if a response is now available. The saga keeps track of how many times the remote status has been checked and uses a sliding interval that increases as the retry count increases. Eventually, the final retry results in a failed transaction and is handle appropriately.

The nice thing about this is there was no custom retry logic required, and a common timeout and message deferral service were used. There are likely other cases within the application that will benefit from this shared functionality.

Transactional Queue Support

With 0.4, the entire method of reading from the endpoints has been redesigned. Previously, a single receive thread was used to receive from the endpoint which then dispatched the message handling to the dispatcher inside the service bus. This has been redesigned to use the dispatcher threads to perform the actual receive from the endpoint, using a transaction (ala System.Transactions) to handle the message reception. This keeps the transaction to a single thread while at the same time allowing concurrent message reception.

The transaction carries over into actions that are part of the message consumer. If a database update is part of the consumer, that database update can cause the entire message to rollback if it fails. If any exception is thrown, the entire reception of the message, any additional database operations, new messages sent, etc. will all be rolled back with the transaction.

Performance Improvements

Dru spent some time in NYC with Ayende Rahien reviewing the MT source code and Oren recommended changing from using locks to ReaderWriterLocks to improve concurrency. The changes in the threading system, along with the elimination of a lot of locking in favor of reader/writer locks has nearly doubled the throughput of messages when using a multi-core system using MSMQ. There have been a number of other internal tweaks as well to improve the concurrency of the bus dispatcher.

Control Bus

To enable competing consumer in a publish/subscribe environment, the control messages need to be on a separate bus from the data messages. To allow multiple services to compete against a single data channel (single MSMQ) in order to load balance and handle failure scenarios, the services cannot compete on the control messages such as subscriptions. The subscription client has been tweaked to allow it to operate on a separate bus from the data bus, at the same time notifying the subscription service of messages handled by the local endpoints of the service.

It’s easy to setup a single service that consumes messages from multiple buses (which in turn each have a specific endpoint being serviced). When a component is created to consume a message, the specific bus/endpoint that received the message is injected into the component (via setter injection) so that any subsequent messages can be published to the appropriate bus.

Health Service

The health service has been added making it easy to monitor endpoints and identify when an endpoint goes down. Periodic heartbeats are sent to the service and when a heartbeat hasn’t been received in a while, it marks that endpoint as down and attempts to directly ping it to get a response. The heartbeats can be subscribed, so a monitoring tool can keep track of which endpoints are there and what they are handling.

Configuration Model

To make it easier to use the bus in different containers, a new configuration model has been added to build and configure a service bus instance. This will ultimately result in moving a lot of the code used by build a service bus out of the container-specific facilities (such as the Windsor container).

Host Improvements

The ability to create and deploy Windows services has gotten easier with the updates to the Host assembly in MassTransit. With only a few files to define the lifecycle of a service, it is easy to get the ability to run, test, install and deploy a service. This includes services that are using the service bus. There is little to no coupling between the host and service bus, making it usable for a variety of purposes.

Learning MassTransit

A lot of requests have come for information on how to learn to use MassTransit. During Tulsa TechFest this week, we’re going to record our presentation and make it available online within a few days. This should give at least some introduction on how to use MassTransit (the presentation is mainly on distributed architecture, but we’re using MT for the demo bits). We’re also talking about doing a couple of podcasts on how to use it as well. Depending upon how that goes, we’ll try to do a couple of screencasts on “creating your first project with MT.”

The best way to discover how to use the code is to review the samples. The WinFormSample gives an overall example of how a variety of features are used. The HeavyLoad shows how many of the pieces work as well. The samples folder has a few others that demonstrate how to use MT in other scenarios.

So, check out the new release and give us some feedback on how the new features are working. We’ve already got a few backlog items that we’re slating for 0.5 based on some other contexts that have come up in our applications. Feel free to post on the message group or send either Dru or I an e-mail or tweet if you have any questions.

One of the first applications we built with MassTransit provides messaging for a long-running transaction started by an application submitting a request. The request is formatted into a X12 envelope and sent to a web service. An intermediate response is returned (a X12 997) with a correlation identifier for the request. Another web service is polled for the response, which can be the result or an indication that the request is still pending. When the response is received, the X12 document is translated and stored in the database. Finally, the user is notified that the transaction is complete and the result is displayed.

As work proceeded on this application, I started to recognize the need for something to coordinate the different steps in a transaction involving multiple loosely-coupled services. Due to the duration of these transactions (the example above can take anywhere from three to sixty seconds), it is unreasonable to keep a single System.Transactions style transaction open the entire time. I started researching how others approached the problem and found a couple of articles that helped. After reading Sagas by Hector Garcaa-Molrna and Kenneth Salem (© 1987 ACM, PDF) and the chapter on sagas in the upcoming book Practical SOA by Arnon Rotem-Gal-Oz (PDF), I started to think about how this could be implemented within MassTransit.

I should note that NServiceBus (another open source service bus) also supports sagas, but I purposely avoiding taking a look at how Udi Dahan implemented them. Once saga support in MassTransit is complete I plan to review the source for NServiceBus to see how the implementations differ. I spoke with Udi at ALT.NET Seattle and his writing has been both educational and inspirational. A lot of great discussions in the NServiceBus mailing list have been an excellent resource as well.

So after a few weeks of trying to flesh out the structure (using TDD, of course), I finally arrived at what I think will be a highly usable infrastructure for handling sagas. In the project MassTransit.Saga.Tests, I’ve created a test that simulates a user registering for a web site. The class for the registration is shown below.

public class RegisterUserSaga :
	InitiatedBy< RegisterUser >,
	Orchestrates< UserVerificationEmailSent >,
	Orchestrates< UserValidated >,
	ISaga< RegisterUserSaga >
{
	private string _displayName;
	private string _email;
	private string _password;
	private string _username;

	public RegisterUserSaga(Guid correlationId)
	{
		CorrelationId = correlationId;
	}

	public Guid CorrelationId { get; private set; }
	public IObjectBuilder Builder { get; set; }
	public IServiceBus Bus { get; set; }
	public Action< RegisterUserSaga > Save { get; set; }

	public void Consume(RegisterUser message)
	{
		_displayName = message.DisplayName;
		_username = message.Username;
		_password = message.Password;
		_email = message.Email;

		Save(this);
		Bus.Publish(new SendUserVerificationEmail(CorrelationId, _email));
	}

	public void Consume(UserVerificationEmailSent message)
	{
		// once the verification e-mail has been sent, we allow 24 hours to pass before we
		// remove this transaction from the registration queue
		Bus.Publish(new UserRegistrationPending(CorrelationId));
		Bus.Publish(new UpdateSagaTimeout(CorrelationId, TimeSpan.FromHours(24)));
	}

	public void Consume(UserValidated message)
	{
		// at this point, the user has clicked the link in the validation e-mail
		Bus.Publish(new UserRegistrationComplete(CorrelationId));
		Bus.Publish(new CompleteSaga(CorrelationId));
	}
}

At the top of the class, the messages consumed by the saga are specified. InitiatedBy indicates the message initiates a new instance of the saga. Orchestrates is for messages that are part of the saga once it has been initiated. All saga instances are identified by a Guid and all messages consumed by the saga should have the CorrelatedBy< Guid > interface. A saga must also implement the ISaga generic interface to allow certain properties to be set giving the saga instance access to the bus and the object builder.

Once the saga class is added to the bus (via the AddComponent method), any messages consumed by the saga will be dispatched to the saga instance. A generic ISagaRepository must also be registered in the container so that sagas can be persisted between messages. The saga dispatcher uses the repository to either load or create the instance of the saga. Since instances of the saga class are saved, the class can expect the members to also be persisted between messages allowing state to be retained.

There is still some work to be done, including a service to handle timeouts and retries. It will be up to the developer to handle any compensating actions that need to be taken in the case of a failure. Therefore, it is highly suggested that the saga also consume any Fault< T > messages that are published when a message consumer throws an exception — particularly if the consumer is not part of the saga (such as an application or domain service).

The code is currently in the trunk and slated to be part of the 0.3 release.

I was reading through the Xgrid documentation for OS X yesterday after reading an article on Integrating Xgrid Into Cocoa Applications. The article gave me some ideas and I decided to see what it would take to build a distributed processing system on top of MassTransit. The result is a new MassTransit.Grid namespace that includes support for building distributed task processing into an application. The following sections define the language used in the distributed task classes.

Distributed Tasks

A distributed task contains one or more subtasks that need to be processed concurrently across multiple systems. To create a distributed task, create a class that implements IDistributedTask. The input and output types for the subtasks must also be defined by the distributed task class.

public interface IDistributedTask< TTask , TInput, TOutput >
{
    int SubTaskCount { get; }
    TInput GetSubTaskInput(int subTaskId);
    void DeliverSubTaskOutput(int subTaskId, TOutput output);
    void NotifySubTaskException(int subTaskId, Exception ex);
    void WhenCompleted(Action<ttask> action);
}

Subtasks

A subtask is an individual unit of work within a distributed task. Each subtask should be completely standalone and not depend upon the completion of any other subtask within the distributedtask. There is no attempt to execute the subtasks within a distributed task in order. A subtask has specific input and output types, each of which are defined by a class (POCO style). These input types are used to determine which workers are used to process the subtasks.

DistributedTaskController

To insulate the application from the details of coordinating the subtasks, a generic DistributedTaskController is used. This class is built from the class that implements IDistributedTask, along with the input and output types. Once created, the application can call .Start() to being processing the distributed task. The controller performs any initial identification of workers that are available to process the subtasks, along with the coordination to ensure that workers are not overloaded.

public class DistributedTaskController< TTask , TInput, TOutput >

TTask is the class that implements IDistributedTask, TInput is the subtask input type, and TOutput is the subtask output type.

Workers

To make it easy to create workers to handle subtasks, a default worker implementation is available. This worker handles the coordination with the DistributedTaskController, along with the delegation of the messages to the actual subtask worker. For example, a worker that accepts a GenerateFileHash object and outputs a FileHashGenerated object would be setup as shown:

public class FileHashGenerator :
       ISubTaskWorker<  GenerateFileHash , FileHashGenerated >
   {
       public void ExecuteTask(GenerateFileHash input, Action< FileHashGenerated > output)
       {
           string path = input.Path;
 
           // do work here
 
           output(new FileHashGenerated());
       }
   }

The worker can then be added to the container for servers that will be processing the subtasks using:

_container.AddComponent<  FileHashGenerator >();
_bus.AddComponent< SubTaskWorker < FileHashGenerator, GenerateFileHash, FileHashGenerated > >();

This will register the SubTaskWorker for the worker as a message handler for the messages that are used on the transport to transfer the input and output data between the controller and the subtask workers.

Exception Handling

If an exception occurs in a subtask, the worker and controller leverage the built-in fault handling support of MassTransit to notify the distributed task that an exception has occurred. The controller will call the NotifySubTaskException method with the subTaskId and the exception that was thrown by the worker allowing the distributed task to determine the next course of action based on that failure. Options would include simply aborting the distributed task, fixing the input data and adding it to the end of the subtask list, or some other application-defined behavior.

Dynamically Adding Subtasks

To reduce the impact of setup time on the overall duration of a distributed task, it is not necessary to have all of the subtasks loaded before starting the distributed task. This also allows additional subtasks to be added based on the output from other subtasks. For example, a task to parse a remote file system may identify additional folders that need scanned for content. The distributed task could just add those folders to the end of the subtask list and they would be picked up by the controller. By allowing this, the distributed task is responsible for calling the delegate set by the controller to indicate that all of the subtasks have completed. The DistributedTaskController will then release any resources that were in use.

Sample in Unit Tests

A quick sample was built in the unit tests (MassTransit.Grid.Test) that shows an integer factoring service. The distributed task creates a bunch of very large integers and processes them as a distributed task between the workers that are available. Hopefully this demonstrates how the classes are hooked together since this was used to drive out the feature set.

Wrapping Up

This is just a brief introduction to the distributed processing capabilities that were added to MassTransit. There are likely some additional features to add that will hopefully be identified as the feature it put to use. Therefore, it is important to note that this feature is still in development and should go through some considerable testing before putting it into use in a production application. Any feedback is always welcome (including patches) so try it out!

When deploying an application, it is important to consider how the application performance will be monitored. For interactive applications, this may include measurements such as the time to load a page, how long it takes the page to be ready for user input, or even how much data is transferred to the client on each request. In the case of a service, it may be important to know the duration for individual transactions, a batch of transactions, or another unit of work.

The benefits of being able to measure the performance of an application are numerous. First, it helps to track performance over time as updates are made to the application. If a new version of the application “feels slower” to users, the developers would be able to compare the performance of the current version to the previous version. This will either identify where the system has degraded or (unlikely as it may be) show that the user is just mistaken.

Another benefit of knowing the performance criteria for an application related to service-level agreements. If the response time of an application can be measured, those metrics can be used to establish expected performance numbers to share with customers. For example, if a certain operation takes less than two seconds on 97% of all requests, customer expectations could be set using that information. Without accurate measurement, it would be impossible to answer that type of question.

An Example

An customer service application has a feature that submits user input to a remote service. When the user clicks submit, a request is built containing the user input and credentials retrieved from the customer configuration. The request is then sent to the remote service. When the response is received, the data returned is formatted for display and shown to the requesting user.
Within this example, there are several points that could be measured.

• The overall duration of the request from the time the user clicks submit until the response is displayed could be tracked. This would be useful in tracking the overall user response time.
• The time taken to retrieve the credentials from the configuration store could be tracked. This would help identify a slowdown in the database (or whatever type of storage is being used).
• The elapsed time between sending the request to the remote service and receiving the response could be measured. This would help identify latency issues with the remote service.

All of these values could also be compared to each other to separate the total time for the operation into the amount of time for each individual operation.

Method Timer

To measure a method on a class, a stopwatch can be used to measure the elapsed time of the method. To make it easy, I created a class that encapsulates the functionality of the Stopwatch class (from the System.Diagnostics namespace). My class implements IDisposable, allowing it to be automatically disposed at the end of the function with a using {…} block.

public string LoadUpdates(string url, string name)
{
    using (AutoFunctionTimer timer = new AutoFunctionTimer(url, x => _log.Info(x)))
    {
        string updatePage;
        using (timer.Mark())
            updatePage = new PageLoader(url).GetBody();
        using (timer.Mark())
            return GetUpdates(name, updatePage);
    }
}

The delegate specified on the constructor is called when the stopwatch is disposed and the timing measurements are complete. The delegate is passed the stopwatch so that it can be output to a log file for processing by the measurement code. The string output includes the time the method started (in UTC), the duration of the method, and the string passed to the constructor.

Timer Sections

Sections make it possible to measure individual operations within a method. The time for each section can be logged, making it easy to identify the largest time consumers in a method. In the code example above, the timer.Mark method is used to create a section. This section time is then output after the total time, allowing it to be compared to the duration of the method. If multiple sections are created, they are output in creation order for consistency.

The code for the entire class is shown below.

namespace MyNamespace.Core
{
    using System;
    using System.Collections.Generic;
    using System.Diagnostics;
    using System.Text;

    public class AutoFunctionTimer : IDisposable
    {
        private readonly Action<string> _action;
        private readonly string _description;
        private readonly List<stopwatch> _marks = new List</stopwatch><stopwatch>(10);
        private readonly Stopwatch _stopwatch;
        private DateTime _started;

        public AutoFunctionTimer(string description, Action<string> action)
        {
            _description = description;
            _action = action;
            _started = DateTime.UtcNow;

            _stopwatch = Stopwatch.StartNew();
        }

        public void Dispose()
        {
            _stopwatch.Stop();

            _action(ToString());
        }

        public override string ToString()
        {
            StringBuilder sb = new StringBuilder(256);

            sb.Append(_started.ToString("yyyy-MM-dd HH:mm:ss ")).Append(_stopwatch.ElapsedMilliseconds);

            foreach (Stopwatch mark in _marks)
            {
                sb.Append(' ').Append(mark.ElapsedMilliseconds);
            }

            if (!string.IsNullOrEmpty(_description))
                sb.Append(" ").Append(_description);

            return sb.ToString();
        }

        public CheckPoint Mark()
        {
            Stopwatch watch = Stopwatch.StartNew();
            _marks.Add(watch);

            CheckPoint point = new CheckPoint(watch);

            return point;
        }

        public class CheckPoint : IDisposable
        {
            private readonly Stopwatch _stopwatch;

            public CheckPoint(Stopwatch stopwatch)
            {
                _stopwatch = stopwatch;
            }

            public void Dispose()
            {
                _stopwatch.Stop();
            }
        }
    }
}

Logging Performance Metrics

For measuring performance within a single application or service, I recommend using log4net to output performance metrics to a file. It’s easy to setup and configure post-deployment, making it an obvious choice. Additionally, each server should log to a directory on the local drive. Not only does this eliminate the need for a network share and the associated permissions, but it makes it less likely that network latency will impact system performance due to logging.

For distributed applications, the story changes quite a bit. Because of the nature of distributed operations, I’ll cover them in another post.

Over the past few weeks, Dru and I have been working on the next release of MassTransit. The current release (0.2) has been beat on pretty hard and is currently seeing some active use in the field. There were a few tweaks made after the initial 0.2 drop to address some issues that popped up.

One of the really cool new features coming in the next release is the ability to automatically wire publishing into the consumers. By implementing a new interface, the service bus will automatically attach the appropriate plumbing into the consumer at the time of creation to handle the publishing of messages back out of the consumer. The new Produces<T> interface helps to insulate your code from the IServiceBus interface by relying only on the new interface on your message handler class.

For example, take a look at the following class:

public class OrderService :
Consumes<createorder>.All,
Produces<ordercreated>
{
private Consumes</ordercreated><ordercreated>.All _orderCreatedConsumer;

public void Consume(CreateOrder message)
{
/* Do the order work here */
_orderCreatedConsumer.Consume(new OrderCreated());
}

public void Attach(Consumes</ordercreated><ordercreated>.All consumer)
{
_orderCreatedConsumer = consumer;
}
}

The ServiceBus will automatically call Attach() to connect a publishing consumer to the class when it is created from the container (so this only works with AddComponent<> style message handlers). These helpers classes will take the call to Consume(new OrderCreated()) and route it through the bus and out to any subscribed consumers without knowing about the actual bus itself. This also makes it easy to use the same message handler with multiple IServiceBus instances if you are listening to multiple endpoints.

This is just one of the new things coming soon that I’m pretty excited about. There are a few others, but that will be the subject of a later post.

We’ve dropped a new release of MassTransit today, version 0.2 is now available on the main page. There are several new features included in this release. It was great to get some feedback from people who have tried MassTransit, along with the evolution from discussions with Greg Young, Udi Dahan, Ayende, and others at ALT.NET Seattle.

A quick summary of the changes in this release:

Extensible Message Dispatcher

To make it easy to add new messaging patterns to the service bus, MassTransit has an entirely rewritten message dispatcher. The new code is structured in a producer/consumer style that lends itself to easy extensibility, allowing new features such as batch messaging and correlated messages without heavy lifting.

Component-Based Message Handling

In the previous version, handling messages required an object to subscribe to messages types passing methods to handle the messages. Now, a class can implement interfaces to support message consumption and the class itself can be added to the service bus. The service bus will then create objects to handle each message, removing the need to use the same instance for each message received. A class can handle multiple messages types, and can also indicate whether to receive all, selected or correlated messages by implementing the Consumes.All, Consumes.Selected, or Consumes.For interface.

Batch Messaging

The new message dispatcher includes support for message batches. Instead of having to correlate a batch of messages at the application layer, an object or component can consume a batch instead of each individual message. Details of how to use batch messaging are on the wiki.

Container Integration

We finally bit the bullet and started using a container. A new assembly called MassTransit.WindsorIntegration adds a default container derived from WindsorContainer that adds facilities to create ServiceBus instances. A new custom syntax was created to make it easy to configure multiple ServiceBus instances. All of the samples have been updated to use the new container to help understand the integration points.

Plain Old C# Object Message Objects (POCOMOs Anyone?)

The need to have all messages implement IMessage is gone, reducing the footprint of MassTransit in your application code. There are some new interfaces to handle things like correlated messages and batch messages, but those are only needed to use the built-in support for those message patterns.

Better Thread Management

To allow for more control over resources, a new thread manager has been added. While we haven’t exposed the thread configuration yet, a dedicated thread pool for asynchronous message dispatching should allow for more efficient message handling. We also took all the threading code from the endpoint and put it in the service bus, reducing the complexity required for new endpoints. In fact, the endpoint structure has also been redesigned to be more send/receive focused.

Publish/Subscribe Focus

The service bus now has a pure publish/subscribe architecture compared to the previous additional methods, such as Send() and Request(). For applications that need to send messages directly to endpoints, the endpoint now has a Send() method. The publishing of messages takes advantage of the same new subscription code, allowing all messages type information to be cached for better performance (avoiding the reflection penalty on each call to Publish). With the extensibility of the message dispatcher, it’s likely that remote endpoints may some day find there way into the dispatcher, resulting in a single dispatch engine for asynchronous publishing of messages as well.

Request/Reply

Requests are handled in an entirely new way, using a new fluent builder. The fluent builder allows the calling code to subscribe to any responses (directed via the Consumes.* interfaces, indicate whether an asynchronous callback should be allowed (for [WebMethod] style Begin/End usage, PageAsyncTask usage, or MonoRail asynchronous actions), and get a future object that can be used to complete the request upon receipt of a response. The responses are handled by the calling class itself, the future object is used to signal the operation complete which will release any waits or callbacks for the action.

Distributed Subscription Cache

A new distributed subscription cache (backed by memcached) is now available. This is mostly designed for high volume request/reply applications that need to add and remove a lot of correlated subscriptions and maintain a high level of performance. A load test will be added to the HeavyLoad sample soon, but it seems to hold up pretty well so far under regular testing.

Dashboard

Dru has been hard at working making an operations dashboard part of the core product. One of the big things about messaging systems is being able to assess the health of the endpoints at any time. The goal of the dashboard is to provide that single pane of glass to find out which endpoints are alive, what messages they are handling, and indicate any problems to the viewer. There are many plans for this, including the ability to remotely control the endpoints for dynamic adjustments to load handling and perhaps even remote service restarts.

Deployment

Dru has also been working on the deployment story, making it easier to deploy MassTransit into a production system. There is much love needed there, but I’m hoping to dig into it soon to see how things have gotten easier to manage.

I’m sure there are many more features that have been added under the covers. The main thing is that with this release (0.2) we’re pretty happy with the API experience. The consumer code is easy to understand and implement, particularly compared to the earlier version. It is unlikely that we’ll do another major overhaul to the interface like we did with this version.

So give it a shot, and blog about your experiences!