PhatBoyG.com

Over the past few months I have been reviewing many of the products I was involved in creating, both as a developer and an architect, and have assembled an inventory of the technology and architecture used. With a catalog of products spanning more than eighteen years, a diverse set of architectural styles are represented. On one end of the spectrum are client/server systems deployed on-premise and on the opposite end are software-as-a-service (SAAS) browser-based products. Most of these products are line-of-business systems and include both heavy user interaction and background data processing. In fact, two separate products offer a similar feature set targeted at the same market but sit on opposite ends of the architectural spectrum. The first product was built in the 90′s and is a client/server system, the latter was built more recently during the SAAS era targeting the web.

What follows are a few of the common design choices that I encountered, with my take on how appropriate that same choice would be today.

Data Storage

As I looked into each product, I examined how various requirements were addressed given the tools available at the time. For example, I didn’t question the use of a flat file to store reference data in the early client/server products since flat files were perfectly acceptable at the time. However, this led me to question some design choices when looking at SAAS products — including some choices that you might not expect. For instance, why is a flat file not an acceptable design choice for a system developed today? The data is still the same reference data, yet current guidance would suggest this reference data be stored in a database, most likely a relational database.

Is this because developers have become too lazy to write the component to read the file? Surely not, since a component will have to be written to import the reference data into the database. While it can be done fairly easily using database tools, the process still has to be scripted out and repeatable in case the import needs to be repeated on a new database.

Let’s enhance the problem and add a time dimension to the reference data, making updates available every thirty days. Now, not only is an initial import needed, but the import component will also need to support updating the database with the new content. Again, this could be done using database tools — a simple truncate table and repeat the import process. But what if developers have created relationships between the reference data table and other tables in the system? What if those relationships were created using the row id instead of the appropriate business identifier? At that point, the table cannot be simply truncated and the update process must now perform a complete delta of the existing and updated data sets and merge the changes into the database. That certainly doesn’t sound lazy — if anything, it sounds downright painful.

Another question that came to mind when using a relational database to store reference data was “which database?” Now, if the first answer that popped into your head when you read that was “SQL Server,” or even worse “the database,” therein lies the real problem.

A product is not just an application, it is a system composed of one or more applications, multiple components, multiple services, and multiple databases. Consider the earlier example that used a flat file to store reference data. The flat file itself is a separate database. In a system of any complexity there are many different sets of reference data, all of which are stored in their own separate flat files. Therefore, the system has multiple databases, each using the appropriate technology based on how that database is used.

If the reference data had remained in a flat file, when the flat file was updated with the new reference data, the original file is simply replaced and the system continues. No special import or update process is required.

Nested Object Graphs

Another common design I saw, particularly in products that manage a revolving set of accounts, was the use of a deeply nested object graph that is persisted in a relational database. As accounts were accessed, the entire object graph would be loaded from the database and presented to the user. Once the user made whatever changes were necessary at the time, the account was then saved to the database. In order to save the object graph, the nodes at each level in the graph are compared with the database, and deltas are generated to update the database tables.

In early examples of this design, a pessimistic locking system was implemented to track user activity and prevent multiple users from working on the same account at the same time. This was common in the client/server products, since even at that time record locking using ISAM files (or even network file locking) was fairly problematic.

As products moved to the web, a more optimistic locking strategy was used. I found two different conflict resolution methods, the first of which used a timestamp to track modifications to an account. If an update was received and the timestamp didn’t match, the later update was rejected. The second method was “last write wins,” updating the account to whatever was in the later update — possibly and quite commonly losing previous updates from other users. This got real interesting when two updates were performed at the same time.

Neither of these solutions make sense today for SAAS applications. In an environment where multiple users may be interacting with an account at the same time, it’s more important to look at providing users with a task-based user interface that captures the intent of each action on an account. For example, loading an entire account just to change the billing address creates unnecessary data movement that can limit throughput (read: scalability concern). At the same time, preventing a user from adding a charge to an account because another user slipped in behind you to update the phone number creates an unnecessary user burden. If updating the billing address, updating the phone number, and adding a charge to an account were explicit actions (read: commands) that can be performed on an account, they could all be performed simultaneously without conflict.

Note that the Command-Query Responsibility Segregation (CQRS) or even just Command-Query Separation (CQS) architectural styles specifically addresses this type of design.

Stored Procedures

In the example above, a deeply nested object graph was loaded from the database. In a system designed today, a developer would most likely reach for an object-relational mapper (ORM) to deal with loading and saving the object graph to the database. There are many to choose from (Hibernate, NHibernate, and Entity Framework are a few) and they solve the problem of binding object graphs to relational database tables very well. In fact, most ORMs today can generate the DDL needed to create the database objects as well — eliminating the need to write table creation scripts by hand.

At this point, I can hear the blood pressure of many database administrators reading this rising through the roof. With SQL book in hand and years of experience writing stored procedures full of selects and cursors, the story of how a hand tuned stored procedure that returns a sequence of forward-only record sets in a single round trip to the database server is the only way the scalability requirements of the application can be met. I’m not saying that using a stored procedure in this situation is wrong, but making a stored procedure the first tool you pull out the toolbox is very wrong indeed.

Why is it wrong? Creating a stored procedure to read data as the first approach is wrong because it is an optimization. Optimizing components of a system before that particular component has been identified as a bottleneck will lead to increased complexity, and that complexity will breed quickly in the project. And as complexity increases across the project, long term maintainability suffers as the capabilities of the development team are challenged. Yep, you guessed it, the stored procedure first approach is a classic case of premature optimization.

How does using a stored procedure in this way breed complexity? First of all, it establishes a myth that reads are a problem. As functionality is added to the system, developers who have come to believe that any account related reads must be done with a stored procedure else they become responsible for performance inadequacy, create more read procedures. As features continue to be implemented, more data elements are added to the schema, requiring every stored procedure to be updated as the schema changes — creating more work for developers who must now touch features that were complete and tested to ensure they still operate as expected.

The opposite effect of the read myth is that retrieving the entire object graph for an account is so well optimized that it is better to load the entire object and use only the needed data elements rather than create a new read procedure. With an ORM, this is handled very well using projections and fetching strategies. Developers can use the ORM to read a partial object graph, returning on the required data elements and reducing the data movement between the database and application server.

All of this accidental complexity was created based on the superstition that only a stored procedure would be fast enough to support the scalability needs of the product. An optimization that was implemented before a bottleneck was identified.

Considering that most ORMs today are capable of writing very efficient SQL and have dialects specifically tuned for each database platform, the read performance of the ORM is less likely to be a system bottleneck. For example, with Microsoft SQL Server, NHibernate takes advantage of batch queries with ADO.NET to reduce the number of round trips between the database and application servers. The SQL generated is also parameterized, allowing the SQL engine to cache execution plans for better server performance. Given these optimizations have already been done by the ORM, tuning read performance in the database is not likely to create the biggest benefit in system scalability. For example, caching of already loaded objects will likely result in greater overall read performance.

Did I forget to mention that this early decision tightly coupled the product to using a particular database platform? SQL dialects are hardly portable between platforms, so the product now has to decide if it will work with a single platform or create a separate release branch for each database platform supported. The better ORMs support multiple server dialects, including Microsoft SQL Server, Oracle, MySQL, PostgreSQL, and many others.

I said I wouldn’t argue the performance difference between using an ORM and a stored procedure. I will point out, however, that using a stored procedure to tune performance is an optimization for a particular environment and should not be an early choice in system design. Going straight for the stored procedure without considering less complex options is another case where a lot of times, the tool we used yesterday is not always appropriate for a system being designed today.

To Be Continued…

Above I’ve covered a few of the design choices made early in the development of several major products and how that affected the evolution of the product over time as featured were added. I also applied a modern view of how many of the choices we made before all these “great tools” were available are not necessarily bad today. As I get more time, I hope to share a few more stories with you as I undercover them in what has basically become a “career retrospective” for me.

Today I was honored for the second time with the Microsoft MVP award. It’s great to be recognized for my efforts in the .NET community over the past year. The next year is already shaping up to be another great one, with upcoming speaking engagements at Dallas TechFest, Devlink (Nashville, TN), St. Louis Day(s, plural) of .NET, and the Heartland Developers Conference in Omaha, NE.

If you are near any of these great events, I hope you are able to attend, learn a few things, and most importantly meet others that are part of the software development community. I also would encourage you to attend a few sessions outside of your regular development platform to get an idea of how other technologies solve the same problems in their own way. The cost to value of all these events is an absolute bargain, and many have early registration discounts that are only good for a limited time, so be sure to get registered to ensure the best price.

I look forward to meeting some of you over the next few months, so if you are at one of my talks or see me in the hall, be sure to introduce yourself and give a shout out.

(word)

This past week I had the pleasure of attending the Heartland Developers Conference in Omaha, Nebraska. I was already in town visiting family and decided to take a day to see what the local flavor had to offer. I’m particularly grateful to Joe Olsen of PhenomBlue for allowing me to register. I was originally hoping to secure a spot as a presenter, but the list apparently fills up in early March so next year I’ll try to plan for it.

On Wednesday night, a pre-party for attendees was held at the Qwest center. I rolled into the event around 9:00 PM (after watching the first part of the debate) and introduced myself to a few of the people there. Some of the folks I met included Jason Bock, Chris Williams, Joe Olsen, Amanda Laucher, Jeff Julian, and John Alexander. It was nice since there were some drink tickets being passed around and Rock Band 2 was going in the corner. I talked with several of the folks there and left looking forward to the content that was on tap for the coming day.

The morning of the event there was a lot of time to mill about before the sessions got underway (particularly since I got a ride from my mom and was there at 6:45 AM). Early on I found Joe Stagner (blog) and chatted about his recent hard drive upgrade amongst other things. I later met Clint Edmonson, an architect evangelist with Microsoft and discussed the content of the session he was delivering that day. There were a lot of good sessions at overlapping times so I wasn’t able to attend them all unfortunately. Once I settled down for the keynote next to Jason Bock, I caught up on some email and listened to Joe’s presentation.

My first session of the day was Jason Bock (blog, twitter) who spoke on Reflection and IL, including using Cecil for some post-build IL weaving. Very interesting topic for me and a great presentation. I spoke to Jason the night before to see if he had looked at any of the new expression tree tools in 3.5 and how to use them to build code on-the-fly without using Emit. It was an interesting discussion, Jason is a smart guy.

When it was time for lunch, a few of us went over to Farrell’s Bar and had some nachos and a burger. I found it funny since I was the only one from the south, I was also the only one who enjoyed the jalepenos. We chatted in general about community involvement and various events where we all had attended/presented and overall was a great discussion.

After lunch, Drew Robbins (blog) presented on the Microsoft Extensibility Framework (MEF, web). Despite a stuffy head that made MEF sound like “METH” I got a lot of great information and look forward to learning more about this new framework. The way you specify exports and imports really makes it easy to define an extensible application. I’m certainly going to look at ways to use MEF with MassTransit in order to provide a new way to compose services that consume messages.

I had to take a rest for a while, chatting with the Geeks With Blogs guys for a while and generally just taking it easy. After bouncing around I settled into Chris Williams’ (blog, wrap-up) talk on XNA. I’ve never looked at the game development tools for Windows, but I got a brief into as to what to expect. Once this session was up, it was break time. I was planning on doing a podcast with the Geeks With Blogs guys but was trumped when Joe Stagner and Amanda Laucher sat down to do a joint session.

The day was great and I learned a few things. I also got to meet some great people and had some interesting conversation. I had hoped to attend the F# presentation by Amanda Laucher (blog) and the Open Source presentation by Javier Lozano (blog), but I wasn’t able to return for day two of the event. The time on the day I went was well spent and I look forward to attending HDC again in 2009.

In two weeks, the 2008 installment of Tulsa TechFest will be upon us. For two days, Tulsa is going to unleash an impressive array of sessions on all aspects of IT, security, and software development. As I review the broad list of presenters I can’t help but see conflicting sessions where I’m going to have to make some tough choices.

If you are going to be anywhere near the Tulsa area and can manage to slip away from work for a couple of days I highly recommend making an appearance. The breadth of learning opportunities at the unbelievable price ($2/day) make this an incredible way to learn some new skills and sharpen your existing ones.

I will be presenting in two sessions this year. The first session will be on building distributed application using MassTransit (co-hosted by Dru Sellers) and other open-source frameworks for .NET. This is doing to be a deep view on how to build loosely-coupled systems on top of a messaging service (in this case, MSMQ). Advanced topics include asynchronous messaging and sagas (long-lived transactions).

The second session will be an introduction to iPhone development. I’m not a seasoned expert here, but I’m impressed with the platform provided by Apple (Xcode) free of charge for building applications for Mac OS X and the iPhone. This introduction will cover the tools and application structure for building iPhone applications in Objective-C.

If you happen to see me there, feel free to stop me and say hello.

I came up with this a month or two ago, but finally decided to share it. While working on Mass Transit, I was joking with Dru Sellers about how nice it was to have really good test coverage when making design changes to some all-new development code. I’ve had very limited opportunity for a completely new projected started purely from unit tests, so I was just impressed at how easy it was to make code changes knowing that a passing set of tests meant all was well in the world.

You see, not all parking lots are paved with quality asphalt, generally flat, and void of any obstructions like islands and lights (see my other hobby). At work, our application is a lot of vintage C++ code, a ton of stored procedures packed to the hilt with domain logic, and nearly zero percent unit test coverage. Since adapting agile development, it is something that has been missing from our process. In our latest iteration, we’ve started using unit tests (with NUnit) to design our interfaces and classes. At the same time, we’re integrating Mass Transit to support the loosely coupled layer of application services (which include object translation, communication with high-latency remote systems, and lazy auditing of transactions). Aside from a few basic web services to support remote client application support tools, this is the first C#/.NET development that is being done as part of the main application.

So back to our story, my first project with really good test coverage exposed me to a lot of new things. From a TDD perspective, I’d read about it, used it to build some basic tests for various classes, and thought I had a pretty decent understanding of it. In this new project, I also learned how to use Rhino.Mocks (which took the test run time from 40-50 seconds down to 1.83 seconds on average), a very powerful tool for making an interface behave as you would expect an implementation of that interface to behave. The use of mocks has really helped me focus on actually writing tests and building a single class at a time. Prior to using mocks I would jump around creating additional classes as I defined new interfaces just to be able to continue writing my unit tests on the original class. By using a mock, I’m able to simulate the behavior of the other class without losing focus.

As my appreciation for TDD grew, I jokingly dropped a slogan into a chat window (using Skype, of course, aren’t you?):

I got a few chuckles, and thought it would make a great t-shirt to wear to tech events like code camps. So I threw together a quick online store so that I could order one for myself. I showed it to a few others (like Joe Ocampo, who suggested the slightly less offensive, yet subtly more suggestive variant) and decided to make it available to anyone that wanted one. So if you like it, grab one for yourself and maybe I’ll see you wearing it at ALT.NET Seattle!

Over the past two weeks, my department has been working on our first iteration using agile practices. Yesterday, we wrapped up with a retrospective to go over our progress. We used a fish bowl to keep the conversation centered and focused — a method that once again proved to be useful for controlling a discussion without controlling the discussion.

We setup a whiteboard with columns for the following topics:

Start
Things that we should start doing on the next iteration.

Continue
Things that we should continue to do every iteration.

Stop
Things that we should stop doing.

Debt
Things that we did (or didn’t) do that will contribute to our technical debt.

We started with an introduction to the retrospective, a declaration of our goals, and a quick recap of how the fish bowl works. We also identified a remote advocate — a single person who is responsible for coordinating communication with our remote team members. Our company uses Live Meeting for conferencing, so we explained how to use the seating chart and how to use the Raise Hand feature. The advocate also had their IM client up for any out-of-band questions or issues with the meeting client. Once that was up and running, we opened the discussion.

Some of the topics discussed:

• Start making sure the acceptance criteria are well defined before starting the story.
• Start pairing throughout the development of the engineering tasks and not just at the end for review
• Start keeping an audit trail of initials of people who worked on a story or an engineering task
• Stop putting incomplete stories into the backlog
• Continue the daily stand up meeting format

There were many more, but you can see how the structure worked. In all, we identified around 20 items that we need to either start, stop, or continue.

Once that segment of the meeting was over, we went over some of the methods being used to track things like burn down. Our project manager (whom we have yet to designate with a more appropriate agile title) went over the spreadsheet she uses to track story points, engineering task estimates, and actual hours worked on each task. She then showed some web sites from other groups in the company doing Scrum and how they had organized their Wiki, how they posted pictures of their planning board and burn down chart, and their honorary stuffed ScrumMaster.

We did have a few bumps towards the end of the iteration with our test environment and the number of defects coming back from testing (which is why we want to start pairing earlier). We hope to improve with each iteration (of course) but for our first lap around the track I think we did pretty well!

On Monday, we started our first iteration. We spent the previous two weeks discussing the process, preparing stories, and learning how we can adapt extreme programming (XP) practices into our development process.

We kicked off with iteration planning to identify the engineering tasks associated with the stories in the iteration backlog. We gave estimated in ideal hours for each of the tasks and added them to the story cards. Once the planning was done, people picked up tasks they were going to work on (some were assigned, others volunteered) that day. We made a point to assign a peer to work with the task owner until the task is finished.

Every morning at 9 AM, we have a stand up meeting with the team to go over each member’s progress from the previous day, plans for that day, and any roadblocks that are inhibiting forward momentum. Once each person was done, we got down to business and started work on our tasks. Full-time pairing has been limited so far, only a few have been pairing for the engineering tasks. I’m sure it will take time to adjust to what works best for the members of the team.

For project tracking, we are using a physical planning board with the cards pinned to the board. We tried a number of electronic methods to improve collaboration with our remote team members, but the tools just got in the way of the process. We are going to follow up in our retrospective with the remote teammates to determine how we can improve their involvement in the daily workflow. For now, they are communicating via e-mail and phone with our project manager who is working with me for our daily focus (like a scrum master, but in XP terms that I can’t remember).

One of the initial things we found is that it is really important to mark on the cards WHO is working on the engineering tasks associated with the story. Without this step, members of the team sometimes lost track of what they were doing and ended up working on other things. The ownership of the story once picked up really needs to be communicated well with the team to make sure multiple people don’t end up doing the same work.

I paired with another team member on a forward deployed application that is part of our system. I’m fairly experienced with the program whereas my pair-mate (?) had no experience with it. We managed to complete a couple of stories in the first three days of the iteration, some solid progress in my opinion. We spent a lot of time testing various scenarios to ensure the changes we made didn’t break anything. Once each story was complete, we did the paperwork to integrate our code changes into SVN. Yes, we have paperwork for each change to ensure compliance with SOx.

We had a couple of stories that we found did not have a sufficient level of detail in the acceptance criteria to complete. After working with the stakeholder on the story and acceptance criteria, the estimate changed and we’re trying to determine if there is room enough in this iteration to complete it properly.

Our application has been live for several years, so we have members of the team that handle support for our production systems. Starting this week, defects that are found in production are red-carded and put into the Inbox on the planning wall (we are using red cards for defects). We then take a few minutes to estimate the defect and depending on the size either add it to the iteration or put it into the release backlog for the next iteration. This allows us to quickly get production issues fixed and delivered for system testing by our SQA department.

This week we’ve also been working with product management on stories for large upcoming features that are being added to the product. That has been a very good experience, particularly in how we are building our knowledge of the user behavior that is being requested. I’m even thinking of how we could write a series of how-to documents on using the new system features based on the stories we are writing — I’m sure our documentation team with love that!

So I’m encouraged by our progress so far, but overwhelmed with the pace at which we’ve adapted these new practices. It seems like only a couple of weeks ago I was having meetings with our management team about moving to a more agile development process and here we are today almost midway through our first iteration. Let’s hope we can sustain our pace, keep the backlog full, and finish our first iteration with success.

Last week, I started looking into using WATIR for web application testing. In the short amount of time I spent with the tool, I got an initial look at Ruby. Aside from seeing it here or there, this was the first time I actually wrote any Ruby code — but it was mostly cut and paste based on the testing examples.

Today, I realized that Ruby (and Ruby on Rails) is part of Leopard and delivered with OSX. So I fired up a terminal window and started playing around with the console (IRB). Mind you, I was only playing around with it at this point so I didn’t get too deep. I created a few classes, learned the value of open classes, added some methods to the built-in classes (like String.is_your_mom), etc. I was basically playing around without any instructions.

Tonight, I started reading a few articles on the Apple web site about getting started with Ruby on Rails using Mac OSX. I created an application, created some models and a migration, installed and configured MySQL for use in development and test, and created some scaffolding to be able to perform basic CRUD against my domain model. It was at this point that I got an idea of how web applications could be created using Ruby on Rails.

The part that concerns me with what I’ve learned so far is that ActiveRecord is handling all of the mapping of database columns do the domain model. I’m not sure this is how I would really want that to be done, but I’m so new to Ruby I can’t be sure. I’m going to find some examples of applying DDD to Ruby on Rails and see if the two play nicely together. It seems like a lot of the Ruby framework expects a lot of things to be named specific ways or it all breaks down quickly. I do know that I’m not a big fan of having the model defined from the database.

I also know that building an application with a Rails web interface might also lead to building supporting application services in other languages using the same database. In that case, it would seem that having a model-driven architecture that can generate the classes for both Ruby and C# (and likely Java) would be very useful. I have Sparx EA, but I’m not sure if it supports Ruby yet or even if that is how I would want to go.

I’m going to pick up a couple of Ruby books to read more about it — it seems Idiomatic Ruby is a good place to start. Who knows where that will lead!