Microsoft released Exchange 2010 SP2 RU1 (roll-up update 1 for Exchange 2010 Service Pack 2, or version 14.2.283.3) on Monday, February 13, 2012. As you probably know, a roll-up update provides a complete set of fixes for a software product and once applied, brings a product as up-to-date in terms of software as Exchange’s Customer Experience (CXP) organization can deliver. As such, a roll-up update is a very good thing that should be welcomed by Exchange administrators.

A chat with Matt Gervais, site editor of SearchExchange.com, got me thinking about the contents of a roll-up update and how Microsoft decides to include the various fixes in the update. I decided to look through the set of fixes that Microsoft included in SP2 RU1 to gain a better understanding of the kind of background maintenance engineering that has to be performed for a major software product.

Before beginning to look at the fixes, the first thing to consider is that Exchange 2010 is now in the category of battle-hardened software. First released in October 2009 and then “completed” in terms of functionality with Exchange 2010 SP1 in August 2010, the product has been used since on a daily basis by millions of people, a fact that enables Microsoft to receive an enormous amount of customer feedback plus observations from professionals working in the field. Microsoft gain further benefit from the feedback received from internal deployments, both to serve its own requirements as Microsoft’s internal email system and as the basis for Exchange Online in its Office 365 cloud offering. Much of the feedback received since 2010 is reflected in Exchange 2010 SP2 (December 2011). A lot of work occurred to add some new functionality and improve quality in the fifteen months between the SP1 and SP2 releases…

The second thing we need to understand is the sheer size of the Exchange code base. Microsoft has been developing Exchange since 1993 or thereabouts and some twenty-five million lines of code form the current base. Code is rewritten all the time to take advantage of new development languages and frameworks (C# and the .NET Framework are obvious examples), to add new features and functionality, and to remove old code that is no longer required. The first version of Exchange was very influenced by the X.400 and X.500 ITU standards for email interchange and directory services; that code has long since disappeared. More recent code eliminations includes the replacement of WebDAV by Exchange Web Services in Exchange 2010. The point is that Microsoft manages a huge code base that evolves all the time. All code bases are fallible to some degree in that they contain bugs. The question is whether these bugs are important because they affect many different customers or, at the other end of the scale, are minor irritations such as a badly translated string that surfaces in a label in the Thai version of Outlook Web App. Bugs are expected and they are there. Most large software products have to manage a bug database that document thousands of bugs. The question is how the developers address the bugs.

The Exchange team triages bugs on a regular basis to make a call as to how important individual bugs are, how quickly they can be fixed, and when the fixes might appear. Once the decision is made to fix a bug, the responsibility for the work is assigned to individual developers. Eventually, updated code is provided for testing and if everything checks out, for integration into the nightly build of Exchange. Microsoft uses a huge battery of automated tests to validate code and make sure that new code doesn’t introduce new problems.

Sometimes, as in the case in April 2011 when Microsoft was forced to withdraw and then re-release Exchange 2010 SP1 RU4 after a fix exposed a problem that actually made its way through the complete testing and validation cycle to end up on customer systems, the testing suite can’t catch everything. Microsoft took their beating over the quality issue and have since improved their testing processes. So far no one has protested that SP2 RU1 contains a new problem. There’s always a chance that this might happen, so it’s a good idea to scan places like the Exchange 2010 TechNet Forum to see if reports have come in about something that you should be aware of before you rush to test the software yourself.

The overall context that we face with Exchange 2010 SP2 RU1 can therefore be characterized as an update for a mature software product based on the feedback from thousands of deployments and millions of users that has gone through substantial testing before the new code was released. Certainly that’s one way to look at RU1. Another way is to examine the list of the 58 individual fixes included in RU1 that Microsoft lists in KB2645995. When I looked at the list, my eye was taken by the following entries:

• It takes a long time for a user to download an OAB in an Exchange Server 2010 organization: Unsurprisingly, if you have two Active Directory sites connected over a slow link, downloading the OAB could be slow for clients that have to access OAB files in the other site.
• You cannot move certain mailboxes from an Exchange Server 2003 server to an Exchange Server 2010 server: This is a good example of an edge condition that is unlikely to apply to many Exchange 2003 deployments. Essentially, the Mailbox Replication Service (MRS) didn’t have the code necessary to deal with mailboxes that have an ACL property set. They do now. If this had been a big issue in the field, I think that the problem would have been found and addressed a long time ago.
• The cmdlet extension agent cannot process multiple objects in a pipeline in an Exchange Server 2010 environment: The cmdlet extension agent is a neat feature that was added in Exchange 2010 to allow for the execution of site-specific PowerShell code after an Exchange cmdlet runs (see these examples to see how to use cmdlet extensions to set various mailbox settings after creation and automatically provision an archive mailbox). It seems like not many people outside Microsoft have attempted to use the cmdlet extension agent so that it has to process a set of objects piped to it by a PowerShell cmdlet such as Get-Mailbox. Now it can. What relief!
• An Exchange Server 2010 database store grows unexpectedly large. This is an interesting fix because the text of the KB article indicates that bloating seems to occur following the use of journaling to a mailbox or the installation of a third-party archive system (for example, Symantec’s Enterprise Vault). Basically, the background maintenance processes that Exchange uses to clean up databases by permanently removing unwanted items from databases hasn’t been doing its job at times and items have lingered on to become digital zombies. The problem is now fixed and further bloat arising from the same root cause shouldn’t occur in Exchange 2010 databases. However, even after you apply SP2 RU1, some bloat might linger in a database. As you can’t apply colonic irrigation to flush rubbish out of an Exchange database, the best idea is to move mailboxes from the affected database to a new database as this will ensure that the resulting database structure is efficient and bloat-free.
• The Exchange RPC Client Access service crashes when you send an email message in an Exchange Server 2010 environment: This bug occurs when the RPC Client Access service running on a CAS attempts to process some recipient data submitted by a MAPI client that hasn’t been validated. It seems unusual that such a bug would still exist when the RPC Client Access service has been in use for several years, so it must only occur under highly specific conditions.
• The RPC Client Access service may crash when you import a .pst file by using the New-MailboxImportRequest cmdlet in an Exchange Server 2010 environment: This is a classic example of how software is improved based on user feedback. PST import is a relatively new feature (introduced in SP1) and there are many crappy PSTs on user PCs, some of which contain information that is not in the form expected or required for import – so Exchange has a problem when it’s asked to ingest the “badly formed” items.
• The Microsoft Exchange Information Store service crashes in an Exchange Server 2010 environment: A crash in the Store is always worrisome but this one seems to be relatively minor. The problem seems to occur if the Store attempts to get message information from a null table – probably something that doesn’t happen a lot given that users tend to start filling up their mailboxes as quickly as possible after creation.

I don’t want to say that any of these bugs are unimportant. They clearly are to the folks that originally met the underlying problems when they deployed Exchange 2010 into production. Meeting a bug is always frustrating, especially when you’re unsure whether it’s you or the software that’s going wrong. However, my point is that the set of fixes incorporated into SP2 RU1 provide a reasonable indication of the kind of updates that you can expect in any roll-up release. My review of the set of fixes included in Exchange 2010 SP2 RU1 indicates that this is very much a “clean-up and keep maintained release” that doesn’t contain any earth-shattering improvements, unless of course you consider the implementation of prioritization for mailbox move requests as such an improvement. For some, this will be important, but others will simply leave MRS to do its stuff and process move requests as it has always done.

Should you deploy Exchange 2010 SP2 RU1 now? I believe that you should, with the caveat that you should first test the new software by running it within an environment that replicates the essential characteristics of your production systems. That way you’ll find out whether the software works for you and make sure that you don’t encounter one of the edge cases that cause problems for just your users.

Despite its need to compile many various assemblies and images, unless your server is old and slow (like me), the SP2 RU1 installation doesn’t take too long and it is a worthwhile activity to schedule for a weekend or other convenient time slot. Things will be a tad slower if you haven’t yet moved to Exchange 2010 SP2 as that process will require both an Active Directory schema update and a build to build update to SP2, plus a backup or two just to make sure that everything’s secured. Remember to run the installation using an account that has elevated permissions (in other words, run as administrator) to ensure that you don’t run into any problems. Overall, the whole exercise is still very manageable, doesn’t take long, and is worth doing.

– Tony

Update 17 February: After I posted this article, I received news about a glitch that affects Exchange 2010 SP2 RU1 where OWA connections can’t be proxied by CAS servers. The full write-up is on WindowsITPro.com. Microsoft’s version of the situation is posted on the EHLO blog

Follow me on Twitter to receive updates about new posts and other info!