Exchange Unwashed Digest – November 2015

Here is a digest of the posts that appeared on my “Exchange Unwashed” blog on during November 2015. The Thanksgiving holidays distracted some in the U.S., but we kept on trucking along…

Building efficient keyword queries for eDiscovery searches in Exchange and SharePoint (Nov 26): These days it’s somewhat strange when software allows you free rein, which is why it seems funny to be able to build keyword queries for eDiscovery searches from scratch without any assistance from Exchange or SharePoint. Perhaps the developers believe that all administrators are perfectly fluent in the Keyword Query Language (isn’t everyone?). Or it’s their little way of warning the unknowledgeable away from the unintelligible.

Why Exchange 2016 ignores Outlook 2007 (Nov 24): You know what it’s like. You’ve just charted out how a new version of Exchange (in this case, Exchange 2016) can be rolled out into production when someone asks the “what about Outlook” question. No one likes going near user desktops and the thought of having to deploy a new version of Microsoft Office is enough to reduce a strong administrator to a quivering wreck, especially after the Office 2010 “ribbon” caused so much grief for some. But there’s no getting away from the fact that Exchange 2016 will simply not play ball with Outlook 2007. Something more modern is required, like a nice new Outlook 2016 client. That’ll do nicely.

SharePoint 2016, Microsoft Graph, Ingestion, and choosing a new smartphone (Nov 19): A miscellany of topics from a new beta version of SharePoint 2016 to preview versions of third-party archiving ingestion for Office 365 taking in an eBook that seeks to dispel cloud myths before arriving at the question of the moment – Windows Mobile 10 versus iPhone?

Sixty million Office 365 targets light up for developers (Nov 18): Being someone whose last programming attempts were based on Visual Basic 3.0, I don’t normally pay much attention to the world of development. But the Office Graph – now the Microsoft Graph – is a pretty fascinating project to weave an intelligent fabric from the many strands of Microsoft’s cloud services. The Graph isn’t a massive database. Instead, it’s an API and a lot of smarts that link data like user accounts, mailboxes, calendars, groups, and so on in a very approachable method for developers to do their stuff. So it’s interesting. At least, I thought so… Now just how do I connect to the Graph with Visual Basic again?

Why Microsoft decided to keep deleted items in Exchange Online and why it might be the right decision (Nov 17): When they made the original decision, I enjoyed a little rant at Microsoft’s decision to stop the Managed Folder Assistant removing items from the Deleted Items folder in Exchange Online. But after taking some time to reflect on the call, it might be that they made the right business decision because it improves the service for the majority of tenants. It might also have been the right technical decision too, even if some other routes were open. I still don’t like it very much, but I can live with the notion of vast amounts of junk piling up in user mailboxes around the world. Isn’t that why we have these massive mailboxes for anyway?

The not so boring version of how Exchange Online satisfies SEC rule 17A-4 (Nov 12): The pleasure of browsing through complex regulations produced by a government agency should never be underestimated, especially when accompanied by the terse commentary of a legal opinion. I guess Microsoft didn’t expect a huge amount of reaction from the community when they let everyone know how Exchange Online Archiving satisfies Rule 17A-4 Security and Exchange Commission (SEC). After all, a story about how Outlook supports “likes” and “mentions” is much more interesting… But some goodness lies in the information, so here’s the not-so-boring and totally not-a-legal-opinion of the technical side of the story.

Microsoft reduces price for Office 365 Import Service (Nov 10): I couldn’t believe it when I heard that Microsoft planned to charge $8/GB to process data through the Office 365 Import Service. It just didn’t make sense when you consider that it is in Microsoft’s best interests to have customers keep as much data as possible in the service. A protest duly erupted and the result is now seen in a rather dramatic reduction to $2/GB data for drive shipping. Network uploads are free. Which is how it should be.

Getting data into Office 365 is easy; not so straightforward to retrieve (Nov 5): The rush to embrace the cloud is a wonder to behold, but I wonder just how many people really think through the issues involved in making a retreat should such a (horrible) course of action becomes necessary. It is easy to move Exchange mailboxes to Office 365 and ingest other data (PSTs, SharePoint sites, file servers) using the Office 365 Import Service, but how do you realize the promise that “it’s your data” and take it back? It’s a hard question…

Google’s last gasp attempt to stop the Office 365 juggernaut (Nov 3): The news from Mountain View that Google would really like more customers to use Google Apps for Work should come as no surprise to anyone. The fact that they’re willing to let companies who have a Microsoft enterprise agreement to use Google Apps for Work free of charge until that agreement expires seems like a good way to entice companies to move out of the Microsoft orbit, but I’m not sure that these tactics will work. There’s just too much to do to move to Google when Office 365 is so much easier to get to…

I also published two articles this month. First, Multi-Factor Authentication and Office 365 – Better protection, better security on November 16 to discuss how MFA or modern autheFollow Tony @12Knocksinnantication now works pretty smoothly for Office 365 tenants, and A call to action: It’s time to eliminate PSTs on Nov 9, my polemic on everything that so so wrong with PST files…

Now on to December. More holidays, but lots more work to do.

Follow Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Office 365, Outlook | Tagged , , , , , , , , , , , | Leave a comment

Updates and new versions for Office 365 for Exchange Professionals


One year after we started to write the book, several thousand people have copies of “Office 365 for Exchange Professionals”. We are delighted at the feedback and positive comments that we have received. However, some questions have arisen as to how often we update the text and what right anyone has to receive an update, so here’s some information about editions and updates in an attempt to clarify the position.

We have published two editions of “Office 365 for Exchange Professionals” to date:

  • Firstedition: May 2015
  • Second edition: September 2015

The books are available online in EPUB (suitable for Sony Readers and iPads), PDF (best for PCs and Macs), and Amazon Kindle.

We plan to publish the 3rd edition, which will probably be renamed “Office 365 for IT Pros” to indicate the large amount of non-Exchange related material in the book, in March 2016.

Each edition includes a substantial amount of new material to reflect new functionality introduced by Office 365. For example, the second edition has some 270 pages of additional material compared to the first book. As an indication of what we’re thinking about for the third edition, it will cover material such as Delve Analytics and the Office 365 Planner, both features that have not yet been released by Microsoft. In addition, we will revise coverage of the existing content to remove redundant material and to add, expand, or correct information as required. We know that many changes are coming for Office 365 Groups, so chapter 7 is likely to be overhauled (again). We are also consider changing the chapters for the third edition to create a more coherent flow.

Readers get the book in three ways:

  • They purchase from and become a site member. Alternatively, they can opt not to become a site member, which then has consequences when it comes to new editions.
  • They purchase the Kindle format from (or one of its country subsidiaries). Note that both the first and second editions are available on Amazon. The first is priced higher than the second and its descriptive text makes it very clear that this edition is now obsolete. We have kept the first online to retain the reviews as you cannot transfer reviews from one book to another.
  • They receive a copy from our sponsor (Binary Tree) or from a company who buys some copies to distribute to their customers. For example, Microsoft is currently providing copies to select attendees at its cloud roadshow series and Mimecast has bought copies to provide to webcast attendees.

It is important to know that we update the current book on an ongoing basis. We do this to remove “bugs” (in this case, bad grammar or spelling errors), address errors that are found or caused because new information comes to light, or to add some important information that we feel should be in the book. This approach matches the ongoing development that occurs within Office 365. The current version is indicated by a date on the inside cover (see screen shot below). The most recent version is dated 25 Nov 2015.


We want to make updates available to people who have “Office 365 for Exchange Professionals”. Here’s the arrangement:

  • If you buy from, you can download free updates for the edition that you purchase if you took out site membership. You have to log on to download the new content from the site.
  • If you buy a Kindle version from Amazon, you can download free updates from You have to refresh your bookshelf to get the new content.
  • Free updates are not available to people who receive copies distributed by third parties. We provide updated content to companies who buy from us in order that they distribute the latest available text, but we don’t have a way to reach those who receive books in this manner thereafter.

It is also worth noting that if you have site membership of, we will offer you discounts as new editions appear. For instance, site members who had bought the first edition were able to buy the second edition for $10 instead of its full price of $44.95.

I hope that this clarifies the way that we organize editions and updates and helps you find the most up-to-date content.

Follow Tony @12Knocksinna


Posted in Exchange, Office 365 | Tagged , , , , | 2 Comments

Fitbit versus Microsoft: Charge versus Band 2


I’m somewhat of a geek, so the notion of wearing a device that provides data about exercise seemed like a good thing, if only because it would help me convince myself that exercise is something that needs to be fitted into my daily routine. After a small medical mishap at the end of 2013, I bought a Fitbit Flex. This is one of the simplest devices you can wear as its only interface is a set of five lights that illuminate to show progress towards your daily goal. In this case, to take 10,000 steps. offers a nice web site to track progress over time and I’ve been happily watch myself progress to a grand total (as of today) of 6.5 million steps or 4,866 km. I know that I would not have done this exercise without wearing a device, so that’s proof of their worth.

I went through several Fitbit Flex, mostly because they have a fiddly clasp that tends to fall off at the worst time, meaning that I have left devices behind in places like the Nice Airport car-hire facility. This fault and the lack of feedback from the device led me to by a Fitbit Charge, which I like very much. It’s simple to use but acts as a watch and provides feedback on steps taken, stairs climbing, calories consumed, and so on.

And then the Microsoft Band appeared. I didn’t like the first version because it was uncomfortable to wear and seemed expensive for what it delivered, so I waited for Microsoft to improve matters, which is what has happened in Microsoft Band 2 (on the right in picture above). I’ve been wearing the Band 2 along with the Fitbit Charge since the Band 2 was released on October 30 to figure out what I like about both and to make a decision as to what I’ll wear in the future.

The Band 2 is packed full of sensors and I delight in knowing my heart rate at a glance. People have been telling me for years that I am somewhat stressed when flying and this was confirmed by the finding that my heart rate increased from a normal 68-70 beats per minute to closer to 100 when flying.

You pay for the Band’s sensor capabilities with a short battery life. The Band 2 will last 48 hours on a charge, but that shortens to a day or less if you exercise with GPS turned on, which you’ll probably want in order to see maps of your course produced in the Microsoft Health dashboard. The Fitbit is simpler and offers much better battery life. I typically charge the Fitbit once a week whereas the Band 2 needs to be monitored daily to know whether it needs a quick charge. Charging is quick for both devices.

Speaking of dashboards, both Fitbit and Microsoft make data accumulated from the devices available for comparison and observation. Both are well laid out. I like the way Fitbit allows you to add activities that might not be recorded by the device (for instance, if you forget to wear it), while Microsoft’s provides more analysis of the data.

Both devices have companion apps available on all major mobile platforms. The Microsoft app (which I use on Windows Phone) has some additional bells and whistles, such as being able to change the tiles that show up on the device. For instance, I have no interest in accessing Cortana with the Band so I removed that tile. Speaking of tiles, the ability of the Band to display snippets for new messages (text and email) as they arrive is a nice feature. This depends on the presence of a Bluetooth-connected phone to receive the messages and broadcast them to the band. As you might expect, reading messages a couple of lines at a time quickly loses its appeal, but it’s a useful way to recognize when something important arrives and you need to take action.

Many will buy these devices to track the distance they cover on a daily basis through normal activities or exercise. All devices have their own way of measuring distance (the Band 2 uses GPS) and the two seldom agreed on the number of steps taken or the distance covered. That really doesn’t matter as long as you use one device on a constant basis and measure your own progress against whatever data it produces. I measured the GPS distance recorded by the Band 2 against Runmaster Pedometer, a GPS-capable exercise recording app running on my Lumia 1020. The Band 2 and the Lumia agreed within 2-3% most of the time.

I prefer the Band 2’s clasp. It is much more secure than the Fitbit Charge, which tends to pop out from time to time (but much less frequently than the Fitbit Flex). The Band 2 is comfortable to wear and I don’t notice it on my left wrist. I don’t wear a watch anymore and like the “glance” mode of the Band 2, which shows the time if you move your wrist. An important point is that the Band 2 is water resistant, which means that you have to take it off when showering. The Fitbit is happy to be clean and goes into the shower along with its owner.

The devices are obviously worlds apart from it comes to price so it’s a little unfair to measure the Fitbit Charge against the Band 2. There are more sophisticated Fitbit devices that are a better direct comparison with the Band 2, but I can only discuss what I have.

The Microsoft Band 2 is available for $249.99 at or £199.99 at while the Fitbit Charge comes in at $116.05 at and £64.95 at To make the comparison more accurate between the U.S. and UK, you have to convert to a common currency and factor in sales tax, which differs according to the state of purchase. Taking an 8% rate for sales tax, the figures are as show below. As you can see, the Microsoft Band 2 is much cheaper in the U.S. while less of a difference exists for the Fitbit Charge, probably because it has been available for longer and some discounting is present.

  U.S. U.K.
Microsoft Band 2 $249.99 + tax = $269.98 £199.99 = $314.51
Fitbit Charge $116.05 + tax = $125.33 £64.95 = $102.14

Overall, both devices do a good job of helping people become less of a couch potato (if you let them). I like the simplicity of the Fitbit and its battery life. I like the approachability of the Band 2 and the data that it produces and that’s the reason why I will probably persist with the Band 2.

Everyone is different and wearing one of the more sophisticated Fitbit devices might be sufficient to tilt the balance the other way. And of course, newer devices when released will up the ante and set new benchmarks for wearability, usability, battery life, data gathering, applications, and dashboards. In the meantime, I’ll get on with achieving my daily goal for kilometres covered and steps walked.

Update (21 Nov): The small tab on the side of the Fitbit that you press to see the clock or other data has snapped off and can’t be fitted back. I guess it’s the Band 2 from here on.

Follow Tony @12Knocksinna


Posted in Technology | Tagged , , | 2 Comments

Microsoft Cloud Roadshows use “Office 365 for Exchange Professionals”

I’m pleased to let you know that Microsoft has selected the “Office 365 for Professionals (2nd edition)” eBook for distribution to select attendees who attend the Exchange sessions at the Microsoft Cloud Roadshows that begun in Dallas on November 2 and will terminate in Hong Kong on March 10-11. Microsoft is providing copies of the book in PDF and EPUB formats. The version being distributed is up to date as at October 27.

Clearly the writing team is delighted that Microsoft is using our content in this way. We think it’s because the book provides a great overview of the potential of Office 365 for those moving to the cloud from an on-premises environment, including coverage of Delve, SharePoint Online, and OneDrive for Business and comprehensive treatment of hybrid deployments.

Office 365 doesn’t remain static and we are busy keeping track of changes as Microsoft rolls them out into the service. We expect to have a third edition available in the March timeframe that will include coverage of major new features that we expect to see introduced between now and then such as Office 365 Planner and Delve Analytics. I’ll post more news about the third edition as we progress.

If you’d like to get a copy of Office 365 for Exchange Professionals, you can do so from: (EPUB and PDF formats). The EPUB format is suitable for iPad and Sony eReaders. People who buy through can become members of the site and qualify for low-cost updates to new editions as they become available. (Kindle) – or local Amazon sites

Now back to the writing…

Follow Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Office 365 | Tagged , , , , | 1 Comment

All about the second Exchange Oscars award ceremony


The first “Exchange Oscars” happened at MEC 2014 as a way for the Microsoft Exchange product group to come together with the Exchange MVPs and gently roast each other while enjoying some food and drink. Of course, in 2015 MEC succumbed to the Ignite conference and no Oscars were awarded – until last night.

Thanks to the generosity of Netmail and Binary Tree, we took advantage of the annual MVP Summit to gather at the Tavern Hall in Bellevue and celebrate the second Exchange Oscars. I didn’t make it until the award ceremony (such as it was) was nearly ready to begin, but when I arrived it seemed like everyone was very jolly and had enjoyed the food and drink that was on offer. It was great to have Perry Clarke (head of Exchange development) and Phil van Etten (CEO of Netmail) join a good crowd of Exchange and Office 365 MVPs along with many famous members of the Exchange product group. And of course, the remnants of Flat Tony

Greg Taylor and Flat Tony's head (See for more information

Greg Taylor and Flat Tony’s head (hopefully the last appearance of this cardboard entity)

The avuncular and well-spoken Greg Taylor (he of elephant protocol handling fame) acted as master of ceremonies and handed out the Oscars (organized by Justin Harris – @NtExcellence) in the following categories. First, the awards to members of the Exchange group as voted for by the MVPs:

Most Helpful Product Group Member – The most-looking for a good end of year review award: Brian Day (or as he is known more commonly, Brain Day)

Best Product Group Presenter – The loves the sound of their own voice award: Greg Taylor (@GregTaylor_Msft)

Best Advice Given – The I wish I had been a therapist but I failed the exams award: Tim Heeney (who does a great job of helping people master hybrid deployments)

Best Tool – Which I thought was quite rude really, so let’s find out who the biggest tool in the chest is…  ExRCA (the Exchange Remote Connectivity Analyzer): Shawn McGrath and Brad Hughes

Best EHLO Blog Post (please add the post author’s name) – The please please please love me award: Ross Smith IV (who has written many good posts in the last year such as “Enabling BitLocker on Exchange Servers” (@RossSmithIV)

We then had a set of awards for MVPs as voted for by the members of the Exchange product group:

Best at knowledge sharing – Also known as the read my greatness, admire my greatness, even though all I did was re-tweet your greatness award: Jeff Guillet (@expta)

Best code contribution – the I have a real job and can actually write code unlike you bunch of tarts who simply write about code other people have written award: Glen Scales (@GlenScales)

Most promising MVP – The I haven’t yet become a parent and so I can still work all hours award: Andrew Higginbotham (@Ashdrewness)

Best use of social media (please add the author’s name) also referred to as the Tinder award: Michael van Horenbeeck (VanHybrid – @MVanHorenbeeck)


VanHybrid beams as Greg instructs him on the art of social media. MVP Frank Carius remains unconvinced that Twitter will ever succeed

Most influential blog (please reference the name of the blog and the author)  – aka the I’m going to just keep writing and writing until eventually it pays off award: Tony Redmond and his WindowsITPro blog


Kissing Greg on both cheeks to thank him for the Oscar. MVPs Justin Harris (left) and Thierry Deman-Barcelò aren’t too sure about what’s happening

Clearly there are many others who make a huge contribution to Exchange, the ecosystem that surrounds the product, and the community who deserve similar recognition. We’re just the folks who were fortunate enough this year.

According to its wiki entry and as far as I can recall at this remove, Microsoft released the first version of Exchange (4.0) to customers in March 1996. We’re rapidly approaching the 20-year anniversary and some debate occurred as to how this landmark should be recognized. Not many products have survived in such good shape, especially in terms of the transition to the cloud, and it would be nice to have an opportunity to celebrate all of the people who contributed to the success of Exchange. We’ll see what transpires here… Maybe it will be a mega-Oscars.

Follow Tony @12Knocksinna

Posted in Exchange | Tagged , , , | 4 Comments

Exchange Unwashed Digest October 2015

October 2015 started with the launch of Exchange 2016, included a fight about a report issued by a security company to show how user credentials might be exposed by OWA and ended at yet another conference in Las Vegas. Lots of things to discuss… Here’s what happened during the month on my Exchange Unwashed blog at

Understanding what the Exchange 2016 Preferred Architecture really means (Oct 29): As part of their launch of Exchange 2016, Microsoft is taking a harder line about the kind of designs that customers use in their deployments, which is why the Preferred Architecture exists. Of course, it’s only Microsoft’s view and you might have good reasons (like virtualization) to move away from their approach. All explained here.

Transition from on-premises to cloud reflected in Microsoft’s latest results (Oct 27): Microsoft’s Q1 FY16 results were the first using the new product categories so it was harder to make a direct comparison with the past, especially when currency fluctuations also exert an influence. But any look at their results demonstrate that Office 365 is still growing and that on-premises servers are declining.

Veeam and Office 365 – not today, maybe in the future (Oct 26): I visited the VeeamON conference in Las Vegas to get a view of what Veaam is up to and a good discussion about backups for cloud services resulted. In a nutshell, they’re waiting to see whether a true market develops.

What Exchange admins need to know about SharePoint Online backup and restore (Oct 22): You might know that Exchange Online disdains to take backups and depends on Native Data Protection. SharePoint Online is different and some backups are taken. It’s a small but important point to understand.

Creating a common groups platform for Office 365 (Oct 20): A follow up article to a previous post to explain what I would do to create a common technical framework for Office 365 Groups and Yammer to run upon. It seems strange that three years after the Yammer acquisition so much distance still exists between the two most common collaborative applications within Office 365.

Outlook’s likes and mentions really don’t float my boat (Oct 15): It’s Grumpy Old Man (GOM) time again. The news that Outlook and Outlook Web App are going to support Facebook-like “likes” and Twitter-like “mentions” brought a cold chill to my bones. I don’t like the ideas very much at all because I cannot see how likes will improve the quality or effectiveness of email communication. I am more positive about mentions. But at the end of the day, it all depends on how people use the technology. Some will like it (no pun intended). Others, like me, will simply be grumpy.

FUD continues over OWA backdoor exploit (Oct 13): I respect companies who focus on IT Security because their work keeps the bad guys away from essential IT systems. The research that they do leads to innovate techniques that detect, isolate, and deal with threats. It’s a difficult area of the industry because some of the challenges that these companies take on are mind-bendingly hard. And when something good happens, it’s right that a company who makes a breakthrough should benefit from that work. Unfortunately, all the whoo-hah about the OWA backdoor attack reported in early October obscured what seems to be some interesting techniques developed by Cybereason in favor of clickbait, press headlines, and page views. It would have been so easy to better present the offending report but that didn’t happen. Oh well…

The ongoing debate over Office 365 Groups and Yammer (Oct 8): One of the most common questions I see asked by Office 365 tenant administrators in Microsoft’s IT Pro Network is whether they should use Office 365 Groups or Yammer to help users to collaborate and work smarter. This is a complex question that cannot be adequately answered to the satisfaction of all in an article, but it’s a question that deserves some comment. So here goes. I now batten down my hatches in anticipation of some “forthright” comments!

Doubtful security report about OWA flaw gains headlines but offers little real value (Oct 7): FUD is a well-known term in the computer industry and it’s alive and well as evident in a recent report of an OWA vulnerability that emerged from security researchers Cybereason. The report is not well written and contains insufficient detail to know whether the issue it describes is real. I don’t think it is, but it’s hard to make the call without understanding how a hacker magically penetrated an Exchange server. Make your own mind up!

Microsoft intends to charge for the Office 365 Import Service (Oct 6): An interesting FAQ released to Microsoft partners contained the news that Microsoft intends to charge to allow Office 365 tenants to import PSTs and other data using the Office 365 Import Service. That news isn’t stunning because it’s a service and you can make the case that someone has to pay for the infrastructure, etc. etc. What was surprising was the price point. $8/GB is, shall we say, ludicrous? Anyway, now that we know what’s coming, it will be interesting to see how Microsoft prices the Import Service – and how third parties price to compete with the folks from Redmond.

Exchange 2016 debuts to delight on-premises customers (Oct 1): It’s October, it’s three years since the last version of Exchange Server was released, so it must be time for a new version. And so it is as Exchange 2016 makes its debut and becomes generally available to all who would like to install the new software. Of course, installing a new version (or an update) of Exchange is not something that anyone with an ounce of sense rushes to do, even in a greenfield environment, so the time clock starts now for the planning exercises that should lead to a wave of deployments as people who want to stay on-premises and run older versions seek to get to a version that will be supported for the next decade.

November is already proving to be busy. I guess it’s better that way than the other…

Stay in touch by following Tony @12Knocksinna

Posted in Email, Exchange, Office 365 | Tagged , , , , , , , , , , | Leave a comment

Exchange’s underappreciated single-page patching capability

I really like single page patching, the facility first introduced in Exchange 2010 to enable a database to detect that a database page is corrupt and to retrieve replacement data from another database copy. It’s one of those elegant pieces of functionality that have been introduced as Microsoft improves levels of automated management and self-monitoring throughout Exchange.

Single page patching is not something you would necessarily notice. Unless, of course, you are in the habit of scanning the application event log or have configured some software to do the job for you. Being invisible yet effective is a major advantage. What you don’t know won’t worry you – and why should you be worried if holes appear in database pages that can be automatically fixed by Exchange?

Before single page patching came along, administrators lived in fear of the dreaded -1018 error, which indicated that corruption had appeared in a database. Corruption arose from many sources, software bugs and storage controllers often being likely culprits. When a -1018 error struck, the only solution was to restore the database from backup and bring it up to date by replaying transaction logs, an activity that was long, tiresome, and prone to error. No administrator ever got out of bed singing the praises of having to do a database restore to fix an ailing Exchange server.

The advent of the Database Availability Group (DAG) created a different environment. Sure, Exchange 2007 had introduced multiple database copies (well, two…) with CCR and SCR cluster configurations, but CCR/SCR operations were manual and complex.

A DAG is usually a more complex configuration than any CCR/SCR cluster. However, Microsoft did a good job of hiding the complexity that exists in the Windows Failover Cluster underpinnings of the DAG as well as most DAG operations (restoring a lagged database copy remains something that could be more automated). And because a DAG supports up to sixteen copies of a database (more like a maximum of four in practice), it is logical to assume that even if corruption was to strike one copy, an uncorrupted version of the affected data should exist and be available elsewhere within the DAG. This is the central notion that lies behind single page patching.

When a corrupted page is detected in the active copy of a database, the Replication service is able to issue a form of “all-points” bulletin that goes to servers holding the other (passive) database copies to request the data necessary to patch the active copy. The signal (the “page patch request list”) is recognized when it arrives on the servers hosting the passive copies and the required data is transmitted back to the requesting server, which uses the inbound data to patch the active database. Any updates with the same data that arrive from other servers are ignored. Simple yet effective.

Different processing occurs when a corrupt page is detected in a passive database copy. A request for data is sent back to the server hosting the active database, a copy of the good page is sent back, and is applied to the passive copy. Again, simple and effective.

Most of the problems fixed by single page patching are in the -1018 category. Other issues (such as another infamous bugbear, the -1022 problem) can also be fixed using this method.

The best thing about this form of patching is that you will probably remain blissfully unaware that it has occurred. Unless, of course, you scan the application event log and find events like this:

Log Name:      Application
Source:        ExchangeStoreDB
Date:          11/12/2014 7:24:56 PM
Event ID:      129
Task Category: Database recovery
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      xxx
At '11/5/2014 7:24:54 PM' the Exchange store database 'DB1' copy on this server encountered an error. For more detail about this failure, consult the Event log on the server for other "ExchangeStoreDb" or "msexchangerepl" events. Page patching was initiated to restore the page.
Log Name:      Application
Source:        ExchangeStoreDB
Date:          11/12/2014 7:25:00 PM
Event ID:      130
Task Category: Database recovery
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      xxx
At '11/12/2014 7:24:58 PM', the copy of database ‘DB1' on this server encountered an error that it was able to repair. For specific information that may help identify the failure, consult the Event log on the server for other "ExchangeStoreDb" or "MSExchangeRepl" events. The Microsoft Exchange Replication service will automatically attempt to retry the operation.

Log Name:      Application
Source:        ExchangeStoreDB
Date:          11/13/2014 9:03:25 PM
Event ID:      104
Task Category: Database recovery
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      xxx
At '11/13/2014 9:03:24 PM' the Exchange store database DB2 copy onthis server experienced an I/O error that it may be able to repair. For more detail about this failure, consult the Event log on the server for other storage and "ExchangeStoreDb" events. Page patching was initiated to restore the page.

How often do these events occur? Hopefully, not often. A lot depends on the stability and capability of your storage infrastructure. High-end enterprise-class storage systems generally create fewer corruption events than low-end JBOD systems. Generally – not always.

I once asked some of the team that run the Exchange Online servers inside Office 365 how often they see events that result in single page patching. Their enigmatic answer was “often enough for the feature to be valuable”. Make of that what you will. All I’ll say is that the high availability story around Exchange would be very different if single page patching did not exist. And can you imagine how disruptive it would be within Office 365 if a database suddenly went bad and required human intervention to fix a corrupt page? That wouldn’t be good at all… and increasing automation is the reason why features like the Replay Lag Manager exist inside Exchange.

As Tim McMichael, Microsoft’s broken-cluster-and-high-availability fix-up guru (that’s not his real title, but I like it) observed at IT/DEV Connections in September 2014, just about the only downside of single page patching is that it can disguise some underlying hardware problem that hasn’t quite failed yet but will do so soon. Exchange is so good at patching that it can make a faltering storage controller look good… but not for long. After all, that would be a case of applying lipstick to a pig and the pig is still liable to burp. Or something like that.

Follow Tony @12Knocksinna

Posted in Email, Exchange, Exchange 2010, Exchange 2013, Office 365 | Tagged , , , , , , , , | Leave a comment