I flattened an Exchange 2013 server the other day. I don’t mean that I took the physical computer out into the parking lot and drove a large vehicle over it to reduce the server to so many random bits of metal. Instead, I did what Exchange administrators have done ever since Exchange 2000 came along when a server is proving truculent and Exchange (the product) won’t uninstall cleanly. I ran ADSIEdit and blew away the server object. End of story.

But it wasn’t really. In the old days, removing the server object with ADSIEdit was clean and efficient. You could then reinstall Exchange on the server and all would be well. Now, Exchange leaves traces of itself in many different places in Active Directory or the system registry, and generally it’s a real pain to find, remove, and validate that all vestiges of a server have truly been removed. In short, flattening with ADSIEdit is only the start of the process.

The support engineers in the Exchange product group are appalled by such behavior. It’s not polite nor the least bit subtle to remove a server so brutally. They prefer that you run the Setup program and take the uninstall option. This would be nice if it worked all the time but sometimes it just doesn’t. In my case, I had committed a major faux pas that prevented the uninstall process from completing.

When I installed the server, I told Setup to use C:\Exchange as the base directory. That installation failed (it was a beta version), so I restarted. The Exchange setup program is pretty intelligent and uses watermarks to know how far it had progressed before a problem occurred so that it can restart and not redo work. Unfortunately, I failed to input C:\Exchange when prompted by Setup and the program therefore used its default location, C:\Program Files\Microsoft\Exchange Server\V15. Setup ran through to completion but left a confused and bewildered server whose files were merrily scattered across the two directories. Hence the need for uninstall, frustration when uninstall didn’t work, and using ADSIEdit to remove the server from the organization.

A better approach might have been to rebuild Windows on the server and then use Setup’s /RecoverServer option, which takes the information held about a server in Active Directory and uses it to reinstall Exchange. Such an approach might have worked, but I concluded that the reinstalled server would probably have been as confused as the original.

It would be nice if Exchange offered a /DeleteAndRemoveNow switch for Setup that would blow away a server and remove every possible trace through brute force if necessary. Unhappily that request has fallen on deaf ears as the product group doesn’t believe that it’s necessary. But it is, especially in test labs or when administrators (like me) do stupid things to servers.

In any case, I did learn something from the experience. After reinstalling Windows from scratch and then Exchange 2013, I found that some weird results were reported when I ran the command Get-Mailbox –Monitoring to view the set of health mailboxes. You might think that this is an odd command to run and certainly not one used regularly. This is true, but I was investigating the depths of Managed Availability and this command reports the set of health mailboxes created in every database in an organization so that probes can create synthetic messages to test that mail flow and other components are working correctly.

As you can see from the screen shot, EMS reported a corrupted mailbox. In effect, some of the properties required for the mailbox were missing (database being one). Although the screen shot shows just one corrupted mailbox, I started off with many such mailboxes.

Inconsistent health mailboxes

I noticed that the corrupt mailboxes are reported as being associated with an object stored in the Monitoring Mailboxes organizational unit, a child of the well-known Microsoft Exchange System Objects (MESO) organizational unit. This was a surprise because Exchange 2013 used to create the disabled user objects associated with health mailboxes in the Users organizational unit. Apparently the change to MESO was made in Exchange 2013 CU1, something that passed me right by. The change makes perfect sense because most installations don’t like random objects showing up in Users; it’s much better when applications have their own location for data that they use.

But why had I some corrupt health mailboxes? The answer is simple: these are lingering traces of the databases that used to exist on the server that I flattened with ADSIEdit. Because the server had gone away, Exchange was not able to associate the user objects with the mailboxes in the now-departed databases. Hence corruption.

The solution was simple – remove the corrupted objects (using ADSIEdit of course). In this case, I knew that I had a backstop because if I made a mistake and deleted the wrong objects, the Microsoft Exchange Health Service would recreate the mailboxes the next time they are needed. Think of them as zombie mailboxes – they always come back from the dead.

Follow Tony @12Knocksinna