As some of you know, I use Office 365 for my production email service. I also use SharePoint and Lync, but email is often the reason why people move to a paid-for cloud service. You might be surprised that I use Office 365 as clearly I have a long history with on-premises email systems and use them on a daily basis. However, I don’t have the time to take care of a mail server for the five or so people who want to use it (other home IT tasks take priority) and it makes much more sense to hand the task over to Microsoft. I suspect that the same is true for most small to medium businesses whose core competence lies elsewhere than mail.

I’ve been using Office 365 ever since it was first launched and was happy with the P1 plan (for small businesses and individual professionals). P1 delivers a lot of functionality and only misses out on features designed for enterprise customers, such as Data Loss Prevention (DLP), which are restricted to the “E” plans.

For many reasons too boring to mention here, I decided to upgrade my tenant domain from P1 to E3. Microsoft supports plan switching except in specific circumstances. One of these is when a plan is in trial status, which was the case for my tenant because it is linked to my MVP status. In any case, Microsoft suggested that I take out some paid subscriptions, delete the old plan, move to E3, and then revert to trial subscriptions, which is the route used to effect the change.

All seemed well except that I noticed that some of the extended functionality was missing from the Exchange Administration Center (EAC) and Exchange Management Shell (EMS). As it happens, I was specifically interested in transport rules and DLP because I have had to write and speak on the topic recently, so I quickly noticed the lack of DLP.

Exchange 2010, Exchange 2013, and Exchange Online all use Role-Based Access Control (RBAC) to ensure that users only see options that they are authorized to access. When the EAC is started, RBAC evaluates the options that are available to an administrator by checking their membership of RBAC role groups (such as “Discovery Management”) and displays options in the EAC UI based on these results. If you don’t see something in EAC it’s an indication that the RBAC evaluation has determined that you are not authorized to use the option, so it came as a surprise when I couldn’t see the DLP and transport rules because my account is a member of the tenant management role group. [For more information on RBAC, see chapter 4 of my Exchange 2013 Inside Out book]

RBAC evaluations also occur when EMS sets up a remote PowerShell session with Exchange. I checked with EMS and found that cmdlets such as New-DLPPolicy and New-TransportRule were not available, so it seemed like the RBAC evaluation was misfiring for some reason.

On May 13 I logged a call with Microsoft Online Support and was quickly led through a number of troubleshooting steps to define and scope the problem. The initial focus was to ensure that the RBAC configuration for the tenant (shown by Get-OrganizationConfig | Select RBACConfigurationVersion). The reported RBAC version was 15.0.918.8 whereas the proper value should have been higher (today it is 15.0.949.10 – I assume that this value has some relationship with the software version of Exchange Online running inside Office 365).

The RBAC configuration was duly updated but had no effect. Microsoft escalated the problem to second-line support on May 14. After further investigation the support engineer reported on May 16 that it appeared that the tenant had not been fully updated to E3. The necessary magic was performed behind the scenes to complete the upgrade and the DLP and transport rules options appeared in EAC and DLP.

All seemed well, except that an even more puzzling problem now appeared. I reported to Microsoft that I could no longer see any mailboxes through EAC. On May 21, I noticed that a public folder mailbox that had been set up under Plan P1 was missing. However, just like running Get-Mailbox in EMS to see mailboxes, I was able to run Get-PublicFolder and Get-Mailbox -PublicFolder to list all the public folders and public folder mailboxes.

Another strange detail was that when I created a new public folder mailbox, EAC listed it as having a secondary copy of the PF hierarchy. I was able to add new public folders, an action that forces a write to the mailbox containing the primary copy of the PF hierarchy, which proved that this mailbox existed somewhere in the organization even if EAC declined to acknowledge its presence.

Many interactions now took place between me, the support engineer, and (within Microsoft) various engineers to try and track down the problem. A week or so elapsed without much progress. Eventually, on May 28 the focus shifted to the browser and its interaction with Exchange. I was asked to capture network traffic with Fiddler and provide the data to Microsoft. This didn’t show much, but we established that:

• Mailboxes and public folders didn’t display when I ran EAC with Chrome (patched to 35.0.1916.114, which seems to be the latest version)
• Mailboxes and public folders appeared as expected up in IE11
• They also showed up when Chrome was run with an incognito window!

No mailboxes – at least, none that EAC can see

Hmmm… this was strange, especially because I have been using Chrome with Office 365 for the past three years and have never had an issue with EAC. Deleting all the browsing history, cached data, and cookies from Chrome didn’t help.

On May 29 the case was handed over to another support engineer and we started to look at different configurations of Chrome. I logged into Chrome on a MacBook Air and everything worked. I then logged out of (disconnected my account) Chrome on my PC and ran EAC and the problem was still present. I created a new user on my PC, logged into it, ran Chrome and everything worked.

The problem seemed to be linked to some association between my Windows account and Chrome. To test this theory, I removed Chrome from the PC and then reinstalled it from scratch and logged in using my Google account. I then connected to Office 365, ran EAC, and lo and behold, the problem had disappeared.

From a pragmatic perspective, it is satisfying to have a fully-functional EAC. On the other hand, the support situation is not as good. Yes, the call is closed because a solution was found. But we do not know why only the mailbox and public folder parts of EAC were affected when viewed through the Chrome browser. These are both elements that exist in Plan P1 and should not have been affected by the upgrade. We also do not know why everything worked perfectly when EAC was run connected to a Plan P1 tenant domain and why the problem happened after the E3 upgrade.

In short, it’s a mystery. Sixteen days after first registering the support issue with Microsoft Online Support and after many conversations and email interactions with three different Microsoft support engineers, I still do not know the underlying root cause. That’s not a good situation for anyone to be in.

Of course, you could assume that it’s a problem that is unique to Chrome but I don’t think so. The problem manifested itself with Chrome but I didn’t try Safari or Firefox. IE11 worked but Chrome has worked smoothly in the past. And anyway, Office 365 supports multiple browsers and Microsoft does their best to provide feature parity across all the supported browsers. And I do know that Microsoft is still plugging away to look for the root cause and has added some additional scenarios to its test harness to pick up similar problems in the future.

On the upside, I have learned more about how Office 365 support works and the difficulty that exists in debugging problems when you have no control over many of the moving parts that might be associated with an issue. It’s not a job that I envy.

Follow Tony @12Knocksinna