NFS and Exchange: Am I a sheep? Some at Nutanix think so…

Posted on June 25, 2014 by Tony Redmond

A number of people working for Nutanix have been in the vanguard of those who would like to see NFS supported as a valid storage option for Exchange deployments, some of whom cheerfully recommend that “customers continue to run Exchange on Nutanix containers presented via NFS.“ In my mind this is a bad idea because it puts customers into an unsupported configuration. Running what is often a mission-critical application in a deliberately unsupported manner has never really appealed.

As you might be aware, the current situation is that Exchange 2010 and 2013 supports three storage architectures: Fibre Channel, iSCSI, and DAS. These architectures are supported because Microsoft knows that they work well with Exchange and are implemented by a wide range of storage vendors in a predictable and robust manner, all of which creates supportable solutions for customers.

As I have discussed before, Microsoft has a number of technical issues that have to be resolved before NFS can be added to the list of supported storage architectures. We can argue this point to the nth degree, but it’s not me that needs to be convinced that NFS is supportable for solutions created by multiple vendors. At the end of the day, the vendors who want to sell NFS-based solutions have to convince Microsoft. And Microsoft have to be convinced that NFS will be implemented at the same high quality by every vendor – and that those vendors can support their hardware in the field if things go wrong.

I think Microsoft has another concern too: there doesn’t appear to be too much customer demand that Exchange supports NFS. At least, this was the strong impression formed when Jeff Mealiffe queried an audience of several hundred people during his “Virtualization Best Practices” session at MEC last April. Only 3 or so people put their hand up when asked. If the commercial demand existed, you can bet that Microsoft would be all over NFS like flies around cow dung.

Those who want to see NFS join the list of supported architectures are diligent about advancing their case across a range of social media. An example is a Reddit debate. Twitter is also a favourite medium, which we’ll get to shortly. Another is the write-up in a TechNet forum that sets out the case for Exchange to support VMDKs mounted on NFS datastores. This post makes the point that there is a “gap in supporting Exchange in VMware vSphere & KVM with datastores connect via NFS 3.0.” That gap exists but the evidence from the audience pool at MEC is that it is a relatively small gap in the overall context of the Exchange market.

In saying this I accept that the gap (in support) might be terribly important to certain companies who wish to standardize their virtualization platform around NFS. But that’s not evidence that consistent and perceivable customer demand exists in sufficient quantity to warrant Microsoft to do the work necessary to certify that NFS indeed works (in a supportable manner) with Exchange.

Remember that once a storage architecture is supported, that support has to stay in place for an extended period (probably ten years given Microsoft’s normal support policies) and that support must be available on a worldwide basis. Convincing management to sign up for the costs necessary to support another support architecture without evidence that this will do anything to increase either Microsoft’s profitability or market share is a hard case to argue. And this proposal has to be viewed in the context of an on-premises market that will decline over time as a sizeable chunk of the installed base moves to Office 365 or other cloud providers. So the case being advanced is a good technical debate that runs into rough waters when money and market issues are factored into the equation.

It does not seem like a good economic decision for Microsoft to support NFS, but does the technology work? It certainly seems like quite a few companies have followed the Nutanix advice to run Exchange databases on NFS storage. To demonstrate that everything works just fine, Nutanix explained in the Reddit debate how they ran a JetStress test. They reported that “the Exchange Solution Reviewed Program (ESRP) Jetstress test (ran) for 24 hours in a single Windows 2008 VM with 4 PVSCSI adaptors serving 8 x VMDKs hosted on a single NFS datastore and no surprise it PASSED with flying colours.” This is all very well, and I am sure that everyone associated with the test was tremendously happy.

However, running Jetstress is no way to prove supportability, which is the key issue debated here. Jetstress certainly allows you to stress storage subsystems to prove their stability, so the test simply proves that (a specific version of) Nutanix’s implementation of NFS works when exercised by JetStress. It does nothing to prove the case for supportability of NFS within Exchange deployments. More work would have to be done to ensure that a guaranteed standard of implementation and performance was achieved for NFS across the industry. A single test run by a single vendor against a single storage configuration proves nothing.

It’s also true that the configurations that are capable of passing the guidelines laid down in ESRP are often “odd” and impractical in terms of real-life deployments, so saying that a test was passed with flying anything really doesn’t help us understand how NFS would cope with Exchange over a sustained period across multiple software versions deployed by different people around the world.

Some comments from a Nutanix employee on my Twitter feed

Some comments from a Nutanix employee on my Twitter feed

Going back to Twitter, I received some advice from Nutanix employees that I should engage with their initiative to encourage Microsoft to support NFS. I should stop being a sheep and work to convince Microsoft to change their “outdated and embarrassing support policies.” Quite.

I found it odd to see the debate reigniting because Nutanix has a configuration that presents iSCSI storage to Exchange. Given that this configuration exists, I wonder why we have to debate NFS support at all. It would seem much more productive all round for Nutanix to tell customers to use the fully-supported iSCSI approach rather than expending all these cycles trying to convince Microsoft that NFS should be supported too.

In my past, I spent over a decade in Chief Technology Officer or Technical Director jobs at Digital, Compaq, and HP. As such, acknowledging that I have been known to be wrong (many times), I think I have a reasonable grasp of how to assess the import and worth of a technology. So I spent a little time looking at the information available on the Nutanix web site.

Up front I have to admit that I have no hands-on experience of Nutanix products. Their literature indicates that they have some interesting technology created by people with a strong track record in the storage industry. As such, its products are certainly worth checking out if you are interested in virtualization. However, I have some doubts about some of their statements concerning Exchange.

For example, browsing Nutanix’s “Microsoft Exchange Solution Brief,” I was interested to find the assertion “With more than 50% of Microsoft Exchange installations now virtualized, it is critical to select the right server and storage architecture.” Although no one could quibble with the advice, I think they are way off with the preamble. I have never seen any reliable data to indicate that the majority of Exchange deployments are virtualized and no evidence is offered by Nutanix to support this statement. And anyway, what do we mean by an installation being virtualized? Does it mean that a single CAS is virtualized? Or all CAS and no mailbox servers? Or all Exchange servers in an organization?

It’s absolutely certain that virtualization is an important technology for many who deploy Exchange but I don’t think it is so widely deployed as to now be in the majority. I asked Microsoft and they didn’t think so either but couldn’t comment publicly. I accept that the document is intended for marketing purposes but it is not good when the first statement in the text can be so easily queried.

Nutanix and Exchange High Availability

Nutanix and Exchange High Availability

Another question is how Nutanix positions their technology alongside Exchange high availability. The picture above is taken from a presentation given by Nutanix representatives to customers where the recommendation is to use VM replication to copy databases to a disaster recovery site. Another recommendation is to take a separate copy to use for litigation or on-hold purposes.

This arrangement smacks of the old-fashioned arrangements needed before the advent of the Database Availability Group (DAG) in Exchange 2010 where storage subsystems copy Exchange databases to a separate disaster recovery site. It doesn’t reflect current best practice, which is to stretch the DAG to cover servers in the DR site and use normal Exchange replication to keep database copies in the DR site updated so that they can be quickly switched into action should a problem affect the primary site.

Exchange has transitioned from depending on separate hardware replication to being able to provide this level of resilience within the appliance. I would have had no problem with the approach in an Exchange 2007 deployment but I think its value must be questioned in Exchange 2010 and 2013. It seems like a case where a product can do something so we’ll do it, even though a better method is available. I can see how the approach would work, but it’s always best to seek simplicity in DR and it seems better to allow the application to do what the application is designed to do in these situations rather than to layer on an additional technology to serve the same purpose. The case can be argued either way.

I also don’t understand the need for a separate immutable copy for litigation or in-place hold. These are features that are designed into Exchange 2010 and Exchange 2013 and you don’t need any additional hardware or software to gain this functionality. To be fair, some circumstances might require a configuration like this to satisfy a particular regulatory demand, but I can’t think of one right now.

I am not slamming Nutanix’s technology or the value proposition that they advance to potential customers. I rather like their can-do attitude and their willingness to take on the big players in the storage world (an example is in this blog about EMC) with new technology and approaches.  However, I do think that some Nutanix employees are prone to hyperbole and that the advice extended to the Exchange community is sub-optimum and in some cases (as in the solution brief), incorrect.

Perhaps Nutanix would be better to focus on building Exchange solutions based on their technology that embrace, enhance, and extend the functionality that’s already available in Exchange rather than starting crusades to support technology that does not seem to be wanted by many. At least, not by those who attended the recent Microsoft Exchange Conference, a group that I assume represents the core audience to which Nutanix wants to sell their products.

After all, if Nutanix can deliver iSCSI storage (a supported storage architecture for Exchange), why do they want to insist on campaigning for NFS? The “Nutanix Bible” and the blog referenced above indicate that this is possible, so I really don’t know why all the hot air exists around NFS.  I must be missing something!

Follow Tony @12Knocksinna

Posted in Email, Exchange, Exchange 2013 | Tagged , , , , , , | 6 Comments

My top 10 Office 365 Roadmap features

Posted on June 23, 2014 by Tony Redmond

O365Roadmap

As I discussed in an “Exchange Unwashed” post about Office 365 change management, Microsoft has adopted new methods to keep customers informed about new developments in the rapidly-changing world of Office 365. This is a necessary step not only because the cadence of change is much faster in cloud systems than it is for on-premises software but also because customers have no real control over what a cloud provider delivers. Once you sign the contract, you accept what you get, including all the changes.

Last week, Microsoft posted the Office 365 roadmap to describe features that are “launched” (generally available to all customers), “rolling out” (being phased in but not yet available to all – MAPI over HTTP is a good example), “in development” (in software engineering but not yet available even in beta), and “cancelled” (nothing so far).

It’s an interesting list. I took a good look at the 35 features in development with an aim of identifying my top ten. Here’s my list:

Feature Notes
Clutter (OWA) I like the idea of machine learning being deployed to help do a better job of filtering out the important messages from the stream of informative but perhaps unessential email that tends to clog up corporate mailboxes. This feature is one that will help people who receive several hundred messages daily. It won’t if all you get is a couple of new messages. The Clutter feature is for Office 365 only at this point and is not scheduled to be in the next on-premises Exchange release.
Public folder scalability improvements It’s good that Microsoft has listened to the flood of customer complaints that ensued after it was revealed that the modern public folder implementation was crippled by totally inadequate limits. There’s still more to do though. This fix will be provided to Exchange 2013 on-premises customers too.
“Oslo” and Office Graph Due to the demands of integration and the complexity involved in gathering the information presented in the Office Graph, this is another feature that will only appear in Office 365 for now. It’s also a feature that will be of most use in large companies where it’s hard to know who does what and who is the expert in what topic. I rate it highly in the list because I think it’s an interesting technical advance. Quite how users will take to the Graph is quite another thing.
Groups for Office 365 There’s no doubt that Office 365 offers a variety of ways to communicate from email (Exchange) to IM (Lync) to document collaboration (SharePoint) to “jamming” (Yammer). Groups is an attempt to make it easier to communicate across the various products. Microsoft says that “Creating a Group anywhere in Office 365 will automatically provision a Yammer conversation feed, inbox, calendar and document library where members can collaborate and work as a team.” But what happens when you don’t use Yammer?
Data Loss Prevention (DLP) in SharePoint Online I like the DLP features available in Exchange Online and Exchange 2013 but it is obviously better when sensitive data types can be protected both in email and in document libraries. Stretching DLP to protect SharePoint Online libraries is a logical step. No word yet whether this feature will appear in SharePoint on-premises, but I expect that it will.
Document Collaboration (OWA) This feature allows documents circulated in email to be edited by recipients using Office Web Apps and an updated copy automatically attached to a reply. No word yet on whether this will also be an Office 365 exclusive. I expect that it will be for now because not every customer has deployed (or wants to deploy) Office Web Apps because it’s an extra server, etc.
Increasing Recoverable Items Quota Compliance is of ever-increasing interest as society becomes more and more litigious. And when mailboxes are put on hold, they might remain in that status for many months, which means that the Recoverable Items folder rapidly fills up with retained items. Increasing the quota to 100GB makes a lot of sense. It’s an easy change too that you can make by running the Set-Mailbox cmdlet on an on-premises server.
Search Suggestions (OWA) At one of the “Support Unleashed” sessions at MEC, several people complained about the search capabilities of Exchange and remarked that different clients deliver different search results. OWA depends on the Search Foundation to provide it with search but this UI change seems like a good idea to help users find what they really want, without having to tear their hair out several times as unwanted results pop up.
Propose New Meeting Time (OWA) A feature in the “should have been in the product first time around” category, but nice that it’s finally going to happen. If a meeting request contains a bad suggested time, you can reply offering a better time. Sensible.
Preserving DL membership and BCC for in-place holds There’s not much information available about this feature, which is described as “This update that will make DL membership & BCC searchable for users placed on In-place Hold.” But I think anything that helps eDiscovery searches do a better job is  a good thing and I prefer it more than competing features such as “OneNote for Mac support for OneDrive for Business” or “Office Online in Yammer”, so I included it here.

You might well have a different list because your priorities and interest are obviously different to mine. For example, those who care about OneDrive for Business will like the fact that its storage limit is increasing to 1TB while companies who have invested in Security Assertion Markup Language (SAML) will like that the Office 2013 applications will soon be able to use third-party identity providers.

Of course, the devil is very much in the detail when it comes to implementation and we do not have full information about how these features really work and what, if any, requirements they place on end users. However, the sheer fact that we have an indication of what’s coming in Office 365 allows for better planning and preparation, and that can’t be a bad thing.

Follow Tony @12Knocksinna

Posted in Cloud, Exchange, Office 365 | Tagged , , , , , , | 4 Comments

Cloud services mean a very different support experience

Posted on June 16, 2014 by Tony Redmond

My recent experience of working through a problem with Office 365 support confirmed once again just how fundamental the paradigm shift is for IT professionals tasked with resolving problems after workload moves to the cloud.

Overall, the support experience was good. The problem was solved. The people I dealt with were professional and polite and seemed to be more competent than earlier encounters. Microsoft provided the necessary systems to track progress of the problem and I was informed of how things were going at every stage. Microsoft even assigned me an “Advocacy Manager” to keep track of the case. So I felt pampered.

However, the same issues with cloud support persist. Progress towards resolution was too slow. Too many questions were reiterated for no apparent reason. Escalation was slow and the amount of value added by first-level support was not as high as I had hoped. But we got there in the end.

Supporting issues with cloud systems is difficult. Can you imagine working as a front-line support professional who has to deal with customer problems? Gathering the necessary information to form a mental picture of the problem is the first challenge. Understanding its impact on customer operations is another. Figuring out how to proceed to a timely resolution is a third.

Of course, Microsoft has databases that record details of support incidents, their symptoms, and eventual resolutions. Sometimes a quick search will reveal an answer and an incident can be closed quickly. In other cases, like mine, the symptoms are not so clear or a match cannot be found by searching, so the investigation process requires substantially more effort.

Events unfold at a very measured and controlled pace as Office 365 support personnel are obviously guided by some very clear instructions about how to gather information, interpret the data, and move towards resolution. Like many other aspects of Office 365, a structured playbook dictates how support unfolds and escalation from first-level to second-level staff happens. This is how it should be as you can’t afford to have different support personnel take radically different approaches to problem resolution in an environment that supports millions of end users. Such a scenario is a recipe for poor customer satisfaction and erratic problem resolution. It would also likely lead to higher support costs and dissatisfied customers.

It’s also important to understand that none of the frontline people working in Office 365 support have no access to datacenters. These people are responsible for collecting and refining information so that problems are well understood, resolved if possible, and then passed on to engineering if a solution cannot be found. Complex problems will probably involve a number of interactions between support personnel and engineering. In these cases, the support personnel retain responsibility for communication with the customer and engineering works in the background.

The highly structured approach to problem resolution means that cases often take longer to resolve than you might anticipate. The slowness in steps unfolding is often the most frustrating aspect for IT professionals. In the on-premises world everything is available and all components can be interrogated and tested in a drive towards a fix. With the cloud, you’re dealing with an amorphous blob where any control that you have terminates as packets leave your network en route across the uncontrollable Internet to a cloud datacenter. You can change client or device settings, but that’s it. Everything associated with server processing is in the cloud and hidden from you.

All of this means that IT professionals have a very different role to play during cloud support incidents. Instead of taking responsibility for pursuing a problem from client to server to resolution, your role is to gather information and provide it to cloud support. The better and more accurate the information you provide, the sooner the problem will be resolved. It is all too easy to lead cloud support down a dead end and even easier to spend days waiting for them to figure out that they are in a dead end. Responding to support with accurate and timely information is the only real way you have of controlling how quickly resolution will happen.

User communication needs increased attention from IT professionals too. You might understand that control over problem resolution has been ceded to cloud support but an end user is unlikely to understand the complexities and nuances involved in moving work to the cloud. All they know is that their email doesn’t work or they can’t get to a document in a SharePoint library. It seems pretty clear therefore that IT professionals have to do more work to reassure end users that their issues really are being worked.

Some might decry the transition of responsibility to the cloud. But similar changes have occurred before in both IT and elsewhere. We don’t have to load programs from paper tape now and cars are less maintainable than before because they use so many computers. You don’t see people worrying that they can’t use paper tape programs and few really mind the passing of fallible carburettors that tended to fail on damp mornings. We adapt and get on with the new situation.

Follow Tony @12Knocksinna

Posted in Cloud, Office 365 | Tagged , , , | 9 Comments

Progress in raising public folder limits – but we’re not there yet

Posted on June 11, 2014 by Tony Redmond

It was great to see the EHLO blog informing the Exchange community that work is progressing to increase the scalability of modern public folders and raise the somewhat pathetic limits revealed last March. As you’ll recall, the old limits were 10,000 public folders in the hierarchy and 100 public folder mailboxes.

To be fair to the public folder team, they walked into the lion’s den at the Microsoft Exchange Conference (MEC) when they faced an unhappy crowd at the “Unplugged” session for public folders and site mailboxes. Not that site mailboxes were discussed much at that session, which probably reflects the current level of usage for site mailboxes in the wild.

In any case, public folders program manager Kanika Ramiji kept calm and noted the three issues that customers wanted Microsoft to deal with in modern public folders. The first is scalability, and that’s being worked on right now. It’s good that the limit will be raised soon to allow Office 365 tenants to use 100,000 public folders. It’s even better that the scalability work will continue and show up in Exchange 2013 CU6 (due in less than a quarter now) and that the final goal is to support more than 1 million public folders.

As noted in the blog post, scalability improvements take time because they are complex. Microsoft has already enjoyed some robust customer feedback over the initial limits; they won’t want to experience the same reaction if modern public folders fail to scale properly again.

But you’ll notice that nothing was said in the blog post about raising the limit of 100 public folder mailboxes within an organization. On the surface, this shouldn’t be a huge issue as a public folder mailbox is able to hold an awful lot of information.

Office 365 limits public folder mailboxes to 50GB and automatically splits public folders across new mailboxes when that threshold is reached. On-premises deployments can support public folder mailboxes of up to 100GB. However, this is a “soft” limit and a mailbox won’t stop working if it is forced to hold 100.1GB. Rather, it’s a matter of how supportable, reliable, and performant mailboxes are when they hold very large amounts of data.

I imagine that the need to restrict hierarchy updates is the reason why Microsoft is holding to 100 mailboxes for now. The primary public folder mailbox handles all updates to the hierarchy and then fans out updates to all of the other mailboxes, each of which holds a secondary read-only copy of the hierarchy. Updates are very smooth when servers are connected with high-speed reliable networks, but it’s easy to see how things might not work so well if Exchange had to process public folder hierarchy updates across thousands of mailboxes distributed in databases around a large network where some of the links aren’t quite as good as you’d like.

So you could be faced with the need to stuff a one million folder deployment into 100 public folder mailboxes. Hopefully everything will fit in 10TB. If not, it’s time to take out the pruning scissors before you start the migration process to move old public folders to Exchange 2013.

The other issues identified at MEC were lack of support for public folders holding calendar and contacts items in Outlook Web App (OWA) and the lack of reporting and management tools for public folders. The EHLO post says that the public folder team have the OWA limitation on their radar and will get to it after the scalability issues are fixed. If the scalability fixes come to on-premises customers in Exchange 2013 CU6, you can assume that it will take one or two further updates before support for calendar and contacts folders show up.

No word is available on what might or might not happen for reporting and management tools. Given the sad history of neglect on this front for public folders and other elements in Exchange, it would come as no surprise if progress is slow here. But we live in hope!

Follow Tony @12Knocksinna

Posted in Email, Exchange, Exchange 2010, Exchange 2013 | Tagged , , , , | 3 Comments

GAMME – an unfortunate name for Google’s new migration utility

Posted on June 4, 2014 by Tony Redmond

I know (because I have visited the Dublin Googleplex several times) that Google employs several thousand people in Ireland. Couldn’t one of them have looked at the acronym for their new Google Apps Migration for Microsoft Exchange (GAMME) service and raised a red flag that perhaps a second meaning can be attached to the name? Especially in Dublin, of course, where anything slightly broken or non-functional has long been referred to as “gammy.”

And the Urban Dictionary offers a definition for “gammy” as an “Irish word for useless” or “The sudden overwhelming shock of confusion, which is sometimes shortly followed by a groan of delirium.” I imagine that neither meaning was quite in the minds of the Google marketeers when they came up with the name for their new migration service. Or perhaps they never bothered to search…

Naming products can be problematic at times as I discovered when the “Electronic Business Documents” product that I worked on at Digital around 1990 was given a marketing make-over and transformed into “TeamRoute.” We then discovered that the name had a most unfortunate meaning in Australia. Unfortunately true knowledge only sank in when the audience at a DECUS conference in Sydney dissolved into a fit of giggles, much to the bemusement of the unfortunate presenter, who really just wanted to discuss how documents could be routed by email and didn’t realize just how amusing frequent references to “TeamRoute” were to his audience.

In any case, Google’s new service uses both Exchange Web Services (EWS) and IMAP4 to migrate mailboxes from Exchange to Gmail. IMAP4 is very much a lowest common denominator as its fidelity and capability in dealing with the Exchange Information Store are much more restrictive than EWS. But Exchange 2003, a major source of migrations at present since Microsoft stopped supporting this version last April, doesn’t support EWS and IMAP4 is the only way that you can move information from mailboxes on those servers to Gmail.

I wonder how many corrupt and unreadable items turn up when mailboxes move to Gmail and if GAMME has the equivalent of the “bad items limit” supported by the Exchange Mailbox Replication Service (MRS). The fact that MRS is happy to allow you to move a mailbox and have up to 50 corrupt items dropped (or “cleansed”) during the move without having to pass the “accept large data loss” parameter is telling in itself.

Indeed, one of the very best things that MRS does is to purge lingering item corruption from mailboxes. Exchange 2013 and Exchange Online are pretty good at maintaining the integrity of the MAPI properties for mailbox items but this was not always the case in the past where items were often cheerfully corrupted by Exchange itself, mobile clients (BlackBerry and calendar items being a major culprit), and Outlook add-ins. No doubt GAMME will perform a most wonderful spring clean for any mailbox it is called upon to move.

That is, of course, if GAMME is ever used in real volume. Although Google’s pricing is competitive, I personally feel that their client strategy is flawed because so many end users organize their working lives around Outlook. Things might be different if Google had a better Outlook integration than Gmail’s current IMAP4 connection, but they don’t and Outlook therefore remains a blocking factor in many instances.

Some companies have moved from Exchange to Gmail  and some of whom are even happy with their new email platform. There are some well-publicized examples of large organizations who have decided to switch from Exchange to Gmail, such as the case of the City of Boston that was announced in May 2013. And to be fair, that migration appears to have gone reasonably well (all migrations are painful) and has become a case study that is often cited for cloud usage in the public sector.

But I don’t see many companies that I know making a similar move. If anything, the move is from Gmail to Office 365, a process that Microsoft and a host of partners will be happy to help with. There’s certainly an opportunity at present to migrate a heap of Exchange 2003 servers to the cloud, a move that makes eminent sense for any small to medium company as they’ll probably get much better service from either Google or Microsoft than they can extract from an aging and now-unsupported Exchange 2003 server.

Apart from its awful name, GAMME is a good thing for Google to offer. I bet Glen Scales (master of EWS programming for Exchange) could build a GAMME equivalent in a couple of hours from the collection of EWS code snippets that he has published on his blog. Perhaps that’s where Google got some of the code ideas for GAMME?

Follow Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Office 365, Technology | Tagged , , , , , , , , , , , , | 2 Comments

Exchange Unwashed Digest – May 2014

Posted on June 2, 2014 by Tony Redmond

May 2014 brought a TechEd North America event and produced some interesting news from that conference. But there was lots of other stuff to discuss during the month. Here’s what I published on my Exchange Unwashed blog on WindowsITPro.com.

Learning about DLP in SharePoint Online – a most unusual activity for an Exchange person (May 29): I’ve covered quite a lot of Data Loss Prevention (DLP) content recently and went back to the well to listen to a TechEd session that discusses a couple of interesting points about the implementation in Exchange 2013 and how DLP will be extended to cover SharePoint 2013 in the near future.

Exchange 2013 CU5 – the totally unremarkable update (in a good way) (May 27): Thirteen weeks after Microsoft released Exchange 2013 SP1 the next cumulative update arrived. In this case, it was a blessed relief to find that the quality was good and that no major upheaval was expected during deployment. A number of points have emerged since CU5 appeared and I have updated the post with new information as items appeared.

How new stuff is introduced through Office 365 Change Management (May 22): It takes a lot of work to keep a cloud service like Office 365 constantly refreshed. But that work is much easier when you run the service. It’s harder for every customer to know what’s coming down the line. Microsoft has acknowledged that this is the case and have vowed to improve matters. I guess we’ll see…

Microsoft pulling back on some MEC commitments and a rather delicious rumour (May 20): The MEC commitments were to provide the “Office Graph” and “Clutter” features to on-premises customers in the next major release of Exchange. That appears to be impossible for now because of the resource demands of machine learning and the need to refine algorithms and data very frequently (something that might be difficult for on-premises customers to manage). So these features will be Office 365-only for the foreseeable future. The rumour is that Microsoft will announce a new on-premises version of Exchange at TechEd EMEA in Barcelona at the end of October. Everything has been very quiet since the rumour appeared. I’m not sure if it will happen. But it might.

The case of the apparent log roll (May 15): Log roll is a technique that is used in Exchange 2007 and Exchange 2010 to force a closure of the current transaction log for a mailbox database and the subsequent replication to servers that host database copies. It is not used in Exchange 2013 because techniques like block mode replication have rendered the mechanism obsolete. But I saw, or thought I saw, log roll on an Exchange 2013 server. In fact, it was all to do with Managed Availability…

If Gmail had a reasonable client (like Outlook) (May 13): I’ve long thought that Google Apps and Gmail would be a much better contender in the enterprise space if they had a good integration with Outlook. The IMAP4 integration is barely acceptable as it’s based on a very old protocol and doesn’t exploit the capabilities of either server or client. But they don’t, so I think Google will continue to struggle against Office 365.

The irritation of mailbox database pings and event 6002 (May 8): There are places in Exchange 2013 where the fit and finish is not what it should be. In this case, tons of 6002 events are generated by pings that deliver absolutely zero value. The good news is that a fix is apparently on the way, which proves that complaining about this stuff sometimes works. I think!

Why you absolutely cannot upgrade Windows on an Exchange server (May 6): The arrival of Windows 2012 R2 support in Exchange 2013 SP1 caused a flurry of speculation that it would be good to be able to update Windows 2012 servers to R2 and then upgrade Exchange 2013 to SP1 (or even CU5 now that it’s available). But you can’t. It’s a rotten idea and you’ll fail if you try. So there. Just don’t do it.

Strong speaker line-up for Exchange Connections 2014 (May 1): I am the chair for the Exchange Connections conference at the Aria Hotel in Las Vegas next September 15-19. The great thing is that we’ve been able to secure a fantastic line-up of speakers that underline the aspiration of the conference to be the premier independent event for all things related to Exchange. It will be very different to TechEd or MEC. Mostly because the Exchange MVPs will go wild… People like “Exchange Server Pro” Paul Cunningham (who seldom speaks outside Australia) or Michael Van Hybrid. Regretfully we won’t have Steve “PST” Goodman because he has a pressing family event, but the team will more than make up for him.

Now we’re into the summer months but the news about Exchange, Exchange Online, Office 365, and even a little SharePoint continues unabated. There’s lots to do, so I had better get started.

Keep up to date by following Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Exchange 2013, Office 365, Outlook | Tagged , , , , , , , , , , | 2 Comments

The odd case of Chrome and the disappearing Office 365 mailboxes

Posted on May 30, 2014 by Tony Redmond

As some of you know, I use Office 365 for my production email service. I also use SharePoint and Lync, but email is often the reason why people move to a paid-for cloud service. You might be surprised that I use Office 365 as clearly I have a long history with on-premises email systems and use them on a daily basis. However, I don’t have the time to take care of a mail server for the five or so people who want to use it (other home IT tasks take priority) and it makes much more sense to hand the task over to Microsoft. I suspect that the same is true for most small to medium businesses whose core competence lies elsewhere than mail.

I’ve been using Office 365 ever since it was first launched and was happy with the P1 plan (for small businesses and individual professionals). P1 delivers a lot of functionality and only misses out on features designed for enterprise customers, such as Data Loss Prevention (DLP), which are restricted to the “E” plans.

For many reasons too boring to mention here, I decided to upgrade my tenant domain from P1 to E3. Microsoft supports plan switching except in specific circumstances. One of these is when a plan is in trial status, which was the case for my tenant because it is linked to my MVP status. In any case, Microsoft suggested that I take out some paid subscriptions, delete the old plan, move to E3, and then revert to trial subscriptions, which is the route used to effect the change.

All seemed well except that I noticed that some of the extended functionality was missing from the Exchange Administration Center (EAC) and Exchange Management Shell (EMS). As it happens, I was specifically interested in transport rules and DLP because I have had to write and speak on the topic recently, so I quickly noticed the lack of DLP.

Exchange 2010, Exchange 2013, and Exchange Online all use Role-Based Access Control (RBAC) to ensure that users only see options that they are authorized to access. When the EAC is started, RBAC evaluates the options that are available to an administrator by checking their membership of RBAC role groups (such as “Discovery Management”) and displays options in the EAC UI based on these results. If you don’t see something in EAC it’s an indication that the RBAC evaluation has determined that you are not authorized to use the option, so it came as a surprise when I couldn’t see the DLP and transport rules because my account is a member of the tenant management role group. [For more information on RBAC, see chapter 4 of my Exchange 2013 Inside Out book]

RBAC evaluations also occur when EMS sets up a remote PowerShell session with Exchange. I checked with EMS and found that cmdlets such as New-DLPPolicy and New-TransportRule were not available, so it seemed like the RBAC evaluation was misfiring for some reason.

On May 13 I logged a call with Microsoft Online Support and was quickly led through a number of troubleshooting steps to define and scope the problem. The initial focus was to ensure that the RBAC configuration for the tenant (shown by Get-OrganizationConfig | Select RBACConfigurationVersion). The reported RBAC version was 15.0.918.8 whereas the proper value should have been higher (today it is 15.0.949.10 – I assume that this value has some relationship with the software version of Exchange Online running inside Office 365).

The RBAC configuration was duly updated but had no effect. Microsoft escalated the problem to second-line support on May 14. After further investigation the support engineer reported on May 16 that it appeared that the tenant had not been fully updated to E3. The necessary magic was performed behind the scenes to complete the upgrade and the DLP and transport rules options appeared in EAC and DLP.

All seemed well, except that an even more puzzling problem now appeared. I reported to Microsoft that I could no longer see any mailboxes through EAC. On May 21, I noticed that a public folder mailbox that had been set up under Plan P1 was missing. However, just like running Get-Mailbox in EMS to see mailboxes, I was able to run Get-PublicFolder and Get-Mailbox -PublicFolder to list all the public folders and public folder mailboxes.

Another strange detail was that when I created a new public folder mailbox, EAC listed it as having a secondary copy of the PF hierarchy. I was able to add new public folders, an action that forces a write to the mailbox containing the primary copy of the PF hierarchy, which proved that this mailbox existed somewhere in the organization even if EAC declined to acknowledge its presence.

Many interactions now took place between me, the support engineer, and (within Microsoft) various engineers to try and track down the problem. A week or so elapsed without much progress. Eventually, on May 28 the focus shifted to the browser and its interaction with Exchange. I was asked to capture network traffic with Fiddler and provide the data to Microsoft. This didn’t show much, but we established that:

  • Mailboxes and public folders didn’t display when I ran EAC with Chrome (patched to 35.0.1916.114, which seems to be the latest version)
  • Mailboxes and public folders appeared as expected up in IE11
  • They also showed up when Chrome was run with an incognito window!
No mailboxes - at least, none that EAC can see

No mailboxes – at least, none that EAC can see

Hmmm… this was strange, especially because I have been using Chrome with Office 365 for the past three years and have never had an issue with EAC. Deleting all the browsing history, cached data, and cookies from Chrome didn’t help.

On May 29 the case was handed over to another support engineer and we started to look at different configurations of Chrome. I logged into Chrome on a MacBook Air and everything worked. I then logged out of (disconnected my account) Chrome on my PC and ran EAC and the problem was still present. I created a new user on my PC, logged into it, ran Chrome and everything worked.

The problem seemed to be linked to some association between my Windows account and Chrome. To test this theory, I removed Chrome from the PC and then reinstalled it from scratch and logged in using my Google account. I then connected to Office 365, ran EAC, and lo and behold, the problem had disappeared.

From a pragmatic perspective, it is satisfying to have a fully-functional EAC. On the other hand, the support situation is not as good. Yes, the call is closed because a solution was found. But we do not know why only the mailbox and public folder parts of EAC were affected when viewed through the Chrome browser. These are both elements that exist in Plan P1 and should not have been affected by the upgrade. We also do not know why everything worked perfectly when EAC was run connected to a Plan P1 tenant domain and why the problem happened after the E3 upgrade.

In short, it’s a mystery. Sixteen days after first registering the support issue with Microsoft Online Support and after many conversations and email interactions with three different Microsoft support engineers, I still do not know the underlying root cause. That’s not a good situation for anyone to be in.

Of course, you could assume that it’s a problem that is unique to Chrome but I don’t think so. The problem manifested itself with Chrome but I didn’t try Safari or Firefox. IE11 worked but Chrome has worked smoothly in the past. And anyway, Office 365 supports multiple browsers and Microsoft does their best to provide feature parity across all the supported browsers. And I do know that Microsoft is still plugging away to look for the root cause and has added some additional scenarios to its test harness to pick up similar problems in the future.

On the upside, I have learned more about how Office 365 support works and the difficulty that exists in debugging problems when you have no control over many of the moving parts that might be associated with an issue. It’s not a job that I envy.

Follow Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Office 365 | Tagged , , , , , , , , | 4 Comments

Some factors that make it much easier to introduce new features in the cloud

Posted on May 28, 2014 by Tony Redmond

As you’re probably aware, Microsoft uses the same code base to deliver both Exchange 2013 on-premises and Exchange Online. New builds are generated from the base and first appear in Exchange Online and later in a quarterly cumulative update. It all seems like a reasonable arrangement that ensures that same the features will be delivered to all customers.

But we also know that this is not the case for major new features like the Office Graph, meaning that a functionality gap will soon open up between the on-premises and cloud versions, if it not already has. In other posts, I have noted that the complexity and cost of delivering something like the Office Graph underpins Microsoft’s logic in keeping these features cloud-only, but more pragmatic reasons might exist.

Take resources for instance. Microsoft can easily afford to take on the delivery of more complex software to cloud consumers because they have the cash, software engineering talent, hardware resources, instrumentation, and datacenters necessary to develop, debug, and refine software. Eventually the software might be in good enough shape to deliver to on-premises customers and if so, it can be slipstreamed into a cumulative update or new version.

In addition, the scale of Office 365 operations allows Microsoft to test and refine new software by selectively deploying it to small user communities running in a highly structured and instrumented environment. Although MAPI over HTTP is already available to on-premises customers, Microsoft is using an incremental roll-out to determine its performance characteristics and sort out final details, like allowing the protocol to access legacy public folders.

Tens of thousands of users connect to Exchange Online with MAPI over HTTP today and more will be connected over the coming months. The switchover will occur gradually to allow Microsoft to tweak code to increase stability and throughput. The results will then be seen in updated code provided to on-premises customers in addition to better documentation, knowledge about network performance, and so on.

The Office 365 change management session at the recent TechEd North America conference described how tenants will be able to nominate themselves to access features earlier than others. This is another way to ease the roll-out of new code in a very structured and observed manner to cloud users that isn’t possible when software is released to on-premises customers.

Another factor is that Microsoft doesn’t have to deal with third-party add-on products inside Office 365. Removing these variables makes it easier to assess and test new code and move it to a point where it will be better able to cope with all of the different ways that Exchange is deployed inside on-premises environments.

Having Microsoft do all the work to refine new software before it is exposed outside Office 365 seems to be a good deal for on-premises customers. After all, no one likes to deploy buggy and immature code. It’s so much better when code works as it should. The days are long gone when it was amusing to have to track down an obscure registry hack that was necessary to make software work properly (or at all).

Of course, Microsoft could release code to their “Technology Adoption Program” (TAP), which allows customers to access new versions very early. This mechanism has worked well in the past and has been the source of valuable information about how Exchange behaves inside very different environments, including its interaction with third-party products. However, the TAP is now running into problems because of the accelerated cadence of software delivery. TAP participants are expected to deploy new builds into their environments to provide feedback to Microsoft about how the software behaves when exposed to different circumstances.

Getting the time and resources necessary to deploy new builds of major products like Exchange is reasonably easy when a new build is provided every few months over a year-long TAP such as the one that ran for Exchange 2007 or Exchange 2010. Getting the same time and resources every few weeks to deal with cumulative updates of Exchange 2013 (which are in effect brand-new versions of Exchange) is a different matter and I think that the lack of feedback from customer testing has not helped Microsoft achieve stability in some of the cumulative updates issued to date.

All in all, it makes a heap of sense to use Office 365 as the testing ground for complex new features like Office Graph before they are unleashed upon on-premises customers. It’s not as if customers are clamouring for Office Graph – and if they really, really want Office Graph in the near future, they can sign up for Office 365.

Follow Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Office 365 | Tagged , , , , | 3 Comments

New technology changes comparison between on-premises and cloud software

Posted on May 21, 2014 by Tony Redmond

When cloud services first became available, it was said that the most dangerous weapon in IT was the article in an airline magazine that attracted the attention of a CIO to a cheap monthly payment per user. Enthused by the notion that they could replace unpredictable and arbitrary costs for running IT systems with a nice, clean monthly bill from a service provider, the CIO returned to their company with all of the zealot passion of the converted and caused havoc for their staff.

Of course, this is a simplistic view of the world and CIOs don’t make long-term strategic decisions on the basis of some dubious assertions by journalists. The articles never explored the costs that lurked behind the monthly bill, but that’s not the kind of detail you can expect over three pages and 1,200 words. For whatever reason, the notion of a cheap fixed price took hold and we’ve been having discussions on that basis ever since.

Indeed, all cloud providers use easily understandable pricing to attract customers, whether it’s the $6/mailbox per month from Office 365 or $50/year from Google Apps. All so much more straightforward than having to grapple with complicated budget calculations involving capital purchases (servers, network equipment, and so on), people costs, and other ongoing expenses like software licenses.

The fixed cost element of using cloud services is easy to understand, easy to bill, and easy to pay and there’s goodness in all of that. Unfortunately, as those who have moved to cloud services understand, a host of other costs hide behind the up-front supermarket-like headline charge. Migrations are seldom joyful and always expensive. Network costs are different because traffic is diverted to external destinations rather than saying within company boundaries. Hybrid connectivity might have to be set up and managed. Single sign-on is highly desirable and needs to be done. The list goes on.

It’s true that a time comes when cost stabilizes and it becomes much easier to understand and measure the true cost of cloud services. It’s also true that this cost is often less than the equivalent of running on-premises servers, if you are able to accurately measure the true on-premises cost. If you are happy with the functionality delivered by the cloud service and its relative lack of flexibility (in terms of customization and the ability to deploy add-on software), then the decision to move to cloud-based services is a good one.

Many small to medium companies are in this situation. They were probably not very good at running IT anyway because it’s not the core competence of the company. Talented individuals working very hard cannot overcome the under-resourcing and aging equipment often found in companies where IT is not a priority. Moving over to cloud services probably results in more functionality, higher availability, and lower costs, all of which are good things. Even if support has improved over the last few years, it remains the Achilles Heel of cloud services and that’s a challenge that the IT department still faces after the transition.

The situation is different for larger companies, those that are more complicated because their operations are more widespread and distributed and cover more areas of activity. These companies tend to have more IT staff and better operations and a fair amount of bespoke code to deal with specific business requirements. It takes a lot more planning, effort, and expense to move these companies to cloud services. The fixed cost argument is not as attractive because in many cases these companies enjoy substantial discounts from IT suppliers for their current systems.

The cloud services provided to date are often pretty utilitarian. The core email functionality of sending and receiving messages is very much a utility operation. Even the more sophisticated aspects such as transport rules or even Data Loss Prevention policies can be delivered in a highly structured and automated manner. All of this is the reason why email is one of the first services that companies consider moving to the cloud. And because the functionality is well-understood and everyone shares many of the same requirements when it comes to email, service delivery is usually pretty good. Moving applications to SharePoint Online takes more effort.

But I think we’re on the cusp of an inflexion point when far more complex software can be better delivered (or only delivered) by cloud services. Software like the Office Graph (project “Oslo”), which Microsoft has talked up recently. The latest news is that it appears very unlikely that Office Graph will appear in a form that can be installed on-premises any time soon. In an interview with Network World, head of Office Technical Marketing Julia White said that Office Graph depends on the “massive compute power” that only exists in the cloud.

The news did not come as a shock because when you think about it, the usefulness of Office Graph depends on achieving an understanding of the many complex and interconnected links that people share within a company. The amount of machine learning that’s necessary to make something like Office Graph work well outside the restrictive boundaries of a software laboratory must indeed be “massive.” More interestingly was the statement that the “intelligence fabric” that represents Office Graph is only possible “because we can update the service with learning every single day.”

Three requirements would have to be met before something like Office Graph could be deployed in an on-premises environment. First, you’d have to be willing to deploy all the necessary software components. Second, you’d have to make large amounts of system resources available. My guess is a pool of dedicated servers and storage to process feeds from systems like Exchange and SharePoint (and other indeterminate sources including Lync and Yammer) that can be used to build the map of connected data presented by Office Graph. Third, you’d probably have to be willing to perform software updates on a very regular basis as Microsoft tweak the algorithms used by Office Graph.

Some companies would be interested and willing to plunge in and run their own Office Graph. I suspect that most others will be quite happy to leave all that work and investment to Microsoft, especially if the new feature is delivered as “evergreen software” covered their subscription (these features will likely only appear in the enterprise Office 365 plans). From the same interview, it seems like the “Clutter” feature, which also depends on machine learning to understand how people deal with new messages arriving in their inbox, falls into the same category.

So, you won’t benefit from new features like Office Graph unless you use Office 365, which then leads us to the conclusion that over time a growing gap is likely to emerge between the functionality that Microsoft delivers to its on-premises customers and that available in the cloud. Over time, it is possible that Microsoft will be able to ready features for transfer to on-premises servers (as they did with Managed Availability in Exchange 2013), but no guarantees exist at this point.

The financial debate about moving to the cloud is therefore not simply about comparing fixed versus variable costs. Some functionality that your company might want to use may be only available in the cloud. I don’t know whether this change will significantly affect the way that companies evaluate cloud services as a lot depends on the exact nature of the functionality gap that develops. That process won’t be fast and we won’t be able to evaluate the impact for a few years yet.

Follow Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Office 365 | Tagged , , , , , , | 6 Comments

Enterprise-grade cloud services are the new battleground between Microsoft and Google

Posted on May 14, 2014 by Tony Redmond

The first day of any large vendor-centric technology conference is invariably accompanied by a great deal of hype as the vendor takes full opportunity to strut their stuff and spread the good news. Keynotes full of bold new announcements are backed up by a blizzard of press releases, blogs, and other information, all of which is intended to result in positive coverage for the company and warm feelings all round.

I therefore wasn’t very surprised to see what happened on the opening day of TechEd U.S. last Monday. I watched Brad Anderson’s keynote online and thought it to be well up to the standard (his shoes are pretty amazing, however) with lots of attention given to new developments in Azure.

The keynote was then followed by the expected announcements and it’s taken a few days to separate the good stuff out. TechEd is a monster conference because it covers so much technology. I applied my usual filter and looked at the Office 365 information to get a sense of the messaging Microsoft is trying to communicate with its customers.

Security is the big theme. I guess this is unsurprising because last year’s PRISM revelations created a lot of difficulty for all cloud providers. Microsoft responded with a new Office 365 Trust Center to lay out how user data is protected in Office 365. The site contains a lot of interesting information as well as the obligatory swipe at Google (“Your data is not used for our advertising”) that is reinforced in their “10 questions you should ask your cloud provider” document.

The Office 365 Trust Center also documents a pretty impressive SLA figure (99.96%), which is in line with the quarterly SLA figures that Office 365 has delivered for the last two years. Gmail started off with a better SLA than Office 365 (largely due to some initial glitches in Office 365 management that have not reappeared), but things have gone awfully quite on the Gmail front recently and I can’t find any public record of Gmail’s current performance against SLA, even using Google Search. Google does maintain an Apps Status dashboard but I can’t get any SLA data from it.

Of course, the normal caveat applies in that all cloud services measure SLA performance at the boundary of their datacenters. In other words, that annoying but oh-so-essential thing called the Internet that connects users to cloud providers is factored out of the SLA equation.

Some observers have attempted to calculate “real SLA performance” using the reports published by cloud providers when incidents occur. These calculations are problematic because so little context is available to understand the exact impact of any reported problem. Nevertheless, zealots on both side of the argument do attempt to take a little data and make it into a story, like this only ever-so-lightly biased comparison of Exchange Online availability against Gmail published in 2013 by Cloud Sherpas, a company that I have noted before in terms of their ability to take information and present it in new and interesting ways.

Stripping away all of the marketing hoo-hah and fables, the key point is that all of the major cloud providers are delivering SLA figures for utility applications like email that few large corporate datacenters could match, which is a good thing.

The Office 365 Trust Center is also referenced in an accompanying blog post by Rajesh Jha that emphasizes the security capabilities required for “enterprise grade cloud services.” Again, this is a competitive poke at Google whose capabilities could be considered to be more focused on the needs of users and small businesses. Microsoft lays out its case for encrypted storage, mobile device management, and Data Loss Prevention for SharePoint (taking a similar approach to the DLP features available in Exchange).

Written words are all very well but a cogent message delivered with verve is usually a much better way of getting the point across. I enjoyed Vivek Sharma’s 5-minute video explaining the layers of security that protect Office 365 data at rest that you can find in the “Inside the Cloud” post. Vivek is a great presenter with bags of knowledge. His “Behind the Curtain: How we run Exchange Online” video from MEC is well worthwhile watching by anyone who wants to understand how a 100,000-server Exchange 2013 environment is automated to the nth degree.

Anecdotal evidence indicates that Microsoft has been very successful in moving small to medium businesses to the cloud. Larger businesses are usually more complex, distributed across multiple countries and more regulated, so they represent a bigger challenge for any cloud vendor, especially when questions about data security and privacy are raised. It seems pretty clear that Microsoft has identified enterprise-grade services as a good battleground to take on Google and that reassuring people that their data is secure at all times with Office 365 is a big part of that war.

Follow Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Office 365 | Tagged , , , , , , , | 1 Comment