SQL, OFS, Exchange file system, WinFS, and other murky dealings

Posted on March 11, 2013 by Tony Redmond

Like many people who are interested in Microsoft Exchange, I have followed the ESE versus SQL debate with some interest for as long as it’s been going on, or just about the entire lifetime of the product. ESE stands for Extensible Storage Engine (also known as JET-Blue, the Joint Engine Technology) and is the database engine on which the Exchange Information Store runs.

A recent post by Hal Berenson, a retired Microsoft Distinguished Engineer, illuminated some of the murky corners of the internal debate that occurred between different engineering factions in Microsoft in the late 1990s and early “naughties” as the company grappled with the challenges of unified storage. It’s worth the read, if only to learn about how technology sometimes evolves.

People have asked why Exchange doesn’t use SQL for years. On the surface, it does not make sense for Microsoft to have two competing database engines. Combining all resources to focus on the development and exploitation of a unified database engine would liberate a lot of engineering resources, reduce the need for product support, save money, and accelerate the development of new features, or so the convenient wisdom goes.

Of course, a lot of work would be required to move a product like Exchange from one database engine to another. The usual response from the Exchange development group is that they have experimented with SQL but that ESE continues to be the best choice for the kind of ad-hoc transactions typical of an email server. But technology has a habit of being adaptable to meet different needs, providing sufficient effort is expended, so I guess that it would be possible to create a SQL-based version of Exchange, should that decision be taken. For now, a change does not seem likely as the mantra is “The future is ESE. ESE is easy, SQL squeals like a pig.“, repeated by Microsoft speakers at events such as the late-lamented TEC and the Microsoft Exchange Conference.

But back to the past, it is fascinating to learn about the Object File Store (OFS) and how an early version of Exchange was supposed to move from ESE (described as an “interim store”) to use OFS. An Exchange General Manager (perhaps Brian Valentine) then revealed in email that he thought that Exchange would never move to OFS. Later on, the story evolves to JAWS (Jim Allchin’s Windows Storage), another candidate for the Exchange Store, and then a discussion about how Outlook might have used SQL Server 2000 (clearly not the full-fledged version) as a replacement for the PST structure that is still used today.

Lots of information about internal Microsoft politics is revealed. The struggle between Bob Muglia’s organization (including Exchange) and another (including SQL) is interesting, especially in how it influenced the development of the Exchange File System (ExIFS), a variant of which that shipped with Exchange 2000 and Exchange 2003 to provide the infamous “drive M:” that exposed the contents of the Exchange Store as a drive and encouraged weird behavior such as attempting to take file-level backups through drive M: – and be then disappointed when the backups could not be restored.

Later on, other efforts occurred to try and make development groups co-operate around a single vision of unified storage. SharePoint, described as “idiosyncratic” users of SQL, Outlook, Exchange, and Active Directory seemed to line up to work on a joint approach to move everything to SQL Server and then evolve towards a fully integrated unified storage system (later to become the WinFS project). During this period Exchange dabbled with SQL Server and even told partners that they would move to that platform in the “Kodiak” project. That never happened, possibly because Exchange then decided to wait for a server-based version of WinFS. As Berenson remarks “Apparently they are still waiting.”

As a partner working at Digital, Compaq, and HP, I followed the debate through many product briefings and other more informal (but highly informative) discussions. But outsiders can only ever have partial knowledge of what happens inside Microsoft as the more salubrious details are kept in-house. Hal Berenson’s story fills in a lot of the gaps. I suspect that many who currently work on Exchange don’t realize exactly what happened in the past. Hal’s position in the Microsoft engineering hierarchy equipped him with a unique perspective. I enjoyed reading his piece very much. You might too.

Follow Tony @12Knocksinna

Posted in Exchange | Tagged , , , , , , , | 3 Comments

Some side-effects of the Exchange 2013 cumulative update strategy

Posted on March 8, 2013 by Tony Redmond

Now that we have had a chance to digest the ramifications of the new cumulative update strategy for Exchange 2013 as laid out by Microsoft, it’s a good time to speculate how things will work in a practical sense.

Three points deserve some debate. The first is the cadence for updates. Three months is an eminently sensible period for “the service” (the name Microsoft uses for Office 365). The nature of cloud services and the competitive imperative to stay fresh means that new features “light up” on a frequent basis. I have no problem with Exchange Online and the other Office 365 products being updated every thirteen weeks, but I wonder whether the IT staff responsible for running on-premises Exchange servers will be quite so happy to be forced to maintain the same pace. After all, no other company has the same technical resources as Microsoft, and no other company has invested so heavily in standardization and automation within their datacenters, two of the basic principles that facilitate frequent updates.

Having a new cumulative update delivered every three months is unlikely to be welcomed. Being forced to update because the support policy dictates that only the two most recent updates are covered. The exact text in Microsoft’s statement is “A CU will be supported for a period of three (3) months after the release date of the next CU”. Three plus three equals a short six month period during which you had better deploy if you don’t want to be met with the stock answer of “deploy the current update” anytime you report a problem to Microsoft support.

So that’s two out of three issues that are pressing on my mind. The last is the effect that the new Microsoft-dictated schedule will have on ISVs. Exchange is not an island and few Exchange servers function without some form of non-Microsoft software that helps to keep the show running. Simple stuff like backup software, for instance. System administrators tend to like ISVs to verify their code against shipping software, which means that ISVs can only really begin to validate their products after Microsoft has shipped a new cumulative update. Of course, the magical mystery three-month clock has started at that point and you can bet your bottom dollar (or currency of choice) that it will take ISVs at least a couple of weeks before they make absolutely sure that their code runs well with Exchange 2013 CUx. So now we have maybe ten weeks to deploy, maybe less if one of the ISVs runs into some trouble with an API or other component. In short, it’s a bit of a mess.

The other thing to remember is that no one signed up for a frenetic update schedule when they committed to buy and install Exchange. No one. No customer. No ISV. No partner. No one. Microsoft has taken it upon themselves to impose a schedule that makes perfect sense for their Office 365 service while leaving everyone else gasping on the ground. It must be a new way of generating customer loyalty through exertion. You know… “the more software installs you do, the happier you’ll be with Exchange”.

Don’t get me wrong. There are some good things in Microsoft’s new update strategy such as the ability to deploy the latest CU in a one-time operation to install Exchange rather than having to deploy the latest service pack and then the latest update as happens with Exchange 2010. But the pain caused by the three-month cadence is too much and I think that Microsoft has to figure out a better approach.

No one will object if they continue to light up new features for Exchange Online every three months – but that’s a highly optimized and standardized massively scalable multi-tenant environment that is unique and totally unlike any customer on-premises infrastructure. You cannot fit a square peg into a round hole and you can’t force unnatural update practices onto Exchange administrators. Over to you, Microsoft – come back and tell us when you’ve figured out how to make on-premises customers happy. And by the way, three months should not figure in your new answer.

Follow Tony @12Knocksinna

Posted in Exchange 2013, Office 365 | Tagged , , , , | 23 Comments

Can Windows servers consume untreated Internet traffic safely?

Posted on March 6, 2013 by Tony Redmond

Last September, Microsoft dropped a bombshell when they announced that they were dropping development of the Threat Management Gateway (TMG) product along with their decision to cease production of on-premises ant-virus products. The problem for the Exchange community was that TMG had become the de facto choice as a reverse proxy deployed alongside Internet-facing Client Access servers to handle inbound client traffic.

Since the original announcement, Microsoft has done its best to reassure customers and explain that TMG support remains in place until April 2015. In a nutshell, although no more TMG licenses can be bought, you can continue to run TMG alongside Exchange 2007, 2010, and 2013 until support expires.

But thinking about the situation after a thought-provoking discussion with Greg Taylor of Microsoft, I wonder whether the function served by TMG and ISA Server, its immediate predecessor, is focused on the needs of the past rather than the present. If you go back to a time when Outlook Anywhere started to popularize HTTPS connectivity instead of running MAPI RPCs over a VPN, the target infrastructure was Windows 2003 servers and Exchange 2003 SP2. External threats abounded as hackers attempted to penetrate past corporate firewalls to attack unhardened internal systems, including Exchange.

So it was logical to deploy multiple levels of protection, starting at the firewall and going through servers to perform tasks such as packet inspection before any traffic was allowed to go to an internal server. The approach worked and has served IT well as long as IT exerted strict control over networks, devices, and servers.

The same conditions do not exist today. On the plus side, the latest version of Windows and application servers like Exchange are more secure than they were in the past, thanks to customer pressure to drive improvement and changes in Microsoft’s engineering practices to enforce “secure by design”. On the downside, infrastructures have to cope with connections coming in from a multitude of device types, not all of which are “approved” because of the popularity of BYOD.

The latest versions of Exchange demand nothing more than TCP (port 443) to be open on corporate firewalls before clients can connect. The question then is what additional processing needs to happen before a sanitized traffic stream from the firewall hits an Exchange server. And as it turns out, the answer is “not much”, largely because Windows and Exchange have the capability to protect themselves against suspect packets and because the latest generation of firewall-cum-load balancer products are capable of doing much more than simply blocking inbound traffic. If this assertion is true, then what value does a product like TMG or UAG deliver? And is that additional product even required to maintain a secure environment?

Strong opinions will no doubt be voiced on this topic. Security professionals take their job very seriously and abhor anything that might expose a company to risk. But in defence of advocating the heresy of passing traffic direct from firewall to Exchange, I point out that some in the security community have considered that erecting strong barriers and depending on them for protection against network threats has been a fool’s errand for many years. The Jericho Forum, part of the Open Group has led the charge to encourage the development of systems that can function without risk as part of the Internet without the kind of traditional barriers that have been erected to date. To get an insight into their work, you could do worse than reviewing a presentation called “The business case for removing your perimeter” given at the RSA conference in April 2008. It makes interesting reading.

I was responsible for HP’s security strategy during the 2004-2007 period. When I worked in that role I had the chance to debate the changing nature of security with members of the Jericho Forum. I always thought that they had interesting but maybe impractical ideas. Now it seems that their thinking might have been a little ahead of its time. Perhaps it is now appropriate to ask the question whether the now-traditional approach should be applied to protecting modern versions of Exchange and other Windows applications that are built to consume and filter HTTP traffic.

Security traditionalists and those who worry about protecting infrastructures against penetration will probably still argue that strong barriers have to be maintained. Their concerns should be taken into account when any security strategy is constructed as threat evolves and flexes all the time – and drives an entire industry dedicated to protection against malware, trojan horses, viruses, and the like. At the end of the day, the decision as to how deploy and protect servers depends on the security requirements and profile of individual companies, but I think that it’s worth thinking about how the attack surface of modern Windows servers differ from their predecessors and whether this influences your protection strategy.

Follow Tony @12Knocksinna

Posted in Email, Exchange, Exchange 2010, Exchange 2013 | Tagged , , , , , , , | 7 Comments

Exchange Unwashed Digest February 2013

Posted on March 4, 2013 by Tony Redmond

February was quite an interesting month, if you like tracking various bugs and arguing whether Exchange 2013 is fit for deployment. Here’s what happened in my “Exchange Unwashed” blog on WindowsITPro.com during the month:

Exchange 2010 Datacenter switchover tool (February 26): It’s often difficult to decide on the best possible way to convey information. PowerPoint can be overused and we’ve all heard about “death by PowerPoint”. But this tool is a rather nice way of using PowerPoint to bring administrators through the steps involved in performing an Exchange 2010 DAG datacenter switchover. Worth looking at!

Next Microsoft Exchange Conference scheduled for April 2014 (February 25): We might be all exhausted by the time the next MEC comes around because it will then be fourteen months of anticipation since Microsoft started the teaser campaign to inform people that yes, they will run another MEC, and no, it won’t be in 2013. Good things come to those who wait. We’ll just have to see…

Cloud Computing Security Handbook (February 21): I like giving praise where praise is due and I really like the new security handbook for cloud computing environments that John Rhoton et al. have released. Worth reading if you’re facing into the management of a cloud infrastructure. Your own or using someone else’s.

Office 365: Prepare Yourself to be Upgraded (February 19): The applications running in Office 365 are about to be refreshed so that tenants will run the Wave 15 versions (Exchange 2013, SharePoint 2013, etc.). This might or might not be a welcome surprise. I suspect that most people will welcome the update, even if they don’t realize that it has happened (for such is the promise of evergreen technology a la the cloud).

Apple iOS woes reveal the seamy underside of BYOD (February 14): We got excited about BYOD from 2007 onward – or at least, users did. Administrators thought that BYOD was the work of the devil and the latest batch of iOS problems that we have all been suffering through has blackened the BYOD concept. Some more control is needed. The question is how to apply the control and how users will accept that control.

Exchange 2010 SP3 is released – almost ready for Exchange 2013 deployments (February 12): Microsoft delighted and excited everyone by releasing Exchange 2010 SP3 and Exchange 2007 SP3 RU10. All seemed clear and ready to go with Exchange 2013 deployments until we all realized that it wasn’t quite the case. Exchange 2013 CU1 is required to complete the picture, but two out of three isn’t bad. Or is it?

Apple iOS 6.1 upgrades result in excessive transaction log growth (February 7): It’s been a bad period for iOS updates. Each one seems worse than the last and none cure the underlying problem. We’re still not altogether sure whether the latest code from Cupertino is acceptable. I’m sure that iOS will get there in the end, but it’s depressing that the problems are quite so long-lasting.

Touchdown–a solution for BYOD email? (February 7): I wrote this review after testing Touchdown on my Windows 8 PC. I could have waited and tested it on a Surface Pro, but in any case the same result would probably have occurred. Touchdown is an interesting email application that’s better and more secure than the standard Windows 8 mail app. But then again, who cares when you have Outlook?

Performance Monitor and Exchange DAG replication queues (February 5): I was rather disappointed that this post didn’t get as many views as I thought it deserved. The salient fact is that the only way that you can be sure of knowing what your replication queues are is to consult Performance Monitor counters. EMC and EAC both lie, but only because they’re stuck in a certain point in time. Haven’t we all been in similar situations?

Office 365 outages due to “routine maintenance” (February 1): Office 365 has received quite a bit of publicity recently, and rightly so because a lot of very interesting developments are happening in that space. But interesting developments turn to dust when the service is unreliable. Outages caused by routine maintenance don’t really create a whole heap of confidence. On the other hand, Office 365 usually provides an excellent service, so let’s hope that this is a glitch that won’t be repeated.

I have no idea what March will bring. Perhaps Exchange 2013 CU1?

Follow Tony @12Knocksinna

Posted in Cloud, Email, Exchange, Exchange 2010, Exchange 2013, Office 365 | Tagged , , , , , , , | Leave a comment

Bricked Lumia returns, great joy all round

Posted on March 2, 2013 by Tony Redmond

My bricked Nokia Lumia 800 returned from maintenance on February 25, some 27 days after I left it in because the phone had decided that it did not want to turn on. Nokia made several attempts to fix the problem and eventually had to replace the main board, work that was fortunately covered by the two-year warranty covering Nokia phones sold in Western Europe. The phone was wiped of data and applications but came back updated to Windows Phone 7.8, so some work was necessary to return it to my preferred configuration.

During the week of February 18, I was in Redmond for the annual MVP Summit, a gathering organized by Microsoft for their Most Valuable Professional community, basically a geek-fest that allows the MVPs to be updated about the latest and greatest developments in various Microsoft technologies. The trip was the first time since 1993 that I had been on the road without a mobile phone and the first since 2002 since I had lacked a phone with email capability. I could just about tolerate being without a phone in Dublin but being away was very different. In short, I was disconnected. In a big way.

To cheer myself up, I visited the Microsoft Store in Bellevue and bought a 64 GB Surface Pro, a device that is not yet available in Ireland. The Surface RT was launched here on February 14, some three months behind our U.K. neighbor. Even though Ireland is a reasonably small market in global terms, I thought this was a little strange because Microsoft does a lot of software engineering in Dublin, roughly a mile away from my house.

In any case, buying the 64 GB version seemed like the way to go. Many people have been worried about the lack of disk space available for user data and applications, but after moving the recovery partition to a bootable USB, installing Office 2013 Professional, and synchronizing my 2 GB mailbox with Exchange Online, I have 29.8 GB free. I think this is quite enough for what I view strictly as a companion device, something that will be much easier to take on the road for a few days. Although the Surface Pro boasts an excellent screen, I doubt I will do much photo editing on it. Likewise, any serious document creation or editing will be done on my “large” laptop. And even though some folks have experimented with running a little datacenter on their Surface Pro, I think I shall leave my virtualized activity to other more capable boxes. And in any case, I bought a 64 GB MicroSD card to boost storage for files like photos and music.

All in all, I’m happy with the Surface. The handwriting recognition with OneNote is particularly good, even if my ability to read electronic scrawls is at a level with my handwritten efforts. I bought the Type cover instead of the Touch as I thought this would be better for someone who is largely keyboard driven. The keyboard is fine as long as it is on a firm surface. Things are a little more “interesting” when it’s balanced on my lap as it seems that this causes the keyboard to miss keystrokes from time to time. At least, I type, but nothing shows up on screen. This also happens when the cover is on a flat, firm surface but less often. I figure this must be something to do with the pressure I put on keys as I type and that it will improve with practice. Here’s hoping…

While I was in Seattle, I picked up 16 GB of memory to upgrade my HP Envy 17 (3002ea) laptop. All of the online documentation and the configuration utilities for memory vendors that I checked (for instance, this one) indicate that the Envy is limited to a maximum of 8 GB. However, some online forums indicate that it was possible to go higher. After searching around, I settled on some Corsair Vengeance 16GB DDR3 1600 MHz (PC3 12800) memory and Amazon.com delivered the modules to my hotel. I’ve installed the memory and the Envy is flying as it wallows in memory. Running four virtual Windows 2012 servers with VMware Workstation is very much easier than before.

Back to the phone, despite the efforts of others to dazzle me with Nokia Lumia 920 (large, heavy, but delightful) and HTC 8X (light and impressive), I think I shall stay a little while longer with my refurbished Lumia 800. It does everything that I want and the Windows Phone 7.8 upgrade appears to be a little snappier than I remember (at a distance) Windows Phone 7.5 was. The Lumia 800 also includes an FM radio, which I like. So I will wait and see what happens later on in 2013 when the price of the current models drop and some new models appear. There’s no rush…

Follow Tony @12Knocksinna

Update 26 March: Seems like FM Radio capability is coming for the latest generation of Nokia Lumia phones. I might just upgrade after all!

Posted in Technology | Tagged , , | Leave a comment

“Can’t delete email” problem emerges for Exchange 2010 SP2 RU6 and Exchange 2010 SP3

Posted on February 28, 2013 by Tony Redmond

Microsoft’s new cumulative update strategy for Exchange 2013 calls for updates to be shipped to customers every thirteen weeks. An expectation exists, at least on the part of Microsoft, that customers will then install the updates soon afterwards, if only to keep their systems in a supported state.

Delivering updates at a rapid cadence is all very well, assuming that the new software is of a high quality. Regretfully, recent experiences of Exchange 2010 roll-up updates has not been great. And as we look forward to Exchange 2013, a new problem has emerged to reinforce the necessity for quality code.

I was first acquainted with the problem when Kevin O’Brien posted a comment to my post about the new cumulative update strategy. Since then some other evidence has emerged to indicate that the issue is being hit by people who have installed Exchange 2010 SP2 RU6 or Exchange 2010 SP3. You can read posts describing experiences here and here. It does not surprise me that both versions exhibit the same problem as Microsoft has released both recently and the root cause probably lies in an update that’s included in both SP2 RU6 and RU3. It would be surprising if two entirely different bugs caused the same problem.

In a nutshell, it seems like clients working in online mode cannot delete certain items. Outlook clients configured in cached Exchange mode do not exhibit the same problems because delete operations are processed on the client and synchronized back to the server. Evidence suggests that some linkage exists with messages that have PDF attachments, but other indications suggest that this is not the only cause. For instance, it might be possible that messages generated by Cisco Unity voicemail servers are affected too.  The only solution is to deinstall Exchange 2010 SP2 RU6 (or in the case of SP3, either wait for Microsoft to provide a fix or rebuild the server).

This is not a good situation for Exchange to be in. Not only has another quality issue occurred with an update, it seems strange that the problem should be in a fundamental messaging operation like client deletes. This is not an esoteric backroom bug. It’s something that stares users in the face and should have been caught by the massive suite of automated tests that Microsoft uses to validate Exchange builds.

What’s more worrying is that the rapid cadence for Exchange 2013 CU releases might encourage administrators not to test CUs thoroughly before putting new software into production. After all, when you only have thirteen weeks before the next CU is released, something has to give if everything is to be done, including the normal day to day administration required of Exchange servers.

Seeing a new problem emerge in an update released for Exchange is disappointing. No word is yet available from Microsoft as to the root cause or whether we will see Exchange 2010 SP2 RU6-v2 or an updated SP3. The cynics will say that this is par for recent update quality. I sincerely hope that things improve for Exchange 2013.

Follow Tony @12Knocksinna

Posted in Exchange 2010, Exchange 2013 | Tagged , | 17 Comments

Why even bulletproofed Exchange ActiveSync is no guarantee of future iOS client happiness

Posted on February 25, 2013 by Tony Redmond

A comment posted to my post wondering whether Apple’s release of iOS 6.1.2 would provide a long-term fix for the problem that causes excessive transaction log growth on Exchange servers (without introducing any new problems) asked a pertinent question:

“Maybe Microsoft could fix their ActiveSync protocol so this doesn’t happen again. This has been going on for YEARS now and Microsoft has simply shuffled the issue under the rug. The real cause of the issue is that ActiveSync has security bugs in it STILL after all these years. The fact that a client app can cause trouble for an Exchange server is indicative of a failure on the back-end services, which should handle the problem.”

I don’t believe that Exchange ActiveSync (EAS) running on Exchange 2010, Exchange 2013, or Exchange Online (Office 365) contains any security bugs that are known to compromise devices or force devices to behave strangely. At least, the current batch of iOS problems are not security related, unless you consider the imposition of excessive load by clients on a server to be a security bug. I can see how people might consider that a malfunctioning client could be a security concern, but it’s really not. Instead, I believe that the core issue is how can EAS provide a bullet-proof platform that cannot be compromised by misbehaving clients.

I’ve written before about the need for Microsoft to step up to the plate and take a more proactive stance with ActiveSync licensees. That article appeared when we were coping with the effect of the calendar hijacking problem, which seemed to have been knocking around since iOS 4. It’s hard to say that anything has really improved since October 2012, although I think the fact that Apple acknowledged an issue in the way that iOS interacts with Exchange is a positive step because it’s the first time that I can ever recall Apple admitting that the problem lies on their side. Indeed, the statement contained in Apple’s article that “the device may begin to generate excessive communication with Microsoft Exchange Server” is illuminating because it illustrates a problem that might continue even if Microsoft removes every single bug from the EAS code.

Exchange handles client connections using a number of protocols (HTTP, IMAP, POP, MAPI, EAS, and EWS). Even if the server implements its protocol stacks and code to handle connections perfectly, or as close to perfect as software engineering can ever achieve, the server is still exposed to a load generated by a client that runs amok. Think of what happens when a client sends data to Exchange to update a message (for example, set the status to Read) or calendar item (for example, accept a meeting request). These are transactions that have to be logged and recorded in a database. If all goes well, the client issues the relevant command, Exchange executes it, and a single transaction occurs. Or perhaps multiple transactions in certain instances, as in when a meeting request is updated and leads to messages being sent to inform attendees about the update. In any case, the number of transactions for specific client operations is predictable under normal circumstances. But if the client code is buggy, it’s possible that the simple flow of transactions to the server might become the “excessive communication” referred to in Apple’s article. And excessive communication leads to transactions, leading to transaction log accumulation and additional server load.

Another example of where ActiveSync clients behave in a way that can create an excessive load on servers is if they synchronize folders at short intervals. The logic here is that the client wants to make new messages and updates in the mailbox available to the end user as quickly as possible, so it continually checks with Exchange to determine if new mail has arrived or other updates have occurred. A client that synchronizes every minute is going to make sixty requests to the server every hour, something that might be acceptable if just a few clients are connected, but clearly a problem as you scale up to support hundreds or thousands of mobile devices. Think of how much worse the situation becomes if a client now attempts to synchronize every thirty seconds… or twenty seconds…

The server could, of course, attempt to recognize situations where a client is generating excessive load and terminate the connection, but the client could simply reconnect and start over. Exchange deals with this kind of problem with throttling policies that measure the load generated by clients and block connections when abnormal activity is detected. Although EAS incorporates comprehensive device access policies that can be deployed to block individual clients or complete families of clients, it’s not EAS that suppresses hyperactive clients. Instead this is done by the throttling policies operated by Exchange’s workload management.

I’m pretty sure that the Exchange developers are doing what they can to make sure that EAS behaves properly and doesn’t expose itself to client-initiated problems such as calendar hijacking. Fixes for these kind of issues will appear in the updates that Microsoft releases regularly for Exchange. But Microsoft faces a problem in that there are so many ActiveSync client applications running on devices that there’s no guarantee that a client might not expose a new problem at any moment. Apple does a good job of pushing out iOS updates to devices. And although there have been quite a few recently (iOS 6.1, 6.1.1, 6.1.2 and 6.1.3 coming soon), at least the updates are pushed out effectively via iTunes in such a way that a large percentage of devices in use run the latest version of iOS. The same situation does not pertain for Android clients.

At the end of the day, client-server systems can only work well when solid code runs on both platforms. It’s less complicated when a single company owns both clients and servers, which probably accounts for why you see fewer EAS problems with Windows Phone devices. But we live in the era of BYOD, a development that looks less attractive when clients run wild.

Even if Microsoft succeeds in making EAS relatively bullet-proof and also takes a more aggressive stance with ActiveSync licensees to make sure that their client code works as well as possible, I suspect that we will still have to be prepared to deal with client-provoked ActiveSync issues in the future (or on an ongoing basis, as some evidence suggests that the Exchange transaction log problem is not quite fixed in iOS 6.1.2).

If you accept that this is the case, you should acquaint yourself with the advice given in the very good “ActiveSync Troubleshooting Chalk Talk” session given by Austin McCollum of Microsoft Support as well as read up about how to manage iOS devices effectively when deployed with Exchange. Otherwise we might end up moving from BYOD meaning “bring your own device” to “bring your own disaster”.

Follow Tony @12Knocksinna

P.S. Following the news that LG has acquired webOS from HP to use in their “smart TV” platform, perhaps they’ll write a good ActiveSync client and show how it’s really done. Or maybe HP’s new Android-powered Slate 7 will include a properly functioning ActiveSync client. Just thinking…

Posted in Email, Exchange, Exchange 2010, Exchange 2013, Office 365 | Tagged , , , , , , , , , | 3 Comments

iOS 6.1.2 appears – it might fix the bug that screws up Exchange

Posted on February 19, 2013 by Tony Redmond

It looks like Apple has released iOS 6.1.2 in an attempt to fix the calendar synchronization problem that causes Exchange servers to log a vastly increased number of database transactions and results in more transaction logs being generated, log replication within Database Availability Groups (DAGs) and potentially service disruption if disks fill up.

This problem has lasted for far too long and it follows on other issues with iOS such as calendar hijacking. In fact, dealing with a calendar seems to pose a technical challenge for Apple, possibly because their experience and focus is centered around the development of consumer applications rather than enterprise-ready software.

I don’t know if the latest fix will work or whether some new bugs are lurking and will be exposed when the fix is applied. No one knows how Apple tests their code against Exchange. If we did, we might have more confidence in the new build. Until solid proof emerges to demonstrate that it does (and that no other problem is uncovered by 6.1.2), I recommend that Exchange administrators are cautious and consider taking steps to control iOS clients. Later on, if the problem really is solved, controls like ActiveSync device access rules can be eased back to allow more user choice. If these problems have taught us one thing, it is that BYOD needs to be managed as otherwise chaos can reign.

Let’s hope that the folks in Cupertino have read the ActiveSync protocol command specification and have finally figured out how the iOS mail app should interact with Exchange. It would be nice if an iOS upgrade became something to which users looked forward with anticipation rather than trepidation. Time will tell.

Follow Tony @12Knocksinna

Posted in Email, Exchange 2010, Office 365 | Tagged , , , | 9 Comments

iOS problems continue, good tip about VM deduplication, and the MVP Summit

Posted on February 18, 2013 by Tony Redmond

Apple’s continuing struggle to make iOS work properly with Exchange was a prominent topic of discussion last night when some Exchange MVPs assembled for dinner. We’re in Redmond for the annual MVP Summit run by Microsoft and I was delighted to meet a number of people in person for the first time, including new MVPs Steve Goodman, Paul Cunningham, and Michael van Horenbeeck. I imagine that Steve’s book “iPhone with Microsoft Exchange Server 2010 – Business Integration and Deployment” should become popular reading with those who want to manage iOS devices more effectively in an Exchange (on-premises or Office 365) environment. If it, it should be because it’s a very good book.

Paul is working on an Exchange 2013 book about PowerShell that he will self-publish. Apparently it will focus on helping people who are still grappling with the concept of using PowerShell to manage Exchange rather than diving into the depths of scripting complexity. The book will appear later on this year and I’ll keep a keen eye on its progress.

Although not all of the old-time Exchange MVPs are in town, it was good to meet up with people like Jeff Guillet and Paul Robichaux again. Jeff and I bumped into each other in the Bellevue Square mall, where I had gone to buy a Surface Pro (not yet available in Ireland – I will report back soon on my experiences of using the Surface) at the Microsoft Store. He waxed lyrical about recent experiences of running Windows Server 2012 with deduplication enabled so that he could recover disk space and run more virtual machines on a server. Some steps are necessary to make sure that the deduplication process does not interfere with the smooth working of Hyper-V, but it seems like a very nice way to fit more VMs on the server that he assembled himself. The technique should work with VMware too. Interesting reading!

Paul Robichaux and I took the chance to catch up with our Microsoft Press editor and discuss the progress of Microsoft Exchange Server 2013 Inside Out, the two-volume set that we’re working on with a target publication date in October 2013. Things are going pretty well on the writing front and we wanted to discover the plans Microsoft Press had to push the book along through the copy editing and other processes that take books from raw text in a Word document to final printing. We want to do is to release chapters early through the Microsoft Press “PrePress” program so that text is available much earlier. If things go well, we should have something online in April – I hope!

Microsoft Press is also going to bring out an Exchange 2013 Best Practices book, scheduled in roughly the same timeframe as the Inside Out books. MVP Siegfried Jagott is working hard on this project, or so he tells me. I’ll see when the text starts arriving for technical review.

To come back to Apple’s iOS problems, perhaps they’ll accept the offer made by Paul Robichaux in an open letter to Tim Cook to come down and help them sort out their code. Paul is a long-time user of Apple products, but I think that his faith is wavering as he’s started to use a Surface Pro inside of his MacBook Pro and a Nokia Lumia 920 instead of an iPhone. Technology buying patterns change quickly when problems occur, and I sure hope that Apple starts to get a hold on the problems that they are having soon. The continuing flood of stories on this topic indicates that they are floundering. Maybe it really is time for Paul to visit Cupertino… Or for companies to take a stronger approach on the BYOD question, which is why you might want to buy Steve Goodman’s book.

Follow Tony @12Knocksinna

Posted in Email, Exchange 2010, Exchange 2013, Office 365 | Tagged , , , , , , , | Leave a comment

Weak Apple advice about iOS 6.1 bug – problems continue

Posted on February 13, 2013 by Tony Redmond

I love this recommendation from Apple’s support article “iOS 6.1: Excess Exchange activity after accepting an exception to recurring calendar event” advising that one cure for the ActiveSync problem that causes Exchange servers (all versions) to be swamped by excessive transaction log generation is:

  1.     Go to Settings > Mail, Contacts, Calendars
  2.     Select the Exchange account from your Accounts list.
  3.     Turn the switch for Calendars to OFF.
  4.     Wait ten seconds.
  5.     Turn the switch for Calendars back to ON.

Hmmm… Although I strongly believe in the age-old adage that many IT problems can be solved by a reboot or by switching a device off and on, I remain unconvinced that switching calendars off and then back on again can cure a problem with code running in Apple’s mail app on iOS 6.1 devices, even if strong evidence suggests that the bug is calendar-related. The facts are that Apple has had many problems in their mail app code, which is based on a publicly available protocol specification (last updated on 11 February 2013). At this point, you’d wonder if the programming whiz-kids in Cupertino understand ActiveSync at all. Perhaps not.

Word on the street is that the iOS 6.1.1 release that Apple rushed out doesn’t fix the Exchange problem. I don’t think this is strange as the bigger problem was that Vodafone had started to tell customers not to upgrade their devices to iOS 6.1 because of poor cellular performance. Apple seems to respond much faster and more comprehensively to pushback from mobile operators because of its potential impact on consumer buying decisions than they do when bugs affect email.

For the moment, the recommendation is to continue monitoring the situation and use some of the techniques explained in my post explaining how to control problematic iOS 6.1 devices and Microsoft’s KB2814847.

Eventually the bug will be fixed and transaction log generation will return to normal. And maybe then Microsoft might have a heart-to-heart meeting with Apple to explain just how ActiveSync works so that the mail app doesn’t run into future problems when it attempts to process a calendar request (or do anything else really).

Follow Tony @12Knocksinna

Posted in Email, Exchange, Exchange 2010 | Tagged , , , , , , | 4 Comments