The dirty little secret about migration to modern public folders


Have you heard about a migration to modern public folders that has gone well? I haven’t. Apart that is from the demo migrations that appear at trade shows and conferences to show that all is well and that Microsoft hasn’t reneged on their promise to bring the cockroaches of Exchange kicking and screaming into the world of highly available mailboxes.

Microsoft has published a public folder migration checklist and written a number of scripts to support that checklist that you can download from their site (the scripts are also available in the $ExScripts folder on an Exchange 2013 server). The scripts support migration from legacy environments running on Exchange 2007 or 2010 servers to Exchange 2013 on-premises or Exchange Online (Office 365).

In theory, the procedure is straightforward.

  1. Wait until all mailboxes are moved to Exchange 2013. This step is mandatory because whereas Exchange 2013 mailboxes can access legacy public folders, the reverse is not true.
  2. Enumerate the legacy public folders and hierarchy and create a CSV file using the Export-PublicFolderStatistics.ps1 script.
  3. Examine the CSV output (a list of public folders and their sizes) to decide whether any adjustments need to be made. Pay attention to the limitations of modern public folders to make sure that you keep under the supported limits of 100 mailboxes and 10,000 folders after migration is complete.
  4. Run the PublicFolderToMailboxMapGenerator.ps1 script to process the CSV file containing details about the legacy public folders and to map them to new public folder mailboxes. Essentially, the script attempts to break up the folders and divide them in a logical manner across a new set of public folder mailboxes. You control how many mailboxes are necessary by setting a maximum for the size of each mailbox. For instance, if you have 100GB of legacy public folder data to migrate, you could set the maximum size to 10GB and end up with 10 public folder mailboxes.
  5. Review the CSV file to decide whether public folders have been allocated in a reasonable manner.
  6. Create the public folder mailboxes, taking care to place them in databases that are capable of supporting the load that user connections will generate after you switch over to use modern public folders.
  7. Create a public folder migration request (only one can exist at any time) to process the CSV file that maps the legacy public folders to new public folder mailboxes. The processing is similar to a mailbox move in that the processing is performed by the Mailbox Replication Service (MRS) in the background. MRS connects to a public folder database, enumerates what it finds there (like a source mailbox), and transfers the data to the target (in this case, the public folder mailboxes).
  8. When MRS has moved the data, it halts (similar to a mailbox move that is auto-suspended).
  9. If everything seems to be good with the PF move request, you lock public folders for the organization (which has the effect of disconnecting users from legacy public folder databases), tell MRS to complete the move with Resume-PublicFolderMigrationRequest, wait for MRS to perform its incremental synchronization to move the last remaining vestiges of data from the legacy folders.
    Get-PublicFolderMigrationRequest | Set-PublicFolderMigrationRequest -PreventCompletion:$false
    Get-PublicFolderMigrationRequest | Resume-PublicFolderMigrationRequest
  10. Once MRS considers the move to be finished (the status shown by Get-PublicFolderMigrationRequest is “Complete”), you can validate that the data has been moved to the new public folder mailboxes by connecting to one of them with a test mailbox. Logically, you cannot connect to a modern public folder mailbox until the migration job is complete. The best way to do this is to run the Set-Mailbox cmdlet to override the normal mechanism used by Exchange to locate the public folder hierarchy using the command shown below. You can also run some scripts to compare what exists on the legacy side with what has been moved.                                                                 Set-Mailbox -Identity "Some Test Mailbox" -DefaultPublicFolderMailbox "PFMailbox1"
  11. If you are happy that everything has been properly moved to the mailboxes used by modern public folders, you can commit the organization to using modern public folders by performing a final switchover. To do this, you make sure that all the new public folder mailboxes are capable of providing the public folder hierarchy to clients and then run Set-OrganizationConfig to update the configuration and switch from old to new.
    Get-Mailbox -PublicFolder | Set-Mailbox -PublicFolder -IsExcludedFromServingHierarchy $False
    Set-OrganizationConfig -PublicFolderMigrationComplete:$True
  12. Have a beverage of your choice to recover while basking in the admiration of users who are now totally happy to be connected to the new, improved, and most modern of public folders.

Update 14 July 2014: As MVP Andrew Higginbotham points out, if you don’t do everything properly, you might end up by forcing all clients to see the infamous “The Administrator has made a change that requires you to restart Outlook.” Check out his blog for details.

It all sounds good. It all seems like it might even work. And it does, in a test environment where you have 10 public folders to migrate.

The procedure runs into difficulty when confronted by the average public folder hierarchy. Invariably, the hierarchy is old, badly organized, unmanaged, and contains enough rubbish to occupy several skips. The problem you see is that public folders have been around for a very long time and during their existence it is extremely likely that many people, both users and administrators, have had their fingers in the pie. Folders exist for no reason; folders exist that once had a reason but one that is now long forgotten; folders exist that contain useful information that no one knows about; and public folders exist that contain the greatest collection of old rubbish and corrupt items that you will ever encounter in your career.

And the good news is that the procedure as laid down will faithfully attempt to migrate all the old, unwanted, unhappy, unreadable, and undesirable information from legacy public folders to their new homes. That is, unless you take steps to stop this happening.

We know that a CSV file is generated in step 2 of the procedure to enumerate the public folder hierarchy and its folders. The CSV file contains two columns – folder name and folder size, so you end up with something like this:

“FolderName”, “FolderSize”
“\IPM_Subtree”,”0”
“IPM_Subtree\Contoso”, “248844”
“IPM_Subtree\Contoso\Departments”, “12487744”
“IPM_Subtree\Contoso\HQ”, “45254”
“IPM_Subtree\Contoso\HQ\CEO Office”, “147755”
“IPM_Subtree\Contoso\HQ\CEO Office\2007 Business”, “477444”

Reviewing the contents of a CSV file to identify folders that should be pruned from the hierarchy is a tiresome business if you have a couple of hundred folders to examine, especially when you don’t know a lot about the content of the folders, why they were set up, who uses them, and when they were last accessed in any meaningful manner.

But many hierarchies have a few thousand or more folders and the thought of having to go through the full set becomes something that would only excite a lonely monk in a cell on an island in the middle of the Atlantic. Most administrators would prefer to tear out their own fingernails than process this CSV file, which is a real pity because the big flaw of the migration process is that if you don’t take the time to prune, remove, and destroy old rubbish from the public folder hierarchy before MRS starts to move data, all of the rubbish will be moved.

You can’t even take the obvious approach of deleting large batches of folders from the CSV file in the hope that MRS will ignore them during the migration. MRS is pedantic and exact. It will migrate everything it finds in the hierarchy even if you remove all reference to a folder from the input CSV file.

The next issue is that old public folders have a habit of storing old items (naturally), some of which fall into the category of “malformed MAPI items”. In other words, they are corrupt in the eyes of modern MAPI clients and servers. Legacy public folders are not to be faulted for storing this information. Folders store items and the items were good when they were stored. But MRS will complain if it encounters one of these “bad items” and they will be discarded en route to modern public folders. You control the tolerance MRS has for bad items by setting a bad item limit when you create the public folder migration request, but it’s easy to blow the normal limit (10 or thereabouts) that you might use for mailbox moves with the rubbish lurking in the deep and dank corners of old public folders. Eventually you can force MRS to process everything, but be prepared to lose some data and to restart the public folder migration request several times after upping the bad item limit.

The dirty little secret of public folder migrations is not that the process does not work. It does, but it is excruciatingly manual in nature and execution and requires more time and effort from an administrator than you might possibly imagine.

We have not yet seen many public folder migrations because few companies have yet completed step 1 to move all their mailboxes to Exchange 2013. We will see more as time goes by. The potential for pain is out there…

Follow Tony @12Knocksinna

About these ads

About Tony Redmond

Exchange MVP, author, and rugby referee
This entry was posted in Email, Exchange, Exchange 2013, Office 365 and tagged , , , , , , , . Bookmark the permalink.

35 Responses to The dirty little secret about migration to modern public folders

  1. Ai says:

    Hi Tony,

    Hmm, I thought Public Folders were Exchange’s dirty little secret in as far as Microsoft have been concerned these past several years. If you ignore them long enough maybe they’ll go away was the consideration perhaps? Then again as you recently mentioned 2003 being a dead and forgotten piece of software to them, I imagine Public Folders will go away as by Exchange 201x release date Exchange 2010 will be a non-existent piece of software that Microsoft deny the existence of and as people won’t have been able to migrate away it becomes an SEP.

    I have to say as ever, I’m fighting to resist being your eternal comment writer and expanding my blog when I can to avoid keep airing my opinions here. So saying, we do seem to hit similar topics on occasion and given my working on email since Network Courier I suppose we’ve followed similar paths. I tend to err on the less official path and am slightly more acerbic when it comes to the inequities of vendors I fear. Maybe it stems from hearing the words “Surely Microsoft have a plan for this?” too many times.

    Public Folders do get me going more than many things though, as do migration tools and PST hunting software. There’s a common thread to all of these that have earned me a great deal of money and a no few grey hairs. The problem? Microsoft distancing themselves from Public Folders and ignoring the massive headache this causes for people using their software. Having just dealt with my first 50Gb OST and also finished cleaning up a 5.5(Yes Ms, it still exists out here in the real world) migration that was truly botched by yet another Gold Partner, I truly am fed up with this expanding lack of care.

    Again in common with the PST and migration tools, I believe that Microsoft hoped and prayed that 3rd parties would get to grips with them and come up with a product that migrated them to Sharepoint so they could wash their hands completely and not include Public Folders in later releases. As that didn’t succeed, Public Folders continued to exist until they were removed in 2013 and replaced with slightly modified mailboxes.

    Yet still there is no clear migration process, cogent purpose and definition document, nor tool that provides a mechanism to migrate effectively. Your document outlines just how painful this process could be. I also wonder who has migrated from pre-2013 PF to MPF yet as from a comment on my blog, responding to a CU1 comedy of errors post I made, mentioned that there is a bug preventing migration to MPFs that won’t be fixed until CU3, yet I saw no note of the bug in the list for CU3 at all.

    One of my main questions to ask Microsoft is where there are any real-world test cases that they have undertaken that demonstrates how well a large PF>MPF migration works? The non-coexistence heralded as coexistence in CU1 continues to demonstrate a couldn’t care much approach to moving PFs as it is effectively suggested that one moves all mailboxes to 2013 and then test out your PF migration in one big hit hoping that it works.

    I must admit I have still failed to see what value Modern Public Mailb…oops.. Folders bring and why anyone would migrate to them. Public Folders have historically been used for the following reasons.

    Shared Calendar for room bookings, resource bookings and team calendars.
    Shared mailboxes
    File stores
    Hidey holes for mailbox data on badly administered systems that impose DB restrictions but leave PFs open to user creation (heh and PSTs)
    Public facing addresses or internal system mailing points
    Company directory or shared phone lists
    Custom developed workflow

    So barring the custom dev stuff which I’ll bet you those carefully crafted scripts that call (hmm what executable to CSV could it be?) PFDavAdmin don’t cater for, all can be taken care of by other mailbox types and so we return to the crux of the matter. If Microsoft had taken the time to look at their customers who use PFs extensively and worked with them to develop migration plans to Sharepoint rather than cobbling together another type of mailbox that pretends to be a PF, this problem wouldn’t exist today and we wouldn’t need MPFs.

    Many organisations gave up moving from PFs based on the reasons you outline and I’ve seen some of the environments holding hundreds of gigs in tens of thousands of folders so know how insurmountable the migration task can seem. Then again what migration of tens of thousands of mailboxes from multiple organisation into one was ever an easy task? For this Excel and techies were created!

    I don’t currently have an answer but I am working with some people that may. Once I’ve input some detail on real world scenarios into a tool that’s been developed for another purpose, there may be a good PF/PST migration tool that can be used more effectively to allow organisations to never have to use MPFs and abandon PFs. If and when it comes about I’d be keen to get some input on how well people such as yourself view its capabilities.

    In any case it will be interesting to see how much this aspect of legacy migrations delays the take-up of 2013. I will still go with my maxim of ‘avoid interim development versions of Exchange’ i.e. every alternating version since 5.5 and look forward to the next major release!

    All the best and keep up with the helpful and clear insights, please also never apologise for expressing your ire as I noted when your frustration becomes apparent in some posts this year, you are always diplomatic, polite and inscrutable so if you’re annoyed with something in tech you can be sure the majority of techs out there are with you and appreciate your candour!

    Ai
    ____________________________________________
    Just for your reference Tony, I’m not that keen on publicising my blog or myself(yes I know so why have one?), this was my take on CU1 that touched on PFs. (I must admit I have worried as I put a link to your blog in another one of my posts and not knowing the blogging etiquette still wonder if one should let people know when commenting about them!) In any case I hope it reads as an amused onlooker who has worked at the heart of this industry rather than a tired old cynical hack having a go. (hang on aren’t they the same thing?) http://napoleonmoses.wordpress.com/2013/05/10/poor-old-microsoft-exchange-it-is-in-the-wars/

    • No problem commenting as we all have perfectly valid opinions. The key is to back up comments with facts, which is what I try to do (not always successfully). Maybe you’ll like my interview with Perry Clarke at http://windowsitpro.com/blog/talking-exchange-server-microsofts-perry-clarke… It addresses some of the issues that I think you are concerned with.

      • Ai says:

        Thanks Tony, I’ve had a chance to read through that interview finally. The fact that the technical teams at Microsoft are second to none is the cause of my frustration when their message is distorted by corporate policy and third parties selling by spreading fear and confusion. (I’ll admit some shell-shock from spending a career picking up the pieces resulting from such practices)

        The concerns Perry raises about the disparity between the technical message and the sales message coming from large software vendors seems most relevant.

        Lets hope the recent appointment in Redmond paves the way for a clearer message in the future and the return to letting the quality of the software sell itself rather than a massive advertising department. An optimistic start to 2014 in any case.

  2. Pingback: Exchange Public Folders | kay4ni

  3. Pingback: Migrating to Exchange 2013 public folders | SME IT guy

  4. Pingback: interesting things i have seen on the internet 30/12/2013 | 503 5.0.0 polite people say HELO

  5. Tim Clement says:

    I’m working through a hybrid migration to O365 EOL. I’ve ran the New-PublicFolderMigrationRequest (your step 7). I would like to verify as you mention in step 8 but can’t figure out how to connect to the PF mailbox. Do you have any suggestions ?

    thanks
    Tim
    Austin. Tx.

  6. Tim Clement says:

    Thanks Tony. After running the set-mailbox command, then trying to add a PF through O365 OWA, I receive a prompt saying “can’t complete your request no public folders are available. check that a public folder deployment exists in this exchange organization…”

    Here is the status of my migration request:

    Status : AutoSuspended
    SourceServer : mail.txsg.state.tx.us
    SourceVersion : Version 14.3 (Build 174.0)
    BytesTransferred : 2.679 GB (2,876,674,062 bytes)
    ItemsTransferred : 3328
    PercentComplete : 95
    Message : Informational: The move request has been automatically suspended because the PreventCompletion pa

    [PS] C:\scripts>Get-o365Mailbox -PublicFolder

    Name Alias ServerName ProhibitSendQuota
    —- —– ———- —————–
    Mailbox1 Mailbox1 blupr02mb132 49.5 GB (53,150,220,288 bytes)
    TXSGPF TXSGPF bl2pr02mb468 49.5 GB (53,150,220,288 bytes)

    Any suggestions for validating the migration request ?

    thanks
    Tim

  7. Tim Clement says:

    Hi Tony,

    Let me rephrase my questions.

    1. Can I validate the migration (connect a test mailbox) before setting the PublicFolderMigrationComplete property to True , completing/resuming the migration request and unlocking the public folder hierarchy in EOL ?

    2. Should I be able to see the migrated public folders in the EOL EAC before completing these steps ?

    thanks
    Tim

    • Hi Tim,

      From your last comment, it looks like MRS has performed the initial synchronization from old to modern public folders (it has reached 95% completion and the move then auto-suspended). Before you can access the moved data, you need to allow MRS to complete the move by performing an incremental synchronization to capture all of the changes, additions, and deletions done since the initial synchronization was kicked off. This will ensure that you have a full and complete copy of the data on both old and new sides. You have to lock the public folder environment to complete the move too because you don’t want users adding new stuff or accessing the old public folders while MRS performs the incremental update. Exchange won’t allow you to access the modern public folders until the migration job has completed processing.

      I never intended the post to be a step-by-step guide to migration because there are many other posts on this topic elsewhere. Instead, I wanted to call attention to the fact that substantial care and attention is required to prepare for the migration. However, I think that some extra detail is warranted in the post to provide a more accurate guide so I have updated it with some of the commands that are required. Can you look at the post again to review the changes and see if they answer your questions?

      TR

  8. Pingback: Exchange 2010 Public Folders: Part 1 | Thoughtsofanidlemind's Blog

  9. Scott Schering says:

    We are wrapping up a public folder move from Ex 2007 to 2013 modern public folders.
    Everything up to step 12 has gone very smoothly. Well all except for that CU2 bug that stopped the migration from completing for a few months.

    Post migration we are sitting on 180 gigs of data in 17,000 folders spread across 60 public folder mailboxes. My goal was to keep PF mailbox sizes down to 3 gig each.

    We have one tiny little issue. Unless a user is set to use the Primary Public folder hierarchy mailbox AND the the folder itself is stored in the primary PF mailbox the user cannot create new sub folders from Outlook. They get a “Cannot create the folder” error after a short delay. I can create sub folders in the ECP or powershell but you can’t create any of the other folder types, Calendar, tasks and so on unless everything is set to the primary PF mailbox.

    Everything I’ve read says these requests should proxy to the Primary Mailbox but appears to be failing..

  10. Pingback: Outlook Web App 2013 and mail public folders | Thoughtsofanidlemind's Blog

  11. Pingback: Exchange 2013 public folders crippled by newly revealed limits | Thoughtsofanidlemind's Blog

  12. Brent says:

    Scott, i am having the same issue you are experiencing. Did you ever get a resolution to it? Users at my company cannot create new public folders, or delete current ones. I can do it fine from the EMC and through powershell though.

    • Scott Schering says:

      I posted this to the new thread on 2013 Pf limits but felt it needed a response here too.

      We opened a ticket with MS support about users not being able to create new public folders from outlook. They determined that the large Hierarchy is causing the issue. The primary Pf mailbox is taking to long to return the Hierarchy causing new folder requests to time out and fail.

      We don’t have a solution yet but Support is working on it. We can work around it for now with the esc and moving folders & users to the primary mailbox when needed.

  13. Pingback: What caused the crippling of Exchange 2013 modern public folders? | Thoughtsofanidlemind's Blog

  14. aurum says:

    this is so true.
    Im in the process of Migration a PF Hierarchy with 1200+ folders and 15GB.
    Been working on it for a week and havent got further than starting the migration attempt which immediatly fails. Complaining about unsupported characters, on folder which dont exist anymore!.
    Gone through all the folders manually now and removed spaces and slashes and so on.
    Still no luck

  15. AAA says:

    I have over 180,000 public folders with around 2.5 TBs of data. I am not looking forward to having to migrate these to modern public folders. The effort right now is to clean them up as much as possible, but I am sure when the time comes to move to Exchange 2013, I will still have well over 100,000 folders.

    Shoot me now :D

  16. lebricoleur974 says:

    Hi

    I m migrating public folder 2010 to 2013

    When i do :

    Get-PublicFolderMigrationRequest | Get-PublicFolderMigrationRequestStatistics -IncludeReport | Format-List

    I have a lot of line like this

    00-00-00-00-1A-44-73-90-AA-66-11-CD-9B-C8-00-AA-00-2F-C4-5A-03-00-36-B9-C6-5D-38-99-D6-11-9B-D6-00-08-02-55-A6-70-00-00-00-18-33-94-00-00 ». Ce dossier peut être
    lié manuellement en exécutant la cmdlet Enable-MailPublicFolder une fois la migration terminée.
    23/05/2014 19:18:02 [serveur] Avertissement : échec de recherche ou de liaison de l’objet destinataire « 96-7D-85-6C-DC-A6-83-42-AC-77-A4-5E-BD-CD-52-9B » dans
    Active Directory pour le dossier public à extension messagerie « Public Root/IPM_SUBTREE/Secrétariat RC/PAYS/VENEZUELA/2006/SINCOR » avec l’ID d’entrée «
    00-00-00-00-1A-44-73-90-AA-66-11-CD-9B-C8-00-AA-00-2F-C4-5A-03-00-36-B9-C6-5D-38-99-D6-11-9B-D6-00-08-02-55-A6-70-00-00-00-13-4D-8A-00-00 ». Ce dossier peut être
    lié manuellement en exécutant la cmdlet Enable-MailPublicFolder une fois la migration terminée.
    23/05/2014 19:18:52 [serveur] Avertissement : échec de recherche ou de liaison de l’objet destinataire « 25-86-EC-2D-ED-25-C6-49-83-B6-5F-E9-8C-2E-34-A4 » dans
    Active Directory pour le dossier public à extension messagerie « Public Root/IPM_SUBTREE/Secrétariat RC/Repertoire 2007 » avec l’ID d’entrée «
    00-00-00-00-1A-44-73-90-AA-66-11-CD-9B-C8-00-AA-00-2F-C4-5A-03-00-36-B9-C6-5D-38-99-D6-11-9B-D6-00-08-02-55-A6-70-00-00-00-11-F5-9D-00-00 ». Ce dossier peut être
    lié manuellement en exécutant la cmdlet Enable-MailPublicFolder une fois la migration terminée.
    23/05/2014 19:18:57 [serveur] Avertissement : échec de recherche ou de liaison de l’objet destinataire « ED-80-51-1F-67-6A-B9-41-AA-B8-0C-14-C4-BF-07-A3 » dans
    Active Directory pour le dossier public à extension messagerie « Public Root/NON_IPM_SUBTREE/EFORMS REGISTRY/Organization(409) » avec l’ID d’entrée «
    00-00-00-00-1A-44-73-90-AA-66-11-CD-9B-C8-00-AA-00-2F-C4-5A-03-00-36-B9-C6-5D-38-99-D6-11-9B-D6-00-08-02-55-A6-70-00-00-00-00-1F-49-00-00 ». Ce dossier peut être
    lié manuellement en exécutant la cmdlet Enable-MailPublicFolder une fois la migration terminée.

    it scared me, what does it means ? Do i do something wrong?
    I have let it work
    And when i check status i have this :

    Statut: Failed
    Détails status : Failed other
    Syncstages : Copying message
    Estimed transfert size : 0
    Pourcentcomplete : 95

    What do i do? Can You help me? I m a little scared right now

    Thanks

    • Your PF migration request is stalled because errors have been encountered. You will have to check each one of the folders to see what the problem might be. It might be that the mail-enabled properties are incorrect, it might be that the name of the folder contains one of the special characters that cause the migration to halt. If possible, delete the folders (if you don’t need them). After you fix everything up, restart the migration.

      TR

  17. Pingback: Progress in raising public folder limits – but we’re not there yet | Thoughtsofanidlemind's Blog

  18. Phil Ready says:

    Hi Tony
    A great article. I need to migrate 2500 PF’s totalling about 50GB. There is a few PF’s over 2GB and one over 9GB. Trouble is I don’t really want to make a 10GB limit for the migration. Is there a way I can move the big PF’s first and then set a 1GB limit for the rest of migration?
    Thanks in advance
    PR

  19. Pingback: Exchange 2010 to 2013 Migration - Moving Public Folders

  20. BMcDonald says:

    I would be interested in anyone who has migrated Public Folders from 2010 to 2013 – Step #9 – where the public folders are locked – how much down time do you incur. Some of you have mentioned PF DB’s over 100GB – what was your experience.

  21. neil says:

    I am doing an exercise migration from exch2010 to 2013. I am at the pf migration steps. I stepped through many of the steps and to say its tedious and painful would be an understatement. I immediately started looking around for other options, you would think someone would make a tool for this?

    • You know, I have suggested to a couple of ISVs that they could usefully create a tool to assist PF migration. But all declined on the basis that Microsoft might come along and replace their code with some “approved” solution. I still think that it would be a good thing to do, even just to create some visibility in the market.

  22. eric says:

    I ready to migrate my pf’s from 2007->2013. I ran the prep scripts and so far so good. Couple of questions. The publicfoldertomailboxmapgenerator.ps1 script output calls for only one mailbox. Is this because I don’t have a huge pf hierarchy? What determines how many mailboxes are needed to house pf’s? Most of my pf’s are mail enabled. Does this attribute get migrated along with permissions? Will I be able to email the migrated pf’s immediately after they are migrated?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s