The Mailbox Replication Service (MRS) is an essential component of any Exchange 2010 deployment as it controls the movement of mailboxes between databases. MRS runs on all Exchange 2010 Client Access Servers (CAS).
MRS gets involved in migrations from Exchange 2003 or 2007 to Exchange 2010 because moving mailboxes is the only practical way to get user data into mailboxes. You could export and then import mailbox data and this may be done in situations where no connection is available between Exchange 2010 and the legacy servers. Even so, MRS will be involved as Exchange 2010 SP1 introduces the New-MailboxImportRequest cmdlet to replace the Import-Mailbox cmdlet. MRS manages the work done by New-MailboxImportRequest, an example of how the influence of this service has spread within Exchange.
MRS is a “black box” service in that there’s no management utility provided in Exchange 2010 to control how it works or to tweak its performance. You can create, modify, view, and clear mailbox move requests through EMC but that marks the end of management interaction with data used by MRS. In fact, all you’re doing here is working with the Active Directory attributes for mailboxes that are being moved or the XML data that describe the move requests that Exchange stores in system mailboxes within databases. You’re not really doing much with MRS at all.
It’s likely that this situation exists by design because the view of the Exchange developers is probably that system administrators don’t have sufficient knowledge or data to allow them to make changes to a service that runs in the background and if they provide a management interface it’s possible that people will make mistakes that compromise Exchange’s ability to move mailboxes.
I think this is a reasonable stance to take – for now. MRS is new to Exchange and it will take time for administrators to become used to how it works. However, over time, you’d hope that a management interface will be added to EMC to allow administrators to control how MRS works and remove the black box label.
Behind the scenes, MRS operation on a CAS is controlled by an XML-format configuration file called MSExchangeMailboxReplicationService.exe.config that you’ll find in the \bin directory (for more information, see page 708 in chapter 12 of Microsoft Exchange Server 2010 Inside Out
MRS reads in its configuration file when the service starts. The two major areas of the file that we might be interested in are the section that describes the default, minimum value, and maximum value for each setting (see below) and then the section (MRSConfiguration) that determines the current setting for MRS on this server (next figure).
MRS configuration file values
Although it is interesting to know what settings are active for MRS, I would not rush to make changes without knowing exactly why you want to make the change and what is the desired outcome. The Exchange developers made one significant change for Exchange 2010 SP1 when they reduced the MaxActiveMovesPerTargetMDB setting from 5 to 2. This setting controls the maximum number of active moves that an MRS instance can direct to a target database and the reduction is designed to reduce the pressure that a mailbox database can come under as it processes incoming mailbox content. Remember that Exchange 2010 performs content indexing as mailbox contents are transferred to ensure that a “full client experience” (marketing-speak that means that searches are possible) is available after the mailbox pointer is switched in Active Directory to the target mailbox. You can therefore understand that considerable CPU and I/O resources are consumed to process a new mailbox and that if MRS was allowed to process more than a few moves to a single database the risk exists that the disk holding the database might become very busy and reduce its ability to service other client requests.
Viewing the MRS configuration settings that are active for a server
The MaxActiveMovesPerTargetServer setting is set to 5 to ensure that MRS can’t swamp a target server with masses of incoming mailbox moves. You might be surprised that these settings are set reasonably low but the logic is good because it’s more important for a server to maintain good client responsiveness than to become preoccupied with mailbox moves. Remember that these settings only apply to the MRS running on a single CAS. If you have more than one CAS in an Active Directory site, it’s possible that a target server or target database will be asked to handle more than five incoming mailbox moves (for the server) or two (for a database) as apart from knowing what MRS is handling a move request, each MRS processes mailbox moves independently.
It’s possible that you might want to increase these values, but only when a good reason exists. For example, let’s assume that you are in the middle of a migration project and the Exchange 2010 servers aren’t doing much work yet because mailboxes haven’t been moved across from Exchange 2003 or 2007. In this scenario it would be reasonable to increase the settings that control maximum moves per server and per database to accelerate mailbox moves. However, once you’ve reached a condition where the Exchange 2010 mailbox servers have picked up the majority of the user load, you should throttle back the settings to the recommended values. See this post about batching mailbox move requests and this one about how to clear out mailbox move requests once they are complete.
You’ll have to restart the Mailbox Replication service to make any change to a setting effective. Remember that MRS runs on every CAS, so if you want a setting to be effective for MRS throughout the organization, you’ll have to apply the change to each and every CAS. In addition, you will have to check that the change is required and then reapply it (or not) following software upgrades (maybe not for a roll-up update, definitely required for a service pack).
Follow Tony @12Knocksinna
Thoughtsofanidlemind's Blog 
Changing EWS webconfig in order to enable MRS Proxy – this should be added in the article. Great stuff, congrats for Tony.
– Caio Ribeiro Cesar
* forgot to add that I am talking about MRS Proxy config for O365.
I have a situation where the local move sits at 95 percent with “Waiting for mailbox changes to replicate.” shown in the log at that point, then after about 30 minutes the log shows “Fatal error MailboxDataReplicationFailedPermanentException has occurred.” followed by “at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.WaitForMailboxChangesToReplicate(Object[] wiParams)” and “at Microsoft.Exchange.MailboxReplicationService.CommonUtils.CatchKnownExceptions(GenericCallDelegate del, FailureDelegate failureDelegate)” then finally relinquishing job.
I will investigate the config and see if I can relate these messages to actual values.
Thank you for the insight this article has provided.
The problem is that MRS is unable to complete the mailbox move because the copies for the target database are not healthy. When a target database is in a Database Availability Group (DAG), MRS checks with Active Manager to ensure that database replication is proceeding normally before it proceeds with the final step of switching the Active Directory pointers for the mailbox from the source to target database. In this case, Exchange is telling you that the data constraint setting for the target mailbox database cannot be satisfied. Typically this means that the data transferred to move the mailbox has not been replicated to a second database copy in the DAG (or if you’re in a stretched DAG across two Active Directory sites, to a database copy in the other Active Directory site). MRS will pause for 30 seconds before checking with Active Manager to see whether replication (as measured by copy and replay queue lengths) is healthy and if not, it will pause for another 30 seconds. Eventually MRS will time out the move operation because DAG replication obviously hasn’t gotten any better and that indicates a real problem that has to be addressed by the administrator. The reason why this happens is that MRS doesn’t want to move a mailbox if there is any chance that the mailbox move might be later compromised by a crash/outage of a replicated database that isn’t fully up to date. If a database is up to date across its copies (the signs of a healthy DAG), then it’s OK to complete the move as the data is protected by the other copies.
TR
Hi Tony, I was thinking it might be related to a slow connection between our main and DR site. I am trying to move the last mailboxes from the Exch2003 box to the new Exch2010. I figured the problem was the VPN link because we moved mailboxes up to 5Gb in size when the DR machine was still being setup at the main site locally. The problem we are having has only occured since the DR machine was physically moved to the DR site.
I had thought of turning off the replication but the move request queues until you enable replication again, even if I managed to stop replication and do the local moves I foresee the replication failing due to the slow VPN. Frustrating.
We know the database is healthy because we had a failure at the main site the other day and the whole system switched over to the DR databasae (as it should) and we had no loss of mail. We are undertaking steps to reduce bottlenecks in the VPN and at the DR site.
Thank you.
Scott.
I have worked with MS Support on the same issue..
Problem:
==========
When moving mailboxes from one Exchange 2003 server to another Exchange 2010 server in the same domain, the moving process fails and return an error – MailboxDataReplicationFailedPermanentException
Cause:
==========
The SecondCopy constraint on the mailbox database. Basically, move mailbox operations that involve databases in a DAG environment are different than move mailbox operations on standalone databases. The active database, the passive database and log shipping are factors that affect move mailbox in a DAG environment. MRS checks with the Active manager component of the Exchange replication service before, during and right before completing a move request to see if the active copy is up, if log shipping is not lagging behind and if the passive copies are keeping up. The action taken depends on a property of the database called DataMoveReplicationConstraint. If this value is not set to SecondCopy, then the move operation assumes the database has a copy. That is the move operation does take into consideration log shipping and the passive copies. So if MRS thinks the data replication to the passive database copy is not fast enough, the move operation will fail.
Resolution:
==========
Set the SecondCopy constraint to none during migrations and reverted the Data Move Replication Constraint back after all migrations are complete
Set-Mailboxdatabase DB1 -DataMoveReplicationConstraint None
Addition info:
==========
MRS maximum wait time is configurable via the DataGuaranteeTimeout in msexchangemailboxreplication.exe.config file which is located c:\program files\Microsoft\Exchange Server\V14\Bin folder on the Exchange 2010 mailbox server. By default, it is set to 30 minutes. We may change to 1 hour for example: DataGuaranteeTimeout = “01:00:00”. In this case, the MRS maximum wait time will be 1 hour.
And restarted ‘Microsoft Exchange Replication’ service and do this configuration change on all your CAS servers
There are a couple of CAS servers in my environment used exclusively to perform Domino to Exchange migrations, using the Binary Tree tool set. We are also moving mailboxes from Exchange 2003 servers to Exchange 2010 using the new-moveRequest CmdLet. We would like to ensure that the MRS service on the CAS servers used for Domino Migrations are never used to perform Exchange to Exchange mailbox moves. Can I simply disable the MRS service on those two CAS servers? Would disabling that service cause any issues? Or, can I set the MaxTotalMovesPerMRS value to ‘0’ in the MSExchangeMailboxReplicationService.exe.config file on those two CAS servers? Would that prevent the MRS from doing mailbox moves?
Thank you,
Lex
You could disable MRS on those CAS servers. I can’t see that it would cause a huge heap of problems. Tweaking the replication configuration file to prevent the MRS instances picking up new jobs would also work.
I’d test this first though!
TR
Thanks Tony…I will test in a POC environment and provide feedback…
Lex
We confirmed in a lab environment that setting the ‘MaxTotalMovesPerMRS’ value to ’0′ in the MSExchangeMailboxReplicationService.exe.config stops the MRS from picking up mailbox move jobs…
Lex
Pingback: PowerShell Scripting in E14: EWS and the IPM.Configuration.Owa.UserOptions item | Digital Banterings
Pingback: Diagnosing and fixing slow migration times to Office 365 from Exchange 2010. – Get-SysadminBlog