A recent email discussion between MVPs focused on a topic that I totally missed in my Microsoft Exchange Server 2010 Inside Out book
Database portability was a major focus for Exchange 2010 as it laid the foundation for the native high availability features centered on the DAG that now exist in the product. The unbreakable connection that exists in all previous versions of Exchange to tie a database to a specific server no longer exists and the active copy of a database can be mounted on any server within a DAG. But a limitation still exists in that Exchange defines the boundary for a database’s “switchability” to be the DAG that “owns” the database (beneath the surface, the Active Directory object for the DAG maintains a list of all the databases in the DAG). This small but important detail means that an Exchange 2010 mailbox database is much more portable than previously but boundaries still exist in some circumstances. The notion of a totally portable database is not yet implemented.
Databases owned by a DAG
Every mailbox database is “owned” by a mailbox server or a DAG. When a mailbox database is created on a standalone server, its owner or “master” is set to be that server. If the mailbox server is then added to a DAG, the databases owned by the server are taken over by the DAG as this then allows the DAG to move the database between servers in the DAG once copies are created.
The obvious solution is to use the move mailbox function to move all the mailboxes from the database to a database or databases in the second DAG. This is easy for a small number of mailboxes but requires some planning for databases that hold hundreds or even thousands of mailboxes, but it’s very feasible. In fact, Microsoft moves mailboxes all the time inside their Office 365 datacenters to balance mailboxes across the databases used for their Exchange Online service.
The steps involved are:
- Identify the mailboxes to be moved. You can use the Get-Mailbox -Database command to create a list of the mailboxes and output it to a text file or simply sort the display of EMC by database.
- Decide how and when to move the mailboxes. One approach is to move the mailboxes in batches and use the “suspend when ready to complete” flag. This tells the Mailbox Replication Service (MRS) to work in the background to create the new mailbox in the target database, enumerate the data in the source mailbox and copy it to the new mailbox and then pause to wait for you to release the move to completion. The copying of the data might be performed overnight or over a weekend when overall system and network demand is low. You can then check the move reports to ensure that all has gone well before resuming the moves. When the moves are resumed, MRS enumerates the source mailbox again to discover what changes have occurred since the move and then copy the delta to the new mailbox before switching pointers in Active Directory to complete the move.
- Check whether any arbitration mailboxes or discovery search mailboxes need to be moved. Remember that discovery search mailboxes can hold very large amounts of information retrieved through searches so these mailboxes can take a long time to move.
- After all the mailboxes have been moved, you can remove the mailbox database and its copies from the DAG and eventually delete it.
Moving mailboxes works well but can become a mundane exercise in shuttling data to and fro. A second idea was proposed by Tim McMichael of Microsoft. He thought that it would be possible to use a “swing server” to transport the database from one DAG to another. In this scenario, you would commission a specific mailbox server and introduce it into the first DAG. You then transfer a copy of the mailbox database to be transferred to that server and make it active. Next, you remove all other copies of the mailbox database so that the only copy is now present on the swing server. You then remove the swing server from the first DAG and add it to the second DAG before completing the process by creating whatever number of additional copies of the database are necessary in the second DAG. Finally, you remove the copy of the database from the swing server and remove it from the second DAG.
There’s no doubt that the swing server technique will work and I see a great deal of value in it in a situation where either multiple databases have to be moved between DAGs or the volume of the data in the mailboxes to be relocated is so large that you cannot use the move mailbox technique to accomplish the goal in a reasonable timeframe. However, there are some downsides to the swing server technique. First, it obviously requires additional hardware. Second, there will be a time when the mailbox database (or databases) lose redundancy as you have to reduce them to just one copy before it’s possible to move between DAGs. The redundancy can be reintroduced as soon as the databases join the second DAG but even so, it will take time for the copies to be seeded. Last, this is a moderately complex multi-step technique that I am loathe to recommend within a production environment.
The complexity can be largely addressed by scripting the steps to move databases, remove copies, remove from the first DAG, join the second DAG, and recreate copies but that script does not exist today and it needs to be developed, tested, and debugged. By comparison, we have lots of experience of moving mailboxes and Exchange 2010 allows the moves to be performed offline in the background, paused before final completion, and batched in convenient sets for movement. I therefore prefer the move mailbox approach.
Now that I have poured cold water on Tim’s idea, let me conclude by telling you that he is an acknowledged expert in all matters to do with Windows Failover Clustering and DAGs. Therefore, you should consider coming to listen to Tim and many other experts (including Kevin Allison, General Manager of the Microsoft Exchange engineering group) speak at the Fall 2011 Exchange Connections conference in Las Vegas (October 31 to November 3). An early bird discount of $100 is available until August 1. If you add the code “SPKR” to your registration, you’ll get an additional $50 discount.
– Tony
Thoughtsofanidlemind's Blog 
Pingback: Good read: Moving a database between DAGs « You've got mail!
Hi Tony, my scenario is related… I currently have a 4-node DAG with two members each in a west coast and east coast datacenter. East coast users DBs are active on the east coast and west on the west, and activation of the opposite coast DBs are blocked from autoactivating on the opposite coasts’ servers (we will fail over manually if necessary).
The problem as you might imagine is the scenario where the datacenters get cut off from each other – the west coast stays up (witness server is there) and east coast goes down.
I am planning to split things into two distinct DAGs, each with one server on the opposite coast (so a total of 6 servers, 3 for each dag split 2-1 between the respective active and passive datacenter). I have created the two new servers, and am seeding copies from the current west coast servers to the new server on the west coast (part of the “east” DAG ultimately).
This will be simple enough for this first DAG – the tricky part is what to do about actually splitting off the new DAG. What I’m wondering is how survivable database copies are when removing and adding DAGs… Basically I am hoping to seed the new server, then remove it and the other two dag members from the original DAG, and create a new DAG for the 3 of them… While hopefully then being able to use the already-seeded db copy. The idea of course being to avoid seeding 250 GB of data across the WAN again!
Any idea if this is possible, or am I going to have to suck it up and reseed the new DAG’s DBs?
Thanks!
Wes
Hi Wes,
You can certainly avoid sending the 250GB of database across the WAN to seed a new database copy. To do this, you’d run Add-MailboxDatabaseCopy and specify the SeedingPostponed parameter. This tells Exchange that you’ll take care of setting up the database copy from data that you’ve retrieved in some other way, such as sending a copy via FedEx or your shipper on choice! However, in this case, I think that you want to create a brand new DAG using a database from another DAG.
So you could…
1. Take a backup of the database that you want to transfer and make sure that the backup is successful.
2. Run Remove-MailboxDatabaseCopy to remove all copies of the relevant database copy from the servers in the East DC.
3. Run the same cmdlet to remove all databases from Server4, which we want to move to the new DAG.
4. Once all database copies have been removed from Server4, run Remove-DatabaseAvailabilityGroupServer to remove Server4 from the original DAG. It now becomes a standalone server.
5. Create the new DAG in the West DC and add Server4 to this DAG.
6. Run New-MailboxDatabase to create a new database on Server4.
7. Dismount the new database
8. Restore the copy of the database from the original DAG to the location of the database that you’ve just created on Server4.
9. Mount the database. You might have to run ESEUTIL to convince Exchange that the database has been closed off cleanly and therefore can be mounted.
Caveats:
a) I haven’t tested this. You obviously need to run a test to verify everything.
b) Advice given on a web site is worth the price you pay for it. In this case, zero.
– Tony
Thanks Tony! Not sure I quite explained myself properly though… As it stands now I have two west coast servers w1 and w2 and one east coast server e3 in the West DC, and two east coast server e1 and e2 and one west coast server w3 in the East DC. E1/e2/w1/w2 have copies of all DBs currently. I have also added copies of all East dbs to e3, having sourced their seeding from w1 to avoid WAN traffic. So East is all set!
Now I need to figure out how to break w1/w2 out of the original DAG and create a new one, adding in w3 – but taking advantage of the fact that e1/e2 currently have up-to-date copies of all the West DBs… So your solution makes sense initially but I’m not understanding how it is that I get w1/w2 out of the original DAG and into the new DAG with w3 and still have up-to-date copies on all 3 servers without any reseeding…
Let me see if I follow:
1) I can remove copies from w2, making w1 the sole owner of all west DBs
2) break out w1/w2 and create a new DAG along with w3
3) re-add copies to w2, seeding on the LAN relatively quickly
4) add copies to w3, seeding via a db backup from e1??
Step 4 is where I’m fuzzy… thanks!
-Wes
You have to remove all database copies from the W1/W2 servers before you can evict them from the original DAG. You can leave the West databases that you want to move on W1 because it will be the sole owner.
You can then bring W1 and W2 into the new West DAG. The databases owned by W1 are now available to the West DAG. You can also add the new W3 server to the West DAG to form the 3-server DAG. Create as many copies as you need within the new West DAG to protect the databases.
W3 is in the East DC. You can add a database copy to it from a database copy in the West DAG but state the SeedingPostponed parameter. Then, take database files that you have in the East DC (from a backup before you removed the copies to evict W1/W2 from the original DAG) and use them as the database files for W3.
Clear as mud, and testing 1000% required.
TR
Got it! I guess my confusion comes from the disparity that will occur durin the time lapse between the backup from e1, and getting things up and running in the new DAG… will Exchange be smart enough to use that backup as a base and simply add to it when the database copy is resumed? My experience in the past is that exchange is picky, often ending up requiring a full reseed even when one doesn’t seem to be warranted… Are there specific things you recommend to keep w1’s active database “in line” so that the e1 backup will be sure to be a suitable seed?
Thanks again!
1. Minimize the amount of user activity while you perform all the actions to reduce the divergence between the active and newly seeded copies.
2. Have lots of coffee to hand.
3. Test everything and write down the steps in great detail so that you know exactly what to do when. Even better, script everything so that it’s as automatic as possible.
Thanks Tony. I unfortunately didn’t have the luxury of testing on this one – but I am working from Asia so scheduling a brief downtime at 2am was relatively painless!
I went ahead and did the following:
Removed west coast DB copies from e2.
Removed all w2’s DB copies (but didn’t delete the files) and removed it from DAG1.
Created DAG2 and added w2 to it.
Deleted my generic server resource from DAG1 and recreated it within DAG2 so that CAS services were unaffected (we use a generic cluster resource/ip address to achieve failover for CAS without expensive hardware LBalancers – we miss out on load balancing this way but it isn’t necessary for our load)
Added w3 to DAG1 and created copies of all west coast DBs, seeding from e1.
Removed west coast DBs from e1.
Dismounted all DBs on w1.
Removed all DB copies from w3 (but didn’t delete the files).
Removed w1 and w3 from DAG1, added them to DAG2.
Readded all DB copies to w3 with “-seedingpostponed” – after a minute or two exchange realized the previous files existed and everything showed up as healthy.
Readded all DB copies to w2 with “-seedingpostponed” – w2 was a bit behind since it was removed 24 hours before the rest, but it soon caught up and all DBs became healthy.
Worked great! Thanks again for your help…
Fantastic! Well done… and thanks for sharing with the Exchange community.
TR
It was cool especially the scenario in the comments 🙂
Hi there Tony. This is a bit off topic, but definitely on topic of DAGs. I have a situation where I need to change the IPs on a 2 node DAG due to some networking requirements that have come about. Is this a supported operation that you know of? If so, what would I do first? I’m thinking the most obvious steps are to re-ip the NICs on each server, then via the EMC to re-IP the DAG itself and ensure DNS is up to date. Any insight into this potentially precarious (because I’ve never tried it) situation would be greatly appreciated! Thanks for your blog and advice!!! Regards, Ryan
Hi Ryan,
I haven’t done this before, but I think the approach you’ve outlined sounds reasonable. I would do this at the weekend (obviously) and I’d do it after testing the steps on a test DAG too. You should be able to set up such a DAG as virtual machines on a laptop and see how it goes there… and then do it in practice!
Take care.
TR
Thanks Tony for the advice. Once I test, I’ll let you know the outcome here! Regards, Ryan
Pingback: Moving A Database Between DAGs | Fix My Exchange MonkeyBoy
Tony,
I realize this is an old thread but I have a question. I have a DAG (DAG1) in our corp site and several stand alone servers at field offices. I need to move roughly 300GB of mail to the corp office from one of the field offices. My plan was to:
– send a server to the field office
– create a DAG (DAG2) with the old and new server
– add a copy of the db to the new server
– ship the server back to corp and let the db sync
– change the RPCClientaccess to the new server
– remove the db from the field server
– Remove new server from DAG2 and add to DAG1
– start moving mailboxes into the corp mailbox servers.
I only have a 6MB pipe to the field office so individual mailbox moves take too long. Does my process make sense? Am I leaving out any steps?
Thanks,
Jack
Hi Jack,
To make sure that I understand you…
>>> send a server to the field office
So you already have a server in the field office with the 300GB of email in its databases.
>>> create a DAG (DAG2) with the old and new server
Hopefully the two servers (old and new) run the same operating system and version of Exchange.
>>> add a copy of the db to the new server
No problem. You will let the full database synchronize to its copy on the new server and leave the copy on the old server as the active.
>>> ship the server back to corp and let the db sync
So this is an incremental synchronization and hopefully you will be able to ship the server back quickly so that there isn’t a lot of data to move over the 6MB pipe.
>>> change the RPCClientaccess to the new server
What precisely do you mean by this (what commands are you executing and what do you intend)?
>>> remove the db from the field server
So you will activate the copy on the new server (now located in HQ) and then remove the now passive copy on the old server. Will the 6MB pipe be capable of handling the client traffic to the new server?
>>> Remove new server from DAG2 and add to DAG1
Once there is only a single copy of every database on the new server, it can be removed from DAG2. It can then be added to DAG1 and its database becomes owned by DAG1.
>>> start moving mailboxes into the corp mailbox servers.
Presumably the corporate mailbox servers are part of DAG1 and so the mailbox moves will be protected by replication. In fact it doesn’t make too much difference as MRS is pretty good at protecting mailbox data during moves and the moves will happen locally.
>>> Am I leaving out any steps?
I’d be worried that the pipe from the field office back to HQ will not handle the client traffic. Do you have any idea what kind of client traffic is normally generated at peak time? And will it contend with any existing traffic that passes across the link?
As to the procedure, it seems pretty OK to me.
TR
>>> send a server to the field office
So you already have a server in the field office with the 300GB of email in its databases.
-> Yes. The server has 450 GB of mail. 150GB of the mail belongs to users at that site. The other 300GB belongs to users at different sites that don’t have direct connections to that site.
>>> create a DAG (DAG2) with the old and new server
Hopefully the two servers (old and new) run the same operating system and version of Exchange.
-> Absolutely.
>>> ship the server back to corp and let the db sync
So this is an incremental synchronization and hopefully you will be able to ship the server back quickly so that there isn’t a lot of data to move over the 6MB pipe.
-> I would run a quick backup on a Friday to truncate the logs and have it fedexed next day. Hopefully not much to sync up.
>>> change the RPCClientaccess to the new server
What precisely do you mean by this (what commands are you executing and what do you intend)?
-> The RPC Client Access server for the database is currently the old server. So, outlook clients connect to it to get to the database. I am assuming I would have to change it before I removed the copy from the old server. Outlook keeps an affinity to the original cas server until you move it to a different database. I can probably get around this by creating another database and making the new server the client access – so disregard the statement/question. The command I was referring to is:
Set-MailboxDatabase “DatabaseToMove” -RpcClientAccessServer newserver.mydomain.com
>>> remove the db from the field server
So you will activate the copy on the new server (now located in HQ) and then remove the now passive copy on the old server. Will the 6MB pipe be capable of handling the client traffic to the new server?
-> None of the users will be using the 6MB pipe to get to their mailboxes – they are at different sites. Should be good.
Thank you, Sir. I really appreciate this blog and your quick response.
Tony,
First, love your blog and been reading Win IT Pro for years!
Although the -SeedingPostponed switch has been around now for a while, there is little documentation on how to execute it. I understand the concept and the command line, but the procedure is hazy. I have 2 local servers in a DAG, one active one passive. I want to add a lag copy on a 3rd that is geographically distant. Question is, can I use the passive copy to seed with? If so, I am not sure of the specific order to do it. Here is what I would think:
1- suspend replication
2- copy the passive DB to a portable HDD
3- create lag copy in powershell (this is where it gets fuzzy)
4- copy the lagged copy, (now a day old) to the new box and run Update-MailboxDatabaseCopy
Will this work? Any reason to use a lag copy over a 2nd passive one these days?
Hi Ian,
Like any reasonably complex operation that you haven’t tried before, this is one that I would attempt in a lab first. Your logic seems fine but it is so long since I tried this (to create a “Fedex” copy) that I can’t be 100% sure. That, plus the fact that I am traveling and don’t have access to lab systems, makes me a tad cautious.
As to a lab copy versus 2nd passive: Exchange 2013 makes lag copies more manageable, I think, and some advantages exist in terms of being able to keep physical corruptions at arm’s length. But I think I would prefer to have a 3rd database copy so that it’s available for things like single page patching.
TR
Tony,
To update this post, I wanted to share my findings: I found a technet article which I was able to test in my lab and was successful. Of course, a 300MB database is always easier than a 300GB database, but the steps are the same. See under Manually Copy an Offline Database: http://technet.microsoft.com/en-us/library/dd351100(v=exchg.150).aspx
Okay, I have yet to find definitive steps to UPGRADE ( –> migrate) from a 2010 DAG to a 2013 server / DAG scenario. Does anyone have links? Steps? Tips? It’s like Microsoft went BRAIN-DEAD when they talk about “upgrade/migrate from 2010 to 2013.” They assume NON-DAG in all their scenarios. My 2013 is up, and is standalone – my two 2010 nodes are up, and are DAG1. Do I set up a WFS member server for the 2013 box and then somehow introduce it into the 2010 DAG? Inquiring minds need to know! FYI, ZERO useful steps found on moving a mailbox from a DAG to a standalone server (maybe that’s in that last “Ian Fischer” comment above) – I could go that route; move all 2010 DAG mboxes to 2013 standalone – offline, I guess? Then, create a new 2013 DAG, once we have the new 2nd server for 2013. Meanwhile, we would be running standalone, no DAG on 2013, making sure all of it gets backed up. Tips? Links? Steps? Anybody? Thanks in advance!
One more update: Old 2010 2-node DAG is on 2008 R2 SP1; new 2013 server is on Windows Server 2012 R2 SP1 – I just saw one note that said we cannot run mixed operating systems in a DAG. So, if that’s true, it kind of seals my fate. HELP ????
You absolutely cannot combine different versions of the O/S or Exchange within a DAG. All DAG members have to run the same version of the O/S (required by the underlying cluster) and Exchange (a difference of one in cumulative updates is OK to allow for DAG member updates).
The normal approach is to create a new DAG (Exchange 2013) and then transfer mailboxes from the old (Exchange 2010) DAG to the new. You can start off with a two-node Exchange 2013 DAG. As workload is transferred from old to new, you decommission servers from the old DAG and perhaps upgrade those servers to the new O/S and Exchange versions and then reintroduce them into the Exchange 2013 DAG.
TR