The Store Driver is a very important Exchange component. Running on all hub transport servers, its function is to provide the mechanism to deliver inbound messages to mailbox databases. Unlike Exchange 2003 and previous versions, all messages go through a hub transport server, even if they are sent between two mailboxes in the same mailbox database.
Exchange 2010 SP1 includes new code to throttle or manage connections from the Store Driver to mailbox servers as it delivers messages in such a way that Exchange can isolate and limit the effect of any faults that occur. Most of the time, you’ll be unaware that throttling applies to the Store Driver. However, if you receive non-delivery notification (NDR) messages that report problems from “STOREDRV” (the Store Driver), your Exchange hub transport servers might be hitting a limit. For example, an entry in Microsoft’s Exchange 2010 forum reports this error:
432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded
As a quick search for “STOREDRV 432” reveals, the Store Driver issues a 432 NDR when it encounters other problems, so the important thing here is the text reporting that the “recipient thread limit exceeded”.
In most cases, the problem seems to be encountered with mailboxes that receive very heavy traffic such as those used for journaling. As you probably know, Exchange allows you to capture journal reports to any valid SMTP address. This facility allows you to direct journal traffic to a third-party journaling application but if you send journal reports to an Exchange mailbox, perhaps as an interim step before the reports are later moved to an archiving application, it’s possible that Exchange will encounter a throttle limit as the Store Driver attempts to deliver messages concurrently at times of peak demand. By default, Exchange 2010 SP1 will only attempt a single concurrent delivery to a mailbox to ensure that all of the mailboxes in a database receive a similar quality of service. This is what you’d want, but you can see that a journal mailbox might have many messages directed towards it concurrently.
The solution is to update the EdgeTransport.exe.config XML configuration file on the hub transport server to add two new keys to force Exchange to allow more than one concurrent delivery to a mailbox. Any text editor can be used to update the configuration file but it is best to update a test server first and measure the effectiveness of the change before you introduce it into production.
It is also best to make the update to all hub transport servers in the site that supports the journal mailbox so that the Store Driver has the same behaviour on all servers.
The new keys and their values are:
<add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />
You should restart the Exchange Transport service after making the change to force the Store Driver to respect the new setting.
As documented on Page 397 in chapter 7 of Microsoft Exchange Server 2010 Inside Out
| Connection | Limit |
| Hub transport server to mailbox server | Concurrent delivery to a mailbox: Only one message can be delivered concurrently to the same mailbox. This limit is in place to ensure that all the mailboxes in a database receive a similar speed and quality of service. |
| Concurrent delivery to a database: Only two messages can be delivered concurrently to the same mailbox database. This limit prevents a problem with one database affecting all the other mailbox databases on a server. The problem may be caused by an underlying hardware issue such as a failing disk or storage controller or a software bug. Exchange may reduce this limit automatically if it senses that the performance of a mailbox database is lower than normal. | |
| Concurrent delivery to a mailbox server: Only twenty-four messages can be delivered concurrently to the same mailbox server. A hub transport server can deliver to any mailbox server in the site. This limit prevents a problem with one mailbox server (that might be experiencing heavy load) from affecting delivery to all other mailbox servers in the site. | |
| Concurrent delivery per message across all mailbox servers: Only twelve concurrent deliveries per individual message can be attempted to all mailbox servers in the site. This limit prevents a problem with a corrupt or expensive message (such as one with a number of large attachments or several hundred recipients in the message header) from absorbing all available connections and therefore stopping or slowing down delivery from a hub transport server to the mailbox server within a site. | |
| Mailbox server to hub transport server | Concurrent submission to a hub transport server: The limit for the maximum number of connections is calculated as five times the number of available mailbox processor cores. This limit controls how many concurrent submissions can be made from a mailbox server to all hub transport servers in the site. |
| Concurrent submission from a mailbox database: Only four connections can be performed concurrently from a mailbox database | |
| Concurrent submissions from a single mailbox server: A hub transport server will only process at most twelve concurrent submissions from any single mailbox server. | |
| ` | Concurrent submissions to a hub transport server: A mailbox server will only make a maximum of fifteen concurrent submissions through any single hub transport server. |
| Site-wide hub transport to mailbox server | The upper limit for the number of connections from hub transport servers to mailbox servers across the site is 20 x the number of processor cores. For example, if there are two hub transport servers in the site, each equipped with four processor cores, the limit is 160 connections. |
| Site-wide mailbox servers to hub transport servers | The upper limit for the number of connections from mailbox servers to hub transport servers across the site is five times the number of processor cores. For example, if there are six mailbox servers in the site, each equipped with four processor cores, the limit is 100 connections. |
Connection limits imposed by Exchange on the Store Driver
Thoughtsofanidlemind's Blog 
This is exactly what Microsoft told us, but sadly even with a RecipientThreadLimit value of 2, Exchange 2010 doesn’t have the ability to handle the same message rate for journaling to a single mailbox that Exchange 2007 does on hardware with half the performance.
Any chance this change could be responsible for the terribly slow performance of mailbox moves (from a 2003 server)? seemed that moves were going MUCH faster for me just prior to installing SP1.
Hi Greg,
The simple answer is I don’t know if Store Throttling is responsible for the lower performance that you’re seeing with mailbox moves. Have you data to show whether moves are really going slower (time required to move a 500MB mailbox when servers are under similar load, for instance) or is it just a feeling in your water?
TR
unfortunately no benchmark. i can say that days prior to installing SP1 I was able to move (approx) a 1 GB mailbox in about an hour. Same server, three days later (after SP1 install) a 150MB mailbox is taking 3 hours (no exaggeration). I haven’t seen this on other servers I’ve worked on so it’s probably not the store throttling, but it’s hard not to think it has something to do with SP1 since that’s the only variable that changed.
I’ll be receiving Exchange 2010 Inside Out in a few days so hopefully I won’t have to spend as much time searching for solutions online anymore!!!
Hopefully you won’t be bored by the book!
I’m bemused at the slow down in throughput. I’ll ask some folks… There is an updated MRS configuration file used by SP1 that reduces the number of moves that can be going on concurrently for a target mailbox database and server (for example, reduction in active moves to a database from 5 to 2) but this is hardly causing a problem if you’ve only got a single 150MB mailbox to move.
TR
I checked with the MRS developers and there is nothing in SP1 throttling that would cause a reduction of throughput. The problem is more likely to be something due to an issue with the source mailbox – maybe MRS has not been able to read data as smoothly as the other times. Have you compared the mailbox move reports to see what happened on the two occasions? Hopefully this will identify where the issue lies.
TR
Yeah, i think i can safely assume the problem was with the source server. I can’t really complain, we deserve it for what we put our source server through – little guy should have given up years ago!
I only had two (in production) mailboxes left to move so I just had them exported manually. Just to drive myself crazy i ran another move on an old support mailbox of ours that had about 1GB of data – naturally the process flew and completed moving in about 20min.
If anyone else stumbles upon this problem, for me it seemed to get better after updating the source server NIC drivers, then disabling the TCP chimney feature on both servers. As it happens all too often we got through a workaround (manual export) before solving the actual problem.
Tony – thanks for the follow up. Can’t wait to dig into the book!
You mentioned Journaling mailboxes. This MS Support article seems to point out this fact as well.
http://support.microsoft.com/kb/2536313 – Slow message delivery and mailbox access for journaling mailboxes on an Exchange Server 2010 server
I have made the changes but still facing the same issue can anyone review my “EdgeTransport.exe.config” once
http://www.2shared.com/file/TbbPaSS_/EdgeTransportexe.html
Pingback: 432-4.3.2 STOREDRV and Store Driver throttling | Ilantz's Weblog
Came across this blog because we were getting “400 4.4.7 Message delayed, too many concurrent deliveries of this message at this time” in the Message Tracking Log and “Internet message id has reached thread limit 40.” In ExTra for Local Delivery. Your blog states, “Concurrent delivery per message across all mailbox servers: Only twelve concurrent deliveries per individual message can be attempted to all mailbox servers in the site.”, but the limit is calculated as five times the number of available hub processor cores. Increasing the processor cores from 8 to 16 increased the limit from 40 to 80.