Weird NDRs after migration to Exchange 2010

Recently we finished an E2007 migration to E2010.
No major issues, overall a smooth migration.
Users did not complain and did not report errors.
So we started to fear the worst, like everything was going to fall apart once we finished.
Well…It didn’t :)
But we got reports of a weird issue.

When some secretaries send out meeting requests they received an NDR like.

===================================================================
Your message could not be delivered to the following recipients:
Username.lastname@domain.com

The e-mailsystem of the recipient does not receive messages now. Try sending the message at a later time or try to contact the recipient directly.

Diagnostic data for administrators:

Username.lastname@domain.com
#550 4.3.2 Transport.Rules; message is deleted by mailbox rules ##

===================================================================

So initially we though; a user had a mailbox rule which deletes messages from specific users.
Embarrassing but not really an issue...
Well it turned out; the user did not have any rules on his mailbox. And more and more users started to complain about this issue.

After investigation I found that the NDRs were generated on the HUB server and it concerned only a few mailboxes. There were no transport rules defined on the HUB transport servers and no mailbox rules defined on the mailboxes.

We tried boosting the diagnostic logging on the HUB server but did not find anything useful.
So I started to look at the mailboxes.
I used MAPIMFC to open the mailbox and see what rules were defined on the inbox. I did find mailbox rules on the inbox although the user did not create mailbox rules.
As you can see in the picture below the name of the rule is “Schedule+ EMS interface”

If you open the PR_RULE_ACTIONS attribute you will see something like the listing below.

lpActions->ulVersion = 0x00000001 = EDK_RULES_VERSION
lpActions->cActions = 0x00000002
lpActions->lpAction[0x00000000]:
lpAction->acttype = 0x00000008 = OP_DELEGATE
lpAction->lpRes:
lpRes was NULL
lpAction->ulFlags = 0x00000000 = (null)
lpAction->lpadrlist:
lpAdrList->cEntries = 0x00000001
lpAdrList->aEntries[0x00000000].cValues = 0x0000000C
lpAdrList->aEntries[0x00000000].rgPropVals[0x00000000].ulPropTag = Tag: 0x0FFF0102
Type: PT_BINARY
Property Name(s): PR_ENTRYID, PR_MEMBER_ENTRYID, PidTagEntryId, PidTagMemberEntryId, ptagEntryId
DASL: http://schemas.microsoft.com/mapi/proptag/0x0FFF0102
Property Name(s): PR_DISPLAY_NAME, PR_DISPLAY_NAME_A, ptagDisplayName
Other Names: PR_DISPLAY_NAME_W, PidTagDisplayName
DASL: http://schemas.microsoft.com/mapi/proptag/0x3001001E
...
...
...
lpAction->ulActionFlavor = 0x00000000 = (null)
lpAction->lpPropTagArray = NULL
lpActions->lpAction[0x00000001]:
lpAction->acttype = 0x0000000A = OP_DELETE
lpAction->lpRes:
lpRes was NULL
lpAction->ulFlags = 0x00000000 = (null)
lpAction->ulActionFlavor = 0x00000000 = (null)
lpAction->lpPropTagArray = NULL

So we know now the rule was created by exchange and has to do with mailbox delegations.
When I opened a mailbox and checked the delegates at first I did not see anything out of the ordinary.

Delegate was added and the permissions were also set as default.
So everything seems OK, or not?
Well there is one difference with the defaults.
The default setting for request handling is:
“My delegates only, but send a copy of meeting requests and responses to (recommended)”
But here it is set to:
“My delegates only”
What does this mean?

Well when UserA has a delegate UserB, UserB normally has the calendar of UsersA in outlook (typically like a secretary).
So now UserA want to schedule an appointment and asks UserB to schedule a meeting.
UserB will create a meeting request for UserA and sets UserA as the owner of the meeting.
UserA is also in the recipientlist.
When UserB sends the meeting request, the exchange HUB transport server will process the message and he delegation mailbox rule which states “send meeting request and responses to my delegates and not to me”. So the exchange HUB transport deletes the message which was send to userA.

This seems like default behaviour only exchange 2010 sends an NDR. Which will cause confusion with the users. It looks like it's a bug in Exchange 2010.

To solve this issue just tick the “My delegates only, but send a copy of meeting requests and responses to (recommended)” setting in the delegates setting. Like the screenshot below.


There will still be a “Schedule+ EMS interface” mailbox rule on the mailbox; the only difference is that the OP_DELETE entry has disappeared from the rule.
Since we changed the setting we have not had any reports from users about these NDRs.

Public Folder Mayhem Exchange 2010

I have been working at a costomer on a Exchange 2010 implementation in an existing Exchange 2007 environment.

All went pretty smooth until we came to the public folder replication.

For some reason we can't replicate public folders.
We even could not replicate public folder hierarchy.

The HUB Transport server dropped the update message with the error:

554 5.6.0 STOREDRV.Deliver.Exception:ObjectNotFoundException;
Failed to process message due to a permanent exception with message The Active Directory user wasn't found.

In the event log we received an event:
Error
4-11-2009 12:52:23
MSExchange Store Driver
1020
MSExchangeStoreDriver

The store driver couldn't deliver the public folder replication message "Status Request (PublicFolder@domain.local)" because the following error occurred: The Active Directory user wasn't found..

For some reason the HUB Transport server is unable to resolve the recipient.

We ran into brickwalls over and over again and finally descided to scratch Exchange 2010 completely and start all over again....

Removing the last exchange 2010 mailbox server in your environment is a bit different than it was in exchange 2007.

Exchange 2010 uses so called arbitrary mailboxes.
These mailboxes can't be retrieved by the normal command Get-Mailbox.

You have to use:

get-mailbox -Arbitration

Name Alias ServerName ProhibitSendQuota
---- ----- ---------- -----------------
SystemMailbox{1f05a... SystemMailbox{1f0... exch-ut-mbx unlimited
SystemMailbox{e0dc1... SystemMailbox{e0d... exch-ut-mbx unlimited
FederatedEmail.4c1f... FederatedEmail.4c... exch-ut-mbx 1 MB

So to remove these mailboxes you have to use:

get-mailbox -Arbitration Remove-Mailbox -RemoveLastArbitrationMailboxAllowed -Arbitration

Then all mailboxes are removed and you can uninstall exchange 2010.

But because you removed some system mailboxes you cannot install exchange 2010.

It will fail because the federation mailbox and discover mailbox are missing.

To fix this issue you have to run setup.com /prepareAD.
Even if you already had prepared the AD you need to rerun this command.
The PrepareAD will create these accounts again so you can re-install Exchange 2010.

So back to my initial problem.
The public folder replication issue.

In our first configuration we had three servers.
In the main AD site:
1 HUB / CA server
1 Mailbox server

In a remote AD site:
1 Mailbox / HUB / CA server

To make sure PF replication is working we decided to begin with just one server running all roles.

Installation went smooth and all was up and running.
Then we configured the E2010 public folder database and setup replication.

NOTHING!!!
Same error, same problem.

Until now we have not figured out what the problem is and how we can fix it.

We see a couple of things:

1. Exchange 2007 deliveres the message to the Exchange 2010 HUB server
2. The event entry saying message could not be delivered.
3. The error:
   
   554 5.6.0 STOREDRV.Deliver.Exception:ObjectNotFoundException; Failed to process message due to a permanent exception with message The Active Directory user wasn't found.
   
4. in the connectivity logs we also see:
   
   08CC2B6692C30788,MAPI,mbx.domain.local,>,Starting delivery
   08CC2B6692C30788,MAPI,mbx.domain.local,>,Connecting to server mbx.domain.local session type Public Folder
   08CC2B6692C30788,MAPI,mbx.domain.local,>,Failed to connect to server mbx.domain.local
   08CC2B6692C30788,MAPI,mbx.domain.local,-,Messages: 0 Bytes: 0 Recipients: 0

At this point we are still trying to solve this issue.

I will keep you posted.

Combining two commands get-mailbox and get-mailboxstatistics

Did you ever needed to supply a list of mailboxes containing the e-mail address, DisplayName, Database, TotalSize of mailbox and total Items of the mailbox.
There are various ways of obtaining this information but the easiest way would be with powershell.
The information you need is supplied by two commands:

1. Get-mailbox
2. Get-mailboxstatistics

So how do you get the information of those two commands in a single output?

Get-mailbox Name select-Object DisplayName,PrimarySmtpAddress,Database
Get-Mailboxstatistics Name Select-Object TotalItemSize,ItemCount

I used to create arrays and then merge them together but for this relatively simple task you need at least 15 lines of code.
So I looked for other ways doing this.

To get the size of a mailbox you use the command:

Get-MailboxStatistics MBXName select TotalItemSize,ItemCount

This will produce:
TotalItemSize is displayed in Bytes which is not easy to work with so you can use the following expression to get it in another format.

Get-MailboxStatistics MBXName select {$_.TotalItemSize.value.toMB()},ItemCount

This will produce:

This is still not easy to work with. We need a different header for the mailbox size.

Get-MailboxStatistics MBXName select @{n="Size(MB)" ;E = {$_.TotalItemSize.value.toMB()}},ItemCount

This will produce:
So we can use an expression to reformat the output. So can we use another command in the same way we reformatted the output? Yes you can.

Get-Mailbox MBXName Select-Object name,primarysmtpaddress,DisplayName,Database, @{e = {Get-MailboxStatistics $_.name select totalItemsize}}

This will produce:
The columnheader of "Get-MailboxStatistics ......" is actually:

Get-MailboxStatistics $_.name PipelineBreakpointerE4078E3092DF4dd9A469F3DC0CBB505C(00000000000000000110) select totalItemsize

This is stil not easy to work with.
If you combine the statement to convert-to-MB with the get-mailboxstatistics you can get something like this:

Get-Mailbox MBXName Select-Object name,primarysmtpaddress, DisplayName,Database,@{n="Size(MB)";e = {$MBXstat = Get-MailboxStatistics $_.name; $MBXstat.totalItemsize.value.toMB()}},@{n="Items"; e = {$MBXstat = Get-MailboxStatistics $_.name ; $MBXstat.itemcount}}

This will output:
If you want to create an overview of a complete database you would use the following command:

Get-Mailbox –Database SRV1\SG01\DB01 Select-Object name,primarysmtpaddress,DisplayName,Database,@{n="Size(MB)";e = {$MBXstat = Get-MailboxStatistics $_.name; $MBXstat.totalItemsize.value.toMB()}},@{n="Items"; e = {$MBXstat = Get-MailboxStatistics $_.name ; $MBXstat.itemcount}}


Remco

Update-StorageGroupCopy : This task can only be performed locally on the source or target computer.

Ever experienced the following?

Update-StorageGroupCopy : This task can only be performed locally on the source or target computer. Whilst you are logged on the passive node of an Exchange 2007 CCR cluster.

Well I did, we couldn't figure out why this kept failing.
Even opened a support call with PSS.
In the end we found out the reason why this was failing.

In a cluster configuration there are two type of owners:

1. Preferred Owner which is set on cluster groups
2. Possible Owner which is set to cluster resources

Now it turns out when you normally configure say a two node cluster, you configure one node to be the preferred owner and you set both nodes to be the possible owner.

Well not in a CCR cluster, you have to set both nodes to be the preferred owner.

You can change the order in which they are used but both nodes should be configured as preferred owner.Once we set the preferred owners everything worked fine.

unfortunately there is no event or tool that reports that the passive node is missing from the preferred owner tab hopefully this will be highlighted in future releases of EXBPA.