MSExchange ActiveSync Event ID 1016

Customer with a single Exchange 2010 completely down.  The following error was spammed throughout the event log:

Log Name:      Application
Source:        MSExchange ActiveSync
Date:          18/01/2017 7:31:46 PM
Event ID:      1016
Task Category: Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EXCHANGE2010
Description:
Exchange ActiveSync has encountered repeated failures when it tries to access data on Mailbox server [EXCHANGE2010.domain.local]. It will temporarily stop making requests to the Mailbox server for [60] seconds to reduce load on that server. This delay may occur if the Mailbox server is overloaded. If this event is logged frequently, review the Application log on this server and the Mailbox server noted above for other events that could indicate the root cause of performance problems.
Additional information:
"serverFQDN=EXCHANGE2010.domain.local
Error 0:

ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=EXCHANGE/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=User.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273 
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=140]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081 
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065 
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805 
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583 
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778 
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730 
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
Error 1:

ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=sfso.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273 
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=133]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081 
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065 
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805 
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583 
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778 
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730 
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
Error 2:

ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=sfso.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711f
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273 
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=133]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081 
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065 
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805 
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583 
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778 
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730 
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
errorCount=3, backingOff=True".



I know there are a few causes of this error however in my instance it was due to Microsoft Exchange Replication Service not started.  The service had been misconfigured to run as a service account with invalid credentials.

Setting it back to local system resolved the issue and getting the customer back online.

The Active Sync service queries "Active Manager" which is part of the Exchange Replication Service to determine where the users active mailbox copy in a DAG resides - and still follows this model even if your not using DAG's in your environment.  Hence if Active Manager is not available, no one can locate their Active Mailbox.
Previous
Next Post »