A customer running two Exchange 2013 CU9 multi-role servers in a cluster with an F5 load balancer contacted me this morning reporting issues where they could not create new Outlook Profiles for users. Users with existing Outlook profiles were not affected by the issue.
Users primarily connect with MAPI over HTTP which is enabled on the Exchange Organization configuration.
When a user attempted to create a new profile, they would receive the following error:
The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.
When running the Microsoft Exchange Remote Connectivity Analyzer against the Exchange organisation, the following errors were captured:
Testing MAPI over HTTP connectivity to server webmail.company.com
MAPI over HTTP connectivity failed.
Additional Details
HTTP Response Headers:
request-id: 1f79419f-0c42-4a38-bf07-99b1c6882928
Set-Cookie: ClientId=MUZOXYKGXN9GLA; expires=Sat, 13-Aug-2016 02:50:02 GMT; path=/; HttpOnly
Server: Microsoft-IIS/8.5
WWW-Authenticate: Negotiate,NTLM
X-Powered-By: ASP.NET
X-FEServer: SERVER2
Date: Fri, 14 Aug 2015 02:50:02 GMT
Content-Length: 0
Elapsed Time: 7764 ms.
Testing the MAPI Address Book endpoint on the Exchange server.
An error occurred while testing the address book endpoint.
Additional Details
Elapsed Time: 5784 ms.
Test Steps
Testing the address book "Check Name" operation for user TestUser@company.com against server webmail.company.com.
An error occurred while attempting to resolve the name.
Additional Details
A protocol layer error occured. HttpStatusCode: 500
FailureLID: 47372
FailureInfo:
###### REQUEST [2015-08-14T02:50:03.1064218Z] ######
POST /mapi/nspi/?mailboxId=e110343c-c351-4dac-b066-3b552417a51b@company.com HTTP/1.1
Content-Type: application/octet-stream
User-Agent: MapiHttpClient
X-RequestId: 57959cb1-fd5d-4ae1-ac55-4494f0a61748:1
X-ClientInfo: 6016fb10-ca94-4a67-847c-549dd99bacb8:1
X-ClientApplication: MapiHttpClient/15.1.243.0
X-RequestType: Bind
Authorization: Negotiate [truncated]
Host: webmail.company.com
Cookie: ClientId=SGMTABEYKDADLXOTCG
Content-Length: 45
--- REQUEST BODY [+0:05.159] ---
..[BODY SIZE: 45]
--- REQUEST SENT [+0:05.159] ---
###### RESPONSE [+0:05.781] ######
HTTP/1.1 500 Internal Server Error
request-id: 9653be79-d895-4cf3-8169-98ceeb218197
X-CalculatedBETarget: Server1.company.local
X-DiagInfo: Server1
X-BEServer: Server1
X-FailureContext: BackEnd;500;NTAw;U3lzdGVtLk5ldC5XZWJFeGNlcHRpb246IFRoZSByZW1vdGUgc2VydmVyIHJldHVybmVkIGFuIGVycm9yOiAoNTAwKSBJbnRlcm5hbCBTZXJ2ZXIgRXJyb3IuDQogICBhdCBTeXN0ZW0uTmV0Lkh0dHBXZWJSZXF1ZXN0LkVuZEdldFJlc3BvbnNlKElBc3luY1Jlc3VsdCBhc3luY1Jlc3VsdCkNCiAgIGF0IE1pY3Jvc29mdC5FeGNoYW5nZS5IdHRwUHJveHkuUHJveHlSZXF1ZXN0SGFuZGxlci48PmNfX0Rpc3BsYXlDbGFzczJjLjxPblJlc3BvbnNlUmVhZHk+Yl9fMmIoKQ==;;ProtocolError;
Persistent-Auth: true
X-FEServer: Server1
Transfer-Encoding: chunked
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Fri, 14 Aug 2015 02:50:08 GMT
Set-Cookie: X-BackEndCookie=e110343c-c351-4dac-b066-3b552417a51b=u56Lnp2ejJqBz5nPyM2cxpnSncebmdLLyZue0sfJnJ7SnM2Znc7Iys7LnJnMgYHNz87K0s/G0s7Mq8/NxcrPxc/H; expires=Sun, 13-Sep-2015 02:50:08 GMT; path=/mapi; secure; HttpOnly
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Also on only on Server1, the following error was experienced in the event log.
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 14/08/2015 5:54:03 PM
Event ID: 1309
Task Category: Web Event
Level: Warning
Keywords: Classic
User: N/A
Computer: Server1
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 14/08/2015 5:54:03 PM
Event time (UTC): 14/08/2015 9:54:03 AM
Event ID: 6e8a15487387414b9279078f9f6aba51
Event sequence: 2
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/2/ROOT/mapi/nspi-3615-130840196323691514
Trust level: Full
Application Virtual Path: /mapi/nspi
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\mapi\nspi\
Machine name: Server1
Process information:
Process ID: 13528
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM
Exception information:
Exception type: HttpException
Exception message: RpcObjectSetType
at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
RpcObjectSetType
at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
at Microsoft.Exchange.Rpc.ProcessAccess.ProcessAccessRpcServer.RegisterInterface(Void* ifSpec, ValueType mgrTypeGuid, _GUID* pMgrTypeUuid, Void* pMgrEpv, UInt32 flags, UInt32 maxCalls)
at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)
Request information:
Request URL: https://localhost:444/mapi/nspi
Request path: /mapi/nspi
User host address: ::1
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM
Thread information:
Thread ID: 81
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
After further investigation, we discovered that only mailboxes which resided on an Active database on Server1 experienced the issue creating Outlook profiles, the same server which experienced EventID 1309. After failing mailbox databases over to Server2, users were able to login to Outlook without issues.
After this we identified the issue was only present with Server1.
On Server1 we tested the NSPI website by navigating to the Exchange backend in IIS Manager and selecting Browse:444 (https).
On Server2 when repeat the same process.
On Server2 it prompted as authentication as normal.
This shows the MAPI/NSPI results as expected.
Resolution:
We rebooted Exchange Server 1 and after a reboot, it started responding to MAPI/NSPI requests as expected. We believe it was a side affect after the CU9 upgrade, however will monitor the situation and update this blog thread in the event the issue reoccurs.
Users primarily connect with MAPI over HTTP which is enabled on the Exchange Organization configuration.
When a user attempted to create a new profile, they would receive the following error:
The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.
When running the Microsoft Exchange Remote Connectivity Analyzer against the Exchange organisation, the following errors were captured:
Testing MAPI over HTTP connectivity to server webmail.company.com
MAPI over HTTP connectivity failed.
Additional Details
HTTP Response Headers:
request-id: 1f79419f-0c42-4a38-bf07-99b1c6882928
Set-Cookie: ClientId=MUZOXYKGXN9GLA; expires=Sat, 13-Aug-2016 02:50:02 GMT; path=/; HttpOnly
Server: Microsoft-IIS/8.5
WWW-Authenticate: Negotiate,NTLM
X-Powered-By: ASP.NET
X-FEServer: SERVER2
Date: Fri, 14 Aug 2015 02:50:02 GMT
Content-Length: 0
Elapsed Time: 7764 ms.
Testing the MAPI Address Book endpoint on the Exchange server.
An error occurred while testing the address book endpoint.
Additional Details
Elapsed Time: 5784 ms.
Test Steps
Testing the address book "Check Name" operation for user TestUser@company.com against server webmail.company.com.
An error occurred while attempting to resolve the name.
Additional Details
A protocol layer error occured. HttpStatusCode: 500
FailureLID: 47372
FailureInfo:
###### REQUEST [2015-08-14T02:50:03.1064218Z] ######
POST /mapi/nspi/?mailboxId=e110343c-c351-4dac-b066-3b552417a51b@company.com HTTP/1.1
Content-Type: application/octet-stream
User-Agent: MapiHttpClient
X-RequestId: 57959cb1-fd5d-4ae1-ac55-4494f0a61748:1
X-ClientInfo: 6016fb10-ca94-4a67-847c-549dd99bacb8:1
X-ClientApplication: MapiHttpClient/15.1.243.0
X-RequestType: Bind
Authorization: Negotiate [truncated]
Host: webmail.company.com
Cookie: ClientId=SGMTABEYKDADLXOTCG
Content-Length: 45
--- REQUEST BODY [+0:05.159] ---
..[BODY SIZE: 45]
--- REQUEST SENT [+0:05.159] ---
###### RESPONSE [+0:05.781] ######
HTTP/1.1 500 Internal Server Error
request-id: 9653be79-d895-4cf3-8169-98ceeb218197
X-CalculatedBETarget: Server1.company.local
X-DiagInfo: Server1
X-BEServer: Server1
X-FailureContext: BackEnd;500;NTAw;U3lzdGVtLk5ldC5XZWJFeGNlcHRpb246IFRoZSByZW1vdGUgc2VydmVyIHJldHVybmVkIGFuIGVycm9yOiAoNTAwKSBJbnRlcm5hbCBTZXJ2ZXIgRXJyb3IuDQogICBhdCBTeXN0ZW0uTmV0Lkh0dHBXZWJSZXF1ZXN0LkVuZEdldFJlc3BvbnNlKElBc3luY1Jlc3VsdCBhc3luY1Jlc3VsdCkNCiAgIGF0IE1pY3Jvc29mdC5FeGNoYW5nZS5IdHRwUHJveHkuUHJveHlSZXF1ZXN0SGFuZGxlci48PmNfX0Rpc3BsYXlDbGFzczJjLjxPblJlc3BvbnNlUmVhZHk+Yl9fMmIoKQ==;;ProtocolError;
Persistent-Auth: true
X-FEServer: Server1
Transfer-Encoding: chunked
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Fri, 14 Aug 2015 02:50:08 GMT
Set-Cookie: X-BackEndCookie=e110343c-c351-4dac-b066-3b552417a51b=u56Lnp2ejJqBz5nPyM2cxpnSncebmdLLyZue0sfJnJ7SnM2Znc7Iys7LnJnMgYHNz87K0s/G0s7Mq8/NxcrPxc/H; expires=Sun, 13-Sep-2015 02:50:08 GMT; path=/mapi; secure; HttpOnly
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Also on only on Server1, the following error was experienced in the event log.
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 14/08/2015 5:54:03 PM
Event ID: 1309
Task Category: Web Event
Level: Warning
Keywords: Classic
User: N/A
Computer: Server1
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 14/08/2015 5:54:03 PM
Event time (UTC): 14/08/2015 9:54:03 AM
Event ID: 6e8a15487387414b9279078f9f6aba51
Event sequence: 2
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/2/ROOT/mapi/nspi-3615-130840196323691514
Trust level: Full
Application Virtual Path: /mapi/nspi
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\mapi\nspi\
Machine name: Server1
Process information:
Process ID: 13528
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM
Exception information:
Exception type: HttpException
Exception message: RpcObjectSetType
at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
RpcObjectSetType
at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
at Microsoft.Exchange.Rpc.ProcessAccess.ProcessAccessRpcServer.RegisterInterface(Void* ifSpec, ValueType mgrTypeGuid, _GUID* pMgrTypeUuid, Void* pMgrEpv, UInt32 flags, UInt32 maxCalls)
at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)
Request information:
Request URL: https://localhost:444/mapi/nspi
Request path: /mapi/nspi
User host address: ::1
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM
Thread information:
Thread ID: 81
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
After further investigation, we discovered that only mailboxes which resided on an Active database on Server1 experienced the issue creating Outlook profiles, the same server which experienced EventID 1309. After failing mailbox databases over to Server2, users were able to login to Outlook without issues.
After this we identified the issue was only present with Server1.
On Server1 we tested the NSPI website by navigating to the Exchange backend in IIS Manager and selecting Browse:444 (https).
The browse failed on Server1 with an "unhandled exception error". This was aligned with the HTTP 500 exception shown in the Event Logs on Server1 and the Exchange Remote Connectivity Analyzer.
RpcObjectSetType
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: Microsoft.Exchange.Rpc.RpcException: RpcObjectSetType
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
On Server2 it prompted as authentication as normal.
This shows the MAPI/NSPI results as expected.
Resolution:
We rebooted Exchange Server 1 and after a reboot, it started responding to MAPI/NSPI requests as expected. We believe it was a side affect after the CU9 upgrade, however will monitor the situation and update this blog thread in the event the issue reoccurs.
This is dummy text. It is not meant to be read. Accordingly, it is difficult to figure out when to end it. But then, this is dummy text. It is not meant to be read. Period.
ConversionConversion EmoticonEmoticon