AspNetCoreModule returns "The service is unavailable." on IIS 7.5

[andersnm]

Hi!

We have two servers "staging" and "live" running an ASP.NET Core RC1 application currently using HttpPlatformHandler. Our app has run fine for months on RC1.

"Staging": Windows Server 2012, IIS 8
"Live", Windows Server 2008 R2, IIS 7.5

Now we have updated our app to RTM and want to deploy-

First we installed AspNetCoreModule and deployed our RTM-updated app on staging, this went really smooth and had no problems whatsoever. Our app runs fine on the staging server.

Then we tried to follow the same steps on our live server, but it fails and now after days of studying the docs and binging(/googling) I'm out of ideas.

We installed AspNetCoreModule in the live environment and it installed without problems: It appears in IIS' list of "Modules", and the aspnetcore.dlls are indeed present in the filesystem. But our app doesn't start, the server simply returns "The service is unavailable".

Some observations:
- The application pool is started (common cause of 503)
- Cant see any permissions issues, because HttpPlatformHandler works (also common cause of 503)
- App pool configurations are identical on both servers
- Event Viewer has zero relevant log entries
- HTTPERR/IIS logs has zero relevant entries
- Setting stdoutLogEnabled="true" on in web.config does not produce any logs
- If I try to put an "App_Offline.htm" in the root folder, there is no change in behaviour
- IIS' Failed Request Tracing confirms the AspNetCoreModule indeed processes IIS messages and returns "The server is unavailable", but not why

I belive the sum of these issues indicate a sort of problem with the AspNetCoreModule, as it doesn't even attempt to load our app.

Not sure if it is related, but can mention the "Service Service" stopped by itself after installing the ASP.NET Core Module for IIS and rebooting. We were not able to restart it, giving the message "Error 86: The specified network password is not correct". After restoring the server from a week old backup, we reinstalled ASP.NET Core Module for IIS, rebooted, and now the "Server Service" runs, but are still getting "The service is unavailable" from IIS+ANCMI trying to show an "App_Offline.htm".

Any ideas where to begin diagnosing this issue?

[Tratcher]

@pan-wang?

[pan-wang]

Did the Krestrel process start? Could you please take a look at Windows event log to see whether there is anything suspicious?

[shirhatti]

@andersnm Can you share your FREB logs as well?

[andersnm]

There are no suspicious entries in the Windows event log / Event Viewer.

Here is a fresh FREB log requesting from the site with App_Offline.htm in place:
http://pastebin.com/aPY50tK8 (expires in a month)

After further testing, I have found the following;

1. My report above is likely what happens when the "Server Service" is stopped: My guess is ANCM fails with "Service unavailable." and zero hints anywhere when the "Server Service" is stopped. This is different behaviour from HttpPlatformHandler, which works fine without the "Server Service". Alas I don't have a spare server to verify my theory, and I can not find the ANCM source codes to check if it uses e.g UNC paths.
2. I redeployed our application to a new IIS site on the same server, and behold - it starts and runs! However, as soon as I put in the App_Offline.htm, its back to "Service unavailable" and zero hints in any logs. I wrote my initial post after a day diagnosing the server when the "Server Service" was stopped. After we restored the "Server Service" and reinstalled ANCM, I didn't try to launch the app, but only tried serve App_Offline.htm, which failed in the same manner, so I assumed the same problem persisted.

A better title for this issue might thus be "ANCM+App_Offline.htm fails with 'Server unavailable' on IIS 7.5". It is still unresolved.

Also does anybody know if ANCM has a dependency on the "Server Service" which HttpPlatformHandler lacks?

[pan-wang]

If app_offline.htm exists in the application root folder, ANCM will set the response status to 503. So 503 is expected instead of a surprise here. Did the content inside app_offline.htm file server? Was the FREB log captured with app_offline.htm presented? Was Krestrel process created if you rename the app_offline.htm.
You mentioned "server service", could you please be more specific about the service name? Please also note that HttpPlatformHandler's configuration is not fully compatible with that of ANCM, and make sure you did not accidently break the configuration.

[pan-wang]

I tried hosting asp.net core application on Win7 using IIS. It works after installing https://support.microsoft.com/en-us/kb/2533623 mentioned in https://github.com/dotnet/core-setup/issues/191. If you uses UNC, please make sure the identity of the application pool does have the access right to the corresponding share.

[andersnm]

When App_Offline.htm is present in the root folder, the server returns the string "The service is unavailable." (HTTP status code 503) and not the contents of App_Offline.htm.

If I rename App_Offline.htm, our app runs fine (as served by Krestrel?). The FREB was captured with App_Offline.htm in place.

I have assumed we're seing a "real" 503 where some underlying service being unavailable, f.ex like "Stopped application pool" as described here: https://docs.asp.net/en/latest/publishing/iis.html . We see different behavior in mostly identical live and staging environments, main difference being the OS/IIS version. I'm not too happy about deploying to a misbehaving environment.

To summarize what we saw:

1. Server runs normally for a long time
2. Install ANCM from cmd.exe: DotNetCore.1.0.0-WindowsHosting.exe OPT_INSTALL_REDIST=0 - reboot
3. "Server service" stops by itself and ANCM does not serve contents of App_Offline.htm
4. Restore server from backup pre-ANCM
5. Install ANCM from cmd.exe: DotNetCore.1.0.1-WindowsHosting.exe OPT_INSTALL_REDIST=0 - reboot
6. "Server service" does not break, but ANCM does still not serve contents of App_Offline.htm

The "Server service" is not directly a problem anymore and likely irrelevant to why App_Offline.htm isn't served. The exact name is "Server", under Administrative Tools -> Services. It's related to serving UNC paths. We're not using UNC paths. I mention this because the service stopped strangely/confusingly only after installing ANCM. It caused us a lot of problems, and I'm speculating about side effects.

Afaict, KB2533623 does not seem relevant.

[pan-wang]

When you put app_offline.htm, you saw "The service is unavailable" from remote machine, right?
This is by-design as for security purpose IIS does not flow the error details. If you want to remote client see the 503 error details, you need to follow this blog to turn on detailed error. https://blogs.msdn.microsoft.com/benjaminperkins/2012/06/10/displaying-detailed-error-messages-in-iis/

[andersnm]

As described in the blog, I went to "Error Pages" in IIS, selected "Edit Feature Settings..." and changed from "Custom Error Pages" to "Detailed Errors". And now App_Offline.htm shows. Thanks for that tips.

I had expected App_Offline.htm would serve to all users without special configuration (in IIS 7.5?).

This basically solves the issue for me: Ultimately it doesn't matter what message is shown to the end user as long as I understand "why" it's different between the OS/IIS versions.

[guardrex]

@pan-wang Do you want to address this in https://docs.asp.net/en/latest/hosting/aspnet-core-module.html#asp-net-core-module-app-offline-htm ?

Safe: Explain that remote clients will receive a 503 Service Unavailable remote error page when the app_offline.htm file is in place, while local clients will receive the app_offline.htm page ... and leave it at that.

Less safe: Explain the bit about the default behavior of IIS Error Pages ☝️ ... but add that in order to display the contents of the app_offline.htm file to remote clients, detailed error pages must be set (and/or cross link to that blog post). This seems like a bad idea to me: If they have something special that they want to say for a 503, then shouldn't they leave the default setting in place and create a custom 503 page for IIS?

cc/ @Rick-Anderson @spboyer @blowdart

[pan-wang]

It seems when user puts app_offline.htm, he does want pass the content to the remote client. I think we should change ANCM code to flow the content without asking user to config IIS error page setting. If you all agree, I will add this into vNext release. @guardrex @Tratcher @andersnm @shirhatti

[guardrex]

Cool, thx @pan-wang. @Rick-Anderson @spboyer let me know if we need something temporary there. I can add a "safe" explanation from my comment above if you like ... or maybe this issue isn't being raised enough to sweat it.

[blowdart]

I'd be ok with serving the contents because in a lot of host environments won't let you set custom 503 remote errors. Also this would match the behaviour on asp.net 4.5 (I believe)

[pan-wang]

the change has been checked in