Restarted host is not picking up newly-deployed assemblies

[brettsam]

Spawning this from #1690, as it is a separate issue.

While investigating #1690, I noticed that a customer was seeing ~20 simultaneous requests to shutdown the host, yet the process never recycled. The newly deployed assembly was never loaded so the site had to be restarted to pick up the changes.

Details on finding site name:

| where PreciseTimeStamp between (datetime(2018-03-08 11:00) .. datetime(2018-03-08 12:10))
| where Summary contains "c67d3f04-1134-4b2d-a4c3-266878710784"

And then for seeing the full timeframe of this occurrence:

| where PreciseTimeStamp between (datetime(2018-03-08 11:00) .. datetime(2018-03-08 12:10))
| where AppName == "{sitenamefoundabove}"
| project PreciseTimeStamp, Pid, Details, Summary, FunctionName  

You'll see that an environment restart was requested around 11:05, yet the process id never changed. That didn't happen until the next request around 12:05, at which point the host shuts down and the process id changes.

@simonness, FYI

[brettsam]

@fabiocav -- FYI, this is what we were chatting about earlier.

[paulbatum]

@brettsam This issue is relevant to V1 only, right?

[brettsam]

Yes -- v2 exits the process itself rather than signalling for an environment shutdown.

[brettsam]

I'm seeing more and more cases of this -- another just happened where the Health Monitor is trying to shut down the host over-and-over. We see the log that "Environment shutdown has been triggered," but the Pid never changes. Repeat forever.

[paulbatum]

I do not think the process ID is supposed to change in this case - the app domain should recycle and that should be sufficient. Now we're supposed to log the app domain during host start. Can you check the log to see if the app domain was changing after each HostingEnvironment.Shutdown ?

cc @pragnagopa

[brettsam]

Yeah, you're right. I see the AppDomain Id changing in this case, yet the site still fails the very first Host Health check, before any functions are executed. I think this may be a separate issue, then.

[paulbatum]

Can you send @mathewc a kusto query so that he can take a look.

[brettsam]

I'm re-titling this as I misunderstood how that API worked. It looks like the shutdown is happening -- but rather than recycling the process, it recycles the AppDomain. But that doesn't explain why we see apps occasionally using old assemblies after an AppDomain recycle.

We've had one internal report of this as well, and there are likely others that just don't realize it.

There's something fishy going on there.

[paulbatum]

I have a suspicion that this is due to VSTS default usage of the old kudu ZIP api. Things to check:

• does this issue ever occur if webdeploy is enabled
• does the old zip API block on full extraction, or does it return early? If it returns early, that could explain a test seeing the old assembly after deployment.

[simonness]

Hi @brettsam @paulbatum thanks for your help on this one so far. Was there a resolution?

I've seen something similar again today, this time 2 different versions of dlls running at the same time for quite a while after a deployment. I've raised #3119 for it. Much appreciated if you could take a look!

[brettsam]

@mathewc / @fabiocav -- I think we've discussed this in the past (#3119 is another instance).

In this case, it was a Web Deploy. I can see the host restarts and recycles the AppDomain, yet the previous file version is loaded. We can't see that in the logs, but the customer is reporting it. Could there be some file caching issue upon restart?

[fabiocav]

Going through some old threads and noticed this was waiting for a response. Is this still an issue?