Worker fails to initialize from Core Tools when Worker Indexing is enabled in PowerShell 7.2 worker

[andystaples]

I am working on a version of the PowerShell worker which supports indexing of functions from within the worker, similar to PyStein. When the PowerShell 7.2 worker and the Function App report compatibility with worker indexing, and extension bundles are enabled, the Core Tools reports an issue starting the Host. Here is the error I get:

Investigative information

Log output from func host start --verbose:

Azure Functions Core Tools
Core Tools Version: 4.0.4590 Commit hash: N/A (64-bit)
Function Runtime Version: 4.5.2.18383

<other stuff>

[2022-07-20T18:00:26.555Z] Initializing Host. OperationId: '22148f0c-ba24-4fcc-82d7-dac02bdffa41'.
[2022-07-20T18:00:26.573Z] Host initialization: ConsecutiveErrors=0, StartupCount=1, OperationId=22148f0c-ba24-4fcc-82d7-dac02bdffa41
[2022-07-20T18:00:26.607Z] Loading functions metadata
[2022-07-20T18:00:26.611Z] Reading functions metadata
[2022-07-20T18:00:26.646Z] A host error has occurred during startup operation '22148f0c-ba24-4fcc-82d7-dac02bdffa41'.
[2022-07-20T18:00:26.666Z] Microsoft.Azure.WebJobs.Script: _dispatcher.
[2022-07-20T18:00:26.675Z] Stopping JobHost
[2022-07-20T18:00:26.679Z] Closing local http RPC endpoint: http://127.0.0.1:17071/durabletask/. InstanceId: . Function: . HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.7.1. SequenceNumber: 1.
[2022-07-20T18:00:26.704Z] Failed to stop host instance 'ed26e5ee-ced0-48fc-8169-6a9132fa4c96'.
[2022-07-20T18:00:26.704Z] Microsoft.Azure.WebJobs.Host: The host has not yet started.
Value cannot be null. (Parameter 'provider')
[2022-07-20T18:00:26.737Z] Initialization cancellation requested by runtime.
[2022-07-20T18:00:26.741Z] Stopping host...
[2022-07-20T18:00:26.762Z] Host shutdown completed.

Host error is reported immediately after reading the functions metadata, something to do with _dispatcher

Repro steps

Provide the steps required to reproduce the problem:

• Setup a PowerShell 7.2 Function App, include "AzureWebJobsFeatureFlags":"EnableWorkerIndexing" in the local.settings.json
• Modify the PowerShell worker to report compatibility with worker indexing by modifying worker.config.json to include "workerIndexing": "true" in both 'Azure Functions Core Tools\workers\powershell' and 'Azure Functions Core Tools\workers\powershell\7.2'
• Attempt to run the Function App

Expected behavior

When the PowerShell 7.2 worker and the Function App report compatibility with Worker Indexing, the Host should start from Core Tools.

Actual behavior

Core tools reports "A host error has occurred during startup operation "

Known workarounds

Removing the extensionBundle from host.json will prevent this error

cc @surgupta-msft @Francisco-Gamino
FYI @AnatoliB

[surgupta-msft]

@andystaples any reason why we are testing with Host 4.5.2 and not the latest one (4.8.0 or 4.7.x)?

Also, can you share all logs with verbose flag (either attach the file here or email me) as I cannot see if worker indexing was enabled in logs.

[andystaples]

Here is a log file containing full logs output from the latest Core Tools version at https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=v4%2Cwindows%2Ccsharp%2Cportal%2Cbash
Let me know if there is another version of the Core Tools/Host you want me to try.
Core Tools output verbose.log

[fabiocav]

Moving to sprint 128 so we can continue tracking the work, but this seems related to the Durable Functions extension

[ejizba]

I don't think this is related to durable. It reproduces for me based on this code path, aka I think any core tools, consumption, or premium environment would hit the bug. This is regardless of langauge - as I could repro on both Node.js and Python. For core tools, you can reproduce in the host by setting FUNCTIONS_CORETOOLS_ENVIRONMENT to true.

@surgupta-msft and I were working on this offline and we had some potential fixes already. I'm hoping we can get something merged for this soon as I think it's a pretty big blocker for all new programming models for all languages. Surbhi, let me know if you want to submit a PR or if you want me to do it

[surgupta-msft]

@ejizba Sorry for not being able to keep you in loop as there are multiple discussions happening around this issue.

The main observation is that Stein apps are working fine with core tools if Host.json does not have extension bundles. If a dependency would be missing in code, it should not work with and without extension bundles. We also found that if we just delete DurableTask.Netherite.AzureFunctions from extensions.json file then it all works fine with and without extension bundles. That is why we are investigating in that direction.

This is Netherite extension code
class NetheriteProviderStartup : FunctionsStartup

This is DurableTask extension code
internal class DurableTaskWebJobsStartup : IWebJobsStartup

We need extensions to implement IWebJobsStartup instead of FunctionsStartup and it will fix the issue. We are discussing with Netherite team to help us here. I will keep this thread updated.

To unblock your development, the workaround is to either remove extension bundles from Host.json or just remove Netherite from extensions.json.

[ejizba]

I still think it would be best if this was fixed on the host. If one random extension can cause this bug, then we are at risk of it reoccurring in the future. In addition, users with old extensions could still hit the bug and the error message is not at all helpful in terms of how to fix.

Lastly, what is the difference between FunctionsStartup and IWebJobsStartup? If they're both supported for different scenarios, then this definitely needs to be fixed in the host.