Nothing changed, pinned versions, functions still broke at some point in time

[paulschroeder-tomtom]

🐛 Describe the bug
We are using Python flavored Azure Durable functions.

We realized some days ago that the local testing via the func tool did not work anymore:

$ func start --useHttps --verbose
...
[2023-12-22T13:33:52.512Z] Worker failed to index functions
[2023-12-22T13:33:52.512Z] Result: Failure
[2023-12-22T13:33:52.512Z] Exception: AttributeError: 'Blueprint' object has no attribute 'function_name'         
[2023-12-22T13:33:52.512Z] Stack:   File '/usr/lib/azure-functions-core-tools-4/workers/python/3.10/LINUX/X64/azure_functions_worker/dispatcher.py', line 345, in _handle__functions_metadata_request
...
[2023-12-22T13:33:52.512Z]   File '.../function_app.py', line 6, in module
[2023-12-22T13:33:52.512Z]     from landing import landing_page_bp
[2023-12-22T13:33:52.512Z]   File '.../orchestration-api/landing.py', line 15, in module
[2023-12-22T13:33:52.512Z]     @landing_page_bp.function_name(name='z_landing')
[2023-12-22T13:33:52.512Z] .
[2023-12-22T13:33:52.513Z] 0 functions found (Worker)
[2023-12-22T13:33:52.520Z] 0 functions loaded

Without us changing any code.

We are using the following versions:

• Python: 3.10.12
• Core Tools Version: 4.0.5455 Commit hash: N/A (64-bit)
• Function Runtime Version: 4.27.5.21554
• azure-functions-durable Python Module: 1.2.4

We were quite surprised to find out, that non of these versions have changed since last known working state to the currently broken state (we though we did our best not to run in such problems by pinning all of those versions).

Then we found out that there are newer versions of the azure-functions-durable module and we increased it subsequently to 1.2.8. With the intermediate versions we were facing the following behavior:

1.2.5

Same as 1.2.4.

1.2.6

Same as 1.2.4, 1.2.5:

1.2.7

While installing it we got following warning:

| Warning: The locked version 1.2.7 for azure-functions-durable is a yanked version. Reason for being yanked: This release is faulty and causes failures at startup. Please install `1.2.8` instead,

so we refrained from using it, though it worked:

$ func start --useHttps --verbose
...
[2023-12-22T13:54:25.568Z] Reading functions metadata (Custom)
[2023-12-22T13:54:25.577Z] 1 functions found (Custom)
[2023-12-22T13:54:25.582Z] 18 functions loaded
...

1.2.8

Finally did the trick.

So in essence this is solved BUT(!) the problem just appeared out of the blue and we want to know why, to avoid being surprised in the future, so:

• What caused the breakage?
• Did Microsoft change something?
• If so, what?
• Why did it happen?
• How do we prevent such breakages in the future?

🤔 Expected behavior
No errors popping up, out of the blue.

☕ Steps to reproduce

Any minimal reproducer we can use?

Here you'll find a minimal working example: #460 (comment) with the corresponding version it produces the given results.

What Durable Functions patterns are you using, if any?

None

Are you running this locally or on Azure?

Both

[davidmrdavid]

Thanks for the report, @paulschroeder-tomtom. This is concerning, but I think I know the root problem - I believe there was a breaking release of the Python worker that has made it to the latest core tools release. Let me reach out to the appropriate team to try to correct this.

[davidmrdavid]

Hi @paulschroeder-tomtom - apologies for the delay here. Can you confirm that your azure-functions version was also pinned? To my understanding, there was an accidental breaking change in that library that caused this error to start surfacing, and as a result we made a release to azure-functions-durable that circumvented the error. I'm working with the azure-function library owners to try and provide a backwards compatibility later, though it may take a bit to roll out. My apologies for the inconvenience here, our automation should have caught this (and we're strengthening it to preventing this kind of regression in the future).

[paulschroeder-tomtom]

@davidmrdavid We are managing our (sub)dependencies with a pyproject.toml/poetry.lock combination.

It should be locked, but since azure-functions seems to be a sub-dependecy of azure-functions-durable it resides in the poetry.lock which is automatically generated. It was not actively pinned by us.

It can happen, while doing a poetry update that sub-dependencies are updated according to the version constraints(!) of the dependencies (i.e. azure-functions-durable).

How would you suggest going forward?

[davidmrdavid]

Hi @paulschroeder-tomtom. Sorry for the delayed response, we've had a few competing threads.

How would you suggest going forward?

The azure-functions library owners are driving the fix for this issue here though unfortunately it's taken longer than expected. I've pinged them again to help drive the fix to merge and release, which should help re-establish backwards compatibility with prior versions azure-functions-durable.

It can happen, while doing a poetry update that sub-dependencies are updated according to the version constraints(!) of the dependencies (i.e. azure-functions-durable).

Unfortunately, I haven't had the opportunity to reproduce this behavior in poetry yet. However, the key point here is that the azure-functions library accidentally made a breaking change within a minor version release, which means that the azure-functions-durable version constraints failed to protect against this issue. That was a mistake on our end, and we're working with our partner teams to help prevent this from happening again.