Http request with header transfer-encoding set to chunked is not handled as expected

[soninaren]

Function runtime: v2
Environment: Core-tools and Azure

Repro steps:

1. Create a function app
2. Create a function using httpTrigger template
3. Send a http request with encoding set to chunked via the following this console app: repro

It works as expected for C# and csx functions apps. but for node the function runtime processes the request without the request body which results in a 400 Bad request.

[dkovela]

This is basically blocking the proposed design and implementation for an important project. If this is not fixed, we will have to re-design and implement the planned integration. Can you please fix this as priority ASAP or please let us know ETA if you have it.

[crklein]

Can we please get an update? Our project is in day by day slip without this resolved.

[soninaren]

cc: @fabiocav , @ColbyTresness

[ColbyTresness]

@mhoeger as well

[crklein]

We are still stuck and haven't heard anything regarding this issue. Can we please get a status update?

[joain946]

I'm experiencing the same issue except that I'm running with Java locally.
If I set a Content-Length then it works. But of course that should not be needed for a chunked transfer.
Any plan to create a correction for this, since it is a year old ticket?

My environment:
Java 11
Azure Functions Core Tools (3.0.2798 Commit hash: d9d99872a51c04b1c2783198b1ee1104da6b064f)
Function Runtime Version: 3.0.14191.0

[rkaw92]

Having the same problem with a Node.js function - a POST with chunked encoding results in both req.body and req.rawBody being undefined, so there doesn't seem to be a way to actually access the data. This is on a local environment started with func start - tested with both Functions 2.0 and the Functions 3.0 runtimes.

[fabiocav]

@soninaren (and others on the thread), there was an update recently made to the HTTP binding that could potentially help with this, so we need to validate whether that is indeed the case.

This is an opt-in behavior (as it does have security implications). In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

[fabiocav]

Triaging for assignment (for validation only)

[fabiocav]

@soninaren could you also follow up with a documentation issue to make sure this is publicly documented (with the warnings). @alrod should be able to assist here as well.

[tom-odon]

@soninaren (and others on the thread), there was an update recently made to the HTTP binding that could potentially help with this, so we need to validate whether that is indeed the case.

This is an opt-in behavior (as it does have security implications). In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

@fabiocav Can you detail more about what the security implications are?

[dvdgutzmann]

Hi!

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

@fabiocav Was this improvement only made to http binding for the v2 Function runtime or is this extensions configuration also available for v3 and v4? What about in-process vs. isolated? I'm unable to find any documentation on this config option.

We' are currently blocked by this with a dotnet-isolated function running .NET 5 on runtime v3. See Azure/azure-functions-dotnet-worker#713

Thanks! David

[dvdgutzmann]

Hi!

{ "extensions": { "http": { "EnableChunkedRequestBinding": true } } }

@fabiocav Was this improvement only made to http binding for the v2 Function runtime or is this extensions configuration also available for v3 and v4? What about in-process vs. isolated? I'm unable to find any documentation on this config option.

I just did a quick test with

curl -ivsS --trace -H "Transfer-Encoding: chunked" --data 123 -X POST http://localhost:7071/api/fn

and this setups

• v3 runtime, dotnet, .NET Core 3.1
• v3 runtime, dotnet-isolated, .NET 5
• v4 runtime, dotnet, .NET 6
• v4 runtime, dotnet-isolated, .NET 6

Observations:

1. The EnableChunkedRequestBinding option has no effect at all in v3 and v4 setups.
2. The functions running out-of-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated" pass on NullStreams to the functions - therefore the request body is NOT accessible by the function.
3. The functions running in-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet" behave like all of us expect them to do - the request body is accessible by the function.

Maybe I'm going out on a limb but I do consider the mishandling of chunked transfers in workers running out-of-process a bug - especially when none of it is documented anywhere.

@fabiocav Do you agree with my analysis? Even if you don't concur with my conclusion, did I get the facts straight?

Thanks! David

[Zhaph]

Just to add more keywords here:

This issue means that you cannot send requests to an HTTP Trigger on the function runtime using the HttpClient PostAsJsonAsync extension method as per aspnet/AspNetWebStack#252 and instead need to manually create a StringContent body when sending JSON requests to a Function App.

This is considered a "server side fix and issue".

[mrurb]

This is still an ongoing issue. Is there any progress to fix this issue?
We were looking into upgrading our function to .Net6 isolated, but this is currently stopping us.

[Rashair]

Same issue with Python - Content-Length header is required, otherwise the request is not parsed.

[MitchBodmer]

I feel like it's important to stress here again that this bug in combination with the fact that PostAsJsonAsync and PostAsync with JsonContent do not set the Content-Length means you can't post JSON to an isolated HTTP Azure Function without a non-obvious workaround. This also still does not seem to be documented anywhere, but I may have missed something. It also presents itself in a confusing way: as a completely missing request body with no warning.

I really can't think of a use case for an HTTP Triggered Azure Function that I've seen more often than "post JSON to it from a C# application", so I'm confused why this issue has languished for three years. Now that we're five days away from developers not being able to migrate to .net 7 functions unless they also move to isolated functions, this seems especially urgent.

@fabiocav am I missing something here?

[MitchBodmer]

#Azure/azure-functions-dotnet-worker#374

Here is another issue where a number of developers wrestled with this issue and never got to the root cause, though they did find the workaround.

[mrurb]

@cloudmelon I was made aware that you might be able to help us with this issue.
It is currently stopping our and other companies from upgrading to Isolated Workers.

[cloudmelon]

Thanks, @mrurb we'll take a look at this and get back to you.

[cloudmelon]

Hi @mrurb i checked with the team, and this is an opt-in behavior that's not enabled by default due to the security implications. In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

And this should be able to resolve your issue, let us know if that works and hopefully unblock your customers. I am tagging @fabiocav here for vis.

[mrurb]

Hi @mrurb i checked with the team, and this is an opt-in behavior that's not enabled by default due to the security implications. In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

And this should be able to resolve your issue, let us know if that works and hopefully unblock your customers. I am tagging @fabiocav here for vis.

Hi @cloudmelon as mentioned earlier in this thread, the configuration in the host.json seems to have no effect.

Hi!

{ "extensions": { "http": { "EnableChunkedRequestBinding": true } } }

@fabiocav Was this improvement only made to http binding for the v2 Function runtime or is this extensions configuration also available for v3 and v4? What about in-process vs. isolated? I'm unable to find any documentation on this config option.

I just did a quick test with

curl -ivsS --trace -H "Transfer-Encoding: chunked" --data 123 -X POST http://localhost:7071/api/fn

and this setups

• v3 runtime, dotnet, .NET Core 3.1
• v3 runtime, dotnet-isolated, .NET 5
• v4 runtime, dotnet, .NET 6
• v4 runtime, dotnet-isolated, .NET 6

Observations:

1. The EnableChunkedRequestBinding option has no effect at all in v3 and v4 setups.
2. The functions running out-of-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated" pass on NullStreams to the functions - therefore the request body is NOT accessible by the function.
3. The functions running in-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet" behave like all of us expect them to do - the request body is accessible by the function.

Maybe I'm going out on a limb but I do consider the mishandling of chunked transfers in workers running out-of-process a bug - especially when none of it is documented anywhere.

@fabiocav Do you agree with my analysis? Even if you don't concur with my conclusion, did I get the facts straight?

Thanks! David

Just to make sure that anything has not changed since last tested, I just tested it and it still has no effect:
with v4 runtime and dotnet-isolated and EnableChunkedRequestBinding set to true any chunked request still returns a null stream:

Running a request without chunked works fine:

[yggdrasil-tynor]

@cloudmelon @fabiocav
We really need some clarification around this bug. I hope that this will eventually be fixed, but if there are no plans you should really let the community know so we can settle on other solutions. This was opened 3.5 years ago!! 👎

[cloudmelon]

Hi @mrurb i checked with the team, and this is an opt-in behavior that's not enabled by default due to the security implications. In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

And this should be able to resolve your issue, let us know if that works and hopefully unblock your customers. I am tagging @fabiocav here for vis.

Hi @cloudmelon as mentioned earlier in this thread, the configuration in the host.json seems to have no effect.

Hi!

{ "extensions": { "http": { "EnableChunkedRequestBinding": true } } }

@fabiocav Was this improvement only made to http binding for the v2 Function runtime or is this extensions configuration also available for v3 and v4? What about in-process vs. isolated? I'm unable to find any documentation on this config option.

I just did a quick test with

curl -ivsS --trace -H "Transfer-Encoding: chunked" --data 123 -X POST http://localhost:7071/api/fn

and this setups

• v3 runtime, dotnet, .NET Core 3.1
• v3 runtime, dotnet-isolated, .NET 5
• v4 runtime, dotnet, .NET 6
• v4 runtime, dotnet-isolated, .NET 6

Observations:

1. The EnableChunkedRequestBinding option has no effect at all in v3 and v4 setups.
2. The functions running out-of-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated" pass on NullStreams to the functions - therefore the request body is NOT accessible by the function.
3. The functions running in-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet" behave like all of us expect them to do - the request body is accessible by the function.

Maybe I'm going out on a limb but I do consider the mishandling of chunked transfers in workers running out-of-process a bug - especially when none of it is documented anywhere.
@fabiocav Do you agree with my analysis? Even if you don't concur with my conclusion, did I get the facts straight?
Thanks! David

Just to make sure that anything has not changed since last tested, I just tested it and it still has no effect: with v4 runtime and dotnet-isolated and EnableChunkedRequestBinding set to true any chunked request still returns a null stream:

Running a request without chunked works fine:

The current HTTP model for isolated doesn't all head chunking scenarios, but we're working on a new design that will help resolve related challenges here

[mattchenderson]

Confirmed that EnableChunkedRequestBinding does not seem to work. We need to retriage this.

In addition to fixing the behavior, part of the work here is to ensure that this gets properly documented.

[MitchBodmer]

@fabiocav @mattchenderson Is there a reason this was moved out of Sprint 147, but not into any other sprint? I was looking forward to this being addressed.

[mattchenderson]

We've changed our sprint process and are trying to shift milestones to deliverables rather than sprints. As part of this, we clsoed out a bunch of sprint-related milestones, so that's what you're seeing here. It's still a work in progress. This item is still on our radar, although I don't have a specific ETA attached to it quite yet.

[aishwaryabh]

After looking through this issue, we've concluded that for dotnet-isolated that the currently used HTTP model doesn't support chunking, which is why the body content does not propagate.

However a workaround for this issue for .NET isolated is to use the new ASP.NET core integration HTTP model (which is still in preview). Note the feature flag does not have to be set in the host when using this new HTTP model. As for the other language workers, we will continue working on adapting this new HTTP model.

[m-khooryani]

Disappointing from Microsoft! It does not support basic features!
I will consider switching from Azure Functions in our application.

[paulsmithkc]

@aishwaryabh What's the ETA on finally getting streaming/chunking support in Node/JS?

This lack of support makes large data transfers impossible as an input or output of Azure Functions.

[hholst80]

Sorry but is this related to this issue?

https://techcommunity.microsoft.com/t5/apps-on-azure-blog/azure-functions-support-for-http-streams-in-node-js-is-now-in/ba-p/4066575

[magurotuna]

@hholst80 Thanks so much, that saves my life

[mnelsonwhite]

Is chunked transfer encoding still not supported?

[dkapadia-jb]

This still seems to be an issue. Is there a fix planned for those using .NET 8 isolated worker model with C#?

[mnelsonwhite]

As far as I can see we should be using Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore not Microsoft.Azure.Functions.Worker.Extensions.Http. That introduces a range of problems.

For middleware, buffering the body stream is a little tricky since the EnableRewind extension does not work (or I couldn't get it to work). The solution for me was to copy the source for this extension method into my code, which thankfully was pretty easy because the required fields and types weren't internal (so rare!).

Common middleware probably needs to try and differentiate between grpc request data, null stream, and aspnet request data.

Transitioning is an all or nothing approach since you can only use the host builder with either ConfigureFunctionsWebApplication or ConfigureFunctionsWorkerDefaults, but cannot work together.

The problem for us was compounded with Azure API started applying chunked transfer encoding to requests without it, and without and configuration to disable or control that behaviour, and no way to implement a policy to prevent it.

This does not give the perception of competence or value to customers when poor design and implementation hinders progress frequently.

[johnthom]

I just got bit by this. Is the solution really switching to asp.net bindings? This is a huge problem for me. We just upgraded 14 function apps to .net8 isolated and all have HttpTrigger endpoints. I have no control over consumer's transfer-encoding's!

[johnthom]

I just got bit by this. Is the solution really switching to asp.net bindings? This is a huge problem for me. We just upgraded 14 function apps to .net8 isolated and all have HttpTrigger endpoints. I have no control over consumer's transfer-encoding's!

I was able to work-around this issue as follows...

1. Replace the following package reference...
   "Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.3.0" with
   "Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="2.0.1"
2. Update the Program.cs builder code…
   var builder = new HostBuilder()
   .ConfigureFunctionsWebApplication()
   //.ConfigureFunctionsWorkerDefaults()