[Bug] Ray Serve End-to-End tutorial not working with Ray 1.9.0 on Windows

[pcmoritz]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Serve

What happened + What you expected to happen

When I run through the tutorial https://docs.ray.io/en/latest/serve/tutorial.html, at the part where it says

curl -X GET localhost:8000/Counter/

I'm getting this in the backend:

 pid=2676) ERROR:    Exception in ASGI application
 pid=2676) Traceback (most recent call last):
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 384, in run_asgi
 pid=2676)     result = await app(self.scope, self.receive, self.send)
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__
 pid=2676)     return await self.app(scope, receive, send)
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\ray\serve\http_proxy.py", line 262, in __call__
 pid=2676)     await _send_request_to_handle(handle, scope, receive, send)
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\ray\serve\http_proxy.py", line 85, in _send_request_to_handle
 pid=2676)     await result(scope, receive, send)
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\starlette\responses.py", line 153, in __call__
 pid=2676)     "headers": self.raw_headers,
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 459, in send
 pid=2676)     status_code=status_code, headers=headers, reason=reason
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\h11\_events.py", line 56, in __init__
 pid=2676)     self.headers, _parsed=_parsed
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\h11\_headers.py", line 81, in normalize_and_validate
 pid=2676)     raise LocalProtocolError("multiple Content-Length headers")
 pid=2676) h11._util.LocalProtocolError: multiple Content-Length headers
 pid=2676) ERROR:    Exception in ASGI application
 pid=2676) Traceback (most recent call last):
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 384, in run_asgi
 pid=2676)     result = await app(self.scope, self.receive, self.send)
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__
 pid=2676)     return await self.app(scope, receive, send)
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\ray\serve\http_proxy.py", line 262, in __call__
 pid=2676)     await _send_request_to_handle(handle, scope, receive, send)
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\ray\serve\http_proxy.py", line 85, in _send_request_to_handle
 pid=2676)     await result(scope, receive, send)
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\starlette\responses.py", line 153, in __call__
 pid=2676)     "headers": self.raw_headers,
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 459, in send
 pid=2676)     status_code=status_code, headers=headers, reason=reason
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\h11\_events.py", line 56, in __init__
 pid=2676)     self.headers, _parsed=_parsed
 pid=2676)   File "C:\ProgramData\Anaconda3\lib\site-packages\h11\_headers.py", line 81, in normalize_and_validate
 pid=2676)     raise LocalProtocolError("multiple Content-Length headers")
 pid=2676) h11._util.LocalProtocolError: multiple Content-Length headers

and curl returns

curl: (52) Empty reply from server

Versions / Dependencies

Ray 1.9.0

Reproduction script

See instructions above

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[simon-mo]

can you

• curl -v to show the request
• pip freeze | grep h11

[simon-mo]

likely related: python-hyper/h11#92

[pcmoritz]

(base) C:\Users\ray>curl --version
curl 7.68.0 (x86_64-pc-win32) libcurl/7.68.0 Schannel zlib/1.2.11 libssh2/1.8.2
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI UnixSockets libz

[simon-mo]

sorry i meant

curl -v -X GET localhost:8000/Counter/

for verbose output

[pcmoritz]

h11 version is 0.9.0

[pcmoritz]

(base) C:\Users\ray>curl -v -X GET localhost:8000/Counter/
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying ::1:8000...
* TCP_NODELAY set
*   Trying 127.0.0.1:8000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /Counter/ HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.68.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

[simon-mo]

aha this is fixed in v0.11.0 of H1 apparently?

2.5.2 v0.11.0 (2020-10-05)
New features:
• h11 now stores and makes available the raw header name as received. In addition h11 will write out header
names with the same casing as passed to it. This allows compatibility with systems that expect titlecased header
names. See #31.
• Multiple content length headers are now merged into a single header if all the values are equal, if any are unequal
a LocalProtocol error is raised (as before). See #92.

serve depends on uvicorn -> h11>=0.8.

maybe just upgrading h11 should help here.

[pcmoritz]

Thanks, let me try to upgrade my python installation :)

[pcmoritz]

Thanks, that indeed fixed the problem!