[ProxyPoolPlugin] Fails on HTTPS requests

[eugen1j]

Unable to send HTTPS requests using ProxyPoolPlugin.

Steps to reproduce:

1. Run proxy on ports 9000, 9001.
   proxy --port 9001 & proxy --port 9000
2. Run proxy with proxy ProxyPoolPlugin on default port.
   proxy --plugins proxy.plugin.ProxyPoolPlugin
3. Send the HTTPS request via curl to trigger an error.
   curl -vx http://localhost:8899 https://google.com/
   Request hangs on the CONNECT stage.

I expect the request to complete correctly.

Environment:

• proxy.py from develop branch.
• Python 3.7.5
• macOS Catalina 10.15.4

Details:

HTTPS request curl log:

 ~  curl -vx http://localhost:8899 https://google.com/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.64.1
> Proxy-Connection: Keep-Alive
>
* Proxy CONNECT aborted
* CONNECT phase completed!
* Closing connection 0
curl: (56) Proxy CONNECT aborted

HTTPS request proxy.py log:

2020-06-05 10:10:16,145 - pid:43789 [E] run:415 - Exception while handling connection <socket.socket fd=76, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 8899, 0, 0), raddr=('::1', 53503, 0, 0)>
Traceback (most recent call last):
  File "/Users/macbook/test-proxy/venv/lib/python3.7/site-packages/proxy/http/handler.py", line 405, in run
    teardown = self.run_once()
  File "/Users/macbook/test-proxy/venv/lib/python3.7/site-packages/proxy/http/handler.py", line 390, in run_once
    teardown = self.handle_events(readables, writables)
  File "/Users/macbook/test-proxy/venv/lib/python3.7/site-packages/proxy/http/handler.py", line 193, in handle_events
    teardown = self.handle_readables(readables)
  File "/Users/macbook/test-proxy/venv/lib/python3.7/site-packages/proxy/http/handler.py", line 349, in handle_readables
    upgraded_sock = plugin.on_request_complete()
  File "/Users/macbook/test-proxy/venv/lib/python3.7/site-packages/proxy/http/proxy/server.py", line 259, in on_request_complete
    r = plugin.handle_client_request(self.request)
  File "/Users/macbook/test-proxy/venv/lib/python3.7/site-packages/proxy/plugin/proxy_pool.py", line 49, in handle_client_request
    self.tunnel(request)
  File "/Users/macbook/test-proxy/venv/lib/python3.7/site-packages/proxy/plugin/proxy_pool.py", line 66, in tunnel
    response = self.conn.recv(DEFAULT_BUFFER_SIZE)
socket.timeout: timed out
2020-06-05 10:10:16,153 - pid:43789 [I] access_log:332 - ::1:53503 - CONNECT None:None - 0 bytes - 10025.00 ms

How can I modify proxy.plugin.ProxyPoolPlugin to work properly with HTTPS?

[abhinavsingh]

@eugen1j Thanks for reporting this.

Took a quick look and it fails for HTTPS because of wrong re-construction within rebuild_original_path. See here https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/plugin/proxy_pool.py#L70

For HTTPS requests rebuild_original_path returns b'://httpbin.org:443httpbin.org:443' which indeed ain't gonna work.

I am going to address this over the weekend. Meanwhile feel free to fix locally, contribute back if possible :)

Cheers!!!

[eugen1j]

@abhinavsingh thanks a lot for the quick and accurate answer 👍

I managed to fix it locally with a workaround

@staticmethod
def rebuild_original_path(request: HttpParser) -> bytes:
    """Re-builds original upstream server URL.

    proxy server core by default strips upstream host:port
    from incoming client proxy request.
    """
    if not request.url.scheme:  # Here the trick
        return request.path
    assert request.url and request.host and request.port and request.path
    return (
        request.url.scheme +
        COLON + SLASH + SLASH +
        request.host +
        COLON +
        str(request.port).encode() +
        request.path
    )

Now CONNECT request to the proxy works, but nothing happens next.

$> curl -vx http://localhost:8899 https://google.com/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.64.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to google.com:443
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to google.com:443

proxy.py log

2020-06-05 19:21:11,580 - pid:75892 [I] access_log:332 - ::1:57499 - CONNECT None:None - 0 bytes - 10070.92 ms

Request hangs until timeout. I suggest that we just send CONNECT request to create a tunnel. Now we need to send target HTTP request via this tunnel here https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/plugin/proxy_pool.py#L46

def tunnel(self, request: HttpParser) -> None:
    """Send to upstream proxy, receive from upstream proxy, queue back to client."""

    assert self.conn
    self.conn.send(request.build())
    response = self.conn.recv(DEFAULT_BUFFER_SIZE)
    self.client.queue(memoryview(response))

I tried to wrap self.conn into TLS using OpenSSL like this https://stackoverflow.com/a/9738403/8153147 and send HTTP request with no luck. Here an example of non-working code:

def tunnel(self, request: HttpParser) -> None:
    """Send to upstream proxy, receive from upstream proxy, queue back to client."""
    assert self.conn
    self.conn.send(request.build())
    response = self.conn.recv(DEFAULT_BUFFER_SIZE)
    self.client.queue(memoryview(response))

    def verify_cb(conn, cert, errun, depth, ok):
        return True

    ctx = SSL.Context(SSL.SSLv23_METHOD)
    ctx.set_verify(SSL.VERIFY_PEER, verify_cb)
    ss = SSL.Connection(ctx, self.conn)

    ss.set_connect_state()
    ss.do_handshake()
    cert = ss.get_peer_certificate()
    ss.send(b'GET http://google.com:80/123 HTTP/1.1\r\nHost: google.com\r\nUser-Agent: curl/7.64.1\r\nAccept: */*\r\nProxy-Connection: Keep-Alive\r\n\r\n')
    response = ss.read(DEFAULT_BUFFER_SIZE)
    ss.shutdown()
    ss.close()

Can you suggest how to rewrite the tunnel method for HTTPS support?

[abhinavsingh]

@abhinavsingh thanks a lot for the quick and accurate answer 👍

I managed to fix it locally with a workaround

@staticmethod
def rebuild_original_path(request: HttpParser) -> bytes:
    """Re-builds original upstream server URL.

    proxy server core by default strips upstream host:port
    from incoming client proxy request.
    """
    if not request.url.scheme:  # Here the trick
        return request.path
    assert request.url and request.host and request.port and request.path
    return (
        request.url.scheme +
        COLON + SLASH + SLASH +
        request.host +
        COLON +
        str(request.port).encode() +
        request.path
    )

Awesome. Though after #360 this hack should no longer be required. PTAL.

Now CONNECT request to the proxy works, but nothing happens next.

$> curl -vx http://localhost:8899 https://google.com/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.64.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to google.com:443
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to google.com:443

proxy.py log

2020-06-05 19:21:11,580 - pid:75892 [I] access_log:332 - ::1:57499 - CONNECT None:None - 0 bytes - 10070.92 ms

Request hangs until timeout. I suggest that we just send CONNECT request to create a tunnel. Now we need to send target HTTP request via this tunnel here https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/plugin/proxy_pool.py#L46

def tunnel(self, request: HttpParser) -> None:
    """Send to upstream proxy, receive from upstream proxy, queue back to client."""

    assert self.conn
    self.conn.send(request.build())
    response = self.conn.recv(DEFAULT_BUFFER_SIZE)
    self.client.queue(memoryview(response))

I tried to wrap self.conn into TLS using OpenSSL like this https://stackoverflow.com/a/9738403/8153147 and send HTTP request with no luck. Here an example of non-working code:

def tunnel(self, request: HttpParser) -> None:
    """Send to upstream proxy, receive from upstream proxy, queue back to client."""
    assert self.conn
    self.conn.send(request.build())
    response = self.conn.recv(DEFAULT_BUFFER_SIZE)
    self.client.queue(memoryview(response))

    def verify_cb(conn, cert, errun, depth, ok):
        return True

    ctx = SSL.Context(SSL.SSLv23_METHOD)
    ctx.set_verify(SSL.VERIFY_PEER, verify_cb)
    ss = SSL.Connection(ctx, self.conn)

    ss.set_connect_state()
    ss.do_handshake()
    cert = ss.get_peer_certificate()
    ss.send(b'GET http://google.com:80/123 HTTP/1.1\r\nHost: google.com\r\nUser-Agent: curl/7.64.1\r\nAccept: */*\r\nProxy-Connection: Keep-Alive\r\n\r\n')
    response = ss.read(DEFAULT_BUFFER_SIZE)
    ss.shutdown()
    ss.close()

Can you suggest how to rewrite the tunnel method for HTTPS support?

Indeed the problem is new_socket_connection creates a plain connection. Check proxy/server.py for examples on SSL connection wrapping. Let me know how it goes.

[eugen1j]

Awesome. Though after #360 this hack should no longer be required. PTAL.

Thank you, works great.

Indeed the problem is new_socket_connection creates a plain connection. Check proxy/server.py for examples on SSL connection wrapping. Let me know how it goes.

I decided to solve my problem another way, without ProxyPoolPlugin. I will try this later.

[abhinavsingh]

I decided to solve my problem another way, without ProxyPoolPlugin. I will try this later.

Awesome. Note that all the plugins are merely examples and demonstration of plugin capabilities. They are in no way production ready, but I would love to get them there with time :)

I'll keep this open and revisit at some point later. Thank you!!!

[abhinavsingh]

@eugen1j Looking back at it today, I realize issue is not related to SSL connection. Once primary proxy.py is able to proxy the request to secondary proxy.py, things should work just fine. Unfortunately, within server.py on_client_data callback, plugin callbacks are no longer invoked if an upstream server connection was not established, which happens with our proxy pool plugin, as we skip direction connection to upstream and instead re-proxy the request to secondary proxies.

Related code block is here

https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/http/proxy/server.py#L208-L236

To make https based proxy pool to work, core must continue invoking plugins even if upstream server connection was not established. Currently, proxy.py processes the request but doesn't invoke plugins resulting into https proxy pool failure.

[abhinavsingh]

Marking this issue as feature request as this is less of a bug right now. Fixing this requires changes to core wiring.

[arache]

When do we want to fix this please?