Support proxy configuration in HttpClient #5468

sgjesse · 2012-09-26T08:49:09Z

The HttpClient should support proxy configuration.

sgjesse · 2012-09-26T08:51:32Z

Marked this as blocking #5454.

sgjesse · 2012-10-02T13:00:03Z

I propose to make the proxy configuration be very much like the PAC scripts used for web browsers. Of cause using Dart code instead of JavaScript code. This could be done by having the ability to set proxy resolver function on an HttpClient, e.g.

abstract class HttpClient {

...

  void set findProxy(void finder(Uri url));

  ...
}

The function findProxy will return a string in the same format as is used by PAC scripts. We then need to make the functions available to PAC scripts (see http://findproxyforurl.com/pac-functions/) available in Dart.

The default, if findProxy is not set, will be not to use a proxy, which is the same as the following function:

HttpClient client = ...
client.findProxy = (_) => "DIRECT";

Always using the same proxy is also simple:

HttpClient client = ...
client.findProxy = (_) => "PROXY proxy.example.com:8080";

sgjesse · 2012-10-04T14:30:31Z

Set owner to @sgjesse.
Added Started label.

sgjesse · 2012-10-05T12:31:02Z

http://codereview.chromium.org/11066041/

sgjesse · 2012-10-10T10:01:03Z

https://codereview.chromium.org/11092044

sgjesse · 2012-10-12T08:39:24Z

Support for the findProxy approach described above is implemented in https://code.google.com/p/dart/source/detail?r=13348 and
https://code.google.com/p/dart/source/detail?r=13599.

What is now missing is support for tunneling using the CONNECT method when contacting the proxy. This is mostly required for HTTPS, but could also be supported for HTTP.

There is no specified way of configuring when to use a tunnel, but the simple heuristic is to use it for HTTPS connections only. If is should be supported for HTTP as well some kind of additional configuration will be required.

An additional issue is using secure transport between the HTTP client and the proxy server. For that the Chromium is using a extension of the PAC script, where HTTPS is used instead of PROXY, see http://www.chromium.org/developers/design-documents/secure-web-proxy and http://www.chromium.org/spdy/spdy-proxy.

Marking this as blocked on HTTPS support for now.

Marked this as being blocked by #3593.

sgjesse · 2012-12-12T15:02:31Z

Removed this from the M2 milestone.
Added this to the M3 milestone.
Unmarked this as being blocked by #3593.

sgjesse · 2013-02-21T15:03:14Z

Removed this from the M3 milestone.
Added this to the M4 milestone.

sgjesse · 2013-03-19T13:25:42Z

Added builtin support for using environment variables to configure the HTTP proxy in https://code.google.com/p/dart/source/detail?r=20202.

DartBot · 2013-03-19T13:31:05Z

This comment was originally written by [email protected]

When is M4 planned for because this is really a pita?

sgjesse · 2013-04-12T14:42:13Z

Made the environment configuration the default for HttpClient in https://code.google.com/p/dart/source/detail?r=21332.

sgjesse · 2013-04-12T14:48:02Z

Proposed change to proxies requiring authentication in https://codereview.chromium.org/13915007/.

sgjesse · 2013-04-15T12:44:03Z

Proxy authentication added in https://code.google.com/p/dart/source/detail?r=21417

Only CONNECT support is now missing.

Removed this from the M4 milestone.
Added this to the M5 milestone.

sgjesse · 2013-04-30T07:14:47Z

Support for the CONNECT method was added in https://code.google.com/p/dart/source/detail?r=22119.

Added Fixed label.

DartBot · 2013-06-12T10:38:24Z

This comment was originally written by [email protected]

I still get error even when https_proxy is configured and

curl -v https://pub.dartlang.org/packages/web_ui.json

works and I am using untested dart-sdk which includes r22119 change.

Here is the error:

$ pub.bat -v install
FINE: Pub 0.1.2+0.r23900
IO : Spawning cmd /c ver
IO : Finished ver. Exit code 0.stdout:Microsoft Windows [Version 6.1.7601]Nothing output on stderr.
Resolving dependencies...
SLVR: Solving dependencies:
| - browser any from hosted (browser)
IO : Get versions from https://pub.dartlang.org/api/packages/browser.
FINE: HTTP GET https://pub.dartlang.org/api/packages/browser
| Accept: application/vnd.pub.v2+json
Uncaught Error: SocketException: Failed host name lookup (OS Error: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.
, errno = 11002)
Stack Trace:
#0 _NativeSocket.lookup.<anonymous closure> (dart:io-patch/socket_patch.dart:100:9)
#1 _ThenFuture._sendValue (dart:async/future_impl.dart:316:24)
#2 _FutureImpl._setValue (dart:async/future_impl.dart:153:26)
#3 _SyncCompleter._setFutureValue (dart:async/future_impl.dart:43:21)
#4 _Completer.complete (dart:async/future_impl.dart:12:20)
#5 _SendPortImpl.call.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:106:27)
#6 _ReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:81:92)

Unhandled exception:
SocketException: Failed host name lookup (OS Error: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.
, errno = 11002)
#0 _FutureImpl._scheduleUnhandledError.<anonymous closure> (dart:async/future_impl.dart:192:9)
#1 _asyncRunCallback._asyncRunCallback (dart:async/event_loop.dart:9:15)
#2 _asyncRunCallback._asyncRunCallback (dart:async/event_loop.dart:13:7)
#3 Timer.Timer.<anonymous closure> (dart:async-patch/timer_patch.dart:9:15)
#4 _Timer._createTimerHandler._handleTimeout (timer_impl.dart:99:28)
#5 _Timer._createTimerHandler._handleTimeout (timer_impl.dart:107:7)
#6 _Timer._createTimerHandler.<anonymous closure> (timer_impl.dart:115:23)
#7 _ReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:81:92)

sgjesse · 2013-06-12T11:11:43Z

The Windows error code of 11002 suggests some issues with resolving the pub.dartlang.org name. However it seems like curl can resolve it just file.

Could you try the following:

nslookup pub.dartlang.org

What result do you get?

I get the following:

Non-authoritative answer:
Name: ghs-svc-https-c1043.ghs-ssl.googlehosted.com
Address: 72.14.249.27
Aliases: pub.dartlang.org

If this returns a personable response could you try to add it to the hosts file (%WINDIR%\System32\drivers\etc\hosts)? Adding a line like this:

72.14.249.27 pub.dartlang.org

This could be related to issue #11103.

DartBot · 2013-06-13T08:00:47Z

This comment was originally written by [email protected]

It is definitely related with resolving, nslookup does not work in my corporate environment for public names:

$ nslookup google.cz
Server: dc01.dom01.ad
Address: 171.27.1.102

*** dc01.dom01.ad can't find google.cz: Server failed

DNS resolving for public addresses works only via proxy.

Adding entry to host file helps, so my immediate problem is solved.

However, for dart's sake, I think it would be good to look into why pub does not work without local host file entry, when similar tools like curl or git works. My guess it is related to reverse DNS lookup done by some https security feature.

sgjesse · 2013-06-13T10:02:59Z

Thanks for helping with tracking this down. I have been trying to figure out where in a Dart code a DNS request is made when and https proxy is used, and have not been able to find it yet.

To simplify what we are looking at, could you try the following simpler program, without the pub.dartlang.org entry in the hosts file and with https_proxy set?

import "dart:io";

void main() {
  var client = new HttpClient();
  client.getUrl(Uri.parse('https://pub.dartlang.org/api/packages/browser'))
      .then((request) => request.close())
      .then(HttpBodyHandler.processResponse)
      .then((body) => print(body.body));
}

I assume that it will also fail.

Could you also try the following which uses HTTP instead of HTTPS with http_proxy configured?

void main() {
  var client = new HttpClient();
  client.getUrl(Uri.parse('http://pub.dartlang.org/api/packages/browser'))
      .then((request) { request.followRedirects = false; return request.close(); })
      .then(HttpBodyHandler.processResponse)
      .then((body) => print(body.body));
}

DartBot · 2013-06-14T12:37:14Z

This comment was originally written by [email protected]

Yes,
https version requires local hosts file entry, http does not.

https without local hosts file entry:

$ dart proxytest.dart
Uncaught Error: SocketIOException: Failed host name lookup (OS Error: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.
, errno = 11002)
Stack Trace:
#0 _NativeSocket.lookup.<anonymous closure> (dart:io-patch/socket_patch.dart:100:9)
#1 _ThenFuture._sendValue (dart:async/future_impl.dart:265:24)
#2 _FutureImpl._setValue (dart:async/future_impl.dart:149:26)
#3 _SyncCompleter._setFutureValue (dart:async/future_impl.dart:43:21)
#4 _Completer.complete (dart:async/future_impl.dart:12:20)
#5 _SendPortImpl.call.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:106:27)
#6 _ReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:81:92)

Unhandled exception:
SocketIOException: Failed host name lookup (OS Error: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.
, errno = 11002)
#0 _FutureImpl._scheduleUnhandledError.<anonymous closure> (dart:async/future_impl.dart:180:9)
#1 _asyncRunCallback._asyncRunCallback (dart:async/event_loop.dart:9:15)
#2 _asyncRunCallback._asyncRunCallback (dart:async/event_loop.dart:13:7)
#3 Timer.Timer.<anonymous closure> (dart:async-patch/timer_patch.dart:9:15)
#4 _Timer._createTimerHandler._handleTimeout (timer_impl.dart:99:28)
#5 _Timer._createTimerHandler._handleTimeout (timer_impl.dart:107:7)
#6 _Timer._createTimerHandler.<anonymous closure> (timer_impl.dart:115:23)
#7 _ReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:81:92)

liudonghua123 · 2020-09-17T09:11:01Z

how to make a https proxy, I have a pac script returns HTTPS usa.cn-cloudflare.com:443, but when I set it in findProxy, I got the following errors.

Unhandled exception:
HttpException: Invalid proxy configuration HTTPS usa.cn-cloudflare.com:443
#0      new _ProxyConfiguration.<anonymous closure> (dart:_http/http_impl.dart:3020:11)
#1      List.forEach (dart:core-patch/growable_array.dart:302:8)
#2      new _ProxyConfiguration (dart:_http/http_impl.dart:2976:10)
#3      _HttpClient._openUrl (dart:_http/http_impl.dart:2338:25)
#4      _HttpClient.getUrl (dart:_http/http_impl.dart:2208:48)
#5      main (file:///D:/code/dart/simple_throttle_debounce/proxy.dart:11:30)
#6      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:301:19)
#7      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

The demo code is belows.

import 'dart:io';
import 'dart:convert';

// https://codeburst.io/quick-tip-how-to-make-http-requests-in-dart-53fc407daf31


void main() async {
  HttpClient client = new HttpClient();
  client.findProxy = (Uri uri) => "HTTPS usa.cn-cloudflare.com:443";
  // produces a request object
  var request = await client.getUrl(Uri.parse('https://jsonplaceholder.typicode.com/posts'));
  // sends the request
  var response = await request.close();
  // transforms and prints the response
  await for (var contents in response.transform(Utf8Decoder())) {
    print(contents);
  }
  client.close();
}

sgjesse added Type-Enhancement area-core-library SDK core library issues (core, async, ...); use area-vm or area-web for platform specific libraries. library-io labels Jun 14, 2013

sgjesse self-assigned this Jun 14, 2013

sgjesse added this to the M5 milestone Jun 14, 2013

kevmoo added type-enhancement A request for a change that isn't a bug and removed type-enhancement labels Mar 1, 2016

This issue was closed.

Support proxy configuration in HttpClient #5468

Support proxy configuration in HttpClient #5468

Comments

sgjesse commented Sep 26, 2012

sgjesse commented Sep 26, 2012

Uh oh!

sgjesse commented Oct 2, 2012

Uh oh!

sgjesse commented Oct 4, 2012

Uh oh!

sgjesse commented Oct 5, 2012

Uh oh!

sgjesse commented Oct 10, 2012

Uh oh!

sgjesse commented Oct 12, 2012

Uh oh!

sgjesse commented Dec 12, 2012

Uh oh!

sgjesse commented Feb 21, 2013

Uh oh!

sgjesse commented Mar 19, 2013

Uh oh!

DartBot commented Mar 19, 2013

Uh oh!

sgjesse commented Apr 12, 2013

Uh oh!

sgjesse commented Apr 12, 2013

Uh oh!

sgjesse commented Apr 15, 2013

Uh oh!

sgjesse commented Apr 30, 2013

Uh oh!

DartBot commented Jun 12, 2013

Uh oh!

sgjesse commented Jun 12, 2013

Uh oh!

DartBot commented Jun 13, 2013

Uh oh!

sgjesse commented Jun 13, 2013

Uh oh!

DartBot commented Jun 14, 2013

Uh oh!

liudonghua123 commented Sep 17, 2020

Uh oh!