Bug when rewriting url with %2F%2F

[La0]

Some artifacts have double slashes in their url: like for code-coverage artifacts like public/test_info//code-coverage-grcov.zip

The public url works fine, anyone can download that public artifact through https://queue.taskcluster.net.

But if a Taskcluster task uses the proxy, an url with double slashes will trigger a 301 redirection code toward the url with a single quote. That final url leads to a 404, and bad things happen.

I reproduced the issue locally using the latest release of taskcluster-proxy:

Test with an url with single /

Artifact name: public/logs/live.log ➡️ public%2Flogs%2Flive.log once url encoded

curl -v http://localhost:8080/queue/v1/task/OXhVOEchQlewyyWayyTxjA/runs/0/artifacts/public%2Ftest_info%2F%2Fcode-coverage-jsvm.zip
...
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Age: 2480
< Content-Type: text/plain; charset=utf-8
< Date: Wed, 02 Oct 2019 12:47:28 GMT
< Etag: "c723fd8dbc0169de0a6f8c99c747e783"
< Last-Modified: Wed, 02 Oct 2019 12:45:17 GMT
< Server: AmazonS3
...
Log is served

Test with an url with double //

Artifact name: public/test_info//code-coverage-jsvm.zip ➡️ public%2Ftest_info%2F%2Fcode-coverage-jsvm.zip once url encoded

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /queue/v1/task/OXhVOEchQlewyyWayyTxjA/runs/0/artifacts/public%2Ftest_info%2F%2Fcode-coverage-jsvm.zip HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Type: text/html; charset=utf-8
< Location: /queue/v1/task/OXhVOEchQlewyyWayyTxjA/runs/0/artifacts/public/test_info/code-coverage-jsvm.zip
< Date: Wed, 02 Oct 2019 13:30:44 GMT
< Content-Length: 129
< 
<a href="/queue/v1/task/OXhVOEchQlewyyWayyTxjA/runs/0/artifacts/public/test_info/code-coverage-jsvm.zip">Moved Permanently</a>.

* Connection #0 to host localhost left intact

[petemoore]

Thanks for raising! I can reproduce locally.

Calling queue directly I get a 303 redirect and then a 200 response:

$ curl -LIv https://queue.taskcluster.net/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public%2Ftest_info%2F%2Fwpt_raw.log
*   Trying 54.225.187.115...
* TCP_NODELAY set
* Connected to queue.taskcluster.net (54.225.187.115) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Mozilla Corporation; CN=auth.taskcluster.net
*  start date: Jul 30 00:00:00 2019 GMT
*  expire date: Jul 22 12:00:00 2020 GMT
*  subjectAltName: host "queue.taskcluster.net" matched cert's "queue.taskcluster.net"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> HEAD /v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public%2Ftest_info%2F%2Fwpt_raw.log HTTP/1.1
> Host: queue.taskcluster.net
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 303 See Other
HTTP/1.1 303 See Other
< Server: Cowboy
Server: Cowboy
< Connection: keep-alive
Connection: keep-alive
< Strict-Transport-Security: max-age=7776000000; includeSubDomains
Strict-Transport-Security: max-age=7776000000; includeSubDomains
< Content-Security-Policy: report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';
Content-Security-Policy: report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-For-Request-Id: bf3bf037-8b31-4150-a62d-3a3f97fd1f91
X-For-Request-Id: bf3bf037-8b31-4150-a62d-3a3f97fd1f91
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 900
Access-Control-Max-Age: 900
< Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
< Access-Control-Request-Method: *
Access-Control-Request-Method: *
< Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control
Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control
< Cache-Control: no-store no-cache must-revalidate
Cache-Control: no-store no-cache must-revalidate
< X-Taskcluster-Artifact-Storage-Type: s3
X-Taskcluster-Artifact-Storage-Type: s3
< Location: https://taskcluster-artifacts.net/C_QS_dYFRBWF9ORQGJJqsw/0/public/test_info//wpt_raw.log
Location: https://taskcluster-artifacts.net/C_QS_dYFRBWF9ORQGJJqsw/0/public/test_info//wpt_raw.log
< Vary: Accept
Vary: Accept
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Content-Length: 114
Content-Length: 114
< Date: Wed, 02 Oct 2019 13:43:13 GMT
Date: Wed, 02 Oct 2019 13:43:13 GMT
< Via: 1.1 vegur
Via: 1.1 vegur

< 
* Connection #0 to host queue.taskcluster.net left intact
* Issue another request to this URL: 'https://taskcluster-artifacts.net/C_QS_dYFRBWF9ORQGJJqsw/0/public/test_info//wpt_raw.log'
* Disables POST, goes with HEAD
*   Trying 13.225.78.191...
* TCP_NODELAY set
* Connected to taskcluster-artifacts.net (13.225.78.191) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=taskcluster-artifacts.net
*  start date: Dec 15 00:00:00 2018 GMT
*  expire date: Jan 15 12:00:00 2020 GMT
*  subjectAltName: host "taskcluster-artifacts.net" matched cert's "taskcluster-artifacts.net"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f9179806600)
> HEAD /C_QS_dYFRBWF9ORQGJJqsw/0/public/test_info//wpt_raw.log HTTP/2
> Host: taskcluster-artifacts.net
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
HTTP/2 200 
< content-type: text/plain
content-type: text/plain
< content-length: 81834282
content-length: 81834282
< date: Wed, 02 Oct 2019 13:39:38 GMT
date: Wed, 02 Oct 2019 13:39:38 GMT
< last-modified: Tue, 01 Oct 2019 06:37:39 GMT
last-modified: Tue, 01 Oct 2019 06:37:39 GMT
< etag: "d755e8c6fef4960f5c49a29073ec4893"
etag: "d755e8c6fef4960f5c49a29073ec4893"
< x-amz-version-id: 7YHKPDFd0oA85N6TkgUxVGPuoyZxd0HW
x-amz-version-id: 7YHKPDFd0oA85N6TkgUxVGPuoyZxd0HW
< accept-ranges: bytes
accept-ranges: bytes
< server: AmazonS3
server: AmazonS3
< x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
< via: 1.1 71dbd5706c5b0c7b733248e1171f2d4f.cloudfront.net (CloudFront)
via: 1.1 71dbd5706c5b0c7b733248e1171f2d4f.cloudfront.net (CloudFront)
< x-amz-cf-pop: FRA2-C2
x-amz-cf-pop: FRA2-C2
< x-amz-cf-id: _6xDmIaCbQH7IzujlV9suqbqrnDMqK90z11JLiOPUeiQZqP2x8ETLQ==
x-amz-cf-id: _6xDmIaCbQH7IzujlV9suqbqrnDMqK90z11JLiOPUeiQZqP2x8ETLQ==
< age: 216
age: 216

< 
* Connection #1 to host taskcluster-artifacts.net left intact

Then using taskcluster proxy (locally) I get a 301 (not a 303) and then a 404:

pmoore@Petes-iMac:~ $ curl -LIv http://localhost:8080/queue/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public%2Ftest_info%2F%2Fwpt_raw.log
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> HEAD /queue/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public%2Ftest_info%2F%2Fwpt_raw.log HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Location: /queue/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public/test_info/wpt_raw.log
Location: /queue/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public/test_info/wpt_raw.log
< Date: Wed, 02 Oct 2019 13:43:17 GMT
Date: Wed, 02 Oct 2019 13:43:17 GMT

< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8080/queue/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public/test_info/wpt_raw.log'
* Found bundle for host localhost: 0x7fe285c1d970 [can pipeline]
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 8080 (#0)
> HEAD /queue/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public/test_info/wpt_raw.log HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control
Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control
< Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 900
Access-Control-Max-Age: 900
< Access-Control-Request-Method: *
Access-Control-Request-Method: *
< Cache-Control: no-store no-cache must-revalidate
Cache-Control: no-store no-cache must-revalidate
< Connection: keep-alive
Connection: keep-alive
< Content-Length: 478
Content-Length: 478
< Content-Security-Policy: report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';
Content-Security-Policy: report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Date: Wed, 02 Oct 2019 13:43:17 GMT
Date: Wed, 02 Oct 2019 13:43:17 GMT
< Etag: W/"1de-c+Am28XKJvC0FWM/jkOTqAsXkUs"
Etag: W/"1de-c+Am28XKJvC0FWM/jkOTqAsXkUs"
< Server: Cowboy
Server: Cowboy
< Strict-Transport-Security: max-age=7776000000; includeSubDomains
Strict-Transport-Security: max-age=7776000000; includeSubDomains
< Via: 1.1 vegur
Via: 1.1 vegur
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-For-Request-Id: fcc586a9-22df-4b16-b113-a69273597863
X-For-Request-Id: fcc586a9-22df-4b16-b113-a69273597863
< X-Taskcluster-Endpoint: https://queue.taskcluster.net/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public/test_info/wpt_raw.log
X-Taskcluster-Endpoint: https://queue.taskcluster.net/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public/test_info/wpt_raw.log
< X-Taskcluster-Proxy-Perm-Clientid: mozilla-ldap/pmoore@mozilla.com/dev
X-Taskcluster-Proxy-Perm-Clientid: mozilla-ldap/pmoore@mozilla.com/dev
< X-Taskcluster-Proxy-Revision: 
X-Taskcluster-Proxy-Revision: 
< X-Taskcluster-Proxy-Version: 5.0.1
X-Taskcluster-Proxy-Version: 5.0.1

< 
* Connection #0 to host localhost left intact
pmoore@Petes-iMac:~ $ 

[petemoore]

In the taskcluster proxy logs I see:

2019/10/02 15:50:35 Proxying /queue/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public/test_info/wpt_raw.log | HEAD | https://queue.taskcluster.net/v1/task/C_QS_dYFRBWF9ORQGJJqsw/runs/0/artifacts/public/test_info/wpt_raw.log

This is incorrect, clearly some "path cleanup" has taken place, which shouldn't happen. The source path is unescaped (wrong) and double slashes are stripped (wrong) and this causes an incorrect destination path.