Use direct downloads of packages instead of using `pub cache`

[mkustermann]

We should directly download the packages from the normal https://pub.dartlang.org/packages/<name>/versions/<version>.tar.gz urls. Doing so will avoid running into the memcache issue. Furthermore it seems not particularly nice using the internal caching of pub for other purposes.

[kevmoo]

Hopefully mitigated by dart-lang/pub-dev#338

[isoos]

This is now enabled via #88, pub site can download the package directly and call analyzer on the directory.

[mkustermann]

Is this issue in fact fixed - i.e. does pana now request the url directly (as mentioned above)?

Somehow it's not clear from #88 . We might go back on the memcaching decision.

[isoos]

Pana does not download the URL directly. Pub's analyzer could do that though...

[mkustermann]

So the issue wasn't fixed after all :-/

[isoos]

question is: do we want to handle this in pana or on the pub site?

[mkustermann]

If there is a tool which wants to download a package tarball for a specific version then it can and should use the API to do so. Why would it not? It's in the benefit of the client and the server to request the tarball for the (package, version) tuple directly.

(this is completely independent of any memcaching, it just has the side effect of bypassing it)

[isoos]

At the point of pub get, we'll hit the pub site to fetch dependencies. The difference I see is about memcaching: direct download gets you the given version if memcache is failing for whatever reason. And as far as I can recall, we may have skipped the implementation of the direct download just around the time we had consistent memcache, it may be related. But I agree: to minimize the risk, we can download the package directly as .tar.gz

[mkustermann]

Of course generally speaking, all instances of the service should be consistent and an upload to one of them should affect subsequent requests to all other instances as well.

Though it doesn't mean that if one knows (e.g. due to polling of pub.dartlang.org for new versions) that a particular (package, version) is now available, it has to check again later: It can and probably should use the direct download instead.

It just happens to be the case that we don't test for existence (therefore also not use memcache) on the server and just redirect to the GCS tarball (so far).