Migrate documentation hosting to pub site

[kevmoo]

Line up with new analysis engine. Take over ownership, etc.

[isoos]

Interesting case: https://groups.google.com/a/dartlang.org/forum/#!topic/misc/tc_ou6znKgE
Based on the timestamps I would assume the upload completed later than dartdoc site got its notification, and that was the reason it could find and download the new version.

[kevmoo]

Yup...

[mkustermann]

@isoos How would dartdocs get a notification before the package was uploaded? The only way for dartdocs to find out about new packages atm is to poll e.g. /api/packages (or one of the other apis). Packages only show up there if they're committed in datastore. Our uploads happen within a transaction, so either they are committed or not.

So I'm not quite sure what happened, but seems worth investigating.

[kevmoo]

Could it be that the instance that was accessed had a stale memcache?

Since we have more than one instance...

[mkustermann]

There is a possibility of a memcache issue, where the upload happened on one VM (which invalidates it's memcache) and the fetch from the dartdocs happends to another VM.

Though I believe that if a client is requesting the tarball of a specific (package, version)-tuple, we will simply redirect to a google cloud storage URL (we don't first check if it exists, because GCS with just return 404 if it doesn't). So that's why it's a bit suspicious.

[isoos]

What's the expectation here?

1. Same as analyzer: re-generate docs with new dartdoc version every so often?
2. Generate docs at the time of the upload, and never bother to re-generate them again?

The current dartdocs.org site has docs generated with different versions of dartdoc, but most of it seems to be with a fairly recent version. (~ most packages I've checked was generated within a year or see, even though the package was several years old).

[mkustermann]

What's the expectation here?

Same as analyzer: re-generate docs with new dartdoc version every so often?
Generate docs at the time of the upload, and never bother to re-generate them again?

We currently re-analyze packages "every so often" because:

1. transitive dependencies change (3rd party packages, sdk, flutter sdk)
2. the version of dartanalyzer changes

At least 2) will be the same, if we have a new dartdoc, we would like to re-generate documentation for packages (it would be quite weird otherwise).

The point 1) might not be necessary for dartdoc.

@kevmoo Any comments?

[isoos]

So, re-running the dartdoc every month or two seems to be just fine.

Where shall we store the timestamp of the last run? AFAICT there is no other state we need to store, except to upload the static files to the bucket that serves them eventually.

It could be just another file in the bucket... It contains a JSON map to be extendable, with the timestamp, and we could just fetch the file, check the timestamp, and decide whether a new (if missing) or a re-run (if old) is required. wdyt?