Feature request: store "installed from CI" in the BigTable

[Zac-HD]

Like many other maintainers, I'd love to know how many of my package installs are from CI vs. other use-cases in order to get a better estimate of how many people use Hypothesis. (as a testing tool the CI download stats are useful to decide on e.g. supported versions, but much less useful for counting users)

I was pleasantly surprised when my background reading shows that pip already does some auto-detection of CI and includes that in the user-agent (pypa/pip#6273), and that it's already handled by linehaul (#46)...

So all that's left is to actually store that data ala pypi/linehaul#46 (comment), so that we can query it later!

[di]

Just a pointer that this should probably happen at https://github.com/pypa/linehaul-cloud-function instead. This "linehaul" is currently writing to a dataset that is in the process of being deprecated. It will also require a schema migration beforehand.

[Zac-HD]

Thanks @di - should I open an issue there instead?

[di]

I went ahead and just transferred this one. Thanks!

[jfuruness]

seconded

[di]

The schema has been updated to add details.ci to simple_requests and file_downloads, I think the next step here is updating this function to write that field.

[jfuruness]

Ah gotcha. I saw the details.ci in the file_downloads field, but noticed that it was a null value. Not a priority, I get that this is just open source, just wanted to bump this. I was just digging into the data trying to figure out how many people were actually using my package (since overwhelmingly the downloads appeared to be mirrors or scrapers), and couldn't find a good way to determine this (Although I think the details.ci field would go a long way), so I was just curious. Tbh I'm surprised others aren't more interested in having this statistic.

[Zac-HD]

I poked at this in hopes of opening a PR, but ended up stuck because I don't understand the message format well enough. Some notes which I hope will be useful for someone, no promise of accuracy:

• The core of our function is here: we grab a line from the input file, parse it into one of our neat dataclasses, and then write it out as jsonlines
• So the change we need to make is:
  • Add ci = attr.ib(type=Optional[bool], default=None) to the Download (and Simple?) dataclass(es)
  • Write a bit of parsing code which will extract this from the blob format, and use it to populate the data dictionary in the parse() function
    • This is where I got stuck; I'm not sure where to look for raw blobs to see how "is in CI" is represented. The current test fixtures don't contain any CI info.
  • Add some tests to make sure it keeps working in future :-)

@di if you happen to have some raw data with a CI field handy, I'd be happy to update the test fixtures and code according to this plan 🙂

[di]

Actually, this seems to just be working as expected now that the field exists in the schema, without needing any changes here:

SELECT COUNT(*) AS num_downloads
FROM `bigquery-public-data.pypi.file_downloads`
WHERE details.ci is not NULL
  AND DATE(timestamp)
    BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
    AND CURRENT_DATE()

Row	num_downloads
1	1928588704

SELECT COUNT(*) AS simple_requests
FROM `bigquery-public-data.pypi.simple_requests`
WHERE details.ci is not NULL
  AND DATE(timestamp)
    BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
    AND CURRENT_DATE()

Row	simple_requests
1	2427426581

I made #144 anyways just to ensure we have an example of ci: true in all our test fixtures.

Given that, I think we can close this, but I'll let @Zac-HD confirm.

[jfuruness]

Ah gotcha. I'm not sure if this is a bug or a feature, but when I tried looking up whether details.ci was False it came back with no results. Also when I looked up some of my own packages that I installed from the command line, they all came back with 0 results when checking if details.ci IS TRUE. Maybe I'm just not using it right or not understanding fully this functionality?

[jfuruness]

Awesome that it's already storing this though!

[Zac-HD]

Yep, it's working! I'll give it a few months to populate, and then dig in - maybe at PyCon 🙂

Thanks again @di!