Improve Sentry grouping

[marco-c]

Sentry for code-review and code-coverage is almost unusable because we have too many errors which are not grouped together.

E.g. we have one Mercurial error on diff (error='patching file XXX for each possible file, while we should have only one Mercurial error on diff that groups all of them together.

We should either find a way to make Sentry group all of those issues together, or modify the log messages to avoid including specific information about errors.

Here's where that example warning comes from: https://github.com/mozilla/libmozevent/blob/f1d8c03c418bebe45f468f84dabfb85d4d533685/libmozevent/mercurial.py#L386.

[marco-c]

I asked Armen a while ago.

Armen replied:

there's various ways to adjust grouping. You can start with this page and let me know if it doesn't help you:
https://docs.sentry.io/product/sentry-basics/guides/grouping-and-fingerprints/
I also once attended this presentation and it was helpful (there might be more videos there):
https://www.youtube.com/watch?v=3e1-trZZ3S0

My answer:

Thanks! It looks like we'll have to change the way we log messages, as these grouping methods are all partially manual (that is, we'd basically have to create a new fingerprinting rule for every message we currently have!).

At this point, maybe we'd better log generic messages and then use context or tags for the additional data (https://docs.sentry.io/platforms/python/enriching-events/context/ and https://docs.sentry.io/platforms/python/enriching-events/tags/.

[marco-c]

To achieve that, we could simply stop using the deprecated Raven client and switch to sentry_sdk everywhere, and follow https://docs.sentry.io/platforms/python/guides/logging/ to send extra data from log messages.

Or we can keep Raven, and follow https://raven.readthedocs.io/en/stable/integrations/logging.html#usage to send extra data.

Unfortunately it is not so simple given the complexity of our log module in https://github.com/mozilla/code-review/blob/master/tools/code_review_tools/log.py, we are using both structlog (to prepare the log messages) and Logbook (to send the log messages to stderr and sentry).

[marco-c]

See also #999 (which had a WIP patch).

[La0]

So the plan is:

• drop logbook from code review tools (and all 3 dependant projects: bot + events + backend)
• use official sentry_sdk instead of raven
• update fingerprint data to group by exception type (instead of raw message with unique IDs)

[marco-c]

* update fingerprint data to group by exception type (instead of raw message with unique IDs)

If we group by exception type, we have to be careful to have different exception types for all errors.
Another option could be to still group by message, but have the ability to attach extra data (https://docs.sentry.io/platforms/python/guides/logging/).

[La0]

We discussed with @EvaBardou about this issue, and we have 2 options:

1. remove logbook, keep structlog to avoid updating logging messages, and add structlog-sentry along with sentry_sdk
2. remove logbook & structlog, using internal feature of sentry extra to pass parameters.

As we only have 51 logger calls with structlog parameters, it seems option 2 is totally doable and simpler (this would streamline a lot the logging situation...)

[La0]

We would also need to update libmozevent !

[marco-c]

I like option 2, the less dependencies the better.

[La0]

I like option 2, the less dependencies the better.

Just keep in mind we'll need to also remove structlog from libmozevent

[marco-c]

Yeah that's fine, we can do the same cleanup on code-coverage