E2E Test Setup

[smeubank]

Idea

There is a gap in our tests, that we do not do a proper E2E test of our consumers using new releases of our SDK. E2E tests should catch those errors that we're unlikely to notice during normal development.

Requirements:

Examples of what an E2E test of an individual package should do:

• Upload current version of SDK to a local fake registry (this is to test tarball contents) (we could use verdaccio as registry)
• Download SDK package from the registry
• Build an example app using the downloaded SDK (this is to test typing and eventual build-time stuff we do, ie. webpack plugin)
• Run the example app
• Intercept any outgoing requests to sentry from that example app
• Do some stuff in the example app (cause errors, trigger transactions)
• Check payloads of intercepted requests and their content

Consider:

• This will have maintenance costs, changes in SDKs need to be reflected in our example apps

Prior Work:

• React Native: https://github.com/getsentry/sentry-react-native/blob/main/.github/workflows/e2e.yml
• Electron: https://github.com/getsentry/sentry-electron/blob/6ed4441d67e9002e3dcb7d5c2db263fa455d48a5/.github/workflows/build.yml#L71-L95

---

Things to discuss and figure out

• Do we intend these tests to be run locally? It would make the whole thing much less complicated running this only in CI because we don't have to worry about cleanup. The downside is that writing tests and adjusting the E2E test becomes a little bit more cumbersome.
  • Yes, it's a requirement to be run locally. It is definitely worth the additional effort.
• When do we run E2E tests? As part of CI or with a cron-job?
  • Let's start off running the E2E as part of every PR. If we notice that they slow us down, we demote the checks to only run on master pushes or even only before doing a release.
  • Cron jobs will only become relevant when we test canaries for major frameworks.
• Will E2E tests be a required check?
  • Discussed: Most likely yes. However, might turn into "no" depending on how fast the tests run.
• Will we run E2E tests on different kinds of OSes? (Checking filepaths, etc.)
  • Yes, we should.
• For step 2 (validating SDK behavior), would it make sense to actually send events to prod Sentry and query for the event IDs to verify the payloads?
  • Flakiness?
  • What if Sentry goes down? Bigger things to worry about.
  • What if Sentry simply has an implementation error?
  • What if the SDK has an error and doesn't return event IDs?
  • Upside: Constant stream of data flowing into Sentry for us to check out and verify things.
  • Decision: Let's send the events to prod Sentry. If we notice it's too flakey, we reevaluate.

Tasks

• Set up fake registry infrastructure (test(e2e): Set up fake registry for E2E tests #5806)
• Set up recipe-based test suites framework (test(e2e): Add E2E Test recipe framework #5836)
• Add first tests (for @sentry/react) that test the typical build-pipeline steps (test(e2e): Test building a create-react-app app with Sentry SDK #5848)
  • What to test?: Installing package, building the app (create-react-app) check typings (run tsc)
• Create guide in README on how to set up new tests

Next up

Verifying SDK behavior with E2E tests. Tracked in #5855

[smeubank]

I see these for electron all the time getsentry/sentry-electron#562

Could we achieve something similar her?

[AbhiPrasad]

I see these for electron all the time getsentry/sentry-electron#562

Could we achieve something similar her?

This would be nice to have when we look at Test multiple versions of major frameworks, which we set up as a stretch goal for now.

[smeubank]

Out of scope of this issue:

Discussion: this would be functionally different, new releases of react would be something like a canary release issue to test not block a release of our SDK

[lforst]

For step 2 (validating SDK behavior), would it make sense to actually send events to prod Sentry and query for the event IDs to verify the payloads?

From an internal discussion with Bruno and Manoel we’ve determined that going forward it makes the most sense for our E2E tests to send events to prod Sentry - but to keep the tests very simple so they stay maintainable.

Some things I’ve learned from the discussion:

• Running against prod Sentry requires us to have an auth key as secret in CI (would be our first I believe) - at first the auth token only needs the read scope but if we want to test source maps we need the write scope
• Running against prod Sentry allows us to verify that ingest doesn’t throw away our events if we somehow messed them up
• Running against prod Sentry allows us to check for post-processing behavior of the platform (eg. if source maps were able to be resolved)
• Running against prod Sentry is not really flakey but just slow sometimes - we can switch to running the E2E tests only on master pushes if it is too slow
• Unity is using a snapshot approach to verify package contents (interesting)
• Because swizzling blows up at runtime the cocoa SDK is downloading an open source app, installs sentry there via git patch, and then runs the open source app’s test suite (also interesting)
• Manoel and Bruno recommended to put the majority of work into a fake server instead of running against prod, but since we are already doing that for our integration-tests I opted for going against prod
• If the protocol evolves and our events fail, we will notice when going against prod