Timeout for `LoadSuite` is not configurable

[nilsreichardt]

test/pkgs/test_core/lib/src/runner/load_suite.dart

Lines 23 to 29 in db8cf09

	/// The timeout for loading a test suite.
	///
	/// We want this to be long enough that even a very large application being
	/// compiled with dart2js doesn't trigger it, but short enough that it fires
	/// before the host kills it. For example, Google's Forge service has a
	/// 15-minute timeout.
	final _timeout = const Duration(minutes: 12);

The timeout for loading the test suit is fixed. I assume this the root cause for the issue that we have with running flutter test integration_test with heavy apps: flutter/flutter#105913:

12:00 +0 -1: loading /Users/runner/work/sharezone-app/sharezone-app/app/integration_test/app_test.dart [E]                                                                                             
  TimeoutException after 0:12:00.000000: Test timed out after 12 minutes.
  package:test_api/src/backend/invoker.dart 333:28  Invoker._handleError.<fn>

As the documentation of the variables says, determines this variable how long the test command tries to compile the application. Therefore, I assume that this is the issue.

People pass the --timeout flag (like --timeout none) to the flutter test command but are confused why their integration test timeout. To proper fix this issue, I think we need to make this timeout configure.

Another workaround would be to add the --ignoreTimeout flag of the Dart test command also the Flutter test command and pass it.

[bkonyi]

@natebosch it would be nice if there was a way to bypass or change this timeout so users wouldn't need to disable all timeouts to work around it.

[natebosch]

@bkonyi - how common is it to have apps that take 12 or more minutes to compile? Should we be exploring better support for pre-compiling tests separately?

[bkonyi]

@bkonyi - how common is it to have apps that take 12 or more minutes to compile? Should we be exploring better support for pre-compiling tests separately?

Unfortunately, I have no clue :(

[nilsreichardt]

@bkonyi - how common is it to have apps that take 12 or more minutes to compile? Should we be exploring better support for pre-compiling tests separately?

I would say for production apps, it happens relativity often when running on slow CI/CD machines (e.g. GitHub Actions). You can see that flutter/flutter#105913 has 45 upvotes. Sorting by 👍 it ranks on place 450 out of 97,422 closed tickets. If you add Firestore as a dependency, you are instantly at 10 - 15 minutes of build time using GitHub Actions VMs (as described in flutter/flutter#105913).

[natebosch]

After thinking some more about this, I think the 12 minute timeout is pretty focused on internal google systems. I'm leaning towards removing the default and using the argument internally to get the 12 minute behavior.