[wip] async cancellation test

[bobrenjc93]

Stack from ghstack (oldest at bottom):

• -> [wip] async cancellation test #157652
• [pc] verify max autotune is in generated source code #157650
• [pc] introduce ProgressiveCompilationState and clear callback #157619
• [pc] migrate progression futures from list to deque #157614

Passes but kinda suspicious:

(/home/bobren/local/a/pytorch-env) [22:38] devgpu035:/home/bobren/local/a/pytorch tlp python test/inductor/test_compile_subprocess.py -k GPUTests.test_async_cancellation 2>&1 | tee ~/data.log
/data/users/bobren/a/pytorch/torch/_inductor/ops_handler.py:741: UserWarning: undefined OpHandler.__getstate__, please add missing op schema
  warnings.warn(f"undefined OpHandler.{name}, please add missing op schema")
.
----------------------------------------------------------------------
Ran 1 test in 11.967s

OK
boop
stats [('calls_captured', 500), ('unique_graphs', 1)]
inductor []
aot_autograd [('total', 1), ('ok', 1)]
graph_break []
W0704 22:39:31.787000 2926981 torch/_inductor/compile_worker/subproc_pool.py:246] Ignored OSError in pool shutdown:  Test didn't exit fast enough, is async compile stalling our exit?
Warning: dedicated_log_torch_trace_4kznlcw_.log is likely an extraneous log; ignoring.
Stats { ok: 78, other_rank: 0, fail_glog: 0, fail_json: 0, fail_payload_md5: 0, fail_dynamo_guards_json: 0, fail_parser: 0, unknown: 0 }
Link to logs: 
https://manifold.edge.x2p.facebook.net/v0/read/tree/logs/bobren/faaac01a-b841-4499-a43b-82474cd02949/custom/index.html?bucketName=tlparse_reports&apiKey=tlparse_reports-key&withPayload=1&timeoutMsec=10000

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157652

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit 752e083 with merge base c808af5 ():

NEW FAILURES - The following jobs have failed:

• Check Labels / Check labels (gh)
  RuntimeError: Error checking labels: PR does not have required labels
• Lint / lintrunner-noclang / linux-job (gh)
  >>> Lint for torch/_inductor/compile_worker/subproc_pool.py:

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

• pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu, unstable) (gh) (#153987)
  MISSING REGRESSION TEST

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.