runtime: add "maymorestack" hook for testing stack growth prologues #48297
Comments
aclements
added
the
FeatureRequest
How do you plan to make this happen? Presumably we don't want to always call the hook in every function. Perhaps I misunderstand. |
|
I plan to always call the hook in every function. The hook call will only be added if the debug option is supplied, so the performance isn't really an issue. And, in fact, I've already implemented this and it has shockingly little performance impact. Unless you were thinking of some other problem? |
|
How is the debug option supplied? |
|
In my current prototype, the compiler takes it as a -d=maymorestack= argument and the assembler takes it as -maymorestack= (because it doesn't have an equivalent of -d). So, e.g., you can supply the preemption-testing hook as But I'm happy to change this. |
|
That seems fine. Thanks. I just didn't find the original description very clear. We have various different kinds of debugging hooks (e.g., |
|
Change https://golang.org/cl/359956 mentions this issue: |
|
Change https://golang.org/cl/359795 mentions this issue: |
|
Change https://golang.org/cl/359797 mentions this issue: |
|
Change https://golang.org/cl/359796 mentions this issue: |
This moves and slightly generalizes the -d debug flag parser from cmd/compile/internal/base to cmd/internal/objabi so that we can use the same debug flag syntax in other tools. This makes a few minor tweaks to implementation details. The flag itself is now just a flag.Value that gets constructed explicitly, rather than at init time, and we've cleaned up the implementation a little (e.g., using a map instead of a linear search of a slice). The help text is now automatically alphabetized. Rather than describing the values of some flags in the help text footer, we simply include it in the flags' help text and make sure multi-line help text renders sensibly. For #48297. Change-Id: Id373ee3b767e456be483fb28c110d025149be532 Reviewed-on: https://go-review.googlesource.com/c/go/+/359956 Trust: Austin Clements <[email protected]> Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: David Chase <[email protected]> Reviewed-by: Cherry Mui <[email protected]>
This adds a maymorestack hook that forces a preemption at every possible cooperative preemption point. This would have helped us catch several recent preemption-related bugs earlier, including #47302, #47304, and #47441. For #48297. Change-Id: Ib82c973589c8a7223900e1842913b8591938fb9f Reviewed-on: https://go-review.googlesource.com/c/go/+/359796 Trust: Austin Clements <[email protected]> Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Cherry Mui <[email protected]> Reviewed-by: Michael Pratt <[email protected]> Reviewed-by: David Chase <[email protected]>
This adds a maymorestack hook that moves the stack at every cooperative preemption point. For #48297. Change-Id: Ic15f9bcbc163345e6422586302d57fda4744caec Reviewed-on: https://go-review.googlesource.com/c/go/+/359797 Trust: Austin Clements <[email protected]> Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Cherry Mui <[email protected]> Reviewed-by: Michael Pratt <[email protected]> Reviewed-by: David Chase <[email protected]>
|
Change https://golang.org/cl/361212 mentions this issue: |
|
How to use use this option in go commands? |
|
@go101, see the comments added in CL 359796. Your usage is close, but you have to also specify which maymorestack hook function to use: |
|
Thanks. Got it and make it work for a custom function. Cool feature! |
|
Er, it looks I misunderstood this option. I thought the hook will be called when stacks grow. In fact, it is not. |
|
@go101, right, it's called whenever they might grow. It's really intended as a way for us to amplify the rare things that happen when that path is (fairly unpredictably) taken. Hence the hooks to "always preempt" and "always move the stack". I'd also like to hook this up with the static lock rank checker so we can mark locks that could be taken by morestack if the conditions were right, but haven't gotten to that. |
|
Change https://go.dev/cl/402157 mentions this issue: |
There are a few tests of the scheduler run queue API that allocate a local []g and test using those G's. However, the run queue API frequently converts between *g and guintptr, which is safe for "real" Gs because they're heap-allocated and hence don't move, but if these tests get a stack movement while holding one of these local *g's as a guintptr, it won't get updated and the test will fail. Updates #48297. Change-Id: Ifd424147ce1a1b53732ff0cf55a81df1a9beeb3b Reviewed-on: https://go-review.googlesource.com/c/go/+/402157 Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Cherry Mui <[email protected]>
These tests run the runtime, reflect, and sync package tests with the two maymorestack hooks we have. These tests only run on the longtest builders (or with GO_TEST_SHORT=false) because we're running the runtime test two additional times and the mayMoreStackMove hook makes it about twice as slow (~230 seconds). To run just these tests by hand, do GO_TEST_SHORT=false go tool dist test -run mayMoreStack Updates #48297. This detected #49354, which was found as a flake on the dashboard, but was reliably reproducible with these tests; and #49395. Change-Id: If785a8b8d6e1b9ad4d2ae67493b54055ab6cbc85 Reviewed-on: https://go-review.googlesource.com/c/go/+/361212 Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Austin Clements <[email protected]> Reviewed-by: Cherry Mui <[email protected]>
|
Change https://go.dev/cl/428656 mentions this issue: |
CL 424257 modified gentraceback to switch gp when jumping from a system stack to a user stack to simplify reasoning through the rest of the function. This has the unintended side-effect of also switching all references to gp.m. The vast majority of the time, g0.m and curg.m are the same across a stack switch, making this a no-op, but there's at least one case where this isn't true: if a profiling signal happens in execute between setting mp.curg and setting gp.m. In this case, mp.curg.m is briefly nil, which can cause gentraceback to crash with a nil pointer dereference. We see this failure (surprisingly frequently!) in profiling tests in the morestack=mayMoreStackPreempt testing mode (#48297). Fix this by making only jumping stacks if doing so will not switch Ms. This restores the original property that gp.m doesn't change across the stack jump, and makes gentraceback a little more conservative about jumping stacks. Fixes #54885. Change-Id: Ib1524c41c748eeff35896e0f3abf9a7efbe5969f Reviewed-on: https://go-review.googlesource.com/c/go/+/428656 Reviewed-by: Michael Pratt <[email protected]> Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Austin Clements <[email protected]>
I plan to add a debugging hook for internal use that will be invoked in the prologue of any function that may call morestack (the stack growth function). This is inspired by the preemption bugs that led to #47302, #47304, and #47441.
In general, testing around morestack is difficult because almost all Go functions can potentially call morestack, but very few do at run time, and which ones do are highly unpredictable. As a result, if a particular function is incorrectly sensitive to whether morestack gets invoked (either to grow the stack or to preempt), it's highly unlikely that general testing will actually trigger the morestack path in that function.
The hook function will be called unconditionally before the stack bound check in any function that could call morestack. This particular placement means the hook can inject preemption or stack growth by simply setting the stack bound to a poisoned value and returning, since the stack bound check will then fail and enter morestack.
This hook is also useful for more precise lock order checking because it lets us model functions that "may" acquire any of the locks acquired by morestack. These are lock edges we only see in the rare cases where morestack is actually called, but lie latent in any function that could call it.
The text was updated successfully, but these errors were encountered: