Detect heterogeneous generic recursion usage pre-monomorphization

[scottmcm]

(Request inspired by #90063 (comment), after I wrote compiles-but-never-works code in https://rust-lang.github.io/rfcs/3058-try-trait-v2.html#the-opscontrolflow-type )

Given the following (minimal example) code: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=584d7b447db0f7890d2f94c3b945640e

pub fn foo<T>(x: T) {
    if rand() {
        foo(&x);
    }
}

The current output is: no errors nor warnings

However, if it's ever called, it'll give an error like

error: reached the recursion limit while instantiating `foo::<&&&&&&&&&&&&&&&&&&&&&&&&&&...&&&&&&&&&&&&&&&&&&&&&&&&&&&&i32>`

It would be nice for the compiler to detect, pre-monomorphization, that this will never actually work because any possible monomorphization will end up requiring instantiating another additional monomorphization.

(Not catching all cases would be fine -- I suspect indirect recursion would be particularly annoying to detect well. Maybe a deny-by-default lint for this? It'd technically be a breaking change to make it a hard error, and I don't think it really needs to be hard error. The lint is enough.)

[yanok]

@rustbot claim

[yanok]

Thanks for filing the bug! I think I have an idea how to handle this. Just need to get more familiar with rustc internal representations to evaluate if it's feasible.