Rollup merge of #115007 - kpreid:alloc, r=Mark-Simulacrum

Correct and expand documentation of `handle_alloc_error` and `set_alloc_error_hook`.

The primary goal of this change is to remove the false claim that `handle_alloc_error` always aborts; instead, code should be prepared for `handle_alloc_error` to possibly unwind, and be sound under that condition.

I saw other opportunities for improvement, so I have added all the following information:

* `handle_alloc_error` may panic instead of aborting. (Fixes #114898)
* What happens if a hook returns rather than diverging.
* A hook may panic. (This was already demonstrated in an example, but not stated in prose.)
* A hook must be sound to call — it cannot assume that it is only called by the runtime, since its function pointer can be retrieved by safe code.

I've checked these statements against the source code of `alloc` and `std`, but there may be nuances I haven't caught, so a careful review is welcome.

Author: matthiaskrgr

library/alloc/src/alloc.rs

@@ -343,18 +343,31 @@ extern "Rust" {
     fn __rust_alloc_error_handler(size: usize, align: usize) -> !;
 }
 
-/// Abort on memory allocation error or failure.
+/// Signal a memory allocation error.
 ///
-/// Callers of memory allocation APIs wishing to abort computation
+/// Callers of memory allocation APIs wishing to cease execution
 /// in response to an allocation error are encouraged to call this function,
-/// rather than directly invoking `panic!` or similar.
+/// rather than directly invoking [`panic!`] or similar.
 ///
-/// The default behavior of this function is to print a message to standard error
-/// and abort the process.
-/// It can be replaced with [`set_alloc_error_hook`] and [`take_alloc_error_hook`].
+/// This function is guaranteed to diverge (not return normally with a value), but depending on
+/// global configuration, it may either panic (resulting in unwinding or aborting as per
+/// configuration for all panics), or abort the process (with no unwinding).
+///
+/// The default behavior is:
+///
+///  * If the binary links against `std` (typically the case), then
+///   print a message to standard error and abort the process.
+///   This behavior can be replaced with [`set_alloc_error_hook`] and [`take_alloc_error_hook`].
+///   Future versions of Rust may panic by default instead.
+///
+/// * If the binary does not link against `std` (all of its crates are marked
+///   [`#![no_std]`][no_std]), then call [`panic!`] with a message.
+///   [The panic handler] applies as to any panic.
 ///
 /// [`set_alloc_error_hook`]: ../../std/alloc/fn.set_alloc_error_hook.html
 /// [`take_alloc_error_hook`]: ../../std/alloc/fn.take_alloc_error_hook.html
+/// [The panic handler]: https://doc.rust-lang.org/reference/runtime.html#the-panic_handler-attribute
+/// [no_std]: https://doc.rust-lang.org/reference/names/preludes.html#the-no_std-attribute
 #[stable(feature = "global_alloc", since = "1.28.0")]
 #[rustc_const_unstable(feature = "const_alloc_error", issue = "92523")]
 #[cfg(all(not(no_global_oom_handling), not(test)))]

library/std/src/alloc.rs

@@ -290,15 +290,29 @@ static HOOK: AtomicPtr<()> = AtomicPtr::new(ptr::null_mut());
 
 /// Registers a custom allocation error hook, replacing any that was previously registered.
 ///
-/// The allocation error hook is invoked when an infallible memory allocation fails, before
-/// the runtime aborts. The default hook prints a message to standard error,
-/// but this behavior can be customized with the [`set_alloc_error_hook`] and
-/// [`take_alloc_error_hook`] functions.
+/// The allocation error hook is invoked when an infallible memory allocation fails — that is,
+/// as a consequence of calling [`handle_alloc_error`] — before the runtime aborts.
 ///
-/// The hook is provided with a `Layout` struct which contains information
+/// The allocation error hook is a global resource. [`take_alloc_error_hook`] may be used to
+/// retrieve a previously registered hook and wrap or discard it.
+///
+/// # What the provided `hook` function should expect
+///
+/// The hook function is provided with a [`Layout`] struct which contains information
 /// about the allocation that failed.
 ///
-/// The allocation error hook is a global resource.
+/// The hook function may choose to panic or abort; in the event that it returns normally, this
+/// will cause an immediate abort.
+///
+/// Since [`take_alloc_error_hook`] is a safe function that allows retrieving the hook, the hook
+/// function must be _sound_ to call even if no memory allocations were attempted.
+///
+/// # The default hook
+///
+/// The default hook, used if [`set_alloc_error_hook`] is never called, prints a message to
+/// standard error (and then returns, causing the runtime to abort the process).
+/// Compiler options may cause it to panic instead, and the default behavior may be changed
+/// to panicking in future versions of Rust.
 ///
 /// # Examples
 ///