Removing the ASAN_SHADOW_SIZE setting

[tlively]

Currently ASAN_SHADOW_SIZE defaults to 32MiB, which supports 256 MiB of real memory, but this can cause problems when users opt into larger memories (e.g. #12125). As far as I can tell, there's no reason not to remove this setting and calculate the necessary shadow memory from INITIAL_MEMORY, ALLOW_MEMORY_GROWTH and MAXIMUM_MEMORY. I'm not sure how to handle back-compat, though. Currently ASAN_SHADOW_SIZE can be set to basically any value, but the LEGACY_SETTINGS mechanism expects legacy settings to accept at most a handful of values. What is the best path forward?

1. Keep ASAN_SHADOW_SIZE but emit warnings when its value isn't optimal.
2. Make the LEGACY_SETTINGS mechanism more robust and complicated.
3. Ignore ASAN_SHADOW_SIZE and emit a warning when it is used without going through the LEGACY_SETTINGS mechanism
4. Something else?

[sbc100]

4. Simply remove the settings resulting in hard error.

How about we do (3) and then wait a release or two and then do (4). The good thing about (3) is that users will actually see a warning. With LEGACY_SETTINGS you only see warnings when combined with STRICT which is a little sad.

[tlively]

Sounds good, thanks!

[trybka]

Are there cases where using ASan would result in attempting to set aside too much memory with this method?

[tlively]

There are two calculations I could use:

max_mem = allow_growth ? init_mem : max_mem
shadow_size = max_mem / 8
init_mem += shadow_size
max_mem += shadow_size
error if init_mem or max_mem are too large (e.g. over 4GiB)

or

max_mem = allow_growth ? init_mem : max_mem
shadow_size = max_mem / 9
error if init_mem < shadow_size

The first is nice because users would never have to change their memory size build flags when turning on ASan and the second is nice because the actual memory limits are the same as the requested memory limits. I think I prefer the first because ASan builds are usually for debugging, so it's ok if they use extra memory. Does that sound reasonable?