Missed niche optimization

[Jules-Bertholet]

I tried this code:

#[repr(u8)]
enum Foo {
    A = 0,
}

#[repr(u8)]
enum Bar {
    A = 1,
}

enum Choice {
    F(Foo),
    B(Bar),
}

fn main() {
    dbg!(core::mem::size_of::<Choice>());
}

I expected to see this happen: Choice has size 1

Instead, this happened: Choice has size 2

Meta

Rust version; 1.74.1

@rustbot label A-layout T-compiler

[programmerjake]

this optimization should apply when variants have data too:

// could be size 2 since A and B's discriminants are in the same spot with non-overlapping contiguous values, but currently isn't
enum Top {
    A(A),
    B(B),
}

#[repr(u8)]
enum A {
    A0(u8) = 0,
    A1(u8) = 1,
}

#[repr(u8)]
enum B {
    B2(u8) = 2,
    B3(u8) = 3,
}

[aochagavia]

I was debugging this today together with @holodorum and observed an unexpected layout difference for the simplest possible inhabited enum: enum Univariant { Value }. When switching from#[repr(Rust)] to #[repr(u8)], the variants field now has a value of Variants::Multiple, even though there's only a single variant.

Is this intended? I'm not that familiar with the compiler code, but at least based on the name (i.e. Multiple) it seems wrong to me. I think this could be causing calculate_filling_niche_layout to return early here (there is a check ensuring that all enum variants have a layout of Variants::Single). Could anyone chime in to comment on this hypothesis?

Full output of #[rustc_layout(debug)]

#[repr(Rust)]

Layout {
    size: Size(0 bytes),
    align: AbiAndPrefAlign {
        abi: Align(1 bytes),
        pref: Align(8 bytes),
    },
    backend_repr: Memory {
        sized: true,
    },
    fields: Arbitrary {
        offsets: [],
        memory_index: [],
    },
    largest_niche: None,
    uninhabited: false,
    variants: Single {
        index: 0,
    },
    max_repr_align: None,
    unadjusted_abi_align: Align(1 bytes),
    randomization_seed: 11354830321609605625,
}

#[repr(u8)]:

Layout {
    size: Size(1 bytes),
    align: AbiAndPrefAlign {
        abi: Align(1 bytes),
        pref: Align(8 bytes),
    },
    backend_repr: Scalar(
        Initialized {
            value: Int(
                I8,
                false,
            ),
            valid_range: 0..=0,
        },
    ),
    fields: Arbitrary {
        offsets: [
            Size(0 bytes),
        ],
        memory_index: [
            0,
        ],
    },
    largest_niche: Some(
        Niche {
            offset: Size(0 bytes),
            value: Int(
                I8,
                false,
            ),
            valid_range: 0..=0,
        },
    ),
    uninhabited: false,
    variants: Multiple {
        tag: Initialized {
            value: Int(
                I8,
                false,
            ),
            valid_range: 0..=0,
        },
        tag_encoding: Direct,
        tag_field: 0,
        variants: [
            Layout {
                size: Size(1 bytes),
                align: AbiAndPrefAlign {
                    abi: Align(1 bytes),
                    pref: Align(8 bytes),
                },
                backend_repr: Memory {
                    sized: true,
                },
                fields: Arbitrary {
                    offsets: [],
                    memory_index: [],
                },
                largest_niche: None,
                uninhabited: false,
                variants: Single {
                    index: 0,
                },
                max_repr_align: None,
                unadjusted_abi_align: Align(1 bytes),
                randomization_seed: 11354830321609605625,
            },
        ],
    },
    max_repr_align: None,
    unadjusted_abi_align: Align(1 bytes),
    randomization_seed: 4262916569509659634,
}

[the8472]

I think Variants::Multiple is used because repr(u8) must encode its tag even if there's only a single variant because its layout is guaranteed. Variants::Single just doesn't have tag encoding.
Perhaps the naming is suboptimal in this case.

The normal niche-filling logic can offset the variant index→tag encoding around to accommodate whatever niche is available. To handle the values of repr(u8) structs which are rigid will require some new logic to check that they're non-overlapping.

Probably the most tricky part would be the tag↔discriminant codegen which would need a switch or lookup table for Choice because its discriminant values don't necessarily match the fixed tags of the inner enums.