Unexpected union evalution when union members' fields have defaults and the union itself is optional

[psychedelicious]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

There seems to be inconsistency in union evaluation, depending on how optionality is annotated. The issue can be seen when the union members have no required fields and extra="ignore" (the default) is used.

import itertools
from typing import Optional, Union

from pydantic import BaseModel, Field, create_model, ConfigDict


class Foo(BaseModel):
    foo: str = Field(default="foo")


class Bar(BaseModel):
    bar: str = Field(default="bar")


class MyModel_Optional_Foo_Bar(BaseModel):
    val: Optional[Foo | Bar] = Field()


class MyModel_Optional_Bar_Foo(BaseModel):
    val: Optional[Bar | Foo] = Field()


print(repr(MyModel_Optional_Foo_Bar.model_validate({"val": {"bar": "bar"}})))
print(repr(MyModel_Optional_Bar_Foo.model_validate({"val": {"bar": "bar"}})))

# Create models with `val` annotated as a union of Foo, Bar and None, in all permutations of order
union_members = [Foo, Bar, None]
orders = itertools.permutations(union_members)

for order in orders:
    order_string = "_".join([obj.__name__ if obj is not None else "None" for obj in order])
    M = create_model(f"MyModel_UnionOrder_{order_string}", val=(Union[tuple(order)], ...))
    print(repr(M.model_validate({"val": {"bar": "bar"}})))

# Output:
# MyModel_Optional_Foo_Bar(val=Foo(foo='foo'))
# MyModel_Optional_Bar_Foo(val=Foo(foo='foo'))
# MyModel_UnionOrder_Foo_Bar_None(val=Foo(foo='foo'))
# MyModel_UnionOrder_Foo_None_Bar(val=Foo(foo='foo'))
# MyModel_UnionOrder_Bar_Foo_None(val=Bar(bar='bar'))
# MyModel_UnionOrder_Bar_None_Foo(val=Bar(bar='bar'))
# MyModel_UnionOrder_None_Foo_Bar(val=Foo(foo='foo'))
# MyModel_UnionOrder_None_Bar_Foo(val=Bar(bar='bar'))

Disregarding other quirks in union evaluation - and which of the results are "correct" - there are some in consistencies:

• I expect MyModel_Optional_Foo_Bar and MyModel_Optional_Bar_Foo to give different results in both "smart" and "left_to_right" union modes.

  MyModel_Optional_Foo_Bar should result in a Foo, while MyModel_Optional_Bar_Foo should result in a Bar, but both result in a Foo.

• I expect MyModel_Optional_Bar_Foo to give the same result as the other models that have Bar ahead of Foo in the union's annotation.

  In other words, MyModel_Optional_Bar_Foo should match MyModel_UnionOrder_Bar_Foo_None, MyModel_UnionOrder_Bar_None_Foo, and MyModel_UnionOrder_None_Bar_Foo. The latter 3 result in a Bar, but the former results in a Foo.

It appears the use of Optional[...] introduces the inconsistent results.

Edit: I had initially confounded this optionality annotation issue with another issue related to extra. I split the extra issue out into feature request #9095 and revised this bug report to only relate to the optionality issue.

[sydney-runkle]

@psychedelicious,

Thanks for reporting. Definitely looks like a bug. Adding this to the union sprint to be addressed with 2.8.