Allow None in RHS in df.loc.__setitem__

[Dr-Irv]

Describe the bug
Should we allow None in RHS of a __setitem__() in a DataFrame ?

On the one hand, people shouldn't be placing None as a value in a DataFrame.
On the other hand, it is accepted by pandas.

Would like opinion of @bashtage and @twoertwein on my question above.

To Reproduce

df = pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4], 5: [6, 7]})
df.loc[df["col1"] > 1, "col1"] = None

pyright reports (after adding this line to test_frame.py:

  c:\Code\pandas-stubs\tests\test_frame.py:174:5 - error: Argument of type "tuple[Series[bool], Literal['col1']]" cannot be assigned to parameter "idx" of type "tuple[_IndexSliceTuple[Unknown], HashableT@__setitem__]" in function "__setitem__"
    "tuple[Series[bool], Literal['col1']]" is incompatible with "tuple[_IndexSliceTuple[Unknown], HashableT@__setitem__]"
      Tuple entry 1 is incorrect type
        "Series[bool]" is incompatible with "_IndexSliceTuple[Unknown]" (reportGeneralTypeIssues)
  c:\Code\pandas-stubs\tests\test_frame.py:174:5 - error: Argument of type "None" cannot be assigned to parameter "value" of type "ArrayLike | S1@__setitem__ | Series[S1@__setitem__] | list[Unknown]" in function "__setitem__"
    Type "None" cannot be assigned to type "ArrayLike | S1@__setitem__ | Series[S1@__setitem__] | list[Unknown]"
      Type "None" is incompatible with constrained type variable "S1"
      Type "None" cannot be assigned to type "ExtensionArray"
      Type "None" cannot be assigned to type "ndarray[Unknown, Unknown]"
      Type "None" cannot be assigned to type "Series[S1@__setitem__]"
      Type "None" cannot be assigned to type "list[Unknown]" (reportGeneralTypeIssues)

Fix is to add None to value argument here:

pandas-stubs/pandas-stubs/core/frame.pyi

Line 176 in 5cfc849

value: S1 | ArrayLike | Series | DataFrame,

and in other places with __setitem__() declarations in core/frame.pyi and core/series.pyi

Please complete the following information:

• OS: Windows 10
• python version: 3.10
• version of type checker: pyright 1.1.273
• version of installed pandas-stubs 1.5.0.221010

[twoertwein]

Seems like pandas accepts potentially anything:

In [18]: df = pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4], 5: [6, 7]})

In [19]: df.loc[df["col1"] > 1, "col1"] = object()

In [20]: df
Out[20]: 
                                col1  col2  5
0                                  1     3  6
1  <object object at 0x7f656ad3c6a0>     4  7

[Dr-Irv]

Yes, you can put any object in a Series . We could allow it for __setitem__(), but for __getitem__(), I think we should return specific types, as we do now, when we know the type of the Series.

Are we OK with the asymmetry there?

[twoertwein]

Are we OK with the asymmetry there?

I would be nice to then change the type of Series, but we just have a None return value. I would be fine with the asymmetry, but I can also see a value in being strict so that people are warned when pandas would silently upgrade an int series to a float series. I can imagine pandas often upgrading the types without users being explicitly aware of it - being strict would make these cases more explicit.

[Dr-Irv]

I would be nice to then change the type of Series, but we just have a None return value. I would be fine with the asymmetry, but I can also see a value in being strict so that people are warned when pandas would silently upgrade an int series to a float series. I can imagine pandas often upgrading the types without users being explicitly aware of it - being strict would make these cases more explicit.

In the original example, we are changing part of a DataFrame, so the type of a Series is not at play.

I think there's two possible outcomes here:

1. In the stubs, we only allow Scalar values to be used in __setitem__() calls, because while arbitrary objects are supported by pandas, it is not great usage of pandas. Then users have to just do # type: ignore when they are OK with putting an object (or None) in as a value.
2. We change __setitem__() to allow Any to be copied in as the value argument.

@twoertwein do you have a preference of those two options? (or any other idea). Would also like the opinion of @bashtage

[twoertwein]

For DataFrame, both options sound good to me.

In case of Series, I would slightly prefer to be more strict as we cannot express a potential type promotions (option 1 with restricting the type to the generic type of the series). I wouldn't object to using Any/object (would then be consistent with the DataFrame).

[bashtage]

I think it is common enough to allow it, even though pd.NA or np.nan are usually better choices.

[Dr-Irv]

I think it is common enough to allow it, even though pd.NA or np.nan are usually better choices.

@bashtage Given the discussion above, I think that means there are 3 options for __setitem__()

1. Allow Scalar
2. Allow Scalar | None
3. Allow Any

Do you have a preference?

[gandhis1]

I'd vote for Scalar | None and evaluate any other possible types on a case-by-case basis. Assigning an arbitrary object is rare enough that it makes sense to disallow for now, but perhaps there may be another data type in the future that is omitted here that resembles a scalar.

[bashtage]

I think Scalar or None is reasonable. Don't forget that for any more exotic use, users can do # type: ignore which lets us handle the more common and helpful cases better.