BUG: pd.DataFrame.from_dict with a numpy masked array of dtype=object produces NaN instead of None.

[nsthorat]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> np_a = np.array(['a', 'b'])
>>> np_masked = np.ma.masked_array(np_a, mask=[0, 1])
>>> np_masked.tolist()
['a', None]
>>> df = pd.DataFrame.from_dict(np_masked)
>>> df.dtypes
0    object
dtype: object
>>> df
     0
0    a
1  NaN

Issue Description

The "None" from a numpy masked array in a dtype=object is being converted to a NaN when using pd.DataFrame.from_dict.

This is causing weird behavior in duckdb: duckdb/duckdb#4066

Expected Behavior

Ideally, this "None" from numpy should be a "None" in the pandas dataframe, not a NaN since this is an object (string) dtype column.

Installed Versions

INSTALLED VERSIONS

commit : 2e218d1
python : 3.9.13.final.0
python-bits : 64
OS : Darwin
OS-release : 22.3.0
Version : Darwin Kernel Version 22.3.0: Thu Jan 5 20:49:43 PST 2023; root:xnu-8792.81.2~2/RELEASE_ARM64_T8103
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.5.3
numpy : 1.22.4
pytz : 2022.7
dateutil : 2.8.2
setuptools : 65.6.3
pip : 22.3.1
Cython : None
pytest : 7.2.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader: None
bs4 : 4.11.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2023.1.0
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 9.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : 2.0.0rc1
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : 0.19.0
tzdata : None

[yeshsurya]

The default value for masked variable in numpy is None , but in pandas it is np.nan . Chaning defailt value in pandas from np.nan to None might need lot of changes.

[nsthorat]

Thanks @yeshsurya, and totally understood.

Without understanding pandas impl details, two possible suggestions:

1. For masked object dtypes, use None, use np.nan everywhere else.
2. If that's still prohibitively complicated, maybe an explicit flag for the masked variable so we can pass None.

Thoughts?

[yeshsurya]

take

[dsmilkov]

Hi! Checking if this fix got merged back to the main repo?

[yeshsurya]

Not yet ; Some with merge privileages need to review and merge.