CategoricalDtype.update_dtype fails with Categorical

[WillAyd]

ref #27318 (comment) it appears that CategoricalDtype might want to accept a Categorical as an argument but fails to handle that properly. Not sure intent here but we either want to fix to allow Categorical or alternately simplify and only allow str/CategoricalDtype input

[jorisvandenbossche]

Looking at the base implementation:

pandas/pandas/core/dtypes/base.py

Lines 232 to 269 in 5a7a8e1

	def is_dtype(cls, dtype) -> bool:
	"""Check if we match 'dtype'.
	Parameters
	----------
	dtype : object
	The object to check.
	Returns
	-------
	is_dtype : bool
	Notes
	-----
	The default implementation is True if
	1. ``cls.construct_from_string(dtype)`` is an instance
	of ``cls``.
	2. ``dtype`` is an object and is an instance of ``cls``
	3. ``dtype`` has a ``dtype`` attribute, and any of the above
	conditions is true for ``dtype.dtype``.
	"""
	dtype = getattr(dtype, "dtype", dtype)
	if isinstance(dtype, (ABCSeries, ABCIndexClass, ABCDataFrame, np.dtype)):
	# https://github.com/pandas-dev/pandas/issues/22960
	# avoid passing data to `construct_from_string`. This could
	# cause a FutureWarning from numpy about failing elementwise
	# comparison from, e.g., comparing DataFrame == 'category'.
	return False
	elif dtype is None:
	return False
	elif isinstance(dtype, cls):
	return True
	try:
	return cls.construct_from_string(dtype) is not None
	except TypeError:
	return False

there is a dtype = getattr(dtype, "dtype", dtype), so it seems rather on purpose that it accepts not only dtype objects itself, but also containers that have a dtype.

(personally I don't really like this polymorphism, I think it would be cleaner to just allow dtype objects and have the called to obj.dtype if necessary)

[jorisvandenbossche]

cc @jschendel

[jschendel]

I think only accepting CategoricalDtype is fine. The docstring states that the input should be a CategoricalDtype, with no mention of Categorical/CategoricalIndex, and the behavior is only tested for CategoricalDtype. If Categorical worked in the past it was only due to happenstance, likely due to is_dtype being a bit confusing in that it returns True for things that aren't actually dtypes (though it appears this is the intended behavior for is_dtype). Is there a tangible benefit to explicitly supporting Categorical here?

[TomAugspurger]

Anything to do here for 1.0, or OK to push?

[TomAugspurger]

FWIW, on master, this does succeed

In [31]: pd.CategoricalDtype(['a']).update_dtype(pd.Categorical(['b'], ordered=True))
Out[31]: CategoricalDtype(categories=['b'], ordered=True)