Description
Code Sample, a copy-pastable example if possible
series_list = [pd.Series({"a": 1}), pd.Series({"b": 2}), pd.Series({"c": 3})]
print(pd.concat(series_list, verify_integrity=True)) # This works fine
# Raises TypeError
print(pd.concat(series_list, keys=["red", "red", "red"]))
# Does NOT raise an exception without verify_integirty.
# However, the object `pd.concat` returns is broken (see below for details).
print(pd.concat(series_list, keys=["red", "blue", "red"]))
# Raises ValueError
print(pd.concat(
series_list, keys=["red", "blue", "red"],
verify_integrity=True)) Expected Output
Since the indices of the Series don't conflict, it should be fine to use the same key for several rows (after all, it works fine without any keys at all). None of these calls should raise an exception and the objects they return should not be broken (see below for what I mean). Expected output:
a 1
b 2
c 3
dtype: int64
red a 1
red b 2
red c 3
dtype: int64
red a 1
blue b 2
red c 3
dtype: int64
Problem description
This is similar to issue #20565 (but even worse, IMO). I think the problem is that pd.concat creates a Series with duplicate entries in its index. This is illustrated by the broken object one of the above calls creates:
> broken = pd.concat(series_list, keys=["red", "blue", "red"])
> broken.index # Below, `levels[0]` should contain one u'red', not two.
MultiIndex(levels=[[u'red', u'blue', u'red'], [u'a', u'b', u'c']],
labels=[[1, 0, 1, 1, 1, 0, 1], [0, 1, 2]])Lots of operations fail on this object. For instance, broken.sort_index() raises "ValueError: operands could not be broadcast together with shapes (7,) (3,) (7,)".
Output of pd.show_versions()
INSTALLED VERSIONS
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.3.5-smp-811.22.0.0
machine: x86_64
processor:
byteorder: little
LC_ALL: en_US.UTF-8
LANG: None
LOCALE: None.None
pandas: 0.22.0
pytest: None
pip: None
setuptools: None
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 2.0.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: 3.1.1
numexpr: 2.5
feather: None
matplotlib: 1.5.2
openpyxl: None
xlrd: 0.9.3
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None