pd.concat fails if given repeated keys

[ilyagr]

Code Sample, a copy-pastable example if possible

series_list = [pd.Series({"a": 1}), pd.Series({"b": 2}), pd.Series({"c": 3})]
print(pd.concat(series_list, verify_integrity=True))  # This works fine

# Raises TypeError 
print(pd.concat(series_list, keys=["red", "red", "red"]))

# Does NOT raise an exception without verify_integirty.
# However, the object `pd.concat` returns is broken (see below for details).
print(pd.concat(series_list, keys=["red", "blue", "red"]))

# Raises ValueError
print(pd.concat(
    series_list, keys=["red", "blue", "red"],
    verify_integrity=True)) 

Expected Output

Since the indices of the Series don't conflict, it should be fine to use the same key for several rows (after all, it works fine without any keys at all). None of these calls should raise an exception and the objects they return should not be broken (see below for what I mean). Expected output:

a    1
b    2
c    3
dtype: int64
red  a    1
red   b    2
red  c    3
dtype: int64
red  a    1
blue   b    2
red  c    3
dtype: int64

Problem description

This is similar to issue #20565 (but even worse, IMO). I think the problem is that pd.concat creates a Series with duplicate entries in its index. This is illustrated by the broken object one of the above calls creates:

> broken = pd.concat(series_list, keys=["red", "blue", "red"])
> broken.index  # Below, `levels[0]` should contain one u'red', not two.
MultiIndex(levels=[[u'red', u'blue', u'red'], [u'a', u'b', u'c']],
           labels=[[1, 0, 1, 1, 1, 0, 1], [0, 1, 2]])

Lots of operations fail on this object. For instance, broken.sort_index() raises "ValueError: operands could not be broadcast together with shapes (7,) (3,) (7,)".

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.3.5-smp-811.22.0.0
machine: x86_64
processor:
byteorder: little
LC_ALL: en_US.UTF-8
LANG: None
LOCALE: None.None

pandas: 0.22.0
pytest: None
pip: None
setuptools: None
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 2.0.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: 3.1.1
numexpr: 2.5
feather: None
matplotlib: 1.5.2
openpyxl: None
xlrd: 0.9.3
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[phofl]

Seems to work no on master. I get the following outputs:

a    1
b    2
c    3
dtype: int64
red  a    1
     b    2
     c    3
dtype: int64
red   a    1
blue  b    2
red   c    3
dtype: int64
red   a    1
blue  b    2
red   c    3
dtype: int64

Process finished with exit code 0