PERF: core/base/IndexOpsMixin duplicated should be changed to use same impl as frame.duplicated

[jreback]

DataFrame.duplicated correctly uses hashtable.duplicated_int64 a specialized routine, while core.series.IndexOpsMixin.duplicated uses lib.duplicated an object based one.

as its almost always better to factorize then use the fast routine, than to do object comparisons.
but YMMV, so needs a couple of perf tests.

In [3]: s = Series(np.random.randint(0,10000,size=100000))

In [4]: %timeit s.duplicated()
100 loops, best of 3: 5.75 ms per loop

In [6]: df = s.to_frame()

In [8]: %timeit df.duplicated()
1000 loops, best of 3: 1.86 ms per loop

[sinhrks]

Nice to have a single flow. Because current series logic looks faster for object dtype, branching the logic depending on dtype?

s = pd.Series(pd.util.testing.rands_array(nchars=10, size=1000000))

%timeit s.duplicated()
# 1 loops, best of 3: 448 ms per loop

df = s.to_frame()
%timeit df.duplicated()
# 1 loops, best of 3: 764 ms per loop

# case which has some duplicates
s = pd.Series(np.repeat(pd.util.testing.rands_array(nchars=10, size=1000), 1000))
%timeit s.duplicated()
# 10 loops, best of 3: 32.4 ms per loop

df = s.to_frame()
%timeit df.duplicated()
# 10 loops, best of 3: 83.3 ms per loop

[jreback]

for sure branch on type
I think create a float_64 one (put in hashtable.pyx) - this can release Gil as well (I'll create this in my Gil branch)

the object one can stay as it is

[sinhrks]

OK. I worked a little and it additionally needs datetime and float handling...

master...sinhrks:dup_perf