Why .sort_values() on column containing same values shuffles entire dataframe ?

[artsheiko]

df = pd.DataFrame({
    'col1': ['A', 'A', 'A', 'A', 'A', 'A']*15,
    'col2': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]*15,
    'col3': [0, 1, 9, 4, 2, 3]*15,
    'col4': ['a', 'B', 'c', 'D', 'e', 'F']*15
})
df.sort_values('col2', ascending=True)

Why this operation disorders the entire dataframe ? The same will be produced if the function will be applied for the col1.

[lithomas1]

@artsheiko I think this behavior is intended for sort_values(Nothing is moved for col1 since its already sorted), and the entire row including index is swapped in the DataFrame during sorting. If you want to reset the index(which I think you're trying to do), you can add .reset_index(drop=True) afterwards or pass ignore_index=True

[artsheiko]

@lithomas1,
reset_index() and ignore_index=True do not archive the expected result. Yes, both alter index column, so it will be 0, 1, ... n, but the dataframe is still shuffled randomly.
We obtain 0, 2, 4 ... in col3, but must be 0, 1, 9 ...

[attack68]

col2 contains the same value so sorting by it as a key is ambiguous. It could leave the dataframe unchanged or it could re-arrange any row to any location based on efficient memory use of the sorting algorithm, since the resultant col2 will still be 'sorted' according to non-decreasing order.

What do you expect?

[artsheiko]

The problem is that the result of sorting is shuffled randomly dataframe which is not so obvious for cases when we do not know in advance whether a column consists of one unique value or not.
The expected could be achieved with df.sort_values('col1', ascending=True, kind='mergesort'). Do we need to take into account any considerations / restrictions in terms of sorting big dataframes using this approach ?

[attack68]

The documentation points to np.ndarray.sort for info on the algorithms. The only option that is stable, where items are retained in relative order is mergesort as you have detected. There does not appear to be many downsides except the workspace is larger than the other two options. Not sure there is anything to do here, pls close if you agree.

[Quetzalcohuatl]

I just got burnt by this. I sorted my dataframe once on GPU Machine 1, then sorted it on CPU Machine 2, and found out that they have different indices! Now I have to write a workaround for my project.

@attack68 Would you be interested if we wrote a warning that sort values is not reproducible across machines, or maybe adding a random_state param to ensure reproducibility? Or is it not worthwhile? It was certainly a surprise to me!!!

[sodisga]

df.sort_values('col2', ascending=True, kind='stable') will not disoder the same values' index