Potential performance issue: Unreliable performance of .loc below pandas 2.1

[TendouArisu]

Issue Description:

Hello.
I have discovered a performance degradation in the .loc function of pandas version below 2.1 when .loc handling big DataFrame with non-unique indexes. When using pandas more than 4 indexes, .loc drastically increases to X1000 times. And I noticed that some parts of the repository depend on the pandas version below 2.1. For example, ML Projects/NLP_WebAPP_Twitter_Sentiment_Analysis_knowledge_graph/venv/Lib/site-packages/pandas/_version.py depends on pandas 2.0.0, ML Projects/Multi Class News Classification Project/requirements.txt depends on pandas 2.0.3. I found that many files in ML Projects/NLP_WebAPP_Twitter_Sentiment_Analysis_knowledge_graph used the influenced api. There may be more files using the influenced api. I am not sure whether this performance problem in pandas will affect this repository. Here are some discussions on GitHub related to this issue, including #54550 and #54746.

Suggestion

I would recommend considering an upgrade to a different version of pandas >= 2.1 or exploring other solutions to optimize the performance of .loc .
Any other workarounds or solutions would be greatly appreciated.
Thank you!

[Mj2603]

t might be beneficial to conduct a thorough review of the DataFrame operations within the affected files. Specifically, look for any repetitive/ inefficient use of .loc that could be optimized or replaced with alternative indexing methods. Try leveraging pandas' built-in functionalities such as iloc or vectorized operations where applicable, as they may offer better performance in certain scenarios. profiling the codebase to identify hotspots and bottlenecks could provide valuable insights for further optimization efforts.

[GaneshRehan]

Thank you for letting me know.
The case is, that with newer versions of Python and its libraries rolling out each month, performance issues do arise. So I have stuck to a bit older and stable version of Python and pandas to avoid such dependency and depreciation problems. The functions used are well supported by the version I have used. So I believe that there might be a difference in the environment we are using that is causing the issue. This is a complete project of SVM, to learn how it works. A sample project. Looking to optimize it to such extremes is not required in such small projects I believe.