The logic should be that if a package component declares a dependency on a library or sub-library in the package, then hs-source-dirs should not overlap. This is because imports that occur within the executable / test-suite / benchmark / sub-library etc will instead recompile the code, which is a waste.
It also potentially has other negative effects - #3109 (comment)