[SLP][REVEC] Make Instruction::Select support vector instructions. #100507

HanKuanChen · 2024-07-25T04:05:01Z

No description provided.

llvmbot · 2024-07-25T04:05:34Z

@llvm/pr-subscribers-llvm-transforms

Author: Han-Kuan Chen (HanKuanChen)

Changes

Full diff: https://github.com/llvm/llvm-project/pull/100507.diff

2 Files Affected:

(modified) llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp (+20)
(modified) llvm/test/Transforms/SLPVectorizer/revec.ll (+30)

diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index b74417f4606e7..9fcfc2a0db3b1 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -9713,6 +9713,17 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
         }
         VecCost = std::min(VecCost, IntrinsicCost);
       }
+      if (auto *SI = dyn_cast<SelectInst>(VL0)) {
+        auto *CondType =
+            getWidenedType(SI->getCondition()->getType(), VL.size());
+        unsigned CondNumElements = CondType->getNumElements();
+        unsigned VecTyNumElements = getNumElements(VecTy);
+        if (CondNumElements != VecTyNumElements)
+          VecCost += TTI->getShuffleCost(
+              TTI::SK_PermuteSingleSrc, CondType,
+              createReplicatedMask(VecTyNumElements / CondNumElements,
+                                   CondNumElements));
+      }
       return VecCost + CommonCost;
     };
     return GetCostDiff(GetScalarCost, GetVectorCost);
@@ -13196,6 +13207,15 @@ Value *BoUpSLP::vectorizeTree(TreeEntry *E, bool PostponedPHIs) {
           False = Builder.CreateIntCast(False, VecTy, GetOperandSignedness(2));
       }
 
+      unsigned CondNumElements = getNumElements(Cond->getType());
+      unsigned TrueNumElements = getNumElements(True->getType());
+      if (CondNumElements != TrueNumElements) {
+        // When the return type is i1 but the source is fixed vector type, we
+        // need to duplicate the condition value.
+        Cond = Builder.CreateShuffleVector(
+            Cond, createReplicatedMask(TrueNumElements / CondNumElements,
+                                       CondNumElements));
+      }
       Value *V = Builder.CreateSelect(Cond, True, False);
       V = FinalShuffle(V, E, VecTy);
 
diff --git a/llvm/test/Transforms/SLPVectorizer/revec.ll b/llvm/test/Transforms/SLPVectorizer/revec.ll
index c2dc6d0ab73b7..a6e1061189980 100644
--- a/llvm/test/Transforms/SLPVectorizer/revec.ll
+++ b/llvm/test/Transforms/SLPVectorizer/revec.ll
@@ -58,3 +58,33 @@ entry:
   store <8 x i16> %4, ptr %5, align 2
   ret void
 }
+
+define void @test3(ptr %x, ptr %y, ptr %z) {
+; CHECK-LABEL: @test3(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x ptr> poison, ptr [[X:%.*]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x ptr> [[TMP0]], ptr [[Y:%.*]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq <2 x ptr> [[TMP1]], zeroinitializer
+; CHECK-NEXT:    [[TMP3:%.*]] = load <8 x i32>, ptr [[X]], align 4
+; CHECK-NEXT:    [[TMP4:%.*]] = load <8 x i32>, ptr [[Y]], align 4
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i1> [[TMP2]], <2 x i1> poison, <8 x i32> <i32 0, i32 0, i32 0, i32 0, i32 1, i32 1, i32 1, i32 1>
+; CHECK-NEXT:    [[TMP6:%.*]] = select <8 x i1> [[TMP5]], <8 x i32> [[TMP3]], <8 x i32> [[TMP4]]
+; CHECK-NEXT:    store <8 x i32> [[TMP6]], ptr [[Z:%.*]], align 4
+; CHECK-NEXT:    ret void
+;
+entry:
+  %0 = getelementptr inbounds i32, ptr %x, i64 4
+  %1 = getelementptr inbounds i32, ptr %y, i64 4
+  %2 = load <4 x i32>, ptr %x, align 4
+  %3 = load <4 x i32>, ptr %0, align 4
+  %4 = load <4 x i32>, ptr %y, align 4
+  %5 = load <4 x i32>, ptr %1, align 4
+  %6 = icmp eq ptr %x, null
+  %7 = icmp eq ptr %y, null
+  %8 = select i1 %6, <4 x i32> %2, <4 x i32> %4
+  %9 = select i1 %7, <4 x i32> %3, <4 x i32> %5
+  %10 = getelementptr inbounds i32, ptr %z, i64 4
+  store <4 x i32> %8, ptr %z, align 4
+  store <4 x i32> %9, ptr %10, align 4
+  ret void
+}

alexey-bataev · 2024-07-25T17:15:40Z

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

@@ -9713,6 +9713,17 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
        }
        VecCost = std::min(VecCost, IntrinsicCost);
      }
+      if (auto *SI = dyn_cast<SelectInst>(VL0)) {


Why this is specific only for selects?

Only select has a special form.
For fcmp and icmp, it is either i1 = cmp <cond> <ty>, <ty> or <i1 x N> = cmp <cond> <ty x N>, <ty x N>.
For select, it can be select i1, <ty>, <ty>, select <i1 x N>, <ty x N>, <ty x N> or select i1, <ty x N>, <ty x N>.

alexey-bataev · 2024-07-25T17:16:41Z

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

+        // When the return type is i1 but the source is fixed vector type, we
+        // need to duplicate the condition value.
+        Cond = Builder.CreateShuffleVector(
+            Cond, createReplicatedMask(TrueNumElements / CondNumElements,


What if CondNumElements > TrueNumElements?

Cannot. CondNumElements <= TrueNumElements.

Add assertion

Add in getEntryCost, including comments.

Add here too

Can it be that TrueNumElements % CondNumElements != 0? Will you handle it correctly?

Cannot. They are the operands of Instruction::Select. The number of elements must be the same.

Add the assertions, please, in all branches

alexey-bataev · 2024-07-30T19:42:36Z

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

+      assert(TrueNumElements % CondNumElements == 0 &&
+             "Cannot vectorize Instruction::Select");


It does not prove that TrueNumElements >= CondNumElements

alexey-bataev

LG

[SLP][REVEC] Pre-commit test.

04a648b

HanKuanChen requested a review from alexey-bataev July 25, 2024 04:05

llvmbot added vectorizers llvm:transforms labels Jul 25, 2024

alexey-bataev reviewed Jul 25, 2024

View reviewed changes

HanKuanChen requested a review from alexey-bataev July 26, 2024 09:28

HanKuanChen force-pushed the slp-revec-select branch 3 times, most recently from fa09007 to 8941946 Compare July 30, 2024 15:33

alexey-bataev reviewed Jul 30, 2024

View reviewed changes

[SLP][REVEC] Make Instruction::Select support vector instructions.

b29ba15

HanKuanChen force-pushed the slp-revec-select branch from 8941946 to b29ba15 Compare July 31, 2024 07:11

HanKuanChen requested a review from alexey-bataev July 31, 2024 07:14

alexey-bataev approved these changes Jul 31, 2024

View reviewed changes

HanKuanChen merged commit b5a7d3b into llvm:main Jul 31, 2024
4 of 7 checks passed

HanKuanChen deleted the slp-revec-select branch July 31, 2024 15:03

		assert(TrueNumElements % CondNumElements == 0 &&
		"Cannot vectorize Instruction::Select");

[SLP][REVEC] Make Instruction::Select support vector instructions. #100507

[SLP][REVEC] Make Instruction::Select support vector instructions. #100507

Uh oh!

Conversation

HanKuanChen commented Jul 25, 2024

Uh oh!

llvmbot commented Jul 25, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-bataev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!