Skip to content

[RISCV] Match strided vector bases in RISCVGatherScatterLowering #93972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 3, 2024
Merged

[RISCV] Match strided vector bases in RISCVGatherScatterLowering #93972

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3f0acb2
[RISCV] Match strided vector bases in RISCVGatherScatterLowering
lukel97 May 31, 2024
62eb32d
Check that the pointer operand is an Instruction
lukel97 May 31, 2024
aa6a40d
Move down call to determineBaseAndStride to avoid generating code wit…
lukel97 May 31, 2024
5303532
Name new GEP based off old GEP
lukel97 Jun 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion llvm/lib/Target/RISCV/RISCVGatherScatterLowering.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,8 +349,28 @@ RISCVGatherScatterLowering::determineBaseAndStride(Instruction *Ptr,

SmallVector<Value *, 2> Ops(GEP->operands());

// If the base pointer is a vector, check if it's strided.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move this down into the if (!ScalarBase) case below, we'd rather find a splatable base if possible.

Please perform the operand check before doing the recursive call. The unwind from the recursion may generate code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried moving this down into the !ScalarBase case but it caused us to miss the splat_base_scalar_offset test case.

The code below bails if GEP doesn't have a vector offset, so putting it up here catches the case where we have a splat base + scalar offset.

Value *Base = GEP->getPointerOperand();
if (auto *BaseInst = dyn_cast<Instruction>(Base);
BaseInst && BaseInst->getType()->isVectorTy()) {
// If GEP's offset is scalar then we can add it to the base pointer's base.
auto IsScalar = [](Value *Idx) { return !Idx->getType()->isVectorTy(); };
if (all_of(GEP->indices(), IsScalar)) {
auto [BaseBase, Stride] = determineBaseAndStride(BaseInst, Builder);
if (BaseBase) {
Builder.SetInsertPoint(GEP);
SmallVector<Value *> Indices(GEP->indices());
Value *OffsetBase =
Builder.CreateGEP(GEP->getSourceElementType(), BaseBase, Indices,
"", GEP->isInBounds());
OffsetBase->takeName(GEP);
return {OffsetBase, Stride};
}
}
}

// Base pointer needs to be a scalar.
Value *ScalarBase = Ops[0];
Value *ScalarBase = Base;
if (ScalarBase->getType()->isVectorTy()) {
ScalarBase = getSplatValue(ScalarBase);
if (!ScalarBase)
Expand Down
12 changes: 4 additions & 8 deletions llvm/test/CodeGen/RISCV/rvv/strided-load-store.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,10 +301,8 @@ define void @constant_stride(<vscale x 1 x i64> %x, ptr %p, i64 %stride) {

define <vscale x 1 x i64> @vector_base_scalar_offset(ptr %p, i64 %offset) {
; CHECK-LABEL: @vector_base_scalar_offset(
; CHECK-NEXT: [[STEP:%.*]] = call <vscale x 1 x i64> @llvm.experimental.stepvector.nxv1i64()
; CHECK-NEXT: [[PTRS1:%.*]] = getelementptr i64, ptr [[P:%.*]], <vscale x 1 x i64> [[STEP]]
; CHECK-NEXT: [[PTRS2:%.*]] = getelementptr i64, <vscale x 1 x ptr> [[PTRS1]], i64 [[OFFSET:%.*]]
; CHECK-NEXT: [[X:%.*]] = call <vscale x 1 x i64> @llvm.masked.gather.nxv1i64.nxv1p0(<vscale x 1 x ptr> [[PTRS2]], i32 8, <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer), <vscale x 1 x i64> poison)
; CHECK-NEXT: [[TMP1:%.*]] = getelementptr i64, ptr [[P:%.*]], i64 [[OFFSET:%.*]]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test case or two with a base we can't match, and one with a matchable base, but non matchable indices to cover the bug noted in my comment above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 458a315

; CHECK-NEXT: [[X:%.*]] = call <vscale x 1 x i64> @llvm.riscv.masked.strided.load.nxv1i64.p0.i64(<vscale x 1 x i64> poison, ptr [[TMP1]], i64 8, <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer))
; CHECK-NEXT: ret <vscale x 1 x i64> [[X]]
;
%step = call <vscale x 1 x i64> @llvm.experimental.stepvector.nxv1i64()
Expand All @@ -321,10 +319,8 @@ define <vscale x 1 x i64> @vector_base_scalar_offset(ptr %p, i64 %offset) {

define <vscale x 1 x i64> @splat_base_scalar_offset(ptr %p, i64 %offset) {
; CHECK-LABEL: @splat_base_scalar_offset(
; CHECK-NEXT: [[HEAD:%.*]] = insertelement <vscale x 1 x ptr> poison, ptr [[P:%.*]], i32 0
; CHECK-NEXT: [[SPLAT:%.*]] = shufflevector <vscale x 1 x ptr> [[HEAD]], <vscale x 1 x ptr> poison, <vscale x 1 x i32> zeroinitializer
; CHECK-NEXT: [[PTRS:%.*]] = getelementptr i64, <vscale x 1 x ptr> [[SPLAT]], i64 [[OFFSET:%.*]]
; CHECK-NEXT: [[X:%.*]] = call <vscale x 1 x i64> @llvm.masked.gather.nxv1i64.nxv1p0(<vscale x 1 x ptr> [[PTRS]], i32 8, <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer), <vscale x 1 x i64> poison)
; CHECK-NEXT: [[TMP1:%.*]] = getelementptr i64, ptr [[P:%.*]], i64 [[OFFSET:%.*]]
; CHECK-NEXT: [[X:%.*]] = call <vscale x 1 x i64> @llvm.riscv.masked.strided.load.nxv1i64.p0.i64(<vscale x 1 x i64> poison, ptr [[TMP1]], i64 0, <vscale x 1 x i1> shufflevector (<vscale x 1 x i1> insertelement (<vscale x 1 x i1> poison, i1 true, i64 0), <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer))
; CHECK-NEXT: ret <vscale x 1 x i64> [[X]]
;
%head = insertelement <vscale x 1 x ptr> poison, ptr %p, i32 0
Expand Down
Loading