Add per tensor fp8 quantization support for conv3d #3215
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3215
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 94c2e60 with merge base 7e5d907 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
|
currently waiting for the fbgemm conv op to be available in nightly
|
jerryzh168
changed the title
jerryzh168
force-pushed
the
add-fp8-conv-weight-quant-support
branch
4 times, most recently
from
2569be1 to
b5c8ca5
Compare
jerryzh168
added
the
topic: new feature
jerryzh168
force-pushed
the
add-fp8-conv-weight-quant-support
branch
from
b5c8ca5 to
2ccc619
Compare
jerryzh168
force-pushed
the
add-fp8-conv-weight-quant-support
branch
from
2ccc619 to
4c6e979
Compare
torchao/quantization/quant_api.py
Outdated
| activation_granularity, weight_granularity = granularity | ||
|
|
||
| if not _fp8_mm_compat(weight): | ||
| if weight.dim() != 5 and not _fp8_mm_compat(weight): |
There was a problem hiding this comment.
5 seems a bit arbitrary here without the context, should we add a comment that this is for conv3d?
| "is_MI300", | ||
| "is_sm_at_least_89", | ||
| "is_sm_at_least_90", | ||
| "is_sm_at_least_100", |
There was a problem hiding this comment.
not super related to this PR but I wonder if we should stop exposing these, we don't expect users to call these themselves
There was a problem hiding this comment.
oh, what should user call?
There was a problem hiding this comment.
oh should we use is_sm_version? but I think we need is_sm_at_least
There was a problem hiding this comment.
oh I mean should users even call these helper functions? They know what GPUs they're running on. If they really want to check then maybe they should just check torch.cuda.get_device_capability() >= (10, 0) themselves instead of importing our utils. I'd like to keep them private if possible (in a separate PR)
There was a problem hiding this comment.
oh I see, makes sense, yeah this should just be dev only, not user facing
| stride, | ||
| padding, | ||
| dilation, | ||
| ): |
There was a problem hiding this comment.
do we also need to check kernel preference? Like if it's "torch" maybe we should throw an exception since we don't support that yet?
jerryzh168
force-pushed
the
add-fp8-conv-weight-quant-support
branch
2 times, most recently
from
3339fd7 to
642cec4
Compare
Summary: att, we added support of quantization conv3d weights, with Float8DynamicActivationFloat8WeightConfig API: ``` config = Float8DynamicActivationFloat8WeightConfig( granularity=PerTensor(), ) _is_conv3d = lambda m, fqn: isinstance(m, torch.nn.Conv3d) quantize_(quantized_model, config, filter_fn=_is_conv3d) ``` Test Plan: pytest test/quantization/quantize_/workflows/float8/test_float8_tensor.py -k test_fp8_conv_variants Reviewers: Subscribers: Tasks: Tags:
jerryzh168
force-pushed
the
add-fp8-conv-weight-quant-support
branch
from
642cec4 to
94c2e60
Compare
Add per tensor fp8 quantization support conv3d Summary: att, we added support of quantization conv3d weights, with Float8DynamicActivationFloat8WeightConfig API: ``` config = Float8DynamicActivationFloat8WeightConfig( granularity=PerTensor(), ) _is_conv3d = lambda m, fqn: isinstance(m, torch.nn.Conv3d) quantize_(quantized_model, config, filter_fn=_is_conv3d) ``` Test Plan: pytest test/quantization/quantize_/workflows/float8/test_float8_tensor.py -k test_fp8_conv_variants Reviewers: Subscribers: Tasks: Tags:
Summary:
att, we added support of quantization conv3d weights, with Float8DynamicActivationFloat8WeightConfig
API:
Test Plan:
pytest test/quantization/quantize_/workflows/float8/test_float8_tensor.py -k test_fp8_conv_variants
Reviewers:
Subscribers:
Tasks:
Tags: