-
Notifications
You must be signed in to change notification settings - Fork 376
feat: Added Python accuracy tests using Nox #750
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Anurag Dixit <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/modules/hub.py
ERROR: Some files do not conform to style guidelinesThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/modules/hub.py
ERROR: Some files do not conform to style guidelinesSigned-off-by: Anurag Dixit <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/modules/hub.py
ERROR: Some files do not conform to style guidelinesThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/modules/hub.py
ERROR: Some files do not conform to style guidelines|
Is it possible to have different nox configurations for the docker container and local so we can use nox for development as well? |
|
Also are the Python API tests like backend, PTQ etc going to be automated via nox? |
We can write all the configurations in a single nox file and run selective sessions for different test cases. |
I have already added PTQ, QAT and API tests in the noxfile. It is included in this PR. nox --sessions qat_test-3 |
|
Cool, this looks like its going to work really well. |
Signed-off-by: Anurag Dixit <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/modules/hub.py
ERROR: Some files do not conform to style guidelinesSigned-off-by: Anurag Dixit <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/modules/hub.py
ERROR: Some files do not conform to style guidelinesThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/modules/hub.py
ERROR: Some files do not conform to style guidelinesThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
peri044
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor changes. LGTM
Signed-off-by: Anurag Dixit <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
Reformatting /workspace/py/torch_tensorrt/_compile.py
Reformatting /workspace/py/torch_tensorrt/_Device.py
Reformatting /workspace/py/torch_tensorrt/__init__.py
Reformatting /workspace/py/torch_tensorrt/_Input.py
Reformatting /workspace/py/torch_tensorrt/_util.py
Reformatting /workspace/py/torch_tensorrt/ptq.py
Reformatting /workspace/py/torch_tensorrt/_enums.py
Reformatting /workspace/py/torch_tensorrt/logging.py
Reformatting /workspace/py/torch_tensorrt/ts/__init__.py
Reformatting /workspace/py/torch_tensorrt/ts/_compile_spec.py
Reformatting /workspace/py/torch_tensorrt/ts/_compiler.py
Reformatting /workspace/py/setup.py
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/modules/hub.py
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api.py
ERROR: Some files do not conform to style guidelinesThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code conforms to C++ style guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some changes that do not conform to Python style guidelines:
Reformatting /workspace/py/torch_tensorrt/_compile.py
Reformatting /workspace/py/torch_tensorrt/_Device.py
Reformatting /workspace/py/torch_tensorrt/__init__.py
Reformatting /workspace/py/torch_tensorrt/_Input.py
Reformatting /workspace/py/torch_tensorrt/_util.py
Reformatting /workspace/py/torch_tensorrt/ptq.py
Reformatting /workspace/py/torch_tensorrt/_enums.py
Reformatting /workspace/py/torch_tensorrt/logging.py
Reformatting /workspace/py/torch_tensorrt/ts/__init__.py
Reformatting /workspace/py/torch_tensorrt/ts/_compile_spec.py
Reformatting /workspace/py/torch_tensorrt/ts/_compiler.py
Reformatting /workspace/py/setup.py
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_qat_trt_accuracy.py
Reformatting /workspace/tests/modules/hub.py
--- /workspace/tests/py/test_api.py (original)
+++ /workspace/tests/py/test_api.py (reformatted)
@@ -32,9 +32,9 @@
def test_compile_script(self):
with torch.no_grad():
trt_mod = torchtrt.ts.compile(self.scripted_model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
@@ -49,9 +49,9 @@
def test_compile_global_nn_mod(self):
with torch.no_grad():
trt_mod = torchtrt.compile(self.model,
- inputs=[self.input],
- device=torchtrt.Device(gpu_id=0),
- enabled_precisions={torch.float})
+ inputs=[self.input],
+ device=torchtrt.Device(gpu_id=0),
+ enabled_precisions={torch.float})
same = (trt_mod(self.input) - self.scripted_model(self.input)).abs().max()
self.assertTrue(same < 2e-2)
Reformatting /workspace/tests/py/test_trt_intercompatability.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api.py
ERROR: Some files do not conform to style guidelines|
Looks like python linter is the one left. Feel free to merge after linter passes @andi4191 |
Signed-off-by: Anurag Dixit [email protected]
Description
Added support for Python accuracy tests using Nox. Using nox, one can run complete python tests more conveniently.
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant and/or add your own.
Checklist: