support Long/Double type IValue

[narendasan]

Description

Type of change

Please delete options that are not relevant and/or add your own.

• New feature (non-breaking change which adds functionality)

Checklist:

• My code follows the style guidelines of this project (You can use the linters)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas and hacks
• I have made corresponding changes to the documentation
• I have added tests to verify my fix or my feature
• New and existing unit tests pass locally with my changes

[narendasan]

@inocsin I dont really understand the point of this change? Seems like its casting the input Tensor based on the already existing type of the Tensor and nothing else really changes.

[github-actions]

Code conforms to Python style guidelines

[github-actions]

Code conforms to C++ style guidelines

[inocsin]

@inocsin I dont really understand the point of this change? Seems like its casting the input Tensor based on the already existing type of the Tensor and nothing else really changes.

Some values in torchscript are stored as at::kLong and at::kDouble, but the tensorRT doesn't support those two type. So in conversion time, it will throw an error. That's why we must convert at::Double and at::kLong to trt support datatype to pass the conversion.

[narendasan]

Isnt this potentially dangerous? We would be truncating values in double precision to single precision and in this case it would be done silently

[inocsin]

Isnt this potentially dangerous? We would be truncating values in double precision to single precision and in this case it would be done silently

Yeah, maybe we can throw a warning? But if we doesn't support the type conversion, the module cann't be compiled by trt

[narendasan]

Are tensors from PyTorch in your target module actually double precision? I think its better in that case that we catch this and fully error out telling users to put in a type cast to floating point so they make the choice explicitly.

[inocsin]

Are tensors from PyTorch in your target module actually double precision? I think its better in that case that we catch this and fully error out telling users to put in a type cast to floating point so they make the choice explicitly.

Yeah, the original values are double precision. In my situation, some users may not want to convert precision explicitly, so that's why I think we should do this for them.

[narendasan]

We can add a setting to the CompileSpec that is something like

CompileSpec {
    ... 
    bool truncate_long_and_double = False;
}

So while by default you will error out, if you explicitly opt in, weights will be truncated for you. Then we can tell if say an accuracy issue may be the result of truncating or some other issue.

[narendasan]

I can PR into your branch the API to do this and the checks if you think this would be sufficient

[narendasan]

@peri044 says we should also put a warning in so even if someone opts in you get a message if it actually truncates.