How to use SfMPerspectiveCameras for renderer?

[wangg12]

I want to use this differentiable renderer to render a LINEMOD object. The .ply model vertices are loaded and then scale from mm to m by a factor 0.001.

The camera intrinsic:

K = np.array([[572.4114, 0, 325.2611], [0, 573.57043, 242.04899], [0, 0, 1]])

Rotation and translation

R = np.eye(3)
T = np.array([-0.1, 0.1, 0.7])

I followed the camera position tutorial to set the cameras and renderers, except that the camera I used is

cameras = SfMPerspectiveCameras(
        focal_length=((K[0,0], K[1,1]), ),
        principal_point=((K[0,2], K[1,2]),),
        device=device,
    )

And since I want to get 480x640 image, I set the image_size=640 and crop to 480x640 after rendering.

However, there is nothing in the rendered results. I could get correct results with a OpenGL renderer with the same settings.

So I wonder how I can correctly use the SfMPerspectiveCameras?

[davnov134]

Hi wangg12,

can I please ask whether you can share the rendering script?

More specifically, it is not obvious what is the position of the LM object in the world coordinates. Furthermore, the "R" matrix you are using does not seem to be a valid rotation, see below:

R = torch.tensor([[-0.5,  0.8,  0],  [0, 0, -1],  [-0.8, -0.5,  0]])
torch.det(R)
> tensor(0.8900) # instead of 1

In case the LM object should be centered around (0, 0, 0), I would suggest setting the camera to a trivial one with a non-zero z-component of the T vector, e.g.:

R = torch.eye(3), T = torch.tensor([0., 0., 2.])

This would point the camera directly at the object center and render it from the distance of 2 world units.

[wangg12]

Sorry, R was not the correct one. It was transformed from axis angle and should be correct. I shared my script here

[nikhilaravi]

@wangg12 could you please share a small script to reproduce the error which we can run easily? For example load and render one mesh with the settings you have mentioned. This will enable us to help you much faster :)

[wangg12]

Finally I got the correct results with a modified OpenGL Real Perspective Camera.
The scripts are here.

BTW, @nikhilaravi How can I render the differentiable depth values?

[aluo-x]

Haven't tried this myself since I am working with RGB, but Line 37 of renderer.py exposes a fragment, which should have a zbuf object.

Also according to documentation here:

The PyTorch3d backward pass returns gradients for zbuf, dist and bary_coords

[nikhilaravi]

@wangg12 Glad you got it to work. I can take a look at your script later! For future, please try to share the minimum amount of code needed to explain the issue so we can review more quickly :)

Thanks @aluo-x for explaining the zbuf/fragments :) As mentioned, we expose the zbuffer as part of the fragments object which is output from the rasterizer. It will be a N x H x W x K dimensional tensor.

[bhadresh74]

@wangg12 Thanks for sharing your script. I am trying to do something similar using your script.
However the rendering is vertically stretched as shown below.

Ideally, it should look like this.

My image size is (960, 1280). I believe it has something to do with rasterizer since it only works for square image.
Can you guide me in the right direction here?

[nikhilaravi]

@bhadresh74 yes the rasterizer currently only works for square images!

[gkioxari]

Hi @bhadresh74

Since 57a22e7 we have updated the cameras and have added a note to explain the space and definitions here:

https://github.com/facebookresearch/pytorch3d/blob/master/docs/notes/cameras.md

Note that by default PyTorch3D cameras assume parameters in NDC space for rendering. We have updated cameras.py such that SfMPerspective and SfMOrthographic can now accept parameters in both NDC and screen space.