R is not a valid rotation matrix

[jahaniam]

Tried to run the 'camera_position_optimization_with_differentiable_rendering.ipynb' with my custom obj. Although it works fine for the provided obj, for my custom obj the look_at_view_transform and look_at_rotation sometime returns :

R is not a valid rotation matrix which is weird!

[nikhilaravi]

Can you share a code snippet? e.g. how are you calling look_at_view_transform and look_at_rotation? The warning is from the Transforms3d class instance which is returned from look_at_view_transform, so it should be independent of the obj that is used.

[jahaniam]

Thank you for your quick reply. Yeah, you are right. If I set:

distance = 10   # distance from camera to the object
elevation = 90.0   # angle of elevation in degrees
azimuth = 10.0  # angle of azimuth rotation in degrees

I will get the warning for teapot.

I'm using the same template notebook as it is in the repository with the teapot camera_position_optimization_with_differentiable_rendering.ipynb. This happens when elevation is 90 or -90 or etc. The renderer also breaks at elevation 90(probably because of invalid rotation).

That's probably why my loss becomes NaN, if in the optimization process the elevation becomes 90 somehow then I'm assuming it will be unstable and won't converge

[nikhilaravi]

@a-jahani ok thanks for sharing more details. I will try to reproduce this and get back to you.

[davnov134]

Hi @a-jahani,

I took a look into this issue and my first guess is that the instability might happen due to the camera getting into a position where the look_at_view_transform cannot be computed.

For instance, if the camera gets to a position (0,0,0) and your look_at vector is (0,0,0) as well,
look_at_view_transform will return a singular matrix R which is not a rotation.

In order to debug this further, would it be please possible to share a bit more of your code? The distance, azimuth and elevation params, as you have shared, do not seem to cause the instability (camera_position_from_spherical_angles returns a correct output), so I guess the problem might be the camera position.

[tau-yihouxiang]

@nikhilaravi I encountered the same issue when predicting the camera parameters from images. The loss equals to NaN with error message 'R is not a valid rotation matrix'.

[tau-yihouxiang]

@nikhilaravi If you change the initial camera position to (1.0, 1.0, 1.0) in tutorial 'camera_position_optimization_with_differentiable_rendering.ipynb', then 'nan loss' and 'R is not a valid rotation matrix' will occur.

[davnov134]

@nikhilaravi If you change the initial camera position to (1.0, 1.0, 1.0) in tutorial 'camera_position_optimization_with_differentiable_rendering.ipynb', then 'nan loss' and 'R is not a valid rotation matrix' will occur.

I tried setting the initial position of the camera as you suggested, however I can't run into the issue explained. For me the initial learning step shows the following:

And the optimization terminates with:

Can you please check what is stored in model.camera_position at the time when either the NaN loss occurs or you get the warning about the rotation matrix?

[nikhilaravi]

@tau-yihouxiang were you able to resolve this issue?

[jahaniam]

@davnov134 Thanks for the quick reply.

Forget about the instability. Here is exactly how to reproduce the warning for R and also wrong rendering:

go to this file: camera_position_optimization_with_differentiable_rendering
find elevation = 40.0 and chenge it to: elevation = -90.0
then run the block again and you will see a result similar to the below screenshot and warnings for R.
I don't think that is related to the look_at vector because this happens with any distance when elavation is exactly 90 or -90.

for elevation = -90.0:

for elevation = -89.999:

Edit2:
R warnings are gone after git pull but still renderer breaks for elevation 90. Found the bug. It is as you said related to the look_at vector. I will fix and pull request.
The bug is here in look_at_rotation function:

    z_axis = F.normalize(at - camera_position, eps=1e-5)
    x_axis = F.normalize(torch.cross(up, z_axis), eps=1e-5)
    y_axis = F.normalize(torch.cross(z_axis, x_axis), eps=1e-5)

when camera_position = up vector then z_axis= up vector then torch.cross(up, z_axis) will be a cross product of two same vector which is zero

[jahaniam]

based on this

The Look-At Method Limitation
The method is very simple and works generally well. Though it has an Achilles heels (a weakness). When the camera is vertical looking straight down or straight up, the forward axis gets very close to the arbitrary axis used to compute the right axis. The extreme case is of course when the froward axis and this arbitrary axis are perfectly parallel e.g. when the forward vector is either (0,1,0) or (0,-1,0). Unfortunately in this particular case, the cross product fails producing a result for the right vector. There is actually no real solution to this problem. You can either detect this case, and choose to set the vectors by hand (since you know what the configuration of the vectors should be anyway). A more elegant solution can be developed using quaternion interpolation.

an easy solution would be to use another up vector:
R, T = look_at_view_transform(distance, elevation, azimuth,up=((0, 0, 1),), device=device)