object depth information

[zoharbarzelay]

Hi,

Thanks for sharing your great work!
I was wondering - can one render a mesh and get per-pixel depth too, like e.g. mesh-renderer allows you to do?

Thanks a lot!
Z.

[nikhilaravi]

Hi @Zozobozo yes you can get the per pixel depth (for the top k faces which overlap with that pixel) from the output of the mesh rasterizer. fragments.zbuf is a (N, H, W, K) dimensional tensor.

To retrieve this output, you can initialize a rasterizer and only use that e.g.

rasterizer = MeshRasterizer(
    cameras=cameras, 
    raster_settings=raster_settings
)

fragments = rasterizer(meshes)

OR if you want the full image as well as the depth, you can extend the MeshRenderer class to create your own renderer which also returns the fragments.zbuf e.g

class MeshRendererWithDepth(nn.Module):
    def __init__(self, rasterizer, shader):
        super().__init__()
        self.rasterizer = rasterizer
        self.shader = shader

    def forward(self, meshes_world, **kwargs) -> torch.Tensor:
        fragments = self.rasterizer(meshes_world, **kwargs)
        images = self.shader(fragments, meshes_world, **kwargs)
        return images, fragments.zbuf

We also have a setting to enable perspective correct depth interpolation (set raster_settings.perspective_correct = True).

If this answers your question, please close this issue! :)

[zoharbarzelay]

thanks for the quick answer! i'll give it a try!
Best,
Z.

[wangsen1312]

@nikhilaravi I have tried Depth render with images, but found they are not in the same coordinate? it seems the y-axis flips? is this an issue or I use change it manually?
Best

[nikhilaravi]

@wangsen1312 this y flip issue has now been fixed - see #78 for further discussion.

[wangsen1312]

@nikhilaravi Got it, Nice work！

[Bob-Yeah]

Is this kind of depth image differentiable? @nikhilaravi

[nikhilaravi]

@Bob-Yeah yes it should be differentiable.

[aluo-x]

I'm actually kind of curious now. For the zbuf output (and we optimize with respect to another 2.5D depth map target), is it differentiable ONLY at pixels where there is a face? Or is this like the SoftSilhouetteShader where the boundaries can also be optimized?

[albertotono]

Following this.

I am using this shader

shader = SoftPhongShader(
    cameras=cameras,
    lights= lights,
    device=device
)

and this

rasterizer = MeshRendererWithDepth(rasterizer = rasterizer, shader = shader)
fragments = rasterizer(mesh)
fragments = fragments[1]

and my output is this.

I was wondering how can I extract something with a white gradient and a dark background like this

What are the main decisions regarding your color choice ( purple and green), how can I change those?!

[bottler]

When you do fragments = fragments[1] you have a tensor, not a picture. You have done something to make the picture, and your question pertains to that something, not pytorch3d. My guess is you have plotted a one-channel image with matplotlib, and it has defaulted to the viridis color scheme. You can change to a different color scheme, or manually convert to a 3 channel RGB image. E.g. expand the tensor from (H,W) to (H,W,3) (I think) so it becomes a 3-channel grayscale image.

[theschnitz]

We are landing a change now that introduces MeshRendererWithFragments that returns images, fragments into the renderer library which you can use in the future.