Object Classification with only 3D models in the training

[albertotono]

❓ Questions on how to use PyTorch3D for specific Object Classification in 3D only Datasets

Thanks again for this amazing project. I think Pytorch3D could enable better object classification tasks, correct me if I am wrong ( similar to this work )

I was wondering: in the case we have only 3D models in our Dataset. (for example only the 3d models from ShapeNET, Pix3D, Pascal or others), but with no real pictures of those objects.

Since we need to perform the training on this completely synthetic environment, I think Pytorch3D could be extremely helpful. In the past we used Blender, but now it could be possible to have everything in the same pipeline. I was wondering if there is any code sample available regarding this task of specific object classification (or reconstruction to specific objects)

For example MeshRCNN, is an amazing project, but it is really focused in a way harder task. It tries to generalize and reconstruct objects that were not present in the training set, ( for unseen classes and objects). My question is way simpler, I want to recognize the object in my dataset.

Let's say I have 3 items that I want to 3d print:

1. Chair Specific Model BXshsda43
2. Table Specific Model Anca34
3. Computer Specific Model Calds54

After I 3D print those objects I want to be able to recognize those when I take a picture.

Could you provide the best example using Pytorch3D to this problem:

• Training Dataset: Only 3D Models
• Inference: Image --> Correct 3D object "reconstruction" or better retrieval.

1. Is there someone else working on this task using Pytorch3d?
2. What do you think is still missing to achieve this pipeline?
3. We can have the depth-map of objects following previous issues here. But is possible to change the render background with another image?!
4. How far is Pytorch3D to replace Blender in the current specific production pipelines or task?

Thanks again for this amazing work and for your even greater availability.

Best Regards

[gkioxari]

Hi Alberto!!

Your question is very relevant and while I don't have example code I am happy to give you some feedback in case it's useful.

So if understand correctly you have a dataset of M object models (these are different 3D shape models e.g. chair model BXshsda43, table model Calds54, etc.). And you want given an image to recognize if any of these models is present in the image. If you have a training set of images and you know which models are present you could train an object classifier or object detector (similar to how we detect object categories in images, see detectron2) but instead of object classes your classes are object models. This is clearly a harder task than object detection, i.e. detecting if an object is a cow or a chair or a hair drier, as in your case you have to perform fine grained recognition, i.e. distinguish chair model X from chair model Y. But if you have enough training examples I think there is hope it would work.

I made a call-for-collaboration label and tagged your issue as you are asking if others are working on this task. Hopefully this tag will attract other users who might be working on it and could chime in.
Regarding, rendering with an image background image, we currently don't support this but you could quickly hack it together using the alpha channel of our renderer output to overlay the object on the image you desire.
Regarding how far PyTorch3D is from replacing Blender, the two libraries try to achieve very different things. Blender is a great tool for forward rendering and if you merely wish to produce object renderings then you should use it. PyTorch3D's goal is to provide users with the ability to learn via rendering, thus re-defining traditional rendering functions so that they admit gradients. So the focus of the two tools is rather different and you should use the one that serves your purpose.

[gkioxari]

Closing this for now! The tag remains and would show up with a search.