You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Official repository for the **_Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation_**[paper :page_facing_up:](http://arxiv.org/abs/2204.07548) selected for an Oral presentation at CVPR 2022.
4
+
Official repository for **_Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation_**[paper :page_facing_up:](http://arxiv.org/abs/2204.07548) selected for an Oral presentation at CVPR 2022.
*Recent works on 3D semantic segmentation propose to exploit the synergy between images and point clouds by pro- cessing each modality with a dedicated network and project- ing learned 2D features onto 3D points. Merging large-scale point clouds and images raises several challenges, such as constructing a mapping between points and pixels, and ag- gregating features between multiple views. Current methods require mesh reconstruction or specialized sensors to recover occlusions, and use heuristics to select and aggregate avail- able images. In contrast, we propose an end-to-end trainable multi-view aggregation model leveraging the viewing condi- tions of 3D points to merge features from images taken at ar- bitrary positions. Our method can combine standard 2D and 3D networks and outperforms both 3D models operating on colorized point clouds and hybrid 2D/3D networks without requiring colorization, meshing, or true depth maps. We set a new state-of-the-art for large-scale indoor/outdoor semantic segmentation on S3DIS (74.7 mIoU 6-Fold) and on KITTI- 360 (58.3 mIoU). Our full pipeline only requires raw 3D scans and a set of images and poses.*
10
+
*We propose to exploit the synergy between images and 3D point clouds by learning to select the most relevant views for each point. Our approach uses the viewing conditions of 3D points to merge features from images taken at arbitrary positions. We reach SOTA results for S3DIS (74.7 mIoU 6-Fold) and on KITTI- 360 (58.3 mIoU) without requiring point colorization, meshing, or the use of depth cameras: our full pipeline only requires raw 3D scans and a set of images and poses.*
13
11
14
12
## Coming very soon :rotating_light::construction:
15
-
-**notebooks** for manipulating multimodal data for S3DIS, ScanNet and KITTI-360, training and testing models and reproduce our papers' main results.
13
+
-**notebooks** for manipulating multimodal data for S3DIS, ScanNet and KITTI-360, training and testing models and reproducing our papers' main results.
16
14
-**pretrained weights** from our best-performing model on S3DIS and KITTI-360
17
15
-**[wandb](https://wandb.ai) logs** of our experiments
18
16
@@ -22,7 +20,7 @@ The following must be installed before installing this project.
22
20
- cuda >= 10.1
23
21
- gcc >= 7
24
22
25
-
All remaining dependencies (PyTorch, PyTorch Geometric, etc) should be installed using the prodived[installation script](install.sh).
23
+
All remaining dependencies (PyTorch, PyTorch Geometric, etc) should be installed using the provided[installation script](install.sh).
26
24
27
25
The code has been tested in the following environment:
28
26
- Ubuntu 18.04.6 LTS
@@ -33,10 +31,10 @@ The code has been tested in the following environment:
33
31
## Installation :bricks:
34
32
To install DeepViewAgg, simply run `./install.sh` from inside the repository.
35
33
- You will need to have **sudo rights** to install [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine) and [TorchSparse](https://github.com/mit-han-lab/torchsparse) dependencies.
36
-
-:warning:**Do not** install Torch-Points3D from the official repository, nor from`pip`.
34
+
-:warning:**Do not** install Torch-Points3D from the official repository, or with`pip`.
37
35
38
36
## Disclaimer
39
-
This is **not the official [Torch-Points3D](https://github.com/nicolas-chaulet/torch-points3d) framework**. This work builds on and modifies a fixed version of the framework and has not been merged with the official repository yet. In particular, this repository **introduces numerous features for multimodal learning on large-scale 3D point clouds**. In this repository, some TP3D-specific files were trimmed for simplicity.
37
+
This is **not the official [Torch-Points3D](https://github.com/nicolas-chaulet/torch-points3d) framework**. This work builds on and modifies a fixed version of the framework and has not been merged with the official repository yet. In particular, this repository **introduces numerous features for multimodal learning on large-scale 3D point clouds**. In this repository, some TP3D-specific files were removed for simplicity.
40
38
41
39
## Project structure
42
40
The project follows the original [Torch-Points3D framework](https://github.com/nicolas-chaulet/torch-points3d) structure.
0 commit comments