Errors in NeRF implementation

[rakhimovv]

Hello, first of all, thanks for the beautiful implementation!

However, certain things seem to be buggy. Please correct me if I am wrong:

Problem 1.

NDCGridRaysampler does not work properly with rectangular images. Since the NDC convention assumes that the XY coordinates lie in [-1, 1] x [u, u] or ([-u, u] x [-1, 1]) depending on which size is shorter.

The simple fix I expect would be:

from pytorch3d.renderer.mesh.rasterize_meshes import pix_to_non_square_ndc
min_x = pix_to_non_square_ndc(image_width - 1, image_width, image_height)
max_x = pix_to_non_square_ndc(0, image_width, image_height)
min_y = pix_to_non_square_ndc(image_height - 1, image_height, image_width)
max_y = pix_to_non_square_ndc(0, image_height, image_width)

A similar problem also persists here when passing arguments:

pytorch3d/projects/nerf/nerf/raysampler.py

Lines 162 to 166 in 9585a58

	self._mc_raysampler = MonteCarloRaysampler(
	min_x=-1.0,
	max_x=1.0,
	min_y=-1.0,
	max_y=1.0,

If the fix is applied, then grid_sample should be fixed to account for it (the longer side should be divided by u)

pytorch3d/projects/nerf/nerf/utils.py

Line 51 in 9585a58

xy_sample = -sampled_rays_xy.view(ba, -1, 1, 2).clone()

also, align_corners should be False since the in NDC convention, the pixel location corresponds to its center

pytorch3d/projects/nerf/nerf/utils.py

Lines 53 to 58 in 6d36c1e

	images_sampled = torch.nn.functional.grid_sample(
	target_images.permute(0, 3, 1, 2),
	xy_sample,
	align_corners=True,
	mode="bilinear",
	)

Problem 2.

deltas in _get_densities do not take into account the norm of ray_directions

pytorch3d/projects/nerf/nerf/implicit_function.py

Lines 109 to 115 in 9585a58

	deltas = torch.cat(
	(
	depth_values[..., 1:] - depth_values[..., :-1],
	1e10 * torch.ones_like(depth_values[..., :1]),
	),
	dim=-1,
	)[..., None]

this does not lead to error since the ray_directions were normalized in NeRFRaysampler

but this leads to another problem: since the ray_directions were normalized, ray_bundle_to_ray_points will now produce points that lie before the near plane if, e.g., I pass ray_lengths = near (so the depth values are not actually depths)

it seems to me it is better to avoid ray_directions normalization and take into account the norm of ray_directions when calculating densities like it is done in an original implementation https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/run_nerf.py#L123

Problem 3.

NeRFRaysampler does not correctly work with batch_size > 1

e.g., here

pytorch3d/projects/nerf/nerf/raysampler.py

Lines 349 to 357 in 8fa438c

	# Take the "sel_rays" rays from the full ray bundle.
	ray_bundle = RayBundle(
	*[
	v.view(n_pixels, -1)[sel_rays]
	.view(batch_size, sel_rays.numel() // batch_size, -1)
	.to(device)
	for v in full_ray_bundle
	]
	)

causes the case when the rays originally pointing to the different batch idx will now point to the same batch_idx

also, some index bound errors would be here:

pytorch3d/projects/nerf/nerf/raysampler.py

Line 329 in 8fa438c

n_pixels = full_ray_bundle.directions.shape[:-1].numel()

pytorch3d/projects/nerf/nerf/raysampler.py

Lines 338 to 347 in 8fa438c

	if chunksize is None:
	chunksize = n_pixels * batch_size
	start = chunk_idx * chunksize * batch_size
	end = min(start + chunksize, n_pixels)
	sel_rays = torch.arange(
	start,
	end,
	dtype=torch.long,
	device=full_ray_bundle.lengths.device,
	)

here, e.g., also no batch dimension in data actually going to the function

pytorch3d/projects/nerf/nerf/nerf_renderer.py

Lines 293 to 294 in 8fa438c

	camera: A batch of cameras from which the scene is rendered.
	image: A batch of corresponding ground truth images of shape

pytorch3d/projects/nerf/nerf/nerf_renderer.py

Lines 210 to 211 in 8fa438c

	camera: A batch of cameras from which the scene is rendered.
	image: A batch of corresponding ground truth images of shape

[rakhimovv]

up

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[rakhimovv]

up

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[sergeyprokudin]

Hi folks, NeRF implementation for non-square images indeed seems to be the problem. E.g. I cannot reproduce 'fern' dataset results, as also mentioned here by other researchers. Any plans on investigating the issues any time soon?..

Again, thanks for the great repo!