Improve frustum culling of skinned meshes through per-joint bounds

[greeble-dev]

Objective

Mostly fix #4971 by adding a new option for updating skinned mesh Aabb components from joint transforms.

skinned_bounds_pr.mp4

This fixes cases where vertex positions are only modified through skinning. It doesn't fix other cases like morph targets and vertex shaders.

The PR kind of upstreams bevy_mod_skinned_aabb, but with some changes to make it simpler and more reliable.

Why A Draft?

I'm fishing for design feedback and any other concerns before I finish up the PR. The current state is functional and mostly tested. The remaining work is documentation, error handling, final testing and release notes. There also needs to be a decision on whether the new option is enabled by default in the glTF importer.

Background

If a main world entity has a Mesh3d component then it's automatically assigned an Aabb component. This is done by bevy_camera or bevy_gltf. The Aabb is used by bevy_camera for frustum culling. It can also be used by bevy_picking as an optimization, and by third party crates.

But there's a problem - the Aabb can be wrong if something changes the mesh's vertex positions after the Aabb is calculated. The most common culprits are skinning or morph targets. Vertex positions can also be changed by mutating the Mesh asset (#4294), or by vertex shaders.

The most common solution has been to disable frustum culling via the NoFrustumCulling component. This is simple, and might even be the most efficient approach for apps where meshes tend to stay on-screen. But it's annoying to implement, bad for apps where meshes are often off-screen, and it only fixes frustum culling - it doesn't help other systems that use the Aabb.

Solution

This PR adds a reliable and reasonably efficient option for updating the Aabb of a skinned mesh from its animated joint transforms. See the "How does it work" section for more detail.

The glTF loader can add skinned bounds automatically if a new GltfSkinnedMeshBoundsPolicy setting is enabled in GltfPlugin or GltfLoaderSettings:

app.add_plugins(DefaultPlugins.set(GltfPlugin {
    skinned_mesh_bounds_policy: GltfSkinnedMeshBoundsPolicy::Dynamic,
    ..default()
}))

This option is currently enabled by default, although that's open to debate.

For non-glTF cases, the user must ask Mesh to generate the skinned bounds, and then add the DynamicSkinnedMeshBounds marker component to their mesh entity.

mesh.generate_skinned_mesh_bounds()?;
let mesh_asset = mesh_assets.add(mesh);
entity.insert((Mesh3d(mesh_asset), DynamicSkinnedMeshBounds));

See the custom_skinned_mesh example for real code.

There's two downsides to the current approach:

1. It's broken by RenderAssetUsages::RENDER_WORLD. I'm assuming that Retain asset without data for RENDER_WORLD-only assets #21732 will fix this.
2. It only accounts for skinning - morph targets, vertex shaders and other things that modify vertex positions are not accounted for.

Bonus Features

GltfSkinnedMeshBoundsPolicy::NoFrustumCulling

This is a convenience for users who prefer the NoFrustumCulling workaround, but want to avoid the hassle of adding it after a glTF scene has been spawned.

app.add_plugins(DefaultPlugins.set(GltfPlugin {
    skinned_mesh_bounds_policy: GltfSkinnedMeshBoundsPolicy::NoFrustumCulling,
    ..default()
}))

Note that PR #21845 is also adding an option related to skinned mesh bounds. I'm fine if that PR lands first - I'll update this PR to include the option.

Gizmos

bevy_gizmos::SkinnedMeshBoundsGizmoPlugin can draw the per-joint AABBs.

fn toggle_skinned_mesh_bounds(mut config: ResMut<GizmoConfigStore>) {
    config.config_mut::<SkinnedMeshBoundsGizmoConfigGroup>().1.draw_all ^= true;
}

The name is debatable. It's not technically drawing the bounds of the skinned mesh - it's drawing the per-joint bounds that contribute to the bounds of the skinned mesh.

Testing

# Press `B` to show mesh bounds, 'J' to show joint bounds.
cargo run --example scene_viewer --features "free_camera" -- "assets/models/animated/Fox.glb"
cargo run --example scene_viewer --features "free_camera" -- "assets/models/SimpleSkin/SimpleSkin.gltf"

# Press `B` to show mesh bounds.
cargo run --example custom_skinned_mesh

# More complicated mesh downloaded from https://github.com/KhronosGroup/glTF-Sample-Assets/tree/main/Models/RecursiveSkeletons
cargo run --example scene_viewer --features "free_camera" -- "RecursiveSkeletons.glb"

I also hacked custom_skinned_mesh to simulate awkward cases like rotated and off-screen entities.

How Does It Work?

Click to expand

Summary

Mesh::generated_skinned_mesh_bounds calculates an AABB for each joint in the mesh - the AABB encloses all the vertices skinned to that joint. Then every frame, bevy_camera::update_skinned_mesh_bounds uses the current joint transforms to calculate an Aabb that encloses all the joint AABBs.

This approach is reliable, in that the final Aabb will always enclose the skinned vertices. But it can be larger than necessary. In practice it's tight enough to be useful, and rarely more than 50% bigger.

This approach works even with non-rigid transforms and soft skinning. If there's any doubt then I can add more detail.

Awkward Bits

There's a few issues that stop the solution being as simple and efficient as it could be.

Problem 1: Joint transforms are world-space, Aabb is entity-space.

• Ideally we'd use the world-space joint transforms to calculate a world-space Aabb, but that's not possible.
• The obvious solution is to transform the joints to entity-space, so the Aabb is directly calculated in entity-space.
  • But that means an extra matrix multiply per joint.
• This PR calculates the Aabb in world-space and then transforms it to entity-space.
  • That avoids a matrix multiply per-joint, but can increase the size of the Aabb.

Problem 2: Joint AABBs are in a surprising(?) space.

• When creating joint AABBs from a mesh, the intuitive solution would be to calculate them in joint-space.
  • Then the update just has to transform them by the world-space joint transform.
• But to calculate them in joint-space we need both the bind pose vertex positions and the bind pose joint transforms.
  • Those are in separate assets - Mesh and SkinnedMeshInverseBindposes - and those assets can be mixed and matched.
  • So we'd need to calculate a SkinnedMeshBoundsAsset for each combination of Mesh and SkinnedMeshInverseBindposes.
  • (bevy_mod_skinned_aabb uses this approach - it's slow and fragile.)
• This PR calculates joint AABBs in mesh-space (or more strictly speaking: bind pose space).
  • That can be done with just the Mesh asset.
• One downside is that the update needs an extra matrix multiply so we can go from mesh-space to world-space.
  • However, this might become a performance advantage if frustum culling changes - see the "Future Options" section.
• Another minor downside is that mesh-space AABBs (red in the screenshot below) tend to be bigger than joint-space AABBs (green), since joints with one long axis might be at an awkward angle in mesh-space.

Pseudo-Code

Here's a pseudo-code version of the update so we can compare against other options.

let mesh = ...;
let inverse_bindposes = ...;
let skinned_mesh_bounds = ...;
let skinned_mesh = ...;

let accumulator = AabbAccumulator::new();

for (joint_index, modelspace_joint_aabb) in mesh.skinned_mesh_bounds()? {
    let joint_entity = skinned_mesh.joints[joint_index];
    let world_from_joint = entities.get(joint_entity).get::<Transform>();
    let joint_from_mesh = inverse_bindposes[joint_index];
    let world_from_mesh = world_from_joint * joint_from_mesh;
    let worldspace_joint_aabb = transform_aabb(modelspace_joint_aabb, world_from_mesh);
    accumulator.add(worldspace_joint_aabb);
}
    
let worldspace_mesh_aabb = accumulator.finish();
let entity_from_world = mesh_entity.get::<Transform>().inverse();
let entityspace_mesh_aabb = transform_aabb(worldspace_mesh_aabb, entity_from_world);

*mesh_entity.get_mut::<Aabb>() = entityspace_joint_aabb;

Future Options

For frustum culling there's a cheeky way to optimize and simplify skinned bounds - put frustum culling in the renderer and calculate a world-space AABB during extract_skins. It's a good fit because skin extraction has already grabbed the joint transform and calculated world_from_mesh. I estimate this would make skinned bounds 3x faster.

let mesh = ...;
let joint_transforms = ...;

let accumulator = AabbAccumulator::new();

for (joint_index, modelspace_joint_aabb) in mesh.skinned_mesh_bounds()? {
    let world_from_mesh = joint_transforms[joint_index];
    let worldspace_joint_aabb = transform_aabb(modelspace_joint_aabb, world_from_mesh);
    accumulator.add(worldspace_joint_aabb);
}

let worldspace_mesh_aabb = accumulator.finish();

Another option is to change main world frustum culling to use a world-space AABB. So there would be a new GlobalAabb component that gets updated each frame from Aabb and the entity transform (which is basically the same as transform propagation and the relationship between Transform and GlobalTransform). This has some advantages and disadvantages but I won't get into them here - I think putting frustum culling into the renderer is a better option.

(Note that putting frustum culling into the renderer doesn't mean removing the current main world visibility system - it just means the main world system would be separate opt-in system)

Performance

Click to expand

Initialization

Creating the skinned bounds asset for Fox.glb (576 verts, 22 skinned joints) takes 0.03ms. Loading the whole glTF takes 8.7ms, so this is a <1% increase.

Per-Frame

The many_foxes example has 1000 skinned meshes, each with 22 skinned joints. Updating the skinned bounds takes 0.086ms. This is a throughput of roughly 250,000 joints per millisecond, using two threads.

The whole animation update takes 3.67ms (where "animation update" = advancing players + graph evaluation + transform propagation). So we can kinda sorta claim that this PR increases the cost of skinned animation by roughly 3%. But that's very hand-wavey and situation dependent.

This was tested on an AMD Ryzen 7900 but with TaskPoolOptions::with_num_threads(6) to simulate a lower spec CPU. Comparing against a few other threading options:

• Non-parallel: 0.141ms.
• 6 threads (2 compute threads): 0.086ms.
• 24 threads (15 compute threads): 0.051ms.

So the parallel iterator is better but quickly hits diminishing returns as the number of threads increases.

Future Options

The "How Does It Work" section mentions moving skinned mesh bounds into the renderer's skin extraction. Based on some microbenchmarks, I estimate this would reduce non-parallel many_foxes from 0.141ms to 0.049ms, so roughly 3x faster. Requiring AVX2 (to enable broadcast loads) or pre-splatting (to fake broadcast loads for SSE) would knock off another 25%. And fancier SIMD approaches could do better again.

There's also approaches that trade reliability for performance. For character rigs, an effective optimization is to fold face and finger joints into a single bound on the head and hand joints. This can reduce the number of joints required by 50-80%. Another option is to use bounding spheres instead of AABBs.

FAQ

Click to expand

Should the glTF importer enable this feature by default?

I think so. Bugs caused by skinned mesh culling have been a regular pain for both new and experienced users. I'm guessing most people would pay the CPU cost to have skinned meshes Just Work(tm). Sophisticated users in search of performance can choose to disable the option and use a fixed AABB.

Why can't it be automatically added to any mesh? Then the glTF importer and custom mesh generators wouldn't need special logic.

bevy_mod_skinned_aabb took the automatic approach, and I don't think the outcome was good. It needs some surprisingly fiddly and fragile logic to decide when an entity has the right combination of assets in the right loaded state. And it can never work with RenderAssetUsages::RENDER_WORLD.

So this PR takes a more modest and manual approach. I think there's plenty of scope to generalise and automate as the asset pipeline matures. If the glTF importer becomes a purer glTF -> BSN transform, then adding skinned bounds could be a general scene/asset transform that's shared with other importers and custom mesh generators.

Why is the data in Mesh? Shouldn't it go in SkinnedMesh or SkinnedMeshInverseBindposes?

That might seem intuitive, but it wouldn't work in practice - the data is derived from Mesh alone. SkinnedMesh doesn't work because it's per mesh instance, so the data would be duplicated. SkinnedMeshInverseBindposes doesn't work because it can be shared between multiple meshes.

The names are a bit misleading - Mesh contains the per-vertex skinning data, while SkinnedMesh and SkinnedMeshInverseBindposes are more like joint bindings one step removed from the vertex data.

Why not put the bounds on the joint entities?

This is surprisingly tricky in practice because multiple meshes can be bound to the same joint entity. So there would need to be logic that tracks the bindings and updates the bounds as meshes are added and removed.

Why SkinnedMeshBounds? Shouldn't it be SkinnedMeshAabbs?

Playing it safe in case things change, e.g. bounding spheres might become an an option.

Why is the bounds update system in bevy_camera? Shouldn't it be in bevy_mesh?

bevy_camera is the owner and main user of Aabb, and already has some mesh related logic (calculate_bounds automatically adds an Aabb to mesh entities). So putting it in bevy_camera is consistent with the current structure, but that could change.

What Do Other Engines Do?

Click to expand

• Unreal: Automatically uses collision shapes attached to joints, which is similar to this PR but fragile and inefficient. Also supports various fixed bounds options.
• Unity: Fixed bounds attached to the root bone. Automatically calculated from animation poses or specified manually (documentation).
• Godot: Unclear. There's an issue that suggests they only support fixed bounds (AABB broken on imported meshes with skeletons. godotengine/godot#85171), but also code in the engine related to per-joint AABBs (MeshStorage::mesh_get_aabb). I don't know what they're using for frustum culling.
• O3DE: Fixed bounds attached to root bone, plus option to approximate the AABB from joint origins and a fudge factor.

An approach that's been proposed several times for Bevy is copying Unity's "fixed AABB from animation poses". I think this is more complicated and less reliable than many people expect. More complicated because linking animations to meshes can often be difficult. Less reliable because it doesn't account for ragdolls and procedural animation. But it could still be viable for for simple cases like a single self-contained glTF with basic animation.

[janhohenheim]

I very much like the approach presented here and think it's a very reasonable default.

The policy enum should be expanded to include @pcwalton's approach: don't generate any AABB, but keep frustum culling, since all AABBs will be artist-authored and inserted manually

[pcwalton]

Oh nice, I just filed #21845 for the rest pose approach.

[greeble-dev]

The policy enum should be expanded to include @pcwalton's approach: don't generate any AABB, but keep frustum culling, since all AABBs will be artist-authored and inserted manually

I'd like to consider some different approaches:

enum GltfSkinnedMeshBoundsPolicy {
    ...
    /// Skinned meshes are assigned this `Aabb` component.
    Fixed { aabb: Aabb },
    /// Skinned meshes are assigned a `SkinnedMeshBounds` component, which will
    /// be used by the `bevy_camera` plugin to update the `Aabb` component.
    ///
    /// The `Aabb` component will be updated to enclose the given AABB relative to the named joint.
    /// If the joint name is `None`, the AABB is relative to the root joint.
    FixedJointRelative { aabb: Aabb3d, joint_name: Option<String> },
}

So Fixed is similar to Unreal's fixed options. FixedJointRelative is basically Unity's approach but more flexible (can be any joint, not just the root). Unfortunately this needs some changes to SkinnedMeshBounds to support bounds being in joint-space.

I'm not sure if I'll try adding these options in this PR as it's already a bit chonky.

[robtfm]

awesome work, really happy to see a solution to such a long-standing issue.

• Dynamic should 100% be the default. I would lean towards replacing NoFrustumCulling with CullingStrategy: Static | Dynamic | None (and possibly other variants)
• joint space / world space / mesh space: would it make sense to consider bone bounding spheres? this would make the transformations very cheap. this could be an extra variant for CullingStrategy above then. considering the aabbs are crossed between spaces anyway, the benefit of using aabbs over spheres is probably minimal?
• separate asset: i did read the FAQ but i still think this should be stored in the mesh
  • I have a pr (Retain asset without data for RENDER_WORLD-only assets #21732) that would remove the RenderAssetUsages::RENDER_WORLD problem (we would keep joint bounds data cpu-side even after the vertex data is extracted - that doesn't need any skin data to compute, just the positions and joint weights from the vertex data)
  • that leaves the issue of having to look up the mesh to perform updates. that could maybe be improved by adding bounding sphere / aabb data directly to the joint entities when they are spawned? i haven't thought through the space issues here but it seems clean / logical
  • we could then auto-apply for non-gltf meshes as well

[greeble-dev]

joint space / world space / mesh space: would it make sense to consider bone bounding spheres? this would make the transformations very cheap. this could be an extra variant for CullingStrategy above then. considering the aabbs are crossed between spaces anyway, the benefit of using aabbs over spheres is probably minimal?

I briefly looked at this, but concluded that joint transforms being affine kinda spoils things - bounding spheres would end up only slightly cheaper. But I didn't work it through in detail or benchmark.

There could also be an option where scale is ignored and the bounding sphere is always at zero in joint-space. That would be significantly cheaper but requires a sophisticated user to understand the risks.

separate asset: i did read the FAQ but i still think this should be stored in the mesh. I have a pr that would remove the RenderAssetUsages::RENDER_WORLD problem

Sounds good to me. I think I'd end up adding a HasDynamicSkinnedMeshBounds marker component so the update only touches meshes with dynamic bounds.

that leaves the issue of having to look up the mesh to perform updates. that could maybe be improved by adding bounding sphere / aabb data directly to the joint entities when they are spawned? i haven't thought through the space issues here but it seems clean / logical

The catch there is that multiple meshes can bind to one joint, so whatever's setting up the joints has to be careful to track the exact set of meshes that are bound to each one and merge the AABBs. This is fairly straightforward if done once in the glTF importer and the user leaves it untouched. But rapidly gets messy if the user starts changing stuff - e.g. swapping clothes/accessories on a character.

[greeble-dev]

So my current state is:

• There's been enough feedback that I feel confident finishing up the PR.
  • This would include dropping the separate asset and moving the data back into Mesh (unless I find a blocker).
  • So I'll wait until Retain asset without data for RENDER_WORLD-only assets #21732 lands.
• I'll try waiting until Take the rest pose into account when computing the AABBs of skinned meshes in glTF files. #21845 lands, but might go ahead if that stalls for some reason.

[robtfm]

if you're eager or 21732 stalls you could go ahead with the mesh embedding knowing it will only work for MAIN_WORLD assets initially. i can amend that to store the joint bounds as well if this lands first.