migrate to ValueMesh accumulator #1486

shayne-fletcher · 2025-10-10T03:15:36Z

Summary: this change replaces the old RankedValues<Status> path with ValueOverlay<Status> updates applied into a ValueMesh<Status> snapshot. GetRankStatus.reply now carries overlays, the comm reducer merges overlays, and the accumulator applies them into a full StatusMesh (seeded to NotExist via a template and ValueMesh::default() + first-update install). agents emit a single-rank overlay (or empty when rank unknown). wait() consumes snapshots and checks completion against NotExist. legacy shims convert StatusMesh back to RankedValues only where error types still demand it. removed the RankedValues accumulator/reducer and its registration. the old RankedValues accumulator and reducer definitions are removed.

Differential Revision: D84318425

meta-codesync · 2025-10-10T03:16:02Z

@shayne-fletcher has exported this pull request. If you are a Meta employee, you can view the originating Diff in D84318425.

Summary: Pull Request resolved: meta-pytorch#1486 his change replaces the old `RankedValues<Status>` path with `ValueOverlay<Status>` updates reduced into a `ValueMesh<Status>` snapshot. `GetRankStatus.reply` now carries overlays; the comm reducer merges them (right-wins) and the accumulator applies them into a full `StatusMesh`, seeded to `NotExist` via the accumulator template (`StatusMesh::from_single(region, NotExist)`) and `ValueMesh::default()` + first-update install. agents emit a single-rank overlay (or an empty overlay when the rank is unknown). `wait()` consumes snapshots, declares completion when no `NotExist` remain for the target ranks, and on timeout returns the last snapshot (falling back to the seed if nothing arrived). termination detection now checks `values().any(is_terminating)`. legacy shims convert `StatusMesh` back to `RankedValues` only where error types still require it. removed the `RankedValues` accumulator/reducer and its registration. Differential Revision: D84318425

Summary: his change replaces the old `RankedValues<Status>` path with `ValueOverlay<Status>` updates reduced into a `ValueMesh<Status>` snapshot. `GetRankStatus.reply` now carries overlays; the comm reducer merges them (right-wins) and the accumulator applies them into a full `StatusMesh`, seeded to `NotExist` via the accumulator template (`StatusMesh::from_single(region, NotExist)`) and `ValueMesh::default()` + first-update install. agents emit a single-rank overlay (or an empty overlay when the rank is unknown). `wait()` consumes snapshots, declares completion when no `NotExist` remain for the target ranks, and on timeout returns the last snapshot (falling back to the seed if nothing arrived). termination detection now checks `values().any(is_terminating)`. legacy shims convert `StatusMesh` back to `RankedValues` only where error types still require it. removed the `RankedValues` accumulator/reducer and its registration. Differential Revision: D84318425

Summary: his change replaces the old `RankedValues<Status>` path with `ValueOverlay<Status>` updates reduced into a `ValueMesh<Status>` snapshot. `GetRankStatus.reply` now carries overlays; the comm reducer merges them (right-wins) and the accumulator applies them into a full `StatusMesh`, seeded to `NotExist` via the accumulator template (`StatusMesh::from_single(region, NotExist)`) and `ValueMesh::default()` + first-update install. agents emit a single-rank overlay (or an empty overlay when the rank is unknown). `wait()` consumes snapshots, declares completion when no `NotExist` remain for the target ranks, and on timeout returns the last snapshot (falling back to the seed if nothing arrived). termination detection now checks `values().any(is_terminating)`. legacy shims convert `StatusMesh` back to `RankedValues` only where error types still require it. removed the `RankedValues` accumulator/reducer and its registration. Reviewed By: pablorfb-meta Differential Revision: D84318425

Summary: this diff introduces a v0 basic run-length-encoded (RLE) internal representation for `ValueMesh<T>` and updates all APIs to operate transparently over either dense or compressed data. the compression support is purely additive. public semantics are unchanged: a mesh still represents a complete mapping from rank -> value, and iteration, slicing, and region order behave identically (note: complexity of `get()` is O(log k) in compressed mode vs O(1) dense). the compressed form is lossless and idempotent. compression is manual in rust because automatic detection isn't possible in a fully generic type. i originally considered doing it automatically, but rust provides no specialization or reflection to determine whether a given `T` has meaningful equality semantics. for many types (e.g. futures, closures) equality doesn't even exist. because of that, compression must be explicitly invoked via `compress_adjacent_in_place()` or `compress_adjacent_in_place_by(pred)` when the caller knows adjacent elements can be merged. in python, compression happens automatically on construction. for `Py<PyAny>`, equality is defined by pointer identity (`a.as_ptr() == b.as_ptr()`), so adjacent references to the same python object are coalesced into RLE runs. this will produce savings for sentinel-rich, categorical, or boolean data (e.g. repeated `None`, booleans, cpython-interned small integers and strings) but will have little effect for freshly allocated numbers or dynamic objects. Differential Revision: D84169361

Summary: this change enables serde serialization and deserialization for `ValueMesh<T>` and its internal representations. it introduces a stable wire format for run-length encoded (RLE) meshes by defining a `Run` struct (`u64` bounds, `u32` id) to avoid platform-dependent `usize` encoding. both dense and compressed representations serialize deterministically and retain their form on round trips Reviewed By: dulinriley Differential Revision: D84197759

Summary: add a set of `ValueMesh` constructors equivalent to the `RankedValues` initialization paths. `from_single` and `from_default` create compressed single-run meshes without allocating a dense vector, `from_ranges_with_default` builds a compact RLE mesh from sparse overrides on a default value, and `from_dense` constructs from a dense vector and compresses adjacent equal elements. includes tests for correctness, edge cases, and internal structure. completes the construction half of the `ValueMesh` parity plan before adding `ValueOverlay` and merge support. Differential Revision: D84259628

Summary: introduces `ValueOverlay`, a sparse representation of `(Range<usize>, T) `runs for assembling or patching `ValueMesh` instances without materializing per-rank data. includes structural validation (`EmptyRange`, `OverlappingRanges`), coalescing of equal-adjacent runs, and construction helpers (`push_run`, `try_from_runs`, `normalize`). adds unit tests covering append, coalescing, overlap detection, unsorted inserts, and empty overlays. no behavior change to `ValueMesh`; merge support will follow in the next diff. Differential Revision: D84266477

Summary: this change factors out RLE and merge logic into a new `rle` module and ports the merge algorithm from `RankedValues::merge_from` into a general `merge_value_runs` that operates over normalized `(Range, T)` lists. the function implements last-writer-wins semantics and handles equal-value coalescing, disjoint runs, and arbitrary overlap spans. `ValueMesh` now uses these primitives for compression (`rle_from_dense`, `rle_from_value_runs`) and sparse overlay updates via `merge_from_overlay,` with `materialized_runs()` added for inspection. comprehensive tests cover dense compression, value-run merging, and overlay application, including multidimensional regions. Differential Revision: D84285073

Summary: adds an accumulator and reducer for `ValueOverlay<T>` with right-wins merge semantics via merge_value_runs. `ValueOverlay<T>` now derives `Named` and `Default`, and `BuildError` implements `Display` and `std::error::Error`. registers a reducer factory for `ValueOverlay<Status>` to enable accumulation of sparse updates into `ValueMesh<Status>`. defines `StatusMesh` and `StatusOverlay` type aliases for per-rank lifecycle tracking. Differential Revision: D84301230

Summary: his change replaces the old `RankedValues<Status>` path with `ValueOverlay<Status>` updates reduced into a `ValueMesh<Status>` snapshot. `GetRankStatus.reply` now carries overlays; the comm reducer merges them (right-wins) and the accumulator applies them into a full `StatusMesh`, seeded to `NotExist` via the accumulator template (`StatusMesh::from_single(region, NotExist)`) and `ValueMesh::default()` + first-update install. agents emit a single-rank overlay (or an empty overlay when the rank is unknown). `wait()` consumes snapshots, declares completion when no `NotExist` remain for the target ranks, and on timeout returns the last snapshot (falling back to the seed if nothing arrived). termination detection now checks `values().any(is_terminating)`. legacy shims convert `StatusMesh` back to `RankedValues` only where error types still require it. removed the `RankedValues` accumulator/reducer and its registration. Reviewed By: pablorfb-meta Differential Revision: D84318425

meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 10, 2025

meta-codesync bot added fb-exported meta-exported labels Oct 10, 2025

shayne-fletcher force-pushed the export-D84318425 branch 2 times, most recently from 2526a5b to 216826b Compare October 10, 2025 03:35

shayne-fletcher force-pushed the export-D84318425 branch 2 times, most recently from 9ec49d8 to a43ea95 Compare October 10, 2025 03:44

shayne-fletcher force-pushed the export-D84318425 branch from a43ea95 to 400e71e Compare October 10, 2025 05:17

shayne-fletcher force-pushed the export-D84318425 branch from 400e71e to 0a0b1b3 Compare October 12, 2025 16:12

shayne-fletcher force-pushed the export-D84318425 branch from 0a0b1b3 to 8607788 Compare October 12, 2025 16:12

shayne-fletcher force-pushed the export-D84318425 branch from 8607788 to a62204d Compare October 13, 2025 17:05

shayne-fletcher force-pushed the export-D84318425 branch from a62204d to 7dc5120 Compare October 13, 2025 17:05

shayne-fletcher added 7 commits October 14, 2025 17:17

shayne-fletcher force-pushed the export-D84318425 branch 2 times, most recently from e30964b to e96bd6e Compare October 15, 2025 00:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

migrate to ValueMesh accumulator #1486

migrate to ValueMesh accumulator #1486

Uh oh!

shayne-fletcher commented Oct 10, 2025

Uh oh!

meta-codesync bot commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

migrate to ValueMesh accumulator #1486

Are you sure you want to change the base?

migrate to ValueMesh accumulator #1486

Uh oh!

Conversation

shayne-fletcher commented Oct 10, 2025

Uh oh!

meta-codesync bot commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant