[vm, gc] Mark through new-space.

 - Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
 - A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
 - Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
 - Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
 - reachabilityBarrier is now just a count of major GCs.

TEST=ci
Change-Id: I8c2c64b120766571b62d3bd8dab37ae81c2dca98
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/319583
Commit-Queue: Ryan Macnak <[email protected]>
Reviewed-by: Siva Annamalai <[email protected]>

Author: rmacnak-google

runtime/docs/gc.md

@@ -57,17 +57,9 @@ FLAG_scavenger_tasks (default 2) workers are started on separate threads. Each w
 
 All objects have a bit in their header called the mark bit. At the start of a collection cycle, all objects have this bit clear.
 
-During the marking phase, the collector visits each of the root pointers. If the target object is an old-space object and its mark bit is clear, the mark bit is set and the target added to the marking stack (grey set). The collector then removes and visits objects in the marking stack, marking more old-space objects and adding them to the marking stack, until the marking stack is empty. At this point, all reachable objects have their mark bits set and all unreachable objects have their mark bits clear.
+During the marking phase, the collector visits each of the root pointers. If the target object has its mark bit clear, the mark bit is set and the target added to the marking stack (grey set). The collector then removes and visits objects in the marking stack, marking more objects and adding them to the marking stack, until the marking stack is empty. At this point, all reachable objects have their mark bits set and all unreachable objects have their mark bits clear.
 
-During the sweeping phase, the collector visits each old-space object. If the mark bit is clear, the object's memory is added to a [free list](https://github.com/dart-lang/sdk/blob/main/runtime/vm/heap/freelist.h) to be used for future allocations. Otherwise the object's mark bit is cleared. If every object on some page is unreachable, the page is released to the OS.
-
-### New-Space as Roots
-
-We do not mark new-space objects, and pointers to new-space objects are ignored; instead all objects in new-space are treated as part of the root set.
-
-This has the advantage of making collections of the two spaces more independent. In particular, the concurrent marker never needs to dereference any memory in new-space, avoiding several data race issues, and avoiding the need to pause or otherwise synchronize with the concurrent marker when starting a scavenge.
-
-It has the disadvantage that no single collection will collect all garbage. An unreachable old-space object that is referenced by an unreachable new-space object will not be collected until a scavenge first collects the new-space object, and unreachable objects that have a generation-crossing cycle will not be collected until the whole subgraph is promoted into old-space. The growth policy must be careful to ensure it doesn't perform old-space collections without interleaving new-space collections, such as when the program performs mostly large allocation that go directly to old-space, or old-space can accumulate such floating garbage and grow without bound.
+During the sweeping phase, the collector visits each object. If the mark bit is clear, the object's memory is added to a [free list](https://github.com/dart-lang/sdk/blob/main/runtime/vm/heap/freelist.h) to be used for future allocations. Otherwise the object's mark bit is cleared. If every object on some page is unreachable, the page is released to the OS.
 
 ## Mark-Compact
 
@@ -103,17 +95,17 @@ But we combine the generational and incremental checks with a shift-and-mask.
 ```c++
 enum HeaderBits {
   ...
-  kOldAndNotMarkedBit,      // Incremental barrier target.
+  kNotMarkedBit,            // Incremental barrier target.
   kNewBit,                  // Generational barrier target.
-  kOldBit,                  // Incremental barrier source.
+  kAlwaysSetBit,            // Incremental barrier source.
   kOldAndNotRememberedBit,  // Generational barrier source.
   ...
 };
 
 static constexpr intptr_t kGenerationalBarrierMask = 1 << kNewBit;
-static constexpr intptr_t kIncrementalBarrierMask = 1 << kOldAndNotMarkedBit;
+static constexpr intptr_t kIncrementalBarrierMask = 1 << kNotMarkedBit;
 static constexpr intptr_t kBarrierOverlapShift = 2;
-COMPILE_ASSERT(kOldAndNotMarkedBit + kBarrierOverlapShift == kOldBit);
+COMPILE_ASSERT(kNotMarkedBit + kBarrierOverlapShift == kAlwaysSetBit);
 COMPILE_ASSERT(kNewBit + kBarrierOverlapShift == kOldAndNotRememberedBit);
 
 StorePointer(ObjectPtr source, ObjectPtr* slot, ObjectPtr target) {

runtime/platform/atomic.h

@@ -146,6 +146,27 @@ class AcqRelAtomic {
   std::atomic<T> value_;
 };
 
+template <typename T>
+static inline T LoadRelaxed(const T* ptr) {
+  static_assert(sizeof(std::atomic<T>) == sizeof(T));
+  return reinterpret_cast<const std::atomic<T>*>(ptr)->load(
+      std::memory_order_relaxed);
+}
+
+template <typename T>
+static inline T LoadAcquire(const T* ptr) {
+  static_assert(sizeof(std::atomic<T>) == sizeof(T));
+  return reinterpret_cast<const std::atomic<T>*>(ptr)->load(
+      std::memory_order_acquire);
+}
+
+template <typename T>
+static inline void StoreRelease(T* ptr, T value) {
+  static_assert(sizeof(std::atomic<T>) == sizeof(T));
+  reinterpret_cast<std::atomic<T>*>(ptr)->store(value,
+                                                std::memory_order_release);
+}
+
 }  // namespace dart
 
 #endif  // RUNTIME_PLATFORM_ATOMIC_H_

runtime/vm/app_snapshot.cc

@@ -875,8 +875,8 @@ void Deserializer::InitializeHeader(ObjectPtr raw,
   tags = UntaggedObject::ClassIdTag::update(class_id, tags);
   tags = UntaggedObject::SizeTag::update(size, tags);
   tags = UntaggedObject::CanonicalBit::update(is_canonical, tags);
-  tags = UntaggedObject::OldBit::update(true, tags);
-  tags = UntaggedObject::OldAndNotMarkedBit::update(true, tags);
+  tags = UntaggedObject::AlwaysSetBit::update(true, tags);
+  tags = UntaggedObject::NotMarkedBit::update(true, tags);
   tags = UntaggedObject::OldAndNotRememberedBit::update(true, tags);
   tags = UntaggedObject::NewBit::update(false, tags);
   raw->untag()->tags_ = tags;

runtime/vm/compiler/runtime_api.cc

@@ -360,6 +360,8 @@ uword MakeTagWordForNewSpaceObject(classid_t cid, uword instance_size) {
              TranslateOffsetInWordsToHost(instance_size)) |
          dart::UntaggedObject::ClassIdTag::encode(cid) |
          dart::UntaggedObject::NewBit::encode(true) |
+         dart::UntaggedObject::AlwaysSetBit::encode(true) |
+         dart::UntaggedObject::NotMarkedBit::encode(true) |
          dart::UntaggedObject::ImmutableBit::encode(
              ShouldHaveImmutabilityBitSet(cid));
 }
@@ -378,8 +380,7 @@ const word UntaggedObject::kNewBit = dart::UntaggedObject::kNewBit;
 const word UntaggedObject::kOldAndNotRememberedBit =
     dart::UntaggedObject::kOldAndNotRememberedBit;
 
-const word UntaggedObject::kOldAndNotMarkedBit =
-    dart::UntaggedObject::kOldAndNotMarkedBit;
+const word UntaggedObject::kNotMarkedBit = dart::UntaggedObject::kNotMarkedBit;
 
 const word UntaggedObject::kImmutableBit = dart::UntaggedObject::kImmutableBit;
 

runtime/vm/compiler/runtime_api.h

@@ -418,7 +418,7 @@ class UntaggedObject : public AllStatic {
   static const word kCanonicalBit;
   static const word kNewBit;
   static const word kOldAndNotRememberedBit;
-  static const word kOldAndNotMarkedBit;
+  static const word kNotMarkedBit;
   static const word kImmutableBit;
   static const word kSizeTagPos;
   static const word kSizeTagSize;
@@ -1493,6 +1493,8 @@ class Page : public AllStatic {
   static const word kBytesPerCardLog2;
 
   static word card_table_offset();
+  static word original_top_offset();
+  static word original_end_offset();
 };
 
 class Heap : public AllStatic {