proposal: spec: add native support for programming persistent memory in Go

[jerrinsg]

Persistent Memory is a new memory technology that allows byte-addressability at DRAM-like access speed and provides disk-like persistence. Applications using persistent memory benefit in a number of ways such as seeing improved performance and faster restart times. More details on this technology can be found at pmem.io.

This is a proposal to add native support for programming persistent memory in Go. A detailed design of our approach to add this support is described in our 2020 USENIX ATC paper go-pmem. An implementation of the above design based on Go 1.15 release is available here. In summary, adding support for natively programming persistent memory requires following capabilities to be added to Go :

• Manage a garbage-collected persistent memory heap
• Provide an interface for applications to allocate objects in persistent memory heap
• Enable applications to make crash-consistent updates to data in persistent memory
• Support applications to recover following a crash/restart

There exists libraries such as Intel PMDK that provides C and C++ developers support for persistent memory programming. Other programming languages such as Java and Python are also exploring ways to enable efficient access to persistent memory. But no language provide a native persistent memory programming support. This proposal attempts to remedy this problem by making Go the first language to completely support persistent memory.

Since adding this support involves significant changes to the language runtime and compiler, we have also prepared a design document that I will attach to this proposal.

[jerrinsg]

Design document - https://go-review.googlesource.com/c/proposal/+/284992

cc: @mohit10verma @pratapsu

[zephyrtronium]

I spend enough time explaining to new Go users why they shouldn't use println. Please don't make me explain why they shouldn't use pmake.

[DarkGhostHunter]

I spend enough time explaining to new Go users why they shouldn't use println. Please don't make me explain why they shouldn't use pmake.

Because...

[fbnz156]

Honestly, I don't like the idea of persistent memory in general, but I especially don't like the idea of it in a GC language like Go. Please don't.

[ianlancetaylor]

Is a single root sufficient? A package that is aware of persistent memory might find it useful to declare its own package-local persistent memory data structures, but it seems awkward to have to tie that into a single program root.

This is likely obvious to people familiar with persistent memory, but from reading the design doc I don't understand how a transaction can fail, or what happens if a transaction does fail. What is the purpose of the txn keyword? And why are there parentheses after txn?

Does anything prevent storing a pointer to non-persistent memory into persistent memory? Presumably doing so would lead to data corruption if the program restarts.

[gopherbot]

Change https://golang.org/cl/284992 mentions this issue: design: persistent memory support in Go

[mohit10verma]

Hi Ian,
Here are the answers:

Is a single root sufficient? A package that is aware of persistent memory might find it useful to declare its own package-local persistent memory data structures, but it seems awkward to have to tie that into a single program root.

There can be multiple roots to persistent memory and our current design allows that. Our example shows a single root, but it is possible for users to make multiple roots (we call it named objects) using the pmem package with pmem.New() function and later retrieve these roots with pmem.Get(). The slice of these named objects is pointed to by runtime.SetRoot()

This is likely obvious to people familiar with persistent memory, but from reading the design doc I don't understand how a transaction can fail, or what happens if a transaction does fail. What is the purpose of the txn keyword? And why are there parentheses after txn?

A transaction can fail for example, if a program or system crashes. The txn keyword is used to demarcate the boundaries of a transaction. There are various alternatives to do this. For example use the //go:transactional pragma or as you mentioned don’t use a parantheses after txn. This is syntactical sugar in our current design and it can change.

Does anything prevent storing a pointer to non-persistent memory into persistent memory? Presumably doing so would lead to data corruption if the program restarts.

In case there is a pointer to non-persistent memory from persistent memory, all these pointers are zeroed out when a program restarts

[ianlancetaylor]

If the language supported generics, then as the design doc notes we could implement pnew and pmake as library functions. Can txn be implemented as a library function? That is, could we have beginTransaction and endTransaction? Or does it have to be implemented in the compiler?

[davecheney]

In case there is a pointer to non-persistent memory from persistent memory, all these pointers are zeroed out when a program restarts

How is that better than pointing to a random area of memory? I guess the program blows up deferences a nil pointer rather than a pointer to random memory, but this doesn’t sound very useful if every reference type is spring loaded with nil even if the original author went to lengths to avoid creating objects with invalid references.

Also, how does persistent memory keep track of everything that is a pointer without confusing it with large numbers, floats, strings, etc?

[mohit10verma]

If the language supported generics, then as the design doc notes we could implement pnew and pmake as library functions. Can txn be implemented as a library function? That is, could we have beginTransaction and endTransaction? Or does it have to be implemented in the compiler?

Within a txn block, we instrument stores to persistent memory and inject calls to an undo log before the actual updates are made. (We have a SSA pass, similar to writeBarrier SSA pass to do this). These logs are needed to recover from crashes should they occur. Even with a library, the compiler needs to be aware of a transactional region and instrument the stores.

Alternatively, instead of a language change we can have pragmas like //go:transactional for functions.

[jerrinsg]

Hi Dave,

Thanks for the questions.

In case there is a pointer to non-persistent memory from persistent memory, all these pointers are zeroed out when a program restarts

How is that better than pointing to a random area of memory? I guess the program blows up deferences a nil pointer rather than a pointer to random memory, but this doesn’t sound very useful if every reference type is spring loaded with nil even if the original author went to lengths to avoid creating objects with invalid references.

Pointing to a random area of memory when an application restarts is much more dangerous than overwriting those pointers with nil pointers. The random area in memory could be an unmapped region of memory which upon accessing will result in the OS killing that application, or it could be a region in a span that is unused which violates the invariants kept by the garbage collector. Even worse, it could point to a valid region in memory, and the application can unitentionally corrupt such memory.
Even though zeroing is not ideal, we felt it is the right compromise we could arrive at. Application that incorrectly store such references will crash on accessing such data. Theoretically, it also facilitates a defensive style of programming where applications can check for the validity of each pointer in persistent memory before accessing such pointers.

Also, how does persistent memory keep track of everything that is a pointer without confusing it with large numbers, floats, strings, etc?

During the regular run of an application, persistent memory heap is managed by the Go runtime exactly like the volatile memory heap. They are managed in 64MB arenas, and the arena stores metadata about that region such as its heap type bitmap, span map, etc (these metadata are stored in volatile memory). In addition to this, we added code to log some of this metadata in a header region of each persistent memory arena. Two types of metadata are logged - heap type bitmaps and span table. Heap type bitmap stores the exact bitmap used by the Go runtime that is used to identify pointers in the heap. The span table stores information about each span in the arena such as its sizeclass, needzero parameter, number of pages (for large spans), etc. On an application restart, the heap type bitmap is restored as such from the persistent memory arena header to its corresponding arena metadata in volatile memory. Spans are recreated using the span table. The GC then uses the restored heap type bits to identify pointers in the heap (just like for volatile memory).

[jerrinsg]

I spend enough time explaining to new Go users why they shouldn't use println. Please don't make me explain why they shouldn't use pmake.

Not sure what is the concern you are expressing here. Language changes such as pnew/pmake are not necessary to add support for programming persistent memory. We have explored an alternative using generics where these functionalities can be provided by functions exported by a package - https://github.com/golang/proposal/blob/master/design/43810-go-pmem.md#compatibility.