Reserve/quarantine address range for contiguous spaces

[qinsoon]

Currently we eagerly mmap the entire address range for contiguous spaces, and the address range for their side metadata (if side metadata is used). We also eagerly initialize the SFT map for the chunks.

We should instead only reserve/quarantine the address range (for spaces and the side metadata). We mmap the pages and update the SFT map when we use the pages.

[qinsoon]

Our requirements are:

1. Our mapped memory should not be replaced by others (if they are sane), and we should not replace others' mapped memory.
2. Our implementation should allow the OS to show the real memory used (RSS). If we reserve memory but do not use them, the OS should not show their usage.

For 1, we need to replace our dzmmap() uses with dzmmap_noreplace(). For 2, we are fine for linux, as Linux supports overcommitting and we do not need to separate reserve/commit operations [1]. However, for other OS/s, we may still need to do a refactoring of separating reserve vs. commit.

If we want to separate reserve vs commit, for Linux, we have two options. 1) reserve = mmap, commit = nop, or 1) reserve = mmap with PROT_NONE, commit = mprotect [2].

[1] https://www.gamasutra.com/blogs/NiklasGray/20171107/309071/Virtual_Memory_Tricks.php (keyword: overcommitting)
[2] https://mjmwired.net/kernel/Documentation/vm/overcommit-accounting#57

[qinsoon]

Another option is: reserve = mmap_noreserve, commit = mmap (basically replace the previous reserved memory). Is this the way you did for side metadata? @javadamiri

[javadamiri]

Another option is: reserve = mmap_noreserve, commit = mmap (basically replace the previous reserved memory). Is this the way you did for side metadata? @javadamiri

Yes, that's how it's being done for side metadata.

[javadamiri]

Our requirements are:

1. Our mapped memory should not be replaced by others (if they are sane), and we should not replace others' mapped memory.
2. Our implementation should allow the OS to show the real memory used (RSS). If we reserve memory but do not use them, the OS should not show their usage.

For 1, we need to replace our dzmmap() uses with dzmmap_noreplace(). For 2, we are fine for linux, as Linux supports overcommitting and we do not need to separate reserve/commit operations [1]. However, for other OS/s, we may still need to do a refactoring of separating reserve vs. commit.

If we want to separate reserve vs commit, for Linux, we have two options. 1) reserve = mmap, commit = nop, or 1) reserve = mmap with PROT_NONE, commit = mprotect [2].

[1] https://www.gamasutra.com/blogs/NiklasGray/20171107/309071/Virtual_Memory_Tricks.php (keyword: overcommitting)
[2] https://mjmwired.net/kernel/Documentation/vm/overcommit-accounting#57

A problem in your first approach (reserve = mmap, commit = nop) is that accessing non-committed memory will not produce an error.

[qinsoon]

Another option is: reserve = mmap_noreserve, commit = mmap (basically replace the previous reserved memory). Is this the way you did for side metadata? @javadamiri

This won't produce an error either. Without mmap_noreserve(), this test gives a segfault.

    #[test]
    fn test_mmap_noreserve() {
        serial_test(|| {
            let res = mmap_noreserve(HEAP_START, BYTES_IN_PAGE);
            assert!(res.is_ok());
            unsafe { HEAP_START.store(42usize); }
        })
    }

I will test all the options and see if they meet the requirements mentioned above.

[javadamiri]

Right, it seems there is no guarantee that a write segfaults when we use the MAP_NORESERVE flag.
The documentation says:

When swap space is not reserved one might get SIGSEGV upon a write if no physical memory is available.

So, it may not exactly do what I thought it would.
Probably I should have changed the mmap protocol from libc::PROT_READ | libc::PROT_WRITE to libc::PROT_NONE.

[qinsoon]

Right, it seems there is no guarantee that a write segfaults when we use the MAP_NORESERVE flag.
The documentation says:

When swap space is not reserved one might get SIGSEGV upon a write if no physical memory is available.

So, it may not exactly do what I thought it would.
Probably I should have changed the mmap protocol from libc::PROT_READ | libc::PROT_WRITE to libc::PROT_NONE.

Yeah. I will test the 3 options quickly, and post my finds here.

[qinsoon]

I have tested the options. It seems all of them work similarly. For my test program (code at the end), the outputs are similar for the three options (see below). The virtual memory usage increases after reserving, but actual used memory only increases after actual writes happen for the memory (i.e. commit does nothing in terms of memory usage).

Initial: size = 1128, resident = 197
Reserve: size = 245269, resident = 197
Commit: size = 245269, resident = 391
Actually writes: size = 245269, resident = 61374
Release: size = 1128, resident = 391

size (1) total program size (same as VmSize in /proc/[pid]/status)
resident (2) resident set size (same as VmRSS in /proc/[pid]/status)

However, one difference is that for Option 3 (reserve = mmap_noreserve, commit = mmap), we can reserve a large amount of memory, but for the other two, we may fail for OOM.

My test code:

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/mman.h>

// 1GB
long size = 1000000000;

// Keep the following line to measure performance
// #define PERF

typedef struct {
    unsigned long size,resident,share,text,lib,data,dt;
} ProcStatm;
/* https://stackoverflow.com/questions/1558402/memory-usage-of-current-process-in-c/7212248#7212248 */
void ProcStat_init(ProcStatm *result) {
    const char* statm_path = "/proc/self/statm";
    FILE *f = fopen(statm_path, "r");
    if(!f) {
        perror(statm_path);
        abort();
    }
    if(7 != fscanf(
        f,
        "%lu %lu %lu %lu %lu %lu %lu",
        &(result->size),
        &(result->resident),
        &(result->share),
        &(result->text),
        &(result->lib),
        &(result->data),
        &(result->dt)
    )) {
        perror(statm_path);
        abort();
    }
    fclose(f);
}
void check_mem_usage(char* msg) {
#ifndef PERF
    ProcStatm s;
    ProcStat_init(&s);
    printf("%s: size = %ld, resident = %ld\n", msg, s.size, s.resident);
#endif
}

void release(void* ptr) {
    int err = munmap(ptr, size);
    if(err != 0){
        printf("UnMapping Failed\n");
        exit(1);
    }
}

void write_each_page(void* ptr) {
    int* cursor = (int*)ptr;
    do {
        *cursor = 1;
        cursor += 4096;
    } while (cursor < (int*)(ptr + size));
}

// * Option1: reserve = mmap, commit = nop

// void* reserve() {
//     int *ptr = mmap (NULL, size,
//             PROT_READ | PROT_WRITE,
//             MAP_PRIVATE | MAP_ANONYMOUS,
//             0, 0 );
//     if(ptr == MAP_FAILED){
//         printf("Mapping Failed: %d, %d\n", (int)ptr, errno);
//         exit(1);
//     }
//     return ptr;
// }

// void commit(void* ptr) {

// }

// * Option2: reserve = mmap PROT_NONE, commit = mprotect

// void* reserve() {
//         int *ptr = mmap (NULL, size,
//             PROT_NONE,
//             MAP_PRIVATE | MAP_ANONYMOUS,
//             0, 0 );
//     if(ptr == MAP_FAILED){
//         printf("Mapping Failed\n");
//         exit(1);
//     }
//     return ptr;
// }

// void commit(void* ptr) {
//     int ret = mprotect(ptr, size, PROT_READ | PROT_WRITE);
//     if(ret != 0) {
//         printf("Mprotect failed\n");
//         exit(1);
//     }
// }

// * Option3: reserve = mmap noreserve, commit = mmap

void* reserve() {
    int *ptr = mmap (NULL, size,
        PROT_NONE,
        MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE,
        0, 0);
    if(ptr == MAP_FAILED){
        printf("Mapping Failed\n");
        exit(1);
    }
    return ptr;
}

void commit(void* ptr) {
    int* ret = mmap (ptr, size,
        PROT_READ | PROT_WRITE,
        MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
        0, 0);
    if(ret == MAP_FAILED || ret != ptr){
        printf("Mapping Failed\n");
        exit(1);
    }
}

int main(){
#ifdef PERF
for (int i = 0; i < 100000; i++) {
#endif
    check_mem_usage("Initial");

    void* ptr = reserve();
    check_mem_usage("Reserve");

    commit(ptr);
    check_mem_usage("Commit");

#ifndef PERF
    write_each_page(ptr);
    check_mem_usage("Actually writes");
#endif

    release(ptr);
    check_mem_usage("Release");
#ifdef PERF
}
#endif
}

[1] https://unix.stackexchange.com/questions/35129/need-explanation-on-resident-set-size-virtual-size

[qinsoon]

A few useful posts I found:

• https://unix.stackexchange.com/questions/441364/what-is-the-purpose-of-memory-overcommitment-on-linux
• https://unix.stackexchange.com/questions/35129/need-explanation-on-resident-set-size-virtual-size
• https://stackoverflow.com/questions/2782628/any-way-to-reserve-but-not-commit-memory-in-linux
• https://stackoverflow.com/questions/8506366/does-mmap-with-map-noreserve-reserve-physical-memory

[qinsoon]

We need to be clear about our definition of contiguous spaces, as it is important for us to know whether we need to reserve address ranges for contiguous spaces.

It is clear that a contiguous space has an explicit start and extent (an address range), and within the the address range, the memory can only be used for one space (one policy). However, it is unclear whether we allow holes (memory that is not mapped by mmtk-core) in the address range. If we allow holes, we do not need to reserve the address range beforehand (we can just skip the holes for a failed mmap). But it is possible that we have assumptions in our code that a contiguous space should not have holes. We should check this.