example R: add the librpma example #3

ldorau · 2022-09-08T06:39:00Z

In this example we want to introduce to you a basic way of accessing Persistent Memory when it is installed in a remote system. Remote Persistent Memory (or RPMem for short) is a way of doing this via a network by making use of the Remote Direct Memory Access (RDMA) technology.

This example is an introduction to RPMem. We will guide you briefly through setting up all required hardware and software components and verifying whether the connection works properly. Having that, we will show you how to access Persistent Memory on a remote system using the librpma library.

After completing this example you will know:

what is RPMem and what it is good for,
how RPMem is different comparing to PMem,
what hardware and software components are required to start using RPMem,
how to verify whether the RDMA network works properly,
how to use librpma API to:
establish a connection,
prepare memory for remote manipulation,
manipulate memory on the remote system,
assure persistency of stores to the remote system.

Step-by-step you will:

test the connection on the basic level using ping,
test the connection RDMA capabilities using rping,
review an application focusing on establishing a connection,
review an application reading remote system's memory and writing it back in the persistent manner.

Co-authored-by: Oksana Salyk [email protected]
Co-authored-by: Jan M Michalski [email protected]
Co-authored-by: Patryk717 [email protected]

Signed-off-by: Lukasz Dorau [email protected]

Ref: pmemhackathon/hackathon#1

This change is

…ftRoCE example R: configure SoftRoCE

Instead fix a way the access is given to the /dev/infiniband device. Ref: pmemhackathon#3

igchor

Reviewable status: 0 of 8 files reviewed, all discussions resolved (waiting on @igchor)

scripts/enable_pmemusers line 97 at r1 (raw file):


# set name of device DAX
my $dax_name = "dax0.0";

My current understanding is that all users will use the same /dev/dax but will map different part of it. However, they can modify the example code and potentially interfere with different users, right?

Wouldn't it be better to to have /dev/dax per user? (Each user would have access to a single dev only, similarly as dirs in create_pmemusers). The machines used for the hackathon are usually fully populated so it should not be a big problem to create a few hundreds of /dev/dax.

I can merge it as it is right now, but I think it could only be used in controlled environment.

scripts/enable_pmemusers line 132 at r1 (raw file):


	my $ip = `./config_softroce.sh`;
	if ($? >> 8) {

I think the docker command should also be under that if. Otherwise it will fail on error gathering device information while adding custom device "/dev/infiniband": no such file or directory.

ldorau

Reviewable status: 0 of 8 files reviewed, all discussions resolved (waiting on @igchor)

scripts/enable_pmemusers line 132 at r1 (raw file):

Previously, igchor (Igor Chorążewicz) wrote…

I think the docker command should also be under that if. Otherwise it will fail on error gathering device information while adding custom device "/dev/infiniband": no such file or directory.

Right, fixed.

ldorau

Reviewable status: 0 of 8 files reviewed, all discussions resolved (waiting on @igchor)

scripts/enable_pmemusers line 97 at r1 (raw file):

Previously, igchor (Igor Chorążewicz) wrote…

My current understanding is that all users will use the same /dev/dax but will map different part of it. However, they can modify the example code and potentially interfere with different users, right?

Wouldn't it be better to to have /dev/dax per user? (Each user would have access to a single dev only, similarly as dirs in create_pmemusers). The machines used for the hackathon are usually fully populated so it should not be a big problem to create a few hundreds of /dev/dax.

I can merge it as it is right now, but I think it could only be used in controlled environment.

I think you are right. It would be better to have a separate DAX device per user. But this is out of scope of this pull request. There are other examples that also use a DAX device, so this issue is an issue of the whole hackathon, not of this example.

lukaszstolarczuk

Reviewed 4 of 8 files at r1, 1 of 1 files at r3, all commit messages.
Reviewable status: 5 of 8 files reviewed, 16 unresolved discussions (waiting on @igchor, @ldorau, and @lukaszstolarczuk)

docker/Dockerfile line 35 at r3 (raw file):

	gtest-devel\
	hub\
	iproute\

can you please make these new deps as a sep. ENV (like RPMA_DEPS, e.g. https://github.com/pmem/rpma/blob/master/utils/docker/images/Dockerfile.fedora-33#L30)

docker/librpma.sh line 4 at r3 (raw file):

# 25.08.2022 Merge pull request #2000 from ldorau/common-1.0.0-release
LIBRPMA_VERSION=1.0.0

I guess you can bump this to 1.1.0

docker/librpma.sh line 7 at r3 (raw file):

ZIP_FILE=rpma.zip

WORKDIR=$(pwd)

you can drop this line and just use pushd <...> + popd (instead of cd <...> + cd <back>)

scripts/config_softroce.sh line 3 at r3 (raw file):

#!/usr/bin/env bash
# SPDX-License-Identifier: BSD-3-Clause
# Copyright 2020-2021, Intel Corporation

-2022, I guess

scripts/enable_pmemusers line 97 at r1 (raw file):

Previously, ldorau (Lukasz Dorau) wrote…

I think you are right. It would be better to have a separate DAX device per user. But this is out of scope of this pull request. There are other examples that also use a DAX device, so this issue is an issue of the whole hackathon, not of this example.

@ldorau, can you please file an issue for that, in this repo...?

templates/examples/R/body.tmpl line 6 at r3 (raw file):

We assume you already know what Persistent Memory (or PMem for short) is and hopefully
you have learned many possible ways of benefiting from using it in your applications.
In this example we want to introduce to you a basic way of accessing the very same

drop "to you" perhaps (it sounds a little unnatural, for me)

templates/examples/R/body.tmpl line 6 at r3 (raw file):

We assume you already know what Persistent Memory (or PMem for short) is and hopefully
you have learned many possible ways of benefiting from using it in your applications.
In this example we want to introduce to you a basic way of accessing the very same

In this example, (missing comma)

templates/examples/R/body.tmpl line 7 at r3 (raw file):

you have learned many possible ways of benefiting from using it in your applications.
In this example we want to introduce to you a basic way of accessing the very same
Persistent Memory but when it is installed in a remote system. Remote Persistent Memory

drop it is

templates/examples/R/body.tmpl line 15 at r3 (raw file):

all required hardware and software components and verifying whether the connection
works properly. Having that, we will show you how to access Persistent Memory
on a remote system using the librpma library (where RPMA stands for

rpma library or librpma

templates/examples/R/body.tmpl line 59 at r3 (raw file):

Alternatively, for development purposes, you can use a software-emulated RDMA network
interface (SoftRoCE) that can be set up using the following
<a href="https://github.com/ldorau/2021-04-21.web/blob/devel/scripts/config_softroce.sh" target="_blank">script</a>.

is this URL correct?

templates/examples/R/body.tmpl line 198 at r3 (raw file):

    <li>checked the basic network connectivity,</li>
    <li>checked whether the network is RDMA-capable,</li>
    <li>established a connection using the librpma library,</li>

. (lib... library)

templates/examples/R/body.tmpl line 214 at r3 (raw file):

    </li>
    <li>RPMem is a way to seamlessly incorporate PMem into modern cloud architectures,</li>
    <li>the librpma library is a ready-to-use simple API incarnating RPMem principles.</li>

incarnating hmm... I'm not sure if this is a proper word...?

templates/examples/R/body.tmpl line 229 at r3 (raw file):

    </li>
    <li><a href="https://pmem.io/rpma/" target="_blank">https://pmem.io/rpma/</a>
         - where you can find the librpma library manuals, configuration guidelines

. (lib... library)

templates/examples/R/body.tmpl line 234 at r3 (raw file):

        <a href="https://www.intel.com/content/www/us/en/developer/articles/technical/persistent-memory-replication-over-traditional-rdma-part-1-understanding-remote-persistent.html" target="_blank">Persistent Memory Replication Over Traditional RDMA</a>.
        <ul>
            <li><b>Note</b> it describes RPMem in the context of the librpmem library

. (lib... library)

templates/examples/R/body.tmpl line 234 at r3 (raw file):

        <a href="https://www.intel.com/content/www/us/en/developer/articles/technical/persistent-memory-replication-over-traditional-rdma-part-1-understanding-remote-persistent.html" target="_blank">Persistent Memory Replication Over Traditional RDMA</a>.
        <ul>
            <li><b>Note</b> it describes RPMem in the context of the librpmem library

perhaps mention that librpmem is now deprecated

templates/examples/R/description.tmpl line 15 at r3 (raw file):

From this example you will learn how to establish an RDMA connection,
prepare your local Persistent Memory for remote manipulation,
and how to modify remote Persistent Memory using the librpma library.

. (lib... library)

ldorau

Reviewable status: 2 of 8 files reviewed, 15 unresolved discussions (waiting on @igchor, @ldorau, and @lukaszstolarczuk)

docker/Dockerfile line 35 at r3 (raw file):