Getting Started

Installation and Requirements

To install pynvm you need to first install PMDK:

  • PMDK (install instructions at Github)

You can then install pynvm using pip:

pip install pynvm

pip will automatically install all dependencies for the Python package. To use pynvm in your program, import the submodule you wish to use via the nvm namespace. For example, to use pmemobj you can do:

import nvm.pmemobj

Using pmem (low level persistent memory)

The nvm.pmem module provides an interface to the PMDK libpmem API, which provides low level persistent memory support. This module provides tools you can use to implement tear-proof persistent memory updates, but working at this level your application is solely responsible for protecting against tears.

See also

For more information regarding libpmem, please refer to the libpmem documentation.

Here is an example of opening a PMEM file, writing to it, and reading from it:

import os
from nvm import pmem
from fallocate import posix_fallocate

# (optional) check the pmem library version
pmem.check_version(1, 0)

# Open file to write and fallocate space
fhandle = open("dst.dat", "w+")
posix_fallocate(fhandle, 0, 4096)

# mmap it using pmem
reg = pmem.map(fhandle, 4096)

# Write on it and seek to position zero
reg.write("lol" * 10)
reg.write("aaaa")
reg.seek(0)

# Read what was written
print(reg.read(10))
print(reg.read(10))

# Persist the data into the persistent memory
# (flush and hardware drain)
pmem.persist(reg)

Here is an example of using context managers for flush and drain and numpy buffers:

import os
import numpy as np
from nvm import pmem
from fallocate import posix_fallocate

fhandle = open("dst.dat", "w+")
posix_fallocate(fhandle, 0, 4096)

# Will persist (pmem_persist) and unmap
# automatically
with pmem.map(fhandle, 4096) as reg:
    reg.write("lol" * 10)
    reg.write("aaaa")

    # This will create a numpy array located at
    # persistent memory (very cool indeed) where you
    # can reshape as you like
    n = np.frombuffer(reg.buffer, dtype=np.int32)
    print(n.shape)

# Flush context will only flush processor caches, useful
# in cases where you want to flush several discontiguous ranges
# and then run hardware drain only once
m = pmem.map(fhandle, 4096)
with pmem.FlushContext(m) as reg:
    reg.write("lol" * 10)
    reg.write("aaaa")

# Will only execute the hardware drain (if available)
m = pmem.map(fhandle, 4096)
with pmem.DrainContext(m) as reg:
    reg.write("lol" * 10)
    reg.write("aaaa")

fhandle.close()

Using pmemlog (pmem-resident log file)

The nvm.pmemlog module provides an interface to the PMDK libpmemlog API, which provides pmem-resident log (append-only) file memory support. Writes to the log are atomic.

See also

For more information regarding the libpmemlog, please refer to the libpmemlog documentation.

Here is an example of creating a persistent log, appending a record to it, and printing out the logged record:

from nvm import pmemlog

# Create the logging and print the size (default is 2MB when not
# specified)
log = pmemlog.create("mylogging.pmemlog")
print(log.nbyte())

# Append to the log
log.append("persistent logging!")

# Walk over the log (you can also specify chunk sizes)
def take_walk(data):
    print("Data: " + data)
    return 1

log.walk(take_walk)
# This will show: "Data: persistent logging!"

# Close the log pool
log.close()

Using pmemblk (arrays of pmem-resident blocks)

The nvm.pmemblk module provides an interface to the PMDK libpmemblk API, which provides support for arrays of pmem-resident blocks. Writes to the blocks are atomic.

See also

For more information regarding the libpmemblk, please refer to the libpmemblk documentation.

Here is an example of creating a block pool and writing into the blocks:

from nvm import pmemblk

# This will create a block pool with block size of 256 and
# 1GB pool
blockpool = pmemblk.create("happy_blocks.pmemblk", 256, 1<<30)

# Print the number of blocks available
print(blockpool.nblock())

# Write into the 20th block
blockpool.write("persistent block!", 20)

# Read the block 20 back
data = blockpool.read(20)
blockpool.close()

# Reopen the blockpool and print 20th block
blockpool = pmemblk.open("happy_blocks.pmemblk")
print(blockpool.read(20))

blockpool.close()

Using pmemobj (persistent objects)

The nvm.pmemobj module provides an interface to the PMDK libpmemobj API, which provides transactionally managed access to memory that supports allocating and freeing memory areas. In this case, rather than providing a simple wrapper around the pmemobj API, which by itself isn’t very useful from Python, pynvm provides a full Python interface. This interface allows to you store Python objects persistently.

This is a work in progress: currently persistence is supported only for lists (PersistentList), dicts (PersistentDict), objects (PersistentObject), integers, strings, floats, None, True, and False. This is, however, enough to do some interesting things, and an example (pminvaders2, a port to python of the C example) is included in the examples subdirectory.

Here is an example of creating a PersistentObjectPool and storing and retrieving objects:

from nvm import pmemobj

# An object to be our root.
class AppRoot(pmemobj.PersistentObject):
    def __init__(self):
        self.accounts = self._p_mm.new(pmemobj.PersistentDict)

    def deposit(self, account, amount):
        self.accounts[account].append(amount)

    def transfer(self, source, sink, amount):
        # Both parts of the transfer will succeed, or neither will.
        with self._p_mm.transaction():
            self.accounts[source].append(-amount)
            self.accounts[sink].append(amount)

    def balance(self, account):
        return sum(self.accounts[account])

    def balances(self):
        for account in self.accounts:
            yield account, self.balance(account)

# Open the object pool, creating it if it doesn't exist yet.
pop = pmemobj.PersistentObjectPool('myaccounts.pmemobj', flag='c')

# Create an instance of our AppRoot class as the object pool root.
if pop.root is None:
    pop.root = pop.new(AppRoot)

# Less typing.
accounts = pop.root.accounts

# Make sure two accounts are created.  In a real ap you'd create these
# accounts with subcommands from the command line.
for account in ('savings', 'checking'):
    if account not in accounts:
        # List of transactions.
        accounts[account] = pop.new(pmemobj.PersistentList)
        # Starting balance.
        accounts[account].append(0)

# Pretend we have some money.
pop.root.deposit('savings', 200)

# Transfer some to checking.
pop.root.transfer('savings', 'checking', 20)

# Close and reopen the pool.  The open call will fail if the file
# doesn't exist.
pop.close()
pop = pmemobj.PersistentObjectPool('myaccounts.pmemobj')

# Print the current balances.  In a real ap this would be another
# subcommand, run at any later time, perhaps after a system reboot...
for account_name, balance in pop.root.balances():
    print("{:10s} balance is {:4.2f}".format(account_name, balance))

# You can run this demo multiple times to see that the deposit and
# transfer are cumulative.