pmemobj — Pesistent Python objects

Creating and Accessing a PersistentObjectPool

pmemobj.create(filename, pool_size=MIN_POOL_SIZE, mode=0o666, debug=False)

Return a PersistentObjectPool backed by a file named filename, allocating pool_size bytes for the pool, and setting the mode of the file on the filesystem to mode. Raise an OSError if the file already exists. Pass debug to the PersistentObjectPool constructor.

If filename is in a filesystem backed by persistent memory, the memory will be directly accessed. Otherwise persistent memory will be emulated by memory mapping a disk file.

The actual amount of memory available for objects is smaller than pool_size transaction and object management overhead. The default is the default used by libpmemobj.

pmemobj.open(filename, debug=False)

Return a PersistentObjectPool backed by the file named filename. Raise an an OSError if the file does not exist. If the previous shutdown was not clean, call the PersistentObjectPool.gc method. Pass debug to the PersistentObjectPool constructor.

class pmemobj.PersistentObjectPool(filename, flag='w', pool_size=MIN_POOL_SIZE, mode=0x666, debug=False)

Open or create a persistent object pool backd by filename. If flag is w, raise an OSError if the file does not exist and otherwise open it for reading and writing. If flag is x, raise an OSError if the file already exists, and otherwise create the file and open it for reading and writing. If flag is c, create the file if it does not exist, but in any case open it for reading and writing.

If the file gets created, allocate pool_size bytes for the pool, and set its mode in the filesystem to mode.

If the object pool was previously not closed cleanly, call gc().

Use debug as the default value for the debug parameter to the gc() method.

root

The “root” object of the pool. This can be set to any object that can be persisted, but it is really only useful to set it to a Perisistent collection type. Only objects that are reachable by traversing the object graph starting from the root object will be preserved once the object pool is closed.

gc(debug=None)

Free all unreferenced objects: objects not accessible by tracing the object graph starting at the root object.

new(typ, *args, **kw)

Create a new instance of typ managed by this pool, passing its constructor args and kw. typ must support the Persistent API.

persist_via_pickle(*types)

Add types to the list of types that will be persisted via pickle. Nominated types must be non-container immutable types (this is not currently enforced, but confusing things will happen if you violate it). If a version of pmemobj to which support for a given type has been added is used to open a pool with instances of that type stored via pickle, the object will be resurrected from pickle, but any new instances written to the pool will use the direct support.

transaction()

Return a context manager object that manages a transaction. If the context is exited normally, all changes to objects managed by the pool should be committed; if the context exits abnormally or the program stops running for any reason in the middle of the context, then none of the changes to the persistent objects inside the transaction context should be visible. Note that the transaction does not affect changes to normal Python objects; only changes to Persistent objects will be rolled back on abnormal exit.

close()

Call gc(), mark the pool as clean, and close the underlying file. The object pool lives on in the file that contains it and may be reopened at a later date, and all the objects reachable from the root object will be in the same state they were in when the pool was closed.

Managing Persistent Memory

class pmemobj.MemoryManager(pool_pr, type_table=None)

Create a manager for a PersistentObjectPool’s memory. This class should never be instantiated directly, but instead the automatically created instance should be accessed through a pool object.

All of the methods below are atomic from the point of view of the caller. If the program crashes in the middle of the method it will either have completed or on pool reopen it will be as if it had never been started. All methods may be called from inside a transaction to make them part of a larger atomic unit of change.

new(typ, *args, **kw)

Create a new instance of typ managed by the pool, passing its constructor args and kw. typ must support the Persistent API.

transaction()

Return a context manager object that manages a transaction. If the context is exited normally, all changes to objects managed by the pool should be committed; if the context exits abnormally or the program stops running for any reason in the middle of the context, then none of the changes to the persistent objects inside the transaction context should be visible when the pool is next opened.

otuple(oid)

Ensure that oid is in tuple form. An oid retreived from memory is actually a pointer to the memory the oid was retrieved from, so if contents of that memory location changes the value of the raw oid would change. This method copies the oid data into a tuple not subject to such modification, but which can be assigned to a memory field to store its value at that location.

All MemoryManager methods that return oids return them in tuple form.

alloc(size, type_num=POBJECT_TYPE_NUM)

Return an oid pointing to size bytes of newly allocated persistent memory, passing type_num to libpmemobj as the new memory object’s type. Raise an error if called outside of any transaction().

A Persistent class should use POBJECT_TYPE_NUM for its base memory allocation, but should use a unique number for any non-PObject memory structures it allocates. (There is currently no way to manage allocating these numbers to guarnatee uniqueness, but in fact as long as something other than POBJECT_TYPE_NUM is used, nothing should break even if the number collides with at used by a different Persistent type, you just lose some memory type safety.)

zalloc(size, type_num=POBJECT_TYPE_NUM)

Same as alloc(), but the allocated persistent memory is also zeroed.

free(oid)

Return the persistent memory pointed to by oid to the pool, so that it is avaiable for future allocation. Raise an error if called outside of any transaction.

realloc(oid, size, type_num=None)

Return an oid pointing to size bytes of newly allocated persistent memory and copy the data pointed to by oid into it, truncating or zero-filling as needed. Raise an error if type_num is not None and does not match the pmem type of oid. free() the memory originally pointed to by oid. Raise an error if called outside of any transaction.

zrealloc(size, type_num=POBJECT_TYPE_NUM)

Same as realloc(), but the newly allocated persistent memory is also zeroed.

incref(oid)

Increment the reference count of the PObject pointed to by oid.

decref(oid)

Decrement the reference count of the POjbect pointed to by oid. If the reference count is zero after the decrement, then if the object has a _p_deallocate() method call it, and in any case call free() on oid.

xdecref(oid)

Call decref() on oid if oid is not OID_NULL.

decref() should be used whever possible, so that cases where an oid is unexpectedly null raise an error. If, however, the poitner can legitimately be null, this method eliminates the need for an if test, and this is a common enough case to be worth having extra method.

persist(obj)

Return an oid pointing to the representation of obj in peristent memory, creating that representation if necessary. obj must be one of the directly supported immutable types, or one of the immutable types nominated for persistence via pickle, or a Persistent type.

resurrect(oid)

Return a Python object representing the POjbect stored at oid. This may be a pure Python object if the stored object is a non-container immutable, or is otherwise an object that redirects data accesses to data stored in persistent memory.

direct(oid)

Return the real memory address of the persistent memory pointed to by oid.

Persistent Classes

class pmemobj.Persistent

Persistent is an Abstract Base Class for objects that implement the Persistent interface. All classes that want to store their data in persistent memory and manage it must implement the interface described here, but they are not required to use the ABC as their base.

_p_mm

A MemoryManager instance from the PersistentObjectPool in which this object is stored.

_p_oid

The oid that points to the PObject data structure in persistent memory that anchors this objects data.

_p_new(manager)

Initialize the objects data structures when the object is initially created, and store the provided MemoryManager manager in _p_mm and the oid pointing to the initialized data structures (a POjbect) in _p_oid.

_p_resurrect(manager, oid)

Restore the object’s state from the data located at oid, using manager, storing the manager in _p_mm and the oid in _p_oid.

_p_traverse()

Return an iterable over the oids of all of the objects pointed to by this object.

_p_substructures()

Return an iterable over the oids of all of the non-PObject data structures allocated by this object.

_p_deallocate()

Remove all pointers to any other objects, and free() any allocated data structures. When this method returns, only the memory pointed to by _p_oid should remain allocated.

class pmemobj.PersistentList([iterable])

A Persistent version of the normal Python list. Its behavior should be identical except for being persistent. (Note: currently slices are not supported.)

class pmemobj.PersistentDict([mapping_or_iterable, ]**kwarg)

A Persistent version of the normal Python dict. Its behavior should be identical except for being persistent.

class pmemobj.PersistentObject

Base class for user defined Persistent objects. May not be mixed with any other Persistent type.

As with a normal class, __init__ is called when the object is initially created. It is not called during object resurrection.

_v__init__()

This method is called both when the object is initially created and when the object is resurrected. It does nothing by default, but can be overridden to (re)acquire volatile resources. It is called before __init__ during object creation.