Spritely Goblins v0.17.0: Persistence is better than ever!

We’re excited to announce the release of Spritely Goblins 0.17.0! This release features a new lightning-fast persistence store called Bloblin, improved performance for persistence, and many bug fixes!

Bloblin — a fast streaming file store

Bloblin is a new persistence store designed to be both fast and efficient. We recommend using it as the store of choice for your persistent vats.

To initialize the store, the full object graph is written to a new file, but further updates are then streamed in a compressed delta form upon churn of the vat. It is essentially a binary log of Syrup-encoded data. To keep disk usage under control, a new file is created after a configurable number of churns upon which the full object graph is written out once again and the process repeats. Old log files are automatically deleted.

Prior to Blobin, our main viable store was the Syrup store, which would serialize the entire object graph and write it to disk after each vat churn. This kept the implementation fairly simple for testing purposes, but at the expense of performance. Bloblin is so fast that the store backend is in no way any longer the limitation on persistence performance, and Bloblin plus Goblins can stream many thousands of deltas per second to disk as-is.

Let’s see it in action:

(use-modules (goblins)
             (goblins actor-lib ring-buffer)
             (goblins persistence-store bloblin))

(define-values (vat ring-buffer)
  (spawn-persistence-vat
   ring-buffer-env
   (lambda ()
     (spawn ^ring-buffer 300))
   (make-bloblin-store "my-bloblin-store")))

The above spawns a persistent vat with just one object as the root and all the data is stored in a directory named my-bloblin-store. The Bloblin store is easy to use and the speed boost to existing, persistent Goblins applications is significant.

The above Bloblin store is using the default settings for how often it starts a new log file (every 1000 churns) and how many log files are retained (no limit). Changing these settings is as easy as changing the make-bloblin-store line to something like:

(make-bloblin-store "my-bloblin-store"
                    #:deltas-per-file 500
                    #:max-bloblin-files 3)

Two new actors in actor-lib

Goblins standard library of actors has two new members to the family: ^vector and ^ring-buffer.

The ^vector actor, as you may have guessed, provides an actor interface to Guile’s vector type. Unlike regular vectors, the ^vector actor is also resizable.

The ^ring-buffer actor provides a ring buffer (AKA circular buffer) data structure which comes in handy for queues or logs with a maximum length.

A short detour in delta-efficient actor design

This is a Goblins release post, not a deep dive on actor design, but hopefully this short detour is worth it. When designing these two actors we wanted them to persist efficiently with Bloblin (writing lots of data takes time). This meant thinking a little bit on ensuring changes to these actors produced small deltas when they change.

When an actor changes its behavior (via the bcom capability), Goblins’ persistence system will serialize that actor’s state (what we refer to as a self-portrait) and commit it to a persistence store. For most actors, their self-portrait will just be the arguments passed to its constructor, as is the case for our new ^vector actor. A simple implementation might just receive a vector as an argument and use bcom when there are changes. However, this would mean that the entire vector and all of its contents would be serialized and sent to the store every time a single element is updated.

Instead, the new ^vector actor uses an underlying vector where each element is an instance of a ^cell. Updating a single vector element updates a single cell, which means only that cell is persisted rather than the entire vector and thus the size of each delta is kept small.

Hopefully this detour to look at implementation will help you in design your own persistent actors to run fast on Bloblin and future stores which take advantage of delta writes.

Store conversion

Hopefully by this point you’ve been sold on Bloblin being the new hotness and want to take advantage of it in your project! But perhaps you’ve already have a store with data in it and you’re concerned that the upgrade path to migrate to the new store will be a difficult and messy task. Worry not! This release also contains a new procedure which can migrate data between two persistence stores!

Here’s how to convert from the Syrup store to the new Bloblin store:

(use-modules (goblins)
             (goblins persistence-store syrup)
             (goblins persistence-store bloblin))

(define syrup-store (make-syrup-store "data.syrup"))
(define bloblin-store (make-bloblin-store "bloblin-data"))
(persistence-store-copy! syrup-store bloblin-store)

It’s that simple! persistence-store-copy! takes care of the hard work.

Persistence speedups and fixes

The persistence system in Goblins should be not only faster because of the new Bloblin store, but the actual machinery of the persistence layer has been optimized. There have also been a whole bunch of smaller fixes to the persistence system and define-actor. Two fixes worth highlighting are:

• Actors which upgrade during restoration now have their upgraded state persisted.

• Actors are now restored by traversing the graph from its roots, preventing orphaned objects being unnecessarily (and maybe problematically) spawned on restore.

Getting the release

This release includes all the features detailed above as well as many bug fixes. See the NEWS for more information about all of the changes.

As usual, Guix users can upgrade to 0.17.0 by running the following:

guix pull
guix install guile-goblins

Otherwise, you can find the tarball on our release page.

Oh, and made it this far? We have an extra little treat for you. If you like the animation of our persistence flame-and-crystal mascot Aurie, we have the Blender source file if you happen to be a Blender enthusiast or generally curious how the animation works!

If you’re making something with Goblins or want to contribute to Goblins itself, be sure to join our community at community.spritely.institute! We also host regular office hours where you can come and ask questions or discuss our projects. Information about office hours is available on the forum. Thanks for following along and hope to see you there!