Spritely Goblins v0.13.0: Object Persistence and Easier IO!

We are thrilled to announce the release of our distributed programming environment Spritely Goblins version 0.13.0 for Goblins on Guile! This release has two major new features: a powerful new object persistence mechanism (to save your running programs to disk and wake them up later!), as well as a new abstraction which makes writing IO code much easier!

(Note that there are no major protocol updates to OCapN functionality in this release, and there is not an accompanying release of Racket Goblins this time.)

Persistence comes to Goblins

Imagine you're building out a fantasy dungeon crawler, modifiable by your users at runtime. Your users are building rooms and putting NPCs here and there and your world is evolving. But oh no — it's time to restart your computer! How can you keep track of all these objects and bring them back later?

Now Spritely Goblins has the answer, incorporating a powerful new persistence mechanism (codenamed "Aurie") which empowers you to save relevant parts of a running Goblins program and wake them up later! But that's not all — you can also use Goblins' persistence system to upgrade your saved objects as they evolve, and it even helps making live hacking even more powerful since you can change objects and reload them with upgraded behavior live!

Here's an example of our persistence system in action:

Above we see a user playing a space shooter game named Terminal Phase written on top of Spritely Goblins. In it, you can observe the player saving, quitting, and reopening the game. Although internally, reopening the game initializes a distinct new process, which previously meant a new game state starting at the beginning of the game, with the release of Goblins 0.13.0, we were able to save all the state-setting Goblins objects to disk in the first session and set them as the state of the new one. In other words, the player can resume their session right where they left off!

Unlike persistence systems common to some other Lisps and Smalltalk, Goblins' persistence system does not use "orthogonal persistence". Orthogonal persistence relies on the underlying programming language to implement persistence support and saves a "running snapshot" of the program as the programming language itself sees it. While a powerful approach, orthogonal persistence suffers from generally saving much more than the programmer may need and tends to result in difficult to upgrade systems.

Goblins' persistence system takes a different approach by using a technique which qualifies as "manual persistence", but in reality Goblins has also provided tooling which makes most actors "just work" in terms of persistence, providing the best of both of the worlds of manual and orthogonal persistence.

Goblins' persistence system was also designed so that only the information which needs to be saved can be what is saved, assuming that the author of the relevant object's code can determine what to store and how to restore things. For instance, in the Terminal Phase example above, the entire level need not be stored to memory, only the parts which are on screen and the filename of the current level and progress therein in terms of how far the player has gotten need be stored. Every persisted object is also tagged with a version (defaulting to 0 if none is provided) permitting the user to perform an upgrade at restoration time if appropriate.

For most use cases, adopting Goblins' persistence system will be straightforward. Here is a classic and simple "cell" actor which may be familiar to many Goblins users:

(define (^cell bcom #:optional val)
  (case-lambda
    ;; Called with no arguments; return the current value
    [() val]
    ;; Called with one argument, we become a version of ourselves
    ;; with this new value
    [(new-val)
     (bcom (^cell bcom new-val))]))

The only thing that is needed to convert this cell to a persistence-compatible cell is to switch the top-level define to the new define-actor syntax:

(define-actor (^cell bcom #:optional val)
  (case-lambda
    ;; Called with no arguments; return the current value
    [() val]
    ;; Called with one argument, we become a version of ourselves
    ;; with this new value
    [(new-val)
     (bcom (^cell bcom new-val))]))

That's it! define-actor is a fancy new macro which takes the boilerplate out of writing most persisted objects. The above roughly expands to:

(define (^cell bcom #:optional val)
  (define cell-behavior
    (case-lambda
      ;; Called with no arguments; return the current value
      [() val]
      ;; Called with one argument, we become a version of ourselves
      ;; with this new value
      [(new-val)
       (bcom (^cell bcom new-val))]))
  ;; The self-portrait gives the information necessary to restore this
  ;; object later, by default by applying to the actor's constructor.
  (define (self-portrait)
    (list val))
  (portraitize cell-behavior self-portrait))

However, users in general will rarely need to use portraitize... for most users, define-actor will be enough!

Even though objects self-describe, Goblins' serialization system follows the same capability security properties as Goblins itself. Objects cannot describe themselves with more power than they actually have access to.

Our persistence system also takes advantage of Goblins' transactional nature; we can serialize only the delta of objects which have changed between messages handled in the event loop by introspecting Goblins' transaction data, resulting in faster snapshots and less data saved to disk!

This is the first release of Goblins containing our persistence system; as such, we expect that usage will have some rough edges. In particular, persistence really only works for references to objects contained within one vat. We hope to remove this restriction in a future version of Goblins.

New, simpler IO

One of the most frequent requests for Goblins has been to introduce an IO system which "felt like" using Goblins. Previously, users had to understand the details of programming in Fibers, the event loop system on which Goblins programs run by default.

No longer! We are happy to announce a new addition to Goblins' actor-lib standard library, the ^io actor from (goblins actor-lib io)! This object wraps a protected IO resource and ensures that only one IO operation is handled at a time without blocking the vat (event loop) in which the IO actor is spawned.

Getting the release

This release also includes a number of other bugfixes and minor features. See the NEWS file for more information!

You can also get the latest Guile Goblins release using Guix:

guix pull
guix install guile-goblins

See also the README for more ways to get guile-goblins for your operating system or distribution!

Bonus history!

Goblins' persistence system was a major effort with a long history behind it. The codename "Aurie" (and the flame-and-crystal-character associated with it, named after the Aurora Borealis) came from its predecessor, an independent library for the Racket verison of Goblins.

The original conceptual designs from original-Aurie remain intact within Spritely Goblins, but the implementation and interface have been significantly refactored.

The original version of Aurie was cumbersome to use; it is thanks to Jessica Tallon, Spritely's enchantress of the goblin realm, who took Christine Lemmer-Webber's original proof-of-concept and transformed it into something much easier to use. We are all excited that Spritely Goblins now includes a powerful persistence system.

For more on the ideas behind Aurie, read Safe Serialization Under Mutual Suspicion by Mark S. Miller, by which Aurie is designed. Aurie also borrows heavily from an observation by Jonathan A. Rees made to Mark Miller, that serialization is uneval / unapply... we take exactly this technique in our implementation, as Aurie can be thought of as a metacircular evaluator running in reverse!

Onwards we go!

Of course, there's plenty more to do, and we're already working and looking ahead to the next release! We are planning for v0.14 of Goblins to focus on improvements to networking tooling in particular as well as various quality of life improvements.

If you're making something with Goblins, or want to contribute to Goblins itself, join our community at community.spritely.institute! Hope to see you there!