Spritely Goblins v0.14.0: libp2p and Improved Persistence
Jessica Tallon —
The goblins have been working away all summer to make a new release, and we're excited to announce that version 0.14.0 of Guile Goblins is here! In it, we introduce not only a new libp2p netlayer but also a veritable smörgåsbord of new persistence features.
(Note: There are no major protocol updates to OCapN functionality in this release, and there is not an accompanying release of Racket Goblins this time.)
libp2p
libp2p is a networking stack designed for secure peer-to-peer connections, making it a great fit for Goblins. This netlayer uses protocols like TCP and QUIC which enable libp2p to be fast.
In addition, compared to typical netlayer implementations, such as our TCP-TLS netlayer, which have issues with other peers reaching them when used behind firewalls, libp2p can sometimes mitigate these issues by automatically trying to use NAT hole-punching to open the ports allowing peer-to-peer communication.
Note that, because libp2p doesn't have a Scheme implementation, the Goblins libp2p netlayer uses our own libp2p daemon written in Go. This needs to be run alongside Goblins — similarly to how Goblins uses the Tor daemon whenever the onion netlayer is in use. To learn more about the libp2p daemon, check out its README.
Persistence improvements (Aurie)
Last release, we introduced Aurie, our persistence system which enables writing to actors which can persist across and rehydrate from different stores automatically. In this release, we upgraded Aurie to persist all netlayers and work across different local vats!
Intra-vat persistence
When building Goblins applications, developers often code across several vats with objects that reference other objects which live on different vats on their local systems (whew! Say that ten times fast). These references are known as "far references" (or "far refrs" for short), and up until this release, trying to persist an object which held far references would cause an error to be thrown!
But now there's a new actor in Goblinstown: ^persistence-registry! ^persistence-registry coordinates restoration across local vats. And while this actor itself doesn't persist any information (in other words, it isn't persistence aware), vats can use it to register their own existences at startup time. ^persistence-registry also enables vats to request far references on other vats it needs to restore.
Let's see it in use!
(use-modules (goblins)
(goblins vat)
(goblins actor-lib cell)
(goblins persistence-store syrup))
;; First lets spawn the ^persistence-registry actor!
;; It's not storing anything, so no need for a persistence vat
(define persistence-vat (spawn-vat #:name "Persistence"))
(define persistence-registry
(with-vat persistence-vat
(spawn ^persistence-registry))) ;; provided by (goblins vat)
;; Next let's spawn both our vats using the registry above.
(define-values (a-vat a-cell)
(spawn-persistent-vat
cell-env
(lambda () (spawn ^cell))
(make-syrup-store "a-vat.syrup")
#:persistence-registry persistence-registry))
(define-values (b-vat b-cell)
(spawn-persistent-vat
cell-env
;; Keep a reference to a-cell on a-vat.
(lambda () (spawn ^cell a-cell))
(make-syrup-store "b-vat.syrup")
#:persistence-registry persistence-registry))
Note that the above vats don't need to be spawned in a particular order; any far reference is restored as a promise that resolves when the object on the other vat becomes available. In other words, all far references start out as promises.
Netlayer persistence
Previously, if you wanted to maintain multiple netlayers within a persistent location, you'd have had to save a bunch of data, read it back in somehow, and provide it to all the netlayers.
As of today's release, you can now leverage Goblins' persistence system, just like you do for all the other objects, to persist netlayers. This is especially useful when considering the intra-vat persistence feature detailed above.
Selfish actors
It can sometimes be useful for actors to reference themselves, such as when building a user profile (which might want to create a sturdyref to itself). In the past, there were several ways to accomplish this, usually with the help of (goblins actor-lib selfish-spawn). The selfish-spawn library gives you this fancy selfish-spawn procedure, which everyone who wants to spawn your actor needs to use, instead of the usual spawn that you normally use when spawning actors. It's weird to need to use a special spawn procedure to spawn an actor because, really, the selfish nature of some actors is an internal concern of the actor, not the spawner. Actors shouldn't be imposing strange spawning APIs on everyone who wants to use them. On top of all that, you couldn't use selfish spawn and the persistence system together; you had to rely on other tricks to get a reference to yourself.
But with the release of 0.14.0, define-actor now provides a new #:self keyword that fixes both of these issues! You simply specify the variable you want to bind the "self" reference to and within the actor you can now use that.
Take the following actor whose behavior takes an actor reference in and returns differing commentary depending on whether the reference points to itself or someone else:
(define-actor (^knows-self bcom)
#:self self
(lambda (someone)
(if (eq? someone self)
"that's me!"
"I dunno who that is")))
We can now use the #:self keyword on define-actor. We're able to reference ourself with the identifier we choose (self). This lets our ^knows-self actor use the usual eq? identity predicate to see if the reference passed in really is itself or not!
Migrations macro
The persistence system can be used to not just store actors, but also upgrade them. Longtime readers will recognize that this is nothing new; developers could always optionally specify a version for portrait data (if no version is specified, Goblins will tag the data as version 0) and then pass said data into the restore function. In this way, developers were able to upgrade data prior to spawning the actor, but had to manually write out the code every time.
As of this release, Goblins now has a handy new migrations macro that can successfully apply migrations until it reaches the desired version. Let's look at the idea of a thermostat which initially stored temperatures in Fahrenheit but now has switched to Celsius.
(define-actor (^thermostat bcom temperature)
#:version 2 ;; We're on the second version
;; Specify a restore function which we can use to migrate from previous versions
#:restore
(lambda (version restored-temp)
;; Convert the temperature if needed...
(define temperature
(match version
;; F -> C conversion
[0 (+ (* 9/5 restored-temp) 32)]
;; C, no conversion needed.
[1 restored-temp]))
;; Finally spawn the thermostat
(spawn ^thermostat temperature))
(methods
[(get-current-temp) temperature]
[(set-temp new-temp) ...]))
This technically works, but what if you wanted to see the numbers in Kelvin? Previously, you'd have had to change the initial version 0 to go from Fahrenheit to Kelvin, add a migration from Celsius to Kelvin, and handle the case where you're being handed Kelvin. Let's see how this new migration macro helps.
(define-actor (^thermostat bcom temperature)
#:version 2
;; Instead of #:restore, we can use #:upgrade which isn't
;; responsible for spawning, but is responsible for upgrading
;; the portrait data. This works with the new migrations macro.
#:upgrade
(migrations
;; Version 0 (F) to version 1 (C)
[(1 f-temp) (+ (* 9/5 restored-temp) 32)]
;; Version 2 (C) to version 2 (K)
[(2 c-temp) (+ c-temp 2 273.15)])
;; Same behavior as before
(methods
[(get-current-temp) temperature]
[(set-temp new-temp) ...]))
Notice the similarity between migration syntax and the methods macro:
[(target-version data ...) body ...]
That means in our example when a Fahrenheit temperature (at version 0) is passed in, it'll run the initial F -> C migration, migrating it to Celsius, then it'll run the next to get it into Kelvin. This makes the job of writing migrations far easier as you only have to focus on migrating from one version to the next, not updating how all the migrations work.
Vat root upgrades
As we mentioned previously, Aurie, Goblins' persistence system, provides an upgrade path for actor portrait data.
But did you know it can also be useful to upgrade the roots of a graph when spawning a persistence vat? Similarly to how this works with actors, specify #:version with a version number for the roots and #:upgrade with an upgrade procedure when calling spawn-persistent-vat.
This even works with the new migrations macro!
Let's see it in action, imagine we've got a vat, it just spawns a single object as it's cell:
(use-modules (goblins)
(goblins vat)
(goblins actor-lib cell)
(goblins persistence-store syrup))
;; Version 0
(define-values (vat cell)
(spawn-persistent-vat
cell-env
(lambda ()
(spawn ^cell))
(make-syrup-store "my-vat.syrup")))
We didn't specify any version above, but just like actors, Goblins is tagging this as version 0 so that if we wanted to upgrade in the future, we have a version number to work off. Now, imagine we want to change this vat so that we return two objects as the roots. We want to take the cell above, but instead make it the swappable actor which returns a proxy to the cell and a swapper actor.
Let's see what that looks like:
(use-modules (goblins)
(goblins vat)
(goblins actor-lib cell)
(goblins actor-lib swappable)
(goblins persistence-store syrup))
;; Version 1
(define-values (vat cell swapper)
(spawn-persistent-vat
(persistence-env-compose cell-env swappable-env)
(lambda ()
;; Change the initial spawn procedure so it matches what we expect.
(swappable (spawn ^cell)))
(make-syrup-store "my-vat.syrup")
;; Specify the version now we've changed the roots.
#:version 1
#:upgrade
(migrations
;; Migration from 0 -> 1
[(1 cell)
(define-values (proxy swapper)
(swappable cell))
(list proxy swapper)])))
Just like we saw with the ^thermostat, as required we can keep adding migrations and Goblins will apply them as needed.
Getting the release
This release also includes a number of other bugfixes and minor features. See the NEWS file for more information!
As usual, if you're using Guix, you can upgrade to 0.14.0 by using:
guix pull
guix install guile-goblins
Otherwise, you can find the tarball on our release page
If you're making something with Goblins or want to contribute to Goblins itself, be sure to join our community at community.spritely.institute! Thanks for reading, and hope to see you there!