Spritely Institute

Hoot 0.9.0 released!

2026-05-13T12:00:00Z

We are excited to announce the release of Hoot 0.9.0! Hoot is a Scheme to WebAssembly compiler backend for Guile, as well as a general purpose WebAssembly toolchain. In other words, Scheme in the browser!

This release contains new features and bug fixes and since the 0.8.0 release back in February.

Use Hoot in the upcoming Lisp Game Jam!

On Friday (May 15th 2026), the Spring edition of the Lisp Game Jam will begin! It's a 10-day long game jam where participants make games using their favorite flavor of Lisp.

Does making a small web game in Scheme using Hoot sound appealing to you? Well, then we have just the thing to get you started: the Hoot game jam template! This template project has everything you need to start making an HTML5 game with Hoot quickly.

The template repository includes:

Bindings to the necessary web APIs to make an interactive game with HTML5 canvas
A Makefile for compiling, running a development web server, and generating a .zip bundle for uploading to itch.io
A very simple Breakout-like example game that demonstrates how to put all the pieces together

Thanks to contributor Gonzalo Delgado, the game jam template now features gamepad input support!

For more inspiration, here are some games made with Hoot for past jams:

Now, onto the release notes!

Toolchain changes

Added the concept of host provided types. Useful for Hoot on Wastrel support.
Switched from legacy exceptions to standard Wasm exceptions, which were officially adopted in July 2025 but have been available in browsers for much longer. The --experimetal-wasm-exnref flag is passed to NodeJS in case it is old enough to still require this feature flag (which is currently the case for Guix).
Added support for DWARF custom section.

Compiler changes

Replaced function name/source metadata with DWARF.
Changed default debug level to 1, which now includes emitting DWARF. For a production build stripped of all debug data, use debug level 0 (-g0 in the CLI) or strip the binary afterwards using hoot strip.
Update compiler backend for new primitive bytevector predicates introduced in Guile 3.0.11.
Added support for bitvector literals on big endian host systems.

Runtime changes

Floating point number to string conversion is now implemented in Scheme rather than relying on an import. This makes binaries slightly bigger but makes it easier to support Hoot on Wastrel and non-JavaScript runtimes generally.
Bignum imports have been monomorphized to ease non-JavaScript runtimes (Wastrel, again).
Scheme binaries now export a main function that takes 0 arguments and returns 0 values that invokes the internal $load function. This makes it posible for Wastrel to boot a Hoot program without support for the Scheme reflection interface.
Added DWARF parser to reflect.js.
Removed fsqrt import in favor of using f64.sqrt instruction.

Scheme changes

Range errors now include the range in the exception irritants list.
Added uint8array->bytevector procedure to (hoot typed-arrays).
Vectors are now considered self-evaluating in (hoot expander).
Added internal (hoot module-syntax) module to gather runtime module macros and support code.
define-record-type now works in the Scheme interpreter for record types with up to 8 fields. (Supporting more than 8 fields is planned but will require larger changes to the Hoot runtime.)

CLI changes

guild compile-wasm has been deprecated in favor of the new hoot compile subcommand. Both commands accept the same flags.
Added hoot help subcommand.
Added hoot strip subcommand to remove debugging information from a Wasm binary.
Feature flags have been split out from debug options in hoot compile/guild compile-wasm. For example, -gruntime-modules is no longer valid; use -fruntime-modules instead. The -g flag now exclusively handles debugging data such as whether to emit DWARF or a names section. The -f flag handles things that change program behavior such as whether to include a runtime module system for use with the Scheme interpreter.

Documentation changes

Updated manual to use hoot compile instead of guild compile-wasm.
hoot compile --bundle is now recommended in web deployment section rather than manually copying support files.

Bug fixes

The repl.js file missing from 0.8.0 (which broke hoot repl) is now included in the release tarball. Apologies for the oversight!
Fixed pathologically large function emitted by lower-globals, which Wasm engines such as Wastrel can struggle with. Instead, many smaller functions are emitted.
hash-set!, hashq-set!, etc. now return the passed value. This matches Guile's behavior and allows more existing Guile programs to work as expected.
Fixed bug in parsing zero-length custom sections in Wasm binaries.
Fixed validation of return_call_indirect instruction.
Fixed missing EOF handler in REPL meta-command reader.

Browser compatibility

Compatible with Safari 26 or later.
Compatible with Firefox 121 or later.
Compatible with Chrome 119 or later.

Get Hoot

Hoot is available in GNU Guix:

$ guix pull
$ guix install guile guile-hoot

Also, Hoot is now available in Debian, though it will take awhile for this release to make it there.

Otherwise, Hoot can be built from source via our release tarball. See the Hoot homepage for a download link and GPG signature.

Documentation for Hoot 0.9.0, including build instructions, can be found here.

Get in touch

For bug reports, pull requests, or just to follow along with development, check out the Hoot project on Codeberg.

If you build something cool with Hoot, let us know on our community forum!

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

Aeva Palecek
David Anderson
Holmes Wilson
Jonathan Frederickson
Lassi Kiuru

Gold tier

Alex Sassmannshausen
Juan Lizarraga Cubillos

Silver tier

Austin Robinson
Brit Butler
Charlie McMackin
Dan Connolly
Deb Nicholson
Eric Bavier
Eric Schultz
Evangelo Stavro Prodromou
Evgeni Ku
Glenn Thompson
James Luke
Jonathan Wright
Michel Lind
Mike Ledoux
Nathan TeBlunthuis
Nia Bickford
Noah Beasley
Steve Sprang
Travis Smith
Travis Vachon

Bronze tier

Alan Zimmerman
Aria Stewart
BJ Bolender
Ben Hamill
Benjamin Grimm-Lebsanft
Brooke Vibber
Brooklyn Zelenka
Carl A
Crazypedia No
Ellie High
François Joulaud
Gerome Bochmann
Grant Gould
Gregory Buhtz
Ivan Sagalaev
James Smith
Jason Wodicka
Jeff Forcier
Marty McGuire
Mason DeVries
Michael Orbinpost
Neil Brudnak
Nelson Pavlosky
Philipp Nassua
Robin Heggelund Hansen
Ron Welch
Stefan Magdalinski
Stephen Herrick
Steven De Herdt
Tamara Schmitz
Thomas Talbot
William Murphy
a b
chee rabbits
r g
terra tauri

Until next time, happy hooting! 🦉

Spritely Goblins v0.18.0: Sleepy actors!

2026-04-21T12:00:00Z

We’re excited to announce the release of Spritely Goblins 0.18.0! This release features a new caching layer called “sleepy actors”, OCapN protocol updates, and numerous bug fixes. So get cozy by the fire, pull out a steaming cup of tea, and let’s have a nice relaxing read about this exciting new Goblins release!

Sleepy actors

Remember when we introduced persistence back in Goblins 0.13.0? You’re not sure? Okay, as a quick refresher, Goblins’ persistence system is able to serialize a running Goblins program for you and wake it back up later! Pretty cool!

Goblins is pretty smart about only saving the changes that need to change. But... if we can save actors to disk, do we really need them to be “awake” all at once? What if we let them take a little nap, and just woke them up when it’s time for them to do something? Then they could go back to bed when they aren’t needed anymore!

Well that’s exactly what we’ve built! Sleepy actors are a new, optional caching layer has been added to the core of Goblins. Actors may now go to sleep or be woken up depending on a customizable caching algorithm known as a “sleep strategy”. When an actor goes to sleep, it is saved to the vat’s persistence store but its reference remains live. When an asleep actor receives a message, its state is restored from the vat’s persistence store and the message is processed as usual.

Goblins currently ships with two sleep strategies: an extremely simple strategy where your little goblins head to bed after each and every turn, and a “least recently used” algorithm, which functions as a hot cache where only the most recently activated goblins stay awake, and the rest go take a nap.

For a feature that’s so sleepy, we’re pretty wired about its potential, and we hope you are too!

OCapN protocol updates

The OCapN draft specification has changed in the time since the last Goblins release. The op:deliver-only operation has been dropped in favor of a single op:deliver operation. GC operations now accept a list of export positions instead of a single position so that GC can be done in batches; their operation names have likewise been changed to the plural form (op:gc-export is now op:gc-exports, etc.) The protocol version number has thus been bumped, which means that applications built with an earlier release of Goblins are incompatible with the OCapN shipped in this release.

Notable bug fixes

Fixed a race condition when restoring multiple vats from persisted data. If Alice in vat A is referenced by Bob in vat B but no other actors in vat A then it was possible for Alice to be garbage collected before vat B is restored.
Fixed a signing oracle vulnerability in the WebSocket netlayer’s designator authentication code.

Getting the release

This release includes all the features detailed above as well as many bug fixes. See the NEWS for more information about all of the changes.

As usual, Guix users can upgrade to 0.18.0 by running the following:

guix pull
guix install guile-goblins

Otherwise, you can find the tarball on our release page.

If you’re making something with Goblins or want to contribute to Goblins itself, be sure to join our community at community.spritely.institute! We also host regular office hours where you can come and ask questions or discuss our projects. Information about office hours is available on the forum. Thanks for following along and hope to see you there!

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

Aeva Palecek
Holmes Wilson
Lassi Kiuru

Gold tier

Juan Lizarraga Cubillos

Silver tier

Austin Robinson
Brit Butler
Charlie McMackin
Dan Connolly
Deb Nicholson
Evangelo Stavro Prodromou
Glenn Thompson
James Luke
Jonathan Wright
Michel Lind
Mike Ledoux
Nia Bickford
Steve Sprang
Travis Smith

Bronze tier

Alan Zimmerman
BJ Bolender
Ben Hamill
Benjamin Grimm-Lebsanft
Brooke Vibber
Brooklyn Zelenka
Crazypedia No
Ellie High
François Joulaud
Gerome Bochmann
Grant Gould
Gregory Buhtz
Ivan Sagalaev
Jason Wodicka
Jeff Forcier
Marty McGuire
Mason DeVries
Michael Orbinpost
Neil Brudnak
Nelson Pavlosky
Philipp Nassua
Robin Heggelund Hansen
Ron Welch
Stephen Herrick
Steven De Herdt
Tamara Schmitz
Thomas Talbot
William Murphy
a b
r g
terra tauri

Hoot 0.8.0 released!

2026-02-25T12:00:00Z

We are excited to announce the release of Hoot 0.8.0! Hoot is a Scheme to WebAssembly compiler backend for Guile, as well as a general purpose WebAssembly toolchain. In other words, Scheme in the browser!

This release contains new features and bug fixes and since the 0.7.0 release back in October.

New features

New (hoot repl) module. At long last, there is now a built-in read-eval-print loop implementation! Previous releases added a macro expander, a Scheme interpreter, and a runtime module system, but now it’s possible to do live hacking from a Hoot program inside a WebAssembly runtime!
- To use the REPL, compile your Wasm binary with the necessary debug flag during development: guild compile-wasm -g1. This will include the runtime module system in the resulting binary. Expect compilation time and binary size to increase significantly. The trade-off is that a live hacking workflow will make recompilations fewer and farther between.
While not shipping in Hoot directly, initial support for using the Hoot REPL from Emacs has been added in the new geiser-hoot extension. We have submitted geiser-hoot for inclusion in MELPA and Guix so it will be easy to install in the very near future.
Enhanced (hoot web-server) module. To support the use of REPLs running within a web browser tab, the most common development use case, the web server doubles as a REPL server, proxying TCP traffic from REPL clients (more about that below) over a WebSocket to the connected browser tab.
- These enhancements introduce two new, optional depedencies to Hoot: Fibers and guile-websocket. If either of these dependencies are not present at build time, the (hoot web-server) module will not be built.
- The web server can now be extended with a user-supplied request router. An example of this can be found in our hoot-slides repository.
New (hoot web-repl) module. This module can be imported and compiled into the Wasm binary so that it can act as a REPL server. This is complicated by the fact that a browser client cannot act as a server, it is strictly a client. Instead, it connects to the aforementioned (hoot web-server) which acts as a proxy for all connected REPL clients.
New hoot command-line tool. This command will be used as a place to collect handy Hoot development tools. So far, there are two subcommands:
- hoot repl: Open a REPL running in Node. Useful for quickly trying out basic Scheme expressions in Hoot without having to compile a standalone WebAssembly program.
- hoot server: Conveniently launch the development web server in (hoot web-server).
New (web request) and (web response) modules that export a sliver of the API defined in Guile’s modules of the same names.
New (web socket) module that provides a input/output interface to WebSocket client connections. Mimicks the module of the same name in guile-websocket.
Added customizable module loader interface via new current-module-loader parameter. Two concrete loaders are provided: By default, modules are loaded from the file system by searching a load path. This is useful when running in a non-browser runtime such as NodeJS. When run-web-repl in (web repl) is used, connected REPLs are configured to use an HTTP-based loader. This loader makes HTTP requests to a special endpoint on the development web server to fetch source code.
- Note that modules loaded at runtime are loaded from source and then interpreted. Unlike Guile, where modules are automatically compiled to bytecode, Hoot cannot compile individual modules to Wasm (which would require compiling the compiler to Wasm which is an interesting future possibility).

Community highlights

Check out this chiptune tracker made with Hoot by Vivianne Langdon!

Additionally, check out Wastrel, a Wasm GC to C compiler developed by Andy Wingo. Wastrel notably uses Hoot’s Wasm toolchain. A Wasm program compiled with Wastrel runs faster than the same program on NodeJS!

Documentation changes

Updated Installation chapter to mention new optional dependencies.
Added Modules and REPL sections to the Scheme reference chapter.
Added Development chapter.
Update Status section to remove mention of missing R7RS support that we have now.
Removed docs for obsolete --emit-names flag
Add documentation for -g flag to guild compile-wasm.
Fixed example in the JavaScript reflection section that was using the obsolete load_main signature.

Toolchain changes

Split Wasm validation out of (wasm vm) and into new (wasm validation) module.
Keep data computed within the validation pass in <validated-wasm> records so that data can be used during instantiation rather than redundantly recomputing it.
Added explicit support for representing a “canonicalization”: a world in which structurally equal types are equal.
(wasm vm) types <wasm-func>, <wasm-struct>, <wasm-array> now refer to their types by index into a canonicalized set.
Added untagged <wasm-array> backing stores to (wasm vm) for all simple scalar numeric types, including i8 and i16 packed types.
Modified (wasm vm) to look up named heap type references in the instance’s canonicalization.
Added bytevector->wasm-array, wasm-array->bytevector to (wasm vm).
Added support for some of the “none” bottom types.
Packed array data is now stored signed, wrapped from i32 when set, and only unwrapped to unsigned in get_u functions.
Added string.from_code_point and string.concat lowerings in (wasm lower-stringrefs).
Renamed outdated extern.internalize and extern.externalize to their current names, any.convert_extern and extern.convert_any.
Added new has-wasm-header? procedure to (wasm parse).
Parse core reference types to <ref-type> records rather than symbol abbreviations in (wasm parse).

Miscellaneous changes

Modified schedule-task in (fibers scheduler) (which is implemented using inline Wasm on the target) to be a no-op when called at expansion time on the host i.e. used at the module top-level or from a procedural macro.
Added support for vector and call-with-values primitives to (hoot primitives) module so they can be used in interpreted code.
truncate is now exported from (guile).
Allow exports to clobber each other in module-declare! to support live hacking of modules where define-module forms are often re-evaluated many times.
Extracted JS Uint8Array bindings from internals of (fibers streams) to new (hoot typed-arrays) module.
Implement subset of Guile’s procedural module API for hackable programs (i.e. programs that are built with runtime module support).
Added (hoot config) target-side module for accessing certain build-time constants (currently just the Hoot version string).
Extracted (hoot library) module from (hoot library-group) so that the library parser can be used on the target for live hacking purposes.
Added define-module implementation to (guile) that simply throws an error if used during compilation. A separate implementation is installed for use by the interpreter in hackable programs.
Added #:replace? argument to module-export! to allow replacement of exports for live hacking purposes.
Exported module-root from (hoot modules).
Added module-imported-modules procedure to (hoot modules).
Changed file I/O host functions to return null when a file cannot be opened so a Scheme exception that can be handled by user code rather than a host exception that cannot.
Extracted contents of (scheme file) to new (hoot file) module for use in internal code such as the implementation of the file system module loader in (hoot hackable).
Moved implementation of string-join, string-concatenate, string-prefix?, and string-prefix-ci? from (guile) to (hoot strings).
Moved case-insensitive string procedures from (scheme char) to (hoot strings).
Added string-drop to (hoot strings).
Added every and fold-right procedures to (hoot lists).
Moved implementation of and-map and or-map from (guile) to (hoot lists).
Added symbol-append to (hoot symbols).
Added less verbose custom printer for <module> record type.
Switched from positional to keyword arguments for make-soft-port in (hoot ports).
Added list-index to (guile).

Bug fixes

Fixed format-exception not writing all of its output to the current error port.
Fix eof-object export in (ice-9 binary-ports).
Fixed off-by-one error for procedures with rest args in (hoot eval).
Fixed min/max to only accept real numbers, handle NaNs, and normalize exact zeroes.
Fixed continuation composition leaving an unwind continuation on the stack.
Fixed prompt unwinding in certain join continuation situations.
Fixed compilation of unwind primcalls at join points.
Fixed runtime module system ignoring replacement bindings in Guile modules.

Browser compatibility

Compatible with Safari 26 or later.
Compatible with Firefox 121 or later.
Compatible with Chrome 119 or later.

Get Hoot

Hoot is available in GNU Guix:

$ guix pull
$ guix install guile guile-hoot

Also, Hoot is now available in Debian, though it will take awhile for this release to make it there.

Otherwise, Hoot can be built from source via our release tarball. See the Hoot homepage for a download link and GPG signature.

Documentation for Hoot 0.8.0, including build instructions, can be found here.

Get in touch

For bug reports, pull requests, or just to follow along with development, check out the Hoot project on Codeberg.

If you build something cool with Hoot, let us know on our community forum!

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

Aeva Palecek
David Anderson
Holmes Wilson
Lassi Kiuru

Gold tier

Alex Sassmannshausen
Juan Lizarraga Cubillos

Silver tier

Austin Robinson
Brit Butler
Charlie McMackin
Dan Connolly
Danny OBrien
Deb Nicholson
Eric Bavier
Eric Schultz
Evangelo Stavro Prodromou
Evgeni Ku
Glenn Thompson
James Luke
Jonathan Frederickson
Jonathan Wright
Joshua Simmons
Justin Sheehy
Matt Panhans
Michel Lind
Mike Ledoux
Nathan TeBlunthuis
Nia Bickford
Noah Beasley
Steve Sprang
Travis Smith
Travis Vachon

Bronze tier

Alan Zimmerman
Aria Stewart
BJ Bolender
Ben Hamill
Benjamin Grimm-Lebsanft
Brooke Vibber
Brooklyn Zelenka
Carl A
Crazypedia No
François Joulaud
Gerome Bochmann
Grant Gould
Gregory Buhtz
Ivan Sagalaev
James Smith
Jason Wodicka
Jeff Forcier
Marty McGuire
Mason DeVries
Michael Orbinpost
Neil Brudnak
Nelson Pavlosky
Philipp Nassua
Robin Heggelund Hansen
Rodion Goritskov
Ron Welch
Stefan Magdalinski
Stephen Herrick
Steven De Herdt
Tamara Schmitz
Thomas Talbot
William Murphy
a b
r g
terra tauri

Until next time, happy hooting! 🦉

Mandy: ActivityPub on Goblins

2026-01-06T12:00:00Z

ActivityPub is the protocol that powers the Fediverse. Not only does it allow different instances of the same app to federate with one another, it also allows different apps to federate. For example, a post on a video hosting app could federate to a microblogging app. ActivityPub does a good job of this and is a leap forward from what came before it.

For those unfamiliar, ActivityPub is a decentralized social networking protocol standardized under the W3C. Both Spritely’s Executive Director, Christine Lemmer-Webber and myself (Jessica Tallon) worked on standardizing ActivityPub. The ActivityPub specification left holes in for identity, distributed storage, and more. Since then Spritely has been a continuation of this work, researching and developing the next generation of social web infrastructure.

But where does this leave ActivityPub? Has it been abandoned by Spritely as a stepping stone? No! We’ve long had a project (codenamed Mandy) on our roadmap to implement ActivityPub on top of Goblins. If you open up the ActivityPub specification you’ll actually see mention of actors. The protocol itself is designed with the actor model in mind. Since Goblins is an implementation of the actor model, they should be a natural fit.

The source code for the prototype this blog post is based off can be found here.

Goblins actors over HTTP

ActivityPub is a protocol on top of HTTP, but Goblins doesn’t use HTTP. So, the first step was to make Goblins actors available over HTTP. Fortunately, hooking this up was quite easy. There are many different ways we could do this, but for this prototype I took a fairly simple approach.

Guile has a built in web server. not only that but fibers (the concurrency system Goblins uses) has a backend for this. It means we can pretty quickly start handling requests.

The webserver can be started using the run-server procedure. It takes in a symbol which specifies an implementation ('http would be the built in HTTP server, 'fibers is the one provided by Fibers):

(run-server handler 'fibers)

The handler is a procedure which takes a request and a body and expects the HTTP response as the return value. When writing a typical HTTP server in Fibers we’d suspend the fiber until the response is ready. However, Goblins code is built around sending messages and waiting for promises to resolve. To bridge the API between these two different philosophies, we use a channel.

Channels allow two fibers to send a message to one another. Reading or writing to a channel causes the fiber to suspend until a message is available. We can then send a message to one of our actors and use on to listen to the response, once we have the response we can write our response to our channel.

Goblins vats are event loops which run on a fiber. These event loops manage a queue of messages sent to actors spawned within that vat and are processed sequentially. If we were to just write to the channel, we would suspend underlying fiber for the vat. When the vat’s fiber suspends, it stops it from processing other messages within the queue. To ensure that we’re not blocking the vat by suspending it, we’ll use the helper procedure syscaller-free-fiber which gives us a new fiber outside the vat which can be safely suspended.

(define vat (spawn-vat))
(define (^web-server bcom router)
  (define (handler . args)
    (define response-ch (make-channel))
    (with-vat vat
      (on (apply <- router args)
          (lambda (response)
            (syscaller-free-fiber
             (lambda ()
               (put-message response-ch (vector 'ok response))))
            *unspecified*)
          #:catch
          (lambda (err)
            (syscaller-free-fiber
             (lambda ()
               (put-message response-ch (vector 'err err))))
            *unspecified*)))
    (match (get-message response-ch)
      [#(ok (content-type response))
       (values `((content-type . (,content-type))) response)]
      [#(ok (content-type response) headers)
       (values `((content-type . ,content-type) ,@headers) response)]
      [#(ok response)
       (values '((content-type . (text/plain))) response)]
      [#(err err) (error "Oh no!")]))
  (syscaller-free-fiber
   (lambda ()
     (run-server handler 'fibers `(#:addr ,(inet-pton AF_INET "0.0.0.0")))))
  (lambda () 'running))

(define web-server (with-vat vat (spawn ^web-server (spawn ^router))))

This is a slightly simpler version than the one used in the prototype, but it shows how we’re making asynchronous actors which can return promises accessible to HTTP requests. From the code above, we’ve already bridged into our Goblins actors. This is a pretty flexible bridge as this ^router actor just takes in a request and provides a response, we could dispatch this request in any number of ways. For our prototype, this is the approach we took:

(define-values (registry locator)
  (call-with-vat vat spawn-nonce-registry-and-locator))

(define (^router bcom)
  (lambda (request body)
    (define request-url (request-uri request))
    (match (string-split (uri-path (request-uri request)) #\/)
      [("" "static" filename)
       (static-file (string-append "mandy/web/static/" filename))]
      [("" "object" id)
       (let ((object (<- registry 'fetch (base32-decode id))))
         (<- object 'request))])))

Most web frameworks have a special-purpose routing language that uses strings, which is an inexpressive anti-pattern. We’re fortunate in Scheme to have a powerful pattern matcher that we can use instead. In this case we’re matching a filename for our static files and we’re also making some objects available at /object/<base32-encoded-id>.

We can see above the router we’re spawning something called the nonce registry. This is an actor which provides a mechanism to register any number of actors against some identifier and look them up. The actor handles salting and hashing the IDs and even persistence. This works great for registering ActivityPub objects. Each one gets a unique ID which can be used for lookup later.

These IDs are bytevectors, so we base32 encode them to convert them into a textual form that can be included in a URI. We then just need to add a route to our match clause to look them up. You may notice we’re using <- to send messages which means we get a promise in return. This isn’t a problem for our web server though as the on handler will wait until it’s resolved.

The prototype has more routes and handles slightly more situations than the snippet of code shown above, but the principles introduced are the same.

How does ActivityPub work?

Let’s take a step back and look at ActivityPub itself both because how it works will be useful to keep in mind the rest of the post, and to see if we can see the actor model within the specification.

ActivityPub is actually not too tricky. It has concepts like inboxes and outboxes that you’re probably familiar with from email. It also has activities which describe something the user is “doing”. Activities are the building block of the protocol. Finally, it has objects which are things like Notes, Images, Videos, etc.

ActivityPub is actually two protocols in one. There’s the client-to-server protocol and then the federated server-to-server protocol. These protocols are actually very similar and for the most part mirror one another, but unfortunately the client-to-server protocol gets little love from ActivityPub implementations. Even so, let’s take a look at how I might go about posting a Note (think toot/tweet/microblog text of choice) to my good friend Christine:

POST /outbox/ HTTP/1.1
Host: tsyesika.se
Authorization: Bearer XXXXXXXXXXX
Content-Type: application/ld+json; profile="https://www.w3.org/ns/activitystreams"

{
    "@context": "https://www.w3.org/ns/activitystreams",
    "type": "Create",
    "to": ["https://dustycloud.org/"],
    "object": {
        "type": "Note",
        "content": "Ohai Christine!"
    }
}

The JSON object above represents a Create activity which basically is posting something. Other activities might be Like, Share, Delete, etc. Most activities are transitive (the activity has an object) and our Create activity is no exception. The object inside is a Note with some content.

The activity itself is posted to my outbox as an HTTP POST. The server will assign IDs to both objects and assign me (Jessica) as the author of the note. If you were to do a GET on the outbox, you’d see something like this:

GET /outbox/ HTTP/1.1
Host: tsyesika.se
Authorization: Bearer XXXXXXXXXXX
Content-Type: application/ld+json; profile="https://www.w3.org/ns/activitystreams"

{
    "@context" : "https://www.w3.org/ns/activitystreams",
    "id" : "https://tsyesika.se/outbox",
    "type" : "OrderedCollection",
    "name": "Jessica's Outbox"
    "totalItems" : 1,
    "items": {
        {
            "@context": "https://www.w3.org/ns/activitystreams",
            "type": "Create",
            "id": "https://tsyesika.se/objects/79402654-a9e5-4356-a50d-5109fedbaacc"
            "actor": "https://tsyesika.se"
            "to": ["https://dustycloud.org/"],
            "object": {
                "type": "Note",
                "attributedTo": "https://tsyesika.se"
                "id": "https://tsyesika.se/objects/2f614e93-1fe7-4a8a-ba39-f9e4468ed77f"
                "content": "Ohai Christine!"
            }
        }
    }
}

Reading from a user’s outbox gives you all the things they’ve posted (often it’s paginated, but that was left unimplemented in the prototype). You can see that the note we posted is the only item and the server has assigned IDs the activity, note, and author.

The server also should deliver this activity and note to Christine by looking up her inbox and doing an HTTP POST to it with the activity. I’m not going to go any further into federation but it’s very similar to the client-to-server API.

ActivityPub meets Goblins

If you’ve worked with Goblins or other actor frameworks you might be thinking “these activities look an awful lot like messages between different objects” and you’d be right.

That Create activity we posted above could be written something like this:

(define note (spawn ^note #:content "Ohai Christine!"))
(<- tsyesika-outbox         ; The outbox
    'create                 ; A method called create
    #:object note           ; Create an object
    #:to (list christine))  ; Send it to Christine.

The outbox can then implement whatever things it needs to for its Create activity (assigning an ID), federating the post out, etc. Just like any other Goblins actor would implement a method.

The prototype I’ve built does a fairly simple transformation from the unmarshalled JSON data. The JSON data is accepted and parsed to an association list. Then there is then an unmarshalling step where any nested objects get converted into their corresponding Goblins actors. The result is an activity actor which looks like this:

(define-actor (^as2:activity bcom parent #:key actor object target result origin
                             instrument)
  (extend-methods parent
    ((actor) actor)
    ((object) object)
    ((target) target)
    ((origin) origin)
    ((to-method)
     (string->symbol (string-downcase (assoc-ref ($ parent 'to-data) "type"))))
    ((to-data)
     (append (filter-data `(("actor" . ,actor)
                            ("object" . ,object)
                            ("target" . ,target)
                            ("result" . ,result)
                            ("origin" . ,origin)
                            ("instrument" . ,instrument)))
             ($ parent 'to-data)))))

ActivityPub is based on Activity Streams 2.0 which has a hierarchical structure. The Activity type extends from a base type of Object. This is represented in the Mandy prototype as parent.

We can then use this very simple procedure which takes an activity and converts it to a message:

(define* (activity->message activity #:key send-to)
  (define method-name ($ activity 'to-method))
  (define object ($ activity 'object))
  (define to
    (if send-to
        send-to
        ($ activity 'object)))

  (list to
        method-name
        #:object ($ activity 'object)
        #:actor ($ activity 'actor)
        #:target ($ activity 'target)
        #:self activity))

This produces our message as a list with its method name and a bunch of keyword arguments. The methods can then be defined as normal Goblins methods. If all the keywords aren’t needed for that method behavior, it can include #:allow-other-keys so that everything else can be ignored.

As an example of this let’s see the Collection type which is basically a list of objects. Here’s the implementation in the prototype:

(define* (^as2:collection bcom parent #:optional [items (make-gset)])
  (extend-methods parent
    ((add #:key object #:allow-other-keys)
     (bcom (^as2:collection bcom parent (gset-add items object))))
    ((remove #:key object #:allow-other-keys)
     (bcom (^as2:collection bcom parent (gset-remove items object))))
    ((move #:key object target #:allow-other-keys)
     (define new-items (gset-remove items object))
     ($ target 'add #:object object)
     (bcom (^as2:collection bcom parent new-items)))))

We can see it supports add, remove and move methods. The specification defines the behavior of add as:

Indicates that the actor has added the object to the target. If the target property is not explicitly specified, the target would need to be determined implicitly by context. The origin can be used to identify the context from which the object originated.

In this case, there might be two things which would be important to this method, the first being the #:object keyword and the second being the #:target. Since the collection is being sent the add message, it’s being assumed in the above code that the sender has figured the collection out. The add method then just needs to care about the object, in which case it specifies object as the only key and ignores anything else. Finally, the behavior is straightforward it adds the object to the collection.

Hopefully the above shows how we can take these ActivityPub activities and transform them into Goblins messages. This gives us the desired Goblins ergonomics while implementing ActivityPub objects.

Going further

The prototype that I implemented was a demo trying to explore some of both the Goblins actor HTTP interface and how ActivityPub might be implemented in an actor framework like Goblins. Having helped co-author ActivityPub and then develop Goblins, I’ve had musings of how this implementation might look, but it’s been very exciting to see that they work in practice.

This demo has explored both a HTTP interface with Goblins actors and ActivityPub. I think each one has a lot of potential for future work and I’d love to see Goblins applied in building websites. Goblins could be used both to build the backend of websites by handling the HTTP requests themselves, and in the browser by using Hoot.

There’s a many aspects of an ActivityPub implementation left to explore, for instance Goblins’ persistence system would be well suited to be our database. We could explore adding federation (Goblins being a distributed framework would be well suited to implement). Hopefully in the future we’ll be able to build on this experiment more.

If you found this blog post interesting, both myself and Christine Lemmer-Webber will be giving a FOSDEM 2026 talk on this. We’d love to see you there if you’re attending, but if not the video will be posted shortly after the event.

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

Aeva Palecek
David Anderson
Holmes Wilson
Lassi Kiuru

Gold tier

Alex Sassmannshausen
Juan Lizarraga Cubillos

Silver tier

Brian Neltner
Brit Butler
Charlie McMackin
Dan Connolly
Danny OBrien
Deb Nicholson
Eric Bavier
Eric Schultz
Evangelo Stavro Prodromou
Evgeni Ku
Glenn Thompson
James Luke
Jonathan Frederickson
Jonathan Wright
Joshua Simmons
Justin Sheehy
Michel Lind
Mike Ledoux
Nathan TeBlunthuis
Nia Bickford
Noah Beasley
Steve Sprang
Travis Smith
Travis Vachon

Bronze tier

Alan Zimmerman
Aria Stewart
BJ Bolender
Ben Hamill
Benjamin Grimm-Lebsanft
Brooke Vibber
Brooklyn Zelenka
Carl A
Crazypedia No
François Joulaud
Grant Gould
Gregory Buhtz
Ivan Sagalaev
James Smith
Jamie Baross
Jason Wodicka
Jeff Forcier
Marty McGuire
Mason DeVries
Michael Orbinpost
Nelson Pavlosky
Philipp Nassua
Robin Heggelund Hansen
Rodion Goritskov
Ron Welch
Stefan Magdalinski
Stephen Herrick
Steven De Herdt
Thomas Talbot
William Murphy
a b
chee rabbits
r g
terra tauri

Composing capability security and conflict-free replicated data types

2025-11-24T12:00:00Z

In August, I attended the DWeb Seminar where a small group of builders gathered to discuss the state-of-the-art and open problems in the distributed web space. Some in the group are primarily concerned with distributed data and focus on sync algorithms and local-first use cases. I am mainly concerned with distributed behavior and focus on the object capability security model. Both areas of study are steeped in their own lore and research papers, which makes it difficult for the two camps to communicate effectively with each other.

It is in the interest of unity that I write this blog post. I will show how distributed behavior and data techniques can be composed to build local-first applications that combine the strengths of each paradigm.

Fortunately, the distinction between behavior and data is a false dichotomy; they are two sides of the same coin. This circular relationship is well understood in the Lisp world where we say that “code is data and data is code” in reference to Lisp’s homoiconic syntax.

Messages are both behavior and data. They invoke behavior but are also encoded as a string of bytes and sent across the wire. Our context within the tower of abstraction determines how we look at a message. What is treated as data at one abstraction level may be treated as behavior in another. We need to be equipped to handle both cases.

I’ve crystalized all that I’ve learned recently into a small prototype that combines the following techniques:

On the behavior side, object capabilities (ocaps) and the actor model, both of which are common subjects on this blog.
On the data side, conflict-free replicated data types (CRDTs) and authorization/certificate capabilities (zcaps).

Local-first chat again

I started down the well-trodden path of making a local-first group chat application. Seemingly every other DWeb-adjacent project has one, after all, so why shouldn’t Spritely? I then branched off and went down my own trail, trying to compose tools in ways I hadn’t quite seen before in this context. The result is Brassica Chat!

Brassica Chat is written in Scheme, is built upon Goblins, our distributed programming environment, and uses Hoot to compile it all to WebAssembly so it can be used on the web. Besides just posting messages, it also supports some of the usual features like editing/removing messages and emoji reacts. 🚀

Demo time!

Below is an embedded and simplified demo of Brassica Chat that simulates a conversation between Alice, Bob, and Carol. Messages are sent over a fake network and each user’s network access can be toggled with a button to simulate network partitions and offline usage. Alice is the chat room creator and has the privilege to edit/remove any post. Bob and Carol can only edit/remove their own posts. Okay, hopefully that’s enough context. Try it out!

If you’d like more screen real estate, try this demo on its own dedicated web page. Check out the source code if you’d like.

High-level design

Let’s examine the scenario modeled in the demo more closely. First, Alice creates a new chat room on her computer. She then shares a capability with her friend Bob and another (distinct) capability with Carol that grants them the privilege to send messages to her chat room. Bob and Carol reciprocate by giving Alice capabilities to their respective chat room copies. The resulting network is shown in the diagram above.

Note that Bob and Carol are not directly connected to each other but rather indirectly connected through Alice. This is because Bob and Carol did not exchange capabilities with each other. This is okay! They can all still chat with each other in real time as long as Alice is online. When Alice goes offline, Bob and Carol can still send messages locally. Everything done while in offline mode will be synchronized once Bob and Carol can connect to Alice again. Perhaps Bob and Carol will exchange capabilities with each other later so they can still chat in real time when Alice is offline. The important detail is that Brassica Chat does not try to wire everyone together directly without the active consent of its users.

Each user in the system has a cryptographic identity in the form of a public/private key pair. This key is used for signing messages. In addition to the key, an identity also contains a human-readable, self-proposed name for displaying in the user interface.

Each chat room is an eventually-consistent replica of the distributed chat room state managed using a collection of CRDTs. Chat rooms can propagate locally created or remotely received messages to other replicas of the chat room for which it holds a capability. The replication process works to eventually achieve convergence across all reachable replicas.

At a meta level, these replicas can be thought of as forming a single, conceptual chat room actor. To use some ocap jargon, the chat room is an unum where each presence (replica) communicates by broadcasting messages to the other presences it knows about. In the diagram above, there’s a dotted line drawn around the three replicas to indicate that the chat room is an abstract entity whose canonical form does not live on any single machine. The presences are all co-equal; no single presence has more privilege than any other.

The stack

There are four levels of abstraction in the Brassica Chat architecture. From bottom to top, they are:

Object capabilities: online access control through reference passing.
Actors: online, asynchronous messaging through object references.
CRDTs: eventually consistent, offline messaging.
Authorization capabilities: offline access control through certificate chains.

All objects in the application are represented as actors, including CRDTs. Implementing CRDTs as actors has been done elsewhere, Akka being a notable example.

A reference to an actor is an object capability. In other words, holding a reference to an actor gives you the authority to send messages to it. An actor needs to be online in order to receive messages, however. For offline usage, an object capability variant known as an authorization or certificate capability is used, as well.

Messages are sent between machines using the Object Capability Network (OCapN) protocol, which handles the burden of secure message transport. Messages can be transported over any medium with an associated OCapN netlayer. For this prototype, I used a WebSocket netlayer with a relay in the middle. The CRDT implementation has its own messaging protocol which is defined using actors so that it automatically works over OCapN.

On capabilities

Brassica Chat’s use of capabilities stands in contrast to most existing local-first applications that use the access-control list (ACL) model. In the ACL model, users are associated with groups or roles that grant privileges. When compared to capabilities, the ACL model has many deficiencies:

ACLs are too coarse-grained. It’s difficult to follow the principle of least authority with a limited set of role-based privilege levels so the norm is for users to have more privilege than is necessary. By contrast, capabilities can be arbitrarily fine-grained. Want to make it so that Bob can only moderate Carol’s posts and not Alice’s? It’s easy and natural to make a capability for this but awkward to define a one-off ACL role.
ACLs can’t be safely delegated. Only an administrator may grant or revoke privileges. As a non-admin, your only option is to share your credentials, which is unsafe and hard to audit. Credential sharing happens often in the real world due to the friction involved in doing things “the right way”. With capabilities, it is easy to delegate a subset of your authority to someone else in an auditable, revokable manner without sharing your own credentials or communicating with a central authority.
Most importantly, ACLs have inherent vulnerabilities, such as the confused deputy problem. The “if you don’t have it, you can’t use it” approach of capabilities avoids an entire class of security bugs.

In short, capabilities are safer, more expressive, and more decentralized than ACLs. Now, let’s move on to some implementation details.

The chat room actor

The chat room actor is implemented as a composition of several CRDTs. Rather than using one giant CRDT for the entirety of a chat room’s history, it is partitioned by time into a set of chat log CRDT actors. Each partition covers some uniform number of seconds of real time known as the “period”. This means that all presences must use the same period value in order to converge properly (30 minutes was chosen as a reasonable default). The benefit of this partitioning strategy is that it allows each replica to perform garbage collection (GC) on entire chunks of history without coordinating with the other replicas (GC within a CRDT requires coordination). This ought to keep the append-only log for any individual chunk of history quite small and manageable. Rebuilding the state of a previously deleted chunk from scratch shouldn’t take much time, assuming there is another replica online with the data. For this prototype I didn’t bother to GC old message history as the chat rooms are ephemeral and not persisted to disk (but we could use Goblins’ persistence API to do so in the future).

In addition to the message log partitions, there are two additional CRDT actors that make up the chat room: profiles and certificates. The profiles CRDT contains a mapping from a user’s public key to their self-proposed display name (and could later be extended to include other metadata that a user would like to share with the room). The certificates CRDT contains the set of all zcaps that have been issued for the chat room.

The CRDT actors

CRDTs can be roughly divided into two categories: state-based or operation-based. Brassica Chat uses operation-based CRDTs, which can be thought of like a Git repository with automatic conflict resolution. Each replica of an operation-based CRDT maintains an event log containing all of the operations that have occurred. Due to concurrency in distributed systems, an event may have one or more direct causal predecessors (a fancy term for “parents”). Thus, the log entries form an append-only, directed acyclic graph (DAG), as shown in the diagram above.

An event has the following immutable fields:

ID: Unique ID of the event (SHA-256 hash).
Parent IDs: IDs of all causal predecessors (forming a DAG).
Timestamp: Timestamp from a hybrid logical clock indicating when the event occurred.
Author: Creator of the event (ed25519 public key).
Signature: Crytographic signature of the event.
Blob: Syrup encoded event data (Syrup is the binary serialization format used by OCapN).

Events are delivered in causal order, meaning that an event is not applied to the CRDT’s internal state until all of its predecessor events have been applied. Concurrent events may be applied in any order, so it’s important that operations on the CRDT state are commutative. Despite causal order being encoded in the event graph, a logical timestamp is included in each event. This is important for handling concurrent events and is used to implement common CRDT patterns like the “last write wins” register.

Brassica Chat contains a generic operation-based CRDT actor with prepare, effect and query hooks (straight out of the CRDT literature) for special-purpose CRDTs to implement. The CRDT actor is used as the basis for the chat log, certificates, and profiles actors.

This CRDT implementation, though on the simple side, is Byzantine fault tolerant. A Byzantine fault is best explained by the following scenario: Mallet, a user who is up to no good, sends Alice and Bob an event with the same ID but different contents. When Alice and Bob sync data with each other, they ignore events with IDs that they already have and don’t realize that Mallet has tricked them. The result is that Alice and Bob will never converge to the correct state because their message logs contain different operations.

Divergence due to Byzantine behavior is prevented through content-addressing and cryptographic signing of events, much like Git, as described in Martin Kleppmann’s “Making CRDTs Byzantine Fault Tolerant” paper. Mallet cannot send Alice and Bob events with the same ID but different contents because the ID is the hash of the contents and if the hash doesn’t match then the event is rejected. Events are signed to associate them with the author for use with the authorization capability system and the parent IDs are incorporated into the signature to prevent replay attacks. For this prototype, SHA-256 was chosen for the hash function and ed25519 for signatures.

Any number of Byzantine replicas may be in the network, but as long as Alice and Bob can directly connect to each other, or indirectly connect through a non-Byzantine node such as Carol, the well-behaved nodes will eventually converge to the correct state. While not implemented in this prototype, detection of Byzantine behavior from a replica could be used as the basis for revoking the object capability being used to send such messages, adding a layer of accountability to the system.

Authorization capabilities

With CRDTs in the mix as an offline messaging layer, object capabilities alone are insufficient for access control. The ocap layer controls access to synchronize chat messages between two replicas but it does not (and cannot) control what those messages contain. Why is that? Because the chat messages are at a higher level of abstraction than the actor messages for which the ocaps apply. When Bob writes the message (react alice-message-1 "👋") to his local replica, he is sending a message to the abstract chat room that doesn’t exist in any single location. What if Alice wanted to prevent Bob from reacting to messages? Who even has the authority to impose that restriction when there’s no central server? We’ve traded away strong consistency to support local-first usage, so there is no way for an adminstrator to install an ocap on all replicas such that they are all guaranteed to reject this message from Bob and converge to the same state. Ocaps are online capabilities, but CRDTs use offline messaging. We need an offline capability that can be used to process the offline messages.

This is where authorization capabilities (zcaps) come in. A zcap is a signed certificate that describes what actions a controller of that certificate may perform. Like ocaps, zcaps support delegation which is represented as a chain of signed certificates. A crucial property of a delegated zcap is that it cannot expand privilege, only reduce it. Certificate chains need to bottom out somewhere, so we need to decide upon a root signer. In Brassica Chat, the initiator of the chat room (Alice in our example scenario) is considered to be the root signer for all zcaps used in the chat room. This is just a convention, though, and a user could decide to place their trust in a different root signer.

Certificates in Brassica Chat are inspired by ZCAP-LD and are composed of the following immutable fields:

ID: Unique ID of the certificate (SHA-256 hash).
Parent ID: ID of the previous certificate in the delegation chain.
Signer: The public key used to sign the certificate. The signer must be a controller of the parent certificate to be considered valid.
Controllers: A list of public keys for the users who are allowed to invoke the capabilities of this certificate.
Predicate: An expression that constrains (or attenuates, to use the ocap term) the capabilities granted by the parent certificate. For example, the expression (when-op (edit delete) (allow-self)) says that edit and delete operations can only be used on posts authored by the user invoking the capability (one of the controllers).

Certificates also carry one piece of mutable state: a flag that the signer can flip from false to true to revoke the certificate. Revocation cannot be reversed, making this a trivially monotonic operation within the certificates CRDT.

At first glance, zcaps might appear to have the same problem as ocaps: a zcap cannot prevent Bob from sending a message that is not permitted because there’s no strong consistency. Instead, zcaps specify the rules by which well-behaved clients should interpret the events that have occurred. For example, Bob can send a message that edits the contents of Carol’s post, but if the zcap Bob used for that operation does not grant the capability to edit posts authored by Carol then that edit will simply be ignored when updating the chat room state on a given replica. Since zcaps are encoded as certificate documents, they can be synced amongst all replicas so that the user interface can eventually render the correct view of the chat room. This is a good example of something treated as data at one level of abstraction but behavior at another.

Security considerations

The security implications of sharing a capability to a chat room are rather large. If Alice, Bob, and Carol have replicas of the same chat room then sending Alice a message means indirectly sending Bob and Carol messages, too. Each presence of the chat room is co-equal with all other presences, after all. As a consequence, we cannot perform administration in a centralized manner like we could if there was a single canonical chat room actor living on a single machine. Revocation, for example, is now a communal effort. If Mallet can propagate messages through Bob and Carol (because Mallet holds a capability to both) then Bob and Carol must each revoke their respective capabilities in order to prevent Mallet from sending messages to the chat room in the future. While it’s possible to create a zcap that would cause Mallet’s messages to be ignored by clients, it doesn’t change the fundamental truth that Mallet has the capability to send messages to the chat room until such a time that all previously issued ocaps have been revoked. The formation of complete networks, where each replica holds a capability to sync with every other replica, is thus discouraged in this design. The connectedness of a replica is a function of how trusted the user of that replica is in the real world social group. The more strongly connected a user is, the harder it becomes to remove them later if the social dynamic changes. There is a tension between the risk imposed by a strongly connected network and the desire to maximize availability of the chat room for online users.

The overall security goal for this prototype was to prevent Mallet from irreparably destroying the shared state of the chat room, which was achieved through Byzantine fault tolerance. Additionally, message signing and zcaps provide a means of holding Mallet accountable for anti-social/malicious actions that the system is technically incapable of preventing, giving users some agency over what they see in their client interface. Is this good enough?

Things left undone

This prototype was focused on exploring the core of a minimally viable p2p chat built on capability security principles. It is not production software. I did not concern myself with optimal bandwidth nor memory usage. As mentioned earlier, chat history is not even saved to disk.

Some areas for improvement are:

Decentralized identity and naming. This was deliberately left out to keep the scope of this experiment manageable. Spritely has another project, codenamed Brux, to explore this topic. See also our paper on petnames.
Ergonomic UI/UX for the complexity introduced by decentralization and eventual consistency. What’s a user-friendly way to add and revoke ocaps and zcaps? The UI doesn’t even attempt to allow viewing or editing zcaps right now. How can we clearly communicate what the security properties are/aren’t so that users don’t get false impressions?
History rewriting. If Mallet writes some truly terrible content to the append-only chat log, it’s stuck in there even if it’s hidden in the user interface. Introducing some amount of synchronization to deal with this scenario seems okay. We could take inspiration from Git where the commit graph is append-only but branch names are mutable pointers.
Preventing new members from reading past messages like in Signal groups. This should be an option like it is in other secure chat programs, but it’s a complex topic and exploring it was out of scope.

Conclusion

I hope this was an interesting walkthrough of how ocaps, actors, CRDTs, and zcaps can be composed with each other! Big thanks to the DWeb Seminar organizers for providing the spark of inspiration I needed to dive into the CRDT literature and build this prototype.

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

Aeva Palecek
David Anderson
Holmes Wilson
Lassi Kiuru

Gold tier

Alex Sassmannshausen
Juan Lizarraga Cubillos

Silver tier

Austin Robinson
Brian Neltner
Brit Butler
Charlie McMackin
Dan Connolly
Danny OBrien
Deb Nicholson
Eric Bavier
Eric Schultz
Evangelo Stavro Prodromou
Evgeni Ku
Glenn Thompson
James Luke
Jonathan Frederickson
Jonathan Wright
Joshua Simmons
Justin Sheehy
Michel Lind
Mike Ledoux
Nathan TeBlunthuis
Nia Bickford
Noah Beasley
Shane Redman
Steve Sprang
Travis Smith
Travis Vachon

Bronze tier

Alan Zimmerman
Alexander Poylisher
Aria Stewart
BJ Bolender
Ben Hamill
Benjamin Grimm-Lebsanft
Brooke Vibber
Brooklyn Zelenka
Carl A
Crazypedia No
François Joulaud
Grant Gould
Gregory Buhtz
Ivan Sagalaev
James Smith
Jamie Baross
Jason Wodicka
Jeff Forcier
Marty McGuire
Mason DeVries
Neil Brudnak
Nelson Pavlosky
Philipp Nassua
Robin Heggelund Hansen
Rodion Goritskov
Ron Welch
Stefan Magdalinski
Stephen Herrick
Steven De Herdt
Tamara Schmitz
Thomas Talbot
William Murphy
a b
chee rabbits
r g
terra tauri

GoblinShare: Secure, Peer-to-Peer File-Sharing with Goblins

2025-11-18T12:00:00Z

This year, instead of participating in the Autumn Lisp Game Jam, Spritely decided to make some non-game demos to show off our tech. To that end, I spent the week of November 3rd writing GoblinShare, a secure, peer-to-peer file-sharing utility using the Guile port of Magenc for storage and distribution, relying on Goblins for the peer-to-peer connection abstraction, and delivered over Tor. Thanks to Goblins, this turned out to be super easy to implement. Let me show you how!

🎶️ Demos; glorious demos! 🎶️

We at Spritely absolutely love our technology demos. We know it can be tricky to understand an unfamiliar paradigm, but we also think hands-on demos help. To that end, we build lots of usable demos to try. We particularly like building games during the Lisp Game Jam so we can connect with the broader Lisp community, encourage more developers to use our technology, and end up with something fun that people want to use. After all, we can't build the future of the social web all by ourselves! This year, though, we decided to focus not on showing off work we've already done, but pushing our work in a new direction; and it felt best to do that outside the context of the jam.

My assignment (which I did choose to accept) was tripartite: port Magenc to Guile, implement a simple file-sharing tool (which I decided to somewhat model on Magic Wormhole), and port Crystal to Guile. The Crystal work remains ahead, but I've ported Magenc and built GoblinShare as the file-sharing tool. I treated the Magenc port as prep work and GoblinShare proper as the equivalent of a jam entry, so that will be the focus of this post. First, though, I'll provide an overview of these two projects, what they do, and how they do it.

Magenc

Magenc is an encrypted, distributed, content-addressed data store relying on magnet URLs as capabilities, inspired by Tahoe-LAFS. It does not use Goblins, but it is built with capability security concepts in mind. It consists of three components, each functioning as a subcommand under a single program: magenc serve, which starts a server; magenc put, which encrypts and POSTs files to a server; and magenc get, which GETs files from a server. (Although the architecture is designed to support arbitrary remote stores, only a web store using HTTP has been implemented so far.)

A quick summary

The core of Magenc's functionality is the magnet URL, which looks like magnet://?xt=<url-encoded-urn>&ek=<base64-url-encoded-string>&es=<string> and uniquely identifies a specific file. Decomposing the query parameters (everything after ?), xt refers to the exact topic identifying a specific file, which is simply the sha256 hash of the encrypted manifest (which may be a raw object as discussed below). ek is the encryption key used to encrypt the file. es is the encryption suite used for encryption. The Guile port of Magenc uses AES in Galois/Counter Mode (AES-GCM) for encryption; the Racket version uses AES in Counter Mode (AES-CTR). Of this information, only the exact topic is ever known by the server. Together, the information in a magnet URL is both all that is needed and everything that is required to identify, access, and decrypt a file on a given server. This makes a magnet URL a capability.

The server only has access to encrypted binary data a client sends. When the server receives a POST request with attached binary data, it hashes the provided data using sha256, encodes the unhashed data using base64, and stores the encoded data in a key-value store where the hash ‒ the exact topic ‒ is the key. It then converts the exact topic into a URI object from Guile's (web uri) module before sending the URI back in the content-location field of the response object. As a safety redundancy, the client checks to make sure that the exact topic it gets back is what it expects. When the server receives a GET request with an exact topic in the content-location field, it looks up the associated data, decodes the base64 into binary data, and sends that back as the body of a response. That's all the server knows and does; everything else is handled by the client.

magenc put handles the encryption and generation of cryptographic inputs. When passed a file (and, optionally, server URL), Magenc first chunks the file as necessary, encrypting each chunk to transmit separately so that the server does not necessarily know the chunks are related (though correlation of connections and network traffic by a malicious server or observer could be used to reliably guess interrelation). Then, the client prints a magnet URL which identifies the entrypoint to retrieve the file ‒ a "raw" object if the file fits in one chunk or a "manifest" if it doesn't. As discussed above, the magnet URL also embeds cryptographic information.

A file's magnet URL can be used with magenc get to retrieve that file. When passed a magnet URL (and, if necessary, server URL), magenc get first retrieves and decrypts the object identified by the exact topic. If the object is a "raw" object, Magenc extracts the binary data, decrypts it, and writes it to either standard output or a given file. If the object is a "manifest" object, Magenc extracts the exact topics for each chunk from the manifest, retrieves and decrypts each chunk in order, then writes out the complete file in the same way.

The original version of Magenc includes a more thorough write-up. Aside from the differing encryption suites and commandline interfaces, that write-up also holds for the Guile version.

A simple example

Some things are easier to understand in action, so let's walk through a trivially simple example of storing and retrieving a file using Magenc. We will skip build instructions, which are provided in the repository. For ease of demonstration, we will use the default server configuration which launches a process listening at http://127.0.0.1:8118 (http://localhost:8118) and stores data in memory rather than writing it to disk ‒ Magenc does include a backend relying on Goblins' bloblin persistence store to efficiently store files on disk. We will use what is called convergent encryption, which uses part of the unencrypted file itself as a cryptographic input, to ensure that our example file produces the same magnet URL every time it is stored. (If you are reading this in the future and Magenc's cryptography has changed, the magnet URLs may no longer match.)

First, let's start a server:

magenc serve

This should print the address where the server is listening, like:

Server running at: http://127.0.0.1:8118

We can leave that running in one terminal session and use a different one for everything else. Now let's create our example file:

echo "Hello! I'm an example file!" > example.txt

Next, we'll store the file with magenc put example.txt --convergent. This prints the magnet URL magnet:?xt=urn%3Asha256d%3A4a2TJXrPx83v1DGnOJyPa5b678AkVsPaplsx_LcT06I&ek=_TrAfpNRRLQb7gutF8KKMtj-tPWk8_AapsJQgu6sDeo&es=AES-256-GCM. (Note that argument ordering is relevant; to simplify the implementation of the CLI, Magenc expects option arguments ‒ anything starting with - or -- ‒ after positional arguments.) Finally, we can retrieve the file:

magenc get "magnet:?xt=urn%3Asha256d%3A4a2TJXrPx83v1DGnOJyPa5b678AkVsPaplsx_LcT06I&ek=_TrAfpNRRLQb7gutF8KKMtj-tPWk8_AapsJQgu6sDeo&es=AES-256-GCM"

Make sure to quote the magnet URL otherwise the shell will interpret & as a command. This command produces the output:

Hello! I'm an example file!

And that's it! Simple! There are a few more options available for the various subcommands, each explained with magenc --help.

GoblinShare

In addition to being an application, Magenc is also a library. GoblinShare is implemented using Magenc in this capacity. All it adds is a purpose-built UI and a wrapper around the in-memory store backend which makes it easier to use through a Goblins actor. The sending peer launches a Tor netlayer, generates a sturdyref ‒ a persistent object reference which can be shared out-of-band ‒ for the store actor, and adds that to the magnet URL with the additional acceptable source (as) field. The receiving peer sets up its own netlayer, "enlivens" the sturdyref, and downloads the file. Once the file is retrieved, the sending peer terminates, removing the associated data from memory.

An example

We will perform essentially the same tasks for this example as we did for the Magenc example. There are a few differences to note. First, GoblinShare does not provide an option for convergent encryption. This simply wouldn't make sense for an ephemeral file-sharing tool. For our present purposes, this means that you will almost certainly get a different magnet URL than is produced here. Second, it is necessary to manually run the tor daemon. Build and usage instructions in the GoblinShare repository cover that so the following example assumes a running daemon.

We will reuse the same example file as above, so feel free to reuse the previous command to create it. Then, all we have to do is send it:

goblinshare send example.txt

to get the magnet URL. The sending process will wait for the file to be retrieved, so switch to another terminal session to retrieve the file:

goblinshare receive "magnet://?xt=...&ek=...&es=...&as=..."

Because we're using Tor as our network layer, it can take a few seconds for the connection to be established and the data to be sent, even though we are connecting to a local server. Since our example file is small enough to fit into a single chunk, the delay isn't very long, but a larger file can take quite a while to transfer. Eventually, you will receive the expected output:

Hello! I'm an example file!

At this point, both the sending and receiving processes will terminate, and that's that! You've successfully shared a file with Goblins! Just like Magenc, GoblinShare has a (very) few options you can see with goblinshare --help. Unlike Magenc, option arguments can be supplied before or after positional arguments as long as they follow the associated subcommand.

A note on Magic Wormhole

I mentioned that I took initial inspiration for this project from Magic Wormhole. In practice, though, the only similarity wound up being the names of the subcommands send and receive. Magic Wormhole relies on a central relay server because it uses "wormhole codes" to facilitate oral communication of the relevant capabilities. These codes seem to be keys mapping to a fuller capability and therefore requiring a coordination point. GoblinShare, by contrast, chooses to instead provide less-human-friendly magnet URLs which allow fully peer-to-peer file transfer because they encode all necessary information.

How easy was it, really?

As I mentioned near the beginning of this post, the most surprising part of implementing GoblinShare was how easy it was. The core functionality was implemented in about half a day, though a failed attempt to get GoblinShare to manage the tor daemon stretched the initial implementation out to about a day and a half. In the end, the UI and business logic of GoblinShare together require 250 lines of code including module headers, whitespace, docstrings, and inline comments (but not license headers). That's very little code! Magenc took a bit more work, but a lot of that was getting the cryptography and URL abstractions playing nicely ‒ and, admittedly, there's still room to make Magenc more approachable as a library. Still, the port took a little under a week all told, including time to implement tests, and came out to somewhere in the neighborhood of 1400 lines of code with similar caveats.

Numbers are one thing, but the simplicity of GoblinShare really comes through in the code itself. The core logic comes down to three procedures, two in the send logic (with an additional helper) and one in the receive logic. Let's break these down and walk through them to build up the relevant logic.

As a general note, the UI layers of GoblinShare and Magenc use SRFI-37 to convert commandline arguments into an association list of options and arguments. The main procedures in (goblinshare) and (magenc) simply parse the commandline into the appropriate arguments which they pass to the procedure associated with a given subcommand.

Now, on to the code walkthrough!

send

First, we'll briefly discuss the relevant helper procedure, add-sref-to-magnet-url:

(define (add-sref-to-magnet-url magnet-url sref)
  (match magnet-url
    (($ <magnet-url> xt ek es #f)
     (make-magnet-url xt ek es
                      (uri-encode
                       (ocapn-id->string sref))))))

All this procedure does is decompose the magnet URL we get from Magenc then build a new magnet URL with an additional query parameter holding the sturdyref to the Goblins actor.

Most of the important logic happens in connect-to-goblinshare-server-store, named to match the convention of Magenc's store abstraction:

(define (connect-to-goblinshare-server-store done)
  (define backend (connect-to-backend #:backend-type 'memory))

  (define (put data)
    (store-backend-data backend data))

  (define (get exact-topic)
    ;; CapTP doesn't support records so we turn exact topics into strings
    (retrieve-backend-data backend (string->exact-topic exact-topic)))

  (define (close)
    (close-backend backend)
    (signal-condition! done))

  (connect-to-store* 'goblinshare-send
                     (lambda () (values get put close))))

This procedure does a few interesting things. First, it accepts a done parameter. This is a Fibers condition so that we can wait on a remote peer to collect the shared file before exiting, which is handled elsewhere.

Next, this procedure creates a Magenc memory backend. Backends are simply data stores wrapped in a Scheme record so they can be used with an abstract interface. They provide three procedures associated with three fields of the record type: backend-get, backend-put, and backend-close, accessed using retrieve-backend-data, store-backend-data, and close-backend, respectively. Here we wrap each of the underlying backend's procedures in a new procedure so we can massage our inputs to work over OCapN.

put calls the memory backend's put procedure unmodified.

get does a bit more. Because we are using OCapN to communicate between peers, and because OCapN doesn't have a dedicated type for generic records, we convert exact topics into strings when sending messages between Goblins actors. get thus converts exact topic strings back into exact topic records. We could instead have written a marshaller to convert our record into an OCapN tagged value, but there was little reason to do so.

close calls the underlying backend's close then signals the done condition.

Finally, the last call in this procedure is to Magenc's connect-to-store* helper which constructs a store interface (similar in shape and function to a backend interface) to be passed to chunk-and-store-data. As you can see, it takes two arguments: a symbol identifying its type, and a thunk which returns three values: get, put, and close. (The higher-level connect-to-store supports keyword arguments and is used to construct Magenc's built-in backends.)

The last and likely most interesting piece of GoblinShare's send logic is the part that actually deals with Goblins. All it does is wrap a store in an actor, spawn a Tor netlayer, and return a sturdyref to the wrapper actor:

(define (spawn-client-sref store)
  (define (^client bcom store)
    (methods
     ((get id) (retrieve-data store id))
     ((put . _) (error "cannot put with client capability"))
     ((close) (close-store store))))

  (define mycapn (spawn-mycapn (spawn ^onion-netlayer)))
  (:: mycapn 'register (spawn ^client store) 'onion))

The actor is ^client. As you can see, we override the backing store's put procedure to prevent remote peers from writing unexpectedly. Otherwise, we use Magenc's store interface to manipulate the store normally.

The last two lines pack a lot of logic. First, we spawn an ^onion-netlayer representing a Tor netlayer. We immediately pass this into spawn-mycapn, creating a new ^mycapn object populated with that netlayer. Then, we spawn a new ^client, immediately passing it to our ^mycapn object's register method to get a promise to a sturdyref referencing the object. Whew! All that in only two lines!

We bring all of the send logic together in gs-send:

(define* (gs-send filename #:optional
                  (out-port (current-output-port)))
  (let* ((done (make-condition))
         (store (connect-to-goblinshare-server-store done))
         (magnet-url
          (call-with-input-file filename
            (lambda (in-port)
              (chunk-and-store-data in-port store))
            #:binary #t)))
    (with-vat (spawn-vat #:name 'goblinshare-server)
      (on (spawn-client-sref store)
          (lambda (sref)
            (format out-port "~a~%"
                    (magnet-url->string
                     (add-sref-to-magnet-url magnet-url sref))))))
    (wait done)))

As you can see, gs-send requires a filename and optionally accepts an output port where it will write the resulting magnet URL. This is to facilitate tests and is not exposed in the commandline client.

The procedure starts by creating a Fibers condition which it passes to connect-to-goblinshare-server-store to create a store.

Next, it passes the resulting store to Magenc's chunk-and-store-data with default parameters ‒ so, using AES-GCM encryption and generating a new key appropriate for that cipher ‒ and a port for reading the input file. It assigns the returned magnet URL to a new variable.

Then, gs-send spawns a vat and enters a vat context. There, it spawns a client actor to wrap the store and resolves the sturdyref to the client actor. It adds the resolved sturdyref to the magnet URL and writes out the result.

Finally, gs-send waits for the done condition to be signaled.

receive

The receive command relies on only one procedure for most of its logic:

(define (connect-to-goblinshare-client-store client-sref)
  (define vat (spawn-vat #:name 'goblinshare-client))
  (define mycapn (with-vat vat (spawn-mycapn (spawn ^onion-netlayer))))
  (define client (with-vat vat (:: mycapn 'enliven client-sref)))

  (define (put . _)
    (error "cannot put with client capability"))

  (define get
    (let ((ch (make-channel)))
      (lambda (exact-topic)
        (with-vat vat
          ;; CapTP doesn't support records so we turn exact topics into strings
          (on (<- client 'get (exact-topic->string exact-topic))
              (lambda (val)
                (syscaller-free-fiber
                 (lambda () (put-message ch `(ok ,val))))
                #t)
              #:catch
              (lambda (exn)
                (syscaller-free-fiber
                 (lambda () (put-message ch `(error ,exn))))
                #f)))
        (match (get-message ch)
          (('ok val) val)
          (('error exn) (raise-exception exn))))))

  (define (close)
    (with-vat vat
      (<-np client 'close)))

  (connect-to-store* 'goblinshare-receive
                     (lambda () (values get put close))))

There are more lines here than in connect-to-goblinshare-server-store, but the gist is simple. connect-to-goblinshare-client-store takes a sturdyref, which may be a promise thanks to promise pipelining.

It starts by spawning a vat, spawning a ^mycapn with a new ^onion-netlayer, and enlivening the client actor. These steps mirror the steps used to spawn the client sturdyref above. Rather than registering an actor to get a sturdyref, here we enliven a sturdyref to get an actor.

Next, connect-to-goblinshare-client-store wraps each method of the client actor in a regular Scheme procedure for Magenc's store interface.

put and close are quite simple. The former errors out to avoid unneeded network access should it be called. The latter sends the close message to the remote actor.

get looks complicated, but most of the code is there to resolve a promise into a concrete Scheme value. The core functionality is (<- client 'get (exact-topic->string exact-topic)). This line encodes the exact topic as a string, messages the client actor's get method with that string, and returns the resulting promise. When that promise fulfills or breaks, the surrounding logic propagates the result as normal.

Finally, connect-to-goblinshare-client-store creates a new store with these wrapper procedures using connect-to-store*.

The resulting store is used by gs-receive to get the desired file:

(define* (gs-receive magnet-url #:optional
                     (out-port (current-output-port)))
  (let* ((store
          (connect-to-goblinshare-client-store
           (string->ocapn-id
            (uri-decode
             (magnet-url-acceptable-source magnet-url)))))
         (result
          (call-with-output-bytevector
           (lambda (out-port)
             (retrieve-and-unchunk-data
              out-port store
              #:exact-topic (magnet-url-exact-topic magnet-url)
              #:key (magnet-url-encryption-key magnet-url)
              #:cipher (magnet-url-encryption-suite magnet-url))))))
    (write-bytevector result out-port)
    (close-store store)))

This procedure requires the magnet URL to retrieve, and optionally accepts an output port where it will write the resulting data. This is exposed through goblinshare receive's --output option.

First, gs-receive creates a store as discussed above, extracting the sturdyref string from the magnet URL and converting it into the appropriate Scheme type. Then, it decomposes the magnet URL into its components and passes them, along with the store and a port receiving the result, to Magenc's retrieve-and-unchunk-data. Finally, it writes the result and informs the remote store that it's done.

And that's it! All the other code in GoblinShare is for the UI.

I told you it was simple!

Final thoughts

It's worth mentioning that neither Magenc nor GoblinShare are intended as finished, production software. Notably, the underlying cryptography has not been audited. Additionally, there are some improvements I'd like to make to Magenc's API which would make GoblinShare simpler.

That's okay, though. These projects are demonstrations. Magenc is intended to demonstrate the basic concepts of capability-secure distributed, encrypted, content-addressed data storage. GoblinShare, for its part, is supposed to show how easy it is to implement otherwise-complex functionality with Goblins, which I think it does. As a rough comparison, Magic Wormhole is about 11,500 lines of Python code, counted with similar caveats as those for Magenc and GoblinShare, and relies on a much longer list of dependencies.

All told, I am incredibly happy with how things turned out. GoblinShare is beautifully simple and useful. I hope it shows others how easy Goblins makes networked applications and inspires more neat software.

Happy hacking!

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

Aeva Palecek
David Anderson
Holmes Wilson
Lassi Kiuru

Gold tier

Alex Sassmannshausen
Juan Lizarraga Cubillos

Silver tier

Austin Robinson
Brian Neltner
Brit Butler
Charlie McMackin
Dan Connolly
Danny OBrien
Deb Nicholson
Eric Bavier
Eric Schultz
Evangelo Stavro Prodromou
Evgeni Ku
Glenn Thompson
James Luke
Jonathan Frederickson
Jonathan Wright
Joshua Simmons
Justin Sheehy
Michel Lind
Mike Ledoux
Nathan TeBlunthuis
Nia Bickford
Noah Beasley
Shane Redman
Steve Sprang
Travis Smith
Travis Vachon

Bronze tier

Alan Zimmerman
Alexander Poylisher
Aria Stewart
BJ Bolender
Ben Hamill
Benjamin Grimm-Lebsanft
Brooke Vibber
Brooklyn Zelenka
Carl A
Crazypedia No
François Joulaud
Grant Gould
Gregory Buhtz
Ivan Sagalaev
James Smith
Jamie Baross
Jason Wodicka
Jeff Forcier
Marty McGuire
Mason DeVries
Neil Brudnak
Nelson Pavlosky
Philipp Nassua
Robin Heggelund Hansen
Rodion Goritskov
Ron Welch
Stefan Magdalinski
Stephen Herrick
Steven De Herdt
Tamara Schmitz
Thomas Talbot
William Murphy
a b
chee rabbits
r g
terra tauri

Hoot 0.7.0 released!

2025-10-22T12:00:00Z

We are excited to announce the release of Hoot 0.7.0! Hoot is a Scheme to WebAssembly compiler backend for Guile, as well as a general purpose WebAssembly toolchain. In other words, Scheme in the browser!

This release contains bug fixes and a few new features added since the 0.6.1 release back in May. Just in time for the upcoming Lisp Game Jam!

Make a game with Hoot

The 2025 edition of the Autumn Lisp Game Jam starts on Friday, 10/31! Halloween jam! 🎃

For those who are unfamiliar, the Lisp Game Jam is a casual, 10-day event where participants build small games using the Lisp implementation of their choice. We encourage you to join the jam and make a game with Hoot! Browser games are both easy to distribute and easy for other people to play, which makes Hoot a great choice! itch.io, where the jam is hosted, has built-in support for browser games.

To make it easy to jump right in and start jamming, we have a ready-to-use template repository on Codeberg that you can clone. It takes care of all the boilerplate and provides some useful bindings to essential web APIs like HTML5 Canvas.

Okay, onto the full release notes!

New features

Added Guile’s source module directory to Hoot’s load path by default. It is now easy to import any module from Guile that Hoot does not override in its core implementation. Of course, any Guile module that relies on a C extension or as-of-yet unsupported (guile) procedure/macro will not work with Hoot. Contributors are wanted to help maximize our Guile compatibility!
Removed stub SRFI-1 module in favor of loading it from Guile’s source module directory. Guile now has a pure Scheme implementation of SRFI-1 (no more C!) usable by Hoot.
Hoot modules can now be loaded from JavaScript typed arrays and ArrayBuffer objects in methods such as Scheme.load_main in reflect.js.
Top-level programs must now start with a use-modules or import form. The implicit import of (scheme base) caused more confusion for users than the convenience was worth.
The import form in a top-level program is now interpreted using R7RS syntax rules whereas previously it used R6RS syntax rules.
SRFI library names such as (srfi :1) are now canonicalized as (srfi srfi-1) in R6RS library forms to match Guile.
Added support for integer components in R7RS define-library names. Example: (define-library (srfi 1) ...).
Added support for #:renamer in define-module.
Added filter!, reverse!, and with-fluids* procedures to (guile).
Added printer for record type descriptors.
Documented HOOT_LOAD_PATH environment variable.

Performance improvements

Added optimized implementation of min and max.

Bug fixes

Explicitly set file permissions for reflection library files copied by guild compile-wasm --bundle. This problem was especially noticeable on Guix where the files were read-only, breaking successive bundle invocations.
Fixed hash to return the accumulated hash value when the recursion depth limit is reached. Previously, the result of hashv was returned, causing objects what were equal? to have different hash codes.
Fixed define-record-type to raise a syntax error when an invalid field is specified as a constructor argument.
Fixed R6RS rename syntax rules in library export forms.
Fixed implementation of #:replace in define-module syntax.
Fixed put-string which was not respecting the port argument.
Fixed docstring for ,hoot-run-file meta-command.
Fixed link to Git repository URL in the manual.
Replaced obsolete reference to parse-wat in the manual with wat->wasm.
Fixed compilation of unquote-splicing in non-tail position.
Fixed string-trim in the case where the result should be the empty string.
simple-format now returns the unspecified value rather than zero values.
Fixed select instruction handling in Wasm toolchain.

Browser compatibility

Compatible with Safari 26! 🎉 At long last, Hoot modules now work reliably in Safari and other sufficiently fresh WebKit-based browsers.
Compatible with Firefox 121 or later.
Compatible with Chrome 119 or later.

Get Hoot 0.7.0

Hoot is already available in GNU Guix:

$ guix pull
$ guix install guile-next guile-hoot

(Hoot currently requires a bleeding-edge version of Guile, hence guile-next above.)

Otherwise, Hoot can be built from source via our release tarball. See the Hoot homepage for a download link and GPG signature.

Documentation for Hoot 0.7.0, including build instructions, can be found here.

Get in touch

For bug reports, pull requests, or just to follow along with development, check out the Hoot project on Codeberg.

If you build something cool with Hoot (for the upcoming game jam or otherwise), let us know on our community forum!

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

Aeva Palecek
Daniel Finlay
David Anderson
Holmes Wilson
Lassi Kiuru

Gold tier

Juan Lizarraga Cubillos

Silver tier

Adam King
Austin Robinson
Brian Neltner
Brit Butler
Charlie McMackin
Dan Connolly
Danny OBrien
Deb Nicholson
Eric Schultz
Evangelo Stavro Prodromou
Evgeni Ku
Glenn Thompson
James Luke
Jonathan Frederickson
Joshua Simmons
Justin Sheehy
Michel Lind
Mikayla Maki
Mike Ledoux
Nathan TeBlunthuis
Nia Bickford
Noah Beasley
Shane Redman
Steve Sprang
Travis Smith
Travis Vachon

Bronze tier

Alan Zimmerman
Alex Poylisher
BJ Bolender
Ben Hamill
Benjamin Grimm-Lebsanft
Brooke Vibber
Brooklyn Zelenka
Carl A
Crazypedia No
François Joulaud
Grant Gould
Gregory Buhtz
Ivan Sagalaev
Jason Wodicka
Jeff Forcier
Louis Jackman
Marty McGuire
Mason DeVries
Neil Brudnak
Nelson Pavlosky
Philipp Nassua
Robin Heggelund Hansen
Rodion Goritskov
Ron Welch
Stephen Herrick
Steven De Herdt
Sébastien Rey-Coyrehourcq
Tamara Schmitz
Thomas Talbot
William Murphy
r g
terra tauri

Until next time, happy hooting! 🦉

Spritely Goblins v0.17.0: Persistence is better than ever!

2025-10-16T12:00:00Z

We’re excited to announce the release of Spritely Goblins 0.17.0! This release features a new lightning-fast persistence store called Bloblin, improved performance for persistence, and many bug fixes!

Bloblin — a fast streaming file store

Bloblin is a new persistence store designed to be both fast and efficient. We recommend using it as the store of choice for your persistent vats.

To initialize the store, the full object graph is written to a new file, but further updates are then streamed in a compressed delta form upon churn of the vat. It is essentially a binary log of Syrup-encoded data. To keep disk usage under control, a new file is created after a configurable number of churns upon which the full object graph is written out once again and the process repeats. Old log files are automatically deleted.

Prior to Blobin, our main viable store was the Syrup store, which would serialize the entire object graph and write it to disk after each vat churn. This kept the implementation fairly simple for testing purposes, but at the expense of performance. Bloblin is so fast that the store backend is in no way any longer the limitation on persistence performance, and Bloblin plus Goblins can stream many thousands of deltas per second to disk as-is.

Let’s see it in action:

(use-modules (goblins)
             (goblins actor-lib ring-buffer)
             (goblins persistence-store bloblin))

(define-values (vat ring-buffer)
  (spawn-persistence-vat
   ring-buffer-env
   (lambda ()
     (spawn ^ring-buffer 300))
   (make-bloblin-store "my-bloblin-store")))

The above spawns a persistent vat with just one object as the root and all the data is stored in a directory named my-bloblin-store. The Bloblin store is easy to use and the speed boost to existing, persistent Goblins applications is significant.

The above Bloblin store is using the default settings for how often it starts a new log file (every 1000 churns) and how many log files are retained (no limit). Changing these settings is as easy as changing the make-bloblin-store line to something like:

(make-bloblin-store "my-bloblin-store"
                    #:deltas-per-file 500
                    #:max-bloblin-files 3)

Two new actors in actor-lib

Goblins standard library of actors has two new members to the family: ^vector and ^ring-buffer.

The ^vector actor, as you may have guessed, provides an actor interface to Guile’s vector type. Unlike regular vectors, the ^vector actor is also resizable.

The ^ring-buffer actor provides a ring buffer (AKA circular buffer) data structure which comes in handy for queues or logs with a maximum length.

A short detour in delta-efficient actor design

This is a Goblins release post, not a deep dive on actor design, but hopefully this short detour is worth it. When designing these two actors we wanted them to persist efficiently with Bloblin (writing lots of data takes time). This meant thinking a little bit on ensuring changes to these actors produced small deltas when they change.

When an actor changes its behavior (via the bcom capability), Goblins’ persistence system will serialize that actor’s state (what we refer to as a self-portrait) and commit it to a persistence store. For most actors, their self-portrait will just be the arguments passed to its constructor, as is the case for our new ^vector actor. A simple implementation might just receive a vector as an argument and use bcom when there are changes. However, this would mean that the entire vector and all of its contents would be serialized and sent to the store every time a single element is updated.

Instead, the new ^vector actor uses an underlying vector where each element is an instance of a ^cell. Updating a single vector element updates a single cell, which means only that cell is persisted rather than the entire vector and thus the size of each delta is kept small.

Hopefully this detour to look at implementation will help you in design your own persistent actors to run fast on Bloblin and future stores which take advantage of delta writes.

Store conversion

Hopefully by this point you’ve been sold on Bloblin being the new hotness and want to take advantage of it in your project! But perhaps you’ve already have a store with data in it and you’re concerned that the upgrade path to migrate to the new store will be a difficult and messy task. Worry not! This release also contains a new procedure which can migrate data between two persistence stores!

Here’s how to convert from the Syrup store to the new Bloblin store:

(use-modules (goblins)
             (goblins persistence-store syrup)
             (goblins persistence-store bloblin))

(define syrup-store (make-syrup-store "data.syrup"))
(define bloblin-store (make-bloblin-store "bloblin-data"))
(persistence-store-copy! syrup-store bloblin-store)

It’s that simple! persistence-store-copy! takes care of the hard work.

Persistence speedups and fixes

The persistence system in Goblins should be not only faster because of the new Bloblin store, but the actual machinery of the persistence layer has been optimized. There have also been a whole bunch of smaller fixes to the persistence system and define-actor. Two fixes worth highlighting are:

Actors which upgrade during restoration now have their upgraded state persisted.
Actors are now restored by traversing the graph from its roots, preventing orphaned objects being unnecessarily (and maybe problematically) spawned on restore.

Getting the release

This release includes all the features detailed above as well as many bug fixes. See the NEWS for more information about all of the changes.

As usual, Guix users can upgrade to 0.17.0 by running the following:

guix pull
guix install guile-goblins

Otherwise, you can find the tarball on our release page.

Oh, and made it this far? We have an extra little treat for you. If you like the animation of our persistence flame-and-crystal mascot Aurie, we have the Blender source file if you happen to be a Blender enthusiast or generally curious how the animation works!

Shepherd × Goblins update

2025-09-10T12:00:00Z

The Shepherd is an init system and process manager initially built for GNU Hurd and now used by Guix. It can run with either root or user privileges to launch daemons, execute tasks, and manage processes. As we’ve discussed previously, Spritely has been working to port the Shepherd to Goblins. We’ve been a bit quiet since that announcement, so what’s the buzz?

First, as a quick refresher, the Shepherd is a great project to port to Goblins because it’s already built on the actor model. By switching to Goblins, we can bring the following benefits to the project:

Streamline the codebase by replacing the Shepherd’s ad-hoc actor model implementation.
Reduce the likelihood of concurrency bugs caused by the existing actor model implementation that exposes too much of its CSP foundation using Fibers.
Transform services (and other actors) into object capabilities for fine-grained management of privileges, which will (eventually) make it to possible to unify the currently separate worlds of “system” Shepherds that run as root under PID 1 and “user” Shepherds that run as an unprivileged user.
Enable Shepherd to use the Object Capability Network (OCapN) to open the door for distributed networks of “Communicating Shepherd Processes” in the future.

Since our last post, we’ve done the following:

Wrote Goblins versions of the core actors like the service controller, service registry, and process monitor.
Added unit tests for all of the core actors (there were none before).
Rewrote the public API as a compatibility layer on top of a new Goblins actor API. This new API is private for now.

What this means is that all of the extant Shepherd functionality will soon be available in the Goblins port. We’re currently working out the remaining sneaky, tricky, subtle bugs in order to have a full 1:1 port that passes the existing test suite. We’re getting very close, so we felt it was time to share this update!

What we’ve been up to

To better explain the work we've been doing, we need to discuss some Shepherd internals, particularly how actors work. Shepherd includes an ad-hoc actor model that has notable differences with Goblins. Shepherd actors are implemented as an event loop running in a fiber (a lightweight thread) and send messages to each other over channels. In contrast, Goblins is also built on Fibers but mostly hides this behind an abstraction barrier. Rather than each actor managing its own event loop, many Goblins actors share an event loop known as a vat. Each Goblins actor is mapped to its current “behavior”, a procedure that is called when the actor receives a message. The differences between these two actor model implementations means that porting an actor from one to the other isn’t as straightforward as it might seem.

The central unit of abstraction is the service, which represents something managed by the Shepherd. This could be an external process, a one-off task (known as a “one-shot” service), a timer, or whatever the user wants it to be. Internally, services are represented by a record holding immutable configuration information and a service controller actor which manages the running state. The service controller (named ^service in the code) was the first target of our porting efforts as it helped elucidate the shape of the new architecture.

Going hand-in-hand with the service controller actor is the service registry (^service-registry). The service registry is responsible for mapping the names of services to their associated service controller. Porting the service registry was one of the simplest parts of the project; most of the original actor logic was copied over with minimal changes.

The Shepherd makes heavy use of dynamically scoped variables known as parameters to pass around shared state like the current registry, the current service, the current client socket, etc. This created an issue for us, however, as vats introduce a continuity barrier for parameters. In the existing actor system, actors inherit the dynamic environment in which they are spawned because each actor is a new fiber spun off the current fiber. In Goblins, actors are spawned within a vat’s event loop which has an entirely separate dynamic state from the caller. Furthemore, Goblins discourages the use of parameters because they are inherently ambient and thus not capability-safe.

Removing these parameters would be a backwards incompatible change, so instead we capture the current state of relevant parameters in the compatibility layer before passing those values off to Goblins actors. The most obvious use of this technique is for I/O handling. The Shepherd uses a custom soft port for logging to standard output, a client socket, and/or the system log. A little named actor tentatively named ^writer handles these concerns now.

The most significant change is the introduction of a coordinating actor called simply ^shepherd. This actor is where we pushed all of the logic related to starting, stopping, respawning, etc. Procedures such as start-service and stop-service are now thin wrappers around calls to this actor.

Related to process orchestration, the Shepherd has a process monitor actor, whose job is to watch for the termination of processes associated with services and notify other actors about it. This was also relatively simple to port to a Goblins ^process-monitor actor once all of the shared state was properly captured. Goblins promises somewhat simplified the logic involved in responding to these changing states.

Perhaps the trickiest part of the port was logging. Loggers read from an input port and write timestamped log lines to some destination, perhaps a file or the system log. Port I/O requires some understanding how Fibers and Goblins interact. If you’re not careful, you can suspend a vat’s fiber, potentially stalling the program. Goblins provides the ^io actor to handle many common I/O needs safely, but the needs of Shepherd were beyond what that actor could provide. A first attempt at porting this logic proved too buggy to rely upon, so we’ve recently reworked logging actors into something more robust (and more similar to the original logging actors, too).

Finally, we have made a variety of smaller changes so existing code plays nicely with new Goblins actors. For example, the Shepherd provides a collection of helpers for things like starting and stopping processes (make-forkexec-constructor, for example). The shift to Goblins required the introduction of Goblins message passing and promise handling to keep some of these working as expected. A lot of time has been spent devising ways to keep the public API the same so that existing user code will continue to function as expected, as if nothing has really changed. To support all of this work, we’ve introduced a few of our own helper procedures and macros, and we’ve modified some existing ones to be Goblins-friendly.

Whew, that’s a lot! It’s the culmination of over a year of work, so it can be difficult to take in. If you’d like to try, you can see the current state of the port in our WIP pull request on Codeberg!

Demo time

Okay, but does it work? We’re so glad you asked!

In addition to using Shepherd for its init system (PID 1), Guix provides helpful facilities for running user-level Shepherd daemons through the home-shepherd-service-type in guix home. This is the same kind of user Shepherd daemon mentioned before; Guix just provides a nice, declarative interface to configure and launch the daemon when defining a user's home-environment. We used this functionality to swap in our Goblins Shepherd and manage an Emacs background daemon with it. Here's an actual session running in the context of guix home container:

juli shepherd λ guix home container home-shepherd.scm
substitute: recherche des substituts sur « https://substitutes.nonguix.org »… 100.0%
substitute: recherche des substituts sur « https://bordeaux.guix.gnu.org »…   0.0%guix substitute: avertissement : bordeaux.guix.gnu.org : la connexion à échouée : Connexion refusée
substitute:
substitute: recherche des substituts sur « https://ci.guix.gnu.org »… 100.0%
Les dérivations suivantes seront compilées :
  /gnu/store/ixpviqjakf55j24ag523pdl5g9k8xld7-provenance.drv
  /gnu/store/ynykwmi237h4jxrgdgkwqs4sgvf1h3cc-home.drv

substitute: recherche des substituts sur « https://bordeaux.guix.gnu.org »…   0.0%
construction de /gnu/store/ixpviqjakf55j24ag523pdl5g9k8xld7-provenance.drv...
construction de /gnu/store/ynykwmi237h4jxrgdgkwqs4sgvf1h3cc-home.drv...
WARNING: (guile-user): imported module (guix build utils) overrides core binding `delete'
WARNING: (guile-user): imported module (guix build utils) overrides core binding `delete'
Symlinking /home/juli/.bash_profile -> /gnu/store/9vidh7q8sp353rb1jnrndyif9wl2fjna-bash_profile... done
Symlinking /home/juli/.profile -> /gnu/store/jjvk66x9wwzxw38byk796y9b6kvi21b0-shell-profile... done
Symlinking /home/juli/.bashrc -> /gnu/store/mdp6zf77631kqr8cw26p4m3vvbr7vk01-bashrc... done
Symlinking /home/juli/.config/shepherd/init.scm -> /gnu/store/nag703p683l66s2adad719810xfrhx3w-shepherd.conf... done
Symlinking /home/juli/.config/fontconfig/fonts.conf -> /gnu/store/bqhrpq7na79bxm3sbpmnana10g6sc4d5-fonts.conf... done
 done
Finished updating symlinks.

Comparing /gnu/store/non-existing-generation/profile/share/fonts and
          /gnu/store/yyyn7zy4lx8z9qsb41imkbxb11wrrqqc-home/profile/share/fonts... done (same)
Evaluating on-change gexps.

On-change gexps evaluation finished.

juli@sordidus ~$ herd status
Started:
 + emacs
 + root
juli@sordidus ~$ herd status emacs
Status of emacs:
  It is running since 19:40:05 (9 seconds ago).
  Main PID: 40
  Command: /gnu/store/b6f34g5rsz35z40fc0myimw9zgj654xj-emacs-no-x-30.1/bin/emacs --fg-daemon
  It is enabled.
  Provides: emacs
  Will be respawned.

Recent messages (use '-n' to view more or less):
  2025-09-05 19:40:06 Starting Emacs daemon.
juli@sordidus ~$ emacsclient -c
juli@sordidus ~$ herd stop emacs
juli@sordidus ~$ herd status emacs
Status of emacs:
  It is stopped since 19:40:31 (2 seconds ago).
  Process exited with code 15.
  It is enabled.
  Provides: emacs
  Will be respawned.
juli@sordidus ~$ herd start emacs
Service emacs has been started.
juli@sordidus ~$ herd status emacs
Status of emacs:
  It is running since 19:40:38 (2 seconds ago).
  Main PID: 144
  Command: /gnu/store/b6f34g5rsz35z40fc0myimw9zgj654xj-emacs-no-x-30.1/bin/emacs --fg-daemon
  It is enabled.
  Provides: emacs
  Will be respawned.

Recent messages (use '-n' to view more or less):
  2025-09-05 19:40:38 Starting Emacs daemon.
juli@sordidus ~$ herd restart emacs
Service emacs has been started.
juli@sordidus ~$ herd status emacs
Status of emacs:
  It is running since 19:40:45 (2 seconds ago).
  Main PID: 180
  Command: /gnu/store/b6f34g5rsz35z40fc0myimw9zgj654xj-emacs-no-x-30.1/bin/emacs --fg-daemon
  It is enabled.
  Provides: emacs
  Will be respawned.

Recent messages (use '-n' to view more or less):
  2025-09-05 19:40:45 Starting Emacs daemon.
juli@sordidus ~$ ls -al $(command -v herd)
lrwxrwxrwx 1 65534 overflow 80 Jan  1  1970 /home/juli/.guix-home/profile/bin/herd -> /gnu/store/4l4b2qb91bq3djj9ldg66jx6p98hxvin-goblins-shepherd-1.0.99-git/bin/herd

As simple as this demo is, it demonstrates that the Goblins Shepherd can already handle its basic job, despite the work left to achieve parity with mainline. If you’d like to try this out for yourself, first ensure you have Guix installed and up-to-date, then run the following commands:

git clone https://codeberg.org/spritely/shepherd
cd shepherd
git checkout -b goblins-shepherd-guix-home-demo
guix home container home-shepherd.scm

You can also look on Codeberg to see the home config itself.

“That’s so exciting!” we hear you saying; “When will this be shipping in a Guix distribution near me? When can I use Goblins to boot my operating system?” As exciting as this is, we’re not quite ready for prime time, as we’ll explain below.

Remaining work

Before we deploy Shepherd onto our own systems, and especially before we try it in PID 1, we want to ensure that we reliably pass Shepherd’s existing suite of shell-based tests. Somewhere between 4 and 7 tests fail as of writing; some tests fail every time, others only intermittently, indicating there may be some subtle race conditions lurking.

One key component that remains unsupported is system log support, the lack of which accounts for a considerable chunk of the remaining test failures. Nonetheless, our branch is passing nearly all of the existing tests, which is great progress! Once the test suite issues have been sorted out, we’ll try using our Shepherd build on a real Guix system and see how stable it is over time.

There is also plenty of code to clean up. We've left all the original actor code in place during development to make rebasing on upstream less prone to conflicts, but the time has come to start removing it. There are also numerous refactors that can be done to improve the code style and readability.

Beyond a direct port, though, this work will empower the Shepherd with everything Goblins and OCapN have to offer. So how could those powers be used? Well, we've got some ideas!

Single-system unification

On Guix systems, where Shepherd serves as the init system in PID 1, it is common to run additional Shepherd instances for unprivileged users. One option is to use guix home, as mentioned above. These Shepherd instances are entirely separate from each other. Only users with access to the root user (likely via sudo) can interact with the system Shepherd; it’s all-or-nothing. It would be nice to be able to give unprivileged users access to a subset of the system services, following the principle of least authority. The object capability security model provided by Goblins makes this possible!

herd clients communicate with shepherd daemons using a custom protocol over a Unix domain socket. If the client were to be modified to speak the OCapN protocol instead, users of herd would only be able to interact with services for which they hold a capability. Consider a shared server: the system administrator could give capabilities to other users of the machine that grant access to just a subset of the available system services — and perhaps to only a subset of the available service actions. The fine-grained nature of object capabilities means that access can be scoped to the minimum necessary for each user to do what they need.

As a first step in this direction, we’ve added a prerequisite component to Goblins, a Unix domain socket netlayer for OCapN. This is necessary to have interconnected machines running Shepherd communicate over OCapN at the system layer. Read on to see an example of what this might look like!

Fleet orchestration

Moving on from Shepherd on a single machine, an OCapN-enabled Shepherd will allow for orchestration of entire server fleets. To demonstrate, let’s walk through an example scenario. Carol, a DevOps engineer, is responsible for the web servers running on a small fleet of machines named A and B. Each machine is running Shepherd with a web-server service registered. To model this scenario on a single machine, we’ll use three Goblins vats:

(define a-vat (spawn-vat #:name "Server A"))
(define b-vat (spawn-vat #:name "Server B"))
(define c-vat (spawn-vat #:name "Carol"))

Then we’ll setup some loggers to distinguish which “machine” logged which line:

(define-actor (^prefix-logger bcom prefix)
  (lambda (str)
    (format #t "~a: ~a\n" prefix str)))

(define a-output (with-vat c-vat (spawn ^prefix-logger "A")))
(define b-output (with-vat c-vat (spawn ^prefix-logger "B")))
(define c-output (with-vat c-vat (spawn ^prefix-logger "C")))

Servers A and B have identical configuration with a web-server service that depends on the networking service:

(define (spawn-networking-service)
  (spawn ^service '(networking)
         #:start-handler (const #t)
         #:stop-handler (const #f)))

(define (spawn-web-server-service)
  (spawn ^service '(web-server)
         #:requirement '(networking)
         #:start-handler (const #t)
         #:stop-handler (const #f)))

(define a-registry (with-vat a-vat (spawn ^service-registry)))
(define a-shepherd (with-vat a-vat (spawn ^shepherd a-registry)))
(define a-networking (with-vat a-vat (spawn-networking-service)))
(define a-web-server (with-vat a-vat (spawn-web-server-service)))
(with-vat a-vat
  (all-of (<- a-shepherd 'register a-networking a-output)
          (<- a-shepherd 'register a-web-server a-output)))

(define b-registry (with-vat b-vat (spawn ^service-registry)))
(define b-shepherd (with-vat b-vat (spawn ^shepherd b-registry)))
(define b-networking (with-vat b-vat (spawn-networking-service)))
(define b-web-server (with-vat b-vat (spawn-web-server-service)))
(with-vat b-vat
  (all-of (<- b-shepherd 'register b-networking b-output)
          (<- b-shepherd 'register b-web-server b-output)))

Carol would like to issue a single command to start or stop all of the web servers. To do this, Carol first acquires references to the web-server service actors on each machine. At first glance this might seem to cause a name collision problem as both services have the same name, but fear not! Carol can assign locally meaningful names to these remote services in her local Shepherd. On Carol’s machine, the remote services are registered as web-server-a and web-server-b, respectively.

;; Naive, but enough for demo purposes.
(define-actor (^exported-service bcom writer shepherd service provision)
  (extend-methods service
    ((canonical-name) (car provision))
    ((provision) provision)
    ((requirement) '())
    (start
     (lambda args
       (let-on ((status (<- service 'status)))
         (match status
           ('stopped  `(started ,(apply <- shepherd 'start service writer args)))
           ('starting `(starting ,(<- service 'running)))
           ('stopping `(stopping ,(<- service 'running)))
           ('running  `(running ,(<- service 'running)))))))
    (stop
     (lambda args
       (apply <- shepherd 'stop service writer args)))))

(define c-registry (with-vat c-vat (spawn ^service-registry)))
(define c-shepherd (with-vat c-vat (spawn ^shepherd c-registry)))
(define c-web-server-a
  (with-vat c-vat
    (spawn ^exported-service a-output a-shepherd a-web-server '(web-server-a))))
(define c-web-server-b
  (with-vat c-vat
    (spawn ^exported-service b-output b-shepherd b-web-server '(web-server-b))))

These exported services are actually proxy objects that you can think of like micro-herd clients that only control a single service. In this simplistic example, Carol can only start or stop the exported services, but it would also be possible to allow other actions to be invoked.

To conveniently orchestrate all of the remote web-server services with a single command, Carol binds them together with her own local web-server-fleet service that depends on both web-server-a and web-server-b.

(define c-web-server-fleet
  (with-vat c-vat
    (spawn ^service '(web-server-fleet)
           #:requirement '(web-server-a web-server-b)
           #:start-handler (const #t)
           #:stop-handler (const #f))))
(with-vat c-vat
  (let-on ((_ (<- c-shepherd 'register c-web-server-a c-output))
           (_ (<- c-shepherd 'register c-web-server-b c-output))
           (_ (<- c-shepherd 'register c-web-server-fleet c-output)))
    (<- c-shepherd 'start c-web-server-fleet c-output)))

Now all Carol has to do is run herd start web-server-fleet (which we simulate above with the start method call) and her local Shepherd will report the success or failure of starting all the remote web servers in the fleet! Assembling the logs from all three machines, the event log would look something like this:

A: Service networking has been started.
B: Service networking has been started.
A: Service web-server has been started.
C: Service web-server-a has been started.
B: Service web-server has been started.
C: Service web-server-b has been started.
C: Service web-server-fleet has been started.

Neat, huh?

Guix deployment over OCapN

One final idea we’ll share is for a new Guix feature: a guix deploy agent. This would be a capability-safe take on the modern DevOps practice of deploying through dedicated agents instead of generic SSH. To make this work, there would be a guix-deploy Shepherd service that runs on the target machine with a special deploy action to start the deployment process. The workstation that is invoking guix deploy would receive a capability to that service, perhaps in sturdyref form, and associate it with a Guix machine declaration. That code might look something like this:

(define my-server
  (machine
    (operating-system my-os)
    (environment ocapn-environment-type)
    (configuration (machine-ocapn-configuration
                    (sturdyref "ocapn://pubkey.tcp-tls/s/swissnum?host=example.com&port=8888")
                    (system "x86_64-linux")))))

Any volunteers interested in building this?

Wrapping up

Porting Shepherd to Goblins has been a long time coming, but we’re starting to see encouraging results! If you’d like to discuss this blog post, help us make some of the ideas described above a reality, or talk about anything else Spritely related, consider joining our community forum!

Spritely Goblins v0.16.1 released!

2025-09-03T12:00:00Z

Today we're happy to announce the release of Goblins 0.16.1. This is a small patch release for the 0.16.0 release from a few weeks ago. Unfortunately, shortly after releasing it we discovered some issues which prevented using OCapN (the peer-to-peer networking element of Goblins) with Hoot. This release resolves that along with a couple of minor bug fixes.

For more details about the changes in the release, see the NEWS file.

Bug Fixes

Fixed an issue where (goblins actor-lib io), which is heavily used by our netlayers, used current-scheduler from Fibers. This procedure is however not present in Hoot's fibers API.
Fixed an issue where multiple connections between two OCapN peers could exist if the OCapN Locator's hints differed.
Fixed an issue where multiple connections would occur between two OCapN peers due to a record hashing bug. This could not actually be reached due to the above IO actor bug.
Fixed an issue where resizing the vat event log would lose data. This was due to a bug in ring-buffer-resize! from (goblins utils ring-buffer).

Getting the release

As usual, if you're using Guix, you can upgrade to 0.16.1 by running:

$ guix pull
$ guix install guile-goblins

Otherwise, you can find the source download links on the Goblins homepage.

Get in touch!

If you're making something with Goblins or want to contribute to Goblins itself, be sure to join our community forum! We also host regular office hours where you can come and ask questions or discuss projects. For more information, see the forum. Thanks for following along and hope to see you there!

Spritely Goblins v0.16.0 released!

2025-08-07T10:45:00Z

We are excited to announce Spritely Goblins v0.16.0! This release of Goblins is faster than ever, with two major core speedups benefiting all Goblins-using programs! Furthermore, we have a brand new Unix Domain Socket netlayer, which means our OCapN protocol is now usable for efficient machine-local inter-process communication!

A new Unix Domain Sockets netlayer

Another new netlayer has come to Goblins, this time one based on Unix Domain Sockets! Unix domain sockets are ideal for communication between multiple processes running on the same machine. We think being able to use Goblins and OCapN to wire together a kind of efficient local inter-process communication is pretty neat!

Many users might be familiar with using Unix domain sockets using file paths on the system, however since the file system on Unix(-like) systems use ACLs, this can lead to security vulnerabilities via confused deputy attacks.

Our implementation uses a feature of Unix domain sockets which allow sockets to be sent and received over other sockets. We built an introduction server which you run on your system. You can think of it as a little OCaps kernel in amongst the modern ACL sea that our systems are built on today! The Unix domain socket netlayer can connect to one or multiple of these introduction servers and so long as two netlayers share the same introduction server, they can securely communicate with one another.

We look forward to seeing what you all use this new netlayer for... we have several exciting uses planned ourselves we hope to show off soon!

Speeding, speeding, speeding ahead!

When we're talking speedups in this release, we're not talking in mere single digit percentage speed boosts. No, not even double digit... keep going! Each of our two core speed-boosts bring make Goblins each improve the speed of common Goblins operations 10-20x, benefiting all Goblins-using programs!

The short version is, spawn has gotten faster, and bcom and promises have also gotten faster, all of which are core to all Goblins programs. For the interested reader, we explain more in further detail below. (This may go into more detail than many readers care for; feel free to skip past!)

Speeding up spawn by bypassing the elfs

Once upon a time, when Goblins was being created, a decision was made: debugging programs is important, and so all objects shall carry a debug name, and that debug name shall be, by default, the name of the procedure that constructed the actor! This was a sensible decision, and we believe, generally the correct one: it has served us well.

This decision was made long ago, in early days when Goblins was a Racket library, and we began to focus on speed much more after the port to Guile. But unfortunately, in Guile, asking a procedure "what is your name?" resulted in a journey to the land of elfs.

Or rather, that is all to say, calling procedure-name in Guile on every spawn, which we did for debuggability purposes, turns out to be painfully slow. And the reason it is slow is that normally procedure-name is only called when experimenting at the REPL or when printing a backtrace. While optimizing Goblins programs, we found that tracing an ordinary spawn (using Guile's lovely ,trace tool) was printing reams and reams of pages of lines of code. Guile's internal object file format is (perhaps surprising to some readers!) the very same ELF as, yes, the Executable and Linkable Format used by Linux executables! However, Guile uses this for different purposes; it turns out this format is just very well thought through, and Guile's lead dev Andy Wingo has a nice blogpost explaining why ELF was chosen. What this effectively meant is that ELF-parsing code would be executed all the time when simply trying to grab the name of a procedure while spawning an object.

What to do? Many paths were considered: We could try to optimize this code intended to be rarely-used in Guile itself, or cache the result and attach to the constructor somehow, or evaluate lazily. But each of these had problems: it was slow or otherwise complicated.

We could change spawn to be a macro, and grab the name referred to by the constructor at compile time. Alas, this had its own pitfall: this would break any case where spawn was already being used with apply.

The solution is to support both cases! Here is the new code for spawn:

;; When an actor is spawned and a name is not specified, we default to
;; the name of its constructor.  However, 'procedure-name' is very
;; slow and can involve parsing ELF for compiled code.  To speed
;; things up, we take advantage of the fact that actor constructors
;; are typically specified as identifiers in the source, so we can
;; simply use that identifier as the name.  To preserve the illusion
;; that 'spawn' is just a regular ol' procedure, there is identifier
;; syntax.
(define-syntax spawn
  (lambda (stx)
    (syntax-case stx ()
      ((_ constructor arg ...)          ; fast path
       (identifier? #'constructor)
       #'(spawn-named 'constructor constructor arg ...))
      ((_ constructor arg ...)          ; slow path
       #'(%spawn constructor arg ...))
      (id                               ; identifier syntax; also slow
       (identifier? #'id)
       #'%spawn))))

What this means is that when a Goblins program is compiled, most invocations of spawn will cleverly use the name of the constructor being passed in at compile time. But if this cannot be determined simply at compile time, or if spawn is to be invoked via apply or passed around as if it were a function, we fall back to using spawn as an ordinary procedure (ie, fall back to the internal %spawn procedure, which calls procedure-name as normal).

We still love our friends the elfs, and upon occasion, some Goblins programs might journey into elf land, should they need their help to provide a simple debugging name. But most of the time, we can be much faster now, by looking around where we are at compile time!

Become your new you, faster than ever

Previously we discussed how Goblins actors got much faster with spawn, but this is only part of an actor's journey. First, we are born, and then, we grow and change based upon experience. So it is too with Goblins actors!

When an actor is spawned, its constructor returns what will be its first behavior. But actors may change their behavior based upon experience: in response to a message, a Goblins actor may choose to bcom (pronounced "become") a new version of its behavior.

An actor having many experiences (receiving many messages) may experience a large amount of change, and thus may invoke bcom a lot. The way bcom was implemented used pretty much the same sealers/unsealers technique from the appendix of The Heart of Spritely, itself a technique borrowed from W7, the very security kernel from A Security Kernel Based on the Lambda Calculus!

This is a cool technique, and takes advantage of being able to construct new types at runtime. However, constructing new types at runtime turns out to have some overhead. The details are unimportant, but we moved to a new implementation of sealers which are functionally equivalent but use an encapsulated "cookie" comparison, which turns out to be dramatically faster... about as fast as two accessor calls and an identity-comparison invocation of eq?!

In other words, actors can now change their behavior with bcom quite quickly! And several other aspects of Goblins have gotten faster too with this new sealers technique, in particular several aspects of promises! Zoom zoom!

Getting the release

This release includes all the features detailed above as well as many bug fixes. See the NEWS for more information.

As usual, if you're using Guix you can upgrade to 0.16 by using the following:

guix pull
guix install guile-goblins

Otherwise, you can find the tarball on our release page.

The above features and speedups spoken about in this blogpost refer to the Guile version of Goblins, which is nowadays the primary version of Spritely Goblins. However, we do maintain our older Racket version, which has now also gotten updated to maintain OCapN compatibility with Guile Goblins. Racket users can run the following:

raco pkg install goblins

If you're making something with Goblins or want to contribute to Goblins itself, be sure to join our community at community.spritely.institute! We also host regular office hours where you can come and ask questions or discuss our projects, you can find information about those on our community forum. Thanks for following along and hope to see you there!

Spritely presented spirited speeches spanning the planet

2025-07-31T00:00:00Z

Over the past 6 months, Spritely has been busy bringing our message to new audiences. I thought it might be nice to compile a list for everyone to watch our talks. Christine Lemmer-Webber, the Executive Director of Spritely, has been busy giving most of these presentations, but the entire team has helped as well. The talks cover our technology, our values, our past, and our vision.

Org mode Witchcraft at Spritely

In January, five Spritely members went to the annual FOSDEM conference to talk about the organization and how we all contribute to it. The first talk was actually by me, talking about how we organize our organization, and how we manage the many whitepapers we have put together using Org Mode. I also gave a sneak-peak of the trans-bean program which can use a Magit-style menu to edit plaintext accounting ledgers.

An update from that video is that trans-bean is now available on Codeberg if you want to try it out!

Today's fediverse: a good start, but there's more to do

Unfortunately, this talk is one of those things you had to experience in person. The camera didn't capture the performance. It is still worth a listen though! Our founder, Christine Lemmer-Webber, and the Chief Technologist at Spritely, Jessica Tallon, talk about their view of the fediverse, from their perspective as two of the primary authors to the AcitivityPub spec.

Object-Capability Security with Spritely Goblins for Secure Collaboration

Juliana gave a beautiful presentation of how our shared values regarding individual rights and consent led naturally to the technical choices Spritely has made. She also gives a great overview of Ocaps which is worth watching even if you are familiar with object capability security already.

Minimalist web application deployment with Scheme

Dave is fighting an uphill battle against dependency hell, and he needs your help. Part of the solution is, of course, Guile Scheme! Spritely's scheme-to-webassembly compiler, Hoot, is now mature enough to utilize in your next web application, and Dave thinks you should try it. What you will get in return is true reproducibility and a good bootstrapping story, along with all the web APIs you're used to. He even goes on to show how reactive programming can work through webassembly and Scheme, and plenty of other goodies.

Goblins: The framework for your next project!

Jessica is the lead technologist at Spritely and brings a lot of experience to the table when it comes to defining networking standards. She is a co-author of the ActivityPub. Now, working on Goblins at Spritely, she believes the Goblins library can be used for much more than just social media.

Shepherd with Spritely Goblins for Secure System Layer Collaboration

Juliana's second talk at FOSDEM this year was about her work on bringing the distributed networking power of Goblins to the Shepherd, which is responsible for coordinating services on a Guix system. With this project well underway, system administration, across the internet, can soon be done in a capability-secure way. This talk covers the current status of the project as well as how the Plan 9 system inspired her to start.

Spritely and a secure, collaborative, distributed future

Christine gave the last talk from Spritely at FOSDEM this year, and walked through the larger concepts of Spritely and how they come together, and why we decided to make mascots for all the different components. The Spritely project plan from 6 years ago is still the current plan, despite all the work that was done in between. As more and more characters have been coming to life, we have been getting closer to fulfilling our promise of peer-to-peer application development made easy and secure.

c-base Fireside chat

Christine had an intimate conversation at Berlin's famous c-base space station with Volker Grassmuck, ranging from topics about her personal life, her experience working on the ActivityPub, and the work she is doing now at Spritely. She ends it with a powerful and hopeful message about the future of decentralized networking.

Fediforum keynote

Christine again gave an amazing talk about the values that led to Spritely, most importantly including fun and enjoyment. She talks about the differences between the fediverse and Bluesky, and how each can learn from each other, as well as our current battle against surveillance capitalism. Throughout all of this, she gives an optimistic view of what can be accomplishe through community activism.

What the future holds

We are all putting our heads down to work on delivering the promises we talked about this year so far. In the current environment, the tools we are building are more important than ever. We hope that these talks inspire you to try out our technology and read our papers, maybe even donate! And each month, you can come listen to more of us talk at our monthly Office Hours.

Have a great rest of the Summer!

Goblinville: A Spring Lisp Game Jam 2025 retrospective

2025-06-04T12:00:00Z

Spritely participates in the Lisp Game Jam to make interactive artifacts demonstrating our progress building out our tech stack. The 2025 edition of the Spring Lisp Game Jam recently wrapped up and this time around we were finally able to show off using both Hoot and Goblins together to create a multiplayer virtual world demo! Now that we’ve had a moment to breathe, it’s time to share what we built and reflect on the experience.

But first, some stats about the jam overall.

Jam stats

Out of 26 total entries, 7 were made with Guile Scheme, including ours. Of those 7, all but one used Hoot, our Scheme to WebAssembly compiler. Guile tied for first place with Fennel as the most used Lisp implementation for the jam. We’re thrilled to see that Guile and Hoot have become popular choices for this jam!

Though many entries used Hoot, our entry was the only one that used Goblins, our distributed programming framework. However, David Wilson of System Crafters gets an honorable mention because he streamed several times throughout the jam while working on a MUD built with Goblins that was ultimately unsubmitted.

Our entry was Goblinville and it was rated the 7th best game in the jam overall. Not bad!

About Goblinville

Goblinville is a 2D, multiplayer, virtual world demo. During last year’s Spring Lisp Game Jam we made Cirkoban with a restricted subset of Goblins that had no network functionality. Since then, we’ve made a lot of progress porting Goblins to Hoot, culminating with the Goblins 0.15.0 release in January that featured OCapN working in the web browser using WebSockets.

Given all of this progress, we really wanted to show off a networked game this time. Making a multiplayer game for a jam is generally considered a bad idea, but Spritely is all about building networked communities so that’s what we set out to do. Our goal was to make something of a spiritual successor to the community garden demo I made when I first joined Spritely.

What went well

First, let’s reflect on the good stuff. Here’s what went well:

Having participated in this jam a number of times, we have gotten pretty good at scoping projects down into something achievable.
Goblins made it easy to describe the game world as a collection of actors that communicate asynchronously. Initially, the entire world was hosted inside a single web browser tab. Once enough essential actors were implemented it was a simple task to push most of those actors into a separate server process. Since sending a message to a Goblins actor is the same whether it is local or remote, this change required little more than setting up an OCapN connection.
Communicating with actors over OCapN really helped with creating an architecture that separated server state from client-side input and rendering concerns. This was harder to think about with Cirkoban because there was no network separation.
The Hoot game jam template made it easy to get started quickly. It had been a year since we made our last game, so having a small template project was useful while we were refreshing our memory about the various Web APIs we needed to use.
The vast amount of freely licensed Liberated Pixel Cup (something our Executive Director Christine Lemmer-Webber organized back in her days at Creative Commons) assets allowed us to focus on the code while still having pleasing graphics that felt unified.

As a bonus, David Wilson gave Goblinville a shout out on a System Crafters stream and a bunch of people joined the server while I was online! It was a really cool moment.

What didn’t go so well

Game jams are fast paced (even though the Lisp Game Jam is more relaxed than the average jam) and not everything goes according to plan. A big part of the game jam experience is to practice adjusting project scope as difficulties arise. Issues with the project included:

Time pressure. Unfortunately, we didn’t have as much time to dedicate to this project that we would have liked. We weren’t able to start work until the Monday after the jam started, so we only had 7 days instead of 10. Also, I came down with a cold at the end of the week which didn’t help my productivity. Making something that felt as polished as Cirkoban simply wasn’t possible.
Lack of persistence for the game world. There’s still some amount of pre-planning that goes into writing actors that can persist that we didn’t have time for. Furthermore, while our persistence system is written to support incremental updates, we don’t have a storage backend that supports it yet. Each tick of the game world would trigger a full re-serialization and we felt that was too much of a performance penalty. We hope that by the next jam this will no longer be an issue.
As predicted, multiplayer increased overall complexity. What felt like a stable enough world during local testing was quickly shown to have several performance issues and bugs once it was released to the public and other people started using it. We had to restart the server once every day or so during the jam rating period (though we have resolved these issues in a post-jam update). Since we weren’t persisting the game world, each restart wiped out all registered players and the state of the map.
No client-side prediction to mask lag. For example, when you press an arrow key to move, you won’t see the player sprite move in the client until it receives a notification from the server that the move was valid. In other words, how responsive the controls feel is directly tied to server lag. A production game client would move the player immediately and fix things up later if it receives contradictory information from the server.

Post-jam updates

We did a bit of additional work after the jam was over to sand some of the roughest edges:

Re-architected the server update loop to greatly reduce message volume. Because it was simple to implement, actors in the game world were being sent a tick message at 60Hz to update their internal state. Most of the time, the actors would simply do nothing. A plant that is done growing has nothing left to do, so that’s 60 wasteful messages per second per plant. Instead, a timer system was added to schedule things to happen after so many ticks of the game world and the tick method was removed from all game objects. This greatly improved server stability, especially for worlds with lots of live objects. As of writing, we’ve had a server running for six days without any noticeable increase in lag.
Added a server event log. It was hard to see what was going on in the world during the jam rating period without being connected to the graphical client. Now the server process emits a timestamped log of every event to standard output.
Added character sprite selection. This feature just barely missed the jam submission deadline, but it’s in now! Instead of all players being the same sprite, there are now six to choose from.
Took down the public server. For the jam submission version, we had baked a URI into the itch.io client to a public server we were hosting so the game would “just work”. This was particularly important for the other participants who were rating the submitted games and giving feedback. Since the jam rating period is now over, we took down the public server. If you’re interested in trying out Goblinville, you can follow the instructions in the README to host your own server.

Also, Spritely co-founder Randy Farmer stopped by our updated Goblinville world!

Wrapping up

Goblinville turned out to be more of a tech demo than a true game, but we’re quite happy with the result. We think it’s a good demonstration of what can be built with Goblins and Hoot in a short amount of time. We hope to build on this success to create even more engaging, featureful demos in the future!

Functional hash tables explained

2025-05-21T13:00:00Z

Prologue: The quest for a functional hash table for Goblins

For those of us that use the Lisp family of programming languages, we have much appreciation for the humble pair. Using pairs, we can construct singly-linked lists and key/value mappings called association lists. Lists built from pairs have the pleasant property of being immutable (if you abstain from using setters!) and persistent: extending a list with a new element creates a new list that shares all the data from the original list. Adding to a list is a constant time operation, but lookup is linear time. Thus, lists are not appropriate when we need constant time lookup. For that, we need hash tables.

The classic hash table is a mutable data structure and one that our Lisp of choice (Guile, a Scheme implementation) includes in its standard library, like most languages. Hash tables are neither immutable nor persistent; adding or removing a new key/value pair to/from a hash table performs an in-place modification of the underlying memory. Mutable data structures introduce an entire class of bug possibilities that immutable data structures avoid, and they’re particularly tricky to use successfully in a multi-threaded program.

Fortunately, there exists an immutable, persistent data structure that is suitable: the Hash Array Mapped Trie or HAMT, for short. This data structure was introduced by Phil Bagwell in the paper “Ideal Hash Trees” (2001). HAMTs were popularized in the Lisp world by Clojure over a decade ago. Unfortunately, Guile does not currently provide a HAMT-based functional hash table in its standard library.

Instead, Guile comes with the VList, another one of Phil Bagwell’s creations which can be used to build a VHash. Goblins currently uses VLists for its functional hash table needs, but HAMTs are better suited for the task.

There are various implementations of HAMTs in Scheme floating about, but none of them seem to have notable adoption in a major Guile project. So, we thought we’d write our own that we could ensure meets the needs of Goblins, compiles on Hoot, and that might just be useful enough to send upstream for inclusion into Guile after it’s been battle tested. To that end, we recently added the (goblins utils hashmap) module. This will become the base for a future re-implementation of the ^ghash actor.

Okay, enough context. Let’s talk about HAMTs!

What the hoot is a HAMT anyway?

From the outside, HAMTs can seem rather mysterious and intimidating. That’s how I felt about them, at least. However, once I dug into the topic and started writing some code, I was pleasantly surprised that the essential details were not too complicated.

As mentioned previously, the HAMT is an immutable, persistent data structure that associates keys with their respective values, just like a regular hash table. By utilizing a special kind of tree known as a “trie” plus some nifty bit shifting tricks, HAMTs achieve effectively constant time insertion, deletion, and lookup despite all operations being logarithmic time on paper.

A trie differs from a tree in the following way: tries only store keys in their leaf nodes. If you’re familiar with binary search trees, tries aren’t like that. Instead, the key itself (or the hash thereof, in our case) encodes the path through the trie to the appropriate leaf node containing the associated value. Tries are also called “prefix trees” for this reason.

A binary tree node can have at most two children, but HAMTs have a much larger branching factor (typically 32). These tries are wide and shallow, so few nodes need to be traversed to find the value for any given key. This is why the logarithmic time complexity of HAMT operations can be treated as if it were constant time in practice.

Trie representation

To explain, let’s start by defining a trie node using a branching factor of 32. At its simplest, we can think of a trie node as a 32-element array. Each element of the trie can contain either a leaf node (a key/value pair) or a pointer to a subtrie (another 32 element array).

For example, a HAMT with 3 entries might look like this:

This is nice and simple, but it’s a little too simple. With such a large branching factor, it’s likely that many boxes in the trie will have nothing in them, like in the above example. This is a waste of space. The issue is further compounded by immutability: adding or removing a single element requires allocating a new 32 element array. For the sake of efficiency, we need to do better.

Instead, we’ll use a sparse array that only stores the occupied elements of the trie node. To keep track of which elements of the theoretical 32 element array are occupied, we’ll use a bitmap. Below is an example trie:

In the above example, bits 4 and 10 (starting from the right) are set. This means that of the 32 possible elements, only 2 are currently occupied. Thus, the size of the underlying array is 2.

To get the number of occupied elements in the trie node, we simply count the number of 1s in the bitmap. This is known as the “population count”. (We could also just check the size of the array, but this bit counting idea is about to have another important use.)

To retrieve the value at index 10, we need to perform a translation to get an index into the underlying array. To do this, we compute the population count of the bits set to the right of bit 10. There is 1 bit set: bit 4. Thus, the value we’re looking for is stored at index 1 in the underlying array. If we take a peek, we find bar → 2 there.

Thanks to this sparse storage technique, insertion/deletion operations will allocate less memory overall.

Insertion algorithm

With the trie representation out of the way, let’s walk through the algorithm to insert a new key/value pair. The insertion algorithm covers all the essential aspects of working with a HAMT.

Let’s say we want insert the mapping foo→42 into an empty HAMT. The empty HAMT consists of a single node with an empty bitmap and an empty array:

To figure out where to store the key foo, we first need to compute the hash code for it. For ease of demonstration, we’ll use 10-bit hash codes. (In practice, 32 bits or more is ideal.)

Let’s say our fictitious hash function produces the hash bits 1000010001 for foo.

Each trie node has 32 possible branches. Thus, the range of indices for a node can be represented using a 5-bit unsigned integer. What if we took the 5 most significant bits of the hash code and used that as our index into the trie? Wow, that sounds like a clever idea! Let’s do that!

So, foo gets inserted at index 16. The original trie is empty, so we’ll make a new trie with a single leaf node. To do this, we need to set bit 16 in our bitmap and create an array with just one key/value pair in it. Our output trie looks like this:

Note that we only examined 5 bits of the hash code. We only need to examine as many bits in the hash as it takes to find an empty element.

Let’s insert another key/value pair, this time for key bar and value 17. Our made-up hash code for bar is 0100100001. Repeating the process above, the most significant 5 bits are 01001, so our index is 9, another unoccupied index. Our new trie looks like this:

Because 9 < 16, the entry for bar is stored in array index 0, followed by foo.

As a last example, let’s insert a key/value pair where the most significant bits of the hash code collide with an existing entry. This time, the key is baz and the value is 66. Our made-up hash code for baz is 1000000001.

The most significant 5 bits are 10000, so our index is 16. Now things get interesting because 16 is already occupied; the first 5 bits are not enough to distinguish between the keys foo and baz! To resolve this, we’ll replace the leaf node with a subtrie that uses the next 5 bits when calculating indices. The resulting root trie node will look sort of like this:

In the figure above, subtrie is a placeholder for the new trie we need to construct to hold the mappings for foo and baz.

To create the subtrie, we just recursively apply the same algorithm we’ve already been using but with different bits. Now we’re looking at these the least significant 5 bits of the hash codes:

The index for foo is 17, and the index for baz is 1. So, our new subtrie will have bits 1 and 17 set, and contain 2 leaf nodes:

Putting it all together, the complete trie looks like this:

Each insertion creates a new trie. The new trie shares nearly all of the data with the original trie. Only the root node and the visited subtries need to be allocated afresh. This makes insertion even into large HAMTs quite efficient.

The partial hash collision described above could happen recursively, in which case the trie will grow yet another level deeper upon each iteration. The worst case scenario is that two or more keys have the same exact hash code. A good hash function and lots of hash bits will make this a very rare occurrence, but a robust insertion algorithm needs to account for this case. We’ll gloss over this edge case here, but hash collisions can be handled by “bottoming out” to a special leaf node: a linked list of the colliding key/value pairs. These lists need to be traversed linearly when necessary upon lookup or future insertions that cause more collisions. In practice, these collision lists tend to be quite short, often only of length 2.

Wrapping up

The insertion algorithm explains all of the essential elements of a HAMT and how to work with its sparse storage structure, but we’ll touch upon lookup and deletion very briefly.

Looking up the value for a key follows the same basic process as insertion, but in a read-only manner. If the hash bits point to an unoccupied element of a trie node then the search fails right then and there. If the bits point to a leaf node with a matching key, the search succeeds. If the key doesn’t match, the search fails. Finally, if the bits point to a subtrie, we recursively search that subtrie with the next set of bits.

Deletion is the most complicated operation as it involves path compression when subtries become empty. If you’ve been successfully nerd sniped by this blog post then consider this a homework assignment. Give Phil Bagwell’s paper a read! 🙂

Thanks for following along! Hope this was fun!

Hoot 0.6.1 released!

2025-05-06T12:00:00Z

We are excited to announce the release of Hoot 0.6.1! Hoot is a Scheme to WebAssembly compiler backend for Guile, as well as a general purpose WebAssembly toolchain. In other words, Scheme in the browser!

This small patch release contains bug fixes and a few new features added since the 0.6.0 release back in January. Just in time for the upcoming Lisp Game Jam!

Make a game with Hoot!

The 2025 edition of the Spring Lisp Game Jam starts on Friday, 5/9! The Lisp Game Jam is a casual, 10-day long event where participants build small games using the Lisp implementation of their choice. We’d like to encourage you to join the jam and try making a game with Hoot! Hoot is a great choice for game jams because web-based games are both easy to distribute and easy for other people to play.

To make it easy to jump right in and make a game with Hoot, we have a ready-to-go template repository on Codeberg that you can clone. It takes care of all the boilerplate and provides some useful bindings to essential web APIs like HTML5 Canvas.

Okay, game jam promotion over! Read on for the full release notes.

New features

New --bundle flag for guild compile-wasm that will automatically install runtime libraries alongside the generated binary (reflect.js, reflect.wasm, wtf8.wasm).
Integrated psyntax-based macro expander (hoot expander) into eval.
Add -g flag for controlling compiler debug level.
Changed current-module to require compilation with -g.
Changed (hoot environments) to use run-time module registry.
Added logcount and vector-move-left! procedures, part of Guile's default environment.
Added experimental (hoot time) module for date/time calculations (currently undocumented).
Internal make-port procedure now uses keyword arguments.
Keyword error constructors are now exported from Hoot's Wasm ABI.
Added external-function? and call-external to (hoot ffi).
Added quote-syntax to (hoot core-syntax) and (hoot syntax).
Documented (fibers streams) module.

Bug fixes

Fixed inexact->exact conversion for a subset of flonums.
Fixed number->string for radix 16.
Fixed current-time which was mistakenly using jiffies.
Fixed string-split when delimiter is the first character in the string.
Fixed string-utf8-length opcode emission.
Fixed define-values macro that was causing compilation errors on newer Guile Git builds.
Fixed cond-expand error condition.
Fixed Wasm linker handling of call_ref and return_call_ref.
Fixed Wasm linker handling of catch bodies.
Fixed ,wasm-eval metacommand.
Fixed NodeJS example in the manual. Thanks to Jelle Licht for the patch!
External references now have a printed representation.
Exported sleep from (fibers) to match upstream Fibers API.

Performance improvements

Optimized Wasm validation pass (validate-wasm).
Optimized Wasm linker (add-stdlib).
Optimized Wasm resolver (resolve-wasm).
Optimized Wasm emission in compiler backend (high-level-cps->wasm).

Browser compatibility

Compatible with Firefox 121 or later.
Compatible with Google Chrome 119 or later.
Compatibility with Safari/WebKit is still a bit complicated, unfortunately. We thought everything was in place when Hoot 0.6.0 was released, but then we found a bug in WebKit’s Wasm tail call implementation. This bug was fixed nearly two months ago but the fix has not yet propagated out to stable Safari and iOS releases, as far we’re aware. We verified that some of our larger Hoot programs are now working on a Git build of WebKit, so we are optimistic that Hoot will finally be supported on all major browsers soon.

Get Hoot 0.6.1!

Hoot is already available in GNU Guix:

$ guix pull
$ guix install guile-next guile-hoot

(Hoot currently requires a bleeding-edge version of Guile, hence guile-next above.)

Otherwise, Hoot can be built from source via our release tarball. See the Hoot homepage for a download link and GPG signature.

Documentation for Hoot 0.6.1, including build instructions, can be found here.

Get in touch!

For bug reports, pull requests, or just to follow along with development, check out the Hoot project on Codeberg.

If you build something cool with Hoot (for the upcoming game jam or otherwise), let us know on our community forum!

Special thanks to the MetaMask folks for funding this work!

Until next time, happy hooting! 🦉

Announcing Spritely Oaken

2025-05-05T00:00:00Z

Hi! I’m Daphne Preston-Kendal! You may remember me from such programming languages as the Revised⁷ Report on the Algorithmic Language Scheme. In this guest post, I’m going to explain the work I’m doing for Spritely, funded by NLnet: developing a new secure Scheme sublanguage for running untrusted code, like modifications to an existing app, in a safe way!

Usually, loading some random third-party code into your program is considered a security risk. Games are probably the most popular category of app these days where it’s normal for them to be moddable by adding more code, but we can see what a security disaster this is. Hardly a week goes by without yet another story of some rascal sharing a Minecraft mod pack on a forum or uploading a mod to the Steam Workshop which, in reality, exists to steal browser cookies or passwords.

If fun things like games can’t stop malicious mods, what hope do we have to build more serious moddable apps which get deployed on major server systems? What if a malicious mod got onto servers with sensitive data about thousands or millions of people?

Obviously, though, not all code is evil: probably most code isn’t, at least not deliberately. Do we really have to throw out the entire idea of running third-party modifications to our software just because some small amount of it might do bad stuff?

What code can you trust?

Here’s a pop quiz! What potential security problems could the following Scheme code cause? (Assume that we are working on a fully-conformant implementation of the R6RS, and this is evaluated with just the (rnrs base (6)) library imported.)

(lambda (x) (+ x 1))

Go ahead – it’s not a trick question!

The answer, obviously, is none! Evaluating this expression can’t access any system resources; once evaluated, you get a procedure, and the worst thing that can happen then is that someone gives it something that’s not a number (which will just safely raise an exception). This code will never steal your bank account information, eat all the cheese in your fridge, cause demons to fly out of your nose, etc.

So obviously, we can trust some code. For a simple procedure like this, we can audit it and see what it does! But for more complex code, it’s not reasonable to carefully read thousands of lines to try and find anything that looks nefarious. Attackers also keep finding new ways to hide malicious code from human auditors, from weird tricks with Unicode control characters to just putting loads of spaces at the start of lines so you have to scroll horizontally to see the evil code.

What if we could automatically audit code to be sure that it doesn’t do anything dodgy? Could that avoid human errors (like failing to notice a horizontal scrollbar)?

To do that, we have to go a level further and consider how we came to the conclusion that procedures like the one above are safe. How do we know? Well, + is a pure function: it takes some input value and returns a new, completely different output value. It doesn’t change anything about the input value: if you call ((lambda (x) (+ x 1)) sensitive-number), the value of sensitive-number itself won’t have changed when the add-one procedure is done with it. It doesn’t open any files or connect to any other remote computers, so your sensitive-number won’t be leaked to any bad guys. Also, it doesn’t allocate any memory or loop. Even a pure-functional program can do that – but at scale, that causes resource exhaustion, which could make an important service inaccessible until the tied-up resources are released. Well, we can treat any pure functional program as safe, so pretty much everything in the (rnrs base (6)) library is okay – and maybe when we run it we’ll be sure to watch it to make sure it doesn’t use too many CPU cycles.

So we’ve got our first idea – but, sorry, I’ve gotta interrupt this blog post now because Christine just sent me some code she thinks I should put in my next program. It looks pretty nifty – it’s a square root procedure written in just Scheme! Hey, you know what, let’s take a look together:

(library (dustyweb square-root)
  (export square-root)

Looks good so far!

  (import (rnrs base (6))
          (rnrs io ports (6)))

Uh-oh. Why does a square root procedure need port I/O? Square-root should be a pure function, but that library gives Christine’s code complete access to my filesystem. What’s she up to? Is she trying to read my private data? Or has thinking about all those evil Minecraft mods and how to detect them made me paranoid??

  (define (close-enough? guess x)
    (< (abs (- (* guess guess) x)) 0.01))

  (define (improve-guess guess x)
    (/ (+ guess (/ x guess))
       2))

  (define (square-root x)
    (call-with-port
        (open-file-output-port "sqrt-log.txt"
                               (file-options no-fail)
                               (buffer-mode line)
                               (native-transcoder))
      (lambda (log-port)
        (put-string log-port "finding square root of: ")
        (put-datum log-port x)
        (put-char log-port #\newline)
        (let loop ((guess 1.0))
          (put-string log-port "guess: ")
          (put-datum log-port guess)
          (put-char log-port #\newline)
          (cond ((close-enough? guess x)
                 (put-string log-port "good enough, ship it!\n")
                 guess)
                (else
                 (loop (improve-guess guess x)))))))))

Well, actually, that looks perfectly innocent as far as uses of the filesystem go. Christine probably wanted to log the guesses so she could debug her code. Still, if I already have any file called sqrt-log.txt, this will inadvertently delete and write over it. Also, I wanted to use this procedure in a program to work out how much perspective distortion I should expect with various lenses on some of the weird and wacky film formats I photograph on. (The amount of perspective distortion is proportional to a function of the difference between the focal length and the corner-to-corner diagonal of the image plane.) but what weird and wacky film formats I use is my own private business, and I don’t necessarily want it sent to Christine’s server: imagine if instead of (rnrs io ports (6)), she’d imported (srfi :106) (socket library) and sent all that data off to her server to spy on the square roots I’m calculating! I can’t just trust any code Christine sends me.

Hmm. Maybe we’ve learned something here, though – something we can use to improve our intuition about what code we can trust in general. I’m fine with recording what square roots I take, and what steps were taken to calculate them – on my own computer. Can we reformulate Christine’s library so that she can still have her debugging information, but it still only imports the (rnrs base (6)) library? Then we know we can trust her code just by looking at the import list!

How about something like this?

(library (dustyweb square-root)
  (export square-root)
  (import (rnrs base (6)))

  (define (close-enough? guess x)
    (< (abs (- (* guess guess) x)) 0.01))

  (define (improve-guess guess x)
    (/ (+ guess (/ x guess))
       2))

  (define (square-root x debug-log)
    (debug-log "finding square root of" x)
    (let loop ((guess 1.0))
      (debug-log "guess" guess)
      (cond ((close-enough? guess x)
             (debug-log "good enough, ship it!")
             guess)
            (else
             (loop (improve-guess guess x)))))))

Now when I write my perspective-distortion program, I call square-root like this:

(square-root (+ (square image-width)
                (square image-height))
             ignore)

where ignore is a procedure I wrote and trust that accepts any number of arguments, does nothing with them, and returns nothing of interest. In other words, the logging calls do nothing for me because they’re not very interesting to me. Meanwhile, when Christine is debugging her program, she can write a debug-log procedure that looks like this:

(define (make-debug-log-procedure log-port)
  (case-lambda
    ((message)
     (put-string log-port message)
     (put-char log-port #\newline))
    ((message value)
     (put-string log-port message)
     (put-string log-port ": ")
     (put-datum log-port value)
     (put-char log-port #\newline))))

(define debug-log (make-debug-log-procedure
                   (open-file-output-port "sqrt-log.txt"
                                          (file-options no-fail)
                                          (buffer-mode line)
                                          (native-transcoder))))

What happened here? Square-root can still do I/O, but suddenly we’re able to trust it because we know that the library it’s in only has access to procedures we trust. In other words, we only have to look at that one line in the program to know that we can completely trust the whole program! But, somehow, it can still do the I/O for the debug log Christine wants. The reason is that when we call it, we explicitly give it the ability to do I/O in a limited way: we write code ourselves – code we trust to access libraries like (rnrs io ports (6)) or (srfi :106) – and pass in a procedure from outside that exposes the capabilities of those libraries in a specific, limited way. Even if I did want to log what square-root is doing, using Christine’s debug-log procedure would only let it write to one specific file, and in a very specific way. It couldn’t use that ability to start reading my browser cookies like it could if I gave it the full (rnrs io ports (6)) library. Even better, if you do want to get the log, you can change where it goes just by passing a different port to make-debug-log-procedure!

Make-debug-log-procedure uses a technique called closure so that its own argument, the log-port, is available to the debug-log procedure even though it isn’t an argument to it. (We say that the debug-log procedure is closed over the log-port variable – or more informally, that it’s closed over the port itself.) Once make-debug-log-procedure returns, the debug-log procedure it creates has access to that port and exposes it in a limited way, which means the (dustyweb square-root) library doesn’t need to import (rnrs io ports (6)) any more. But Christine can still use open-file-input-port when she writes a debugging procedure like make-debug-log-port, because she obviously trusts herself to use it responsibly. In fact, anyone who actually uses her library can trust themselves to access their own files.

It would still be nice to be able to use square-root as if it were a pure function, not having to pass in this ugly debug-log argument. That gives me an idea: could we close the whole library over a procedure like debug-log? That would mean Christine could also extend her library to include all sorts of useful functions, like cube roots or inverse square roots – all of them safely debug-logging away – and I would only need to pass in the debug-log argument once when I import her library.

Now we’ve actually arrived at a much better idea than trying to audit all the third-party code we might possibly want to use. Instead of doing that, we just make sure it only uses safe procedures or ones we’ve given it!

Unfortunately, neither R6RS nor R7RS Scheme provides the ability to make an entire library into a closure. But maybe we can design a system that does have this ability – that’s Oaken!

Taming libraries to stop them eating resources

In programming security research, this kind of trick of using language features to turn an unsafe library into a safer one is called ‘taming’. Like humans tamed wolves into dogs to stop them eating their sheep programmers can tame libraries to stop them providing access to the procedures that could create a security vulnerability.

I can go down to the pet shop and buy a dog and train it to recognize me as its owner. As its owner, I’m in charge and I can do all the things the dog isn’t allowed to do in my household – sleep on the human bed, eat food from the dinner table, dig holes in the garden, etc. In security terminology, this kind of general permission is called ‘ambient authority’. But, as a treat, I could let the dog do some of those things sometimes, but only when I allow it. The dog doesn’t get the ambient authority to do these things all of the time: it’s only allowed to do it in the way I let it and when I say so.

We are in charge of our own programs and our own computers. We get to do everything, but the code we pull in from some library should only be able to do things we allow it to do. We’ve seen how we could do this by closing those libraries over procedures giving more limited access to a resource, but we should really have a standard library for systematically controlling access to the resources which could cause security problems if evil code uses them in evil ways.

We’re not going to support every use case in the initial version of Oaken: full taming is a really hard problem. (Like any red-blooded programmer, I initially thought it was going to be easy, and then all the individual little difficulties slowly started to dawn on me. In fact, I still keep realizing new problems almost every day.) But we’ll provide some very basic tamings that should be good enough to start off with: the idea of Oaken is that the library system supports taming as a general concept, so more advanced controls can follow later.

Filesystem and network abuse

It should be obvious that giving untrusted code unrestricted read/write access to the whole computer filesystem is a bad idea. In combination with unrestricted ability to connect to other computers over the internet, it’s an even worse idea.

We already saw a simple idea for taming filesystem access in the square root example, but real-world libraries might legitimately want to create and read their own files to provide some useful functionality to the trusted code which uses those libraries.

We already have a real world example of how this could work. When you upload a file to a website in your browser, the web site can’t directly read the files on your computer. Instead, your browser, which does have that unrestricted filesystem access, pops up a file selection dialogue box and asks you to choose a file. The browser then makes that file available to the website by giving it a File object which it can use to read that file – and only that file. This pattern of passing limited access to some restricted resource to untrusted code is called the ‘powerbox pattern’.

Just as you give a website access to one file, with Oaken you can close a library over a version of the file-system access library which, for example, only gives access to one directory. That means the code could never go snooping outside that directory and steal or destroy information that doesn’t belong to it. Simple!

Unfortunately, file access is really tricky to fully tame. We can easily make it so that a tamed library can only access one directory on the disk, but it’s not so easy to stop that library from writing files in that directory which would eventually eat up all space on the drive. Ideally, you’d be able to limit how much disk space untrusted code can use and not just access to individual files. But that would require looking at what special support the operating system can give us and how to make use of the tools it has to restrict storage abuse, so full taming of filesystem access – beyond restricting code’s access to a single file or directory – is probably out of scope for the initial version of Oaken.

But what about network access? Should we instantiate each library a different time, passing it in multiple times, for each server and port it wants to access? That sounds a bit annoying. Maybe it would be better to use a different mechanism for powerboxing than the one the library system provides. I’ll be thinking a lot about this and trying to come up with a workable design!

Timing attacks

Another surprising source of security issues in apps comes just from giving programs access to a simple system clock. Timing attacks can let untrusted code find out information about what’s going on inside of trusted procedures they shouldn’t be able to extract information from, or see the inner workings of; or they can let two untrusted bits of code communicate with one another when they shouldn’t. Mark S. Miller, an influential researcher in the area of secure programming systems, talks about this as ‘prisoners tapping on the pipes’ – sending messages to one another in their cells with Morse code or whatever. You really do have to be careful to ensure that all potential back-channels for unwanted communication are closed off.

Preventing this particular kind of attack seems quite simple – there’s no real taming to be done. It’s just a matter of making sure a library only has access to the system clock if you explicitly give it access when you instantiate it, which is an all-or-nothing proposition – unlike carefully controlling which individual files or directories a procedure could access and how. But something I’ve already learned in planning this project is that if something looks simple, I probably haven’t thought about it enough yet, so maybe there are more problems lurking beneath the surface here.

Exhaustion attacks

It’s not only external resources like files, the network, or the system clock which can pose a security risk. The computer’s CPU and memory are all you need for another kind of attack: a denial of service. If you tie up all the computer’s memory and processor time with nonsense, you can make a service inaccessible to people who actually need to use it. Typically, attackers find some request they can make to a service that takes massively, disproportionately more resources for the service to respond to than it does for them to make the request.

What tools do you need for this kind of attack? Could we restrict those through the library system too? Unfortunately, that approach won’t work here. In fact, all you need to cause a denial-of-service attack is lambda, as in this example that’s often shown while explaining how the fixed-point combinator is derived:

((lambda (x) (x x)) (lambda (x) (x x)))

By conjuring up two procedures which call each other forever, this cute little Scheme expression will peg one of your CPUs until you find a way to stop it. This isn’t good! But this kind of vulnerability to resource-exhaustion attack can happen entirely by accident: just the other day I was hacking on a procedure and accidentally made it loop infinitely on some input.

It’s mathematically impossible to always detect this kind of bug in advance. Moreover, even if a program eventually stops looping (and there is a useful subset of programs whose halting behaviour is provable), it might still take too much CPU to return its result. (Even just a badly behaved regular expression implementation can make resource exhaustion attacks possible!) So the best we can do is put a limit on the amount of computation an untrusted procedure is allowed to do.

Guile already includes a little library called (ice-9 sandbox) which lets you call code with limits on how much time it can take and how much memory it can allocate. Unfortunately, (ice-9 sandbox) limits wall-clock time, not actual processor resources, which is not ideal. For example, wall-clock time includes time spent waiting for user input! You don’t want your procedure getting killed for DoSing the system when in fact it’s sitting there doing nothing while a hunt-and-peck typist is writing up their life story into a text box. Worse, there’s no way to powerbox time usage with (ice-9 sandbox).

But there might be a solution: engines! Engines are a Scheme idea introduced by Kent Dybvig and Bob Hieb. The basic concept is that the system runtime keeps track of how many units of computation (‘ticks’) have been executed, and when that number exceeds a certain limit (the amount of ‘fuel’ in the engine), it pauses execution and returns the unfinished task back to whoever started it. They can then decide whether to stop running the computation, or give it more fuel and carry on.

The ticks can be measured in any unit – with the right support from the OS, you could even make it actual CPU time – but Dybvig and Hieb suggest counting procedure calls. That works well in Scheme because, once you expand all your macros, everything you could do that might cause CPU exhaustion – in particular, looping – involves a procedure call.

The Guile manual also describes a few problems with the memory limiter in (ice-9 sandbox). For now, we’ll probably have to live with those and a few more, like the fact that it’s also not possible to powerbox memory usage. Guile is getting a new garbage collector soon, thanks to Andy Wingo’s sterling work on Whippet; as the migration from BDW to Whippet progresses, maybe it will become possible to fix some of these issues in future. But here also, many operating systems (cough Linux) make things difficult: we can’t really do anything about the kernel deciding that the Guile process has had quite enough memory for tonight and that it’s time to go home.

Capability attentuation

So far we’ve only considered one level of delegation of trust: you have your dog/program, and the dog/program is only allowed to sleep in the dog bed you gave it/use only the files you gave it. But in reality, software dependencies form a much more complex graph, and it should be possible for the things you trust in some limited way to place their trust in other things in even more limited ways.

Let’s say I have my program, and within it I gave your library access to a key-value store where it gets to store up to 100 MB of data in its own namespace. Your library might want to import some other library, giving it some smaller slice of that data in turn, like 10 MB, and its own subnamespace. So your library needs to be able to take the version of the library you gave it, which is closed over the limit of 100 MB and one namespace, and close it over a different limit and a different, subordinate namespace.

At the moment, I’m still in the early stages of designing the library system, and supporting this use case seems kind of hard! It’s easy to think about a one-level system where the main, trusted program gets to instantiate every (potentially untrusted) library with some fixed limitations. It’s a lot harder to design a system where it’s easy to start with an existing instantiated, untrusted library and impose even stricter limitations on it to create another library with the same interface but even tighter controls.

Onward

I’m looking forward to hacking on Oaken over the next couple of months! Oaken is one of the more intriguing parts of the Spritely Institute’s vision for decentralized internet infrastructure. Imagine spreadsheet macros that won’t make off with your sensitive financial data; imagine not just safe mods to existing games, but even something like the ability to send a whole new multiplayer game to your friends in a chat program, who could then just run it right in place on their computer with no security fears; imagine the Shepherd where each service ran in its own capability sandbox! Oaken will enable all of this by working alongside other Spritely projects like Hoot and Goblins.

Let a thousand Oaken forests bloom!

Successful supporter drive supplements Spritely's story

2025-04-08T00:00:00Z

The Spritely team is overjoyed by the support we've received in our first ever supporter drive. More than just donations, these are individual voices which inspire us to deliver on our promise of a more secure future.

The world needs social media that isn't powered by centralized sources that don't have our best interests in mind. Spritely's mission helps make that more achievable. 🖤

- Josh Mock

Spritely is painting a picture of a hopeful, human future for people and their communication, and have a credible vision to get there. That's a rare thing, and worthy of support.

- David Anderson

My digital life is my life.

Corporations and their centralized web services exist to turn a profit. They’ll eventually do so by extracting what they can from me. And by using their services. I’ve been giving them coercive leverage!

To hold pieces of me hostage.

No mas.

- Moto A

More than five hundred people have decided to donate to Spritely so far, with more than three hundred new voices being heard during this campaign!

We easily surpassed our original goal of $80,000 with weeks to spare! Even though we fell short of our stretch goal we introduced in January, our final amount was $90,000, well in excess of what we expected.

Also, if you donated at the Silver level or higher during the campaign, your name has already been added to the credits for Cirkoban in the shiniest way we could think of. Our founder, Christine Lemmer-Webber, is busy working on doodles to send out to the Diamond tier supporters and those should be received in the mail in the near future. As part of our appreciation for your support, we are keeping the tiered reward system for future donations, regardless of funding drive, and we hope to add even more rewards in the future.

Just because the campaign is over doesn't mean we are slowing down. Just the opposite; your support has given us a second wind. Already, we've released new versions of goblins and hoot and we have even more in store for the near future. On top of this, we gave seven presentations at FOSDEM which laid out our vision and our progress and allowed us to meet with partners and supporters in person.

Despite the recent uncertainty in non-profit funding in the US, we are continuing to work against centralized interests as hard as we can, and all of your support makes that possible. Donor drive or not, we always welcome more donations and we appreciate every dollar and cent.

Thank you to everyone who made this supporter drive a success. We couldn't have done it without you!

Spritely Goblins v0.15.1 released!

2025-03-26T12:45:00Z

We are happy to announce the release of Goblins 0.15.1! This patch release includes many bug fixes, documentation fixes, and quality-of-life improvements made since the 0.15.0 release back in January.

Thank you to the Spritely community for identifying and reporting many of these issues! We’re thrilled to see more people trying Goblins for the first time. Big thanks to Jessica Tallon, who has done nearly all of the hard work sanding the rough edges for this release.

For more detail about the changes in this release, see the NEWS file.

Bug fixes

Fixed sending ghashes over CapTP that have actor references or other complex objects.
Fixed sending messages to far references in persistent vats.
Fixed thread-safety issues in CapTP garbage collection.
Fixed issue where severed connections to peers couldn’t be re-enlivened.
Fixed CapTP crossed hellos mitigation.
Fixed call-with-vat blocking the current vat’s event loop. It now returns a promise when called in this context.
Fixed prelay netlayer not reconnecting after severance.
Fixed prelay severance breaking local references.
Fixed on-sever for WebSocket netlayer on Hoot.
Fixed WebSocket error handling on Hoot.
Fixed (goblins ocapn netlayer base-port) and some of its dependencies not compiling with Hoot.
Fixed make install installing utility libraries that are only for the test suite.

Documentation fixes/improvements

Fixed documentation for spawning ^onion-netlayer and registering sturdyrefs.
Fixed the greeter example in the “Persistence Environments” section.
Added additional explanation of cooperative multitasking, event loops, and common vat pitfalls in response to user feedback.
Added example of passing an actor reference over OCapN.

Quality-of-life improvements

Added race* variant of race joiner in (goblins actor-lib joiners).
Vectors and non-list pairs (such as key/value pairs within association lists) are now serializable over CapTP.
Exported ^persistence-registry from (goblins) module.
The fake netlayer can now be halted, which is useful when testing.
Added on-sever support to the prelay netlayer.
Added a new procedure, timeout, for creating promises that resolve after a certain amount of time has passed. timeout can be found in the new (goblins actor-lib timers) module.
Suppressed warning for the override of Guile core binding spawn when (goblins) is imported.

Getting the release

As usual, if you're using Guix, you can upgrade to 0.15.1 by running:

guix pull
guix install guile-goblins

Otherwise, you can find the source tarball on our release page.

Get in touch!

For bug reports, pull requests, or just to follow along with development, check out the Goblins project on Codeberg.

If you're making something with Goblins or want to contribute to Goblins itself, be sure to join our community at community.spritely.institute! Thanks for following along and hope to see you there!

Spritely Goblins v0.15.0: Goblins in the browser!

2025-01-24T12:45:00Z

We are thrilled to announce a long-awaited goal: Goblins in the browser!!

Hoot can now be used to compile Goblins and Goblins-using applications to WebAssembly! In fact, we even have support for our distributed programming protocol, OCapN working in the browser with Goblins-on-Hoot! Goblins now compiles to Webassembly in Hoot from the same codebase used to run Goblins in Guile's own virtual machine. This is a major step forward toward getting this software in the hands of everyday users. We hope it unlocks a new wave of innovation about what secure collaboration can look like.

As part of this release, we are introducing the websocket netlayer which will allow browser-based applications to easily connect with one another!

Also, a number of performance improvements makes this release the fastest Goblins ever!

Let's get into it!

Seeing is believing: Goblins in the browser IN ACTION!

Long-time readers may remember the goblin-chat demo we've shown off several times before:

Above you can see two Guile goblin-chat programs talking with a Racket based goblin-chat program, demonstrating two completely different language runtimes communicating over OCapN via Goblins. The impressive thing about goblin-chat is that it's only 150 lines of Goblins code to implement both the user code and chatroom code for a peer-to-peer chat program which authenticates messages as they come in!

Cool enough, but didn't we say Goblins works in the browser now? Wouldn't it be cool if you could try two goblin-chat programs communicating in your browser over OCapN... right here, on this page?!

Well, behold!

Peer A

Peer B

Try sending messages between Alice and Bob!

If you don't see anything, note that this demo requires a recent browser supporting all the WebAssembly extensions used by Hoot, including Wasm GC and tail call support. Firefox and Chrome ought to just work. Safari is not expected to work properly at this time.

This is the VERY SAME 150 line goblin-chat program shown in the previous gif, but compiled to Hoot and running in the browser! (Source code for the running Webassembly code executing above; only the ^chatroom and spawn-user-controller-pair code are responsible for the chat protocol, the rest of the code is to provide a user interface.)

These messages really are being delivered through OCapN too. We have a simplified abstract netlayer which simulates the network which we're using here. However, all the ordinary OCapN protocol operations are occuring with each message transmitted from one user to the other. This isn't only real Goblins in the browser, this is real OCapN in the browser too!

More about Goblins on Hoot

If you've been paying close attention, you may have noticed that this release of Goblins was directly preceded by the Hoot 0.6.0 release. (It's also no coincidence that the Hoot 0.6.0 release art showed the Goblins mascot and the Hoot mascot talking about going on an adventure and this release's release art shows them flying together in the sky!) Hoot 0.6.0 had a large amount of work put into it which is useful far beyond Goblins, but Hoot 0.6.0 and Goblins 0.15.0 were actively developed together. This has been many months of hard work from our team, and we are proud to see it pay off. Goblins in the browser is possible at last, opening the door for a future where Spritely's technology is in the hands of everyone!

Moreover, the foreign function interface feature of Hoot allows developers to access any part of the enormous javascript ecosystem within a Goblins app. That includes reactive frameworks, data processing, and visualization libraries that we think will benefit user experiences.

WebSocket Netlayer

We are also releasing a new netlayer specifically designed for browsers and everyday use: WebSockets!

The WebSocket NetLayer is the glue between Goblins applications written for Guile and Hoot. It is currently the first NetLayer available for both and will allow communication between them!

Having two different users across two different browsers talk to each other does require an intermediary. Thankfully our prelay netlayer works well together with the websockets netlayer, allowing users across browser pages to talk to each other.

However, our prelay netlayer is called "prelay" for a reason: it's the "pre-relay" netlayer. The actual "relay" netlayer we would like to provide for Goblins' OCapN code should be properly end-to-end encrypted. The specification for how to do this still needs to be written and agreed upon by the our team and others in the OCapN group so it can be used widely across OCapN implementations.

Because the prelay is not end-to-end encrypted, prelay providers can surveil and modify communication on the wire. This is a known concern, and we are going to use this current implementation as a stepping stone to derive the correct and fully end-to-end encrypted relay netlayer we still have planned.

Speedup!

This release of Goblins has a major speedup: 1.2-2x performance improvements for all Goblins code! Even more performance improvements are planned for upcoming releases; this is just the beginning!

Your support is appreciated!

Excited about Goblins on Hoot, or any of our other projects? Consider showing your appreciation with a donation! We recently hit our initial fundraising goal of $80,000 USD and announced a stretch goal. For the stretch goal, we will use Hoot to build a learning environment for Goblins and advance Hoot’s live hacking abilities in the process.

Monthly supporters at the silver, gold, and diamond tiers will have their names featured in the credits of Cirkoban, a small web game built with Hoot, as a display of our gratitude.

Thank you to everyone who has supported us so far!

Getting the release

This release also includes a number of other bugfixes and minor features. See the NEWS file for more information!

As usual, if you're using Guix, you can upgrade to 0.15.0 by using:

guix pull
guix install guile-goblins

Otherwise, you can find the tarball on our release page.

As a little bonus, if you like the release artwork, you can get a 4k resolution version to use as a wallpaper if you like! The artwork is made in Blender and its Grease Pencil tool; you can also download the .blend file! (Observant readers of our blog may notice that this release art is a sequel to the Hoot 0.6.0 release art... in the previous artwork, the characters discussed going on an adventure together, and here they are!)

If you're making something with Goblins or want to contribute to Goblins itself, be sure to join our community at community.spritely.institute! Thanks for following along and hope to see you there!

Guile Hoot 0.6.0 released!

2025-01-22T13:00:00Z

We are excited to announce the release of Hoot 0.6.0! Hoot is a Scheme to WebAssembly compiler backend for Guile, as well as a general purpose WebAssembly toolchain. In other words, Scheme in the browser!

But first, a humble ask

If you like what Spritely is doing with Hoot or any of our other projects, please consider showing your appreciation with a donation! We recently hit our initial fundraising goal of $80,000 USD and announced a stretch goal. For the stretch goal, we will use Hoot to build a learning environment for Goblins and advance Hoot’s live hacking abilities in the process.

Monthly supporters at the silver, gold, and diamond tiers will have their names featured in the credits of Cirkoban, a small web game built with Hoot, as a display of our gratitude.

Okay, back to the show!

Highlights

Much improved weak hashtable support in (hoot hashtables). Weak key hashtables have been reimplemented in Scheme rather than using JavaScript’s WeakMap and are now iterable, as they are in Guile. Weak key hashtables now have company: Weak value and doubly weak hashtables have been added. The Guile legacy hashtable API has likewise been updated to support all forms of weak hashtables. As a bonus, we also have weak vectors thanks to a contribution from Vivianne Langdon.
Hoot now implements enough of Guile (or provides sufficient alternatives for APIs that are not web compatible) to compile nearly all of the Goblins source code. Stay tuned for another blog post about this progress!
As of December 9th, 2024, WebKit now ships with Wasm GC and tail calls enabled by default. These were the remaining blocker issues for running Hoot programs on Safari and other WebKit-derived browsers. Previously, Hoot couldn’t run on iOS because Safari is the only browser available on that platform. That said, we think there might be an issue with WebKit’s Wasm implementation. We’d like to gather more data so please let us know how things work for you on WebKit!

Try Hoot

Can I interest you in some eval?

Read on for the full release notes.

New features

Added (hoot expander) module with a port of Guile’s psyntax. This was a major undertaking by Andy Wingo but we didn’t have time to hook it up to eval for this release. Stay tuned!
Added (hoot finalization) module.
Added (hoot primitive-eval) module for evaluating macro-expanded, lowered Tree-IL.
Added (hoot environments) module.
Added (hoot modules) module.
Added (hoot weak-refs) module.
Added (ice-9 weak-vector) module. Thanks to Vivianne Langdon.
Reimplemented weak key and added weak value and doubly weak hashtables to (hoot hashtables).
Added weak hashtable support to legacy Guile hashtable API.
Added initial implementation of applicable records, undocumented for the moment while we figure out what the public API should be.
File ports now track their line/column.
syntax->datum and datum->syntax now work on the Wasm target.
Added read-syntax.
Added --user-imports flag to guild compile-wasm for user code that defines Wasm function imports.
Added support for syntax transformers on the Wasm target.
Added vector-binary-search to (hoot vectors).
Added partial, temporary SRFI-14 implementation (character sets).
Added partial implementation of inet-pton to (guile).
Switched eval to use modules as representation of environments.
Added partial implementation of while to (guile).
Replaced define-inlinable stub with real implementation.
Added string-rindex, string-trim, string-trim-both, string-trim-right, and string-reverse to (guile).
capture-stack, stack-height, and print-backtrace are now exported from (hoot error-handling).
Added any, every1, every, filter-map, and find to temporary SRFI-1 module (eventually you will just import the one from Guile).
Added &external-error exception type.
Added const to (guile).
Added issue-deprecation-warning to (guile).
Added ceiling-quotient, ceiling-remainder, ceiling/, euclidean-quotient, euclidean-remainder, euclidean/ procedures to (guile).
Added iota to (guile).
Added read-string implementation to (ice-9 rdelim).
Trimmed default set of imports to compiler to just (scheme base) to reduce compilation time.
Added target-side procedure-property scaffolding.
Added procedure-name procedure.
Partially implement R7RS exit procedure (it exits but doesn’t deal with dynamic-winds in effect.)
map and for-each now support more than two lists, at last.
Added simple terminal REPL demo to the Git repository.

Bug fixes

Fixed auxilary modules not being added to *all-instances* table when running Wasm in Hoot’s VM.
Guile no longer terminates when a Hoot module calls its internal quit function.
(hoot ports) API is now usable at expansion time.
Fixed f32.copysign and f64.copysign instructions in (wasm vm).
Fixed Fix define-exception-type when deriving exception types with fields.
Fixed cond-expand in define-library forms.
Fixed parsing of br_on_cast and br_on_cast_fail instructions in (wasm parse).
string-copy is now usable at expansion time.
Improved hashtable size lower bound calculation to avoid some bad resizing behavior.
Fixed legacy Guile hashtable API to return the unspecified value for hash-set!, hash-remove!, and hash-clear!.
Fixed Wasm generated for s64->f64 primitive.
Fixed unbounded exception raising loop when exception handler prompt is not on the current dynamic stack. This could occur when an exception handler is established before spawning a fiber, for example.
Fix compilation of %inline-wasm that returns 0 or 2+ values.
String comparison procedures are now usable at expansion time.
Fixed memory leak in library group expansion.
Fixed stream I/O implementation on Hoot VM.

Browser compatibility

Compatible with Firefox 121 or later.
Compatible with Google Chrome 119 or later.
Compatible with Safari/WebKit 18.2 or later. Finally!

Get Hoot 0.6.0!

Hoot is already available in GNU Guix:

$ guix pull
$ guix install guile-next guile-hoot

(Hoot currently requires a bleeding-edge version of Guile, hence guile-next above.)

Otherwise, Hoot can be built from source via our release tarball. See the Hoot homepage for a download link and GPG signature.

Documentation for Hoot 0.6.0, including build instructions, can be found here.

Hoot at FOSDEM

I (Dave) will be presenting “Minimalist web application deployment with Scheme” on Sunday, February 2nd in the Declarative and Minimalistic Computing devroom. Hoot will be a major focus of the presentation. This talk, like all FOSDEM talks, will be livestreamed and a recording will be available sometime after the conference.

For all the details about Spritely’s presence at FOSDEM, see our recent blog post.

Get in touch!

For bug reports, pull requests, or just to follow along with development, check out the Hoot project on Codeberg.

If you build something cool with Hoot, let us know on our community forum!

Thanks to Christine Lemmer-Webber for the lovely Hoot pixel art, and a special thanks to the MetaMask folks for funding this work!

Until next time, happy hooting! 🦉