composable virtual file systems

systemd exists, NixOS also exists… both enable you to specify what state your system should reach, rather than how it should be done.

I was familiar with the idea of creating an API exposing a virtual file system through LÖVE, which exposes bindings to PhysicsFS.

I started designing. I knew I at least needed the ability to enumerate and read files, so I needed at least these two functions:

trait Dir {
    /// List all entries under the given path.
    fn dir(&self, path: &VPath) -> Vec<VPathBuf>;

    /// Return the byte content of the entry at the given path, or `None` if the path does not
    /// contain any content.
    fn content(&self, path: &VPath) -> Option<Vec<u8>>;
}

this alone already gave me an insane amount of insight!

with these two functions, the ability to join paths, and remove their prefixes, this is enough to start building interesting things.

interestingly enough, assuming your virtual paths cannot represent parent directories, this already forms the foundation of a capability-based file system.

having a design in mind, I thought it would be interesting to integrate it into a real project. the treehouse seemed like the right thing to test it out on, since it’s effectively a compiler—it transforms a set of source directories into a target directory.

I ended up needing a few more resource forks to implement all the existing functionality.

pub trait Dir: Debug {
    /// List all entries under the provided path.
    fn dir(&self, path: &VPath) -> Vec<DirEntry>;

    /// Return the byte content of the entry at the given path.
    fn content(&self, path: &VPath) -> Option<Vec<u8>>;

    /// Get a string signifying the current version of the provided path's content.
    /// If the content changes, the version must also change.
    ///
    /// Returns None if there is no content or no version string is available.
    fn content_version(&self, path: &VPath) -> Option<String>;

    /// Returns the size of the image at the given path, or `None` if the entry is not an image
    /// (or its size cannot be known.)
    fn image_size(&self, _path: &VPath) -> Option<ImageSize> {
        None
    }

    /// Returns a path relative to `config.site` indicating where the file will be available
    /// once served.
    ///
    /// May return `None` if the file is not served.
    fn anchor(&self, _path: &VPath) -> Option<VPathBuf> {
        None
    }

    /// If a file can be written persistently, returns an [`EditPath`] representing the file in
    /// persistent storage.
    ///
    /// An edit path can then be made into an [`Edit`].
    fn edit_path(&self, _path: &VPath) -> Option<EditPath> {
        None
    }
}

one notable thing about this virtual file system is that it doesn’t allow writing to the virtual files.

it may seem pointless to want to have the treehouse—again, a compiler of sorts—capable of editing source files, but there is one legit use case: I have a fix command, which fills in all the missing branch IDs in a file, optionally saving it in place.

edit_path returns an EditPath, which represents a location somewhere in persistent storage. having an EditPath, you can construct an Edit.

/// Represents a pending edit operation that can be written to persistent storage later.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Edit {
    /// An edit that doesn't do anything.
    NoOp,

    /// Write the given string to a file.
    Write(EditPath, String),

    /// Execute a sequence of edits in order.
    Seq(Vec<Edit>),
    /// Execute the provided edits in parallel.
    All(Vec<Edit>),

    /// Makes an edit dry.
    ///
    /// A dry edit only logs what operations would be performed, does not perform the I/O.
    Dry(Box<Edit>),
}

Edits take many shapes and forms, but the most important one for us is Write: it allows you to write a file to the disk.

the other ones are for composing Edits together into larger ones.

although I said before that the fork-based virtual file system is a leaky abstraction when you introduce writing to the physical file system, I don’t think this particular API is susceptible to this—since it can expose EditPaths for entries that can actually be written (ones with a content), you can disallow writing to directories that way.

thanks to the Dir’s inherent composability, it is trivial to build adapters on top of it. I have a few in the treehouse myself.

for the curious, here’s roughly how the treehouse’s virtual file systems are structured:

source: ImageSizeCache(Blake3ContentVersionCache(MemDir {
    "treehouse.toml": BufferedFile(..),  // content read at startup
    "static": Anchored(PhysicalDir("static"), "static"),
    "template": PhysicalDir("template"),
    "content": PhysicalDir("content"),
}))

target: Overlay(
    HtmlCanonicalize(ContentCache(TreehouseDir)),
    MemDir {
        "static": Cd(source, "static"),
        "robots.txt": Cd(source, "static/robots.txt"),
    },
)

there is one flaw I want to point out with the current implementation of Dir: it uses trait methods to add new resource forks.

one idea I’ve had to fix this was to change the API shape to a single trait method.

pub trait Dir {
    fn forks(&self, path: &VPath, forks: &mut Forks);
}

impl Dir for MyDir {
    fn forks(&self, path: &VPath, forks: &mut Forks) {
        forks.insert(|| MyFork);
    }
}

impl<T> Dir for AdapterDir<T> {
    fn forks(&self, path: &VPath, forks: &mut Forks) {
        self.inner.forks(path, forks);

        forks.insert(|| MyMomsEpicSilverware);
    }
}

but that hasn’t come to fruition yet, as I have no idea how to make it efficient yet object-safe… I’m yet to add profiling to the treehouse, so I don’t want to make risky performance decisions like this at this point.