Compiling Go to WASM

Background #

Last year I wrote a type-checker for Golang (written in Go) whose aim is to expand the language's type inference capabilities by using bidirectional typechecking. (It's just a small "what if" experiment.)

It was my first experience with that technique. In retrospect, I think I did a decent job but I am sure I did not grasp the concept fully at the time. For example, I would've liked to split my AST in the way that Conor McBride describes in this talk on type systems.

As I developed the type-checker, I found myself repeatedly clearing my terminal screen and re-running the tool to check its output. My write-run-debug loop was inefficient. So I put together a tiny web app.

The web app consisted of a frontend and a backend. The frontend makes use of Monaco Editor to display two code editors side by side (a la Compiler Explorer) -- one to edit the source Go code and one to display the tool's output. The backend receives POST /compile requests from the frontend every time the user edits the source code; the input is the user's code as text, and the output is the type-checker's internal logs.

Admittedly, I spent too much time on this (I have a weak spot for webdev), but it worked out pretty well.

...and it wasn't long before I wanted to share my work with friends and colleagues 😄. So I used Google Cloud Run's free tier to build and deploy the app, which gave me access to a public URL for the app.

For the time being, I was satisfied with all the work that I did. And I moved on.

Fast forward to a few weeks ago...

During the holidays, I had enough time and energy to finally pour some love into my personal website. It had been neglected for years, and one of the reasons for such neglect was my annoyance of working with Ruby and Jekyll. I just found it clunky and uninteresting.

On my search for alternative static site generators, I found Lume. It is a dead-simple, batteries-included framework that just works. But more importantly, it just makes sense. I was instantly delighted, so I spent the following days moving every site that I own (mainly GitHub Pages) to Lume. I had lots of fun and was extremely productive using Deno, Lume, TypeScript, and Preact.

And then it hit me: What if I don't need a backend for my type-checker web app? Could I also turn it into a statically-built website?

The answer was yes. I was able to replace the backend call with a direct call to the type-checker compiled as WASM. That allowed me to get rid of the Google Cloud deployment and simply build the app with Lume and deploy it for free to GitHub Pages.

So, that's the why. Now, let's focus on the how.

Compiling and running #

(Note: I'm currently running Go version 1.23.4.)

Compiling Go to WASM (docs) is pleasantly simple and easy:

GOOS=js GOARCH=wasm go build -o build/main.wasm github.com/garciat/gobid/cmd/main

To my surprise, the command ran successfully on the first try without any changes at all to the codebase. Props to the Golang dev team.

In order to execute the code, the documentation asks us to copy the file "$(go env GOROOT)/misc/wasm/wasm_exec.js" into our project and to import it as a helper script:

<script src="wasm_exec.js"></script>
<script>
  const go = new Go();
  WebAssembly.instantiateStreaming(fetch("main.wasm"), go.importObject)
    .then((result) => {
      go.run(result.instance);
    });
</script>

I did so, and it worked. I saw the following logged into the developer console:

wasm_exec.js:22 === CheckPackageDeclarations ===
wasm_exec.js:22 === CheckPackageCycles ===
wasm_exec.js:22 === ResolvePackageNames(main) ===
wasm_exec.js:22 === Loading Package (main) ===
wasm_exec.js:22 === Checking Package (main) ===
wasm_exec.js:22 === Verifying Package (main) ===
wasm_exec.js:22 === Verify ===
wasm_exec.js:22 === iteration 0 ===
wasm_exec.js:22 Context:
{} || {}
wasm_exec.js:22 Steps:
wasm_exec.js:22 === DONE ===
wasm_exec.js:22 {{  }}

But, wait, there's more.

My tool's main executable is meant to be called with file paths passed as commandline arguments. For example:

go run github.com/garciat/gobid/cmd/main hello.go

I needed to be able to:

1. Provide program arguments. (os.Args)
2. Provide readable files.
3. Capture stdout into a JS string, instead of console.log.

I could'ved skipped points 1 and 2 by changing the program to read from stdin instead. That would've been good enough for an MVP. But for whatever reason I chose to stick to its original behavior. How hard could that be?

Providing program arguments #

This is actually easy to do:

const go = new Go();
go.argv.push("hello.go");
// ...

But how did I figure that out? Go's official documentation for WebAssembly does not document the interface of the Go object provided by the support library.

Our only options are to either read the file itself to infer its API, or to read the code of one of the many provided example apps.

As we'll see further on, this will not be our last problem with underspecification.

Providing readable files #

The wasm_exec.js support file (link) gives us a hint as to how filesystem access works in Go-WASM:

if (!globalThis.fs) {
  let outputBuf = "";
  globalThis.fs = {
    constants: {/** ... */},
    writeSync(fd, buf) {
      // ...
    },
    // Several more methods like:
    // write, chmod, chown, etc.
  };
}

That piece of code seems to be defining a Node.js-like filesystem interface. But there are essentially zero direct source references to fs in the wasm_exec.js file itself. How is this being used then?

It turns out that the Go standard library file src/syscall/fs_js.go (link) reads the fs global variable like so:

// NOTE: some lines were redacted away for brevity

package syscall

import (
  "syscall/js"
)

var jsFS = js.Global().Get("fs")
var constants = jsFS.Get("constants")

func Ftruncate(fd int, length int64) error {
  _, err := fsCall("ftruncate", fd, length)
  return err
}

func fsCall(name string, args ...any) (js.Value, error) {
  type callResult struct {
    val js.Value
    err error
  }

  c := make(chan callResult, 1)
  f := js.FuncOf(func(this js.Value, args []js.Value) any {
    var res callResult

    if len(args) >= 1 { // on Node.js 8, fs.utimes calls the callback without any arguments
      if jsErr := args[0]; !jsErr.IsNull() {
        res.err = mapJSError(jsErr)
      }
    }

    res.val = js.Undefined()
    if len(args) >= 2 {
      res.val = args[1]
    }

    c <- res
    return nil
  })
  defer f.Release()
  jsFS.Call(name, append(args, f)...)
  res := <-c
  return res.val, res.err
}

From this, we may infer the following:

• Go expects a global name fs.
• Go redirects filesystem syscalls to method calls on fs.
• The syscall API mimicks Node.js's fs module.
  • Go relies on specific overloads of each function.
  • For example: Node.js's fs.ftruncate (link) may be called with fd, callback or with fd, len, callback, but Go only uses the fd, len, callback overload.
• All syscalls are callback-based.
  • The Go callback specifically expects null when there is no error. I.e. undefined or some other falsey value will not do.

A Google search for "nodejs fs on browser" eventually led me to ZenFS, a browser-compatible filesystem emulation library that is compatible with Node.js's fs API.

I tried doing the following:

import { fs } from "https://esm.sh/@zenfs/core";
globalThis.fs = fs;
const go = new Go();
fs.writeFileSync("main.go", "package main\n");
go.argv.push("main.go");

But it did not work right away. After hours of debugging, I found that I needed to write quite a bit of custom code to get ZenFS to work with Go's implicit expectations and in order to work around some of ZenFS's strange behavior.

In summary, I defined a GoFileSystemAdapter class that wraps a ZenFS FileSystem object and exposes it as the Node.js-like fs API that Go expects.

Capturing stdout #

ZenFS itself has no support for any of the standard streams. I ended up manually creating /dev/stdout and /dev/stderr files and redirecting operations on file descriptors 1 and 2 respectively.

I did not spend time on enabling /dev/stdin, though, because I did not need it.

I also could've implemented a custom ZenFS Device that more accurately mimicks the usual stdin and stdout behavior. But the above was sufficient for my use case.

Upon calling GoFileSystemAdapter#finalize(), I close and fully read those special files and return their contents as JS strings.

Updating wasm_exec.js to JS modules #

I also spent some time modifying wasm_exec.js to make it easier to import as a JS module. More specifically, I made the following changes:

I converted the file extension .ts for TypeScript support. This allowed me to explicitly define the expected API of the filesystem FFI. For example:

export interface GoFileSystemError extends Error {
  readonly errno: number;
  readonly code: string;
}

export type GoFileSystemCallback<T> = (
  err: GoFileSystemError | null,
  value?: T,
) => void;

export interface GoFileSystemFFI {
  // ...

  write(
    fd: number,
    buf: Uint8Array,
    offset: number,
    length: number,
    position: number | null,
    callback: GoFileSystemCallback<number>,
  ): void;

  // ...
}

Then, I changed the constructor of new Go(...) to take in fs and process as parameters instead of reading them from globalThis; and I used export class Go instead of globalThis.Go = class Go.

Sadly, because Go's dependency on globalThis.fs happens in the standard library, it would've been a pain to change this. I had previously hacked on the Go compiler, but I did not want to go down this path at the time. (I will consider it in the future.)

Finally, I renamed the file to wasm_exec@1.23.4.ts in order to make it clear that it is tied to Go version 1.23.4.

Putting it all together #

In the end, my 'compile' function ended up looking like this. I named the function gobid, because that's the name I chose for the type-checker back when I started it (it is a portmanteau of Go and Bidirectional).

import { createFS } from "./go-fs.ts";
import { process } from "./go-process.ts";
import { Go } from "./wasm_exec@1.23.4.ts";

type Path = string;
type GoSourceCode = string;

export async function gobid(
  inputs: Record<Path, GoSourceCode>,
): Promise<string> {
  const fs = await createFS();

  const go = new Go(fs, process);

  for (const [path, source] of Object.entries(inputs)) {
    fs.writeFileSync(path, source);
    go.argv.push(path);
  }

  const source = await WebAssembly.instantiateStreaming(
    fetch(import.meta.resolve("../build/main.wasm")),
    go.importObject,
  );

  await go.run(source.instance);

  const { stdout, stderr } = fs.finalize();

  if (stderr) {
    return stderr;
  }

  return stdout;
}

Although globalThis shenanigans do end up happening, at the very least those details do not leak into the client code.

Check it out #

You can access the web app here: https://garciat.com/gobid/

Conclusions #

Go's WASM support works well, but it is sorely underspecified. It would be great if its JS FFI contract were strictly specified in their documentation and, ideally, with TypeScript interfaces (or at least with JSDoc types).

Go's filesystem JS FFI reliance on a global fs object makes it hard to isolate WASM instances. As an alternative, the syscall code could use the Go object instance itself as a top-level scope and read fs and other FFI dependencies off it directly. GitHub issue #56084 proposed a similar approach back in 2022, but no changes have been landed since.

wasm_exec.js does not leverage modern JavaScript features like modules (aka ESM), and is otherwise too coupled to Node.js API expectations. Also, rather than copying the file off the local Go installation, it would be more convenient to be able to download or import it from NPM or JSR. For example:

import { Go } from "jsr:@go/wasm-ffi@1.23.4"

Finally, implementing a fully-compliant filesystem for the Go JS FFI is non-trivial and there are no ready-made solutions available, as far as I can tell.