πŸ“°

Introduction to WebAssembly on Kubernetes with Krustlet

It's 1625 words long and the reading time is about 8 minutes.

This article was published on June 02, 2020.

krustletkuberneteswasiwasmwebassembly

Welcome! πŸ‘‹

This article will aim to introduce you to something wild and wonderful - running WebAssembly binaries on Kubernetes, with Krustlet.

Why?

WebAssembly is bytecode that can be executed in your browser, regardless of operating system or architecture; this has opened up a, potentially, whole new era of web programming; one that doesn't need transpiled to JavaScript.

Thus far, a few languages have trialed and tested support for compiling their code to WASM for usage within the browser. You can learn about some of these efforts by checking out these links.

  • C/C++
  • Go
  • PHP
  • Rust
  • Swift
  • TypeScript

There are more, but I think that's enough to pique your interest.

ANY language, in the browser ... that's a pretty interesting idea, right? πŸ˜‰

One thing we don't often talk about when discussing the browser ... it's pretty darn secure. Running JavaScript in your browser can't actually do much damage to your host system. This is because the JavaScript engines, such as v8 or SpiderMonkey, run the JavaScript code in a sandbox; much like modern Linux container systems, like containerd.

So what if we want to leverage WASM, in our many languages, and benefit from the sandbox out the browser ... but without the browser?

Enter, WASI and wasmtime.

WebAssembly System Interface (WASI)

WASI provides an interface for running WASM code as a process on your operating system, rather than browsers; which allows us to interact with all the things the browser doesn't: files and filesystems, Berkeley sockets, clocks, and random numbers.

WASI does this in a fashion that still gives us similiar security boundaries that we get within the browser, but through a capabilities model defined by the runtime. WASI is not a runtime, but the interface that runtimes must adhere too. This is handled by a capability-based security model.

How do capabilities work? Lets use an example. Most applications open a file by some reference, which is usually a file path on the filesystem. Any application that has that reference (file path) can interact with that file, provided the Ambient Authority allows.

What does that mean? It means that if user rawkode executes a binary, and rawkode has permissions to read a file; then that binary can read that file.

That's not ideal.

WASI uses a Capability model, rather than ambient authority. When you execute a WASI binary, it can ONLY access the files that you have given that binary the permission to.

What does that mean? Think about container images when executed, they can only access the files within the container image, or explictly bind mounted in. WASI is the same. WASI binaries can only access files if the binary is given permission to access them, usually by providing a directory root.

wasmtime

wasmtime is an implemention of WASI. You can think of wasmtime much like node. In the same way that node allowed us to execute JavaScript outside of the browser, wasmtime does that for WASM.

Krustlet

Krustlet is a Kubelet implementation for Kubernetes that supports running WASM bytecode, with the wasmtime runtime, instead of container images with a container runtime.

Getting Started

Getting a Kubernetes cluster

For today's tutorial, we're going to use minikube. minikube gives us a full Kubernetes cluster, in a VM, on our local machine. With minikube installed, create a cluster with the following command.

1minikube start
2

Building Krustlet

As this is rather experimental / early stages, you'll need to compile the Krustlet from source. Depending on your machine, this can take anywhere from 3 minutes to 10 minutes; so you make want to go make a coffee or something ⏰

1# Clone the Krustlet source code to a local directory and enter it
2git clone https://github.com/deislabs/krustlet
3cd krustlet
4
5# Using Rust's build tool, Cargo, to build the application
6cargo build
7

Bootstrapping

Now that we have the binaries needed to run our Krustlet, we need to perform some initial bootstrapping.

Bootstrapping is important, because Kubernetes restricts access to the control plane. We need to provide enough configuration for our Krustlet to register itself with the control plane, so it can become a "Node" within the cluster.

You can create this bootstrap configuration with the following command. PLEASE ensure your KUBECONFIG context is pointing to your development cluster, and not something in production.

1./docs/howto/assets/bootstrap.sh
2

You'll see some output, like so:

1secret/bootstrap-token-x4jygy created
2Switched to context "minikube".
3Context "minikube" renamed to "tls-bootstrap-token-user@kubernetes".
4User "tls-bootstrap-token-user" set.
5Context "tls-bootstrap-token-user@kubernetes" modified.
6Context "tls-bootstrap-token-user@kubernetes" modified.
7

What does this all mean? πŸ€“

First, this bootstrap script creates a secret inside of our cluster; using our default KUBECONFIG. This will be setup for you when you spin up minikube.

This secret is very similiar to a Kubelets bootstrap secret. It's all very 😴

1auth-extra-groups: # system:bootstrappers:kubeadm:default-node-token
2expiration: MjAyMC0wNy0wMlQxMzozOTo0OVo= # 2020-07-02T13:39:49Z (1 hour from bootstrap)
3token-id: eDRqeWd5 # x4jygy
4token-secret: OWR1enFtamdvM3BlaXQ3YQ== # 9duzqmjgo3peit7a
5usage-bootstrap-authentication: dHJ1ZQ== # true
6usage-bootstrap-signing: dHJ1ZQ== # true
7

Next, the bootstrap script grabs your existing KUBECONFIG, copies it, crates a new user within your cluster, modifies the context to use that new user, and stores it at ~/.krustlet/config/bootstrap.conf. This bootstrap.conf can now be used to provide authentication for our Krustlet to the control plane.

Running the Krustlet

1KUBECONFIG=~/.krustlet/config/bootstrap.conf krustlet-wasi --node-ip $(minikube ip) --port 3000 --bootstrap-file ~/.krustlet/config/bootstrap.conf
2

The first time this runs, it will generate the required TLS certificates. You'll need to approve the CSR in another terminal tab / window.

1kubectl certificate approve $(uname -n)-tls
2

Lets confirm everything looks good, we should see 2 "nodes" now.

1NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
2arch Ready agent 3m15s 0.3.0 192.168.39.37 <none> <unknown> <unknown> mvp
3minikube Ready master 57m v1.18.3 192.168.39.37 <none> Buildroot 2019.02.10 4.19.107 docker://19.3.8
4

Congratulations, you've got a Krustlet running on minikube

Building a WASI Application

Next, we're going to throw together a quick "Hello World" style application. Unfortunately, WASI hasn't defined what networking / Berkeley sockets will look like yet within WASI-land; so we can't run a web server or anything more interesting (yet).

All in good time though 😁

TL;DR

If you don't want to follow the DIY steps below, you can clone my repository and jump straight to the Building section.

1git clone https://gitlab.com/rawkode/wasi-hello-world
2

DIY

First, create a Rust project. This command generates a skeleton Rust project.

1cargo new --bin wasi-hello-world
2

You'll want to update main.rs with the following code. This code prints a message, and sleeps for 5 seconds; it then loops this process indefinitely.

01// Code from
02// https://github.com/deislabs/krustlet/blob/master/docs/intro/tutorial01.md
03use std::thread::sleep;
04use std::time::Duration;
05
06fn main() {
07 loop {
08 println!("Hello, from WASI and Krustlet!");
09 sleep(Duration::from_secs(5));
10 }
11}
12

Building

We need to enable the WASI target, which we'll do with rustup. We also need to build our application, targetting WASI as a runtime.

1# Enable wasm32-wasi as a compilation targeht
2rustup target add wasm32-wasi
3
4# Build WASI bytecode
5cargo build --target wasm32-wasi
6

Running & Publishing

When we build a WASI compatible binary, what we get is a wasm file.

1❯ ls target/wasm32-wasi/debug
2... wasi-hello-world.wasm ...
3

In order to run WASI bytecode, we use a WASI runtime; like wasmtime.

1❯ wasmtime ./target/wasm32-wasi/debug/wasi-hello-world.wasm
2Hello, from WASI and Krustlet!
3^C
4

Oooh, shiny! 🌠

However, that's not what we want to do. We want to run this on Kubernetes using our Krustlet.

In-order to do so, we need to "publish" our image to "somewhere".

Fortunately, there's wasm-to-oci which allows publishing wasm binaries to OCI compatible registries.

Note: Docker Hub and Quay do not support these artifacts at the time of writing, but GitLab, Google Cloud, and Azure Cloud do.

Sadly, we do need to build this ourselves too; fortunately, it's I've prepared the commands πŸ˜ƒ

1git clone https://github.com/engineerd/wasm-to-oci
2cd wasm-to-oci
3make
4sudo cp ./bin/wasm-to-oci /usr/local/bin/wasm-to-oci
5

Now, lets go back to our wasi-hello-world directory and push an image with our WASI binary.

1# You can use this image if you want, it's public
2wasm-to-oci push ./target/wasm32-wasi/debug/wasi-hello-world.wasm registry.gitlab.com:rawkode/wasi-hello-world:latest
3
4INFO[0008] Pushed: registry.gitlab.com/rawkode/wasi-hello-world:latest
5INFO[0008] Size: 1808571
6INFO[0008] Digest: sha256:a9617d29b859994cc4b6f5ea73f7d68096921f7225ed31e645f0dda7b27163ed
7

The Last Kilometer

OK. We're almost there. Thus far, we've:

  • β˜‘ Created a Kubernetes cluster with minikube
  • β˜‘ Cloned, compiled, and ran a Krustlet from source
  • β˜‘ Created a Rust application and compiled it to WASI
  • β˜‘ Converted our WASI binary to an OCI compatible image and pushed it to the GitLab Container Registry
  • ❌ Run our Rust WASI targetting binary on Kubernetes

OK, so now we need to schedule our WASI workload on the Krustlet. Like ALL Kubernetes resources, we need YAML for this.

Save the following to pod.yaml.

01apiVersion: v1
02kind: Pod
03metadata:
04 name: wasi-hello-world
05spec:
06 containers:
07 - name: wasi-hello-world
08 image: registry.gitlab.com/rawkode/wasi-hello-world:latest
09 imagePullPolicy: IfNotPresent
10 tolerations:
11 - key: "krustlet/arch"
12 operator: "Equal"
13 value: "wasm32-wasi"
14 effect: "NoExecute"
15

Apply this YAML file to your Kubernetes cluster.

1kubectl apply -f pod.yaml
2

πŸ’₯ DONE πŸ’₯

We've now deployed a WASI binary, via Krustlet, to our Kubernetes cluster.

Confirm the Awesome

To confirm your WASI binary is running, you can use the Kubernetes tooling to check a few things.

First, is the pod scheduled and running and do we have log output? Run the following commands.

1kubectl get pods
2kubectl logs -f wasi-hello-world
3

Awesome! I hope this helps you on your path to understanding, getting excited, and adopting WASM, WASI, and Krustlet.

If you have any questions, grab me on Twitter.

Until next time 🀘