πŸ“°

SaltStack on Packet with Pulumi

It's 2791 words long and the reading time is about 13 minutes.

This article was published on July 30, 2020.

packetpulumisaltstack

In this article, I'm going to walk through using Pulumi to provision a couple of bare metal servers on Packet, using the user data to provision and configure a working SaltStack setup with a single master and single minion node.

Why Packet?

I joined Packet on July 27th, 2020 (3 days ago, at the time of writing) and I wanted a quick project that would allow me to try out the platform and see what fun I could have. Packet is a hosting provider that has commoditised bare metal compute over an API. Nice, right? πŸ’―

Why SaltStack?

I've been using and advocating SaltStack since around 2015, it's my goto tool for configuration management and provides some very cool features that aren't available in alternatives, such as beacons, the reactor, and salt-cloud.

When you're running bare metal over an API, there's a few convenience factors that aren't available to you that you'd get for vendor-lockin free on other clouds, such as ASGs; so by providing a SaltStack backbone to our compute, we can begin to put these pieces together for ourselves. The old adage of Cattle vs Pets is still aplicable in a bare metal environment, but with different applications. I'll dive into this in a future post.

Why Pulumi?

Most people reach for Terraform. Hell, I used to. Terraform is fantastic. That being said ... while HashiCorp are making good progress on evolving HCL to support more code-like features with 0.12, why not use a programming language straight up? That's what Pulumi is.

Pulumi is a Infrastructure as Code (IaaC) tool that allows you to describe your infrastructure, much like Terraform. Unlike Terraform, Pulumi doesn't impose a specific DSL, HCL, on you; and instead, you can use your programming language of choice ... provided it's supported.

At the time of writing, Pulumi supports:

  • C#
  • F#
  • Go
  • JavaScript
  • Python
  • TypeScript
  • VisualBasic

I know, I know. I'm disappinted Rust isn't there either. Maybe one day.

Walkthrough

Creating the Stack

So to create our stack, we need to generate a new Pulumi project. For this example, we'll use the Pulumi local login, which stores the statefile on our local disk. The statefile is very similar to Terraform state. It is needed to build an execution plan for our apply commands. Using local will suffice today, but you should investigate using alternative options for production deployments.

1pulumi login --local
2

We're going to use the TypeScript template to get started. Unfortunately, there isn't a template for all supported languages, but templates do exist for the following:

1packet-go # A minimal Packet.net Go Pulumi program
2packet-javascript # A minimal Packet.net JavaScript Pulumi program
3packet-python # A minimal Packet.net Python Pulumi program
4packet-typescript # A minimal Packet.net TypeScript Pulumi program
5

If you want to use one of the dotNet languages, you can use a generic template; then add the Packet provider manually. Generic templates for dotNet are called:

1csharp # A minimal C# Pulumi program
2fsharp # A minimal F# Pulumi program
3visualbasic # A minimal VB.NET Pulumi program
4

To create our project from the TypeScript template, let's run:

1pulumi new packet-typescript
2

You'll be walked through a couple of questions to create your stack, after which you'll have a directory that looks like:

1drwxr-xr-x - rawkode 30 Jul 18:07 node_modules
2.rw------- 286 rawkode 30 Jul 18:06 index.ts
3.rw-r--r-- 28k rawkode 30 Jul 18:07 package-lock.json
4.rw------- 201 rawkode 30 Jul 18:06 package.json
5.rw-r--r-- 85 rawkode 30 Jul 18:07 Pulumi.dev.yaml
6.rw------- 100 rawkode 30 Jul 18:06 Pulumi.yaml
7.rw------- 438 rawkode 30 Jul 18:06 tsconfig.json
8

If we take a look inside of index.ts, we'll see:

01import * as pulumi from "@pulumi/pulumi";
02import * as packet from "@pulumi/packet";
03
04// Create a Packet resource (Project)
05const project = new packet.Project("my-test-project", {
06 name: "My Test Project",
07});
08
09// Export the name of the project
10export const projectName = project.name;
11

This TypeScript uses the Pulumi SDKs to provide a nice wrapper around the Packet API. Hopefully it's pretty self-explanatory; you can see that it creates a new project and exports its name.

Exports are similar to Terraform outputs. We use an export when we want to make some attribute from our stack available outside of Pulumi. Pulumi provides the pulumi stack output command, which displays these exports.

We'll use these later.

Cleaning Up the Stack

This step is completely subjective, but I don't like my code just chilling in the top level directory with all the other stuff. What is this, Go? 😏

Fortunately, we can append main: src/index.ts to the Pulumi.yaml file, which tells Pulumi our entry point for this stack lives somewhere else. I'm going to use a src directory.

1mkdir src
2mv index.ts src/
3echo "main: src/index.ts" >> Pulumi.yaml
4

Creating a "Platform"

I like to create a Platform object / type / class that can be used to pass around the Pulumi configuration and some other common types that my Pulumi projects often need. This saves my function signatures getting too gnarly as we add new components to our stacks.

The Platform object I'm using for this is pretty trivial. It loads the Pulumi configuration and stores our Packet Project, which means we can pass around Platform to other functions and it's a single argument, rather than many.

01// ./src/platform.ts
02import { Config } from "@pulumi/pulumi";
03import { Project } from "@pulumi/packet";
04
05export type Platform = {
06 project: Project;
07 config: Config;
08};
09
10export const getPlatform = (project: Project): Platform => {
11 return {
12 project,
13 config: new Config(),
14 };
15};
16

Now we can update our ./src/index.ts to look like so:

01import * as packet from "@pulumi/packet";
02import { getPlatform } from "./platform";
03
04const project = new packet.Project("pulumi-saltstack-example", {
05 name: "pulumi-saltstack-example",
06});
07
08const platform = getPlatform(project);
09
10export const projectName = platform.project.name;
11

Creating the SaltMaster

Now we want to create the Salt Master server. For this, I create a new directory with an index.ts that exports a function called createSaltMaster; which I can consume in our ./src/index.ts, much like we did with our Platform.

Lets start with the complete file, then we'll break it down; sound good? Good πŸ˜€

01// ./src/salt-master/index.ts
02import {
03 Device,
04 IpAddressTypes,
05 OperatingSystems,
06 Plans,
07 Facilities,
08 BillingCycles,
09} from "@pulumi/packet";
10import { Platform } from "../platform";
11import * as fs from "fs";
12import * as path from "path";
13import * as mustache from "mustache";
14
15export type SaltMaster = {
16 device: Device;
17};
18
19export const createSaltMaster = (
20 platform: Platform,
21 name: string
22): SaltMaster => {
23 // While we're not interpolating anything in this script atm,
24 // might as well leave this code in for the time being; as
25 // we probably will shortly.
26 const bootstrapString = fs
27 .readFileSync(path.join(__dirname, "./user-data.sh"))
28 .toString();
29
30 const bootstrapScript = mustache.render(bootstrapString, {});
31
32 const saltMaster = new Device(`master-${name}`, {
33 hostname: name,
34 plan: Plans.C1LargeARM,
35 facilities: [Facilities.AMS1],
36 operatingSystem: OperatingSystems.Debian9,
37 billingCycle: BillingCycles.Hourly,
38 ipAddresses: [
39 { type: IpAddressTypes.PrivateIPv4, cidr: 31 },
40 {
41 type: IpAddressTypes.PublicIPv4,
42 },
43 ],
44 projectId: platform.project.id,
45 userData: bootstrapScript,
46 });
47
48 return {
49 device: saltMaster,
50 };
51};
52

SaltMaster Return Type

Because this is TypeScript, we want to be very explicit about the return types within our code. This allows us to catch errors before we ever run our Pulumi stack. As we're using createSaltMaster function to create our SaltMaster, we want that function to return a type with the resources we create.

1export type SaltMaster = {
2 device: Device;
3};
4

While our function only returns a Device, it's still nice to encapsulate that in a named type that allows for our function to evolve over time.

This is the function signature for createSaltMaster. You can see our return type is the type we just created, SaltMaster.

1import { Platform } from "../platform";
2
3export const createSaltMaster = (
4 platform: Platform,
5 name: string
6): SaltMaster =>
7

Our function also takes a couple of parameters, namely platform and name. The platform is our Platform object with our Pulumi configuration and the Packet Project, so we also need to import those too. The name allows us to give our SaltMaster a name when we create the device on Packet. We could hard code this inside the function as salt-master, but then we can't use createSaltMaster more than once for a highly available set up in a later tutorial. Always be thinking ahead, right? πŸ˜‰

Provisioning with User Data

01import * as fs from "fs";
02import * as path from "path";
03import * as mustache from "mustache";
04
05const bootstrapString = fs
06 .readFileSync(path.join(__dirname, "./user-data.sh"))
07 .toString();
08
09const bootstrapScript = mustache.render(bootstrapString, {});
10

I know ... I know what you're thinking ... but trust me, this will make sense.

As Pulumi allows us to use a feature complete programming language to describe our infrastructure, we also have access to that programming languages ENTIRE eco-system of libraries (npm). As such, if I want to template some user data for a server ... say, to provision and install SaltStack ... I can use a popular template tool, such as mustache, from npm πŸ˜‰ This is useful because we're not only using a programming language we're familiar with to provision our server, but we can use the same libraries we're familiar with too. It's nice when a plan comes together.

My user data for the Salt Master looks like so: The user data for installing the Salt master is pretty simple and we're not actually using the mustache interpolation yet; however, it's good to bake this in early instead of using TypeScript/JavaScript literals for the user-data. Why? Because we can keep the file saved as user-data.sh, which means we get syntax highlighting in our IDE of choice as we make changes.

01# ./src/salt-master/user-data.sh
02#!/usr/bin/env sh
03apt update
04DEBIAN_FRONTEND=noninteractive apt install -y python-zmq python-systemd python-tornado salt-common salt-master
05
06LOCAL_IPv4=$(ip addr | grep -E -o '10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')
07
08cat <<EOF >/etc/salt/master.d/listen-interface.conf
09interface: ${LOCAL_IPv4}
10EOF
11
12systemctl restart salt-master
13

Creating the Server

Lastly, we need to create the device with the Packet API. We use the Pulumi SDK to do so. I'm explicitly importing the required types that I need to use as much of the type system as I can.

Instead of hard coding strings like c1-large-arm, I can rely on enums provided by the SDK to verify that my string is correct. We do this for BillingCycles, OperatingSystems, and IpAddressTypes too. Funky! 🦩

01import {
02 Device,
03 IpAddressTypes,
04 OperatingSystems,
05 Plans,
06 Facilities,
07 BillingCycles,
08} from "@pulumi/packet";
09
10const saltMaster = new Device(`master-${name}`, {
11 hostname: name,
12 plan: Plans.C1LargeARM,
13 facilities: [Facilities.AMS1],
14 operatingSystem: OperatingSystems.Debian9,
15 billingCycle: BillingCycles.Hourly,
16 ipAddresses: [
17 { type: IpAddressTypes.PrivateIPv4, cidr: 31 },
18 {
19 type: IpAddressTypes.PublicIPv4,
20 },
21 ],
22 projectId: platform.project.id,
23 userData: bootstrapScript,
24});
25

Consuming our Function

Next up, we can call our createSaltMaster function from ./src/index.ts and we'll have a server with the correct user data for running our salt-master.

1const saltMaster = createSaltMaster(platform, "master-1");
2export const saltMasterPublicIp = saltMaster.device.accessPublicIpv4;
3

We're going to export it's public IPv4 address; so that we can access it easily later and SSH into the machine later.

Creating the SaltMinion

01// ./src/salt-minion/index.ts
02import { interpolate } from "@pulumi/pulumi";
03import {
04 BillingCycles,
05 Device,
06 Facilities,
07 OperatingSystems,
08 Plans,
09} from "@pulumi/packet";
10import { Platform } from "../platform";
11import { SaltMaster } from "../salt-master";
12import * as fs from "fs";
13import * as path from "path";
14import * as mustache from "mustache";
15
16type SaltMinion = {
17 device: Device;
18};
19
20export const createSaltMinion = (
21 platform: Platform,
22 name: string,
23 master: SaltMaster
24): SaltMinion => {
25 const bootstrapString = fs
26 .readFileSync(path.join(__dirname, "./user-data.sh"))
27 .toString();
28
29 const saltMinion = new Device(`minion-${name}`, {
30 hostname: name,
31 plan: Plans.C1LargeARM,
32 facilities: [Facilities.AMS1],
33 operatingSystem: OperatingSystems.Debian9,
34 billingCycle: BillingCycles.Hourly,
35 projectId: platform.project.id,
36 userData: master.device.accessPrivateIpv4.apply((ipv4) => {
37 return mustache.render(bootstrapString, { master_ip: ipv4 });
38 }),
39 });
40
41 return {
42 device: saltMinion,
43 };
44};
45

I'm not going to go over the createSaltMinion code like I did the master, as they're rather similar. Lets cover the major difference.

Mustache interpolation

Unlike our Salt master user-data, we need to use the mustache interpolation for the minion. This is because our minions need to know where the master is, and this requires accessing some state from the Pulumi provisioning; namely the IPv4 address of our master. For this, we use mustache template syntax:

1cat <<EOF >/etc/salt/minion.d/salt-master.conf
2master: {{ master_ip }}
3EOF
4

Instead of passing {} to the mustache render function, we now need to provide some variables. I know this looks weird, so lets explain.

1userData: master.device.accessPrivateIpv4.apply((ipv4) => {
2 return mustache.render(bootstrapString, { master_ip: ipv4 });
3}),
4

The device.accessPrivateIpv4 variable is a Pulumi Output. That means that the variable isn't available until the server has been provisioned. So we can't access that variable directly, because its an promise that the value will exist later. As such, we need to use the apply function that's provided which defers the execution of the calling code until the server has been provisioned, thus the IP address is known.

This is where most people, including me, trip up with Pulumi at the beginning. Definitely read the programming model guide, which really helps understand what is going on; as well, it provides some good tips for working with outputs and inputs.

Other than this, our code is pretty much the same as the master. Our user data is a little different, but I'm going to assume that's self-explanatory too. Here's the user data, for completeness.

01# ./src/salt-minion/user-data.sh
02#!/usr/bin/env sh
03apt update
04DEBIAN_FRONTEND=noninteractive apt install -y python-zmq python-systemd python-tornado salt-common salt-minion
05
06cat <<EOF >/etc/salt/minion.d/salt-master.conf
07master: {{ master_ip }}
08EOF
09
10systemctl restart salt-minion
11

Spinning up the Stack

Before we spin this up, we need to provide our Packet API key as we create the stack.

1pulumi stack init production
2

This will prompt you for a password which is used to encrypt your secrets, namely your Packet API key. This is because we're using a local provider for state. When you do this for your real production deployments, use the Pulumi managed service or Cloud KMS.

1pulumi config set --secret packet:authToken
2
1pulumi up
2

It'll show you a "plan" which you can now confirm and run if you're happy.

01Previewing update (production):
02 Type Name Plan
03 + pulumi:pulumi:Stack pulumi-saltstack-production create
04 + β”œβ”€ packet:index:Project pulumi-saltstack-example create
05 + β”œβ”€ packet:index:Device master-master-1 create
06 + └─ packet:index:Device minion-minion-1 create
07
08Resources:
09 + 4 to create
10
11Do you want to perform this update? [Use arrows to move, enter to select, type to filter]
12 yes
13 no
14 details
15

That's it! Accept the plan, and you'll have a Salt Master and Salt Minion running on bare metal on Packet cloud.

Confirming the Awesome

Of course, I don't expect you to take my word for it that this "just works" (it does).

Lets use the pulumi stack command to get the public IPv4 address of our Salt master.

1pulumi stack output
2

Which gives us:

1Current stack outputs (2):
2 OUTPUT VALUE
3 projectName pulumi-saltstack-example2
4 saltMasterPublicIp 147.75.81.106
5

Now we can SSH into our Salt master and confirm the setup.

1ssh -l root 147.75.81.106
2

We can check that our Salt minion is configured correctly by checking the key approval was requested.

1salt-key -L
2
3Accepted Keys:
4Denied Keys:
5Unaccepted Keys:
6minion-1
7Rejected Keys:
8

We have one unaccepted key! Awesome. We can automate the approval of that in another session. Lets accept the key and test out our Salt magic.

1$ salt-key -A minion-1
2
3The following keys are going to be accepted:
4Unaccepted Keys:
5minion-1
6Proceed? [n/Y] y
7Key for minion minion-1 accepted.
8

Lets try a test.ping, a Salt command that makes sure the minions can be contacted by the master.

1$ salt minion-1 test.ping
2
3minion-1:
4 True
5

Finally, lets actually have our minion run a command and show us the output:

01$ salt minion-1 cmd.run "ip addr"
02
03minion-1:
04 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
05 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
06 inet 127.0.0.1/8 scope host lo
07 valid_lft forever preferred_lft forever
08 inet6 ::1/128 scope host
09 valid_lft forever preferred_lft forever
10 2: enP2p1s0f2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
11 link/ether 70:10:6f:b9:eb:49 brd ff:ff:ff:ff:ff:ff
12 3: enP2p1s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
13 link/ether 70:10:6f:b9:eb:49 brd ff:ff:ff:ff:ff:ff
14 4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
15 link/ether 70:10:6f:b9:eb:49 brd ff:ff:ff:ff:ff:ff
16 inet 147.75.81.34/30 brd 147.75.81.35 scope global bond0
17 valid_lft forever preferred_lft forever
18 inet 10.80.41.131/31 brd 255.255.255.255 scope global bond0:0
19 valid_lft forever preferred_lft forever
20 inet6 2604:1380:2000:5500::1/127 scope global
21 valid_lft forever preferred_lft forever
22 inet6 fe80::7210:6fff:feb9:eb49/64 scope link
23 valid_lft forever preferred_lft forever
24

Until next time πŸ˜€