πŸ“°

Contributing to nixpkgs

It's 2618 words long and the reading time is about 13 minutes.

This article was published on July 14, 2020.

nixnixpkgs

I've been using NixOS, on and off, for around 2 years now. It's got its challenges, which usually means I switch back to Arch after a few weeks; but not this time ... I'm not switching back.

As I stumble my way through the trials and tribulations of Nix, NixOS, and nixpkgs; I'll document my path so that others can learn from my misery. Starting right now ...

What is Nix / NixOS?

This is from the NixOS website:

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. Share your development and build environments across different machines.

NixOS is a Linux distribution with a unique approach to package and configuration management. Built on top of the Nix package manager, it is completely declarative, makes upgrading systems reliable, and has many other advantages.

Contributing a GNOME Extension to nixpkgs

In this article, I am going to walk you through the steps I've taken to contribute a GNOME Extension to the nixpkgs repository. The extension is one that I just CAN'T live without, the emoji selector 😹🌟πŸ₯°

This wasn't my first contribution to nixpkgs, I've sent 16 pull requests to nixpkgs since August 2018 ... and I still have no idea what I'm doing πŸ˜‚

OK. That's a lie, I know enough to be dangerous, but not enough to be smart.

Lets try this, one step at a time.

Step 1. Clone the Repository

Not much needs said about this, right?

1git clone https://github.com/NixOS/nixpkgs
2

Step 2. Find Something Similiar

I'm not a huge fan of reinventing the wheel. I'm certainly not going to type a bunch of Nix code that shares about 95% of it's logic with many other packages already contributed to the nixpkgs repository.

Fortunately for us, this repository is segmented really well.

First, all packages live under ./pkgs.

01drwxr-xr-x - rawkode 11 Jul 13:34 -- applications
02drwxr-xr-x - rawkode 11 Jul 13:34 -- build-support
03drwxr-xr-x - rawkode 11 Jul 13:34 -- common-updater
04drwxr-xr-x - rawkode 11 Jul 13:34 -- data
05drwxr-xr-x - rawkode 11 Jul 13:34 -- desktops
06drwxr-xr-x - rawkode 11 Jul 13:34 -- development
07drwxr-xr-x - rawkode 11 Jul 13:34 -- games
08drwxr-xr-x - rawkode 11 Jul 13:34 -- misc
09drwxr-xr-x - rawkode 11 Jul 13:34 -- os-specific
10drwxr-xr-x - rawkode 11 Jul 13:34 -- servers
11drwxr-xr-x - rawkode 11 Jul 13:34 -- shells
12drwxr-xr-x - rawkode 11 Jul 13:34 -- stdenv
13drwxr-xr-x - rawkode 11 Jul 13:34 -- test
14drwxr-xr-x - rawkode 11 Jul 13:34 -- tools
15drwxr-xr-x - rawkode 14 Jul 23:40 -- top-level
16

As you can see, there's directories for desktop applications, development stuff, games, and a few other categories.

Inside of desktops, we can see:

01drwxr-xr-x - rawkode 11 Jul 13:34 -- cdesktopenv
02drwxr-xr-x - rawkode 11 Jul 13:34 -- cinnamon
03drwxr-xr-x - rawkode 11 Jul 13:34 -- deepin
04drwxr-xr-x - rawkode 11 Jul 13:34 -- enlightenment
05drwxr-xr-x - rawkode 11 Jul 13:34 -- gnome-2
06drwxr-xr-x - rawkode 11 Jul 13:34 -- gnome-3
07drwxr-xr-x - rawkode 11 Jul 13:34 -- gnustep
08drwxr-xr-x - rawkode 11 Jul 13:34 -- lumina
09drwxr-xr-x - rawkode 11 Jul 13:34 -- lxde
10drwxr-xr-x - rawkode 11 Jul 13:34 -- lxqt
11drwxr-xr-x - rawkode 11 Jul 13:34 -- mate
12drwxr-xr-x - rawkode 11 Jul 13:34 -- pantheon
13drwxr-xr-x - rawkode 11 Jul 13:34 -- plasma-5
14drwxr-xr-x - rawkode 11 Jul 13:34 -- rox
15drwxr-xr-x - rawkode 11 Jul 13:34 -- surf-display
16drwxr-xr-x - rawkode 11 Jul 13:34 -- xfce
17

Pretty much every desktop environment there is ... and if you're shouting in your head "WHAT ABOUT MY SHITTY ESOTERIC TILING WINDOW MANAGER?" ... then I've got you; it's under ./applications/window-managers. I love i3, but it's also quite nice being able to change the volume or pair bluetooth headphones without having to put my beer down and grep some shell history.

Notice how I delicately said "shell history" and not zsh, bash, fish, or nu ... I can't be bothered getting into another shell debate; 2020's been shit enough.

Sorry, I've digressed.

Lets move forward. I've found some similiar packages. I've redacted some of the extensions below ... mostly at random; but you can see that we're in the ./pkgs/desktops/gnome-3/extensions directory and we have a fair number of example packages to use as a base for our new one.

Perfect.

1pwd
2
3/home/rawkode/Code/src/github.com/NixOS/nixpkgs/pkgs/desktops/gnome-3/extensions
4
01ll
02
03drwxr-xr-x - rawkode 11 Jul 13:34 -- appindicator
04drwxr-xr-x - rawkode 11 Jul 13:34 -- arc-menu
05drwxr-xr-x - rawkode 11 Jul 13:34 -- caffeine
06drwxr-xr-x - rawkode 11 Jul 13:34 -- dash-to-dock
07drwxr-xr-x - rawkode 11 Jul 21:39 -- dash-to-panel
08drwxr-xr-x - rawkode 11 Jul 13:34 -- paperwm
09drwxr-xr-x - rawkode 11 Jul 13:34 -- sound-output-device-chooser
10drwxr-xr-x - rawkode 11 Jul 13:34 -- topicons-plus
11

Step 3. Creating Our Package

I need to create a new directory with a default.nix and copy over my example Nix from the sample extension. I started with dash-to-panel, as I like that extension. The code for that looks like so:

01{ stdenv, fetchFromGitHub, glib, gettext }:
02
03stdenv.mkDerivation rec {
04 pname = "gnome-shell-dash-to-panel";
05 version = "31";
06
07 src = fetchFromGitHub {
08 owner = "home-sweet-gnome";
09 repo = "dash-to-panel";
10 rev = "v${version}";
11 sha256 = "0vh36mdncjvfp1jbinifznj5dw3ahsswwm3m9sjw5gydsbx6vh83";
12 };
13
14 buildInputs = [
15 glib gettext
16 ];
17
18 makeFlags = [ "INSTALLBASE=$(out)/share/gnome-shell/extensions" ];
19
20 uuid = "dash-to-panel@jderose9.github.com";
21
22 meta = with stdenv.lib; {
23 description = "An icon taskbar for Gnome Shell";
24 license = licenses.gpl2;
25 maintainers = with maintainers; [ mounium ];
26 homepage = "https://github.com/jderose9/dash-to-panel";
27 };
28}
29

Unfortunately, there was a problem. This extension uses a Makefile to document its build steps (which I recommend for ALL repositories), but unfortunately; emoji-selector doesn't ship with a Makefile 😒

So I'm going to copy the "core" sections and make up the rest from another example shortly.

What I copied looked like so:

01{ stdenv, fetchFromGitHub, glib, gettext }:
02
03stdenv.mkDerivation rec {
04 pname = "gnome-shell-emoji-selector";
05 version = "19";
06
07 src = fetchFromGitHub {
08 owner = "maoschanz";
09 repo = "emoji-selector-for-gnome";
10 rev = "${version}";
11 sha256 = "0x60pg5nl5d73av494dg29hyfml7fbf2d03wm053vx1q8a3pxbyb";
12 };
13
14 buildInputs = [ glib ];
15
16 meta = with stdenv.lib; {
17 description =
18 "This GNOME shell extension provides a searchable popup menu displaying most emojis";
19 license = licenses.gpl3;
20 maintainers = with maintainers; [ rawkode ];
21 homepage = "https://github.com/maoschanz/emoji-selector-for-gnome";
22 };
23}
24

Lets break this down.

1{ stdenv, fetchFromGitHub, glib, gettext }:
2

This first line of Nix is in almost every Nix file you'll work with. It's the imports / dependencies that our Nix script needs from the environment / runtime.

I like to think of it as similiar to JavaScript's destructing syntax; selecting only the values we need from the global list. If that's a terrible way to think about it, I'm sure someone from HackerNews will be along shortly.

1stdenv.mkDerivation rec {
2 ...
3}
4

Next up, we need to create a derivation. That's fancy functional talk for a set of values that allow for some output to be derived.

The set that we need to build our GNOME Extension contain some obvious facts:

  • GitHub Repository
  • Version / Branch
  • Dependencies (Inputs)
  • Meta Description
1 pname = "gnome-shell-emoji-selector";
2 version = "19";
3

We define the package name and the version. The version is a tag / branch name that you can get from the Git repository or the GitHub UI.

1 src = fetchFromGitHub {
2 owner = "maoschanz";
3 repo = "emoji-selector-for-gnome";
4 rev = "${version}";
5 sha256 = "0x60pg5nl5d73av494dg29hyfml7fbf2d03wm053vx1q8a3pxbyb";
6 };
7

fetchFromGitHub is a function that grabs some source code from GitHub. The owner, repo, and rev are hopefully self-explanitory. However, we also have sha256.

Nix requires builds to be idempotent. That means it uses the sha to verify that the downloaded and extracted code is what we expected. If it's different, it'll let us know and we can decide what we want to do.

Warning

Word of warning ... if the sha that you use already exists in the Nix store (meaning you copied it from another example that you had installed), then Nix bypasses the download step and reuses the local files. This means that I ... you, will install some random extension instead of getting the actual download you expected.

How Do We Get the Sha?

The simplest way is to use lots of 0's. When you try to build this derivation, Nix will complain that your sha256 doesn't match the repositories download. You can then copy the sha and update in your derivation.

Another approach is to calculate the actual sha yourself, using nix-prefetch-url.

Go to GitHub and grab the URL for the tar.gz artefact on the releases page for the revision or version that you want to add to the repository. Now you can run:

1nix-prefetch-url --unpack https://github.com/maoschanz/emoji-selector-for-gnome/archive/19.tar.gz
2
3unpacking...
4[1.0 MiB DL]
5path is '/nix/store/694kcbsz38rni0lykffv89ndivgcccks-19.tar.gz'
60x60pg5nl5d73av494dg29hyfml7fbf2d03wm053vx1q8a3pxbyb
7

Meta

This is mostly self-explanitory. Just remember to check the license of the package you're adding and update license =. I use the source code to check the correct way to reference the licenses.

1 meta = with stdenv.lib; {
2 description =
3 "This GNOME shell extension provides a searchable popup menu displaying most emojis";
4 license = licenses.gpl3Plus;
5 maintainers = with maintainers; [ rawkode ];
6 homepage = "https://github.com/maoschanz/emoji-selector-for-gnome";
7 };
8

Dependencies

Understanding the dependencies you need to build a new Nix package can be a little intimidating.

Fortunately for GNOME Extensions, these don't vary too much and our example actually had this listed.

1buildInputs = [ glib ];
2

If you need to work this out for another package and you don't know where to start? Follow these steps:

  • Clone Code
  • Enter directory
  • Create a Nix Shell (nix-shell -p depenendency1 depenendency2 ...)
  • Run Build Command

Repeat this, adding whatever you need to nix-shell -p until the build step works.

If you don't know what your dependency you need is called, search on the Nix Package page.

Example, if I need to build some code that needs Rust, Make, and bash - I'd run: nix-shell -p bash cargo gnumake rustc

To use our repository as an example, if I ran nix-shell with no dependencies and then ran ./install.sh - then our install would have failed with glib-compile-schemas command not found; which is provided by glib.

Skipped Makefile

So I said that we had a problem, the problem was that our example extension used make and our current extension doesn't. We know this because our example Nix contained:

1makeFlags = [ "INSTALLBASE=$(out)/share/gnome-shell/extensions" ];
2

and our extension repository doesn't have a Makefile, it only has ./install.sh.

Drats.

Fortunately, I looked at another example (caffeine) and came across the following code. It turns out that we can manually configure the build steps ourself. Sweet! πŸ₯ž

01 uuid = "caffeine@patapon.info";
02
03 nativeBuildInputs = [
04 glib gettext
05 ];
06
07 buildPhase = ''
08 ${bash}/bin/bash ./update-locale.sh
09 glib-compile-schemas --strict --targetdir=caffeine@patapon.info/schemas/ caffeine@patapon.info/schemas
10 '';
11
12 installPhase = ''
13 mkdir -p $out/share/gnome-shell/extensions
14 cp -r ${uuid} $out/share/gnome-shell/extensions
15 '';
16

Customizing the Build

In the code above, we've removed the makeFlags configuration that we found in the dash-to-panel Nix package, because emoji-selector doesn't have a Makefile. We've instead provided buildPhase and installPhase configuration. These two different "examples" also used slightly different inputs: buildInputs and nativeBuildInputs

Argh. What?

First, buildInputs vs nativeBuildInputs. This was tricky to track down, it's not that well documented. However, I did find the following:

nativeBuildInputs A list of dependencies whose host platform is the new derivation's build platform, and target platform is the new derivation's host platform. This means a -1 host offset and 0 target offset from the new derivation's platforms. These are programs and libraries used at build-time that, if they are a compiler or similar tool, produce code to run at run-timeβ€”i.e. tools used to build the new derivation. If the dependency doesn't care about the target platform (i.e. isn't a compiler or similar tool), put it here, rather than in depsBuildBuild or depsBuildTarget. This could be called depsBuildHost but nativeBuildInputs is used for historical continuity.

That's a mouthful. As I understand it, we use nativeBuildInputs when we expect the inputs to be build for the platform on our local machine; and can use buildInputs for when we don't have such a constraint. I definitely need to understand this more and I'll write more on this parameter soon; once I've done some more digging πŸ˜ƒ

Next, those phases!

From the documentation; we can see that there's many phases in a build:

$prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase installCheckPhase $preDistPhases distPhase $postPhases.

Each of these are called in this order, unless specifically overridden by the Nix package by providing phases = [].

We need to override the buildPhase because by default it runs make.

The default buildPhase simply calls make if a file named Makefile, makefile or GNUmakefile exists in the current directory (or the makefile is explicitly set); otherwise it does nothing.

This is also true for installPhase, only it tries to run the install target.

The default installPhase creates the directory \$out and calls make install.

It's a little less magic when you break it down; right? πŸ§™

Testing Our Package

Using the commands inside of ./install.sh, I was able to piece together the config we needed for our extension to be successfully build and installed by nix.

The only "gotcha" here is that Nix provides $out variable that provides the directory our package should install things into.

01{ stdenv, fetchFromGitHub, glib, gettext }:
02
03stdenv.mkDerivation rec {
04 pname = "gnome-shell-emoji-selector";
05 version = "19";
06
07 src = fetchFromGitHub {
08 owner = "maoschanz";
09 repo = "emoji-selector-for-gnome";
10 rev = "${version}";
11 sha256 = "0x60pg5nl5d73av494dg29hyfml7fbf2d03wm053vx1q8a3pxbyb";
12 };
13
14 uuid = "emoji-selector@maestroschan.fr";
15
16 nativeBuildInputs = [ glib ];
17
18 buildPhase = ''
19 runHooks preBuild
20 glib-compile-schemas ./${uuid}/schemas
21 runHooks postBuild
22 '';
23
24 installPhase = ''
25 runHook preInstall
26 mkdir -p $out/share/gnome-shell/extensions
27 cp -r ${uuid} $out/share/gnome-shell/extensions
28 runHook postInstall
29 '';
30
31 meta = with stdenv.lib; {
32 description =
33 "This GNOME shell extension provides a searchable popup menu displaying most emojis";
34 license = licenses.gpl3Plus;
35 maintainers = with maintainers; [ rawkode ];
36 homepage = "https://github.com/maoschanz/emoji-selector-for-gnome";
37 };
38}
39

So how do we test this to make sure it works?

We can install it 😁

To do that, we need to cd into the nixpkgs directory; then install our package using nix-env and the package name we declared above.

1cd ~/Code/src/github.com/NixOS/nixpkgs
2nix-env -f $(pwd) -i gnome-shell-emoji-selector
3

Assuming all goes to plan, you'll see something like:

1installing 'gnome-shell-emoji-selector-19'
2nix-env -f $(pwd) -i gnome-shell-emoji-selector 5.30s user 0.43s system 97% cpu 5.873 total
3

πŸŽ‰πŸŽ‰πŸŽ‰ We did it! πŸŽ‰πŸŽ‰πŸŽ‰

Our package installed. Well done. Fire open GitHub and submit a PR.

What's Next?

There's still a fair amount to cover. When you submit a PR, you'll be presented with this checklist:

01- [ ] Tested using sandboxing ([nix.useSandbox](https://nixos.org/nixos/manual/options.html#opt-nix.useSandbox) on NixOS, or option `sandbox` in [`nix.conf`](https://nixos.org/nix/manual/#sec-conf-file) on non-NixOS linux)
02- Built on platform(s)
03 - [x] NixOS
04 - [ ] macOS
05 - [ ] other Linux distributions
06- [ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests))
07- [ ] Tested compilation of all pkgs that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review wip"`
08- [x] Tested execution of all binary files (usually in `./result/bin/`)
09- [ ] Determined the impact on package closure size (by running `nix path-info -S` before and after)
10- [ ] Ensured that relevant documentation is up to date
11- [ ] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md).
12

We've only covered 2 measely steps. In the coming articles, we'll look at:

  • Sandbox
  • macOS Builds
  • Nixpkgs on Arch Linux

Until next time