Most documentation for using nix with Haskell concentrates on quite simple use cases such as how to modify or add new dependencies. What about when you want to do something more complicated? In this post we look at combining modifications from different sources. The goal is to create a package set which can be used to compile packages with HEAD.

The main problem which we are going to solve is that many packages need modifying in order to compile with HEAD. There is already a nix package which builds a development version of the compiler and a corresponding package set but the utility is limited as many packages will fail to compile.

For example, at the time of writing this post, if we try and build the primitive package, the build fails due to a missing Semigroup instance.

If you are impatient, here is the complete example we are going to come up with. We create a new top-level attribute new-ghcHEAD which we can use to build packages with ghcHEAD.

let
  nixpkgs = import <nixpkgs> {};
  patchRepo =
    nixpkgs.fetchFromGitHub {
      owner = "mpickering";
      repo = "head.hackage";
      rev = "767dcf11b367ccff4a9fcd55df9c2432cd485fbe";
      sha256 = "1cdrcw7grpc2dyxnb7a5bg9mi1h7qnblcibi91s348034zf7a0vj";};

  patchDir = "${patchRepo}/patches";
  patchScript = "${patchRepo}/scripts/overrides.nix";
in
self: super:
{
  haskellPatches = self.callPackage patchScript
              { patches = patchDir; };

  new-ghcHEAD =
    let ghcPackageOverrides = self.callPackage self.haskellPatches {};
        localOverrides      =
          sel: sup: { haskell-src-exts
                        = sel.callHackage "haskell-src-exts" "1.20.1" {};
                      mkDerivation
                        = drv: sup.mkDerivation (drv //
                             { jailbreak = true; doHaddock = false;});
                     };

    in self.haskell.packages.ghcHEAD.extend
         (self.lib.composeExtensions localOverrides ghcPackageOverrides);
}

A simple overlay

We are going to use an overlay in order to define the new package set. An overlay is the idiomatic way to extend and change the top-level nixpkgs set. It is a function which takes two arguments, the result of performing the modification and the old unmodified package set. It returns a set of attributes which augment the top-level attribute set.

self: super: {
  my-cool-new-attribute = ...;
}

As such my-cool-new-attribute will be in self but not in super. If we want to use my-cool-new-attribute in another attribute we can refer to it as self.my-cool-new-attribute.

When this overlay is installed (see below), we can use my-cool-new-attribute like any top-level attribute defined in nixpkgs. The overlay mechanism surpasses the old packageOverrides mechanism which used to be how to modify nixpkgs.

For example, we could enter an environment with my-cool-new-attribute available with nix-shell -p my-cool-new-attribute as if it were defined in the main nixpkgs repo.

There are two ways to easily install an overlay.

Pass it as an argument when importing nixpkgs.
let pkgs = import <nixpkgs> {overlays = [overlay1 overlay2 ..]}; in ...
Place the overlay in ~/.config/nixpkgs/overlays/

More details about using overlays can be found in the user manual.

Extending a package set

We’ll start from a simple template which demonstrates the concept of extending a Haskell package set like you might find in other documentation. We start from the ghcHEAD package set (haskell.packages.ghcHEAD) and then call the extend attribute with the extension function.

The extension function takes two arguments, the extended haskell package set and the original haskell package set much like an overlay. ghcPackageOverrides is the identity extension function in this example which performs no modifications.

self: super: {
  new-ghcHEAD =
    let ghcPackageOverrides = sel: sup: {};
    in self.haskell.packages.ghcHEAD.extend ghcPackageOverrides ;
}

If we then want to fix a specific version of a package we can modify ghcPackageOverrides like so:

self: super: {
  new-ghcHEAD =
    let ghcPackageOverrides =
          sel: sup: {
            haskell-src-exts =
              sup.callHackage "haskell-src-exts" "1.20.1" {};
          };
    in self.haskell.packages.ghcHEAD.extend ghcPackageOverrides;
}

When using the new-ghcHEAD package set, we will now install haskell-src-exts-1.20.1 rather than haskell-src-exts-1.19.1 if it is needed as a dependency.

More extensions

Manually redefining each package to get it to build is tedious. Instead, we’re going to use the patches from head.hackage in order to generate a lot of modifications automatically. What we need are the patches in head.hackage/patches which enable packages to build with the bleeding edge version of the compiler.

In head.hackage/scripts there are the necessary files in order to use the patches with nix. overrides.nix is a wrapper which runs a haskell executable in order to generate a nix expression explaining the package modifications. The result of the derivation is a file which we can then import into our overlay directly.

Using a local copy of `head.hackage`

You will probably want to clone head.hackage locally so that you can add and modify patches as necessary to get your project to compile.

In that case, instead of specifying a url, we just specify the correct paths for the packaging script and patch directory.

For example:

self: super: {
  haskellPatches = self.callPackage <path-to-overrides.nix>
              { patches = <path-to-patch-dir>; };

  new-ghcHEAD =
    let ghcPackageOverrides = self.callPackage self.haskellPatches {};
    in self.haskell.packages.ghcHEAD.extend ghcPackageOverrides;
}

Using a remote copy of `head.hackage`

If you don’t want to maintain a local clone of the repo then you can also fetch it from wherever you like. For example, here I fetch a specific branch from my clone of the repo which contains some nix specific patches.

let
  nixpkgs = import <nixpkgs> {};
  patchRepo = nixpkgs.fetchFromGitHub {
    owner = "mpickering";
    repo = "head.hackage";
    rev = "767dcf11b367ccff4a9fcd55df9c2432cd485fbe";
    sha256 = "1cdrcw7grpc2dyxnb7a5bg9mi1h7qnblcibi91s348034zf7a0vj";};

  patchDir = "${patchRepo}/patches";
  patchScript = "${patchRepo}/scripts/overrides.nix";
in
self: super:
  {
  haskellPatches = self.callPackage patchScript
              { patches = patchDir; };

  new-ghcHEAD =
    let ghcPackageOverrides = self.callPackage self.haskellPatches {};
    in self.haskell.packages.ghcHEAD.extend ghcPackageOverrides;
}

Combining `head.hackage` modifications with local modifications

Unfortunately, this is not quite enough as it is sometimes necessary to further modify packages in order to get them to compile. The most obvious way is that a lot of packages have restrictive version bounds but compile fine with newer versions of libraries. So, in order to get around this we preemptively need to jailbreak all packages.

Jailbreaking is a process which removes all version bounds from a package. This is safe to do as each package set only contains one version of each dependency anyway.

We introduce a new extension function localOverrides which will contain our specific modifications which we don’t want to include in head.hackage. We will first only contain one specific modification which chooses a new version of haskell-src-exts.

self: super: {
  haskellPatches = self.callPackage <path-to-overrides.nix>

              { patches = <path-to-patch-dir>; };

  new-ghcHEAD =
    let ghcPackageOverrides = self.callPackage self.haskellPatches {};
        localOverrides      =
          sel: sup:
            { haskell-src-exts
                = sel.callHackage "haskell-src-exts" "1.20.1" {};
            };
    in self.haskell.packages.ghcHEAD.extend
        (self.lib.composeExtensions localOverrides ghcPackageOverrides);
}

We need to use composeExtensions so that the attributes we locally modify are then also modified by head.hackage incrementally.

Modifying `mkDerivation`

Finally, we will modify mkDerivation in order to jailbreak packages and not run haddock in order to speed up the builds. mkDerivation is the function which takes a Haskell package description and turns it into a nix derivation. It’s the function which when given the what to build, says precisely how to build it.

self: super: {
  haskellPatches = self.callPackage <path-to-overrides.nix>
              { patches = <path-to-patch-dir>; };

  new-ghcHEAD =
    let ghcPackageOverrides = self.callPackage self.haskellPatches {};
        localOverrides      =
          sel: sup: {
            haskell-src-exts
              = sel.callHackage "haskell-src-exts" "1.20.1" {}
            mkDerivation
              = drv: sup.mkDerivation (drv //
                  { jailbreak = true; doHaddock = false;});
          };

    in self.haskell.packages.ghcHEAD.extend
        (self.lib.composeExtensions localOverrides ghcPackageOverrides);
}

Testing

We can now test our new attribute using nix-shell. For example, let’s try to provision a version of the compiler with primitive available.

> nix-shell -p "new-ghcHEAD.ghcWithPackages (ps: [ps.primitive])"
...

Success!

Conclusion

Thanks to Csongor Kiss, Alessio Zakaria, Sarah Brofeldt and Vaibhav Sagar for comments on a draft.

Appendix

Addendum: What do I do if a patch from `head.hackage` doesn’t apply?

There are some situations where patches from head.hackage fail to apply properly. The usual cause is that patches in the head.hackage repo apply to unmodified tarballs but the nix build process tries to apply patches after patching the cabal file to take into account package revisions.

The workaround for this is to just create a new nix-specific patch in your local directory. 🙂

Addendum: How do I make a patch to contribute to `head.hackage`?

cabal get haskell-src-exts-1.20.1 <--pristine>
cd haskell-src-exts-1.20.1
git init
git add .
<Make the simple change>
git diff > <patchDir>/haskell-src-exts-1.20.1.patch

Use the --pristine flag if you intend to contribute the patch upstream! This will fetch the package without applying cabal file revisions.

Addendum: How does `overrides.nix` work?

overrides.nix is a simple derivation which packages a single haskell script. The haskell script scans a directory and generates the correct overrides for the given patches. The wrapper script then stores the overrides and copies the patch files into the nix store.

{ stdenv
  , haskell
  , patches  # A directory containing patch files used to build packages
             # it can either be a local directory or fetched from the web
}:
let
  ghc = haskell.packages.ghc822.ghcWithPackages (ps: with ps;
          [ ]);
in

stdenv.mkDerivation {

  name = "hs-generate-overrides-0.1";

  src = ./generate-nix-overrides.hs;

  preUnpack = ''mkdir hs-generate-overrides'';
  buildInputs = [ ghc ];

  unpackCmd = ''
    cp $curSrc ./hs-generate-overrides
    cp -r ${patches} ./hs-generate-overrides/patches
    sourceRoot=hs-generate-overrides;
  '';

  buildPhase = ''
    ghc $src -o generate
    ./generate $script/patches patches > patches.nix
  '';

  outputs = ["out" "script"];

  installPhase = ''
    cp patches.nix $out
    ensureDir $script/patches
    cp -r patches $script/patches
  '';

}

Addendum: How is jailbreaking different to cabal’s `--allow-newer`?

The goal of jailbreaking and --allow-newer are the same, to remove restrictive bounds on packages.

Jailbreaking achieves this by parsing and rewriting the cabal file to remove nearly all bounds. This fits better into the nix ecosystem where there is exactly one version of each package anyway so additional version bounds are irrelevant.

--allow-newer influences the solver in order to pick a package set as if the package didn’t have any upper bounds.

Addendum: Why are you using `extend` rather than `override`.

override is a general nix mechanism to override arguments to a function. In the haskell package set case, one of the arguments to the package set is called overrides, which is a function explaining how we want to modify the package set further.

However, there is one big downside to using this mechanism, it is not possible to repeatedly override the package set with more and more extensions as each time you call override you replace the argument.

For more explanation of the difference see this issue on the nixpkgs issue tracker.

Addendum: What to do if `callHackage` doesn’t find the latest version of a package

If callHackage doesn’t find a package which you know is on hackage then it is because the all-cabal-hashes attribute is too old. In order to override it, you can simply add a new line to your overlay with an updated copy.

For example, this version is from 2018-01-05:

all-cabal-hashes = super.fetchurl {
  url    = "https://github.com/commercialhaskell/all-cabal-hashes/archive/e1089e56e666c2a0fe82f840d3cc7f49b9c9fe9b.tar.gz";
  sha256 = "0qbzdngm4q8cmwydnrg7jvipw39nb1mjxw95vw6f789874002kn1";
};

Addendum: How to update the ghcHEAD derivation

If you want to use a more recent version of HEAD then you have to update the ghcHEAD derivation. It can be found at development/compilers/ghc/head.nix.

You then need to modify three things.

Correct the version of the compiler by modifying the version argument.
Set rev to the git commit identifier of the revision you want to build.
Correct the sha256 attribute by running nix-prefetch-git and copying the resulting hash. nix-shell -p nix-prefetch-git --run "nix-prefetch-git --fetch-submodules git://git.haskell.org/ghc.git <rev>"

Using a development version of GHC with nix

A simple overlay

Extending a package set

More extensions

Using a local copy of `head.hackage`

Using a remote copy of `head.hackage`

Combining `head.hackage` modifications with local modifications

Modifying `mkDerivation`

Testing

Conclusion

Appendix

Addendum: What do I do if a patch from `head.hackage` doesn’t apply?

Addendum: How do I make a patch to contribute to `head.hackage`?

Addendum: How does `overrides.nix` work?

Addendum: How is jailbreaking different to cabal’s `--allow-newer`?

Addendum: Why are you using `extend` rather than `override`.

Addendum: What to do if `callHackage` doesn’t find the latest version of a package

Addendum: How to update the ghcHEAD derivation

You always gain by giving love

Privacy is power. What people don’t know, they can’t ruin.

Don’t take criticism from people you wouldn’t take advice from

NO pressure, NO diamonds

A simple overlay

Extending a package set

More extensions

Using a local copy of head.hackage

Using a remote copy of head.hackage

Combining head.hackage modifications with local modifications

Modifying mkDerivation

Testing

Conclusion

Related Links

Appendix

Addendum: What do I do if a patch from head.hackage doesn’t apply?

Addendum: How do I make a patch to contribute to head.hackage?

Addendum: How does overrides.nix work?

Addendum: How is jailbreaking different to cabal’s --allow-newer?

Addendum: Why are you using extend rather than override.

Addendum: What to do if callHackage doesn’t find the latest version of a package

Addendum: How to update the ghcHEAD derivation

You always gain by giving love

Privacy is power. What people don’t know, they can’t ruin.

Don’t take criticism from people you wouldn’t take advice from

NO pressure, NO diamonds

Tags

Using a local copy of `head.hackage`

Using a remote copy of `head.hackage`

Combining `head.hackage` modifications with local modifications

Modifying `mkDerivation`

Addendum: What do I do if a patch from `head.hackage` doesn’t apply?

Addendum: How do I make a patch to contribute to `head.hackage`?

Addendum: How does `overrides.nix` work?

Addendum: How is jailbreaking different to cabal’s `--allow-newer`?

Addendum: Why are you using `extend` rather than `override`.

Addendum: What to do if `callHackage` doesn’t find the latest version of a package