Nix, Rust, Python

This post continues from where I left off in my Rust+Python adventures.

In this post I’ll explain:

How I set up my mixed-language development environment using Nix, Pipenv, and direnv.

The public example repository in on GitHub: robertodr/rustafarian

Power-up the development environment using Nix and direnv

Nix and direnv help keep your development environment tidy and reproducible, but special care needs to be taken when interacting with Python. In addition to Pipfile and Cargo.toml, we also have:

• .envrc, used by direnv to set up the local environment:

use nix -s shell.nix -w Pipfile

I am using this version of the use_nix() function of direnv.

• shell.nix, used by nix-shell and setting up the Rust toolchain and the Pipenv shell.

It is particularly convenient to use a Nix shell to not propagate per-developer dependencies. I use Emacs with language server protocol (LSP). The LSP for Python is provided by some extra packages that are thus my “private” development dependencies. These can be specified in the shell.nix file.

A closer look at shell.nix

I’ll describe the shell.nix file bit by bit.

Pinning nixpkgs and local overlays

Here I pin a specific version of the nixpkgs repository and declare an override for the python-language-server package in an overlay.

with import (builtins.fetchGit {
  name = "nixos-19.09-2019-10-10";
  url = "https://github.com/NixOS/nixpkgs-channels";
  ref = "nixos-19.09";
  # Commit hash for nixos-19.09 as of 2019-10-10
  # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-19.09`
  rev = "9bbad4c6254513fa62684da57886c4f988a92092";
}) {
  overlays = [(self: super:
    {
      python3 = super.python3.override {
        packageOverrides = py-self: py-super: {
          python-language-server = py-super.python-language-server.override {
            providers = [
              "rope"
              "pyflakes"
              "mccabe"
              "pycodestyle"
              "pydocstyle"
            ];
          };
        };
      };
    }
  )];
};

Rust from the Mozilla overlay

We need a nightly build of Rust, which we can get from the Mozilla Nix overlay.

let src = fetchFromGitHub {
  owner = "mozilla";
  repo = "nixpkgs-mozilla";
  # Commit hash for master as of 2019-10-10
  # `git ls-remote https://github.com/mozilla/nixpkgs-mozilla master`
  rev = "d46240e8755d91bc36c0c38621af72bf5c489e13";
  sha256 = "0icws1cbdscic8s8lx292chvh3fkkbjp571j89lmmha7vl2n71jg";
};
in
with import "${src.out}/rust-overlay.nix" pkgs pkgs;

Declare the shell

We separate the packages into nativeBuildInputs and buildInputs. The former specify the Rust version and extensions to install. Furthermore, any additional non-Rust packages upon which crates might depend, should also be listed here. The latter are used to specify the per-developer private dependencies, which might include any libraries needed to compile extensions of Python dependencies (e.g. freetype for matplotlib). The shellHook sets up the Pipenv shell to cooperate with the Nix shell.

mkShell {
  name = "rustafarian";
  nativeBuildInputs = [
    # Note: to use stable, just replace `nightly` with `stable`
    #latest.rustChannels.nightly.rust
   ((rustChannelOf
     {
       date = "2019-08-01";
       channel = "nightly";
     }).rust.override {
       extensions = [
         "rls-preview"
         "rust-analysis"
         "rustfmt-preview"
       ];
     })

    # Build-time Additional Dependencies
    #pkgconfig
  ];

  # Run-time Additional Dependencies
  buildInputs = [
    pipenv

    python3

    # System libraries needed for Python packages
    #freetype # Demanded by matplotlib

    # Development tools
    lldb
    python3.pkgs.black
    python3.pkgs.epc
    python3.pkgs.importmagic
    python3.pkgs.isort
    python3.pkgs.jedi
    python3.pkgs.mypy
    python3.pkgs.pyls-black
    python3.pkgs.pyls-isort
    python3.pkgs.pyls-mypy
    python3.pkgs.python-language-server
    travis
  ];

  # Set Environment Variables
  RUST_BACKTRACE = 1;
  shellHook = ''
    SOURCE_DATE_EPOCH=$(date +%s) # required for python wheels

    local venv=$(pipenv --bare --venv &>> /dev/null)

    if [[ -z $venv || ! -d $venv ]]; then
      pipenv install --dev &>> /dev/null
    fi

    export VIRTUAL_ENV=$(pipenv --venv)
    export PIPENV_ACTIVE=1
    export PYTHONPATH="$VIRTUAL_ENV/${python3.sitePackages}:$PYTHONPATH"
    export PATH="$VIRTUAL_ENV/bin:$PATH"
  '';
}

Profit!

It is now time to allow direnv to do its thing:

$ direnv allow

this will take quite some time: Nix will download and install dependencies and then call Pipenv to set up the virtual environment. Since Nix is not manylinux1-compatible, the last step might require compiling some Python packages (e.g. pandas) from source. Sit tight!

When the environment set up is done, the development workflow will be unchanged. As a bonus, every time you jump into the project’s folder, all dependencies will be on PATH!

Acknowledgments

Thanks for @__radovan for reading and commenting on an early draft.

References