~totaltrash totaltrash.xyz Sun, 06 Oct 2019 00:00:00 UT Nix, Rust, Python totaltrash.xyz/blog/nix-rust-python.html

Contents

I have lately been working on coupled cluster Monte Carlo.1 Our pilot code is pure Python: easy to write, but the performance is not optimal. We are thus in the process of rewriting the most computationally intensive parts of the code in a compiled language.

Originally, that compiled language was going to be C++. pybind11 indeed works perfectly for writing Python extensions. Then I learned about Rust and decided to try that instead:

  1. I thought it would be a good idea to learn a new language;
  2. The memory safety guarantees and their potential benefits for parallel and concurrent programming are very enthusing;
  3. There is not runtime library apart from GLIBC;
  4. The tooling is excellent: unobtrusive and shipped with the language.

Both Python and Rust can talk to C, so that’s ultimately what we’ll have to do to get our extension to work properly. Fortunately, we don’t have to go all the way to C. There are 2 tools that can help in writing Python extensions in Rust:

  • Using CFFI. Slightly lower level and it can involve a lot of boilerplate.
  • Using PyO3. Aims at doing what pybind11 (and Boost.Python before it) did for C++ extensions. It is more bleeding edge and what I chose to work with.

This presentation has more information on writing Python extensions in Rust.

In this post I’ll explain:

  1. How to set up a mixed-language development environment.
  2. How I set up my mixed-language development environment using Nix, Pipenv, and direnv.
  3. How to deploy a manylinux1-compatible package using Travis CI for tagged releases.

The public example repository in on GitHub: [robertodr/rustafarian**

Development environment

Rust dependencies

These are for developers only. First and foremost, we need the Rust toolchain. Easy enough with rustup:

Other dependencies will be specified in the Cargo.toml file and installed when we build the extension.

Python dependencies

These are both for final users (e.g., Click for the command-line interface) and for developers (e.g., pytest for testing). Preferably one should only install Python dependencies in virtual environments, never globally. I am partial towards Pipenv, but [Conda] could work as well.

Running pipenv install --dev with the following Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
maturin = ">=0.7.6"
pytest = ">=4.0"

[packages]
click = ">=7.0"

will create the virtual environment and install all needed packages. With pipenv shell, you can jump in the virtual environment. The maturin package is essential: this is the tool that will orchestrate building the extension and packaging for PyPI.

Power-up using Nix and direnv

Nix and direnv help keep your development environment tidy and reproducible, but special care needs to be taken when interacting with Python. In addition to Pipfile and Cargo.toml, we also have:

  • .envrc, used by direnv to set up the local environment:

I am using this version of the use_nix() function of direnv.

  • shell.nix, used by nix-shell and setting up the Rust toolchain and the Pipenv shell.

It is particularly convenient to use a Nix shell to not propagate per-developer dependencies. I use Emacs with language server protocol (LSP). The LSP for Python is provided by some extra packages that are thus my “private” development dependencies. These can be specified in the shell.nix file.

A closer look at shell.nix

This is a bit involved, so I’ll describe it bit by bit.

Pinning nixpkgs and local overlays

Here I pin a specific version of the nixpkgs repository and declare an override for the python-language-server package in an overlay.

with import (builtins.fetchGit {
  name = "nixos-19.03";
  url = "https://github.com/NixOS/nixpkgs-channels";
  ref = "nixos-19.03";
  # Commit hash for nixos-19.03 as of 2019-08-18
  # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-19.03`
  rev = "67135fbcc5d5d28390c127ef519b09a362ef2466";
}) {
  overlays = [(self: super:
    {
      python3 = super.python3.override {
        packageOverrides = py-self: py-super: {
          python-language-server = py-super.python-language-server.override {
            providers = [
              "rope"
              "pyflakes"
              "mccabe"
              "pycodestyle"
              "pydocstyle"
            ];
          };
        };
      };
    }
  )];
};
Rust from the Mozilla overlay

We need a nightly build of Rust, which we can get from the Mozilla Nix overlay.

let src = fetchFromGitHub {
  owner = "mozilla";
  repo = "nixpkgs-mozilla";
  # commit from: 2019-09-04
  rev = "b52a8b7de89b1fac49302cbaffd4caed4551515f";
  sha256 = "1np4fmcrg6kwlmairyacvhprqixrk7x9h89k813safnlgbgqwrqb";
};
in
with import "${src.out}/rust-overlay.nix" pkgs pkgs;
Declare the shell

We separate the packages into nativeBuildInputs and buildInputs. The latter are used to specify the per-developer private dependencies, which might include any libraries needed to compile extensions of Python dependencies (e.g. freetype for matplotlib). The shellHook sets up the Pipenv shell to cooperate with the Nix shell.

mkShell {
  name = "rustafarian";
  nativeBuildInputs = [
    # Note: to use stable, just replace nightly with stable
    latest.rustChannels.nightly.rust

    # Build-time Additional Dependencies
    #pkgconfig
  ];

  # Run-time Additional Dependencies
  buildInputs = [
    pipenv

    python3

    # System libraries needed for Python packages
    #freetype # Demanded by matplotlib

    # Development tools
    lldb
    python3.pkgs.black
    python3.pkgs.epc
    python3.pkgs.importmagic
    python3.pkgs.isort
    python3.pkgs.jedi
    python3.pkgs.mypy
    python3.pkgs.pyls-black
    python3.pkgs.pyls-isort
    python3.pkgs.pyls-mypy
    python3.pkgs.python-language-server
    travis
  ];

  # Set Environment Variables
  RUST_BACKTRACE = 1;
  shellHook = ''
    SOURCE_DATE_EPOCH=$(date +%s) # required for python wheels

    local venv=$(pipenv --bare --venv &>> /dev/null)

    if [[ -z $venv || ! -d $venv ]]; then
      pipenv install --dev &>> /dev/null
    fi

    export VIRTUAL_ENV=$(pipenv --venv)
    export PIPENV_ACTIVE=1
    export PYTHONPATH="$VIRTUAL_ENV/${python3.sitePackages}:$PYTHONPATH"
    export PATH="$VIRTUAL_ENV/bin:$PATH"
  '';
}

Workflow

OK, now we have a working development environment! To build the extension:

Yes, that’s it. maturin will compile the module and link it to a dynamic shared object (DSO) alongside the Python code. We can thus import it and use it in an interactive Python session spawned in our development environment. The Cargo.toml lists the Rust dependencies of our project and these will be automatically downloaded and compiled if not already available.

Running ldd on the DSO shows that there is no Rust runtime to link to:

Packaging and deployment

It is now time to distribute this little Rust+Python package by deploying it to PyPI. PEP 517 introduced the pyproject.toml file to specify build systems for Python packages. Ours will specify maturin as our build tool and thus look like:

[build-system]
requires = ["maturin"]
build-backend = "maturin"

As you can see, there is no mention of user dependencies (Click, in our case) in this file. Nor of console scripts to be installed alongside the module. This information is contained in the Cargo.toml:

[package.metadata.maturin]
requires-python = ">=3.6"
requires-dist = ["click>=7.0"]
scripts = {rustafarian = "rustafarian.cli:cli"}
classifier = ["Programming Language :: Python"]

Automated deploy to PyPI with Travis CI

maturin has a publish CLI switch to upload a package to PyPI. The process can be automatised: whenever a new tag is pushed to GitHub, Travis will run a CI job. If successful, it will deploy the artifact to PyPI:

The deployment script contains the following:

where the MATURIN_PASSWORD is the password to the PyPI account and is encrypted via the Travis CI web UI. We are building in a Docker container to ensure compliance with the manylinux1 policy.

References

(1) Scott, C. J. C.; Di Remigio, R.; Crawford, T. D.; Thom, A. J. W. Diagrammatic Coupled Cluster Monte Carlo. J. Phys. Chem. Lett. 2019, 10 (5), 925–935. https://doi.org/10.1021/acs.jpclett.9b00067.

]]> Sun, 06 Oct 2019 00:00:00 UT totaltrash.xyz/blog/nix-rust-python.html Roberto Di Remigio Plots with Altair totaltrash.xyz/blog/plots-with-altair.html

Contents

Altair

References

]]> Sat, 09 Mar 2019 00:00:00 UT totaltrash.xyz/blog/plots-with-altair.html Roberto Di Remigio The \(n\)-th time is the charm totaltrash.xyz/blog/the-n-th-time-is-the-charm.html

Contents

What do you know?

\[ \ln x = \int_{-\infty}^x \frac 1 y \, dy \]

With plots:

import matplotlib.pyplot as plt

plt.figure()
plt.plot([0,1,2,3,4], [1,2,3,4,5])
plt.title('This is an example figure')

And references even!1

TODOs:

  • Read the Pijul SHA to generate the footer correctly. It is probably possible to adapt this: githash
  • Refurbish the template to include links to other platforms (Twitter etc.)
  • Generate CV as PDF and HTML
  • Clean up the Haskell source, also looking at this The code is discussed here I want to be able to combine with other compilers without triple somersaults:
  • Try to create a pandoc-bokeh filter based on pandoc-pyplot.

References

(1) Thom, A. J. W. Stochastic Coupled Cluster Theory. Phys. Rev. Lett. 2010, 105 (26), 263004. https://doi.org/10.1103/PhysRevLett.105.263004.

]]> Sat, 05 Jan 2019 00:00:00 UT totaltrash.xyz/blog/the-n-th-time-is-the-charm.html Roberto Di Remigio Redesigning Getkw _or_ reinventing the wheel: 0. State of affairs and requirements totaltrash.xyz/blog/getkw-01.html ]]> Fri, 14 Sep 2018 00:00:00 UT totaltrash.xyz/blog/getkw-01.html Roberto Di Remigio Docker Images for Fun and for Profit totaltrash.xyz/blog/docker-images.html ]]> Thu, 03 May 2018 00:00:00 UT totaltrash.xyz/blog/docker-images.html Roberto Di Remigio Installing NixOS totaltrash.xyz/blog/install-nixos.html

Contents

{% include toc %}

Get NixOS on the USB stick

Formatting the USB stick: 1. Find out the device:

  1. Unmount
  1. Format to FAT32
  1. Detach and reattach
  2. Copy NixOS on the USB

Install NixOS

My laptop is a Lenovo Thinkpad T440s and I’ve followed (mostly) the set of instructions on Chris Martin’s blog.

Booting and partitioning

First of all, I had to switch to only using UEFI in the boot menu and then boot from the USB stick. I wanted to wipe the whole disk and reinstall NixOS on an encrypted partition. The NixOS installer does not (yet!) have a graphical user interface for partitioning, but what I wanted to do is fairly straightforward to achieve using fdisk. I followed the advice posted on StackOverflow1 and ended up creating the following partition table:

The LVM device will contain the OS and will be partitioned into the root folder / and the swap.

Setting up the encryption, LVM groups and volumes

Let’s move on to encrypting the device where the OS will actually live, i.e. /dev/sda3. I am here following the instructions reported on the NixOS installation instructions. The first command encrypts /dev/sda3 and will prompt you to insert a password, the second command opens the encrypted partition:

OK, great! Time to make space for / and the swap:

and to format:

At this point, launching fdisk -l you will see the following output (or at least I do!):

Get the installation going!

Let’s mount the disks and volumes to /mnt:

and activate the swap:

NixOS declaratively defines the configuration of the system. The starting point is to run the following:

that generates hardware-configuration.nix (do not touch) and configuration.nix in the directory /mnt/etc/nixos. My configuration.nix is available on GitHub and I can just clone the repo with and populate the /mnt/etc/nixos directory with the contents of the repo. This will not only configure the system as I like, but also install all the software that I need in one go. Once the configuration.nix is all set run:

and reboot when the process is done.

First login

Login as root user when the system reboots and add the 17.03 channel:

Add whatever you need to configuration.nix and rebuild your system:

Moreover, the user that was created in the configuration.nix file does not have a password set! Let’s do it now:

You can now login as a normal user, instead as root. The system is now ready for use!

  1. Thanks to Radovan Bast for the tip.↩

]]> Tue, 11 Jul 2017 00:00:00 UT totaltrash.xyz/blog/install-nixos.html Roberto Di Remigio Turn On Compiler Warnings! totaltrash.xyz/blog/turn-on-compiler-warnings.html

Contents

{% include toc %}

This post originally appeared on DevCafe

Compiler warnings can be a nuisance, clogging the output with unnecessarily verbose information. This is especially true of C++, even more so when the code uses template magic. However, turning them off can be rather harmful.

TL;DR

Set -Wall -Wextra, or equivalent, as your basic compiler flags, both in debug and in release mode. If you are using CMake, the Autocmake project does it by default

An example from the cnpy library

Consider the following C++11 example, but it applies equally well to earlier standards. The cnpy library for saving/loading C++ arrays to/from NumPy binary format has a basic data structure, called NpyArray. An object of this type is constructed by supplying:

  • the shape of the array, i.e. a std::vector<size_t>. For example, a 2-dimensional array with dimensions 10 and 11 will have: std::vector<size_t> shape({10, 11});
  • the word size of the data type to be dumped to NumPy array. This is either read from the .npy when loading the array, or determined by the result of sizeof(T), where T is the data type, when saving the array.

The constructor will then compute how large a memory buffer is needed and allocate a std::vector<char>. The number of values in the array is computed from the shape array:

or more compactly using std::accumulate:

The type information is encoded in the .npy file format header. When loading the array the user will have to perform a reinterpret_cast to get the correct data type.

Runtime error!

Stripped to its barebones, the NpyArray class looks like this:

Let’s now try to compile it:

The live example on Coliru shows that we get a runtime error because the assertion in the constructor fails.

What’s happening? Well, a very, very stupid mistake. The buffer_ data member is initialized using the nValues_ data member This shouldn’t be a problem, since it’s initialized first in the constructor, right? Wrong! According to the standard, 12.6.2 Section 13.3 data members are initialized in the order they were declared in the class. Thus buffer_ gets initialized first, using an undefined value for nValues_.

Fixing it

The correct struct declaration is thus:

which also honors the tenet of ordering data members in your classes and structs by their size in memory. However, this is something very easily forgotten. How to avoid these kinds of errors?

  1. Do not initialize data members based on other data members. This is, in my opinion, overly restrictive.
  2. Insert assertions in the constructors. Very useful, but assertions only work when -DNDEBUG is not given to the compiler. Most of the times this is not the case when compiling with optimization.
  3. Turn on compiler warnings. -Wall catches this mistake and many others. For an extra layer of warnings, I also turn on -Wextra. This is the output on Coliru
]]> Sat, 08 Jul 2017 00:00:00 UT totaltrash.xyz/blog/turn-on-compiler-warnings.html Roberto Di Remigio Turn On Compiler Warnings! totaltrash.xyz/blog/turn-on-compiler-warnings.html ]]> Sun, 04 Jun 2017 00:00:00 UT totaltrash.xyz/blog/turn-on-compiler-warnings.html Roberto Di Remigio Song of the Week, 10 totaltrash.xyz/blog/sow10_2017.html

I found this out listening to a podcast on MixCloud. The podcast was put together by DJ Wooden Wisdom (Elijah Wood & Zach Cowie) for The Vinyl Factory. Here is the tracklist.

]]> Fri, 10 Mar 2017 00:00:00 UT totaltrash.xyz/blog/sow10_2017.html Roberto Di Remigio Branching Models totaltrash.xyz/blog/branching-models.html ]]> Wed, 01 Mar 2017 00:00:00 UT totaltrash.xyz/blog/branching-models.html Roberto Di Remigio Song of the Week, 21 totaltrash.xyz/blog/sow21_2016.html ]]> Fri, 27 May 2016 00:00:00 UT totaltrash.xyz/blog/sow21_2016.html Roberto Di Remigio Song of the Week, 20 totaltrash.xyz/blog/sow20_2016.html

Leave Me ALone

And a great cover:

]]> Thu, 19 May 2016 00:00:00 UT totaltrash.xyz/blog/sow20_2016.html Roberto Di Remigio Welcome totaltrash.xyz/blog/welcome.html ]]> Wed, 27 Apr 2016 00:00:00 UT totaltrash.xyz/blog/welcome.html Roberto Di Remigio