Experimental Local Overlay Store Issue in k8s pod

[yaroslavkasatikov]

Hello team,

I found interesting article https://blog.replit.com/super-colliding-nix-stores and read https://nix.dev/manual/nix/2.22/store/types/experimental-local-overlay-store#experimental-local-overlay-store.

Our goal is building the system based on local-overlay-store which can work in non-privileged pod (we use OKD and pod should been running under the restricted-scc).

So, I have been inspired your idea to have two stores (big with many packages and local).
My idea is having nfsd server in privileged pod, mount OverlayFS there and export nfs share for merged, upper and lower and mount them into non-privileged nix pod.

Well, but until running this idea I tried to create a privileged pod and run my first overlay-fs store test, but I was failed to start it (or maybe I understood something incorrect (I'm newbie in nix).

Here my steps for reproducing.

1. I have created two empty pvc (for upper and lower) and mounted them into my pod. So it will be something like this:

apiVersion: v1
kind: Pod
metadata:
  name: nix-test-p
spec:
  containers:
  - name: volume-test
    env:
    - name: HOME
      value: /nix/home
    image: nixos/nix
    command:
    - tail
    - -f
    - /dev/null
    imagePullPolicy: IfNotPresent
    securityContext:
      privileged: true
    volumeMounts:
    - name: pvc-upper
      mountPath: /mnt/example/
    - name: pvc-lower
      mountPath: /mnt/example/store-a
    - name: nix
      mountPath: /etc/nix/nix.conf
      subPath: nix.conf
    ports:
    - containerPort: 80
  volumes:
  - name: pvc-lower
    persistentVolumeClaim:
      claimName: lower-p
  - name: pvc-upper
    persistentVolumeClaim:
      claimName: upper-p
  - name: nix
    configMap:
      name: nix

I have mounted nix.conf via the configmap with this content:

apiVersion: v1
data:
  nix.conf: |+
    trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
    experimental-features = nix-command flakes local-overlay-store read-only-local-store
    store = local-overlay://?root=/mnt/example/merged-store&lower-store=/mnt/example/store-a&upper-layer=/mnt/example/store-b&check-mount=false

kind: ConfigMap
metadata:
  name: nix

So I have taken fully example from the docs with the path.

First my surprise was the absence of mount command :

sh-5.2# mount
sh: mount: command not found

Okay, I have installed it with nix-env -iA nixpkgs.mount and it installed mount into /mnt/example/merged-store/nix/store/ (which is not mounted yet), but okay, I have mount now..
So the question 1 - how can I install mount in correct way if loading nix container with overlay-fs store for initial mounting?

btw, I haven't found nothing in $HOME/.nix-profiles/bin related to mount and found it with 'find' util:

sh-5.2# find / -name mount
/mnt/example/merged-store/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount
/mnt/example/merged-store/nix/store/ibgzqljiski2y3fqvg8fnj2g6bjyhqly-util-linux-2.39.4-mount/bin/mount
/mnt/example/merged-store/nix/store/2wvb4326f069mz8zan43yx6nak6lsjqk-util-linux-2.39.4-bin/share/bash-completion/completions/mount
/mnt/example/merged-store/nix/store/2wvb4326f069mz8zan43yx6nak6lsjqk-util-linux-2.39.4-bin/bin/mount
/mnt/example/merged-store/nix/store/lnk0ky82i3rqvxvi3qknky80qffldyi9-mount-util-linux-2.39.4/bin/mount

well, I have mounted overlay fs:

sh-5.2# /mnt/example/merged-store/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount -t overlay overlay   -o lowerdir=/mnt/example/store-a/nix/store   -o upperdir=/mnt/example/store-b   -o workdir=/mnt/example/workdir   /mnt/example/merged-store/nix/store
sh-5.2# ls /mnt/example/merged-store/nix/store

Ok, seems I can install packages.. btw, I mounted overlayfs according the commands from the manual, but it doesn't work without check-mount=false

So ok, I have installed packages:

sh-5.2# nix-env -iA nixpkgs.vim
installing 'vim-9.1.0377'
these 3 paths will be fetched (8.79 MiB download, 42.94 MiB unpacked):
  /nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2
  /nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377
  /nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd
copying path '/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/s83pcskzqdz78qwkpydcm33gm40ckfcx-user-environment.drv'...
sh-5.2# nix-env -iA nixpkgs.nano
installing 'nano-8.0'
these 3 paths will be fetched (0.87 MiB download, 10.89 MiB unpacked):
  /nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45
  /nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0
  /nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info
copying path '/nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/rffg19w0n2irxvfmw8paw6g8bx6hzlm0-user-environment.drv'...

all of them has been installed into correct path:

sh-5.2# find / -name vim
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/applications/editors/vim
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/test/vim
/mnt/example/store-b/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/share/vim
/mnt/example/store-b/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/bin/vim
/mnt/example/store-b/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/share/vim
/mnt/example/store-b/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/bin/vim
/mnt/example/merged-store/nix/store/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/share/vim
/mnt/example/merged-store/nix/store/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/bin/vim
/mnt/example/merged-store/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/share/vim
/mnt/example/merged-store/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/bin/vim
sh-5.2# find / -name nano
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/applications/editors/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/doc/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/bin/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/doc/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/bin/nano

Well next I want to prepare lower storage. I plan to umount overlayfs and copy all data from upper to lower storage, cleanup database/upper and try to install data from lower-store. So I expect it won't download packages from the Network but will reuse them from lower storage.

So:

install mount again:

sh-5.2# nix-env -iA nixpkgs.mount
installing 'mount-util-linux-2.39.4'
....
sh-5.2# /mnt/example/store-b/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/umount   /mnt/example/merged-store/nix/store
sh-5.2# cp -a /mnt/example/store-b/* /mnt/example/store-a/nix/store/

found strange: database on lower layer is updated:

sh-5.2# ls -la ./mnt/example/store-a/nix/var/nix/db
total 8280
drwxr-xr-x 2 root root    4096 Nov  8 15:29 .
drwxr-xr-x 6 root root    4096 Nov  8 15:29 ..
-rw------- 1 root root       0 Nov  8 15:29 big-lock
-rw-r--r-- 1 root root   45056 Nov  8 15:29 db.sqlite
-rw-r--r-- 1 root root   32768 Nov  8 15:39 db.sqlite-shm
-rw-r--r-- 1 root root       0 Nov  8 15:29 db.sqlite-wal
-rw------- 1 root root 8388608 Nov  8 15:29 reserved
-rw-r--r-- 1 root root       2 Nov  8 15:29 schema

It is not under overlayfs - just folder on dist, but it's a strange, I thought this db should not be updated or I'm thinking incorrect?

But okay, I have removed everything from upper layer

sh-5.2# ls -la /mnt/example/store-b/
total 160
drwxrwxr-t 2 root nixbld 159744 Nov  8 15:42 .
drwxr-xr-x 7 root root     4096 Nov  8 15:20 ..
sh-5.2# ls -la /mnt/example/workdir/
total 12
drwxr-xr-x 3 root root 4096 Nov  8 15:20 .
drwxr-xr-x 7 root root 4096 Nov  8 15:20 ..
d--------- 2 root root 4096 Nov  8 15:20 work
sh-5.2# ls -la /mnt/example/workdir/work/
total 8
d--------- 2 root root 4096 Nov  8 15:20 .
drwxr-xr-x 3 root root 4096 Nov  8 15:20 ..

Remount overlayfs and trying to reinstall pacakges which stored on lower layer:

sh-5.2# /mnt/example/store-a/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount -t overlay overlay   -o lowerdir=/mnt/example/store-a/nix/store   -o upperdir=/mnt/example/store-b   -o workdir=/mnt/example/workdir   /mnt/example/merged-store/nix/store
sh-5.2# nix-env -iA nixpkgs.nano
installing 'nano-8.0'
these 9 paths will be fetched (8.13 MiB download, 45.52 MiB unpacked):
  /nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45
  /nix/store/87848rvrg5c7jmplpi0iapvbxyj9kfid-glibc-2.39-52
  /nix/store/ya4arqpf6vwbq4msi449314kqbhdb4l4-libidn2-2.3.7
  /nix/store/503h8rdwzj12dh0fs1b6y643qm26vqsb-libunistring-1.1
  /nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0
  /nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info
  /nix/store/qdbfy1ndpz2n868b8y1y3nlfsixgz106-ncurses-6.4
  /nix/store/19avl2s0rd4whz3b0n118qdrv6yr3b0g-xgcc-13.2.0-libgcc
  /nix/store/2k9k3q1vk8z6w7743k6nb22vnb05xv06-zlib-1.3.1
copying path '/nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/19avl2s0rd4whz3b0n118qdrv6yr3b0g-xgcc-13.2.0-libgcc' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/503h8rdwzj12dh0fs1b6y643qm26vqsb-libunistring-1.1' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/ya4arqpf6vwbq4msi449314kqbhdb4l4-libidn2-2.3.7' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/87848rvrg5c7jmplpi0iapvbxyj9kfid-glibc-2.39-52' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/qdbfy1ndpz2n868b8y1y3nlfsixgz106-ncurses-6.4' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/2k9k3q1vk8z6w7743k6nb22vnb05xv06-zlib-1.3.1' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/rffg19w0n2irxvfmw8paw6g8bx6hzlm0-user-environment.drv'...
sh-5.2# nix-env -iA nixpkgs.vim
installing 'vim-9.1.0377'
these 4 paths will be fetched (9.21 MiB download, 44.48 MiB unpacked):
  /nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26
  /nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2
  /nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377
  /nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd
copying path '/nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/s83pcskzqdz78qwkpydcm33gm40ckfcx-user-environment.drv'...

And nix downloads them from Internet, but I expected them will be reused from lower-layer.

So , could you please tell me when I did a mistake? Unfortunately there are few docs about overlay-local-store, but this seems to be a killer feature.

Thank you very much.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/super-colliding-nix-stores/28462/30

[yaroslavkasatikov]

Hey @tomberek !
Thank you for assigning ticket.
Would it be possible to take a quick look? May be it's the easiest issue and I have done the known mistake. Thank you in advance and sorry for ping :)